Linux 下多路径 multipath 配置
一、multipath 在 linux 中的根本配置：
- 通过命令：lsmod |grep dm_multipath 查看是否失常装置胜利。如果没有输入阐明没有装置那么通过 yum 性能装置一下软件包：yum –y install device-mapper device-mapper-multipath
接着通过命令：multipath –ll 查看多路径状态查看模块是否加载胜利
[root@liujing ~]# multipath –ll 查看多路径状态
Mar 10 19:18:28 | /etc/multipath.conf does not exist, blacklisting all devices.
Mar 10 19:18:28 | A sample multipath.conf file is located at
Mar 10 19:18:28 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Mar 10 19:18:28 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf
Mar 10 19:18:28 | DM multipath kernel driver not loaded —-DM 模块没有加载
如果模块没有加载胜利请应用下列命初始化 DM, 或重启零碎
—Use the following commands to initialize and start DM for the first time:
- modprobe dm-multipath
modprobe dm-round-robin
service multipathd start
multipath –v2
初始化完了之后再通过 multipath -ll 命令查看是否加载胜利
[root@liujing ~]# multipath -ll
Mar 10 19:21:14 | /etc/multipath.conf does not exist, blacklisting all devices.
Mar 10 19:21:14 | A sample multipath.conf file is located at
Mar 10 19:21:14 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Mar 10 19:21:14 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf
DM multipath kernel driver not loaded —- 这个提醒没了阐明 DM 模块已加载胜利。
从下面的提醒能够看到，DM 模块是胜利加载，然而 /etc/ 下没有 multipath.conf 配置文件，下一步介绍如何配置 multipath.conf 文件。
How Disk Multipath works on Linux
2. 配置 multipath：
通过 vi 命令创立一个 Multipath 的配置文件门路是 /etc/multipath.conf , 在配置文件中增加 multipath 失常工作的最简配置如下：
vi /etc/multipath.conf
blacklist {
devnode “^sda”
}
defaults {
user_friendly_names yes
path_grouping_policy multibus
failback immediate
no_path_retry fail
}
编辑实现后保留配置，同时通过命令：
/etc/init.d/multipathd start #开启 mulitipath 服务
如果呈现无奈开启服务的状况，没有提醒 OK 的话如下：
[root@liujing mapper]# service multipathd start
Starting multipathd daemon: 没有提醒 OK
从新开关一下服务就能够解决了。
[root@liujing mapper]# /etc/init.d/multipathd stop
Stopping multipathd daemon: [OK]
[root@localhost mapper]# /etc/init.d/multipathd start
Starting multipathd daemon: [OK] —– 提醒 OK 失常开启服务
通过命令查看：
[root@liujing mapper]# multipath -ll
mpatha (360a9800064665072443469563477396c) dm-0 NETAPP,LUN —- 创立了一个 lun
size=3.5G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=4 status=active
|- 1:0:0:0 sdb 8:16 active ready running —- 多路径下的两个盘符 sdb 和 sde.
`- 2:0:0:0 sde 8:64 active ready running
目录 /dev/mapper/ 下多了两个文件夹 mpatha 和 mpathap1。
[root@liujing mapper]# cd /dev/mapper/
[root@liujing mapper]# ls
control mpatha mpathap1
同时 fdisk –l 的命令下也多了两个设施标识：
没有配置多路径时：
[root@liujing~]# fdisk -l
Disk /dev/sda: 146.8 GB, 146815733760 bytes
255 heads, 63 sectors/track, 17849 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a6cdd
Device Boot Start End Blocks Id System
/dev/sda1 * 1 26 204800 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 26 287 2097152 82 Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3 287 17850 141071360 83 Linux
Disk /dev/sdb: 3774 MB, 3774873600 bytes
117 heads, 62 sectors/track, 1016 cylinders
Units = cylinders of 7254 * 512 = 3714048 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 65536 bytes
Disk identifier: 0xac956c3a
Device Boot Start End Blocks Id System
/dev/sdb1 1 1016 3685001 83 Linux
Partition 1 does not start on physical sector boundary.
Disk /dev/sde: 3774 MB, 3774873600 bytes
117 heads, 62 sectors/track, 1016 cylinders
Units = cylinders of 7254 * 512 = 3714048 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 65536 bytes
Disk identifier: 0xac956c3a
Device Boot Start End Blocks Id System
/dev/sde1 1 1016 3685001 83 Linux
Partition 1 does not start on physical sector boundary.
两个 CAN 网卡获取到同一盘符：
/dev/sde 和 /dev/sdb.
配置后多了 /dev/mapper/mpatha 和 /dev/mapper/mpathap1：
[root@localhost mapper]# fdisk -l
Disk /dev/sda: 146.8 GB, 146815733760 bytes
255 heads, 63 sectors/track, 17849 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a6cdd
Device Boot Start End Blocks Id System
/dev/sda1 * 1 26 204800 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 26 287 2097152 82 Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3 287 17850 141071360 83 Linux
Disk /dev/sdb: 3774 MB, 3774873600 bytes
117 heads, 62 sectors/track, 1016 cylinders
Units = cylinders of 7254 * 512 = 3714048 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 65536 bytes
Disk identifier: 0xac956c3a
Device Boot Start End Blocks Id System
/dev/sdb1 1 1016 3685001 83 Linux
Partition 1 does not start on physical sector boundary.
Disk /dev/sde: 3774 MB, 3774873600 bytes
117 heads, 62 sectors/track, 1016 cylinders
Units = cylinders of 7254 * 512 = 3714048 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 65536 bytes
Disk identifier: 0xac956c3a
Device Boot Start End Blocks Id System
/dev/sde1 1 1016 3685001 83 Linux
Partition 1 does not start on physical sector boundary.
Disk /dev/mapper/mpatha: 3774 MB, 3774873600 bytes
117 heads, 62 sectors/track, 1016 cylinders
Units = cylinders of 7254 * 512 = 3714048 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 65536 bytes
Disk identifier: 0xac956c3a
Device Boot Start End Blocks Id System
/dev/mapper/mpathap1 1 1016 3685001 83 Linux
Partition 1 does not start on physical sector boundary.
Disk /dev/mapper/mpathap1: 3773 MB, 3773441024 bytes
255 heads, 63 sectors/track, 458 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 65536 bytes
Alignment offset: 1024 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/mpathap1 doesn’t contain a valid partition table
multipath -F #删除现有门路 两个新的门路就会被删除
multipath -v2 #格式化门路 格式化后又呈现
15 Linux multipath examples third part
15 Linux multipath examples for Linux admin
3. multipath 磁盘的基本操作
要对多路径软件生成的磁盘进行操作间接操作 /dev/mapper/ 目录下的磁盘就行.
在对多路径软件生成的磁盘进行分区之前最好运行一下 pvcreate 命令:
pvcreate /dev/mapper/mpatha
fdisk /dev/mapper/mpatha 分区时用这个目录 /dev/mapper/mpatha
用 fdisk 对多路径软件生成的磁盘进行分区保留时会有一个报错, 此报错不必理睬.
ls -l /dev/mapper/
[root@liujing mnt]# ls -l /dev/mapper/
total 0
crw-rw—-. 1 root root 10, 58 Mar 10 19:10 control
lrwxrwxrwx. 1 root root 7 Mar 10 20:28 mpatha -> ../dm-0
lrwxrwxrwx. 1 root root 7 Mar 10 20:33 mpathap1 -> ../dm-1
的 mpathap1 就是咱们对 multipath 磁盘进行的分区
mkfs.ext4 /dev/mapper/mpathap1 #对 mpath1p1 分区格式化成 ext4 文件系统
mount /dev/mapper/mpathap1 /mnt/ #挂载 mpathap1 分区
格式化和挂载时用 /dev/mapper/mpathap1
How to Check if a Disk is Busy in Linux
4. 分区磁盘：
下面有提到分区时用目录 /dev/mapper/mpatha
[root@liujing~]# fdisk /dev/mapper/mpatha
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xac956c3a.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won’t be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to
switch off the mode (command ‘c’) and change display units to
sectors (command ‘u’).
Command (m for help): n———————— 新建分区
Command action
e extended
p primary partition (1-4)
p—————————– 主分区
Partition number (1-4): 1
First cylinder (1-1016, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1016, default 1016):
Using default value 1016
Command (m for help): w ——————— 写入列表相当于保留
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
注：如果同一台设施的两个 node 挂同样的盘符，另一个盘符还须要再次写入 w 就行。不须要 n 了。
5. 格式化：
[root@liujing ~]# mkfs.ext4 /dev/mapper/mpathap1
mke2fs 1.41.12 (17-May-2010)
/dev/sdd1 alignment is offset by 1024 bytes.
This may result in very poor performance, (re)-partitioning suggested.
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1 blocks, Stripe width=16 blocks
230608 inodes, 921250 blocks
46062 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=943718400
29 block groups
32768 blocks per group, 32768 fragments per group
7952 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Using Linux Iotop to check disk IO usage Per process
Sort Disk Read Write in Linux Iotop Command
- 挂载 /dev/mapper/mpathap1 到 /mnt
[root@liujing ~]# mount /dev/mapper/mpathap1 /mnt
三、multipath 的高级配置之前的配置都是用 multipath 的默认配置来实现 multipath，比方映射设施的名称，multipath 负载平衡的办法都是默认设置。那有没有依照咱们本人定义的办法来配置 multipath 呢，答案是 OK。
1、multipath.conf 文件的配置
接下来的工作就是要编辑 /etc/multipath.conf 的配置文件
multipath.conf 次要包含 blacklist、multipaths、devices 三部份的配置
blacklist 配置
blacklist {
devnode “^sda”
}
Multipaths 局部配置 multipaths 和 devices 两部份的配置。
multipaths {
multipath {
wwid ** #此值 multipath -v3 能够看到
alias iscsi-dm0 #映射后的别名, 能够轻易取
path_grouping_policy multibus #门路组策略
path_checker tur #决定门路状态的办法
path_selector “round-robin 0” #抉择那条门路进行下一个 IO 操作的办法
}
}
Devices 局部配置
devices {
device {
vendor “iSCSI-Enterprise” #厂商名称
product “Virtual disk” #产品型号
path_grouping_policy multibus #默认的门路组策略
getuid_callout “/sbin/scsi_id -g -u -s /block/%n” #取得惟一设施号应用的默认程序
prio_callout “/sbin/acs_prio_alua %d” #获取无限级数值应用的默认程序
path_checker readsector0 #决定门路状态的办法
path_selector “round-robin 0” #抉择那条门路进行下一个 IO 操作的办法
failback immediate #故障复原的模式
no_path_retry queue #在 disable queue 之前零碎尝试应用生效门路的次数的数值
rr_min_io 100 #在以后的用户组中，在切换到另外一条门路之前的 IO 申请的数目
}
}
在我本地的一个残缺的高级配置如下：
[root@liujing ~]# vi /etc/multipath.conf
blacklist {
devnode “^sda”
}
multipaths {
multipath {
wwid 360a98000646650724434697454546156
alias mpathb_fcoe
path_grouping_policy multibus
path_checker “directio”
prio “random”
path_selector “round-robin 0”
}
}
devices {
device {
vendor “NETAPP”
product “LUN”
getuid_callout “/lib/udev/scsi_id –whitelisted –device=/dev/%n”
path_checker “directio”
path_selector “round-robin 0”
failback immediate
no_path_retry fail
}
}
其中 wwid，vendor，product，getuid_callout 这些参数能够通过：multipath -v3 命令来获取。如果在 /etc/multipath.conf 中有设定各 wwid 别名, 别名会笼罩此设定。
15 Linux multipath command examples
四、负载平衡测试：
能够应用 dd 命令来对设施进行读写操作，并同时通过 iostat 来查看 I/0 状态，流量从哪个门路进来：
DD 命令：dd if=/dev/zero of=/mnt/1Gfile bs=8k count=131072 在下面咱们曾经把磁盘挂载在 /MNT 文件夹下所以咱们在读写磁盘时间接对 /mnt 文件夹间接读写就能够了。
如果想对磁盘反复读写能够用如下语句：
[root@liujing ~]# for ((i=1;i<=5;i++));do dd if=/dev/zero of=/mnt/1Gfile bs=8k count=131072 2>&1|grep MB;done; — 反复读写 5 次这个值能够依据本人测试需要批改。
How to Find disk SCSI ID WWIDS In Linux
另一个控制台输出 iostat 2 10 查看 IO 读写状态：
能够看到 sdc 和 sdd 是两个多路径的盘符，流量平均的负载在两条门路中，负载平衡很胜利。
五、门路冗余备份测试
将其中一条门路的端口 down 掉，所有流量会间接切换到另一个门路中。
Troubleshooting Disk Space Usage and Disk IO Issues In Linux
How to Check Disk space usage Disk utilization in Linux