Device Mapper Multipath
What is Device Mapper Multipathing?
What is Multipath? • Multipath is a storage network design technique that allows for fault tolerance or increased throughput by providing multiple concurrent physical connections (paths) from the storage to the individual host systems.
multipathd and multipath internally use WWIDs to identify devices. WWIDs are also used as map names by default.
———
Device Mapper Multipathing (DM-Multipath) is a native multipathing in Linux, Device Mapper Multipathing (DM-Multipath) can be used for Redundancy and to Improve the Performance. It aggregates or combines the multiple I/O paths between Servers and Storage, so it creates a single device at the OS Level.
For example, Lets say a server with two HBA card attached to a storage controller with single ports on each HBA cards. One lun assigned to the single server via two wwn number of both cards. So OS detects two devices: /dev/sdb and /dev/sdc. Once we installed the Device Mapper Multipathing. DM-Multipath creates a single device with a unique WWID that reroutes I/O to those four underlying devices according to the multipath configuration. So when there is a failure with any of this I/O paths, Data can be accessible using the available I/O Path.
https://www.learnitguide.net/2016/06/how-to-configure-multipathing-in-linux.html
https://ubuntu.com/server/docs/device-mapper-multipathing-introduction
http://www.datadisk.co.uk/html_docs/redhat/rh_multipathing.htm
How to Set Up a Multipath Device
This was done on RHEL7 + EC2 Nitro instance type and RHEL7 + VMWare.
1) Launch a fresh EC2 instance (Nitro/C5) with an extra EBS volume (eg. nvme1n1). If you try with xen/t2, multipathd will fail to get path uid (i.e. this would happen with an /dev/xvdb device).
$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 10G 0 disk ├─nvme0n1p1 259:1 0 1M 0 part └─nvme0n1p2 259:2 0 10G 0 part / nvme1n1 259:3 0 9G 0 disk nvme2n1 259:4 0 8G 0 disk
2) (Optional: You can do this if you want to create a multipath device on an LVM, on EBS) Create PV and VG on /dev/xvdb:
$ sudo yum -y install lvm2 $ sudo pvcreate /dev/xvdb $ sudo vgcreate testvg /dev/xvdb
3) Install multipath:
$ sudo yum -y install device-mapper-multipath
4) Next create /etc/multipath.conf file:
$ sudo mpathconf --enable
5) Configure multipath to create mpath devices on any device attached the server. This is done by removing or commenting out the entry "find_multipaths yes", or setting it to "no".
$ sudo vi /etc/multipath.conf
...
# grep -C 2 find_multipath /etc/multipath.conf
defaults {
user_friendly_names yes
#find_multipaths yes <<<< this entry must be commented
}
6) Restart multipathd. Now you will have a multipath device on top of EBS volume:
$ sudo systemctl restart multipathd $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 10G 0 disk ├─nvme0n1p1 259:1 0 1M 0 part └─nvme0n1p2 259:2 0 10G 0 part / nvme1n1 259:3 0 9G 0 disk └─mpathb 253:0 0 9G 0 mpath nvme2n1 259:4 0 8G 0 disk └─mpathc 253:1 0 8G 0 mpath $ lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT nvme0n1 ├─nvme0n1p1 └─nvme0n1p2 xfs 95070429-de61-4430-8ad0-2c0f109d8d50 / nvme1n1 mpath_member └─mpathb nvme2n1 mpath_member └─mpathc # multipath -ll mpathc (nvme.1d0f-766f6c3032646533643935396435376335653331-416d617a6f6e) dm-1 NVME,Amazon Elastic Block Store size=6.0G features='0' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active `- 2:0:1:1 nvme2n1 259:1 active ready running mpathb (nvme.1d0f-766f6c3032656238336464336134616233316330-416d617a6f6e) dm-0 NVME,Amazon Elastic Block Store size=5.0G features='0' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active `- 1:0:1:1 nvme1n1 259:0 active ready running
- If user_friendly_names was set to no or was disabled, then it would just show the WWID instead of "mpatha, mpathb," etc.
How to Set Up a Multipath Device using iSCSI in vCenter
https://www.altaro.com/vmware/adding-linux-iscsi-target-esxi/ -- need targetcli, but its a dependency hell. Follow the next document, and then follow this again to mount the LUN.
Ubuntu targetcli - https://www.server-world.info/en/note?os=Ubuntu_18.04&p=iscsi&f=1 ---- THIS IS KEY!!!!!!
https://www.youtube.com/watch?v=OBMkP0Vdy6Q
https://masteringvmware.com/how-to-add-iscsi-datastore/
REPRODUCTION (Steps to configure a multipath device on top of multiple disks on EC2) (clean this up)
Main docs we're following: https://linux.dell.com/files/whitepapers/iSCSI_Multipathing_in_Ubuntu_Server_1404_LTS.pdf
https://www.hiroom2.com/2018/05/05/ubuntu-1804-tgt-en/
The ideal network configuration in a multipath environment is to connect each network port on your server to a different subnet. That way, you have additional resilience in case one of your subnets goes down (i.e. bad switch or router).
However, you can also connect both of your network ports to the same subnet if that is all you have, as depicted in Figure 2. In this case, your network subnet becomes a single point of failure, but you still have high-availability capabilities in case one of your network ports fails. To increase resiliency in this scenario, connect each network port to a different switch in your subnet.
For simplicity purposes, I used the network topology shown in Figure 2 with only one subnet. I have a Class C network (192.168.1.0/24) and I used the following IP addresses
1) Spun up ubuntu 18. This instance will act as the iSCSI storage server/target. We will call this "Instance A". Attached a secondary NIC of the same subnet and follow this to set it up so you don't get asymmetric routing: https://repost.aws/knowledge-center/ec2-ubuntu-secondary-network-interface
* Note: **Both private IPs/NIC MUST be reachable**.
* Example of network configuration:
```
$ cat /etc/netplan/51-eth1.yaml
network:
version: 2
renderer: networkd
ethernets:
eth1:
addresses:
- 172.31.29.150/20
dhcp4: no
routes:
- to: 0.0.0.0/0
via: 172.31.16.1 # Default gateway (check your subnet)
table: 1000
- to: 172.31.29.150
via: 0.0.0.0
scope: link
table: 1000
routing-policy:
- from: 172.31.29.150
table: 1000
$ ip r show table 1000
default via 172.31.16.1 dev eth1 proto static
172.31.29.150 dev eth1 proto static scope link
$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
link/ether 02:b4:46:25:01:31 brd ff:ff:ff:ff:ff:ff
inet 172.31.22.88/20 brd 172.31.31.255 scope global dynamic eth0
valid_lft 3311sec preferred_lft 3311sec
inet6 fe80::b4:46ff:fe25:131/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:91:23:27:f3:57 brd ff:ff:ff:ff:ff:ff
inet 172.31.29.150/20 brd 172.31.31.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::91:23ff:fe27:f357/64 scope link
valid_lft forever preferred_lft forever
```
2) Install tgt
``` $ sudo apt install -y tgt ```
3) Create iSCSI target. This article uses a file as logical unit. You can use block device as logical unit.
``` $ sudo mkdir /var/lib/iscsi $ sudo dd if=/dev/zero of=/var/lib/iscsi/disk bs=1M count=1K ``` Create iSCSI target (tid 1). ``` $ sudo tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.2018-05.com.hiroom2:disk ``` Add logical Unit (lun 1) to iSCSI target (Target ID 1). ``` $ sudo tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /var/lib/iscsi/disk ``` Publish iSCSI target (tid 1) to all IP address. You can specify 192.168.11.1 and 192.168.11.0/24 in addition to ALL. ``` $ sudo tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL ``` Save configruation for iSCSI target. If you do not save configuration, configuration will be removed after restarting tgtd. (This command was slightly different than Yaron's/doc) ``` $ sudo tgt-admin --dump | tee /etc/tgt/conf.d/disk.configuration
OUTPUT:
------
default-driver iscsi
<target iqn.2018-05.com.hiroom2:disk>
backing-store /var/lib/iscsi/disk
</target>
```
4) Connect to iSCSI target with open-iscsi which is iSCSI initiator. iSCSI initiator runs on server which is installed iSCSI target. The partition is the following before connecting to iSCSI target (locally).
``` $ cat /proc/partitions
OUTPUT: ------ major minor #blocks name 7 0 24972 loop0 7 1 56972 loop1 7 2 64976 loop2 7 3 54516 loop3 7 4 94036 loop4 202 0 8388608 xvda 202 1 8274927 xvda1 202 14 4096 xvda14 202 15 108544 xvda15 ``` ``` $ sudo apt install -y open-iscsi ``` ``` $ sudo iscsiadm -m discovery -t st -p localhost
OUTPUT: ------ 127.0.0.1:3260,1 iqn.2018-05.com.hiroom2:disk ``` Conect to iSCSI target (locally): ``` $ sudo iscsiadm -m node --targetname iqn.2018-05.com.hiroom2:disk -p localhost -l
OUTPUT: ------- Logging in to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 127.0.0.1,3260] (multiple) Login to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 127.0.0.1,3260] successful. ``` The partition is the following after connecting to iSCSI target. The partition sda is appended. ``` $ cat /proc/partitions
OUTPUT: ------ major minor #blocks name 7 0 24972 loop0 7 1 56972 loop1 7 2 64976 loop2 7 3 54516 loop3 7 4 94036 loop4 202 0 8388608 xvda 202 1 8274927 xvda1 202 14 4096 xvda14 202 15 108544 xvda15 8 0 1048576 sda <---------- ``` Check the WWID of the disk: ``` # /lib/udev/scsi_id --whitelisted --device=/dev/sda 360000000000000000e00000000010001 ```
5) So now I know how to mount it locally. I will mount the device onto another EC2. I spun up another Ubuntu instance (we'll call this Instance B). And ran the following commands. **NOTE**: BE SURE TO ALLOW INBOUND for TCP port 3260 on each ENI of instance A !:
``` $ sudo iscsiadm -m discovery -t st -p 172.31.29.150
OUTPUT: ------ 172.31.29.150:3260,1 iqn.2018-05.com.hiroom2:disk ``` ``` # Connect to the target using the private IP of ENI #1 $ sudo iscsiadm -m node --targetname iqn.2018-05.com.hiroom2:disk -p 172.31.29.150 -l
OUTPUT: ------ Logging in to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.29.150,3260] (multiple) Login to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.29.150,3260] successful. ``` ``` # Connect to the target using the private IP of ENI #1 $ sudo iscsiadm -m discovery -t st -p 172.31.30.116
OUTPUT: ------ 172.31.30.116:3260,1 iqn.2018-05.com.hiroom2:disk ``` ``` $ sudo iscsiadm -m node --targetname iqn.2018-05.com.hiroom2:disk -p 172.31.30.116 -l
OUTPUT: ------ Logging in to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.30.116,3260] (multiple) Login to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.30.116,3260] successful. ``` ``` $ sudo /lib/udev/scsi_id --whitelisted --device=/dev/sda 360000000000000000e00000000010001
$ sudo /lib/udev/scsi_id --whitelisted --device=/dev/sdb 360000000000000000e00000000010001 ```
Done! You now have a multipath device on top of multiple disks/EBS volumes:
```
$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 55.7M 1 loop /snap/core18/2745 loop1 7:1 0 63.5M 1 loop /snap/core20/1891 loop2 7:2 0 24.4M 1 loop /snap/amazon-ssm-agent/6312 loop3 7:3 0 53.2M 1 loop /snap/snapd/19122 loop4 7:4 0 91.9M 1 loop /snap/lxd/24061 sda 8:0 0 1G 0 disk └─mpatha 253:0 0 1G 0 mpath sdb 8:16 0 1G 0 disk └─mpatha 253:0 0 1G 0 mpath nvme0n1 259:0 0 8G 0 disk
├─nvme0n1p1 259:1 0 7.9G 0 part / ├─nvme0n1p14 259:2 0 4M 0 part └─nvme0n1p15 259:3 0 106M 0 part /boot/efi
$ sudo multipath -ll mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw |-+- policy='service-time 0' prio=1 status=active | `- 0:0:0:1 sda 8:0 active ready running `-+- policy='service-time 0' prio=1 status=enabled `- 1:0:0:1 sdb 8:16 active ready running
```
How to get the WWID of a device
On Vmware: https://access.redhat.com/solutions/93943
For RHEL7 and RHEL8
# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sda 36000c2931a129f3c880b8d06ccea1b01
For RHEL6
# scsi_id --whitelisted --replace-whitespace --device=/dev/sda 36000c2931a129f3c880b8d06ccea1b01
For RHEL5
#scsi_id -g -u -s /block/sdb 36000c2931a129f3c880b8d06ccea1b01