Ardika Sulistija: Created page with "====What is Device Mapper Multipathing?==== ---- What is Multipath? • Multipath is a storage network design technique that allows for fault tolerance or increased throughput by providing multiple concurrent physical connections (paths) from the storage to the individual host systems. multipathd and multipath internally use WWIDs to identify devices. WWIDs are also used as map names by default. ——— Device Mapper Multipathing (DM-Multipath) is a native multipat..."

2024-08-21T14:47:03Z

Created page with "====What is Device Mapper Multipathing?==== ---- What is Multipath? • Multipath is a storage network design technique that allows for fault tolerance or increased throughput by providing multiple concurrent physical connections (paths) from the storage to the individual host systems. multipathd and multipath internally use WWIDs to identify devices. WWIDs are also used as map names by default. ——— Device Mapper Multipathing (DM-Multipath) is a native multipat..."

New page

====What is Device Mapper Multipathing?====
----

What is Multipath?
• Multipath is a storage network design technique that allows for fault tolerance or increased throughput by providing multiple concurrent physical connections (paths) from the storage to the individual host systems.

multipathd and multipath internally use WWIDs to identify devices. WWIDs are also used as map names by default.

———

Device Mapper Multipathing (DM-Multipath) is a native multipathing in Linux, Device Mapper Multipathing (DM-Multipath) can be used for Redundancy and to Improve the Performance. It aggregates or combines the multiple I/O paths between Servers and Storage, so it creates a single device at the OS Level.

For example, Lets say a server with two HBA card attached to a storage controller with single ports on each HBA cards. One lun assigned to the single server via two wwn number of both cards. So OS detects two devices: /dev/sdb and /dev/sdc. Once we installed the Device Mapper Multipathing. DM-Multipath creates a single device with a unique WWID that reroutes I/O to those four underlying devices according to the multipath configuration. So when there is a failure with any of this I/O paths, Data can be accessible using the available I/O Path.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/dm_multipath/mpath_devices

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_device_mapper_multipath/index

https://www.learnitguide.net/2016/06/how-to-configure-multipathing-in-linux.html

https://ubuntu.com/server/docs/device-mapper-multipathing-introduction

http://www.datadisk.co.uk/html_docs/redhat/rh_multipathing.htm

====How to Set Up a Multipath Device====
----

This was done on RHEL7 + EC2 Nitro instance type and RHEL7 + VMWare.

1) Launch a fresh EC2 instance (Nitro/C5) with an extra EBS volume (eg. nvme1n1). If you try with xen/t2, multipathd will fail to get path uid (i.e. this would happen with an /dev/xvdb device).

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 10G 0 disk
├─nvme0n1p1 259:1 0 1M 0 part
└─nvme0n1p2 259:2 0 10G 0 part /
nvme1n1 259:3 0 9G 0 disk
nvme2n1 259:4 0 8G 0 disk

2) (Optional: You can do this if you want to create a multipath device on an LVM, on EBS) Create PV and VG on /dev/xvdb:

$ sudo yum -y install lvm2
$ sudo pvcreate /dev/xvdb
$ sudo vgcreate testvg /dev/xvdb

3) Install multipath:

$ sudo yum -y install device-mapper-multipath

4) Next create /etc/multipath.conf file:
$ sudo mpathconf --enable

5) Configure multipath to create mpath devices on any device attached the server. This is done by removing or commenting out the entry "find_multipaths yes", or setting it to "no".

$ sudo vi /etc/multipath.conf
...
# grep -C 2 find_multipath /etc/multipath.conf
defaults {
user_friendly_names yes
#find_multipaths yes <<<< this entry must be commented
}

6) Restart multipathd. Now you will have a multipath device on top of EBS volume:

$ sudo systemctl restart multipathd

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 10G 0 disk
├─nvme0n1p1 259:1 0 1M 0 part
└─nvme0n1p2 259:2 0 10G 0 part /
nvme1n1 259:3 0 9G 0 disk
└─mpathb 253:0 0 9G 0 mpath
nvme2n1 259:4 0 8G 0 disk
└─mpathc 253:1 0 8G 0 mpath

$ lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
nvme0n1
├─nvme0n1p1
└─nvme0n1p2 xfs 95070429-de61-4430-8ad0-2c0f109d8d50 /
nvme1n1 mpath_member
└─mpathb
nvme2n1 mpath_member
└─mpathc

# multipath -ll
mpathc (nvme.1d0f-766f6c3032646533643935396435376335653331-416d617a6f6e) dm-1 NVME,Amazon Elastic Block Store
size=6.0G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
`- 2:0:1:1 nvme2n1 259:1 active ready running
mpathb (nvme.1d0f-766f6c3032656238336464336134616233316330-416d617a6f6e) dm-0 NVME,Amazon Elastic Block Store
size=5.0G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
`- 1:0:1:1 nvme1n1 259:0 active ready running

* If <b>user_friendly_names</b> was set to <i>no</i> or was disabled, then it would just show the WWID instead of "mpatha, mpathb," etc.

====How to Set Up a Multipath Device using iSCSI in vCenter====
----

https://www.altaro.com/vmware/adding-linux-iscsi-target-esxi/ -- need targetcli, but its a dependency hell. Follow the next document, and then follow this again to mount the LUN.

Ubuntu targetcli - https://www.server-world.info/en/note?os=Ubuntu_18.04&p=iscsi&f=1 ---- THIS IS KEY!!!!!!

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.hostclient.doc/GUID-4D0E250E-4F8C-4F86-81C1-EC9D317CE02E.html

https://www.codyhosterman.com/2017/07/setting-up-software-iscsi-multipathing-with-distributed-vswitches-with-the-vsphere-web-client/

https://www.youtube.com/watch?v=OBMkP0Vdy6Q

https://masteringvmware.com/how-to-add-iscsi-datastore/

==== REPRODUCTION (Steps to configure a multipath device on top of multiple disks on EC2) (clean this up)====
----
Main docs we're following:
https://linux.dell.com/files/whitepapers/iSCSI_Multipathing_in_Ubuntu_Server_1404_LTS.pdf
https://www.hiroom2.com/2018/05/05/ubuntu-1804-tgt-en/

The ideal network configuration in a multipath environment is to connect each network port on your
server to a different subnet. That way, you have additional resilience in case one of your subnets goes
down (i.e. bad switch or router).

However, you can also connect both of your network ports to the same subnet if that is all you have,
as depicted in Figure 2. In this case, your network subnet becomes a single point of failure, but you
still have high-availability capabilities in case one of your network ports fails. To increase resiliency in
this scenario, connect each network port to a different switch in your subnet.

For simplicity purposes, I used the network topology shown in Figure 2 with only one subnet. I have a
Class C network (192.168.1.0/24) and I used the following IP addresses

1) Spun up ubuntu 18. This instance will act as the iSCSI storage server/target. We will call this "Instance A". Attached a secondary NIC of the same subnet and follow this to set it up so you don't get asymmetric routing: https://repost.aws/knowledge-center/ec2-ubuntu-secondary-network-interface
* Note: **Both private IPs/NIC MUST be reachable**.

* Example of network configuration:
```
$ cat /etc/netplan/51-eth1.yaml
network:
version: 2
renderer: networkd
ethernets:
eth1:
addresses:
- 172.31.29.150/20
dhcp4: no
routes:
- to: 0.0.0.0/0
via: 172.31.16.1 # Default gateway (check your subnet)
table: 1000
- to: 172.31.29.150
via: 0.0.0.0
scope: link
table: 1000
routing-policy:
- from: 172.31.29.150
table: 1000

$ ip r show table 1000
default via 172.31.16.1 dev eth1 proto static
172.31.29.150 dev eth1 proto static scope link

$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
link/ether 02:b4:46:25:01:31 brd ff:ff:ff:ff:ff:ff
inet 172.31.22.88/20 brd 172.31.31.255 scope global dynamic eth0
valid_lft 3311sec preferred_lft 3311sec
inet6 fe80::b4:46ff:fe25:131/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:91:23:27:f3:57 brd ff:ff:ff:ff:ff:ff
inet 172.31.29.150/20 brd 172.31.31.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::91:23ff:fe27:f357/64 scope link
valid_lft forever preferred_lft forever
```

2) Install tgt
```
$ sudo apt install -y tgt
```
3) Create iSCSI target. This article uses a file as logical unit. You can use block device as logical unit.
```
$ sudo mkdir /var/lib/iscsi
$ sudo dd if=/dev/zero of=/var/lib/iscsi/disk bs=1M count=1K
```
Create iSCSI target (tid 1).
```
$ sudo tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.2018-05.com.hiroom2:disk
```
Add logical Unit (lun 1) to iSCSI target (Target ID 1).
```
$ sudo tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /var/lib/iscsi/disk
```
Publish iSCSI target (tid 1) to all IP address. You can specify 192.168.11.1 and 192.168.11.0/24 in addition to ALL.
```
$ sudo tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
```
Save configruation for iSCSI target. If you do not save configuration, configuration will be removed after restarting tgtd. (This command was slightly different than Yaron's/doc)
```
$ sudo tgt-admin --dump | tee /etc/tgt/conf.d/disk.configuration

OUTPUT:
------
default-driver iscsi

<target iqn.2018-05.com.hiroom2:disk>
backing-store /var/lib/iscsi/disk
</target>
```
4) Connect to iSCSI target with open-iscsi which is iSCSI initiator. iSCSI initiator runs on server which is installed iSCSI target. The partition is the following before connecting to iSCSI target (locally).
```
$ cat /proc/partitions

OUTPUT:
------
major minor #blocks name
7 0 24972 loop0
7 1 56972 loop1
7 2 64976 loop2
7 3 54516 loop3
7 4 94036 loop4
202 0 8388608 xvda
202 1 8274927 xvda1
202 14 4096 xvda14
202 15 108544 xvda15
```
```
$ sudo apt install -y open-iscsi
```
```
$ sudo iscsiadm -m discovery -t st -p localhost

OUTPUT:
------
127.0.0.1:3260,1 iqn.2018-05.com.hiroom2:disk
```
Conect to iSCSI target (locally):
```
$ sudo iscsiadm -m node --targetname iqn.2018-05.com.hiroom2:disk -p localhost -l

OUTPUT:
-------
Logging in to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 127.0.0.1,3260] (multiple)
Login to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 127.0.0.1,3260] successful.
```
The partition is the following after connecting to iSCSI target. The partition sda is appended.
```
$ cat /proc/partitions

OUTPUT:
------
major minor #blocks name
7 0 24972 loop0
7 1 56972 loop1
7 2 64976 loop2
7 3 54516 loop3
7 4 94036 loop4
202 0 8388608 xvda
202 1 8274927 xvda1
202 14 4096 xvda14
202 15 108544 xvda15
8 0 1048576 sda <----------
```
Check the WWID of the disk:
```
# /lib/udev/scsi_id --whitelisted --device=/dev/sda
360000000000000000e00000000010001
```
5) So now I know how to mount it locally. I will mount the device onto another EC2. I spun up another Ubuntu instance (we'll call this Instance B). And ran the following commands. **NOTE**: BE SURE TO ALLOW INBOUND for TCP port 3260 on each ENI of instance A !:
```
$ sudo iscsiadm -m discovery -t st -p 172.31.29.150

OUTPUT:
------
172.31.29.150:3260,1 iqn.2018-05.com.hiroom2:disk
```
```
# Connect to the target using the private IP of ENI #1
$ sudo iscsiadm -m node --targetname iqn.2018-05.com.hiroom2:disk -p 172.31.29.150 -l

OUTPUT:
------
Logging in to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.29.150,3260] (multiple)
Login to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.29.150,3260] successful.
```
```
# Connect to the target using the private IP of ENI #1
$ sudo iscsiadm -m discovery -t st -p 172.31.30.116

OUTPUT:
------
172.31.30.116:3260,1 iqn.2018-05.com.hiroom2:disk
```
```
$ sudo iscsiadm -m node --targetname iqn.2018-05.com.hiroom2:disk -p 172.31.30.116 -l

OUTPUT:
------
Logging in to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.30.116,3260] (multiple)
Login to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.30.116,3260] successful.
```
```
$ sudo /lib/udev/scsi_id --whitelisted --device=/dev/sda
360000000000000000e00000000010001

$ sudo /lib/udev/scsi_id --whitelisted --device=/dev/sdb
360000000000000000e00000000010001
```
Done! You now have a multipath device on top of multiple disks/EBS volumes:
```
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 55.7M 1 loop /snap/core18/2745
loop1 7:1 0 63.5M 1 loop /snap/core20/1891
loop2 7:2 0 24.4M 1 loop /snap/amazon-ssm-agent/6312
loop3 7:3 0 53.2M 1 loop /snap/snapd/19122
loop4 7:4 0 91.9M 1 loop /snap/lxd/24061
sda 8:0 0 1G 0 disk
└─mpatha 253:0 0 1G 0 mpath
sdb 8:16 0 1G 0 disk
└─mpatha 253:0 0 1G 0 mpath
nvme0n1 259:0 0 8G 0 disk
├─nvme0n1p1 259:1 0 7.9G 0 part /
├─nvme0n1p14 259:2 0 4M 0 part
└─nvme0n1p15 259:3 0 106M 0 part /boot/efi

$ sudo multipath -ll
mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 0:0:0:1 sda 8:0 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 1:0:0:1 sdb 8:16 active ready running
```

====How to get the WWID of a device====
----
On Vmware:
https://access.redhat.com/solutions/93943

For RHEL7 and RHEL8
# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sda
36000c2931a129f3c880b8d06ccea1b01

For RHEL6
# scsi_id --whitelisted --replace-whitespace --device=/dev/sda
36000c2931a129f3c880b8d06ccea1b01

For RHEL5
#scsi_id -g -u -s /block/sdb
36000c2931a129f3c880b8d06ccea1b01

Device Mapper Multipath - Revision history