Device Mapper Multipath

From DikapediaV2
Revision as of 14:47, 21 August 2024 by Ardika Sulistija (talk | contribs) (Created page with "====What is Device Mapper Multipathing?==== ---- What is Multipath? • Multipath is a storage network design technique that allows for fault tolerance or increased throughput by providing multiple concurrent physical connections (paths) from the storage to the individual host systems. multipathd and multipath internally use WWIDs to identify devices. WWIDs are also used as map names by default. ——— Device Mapper Multipathing (DM-Multipath) is a native multipat...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

What is Device Mapper Multipathing?


What is Multipath? • Multipath is a storage network design technique that allows for fault tolerance or increased throughput by providing multiple concurrent physical connections (paths) from the storage to the individual host systems.

multipathd and multipath internally use WWIDs to identify devices. WWIDs are also used as map names by default.

———

Device Mapper Multipathing (DM-Multipath) is a native multipathing in Linux, Device Mapper Multipathing (DM-Multipath) can be used for Redundancy and to Improve the Performance. It aggregates or combines the multiple I/O paths between Servers and Storage, so it creates a single device at the OS Level.

For example, Lets say a server with two HBA card attached to a storage controller with single ports on each HBA cards. One lun assigned to the single server via two wwn number of both cards. So OS detects two devices: /dev/sdb and /dev/sdc. Once we installed the Device Mapper Multipathing. DM-Multipath creates a single device with a unique WWID that reroutes I/O to those four underlying devices according to the multipath configuration. So when there is a failure with any of this I/O paths, Data can be accessible using the available I/O Path.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/dm_multipath/mpath_devices

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_device_mapper_multipath/index

https://www.learnitguide.net/2016/06/how-to-configure-multipathing-in-linux.html

https://ubuntu.com/server/docs/device-mapper-multipathing-introduction

http://www.datadisk.co.uk/html_docs/redhat/rh_multipathing.htm


How to Set Up a Multipath Device


This was done on RHEL7 + EC2 Nitro instance type and RHEL7 + VMWare.

1) Launch a fresh EC2 instance (Nitro/C5) with an extra EBS volume (eg. nvme1n1). If you try with xen/t2, multipathd will fail to get path uid (i.e. this would happen with an /dev/xvdb device).

   $ lsblk
   NAME        MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
   nvme0n1     259:0    0  10G  0 disk 
   ├─nvme0n1p1 259:1    0   1M  0 part 
   └─nvme0n1p2 259:2    0  10G  0 part /
   nvme1n1     259:3    0   9G  0 disk 
   nvme2n1     259:4    0   8G  0 disk

2) (Optional: You can do this if you want to create a multipath device on an LVM, on EBS) Create PV and VG on /dev/xvdb:

   $ sudo yum -y install lvm2
   $ sudo pvcreate /dev/xvdb
   $ sudo vgcreate testvg /dev/xvdb

3) Install multipath:

   $ sudo yum -y install device-mapper-multipath

4) Next create /etc/multipath.conf file:

   $ sudo mpathconf --enable

5) Configure multipath to create mpath devices on any device attached the server. This is done by removing or commenting out the entry "find_multipaths yes", or setting it to "no".

   $ sudo vi /etc/multipath.conf
   ...
   # grep -C 2 find_multipath /etc/multipath.conf 
   defaults {
       user_friendly_names yes
           #find_multipaths yes <<<< this entry must be commented
   }

6) Restart multipathd. Now you will have a multipath device on top of EBS volume:

   $ sudo systemctl restart multipathd

   $ lsblk
   NAME        MAJ:MIN RM SIZE RO TYPE  MOUNTPOINT
   nvme0n1     259:0    0  10G  0 disk  
   ├─nvme0n1p1 259:1    0   1M  0 part  
   └─nvme0n1p2 259:2    0  10G  0 part  /
   nvme1n1     259:3    0   9G  0 disk  
   └─mpathb    253:0    0   9G  0 mpath 
   nvme2n1     259:4    0   8G  0 disk  
   └─mpathc    253:1    0   8G  0 mpath

   $ lsblk -f
   NAME        FSTYPE       LABEL UUID                                 MOUNTPOINT
   nvme0n1                                                             
   ├─nvme0n1p1                                                         
   └─nvme0n1p2 xfs                95070429-de61-4430-8ad0-2c0f109d8d50 /
   nvme1n1     mpath_member                                            
   └─mpathb                                                            
   nvme2n1     mpath_member                                            
   └─mpathc

# multipath -ll
mpathc (nvme.1d0f-766f6c3032646533643935396435376335653331-416d617a6f6e) dm-1 NVME,Amazon Elastic Block Store              
size=6.0G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 2:0:1:1 nvme2n1 259:1 active ready running
mpathb (nvme.1d0f-766f6c3032656238336464336134616233316330-416d617a6f6e) dm-0 NVME,Amazon Elastic Block Store              
size=5.0G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 1:0:1:1 nvme1n1 259:0 active ready running
  • If user_friendly_names was set to no or was disabled, then it would just show the WWID instead of "mpatha, mpathb," etc.

How to Set Up a Multipath Device using iSCSI in vCenter


       https://www.altaro.com/vmware/adding-linux-iscsi-target-esxi/ -- need targetcli, but its a dependency hell. Follow the next document, and then follow this again to mount the LUN. 
       Ubuntu targetcli - https://www.server-world.info/en/note?os=Ubuntu_18.04&p=iscsi&f=1 ---- THIS IS KEY!!!!!!



https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.hostclient.doc/GUID-4D0E250E-4F8C-4F86-81C1-EC9D317CE02E.html

https://www.codyhosterman.com/2017/07/setting-up-software-iscsi-multipathing-with-distributed-vswitches-with-the-vsphere-web-client/

https://www.youtube.com/watch?v=OBMkP0Vdy6Q

https://masteringvmware.com/how-to-add-iscsi-datastore/


REPRODUCTION (Steps to configure a multipath device on top of multiple disks on EC2) (clean this up)


Main docs we're following: https://linux.dell.com/files/whitepapers/iSCSI_Multipathing_in_Ubuntu_Server_1404_LTS.pdf

https://www.hiroom2.com/2018/05/05/ubuntu-1804-tgt-en/

The ideal network configuration in a multipath environment is to connect each network port on your server to a different subnet. That way, you have additional resilience in case one of your subnets goes down (i.e. bad switch or router).

However, you can also connect both of your network ports to the same subnet if that is all you have, as depicted in Figure 2. In this case, your network subnet becomes a single point of failure, but you still have high-availability capabilities in case one of your network ports fails. To increase resiliency in this scenario, connect each network port to a different switch in your subnet.

For simplicity purposes, I used the network topology shown in Figure 2 with only one subnet. I have a Class C network (192.168.1.0/24) and I used the following IP addresses

1) Spun up ubuntu 18. This instance will act as the iSCSI storage server/target. We will call this "Instance A". Attached a secondary NIC of the same subnet and follow this to set it up so you don't get asymmetric routing: https://repost.aws/knowledge-center/ec2-ubuntu-secondary-network-interface

  * Note: **Both private IPs/NIC MUST be reachable**.
  * Example of network configuration:
  ```
  $ cat /etc/netplan/51-eth1.yaml 
  network:
    version: 2
    renderer: networkd
    ethernets:
      eth1:
        addresses:
         - 172.31.29.150/20
        dhcp4: no
        routes:
         - to: 0.0.0.0/0
           via: 172.31.16.1 # Default gateway (check your subnet)
           table: 1000
         - to: 172.31.29.150
           via: 0.0.0.0
           scope: link
           table: 1000
        routing-policy:
          - from: 172.31.29.150
            table: 1000
  
  $ ip r show table 1000
  default via 172.31.16.1 dev eth1 proto static 
  172.31.29.150 dev eth1 proto static scope link 
  
  $ ip addr show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host 
         valid_lft forever preferred_lft forever
  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
      link/ether 02:b4:46:25:01:31 brd ff:ff:ff:ff:ff:ff
      inet 172.31.22.88/20 brd 172.31.31.255 scope global dynamic eth0
         valid_lft 3311sec preferred_lft 3311sec
      inet6 fe80::b4:46ff:fe25:131/64 scope link 
         valid_lft forever preferred_lft forever
  3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
      link/ether 02:91:23:27:f3:57 brd ff:ff:ff:ff:ff:ff
      inet 172.31.29.150/20 brd 172.31.31.255 scope global eth1
         valid_lft forever preferred_lft forever
      inet6 fe80::91:23ff:fe27:f357/64 scope link 
         valid_lft forever preferred_lft forever
  ```


2) Install tgt

   ```
  $ sudo apt install -y tgt
  ```

3) Create iSCSI target. This article uses a file as logical unit. You can use block device as logical unit.

  ```
  $ sudo mkdir /var/lib/iscsi
  $ sudo dd if=/dev/zero of=/var/lib/iscsi/disk bs=1M count=1K
  ```
  Create iSCSI target (tid 1).
  ```
   $ sudo tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.2018-05.com.hiroom2:disk
  ```
  Add logical Unit (lun 1) to iSCSI target (Target ID 1).
  ```
  $ sudo tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /var/lib/iscsi/disk
  ```
  Publish iSCSI target (tid 1) to all IP address. You can specify 192.168.11.1 and 192.168.11.0/24 in addition to ALL.
  ```
  $ sudo tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
  ```
  Save configruation for iSCSI target. If you do not save configuration, configuration will be removed after restarting tgtd. (This command was slightly different than Yaron's/doc)
  ```
  $ sudo tgt-admin --dump | tee /etc/tgt/conf.d/disk.configuration
   OUTPUT:
   ------
    default-driver iscsi
   <target iqn.2018-05.com.hiroom2:disk>
       backing-store /var/lib/iscsi/disk
   </target>
  ```

4) Connect to iSCSI target with open-iscsi which is iSCSI initiator. iSCSI initiator runs on server which is installed iSCSI target. The partition is the following before connecting to iSCSI target (locally).

  ```
   $ cat /proc/partitions
   OUTPUT:
   ------
   major minor  #blocks  name
   7        0      24972 loop0
   7        1      56972 loop1
   7        2      64976 loop2
   7        3      54516 loop3
   7        4      94036 loop4
   202        0    8388608 xvda
   202        1    8274927 xvda1
   202       14       4096 xvda14
   202       15     108544 xvda15
  ```
  ```
  $ sudo apt install -y open-iscsi
  ```
  ```
  $ sudo iscsiadm -m discovery -t st -p localhost 
  OUTPUT:
  ------
  127.0.0.1:3260,1 iqn.2018-05.com.hiroom2:disk
  ```
  Conect to iSCSI target (locally):
  ```
  $ sudo iscsiadm -m node --targetname iqn.2018-05.com.hiroom2:disk -p localhost -l
  OUTPUT:
  -------
  Logging in to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 127.0.0.1,3260] (multiple)
  Login to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 127.0.0.1,3260] successful.
  ```
  The partition is the following after connecting to iSCSI target. The partition sda is appended. 
  ```
  $ cat /proc/partitions
  OUTPUT:
  ------
  major minor  #blocks  name
  7        0      24972 loop0
  7        1      56972 loop1
  7        2      64976 loop2
  7        3      54516 loop3
  7        4      94036 loop4
  202        0    8388608 xvda
  202        1    8274927 xvda1
  202       14       4096 xvda14
  202       15     108544 xvda15
  8        0    1048576 sda <---------- 
  ```
  Check the WWID of the disk:
  ```
  # /lib/udev/scsi_id --whitelisted --device=/dev/sda
  360000000000000000e00000000010001
  ```

5) So now I know how to mount it locally. I will mount the device onto another EC2. I spun up another Ubuntu instance (we'll call this Instance B). And ran the following commands. **NOTE**: BE SURE TO ALLOW INBOUND for TCP port 3260 on each ENI of instance A !:

  ```
  $ sudo iscsiadm -m discovery -t st -p 172.31.29.150
  OUTPUT:
  ------
  172.31.29.150:3260,1 iqn.2018-05.com.hiroom2:disk
  ```
  ```
  # Connect to the target using the private IP of ENI #1
  $ sudo iscsiadm -m node --targetname iqn.2018-05.com.hiroom2:disk -p 172.31.29.150 -l
  OUTPUT:
  ------
  Logging in to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.29.150,3260] (multiple)
  Login to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.29.150,3260] successful.
  ```
  ```
  # Connect to the target using the private IP of ENI #1
  $ sudo iscsiadm -m discovery -t st -p 172.31.30.116
  OUTPUT:
  ------
  172.31.30.116:3260,1 iqn.2018-05.com.hiroom2:disk
  ```
  ```
  $ sudo iscsiadm -m node --targetname iqn.2018-05.com.hiroom2:disk -p 172.31.30.116 -l
  OUTPUT:
  ------
  Logging in to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.30.116,3260] (multiple)
  Login to [iface: default, target: iqn.2018-05.com.hiroom2:disk, portal: 172.31.30.116,3260] successful.
  ```
  ```
  $ sudo /lib/udev/scsi_id --whitelisted --device=/dev/sda
  360000000000000000e00000000010001
  $ sudo /lib/udev/scsi_id --whitelisted --device=/dev/sdb
  360000000000000000e00000000010001
  ```

Done! You now have a multipath device on top of multiple disks/EBS volumes:

  ```

$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 55.7M 1 loop /snap/core18/2745 loop1 7:1 0 63.5M 1 loop /snap/core20/1891 loop2 7:2 0 24.4M 1 loop /snap/amazon-ssm-agent/6312 loop3 7:3 0 53.2M 1 loop /snap/snapd/19122 loop4 7:4 0 91.9M 1 loop /snap/lxd/24061 sda 8:0 0 1G 0 disk └─mpatha 253:0 0 1G 0 mpath sdb 8:16 0 1G 0 disk └─mpatha 253:0 0 1G 0 mpath nvme0n1 259:0 0 8G 0 disk

├─nvme0n1p1  259:1    0  7.9G  0 part  /
├─nvme0n1p14 259:2    0    4M  0 part  
└─nvme0n1p15 259:3    0  106M  0 part  /boot/efi
$ sudo multipath -ll
mpatha (360000000000000000e00000000010001) dm-0 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 0:0:0:1 sda     8:0   active ready running
 `-+- policy='service-time 0' prio=1 status=enabled
`- 1:0:0:1 sdb     8:16  active ready running

```

How to get the WWID of a device


On Vmware: https://access.redhat.com/solutions/93943

For RHEL7 and RHEL8

# /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sda
36000c2931a129f3c880b8d06ccea1b01

For RHEL6

# scsi_id --whitelisted --replace-whitespace --device=/dev/sda
36000c2931a129f3c880b8d06ccea1b01

For RHEL5

#scsi_id -g -u -s /block/sdb
36000c2931a129f3c880b8d06ccea1b01