Networking and Routing
This page is mainly about useful networking/routing commands for Linux.
/etc/sysconfig/network
Global default gateway configuration is stored in the /etc/sysconfig/network file. This file specifies gateway and host information for all network interfaces.
Routing
- route command - show the IP routing table on Linux.
- ip command (or ip route) - show routing, devices, routing policies, and tunnels on Linux.
- netstat - show network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.
To display the routing tables, you can use the 'route' or 'netstat' commands like so:
$ netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 172.31.0.1 0.0.0.0 UG 0 0 0 eth0 172.31.0.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0 $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.31.0.1 0.0.0.0 UG 100 0 0 eth0 172.31.0.0 0.0.0.0 255.255.240.0 U 100 0 0 eth0
- The -n in option means that you want numerical IP addresses displayed, instead of the corresponding host names.
- Note that the -n option was used with route so it wouldn’t try to resolve any of these IP addresses into hostnames. For one thing, the command runs more quickly, but more important, you don’t want to cloud your troubleshooting with any potential DNS errors.
- The -r option in 'netstat -rn' specifies that you want the routing table. The -n option is similar to that of the route command.
You can also use 'ip route' to display the main routing table. However, note that it is a significantly different output compared to output of 'route'/'netstat -rn':
$ ip route default via 172.31.0.1 dev eth0 proto dhcp metric 100 172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.0.84 metric 100 $ ip route show default via 172.31.0.1 dev eth0 proto dhcp metric 100 172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.0.84 metric 100 $ ip route list default via 172.31.0.1 dev eth0 proto dhcp metric 100 172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.0.84 metric 100
The network 172.31.0.0/20 is available on eth0 with a scope of link. That means that the network is valid and reachable through this device, eth0. As long as link remains good on that device, we should be able to reach any IP address inside of 172.31.0.0/20 through eth0 interface. You can see the source is 172.31.0.84, which means traffic leaving eth0 is coming from 172.31.0.84.
Notice that the destination which is reachable through a gateway appears in the routing table output with the keyword via. This line is reflected in the first line of the 'route -n' output as seen above.
Troubleshooting
If you don’t see a default gateway configured here, and the host you want to reach is on a different subnet (say, web1, which is on another network), then that is the likely cause of you not being able to reach the destination host.
To fix this, either be sure to set the gateway in /etc/network/interfaces on Debian-based systems or /etc/sysconfig/network_scripts/ifcfg-<interface> on Red Hat-based systems, or if you get your IP via DHCP, be sure it is set correctly on the DHCP server and then reset your interface with the following on Debian-based systems:
# Debian-based systems $ sudo service networking restart # Red Hat-based systems # sudo service network restart
Destination host unreachable
root@rhel8:~# ping 192.168.0.1 PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. From 192.168.0.146 icmp_seq=1 Destination Host Unreachable From 192.168.0.146 icmp_seq=2 Destination Host Unreachable From 192.168.0.146 icmp_seq=3 Destination Host Unreachable
If you've ensured that networking is up, but your VM cannot ping another host on the same network due to the error: "destination host unreachable," check to see if there is any asymmetric routing going on by looking at the route table by running one of the following:
- route -n
- netstat -rn
- ip route
$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.31.0.1 0.0.0.0 UG 100 0 0 eth0 172.31.0.0 0.0.0.0 255.255.240.0 U 100 0 0 eth0 172.31.0.0 0.0.0.0 255.255.240.0 U 100 0 0 virb0
- As you can see in the above there are two routes for the destination 172.31.0.0 via the gateway 0.0.0.0. You must delete the virb0, or which ever, so that asymmetric routing doesn't occur.
To delete a route:
- RHEL7:
# route -del -net 172.31.0.0 gw 0.0.0.0 netmask 255.255.240.0 dev virbr0
- RHEL8:
# route del -net 172.31.0.0 gw 0.0.0.0 netmask 255.255.240.0 dev virbr0
How can I make my secondary network interface work in my CentOS or RHEL EC2 instance?
From k u m o 7874
- IF YOU ARE USING IPv6 SIMPLY REPLACE EVERYTHING WITH IPv6 PRIVATE IPs etc. Karthik R. was able to do it.
Short Description
Adding a secondary network interface to a non-Amazon Linux instance causes traffic flow issues. This is because both the primary and the secondary network interfaces are in the same subnet, and there is only one routing table with one gateway. Traffic that comes into the secondary network interface will try to leave the instance using the primary network interface. But this is not allowed, because the secondary IP address does not belong to the MAC address of the primary network interface.
To make the secondary interface work, create a secondary network configuration file, add an additional routing table, and then set up rules in the custom routing table policy database so that traffic for the secondary interface uses the new routing table. To be sure that the new secondary route and rules are brought up in every boot, create and configure a secondary static route file.
Here is a summary of the steps for making the secondary interface work:
- Create a configuration file
- Create a new routing table
- Set rules in the Routing Policy Database
- Create a static route file
Note: For instructions for Ubuntu instances, click here.
Resolution
All procedures must be done with root user privileges. Either become root with `sudo -i` or execute all commands with `sudo`.
Create a secondary network configuration file
1. Get the name of the primary network interface: (The ^ [ [ : d i g i t : ] ] should be all together, but it doesn't show up right in the code box)
ip a | grep ^ [ [ : d i g i t : ] ]
You should see something like this:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
Important: In the previous example, the primary interfaces are named 'eth0', 'eth1' and so on. However, for instances that support enhanced networking, such as the m4 and m5 family type, you might see a naming inconsistency. For example, the primary might be named 'ens3' while the secondary is named 'eth0'. This naming inconsistency happens when the secondary interface is added while the instance is running.
You can avoid a naming inconsistency by adding the interface at launch time, or rebooting the instance. Or, while the interface is running, you can change the name with:
ip link set eth0 name ens4 && ip link set ens4 up
2. Create or edit the secondary network interface 'ifcfg-eth1' configuration file.
vi /etc/sysconfig/network-scripts/ifcfg-eth1
3. Edit the secondary interface file so it is similar to the following. Adapt the edits to your scenario, including your secondary interface MAC address. You can find the secondary interface MAC address using the `ip a` command.
DEVICE=eth1 NAME=eth1 HWADDR=00:00:00:00:00:00 (not necessary.) BOOTPROTO=dhcp ONBOOT=yes TYPE=Ethernet USERCTL=no NM_CONTROLLED=no DEFROUTE=no IPADDR=172.31.0.162 PREFIX=20 # CIDR range of subnet
If you have more than one IP on your secondary interface, configure it as follows:
DEVICE=eth1 NAME=eth1 HWADDR=00:00:00:00:00:00 BOOTPROTO=none ONBOOT=yes TYPE=Ethernet USERCTL=no NM_CONTROLLED=no DEFROUTE=no IPADDR=172.31.0.162 PREFIX=20 # CIDR range of subnet IPADDR1=172.31.13.162 PREFIX1=20 IPADDR2=172.31.1.151 PREFIX2=20
4. For the primary interface to not lose connectivity, be sure that the default gateway remains on the main routing table. To do so, edit the /etc/sysconfig/network file:
vi /etc/sysconfig/network
Add the following line:
GATEWAYDEV=eth0
Important: By default, Red Hat instances come with NetworkManager enabled. It can interfere with networking settings, specially the routing table. You can ensure it doesn't happen by adding NM_CONTROLLED=no to every ifcfg-ethX file you have, including ifcfg-eth0, or you can disable it altogether. Please see How to disable NetworkManager in RHEL?
5. Restart the network:
systemctl restart network
Create a new secondary routing table
The main routing table has ID 254 on Linux. By default, interface traffic is routed based on this table. You must create a new routing table for the secondary interface.
1. Find and take note of your default gateway:
ip route | grep default
2. Create a new routing table for the secondary interface, and then add the default gateway route using the IP of the gateway you found on the previous step. In this example, the new table is ID 1000, and the IP is "172.31.16.1":
ip route add default via 172.31.16.1 dev eth1 table 1000
3. Be sure that table 1000 has a route for every IP present on the secondary interface.
Here's an example for two IPs:
ip route add 172.31.0.162 dev eth1 table 1000 ip route add 172.31.13.162 dev eth1 table 1000
4. Review table 1000 and be sure it is correct:
ip route show table 1000
Set rules in the routing policy database
1. Set rules for every IP present on the secondary interface in the routing policy database so that traffic coming from these IPs is routed according to table 1000.
ip rule add from 172.31.0.162 lookup 1000 ip rule add from 172.31.13.162 lookup 1000
Be sure to check the connectivity with these IPs. If you have Elastic IPs pointing to these IPs, they can be accessed from the public network as well.
Create a secondary static route file
1. To bring the new routes and rules up with every boot, create and configure the 'route-eth1' static route file.
vi /etc/sysconfig/network-scripts/route-eth1
2. Enter the same routes as you entered when creating the secondary routing table at the command line. The difference is that you omit the "ip route add".
default via 172.31.16.1 dev eth1 table 1000 172.31.0.162 dev eth1 table 1000 172.31.13.162 dev eth1 table 1000
3. Create or edit a rule file for rule-eth1:
vi /etc/sysconfig/network-scripts/rule-eth1
Put in the same routes as you entered on the command line in when creating the new secondary routing table. The difference is that you omit the "ip rule add".
from 172.31.0.162 lookup 1000 from 172.31.13.162 lookup 1000
After configuring the CentOS or RHEL IP settings, routing and rules will persist when rebooting the interface. :)
PLEASE NOTE:
- To test, ping the private IP or the corresponding Elastic IP address to validate the connectivity to the IPs on the secondary network interface.
- If you have Elastic IPs that are pointing to these IPs, then they can be accessed from the public network as well.
- The security groups attached to the secondary network interface should allow ICMP traffic for the ping test to return a successful response !!!!!!!.
How can I make my secondary network interface work in my Ubuntu instance?
How can I make my secondary network interface work in my Ubuntu instance?
- NOTE: for whatever reason, the instructions for Ubuntu 18/20 no longer allows me to connect to both private IP addresses of each NIC if the NICs are in the same subnet... likely because of asymmetric routing. Not sure what changed in the doc, if they removed something or if Ubuntu handles the configuration/instructions differently now. WORKAROUND: create a new subnet in the same AZ, create a new ENI/NIC from that new subnet and attach it to the instance as secondary NIC. Now, following the instructions in the doc above, you should now be able to connect to both private IP addresses of both nics.
ip r show table 1000
Other Random Networking tools/info
Command line interface for testing internet latency using speedtest.net: https://github.com/sivel/speedtest-cli
curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -
If you get "No route to host" after spinning up a VM
Check if ipv6 is enabled. Disable it like so:
# vi /etc/sysctl.conf ........................................................................ net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 # sysctl -p
or
# reboot
References:
Learn more about ip route.
[+] https://learning.oreilly.com/library/view/devops-troubleshooting-linux/9780133035513/ch05.html
[+] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-networkscripts-static-routes
Configure Secondary Interface on RHEL8 Workstation
In this scenario, we used a TPLink router for us to test as a separate subnet (192.x.x.x) on the secondary interface while the primary interface was connected to a different subnet.
This is how we set it up: 1. Connected the TPLink ethernet (USB) to the computer 2. Ran ip addr show. Can see the new interface is enp0s20f0u3 with MAC address: 00:24:9b:80:cb:8f 3. Then created the network configuration file:
dzdo cp /etc/sysconfig/network-scripts/ifcfg-enp0s31f6 /etc/sysconfig/network-scripts/ifcfg-enp0s20f0u3
dzdo chmod 644 /etc/sysconfig/network-scripts/ifcfg-enp0s20f0u3
dzdo vi /etc/sysconfig/network-scripts/ifcfg-enp0s20f0u3 TYPE=Ethernet DEVICE=enp0s20f0u3 NAME=enp0s20f0u3 ONBOOT=yes BOOTPROTO=dhcp DHCP_HOSTNAME=cacirh-st (likely not necessary) PROXY_METHOD=none (likely not necessary) BROWSER_ONLY=no (likely not necessary) DEFROUTE=no IPV4_FAILURE_FATAL=no NAME="System enp0s20f0u3" MAC=00:24:9b:80:cb:8f (Ensure the MAC address matches with the output in ip a) USERCTL=no NM_CONTROLLED=yes
dzdo systemctl restart NetworkManager
dzdo nmcli connection up enp0s20f0u3
(I can't remember if this ran successfully or not, but I ran reload in the next command and that worked
dzdo nmcli connection reload
4. Confirm it has IP address now by running: ip addr show 5. Confirmed we could ping the TPLink router: ping 192.16.0.1 6. Also confirmed we could reach the gateway via browser. 7. Confirmed we could reach a website via the primary interface (10.x.x.x): ping somewebsite.com 8. We checked the route table and confirmed the route for the secondary interface was there and it looked good:
route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.228.200.1 0.0.0.0 UG 100 0 0 enp0s31f6 10.118.200.0 0.0.0.0 255.255.248.0 U 100 0 0 enp0s31f6 192.168.0.0 0.0.0.0 255.255.X.0 U 100 0 0 enp0s20f0u3
References: