MTR
https://www.linode.com/docs/networking/diagnostics/diagnosing-network-issues-with-mtr/
https://www.tecmint.com/mtr-a-network-diagnostic-tool-for-linux/
https://www.datapacket.com/blog/mtr-diagnose-network-issues
ADD THIS::: https://www.datapacket.com/blog/mtr-diagnose-network-issues
BEST TO RUN WITH TCP FLAG INSTEAD OF DEFAULT ICMP: [May 14, 2020, 1:41 PM] Bawezir, Ahmed: "mtr -P <tcp port> -T -w <destination ip>"
When analyzing an MTR output, I look for two things: loss and latency. In your output, we can see there was major packet loss on one hop, hop number 10. nyk-bb2-link.telia.net (62.115.113.20) at 87.5%. This can give the illusion of packet loss when in actuality, there is no loss. In situations where you see packet loss in an intermediate hop, you should then take a look at the next subsequent hop to determine if the loss you are seeing is a real issue. If the subsequent hop shows a loss of 0.0%, then you are likely seeing ICMP rate limiting and not actual loss. In your output this is exactly what we see. If there is an actual issue, the loss would most likely continue to accumulate over the next subsequent hops.
[+] A packet loss at one hop on the path doesn't necessarily mean there is something wrong with the network routing, neither does it mean that the path is congested. If you see one hop showing a loss, then it is usually due to ICMP limits set on that particular router (ICMP rate limiting). It is quite a common practice among some service providers to rate limit the ICMP traffic that MTR uses on a router to not overload router CPU by ICMP requests. Some service provider routers have policies that simply either drop ICMP messages S(especially echo request/pings) or they do not care if they will forward those ICMP packets so they might leave them at the bottom of the packets queue until the router decides they stayed there too long and drop them. Most routers on the INTERNET prefer to drop ICMP packets than to forward them and usually they do not respond to pings (e.g. ICMP flood).
[+] That said, Hop number 10. nyk-bb2-link.telia.net (62.115.113.20) is outside of AWS network and is not an AWS device. From my understanding, Telia is a Swedish telecommunication service provider.
By default, MTR trace uses ICMP echo. A better method of using MTR to trace packets is to specify a TCP trace over a specified port. You can use the flags --tcp to specify TCP, and -P to specify a port. Please see the command below as an example of doing a TCP trace over port 6667 to your instance (sa-east-1) from my instance (us-east-1):
Test performance using MTR
The Linux MTR command provides continual, updated output, which enables you to analyze network performance. This diagnostic tool combines the functionality of traceroute and ping utilities. Most Linux distributions come with traceroute and MTR already installed. However, you can also download it from your distribution’s software package manager.
To install MTR, run the following commands:
Amazon Linux:
sudo yum install mtr
Ubuntu:
sudo apt-get install mtr
To test your network's performance using MTR, run this test bidirectionally between the public IP address of your EC2 instances and your on-premises host. The path between nodes on a TCP/IP network can change if the direction is reversed. Therefore, it's important to obtain MTR results for both directions. You can use a TCP-based trace instead of ICMP, because most internet devices deprioritize ICMP-based trace requests.
Review your packet loss. Packet loss on a single hop usually doesn't indicate an issue. The loss can be the result of a control plane policy that causes the "ICMP time exceeded" messages to be dropped. If you notice sustained packet loss until the destination hop, or packet loss over several hops, this loss might indicate a problem.
Note: It's common to see a few requests time out.
ICMP-based MTR:
mtr -n -c 200 <Public/private IP EC2 instance/on-premises host> --report
TCP-based MTR:
mtr -n -T -c 200 <Public/private IP EC2 instance/on-premises host> --report
The argument -T performs a TCP-based MTR, and the --report option puts MTR into report mode. MTR runs for the number of cycles specified by the -c option. Print the statistics, and then exit.
Note: There is a known issue with some versions of MTR where the final hop reports an incorrect value if TCP is used.