Performing Network Diagnostics with MTR

Today, we’re looking at how to diagnose network issues. We’ll use a specialized tool called MTR, which is basically a combination of two common network tools: ping, and traceroute. MTR lets you identify if and how a breakdown occurs between your server and a server you’re trying to reach.

MTR works on every Linux distribution we tested, including Ubuntu 16 & 18 as well as Debian 8 & 9.

Step 1: Install MTR

In most cases you can install MTR just by using the package manager. For example, on Ubuntu the command is just:

sudo apt-get install mtr

Step 2: Generate an MTR Report

MTR works by trying to reach the desired site multiple times and summarizing the results in a report. To test your connection to Google, for example, you can run:

mtr –report nexonhost.com

And check the output. MTR will try 10 times to reach the site, and collect summary information on the steps it took to get there.

Step 3: Read the Report

The output from MTR contains a few different pieces of information. It should look something like this:

root@test:~# mtr –report nexonhost.com

Start: 2021-12-13T14:56:55-0800

HOST: test Loss% Snt Last Avg Best Wrst StDev

1.|– 5.254.113.197 0.0% 10 0.7 0.6 0.4 0.8 0.1

Let’s unpack this a little. Each line tells you a single hop your network traffic took between your server and the destination server. The “Loss%” line tells you how many packets were lost in between steps.

The “Snt” column tells you how many packets were sent. By default, MTR will send 10 packets, which is usually enough to identify issues. If you’re seeing only intermittent network issues, try bumping the number of packets up by passing the -c flag with the command. For example, to run with 100 packets run:

mtr –report -c 100 nexonhost.com

Finally, the last few columns tell you summary statistics for the latency of network traffic. The most informative are the average and the worst, which should give you an idea of what typical traffic is like.

Step 4: Diagnose the Issue

If you’re having network issues, there are a few ways that MTR can help you see what’s going on. One common pitfall is when one particular link in the network is experiencing high packet loss. This appears in the report as a single row that stands out in the “Loss%” column, like below:

root@test:~# mtr –report nexonhost.com

Start: 2021-12-13T14:22:37-0800

HOST: test Loss% Snt Last Avg Best Wrst StDev

1.|– 5.254.113.193 0.0% 10 24.1 3.1 0.5 24.1 7.4

2.|– 142.250.62.151 0.0% 10 26.2 26.2 26.1 26.3 0.1

3.|– fra16s48-in-f14.1e100.net 3.0% 10 26.0 26.1 26.0 26.3 0.1

Here, it’s pretty easy to identify the server that’s causing the issue, which is a lossy hop along the chain. Once you’ve identified the issue, you’ll need to talk to your network administrator and supply them with the IP address of the lossy link to fix the problem.

One important note: packet loss does not always indicate that there’s a problem. Many switches and routers will rate limit requests coming from MTR to prevent network slowdowns. You can identify this type of rate limiting packet loss when the Loss% is only in one row of the report, but later rows have 0%

Another common issue is when outbound traffic is fine, but the final line has a 100% loss. This type of report looks like this:

root@test:~# mtr –report nexonhost.com

Start: 2021-12-13T14:22:37-0800

HOST: test Loss% Snt Last Avg Best Wrst StDev

1.|– 5.254.113.193 0.0% 10 24.1 3.1 0.5 24.1 7.4

…

4.|– fra16s48-in-f14.1e100.net 100.0% 10 26.0 26.1 26.0 26.3 0.1

Typically, this is indicative of a firewall or IP table configuration issue. The traffic is successfully reaching the destination, but the response coming back from the destination server is getting lost. To fix this, you’ll need to change your firewall settings to allow the response to be received.

Advanced MTR Techniques

For advanced users, MTR includes several additional diagnostic tools that may come in handy. First, as we mentioned above, you can vary the number of packets MTR sends when generating a report with the -c flag. You can also vary the rate at which MTR sends the packets using the -i flag. The default is 1 second.

For example, to send 100 packets at 10 packets per second, use the command:

mtr –report -i 0.1 -c 100 nexonhost.com

If you’re being rate limited, you may also want to send packets more slowly. For example, once every two seconds:

mtr –report -i 2 -c 100 nexonhost.com