Emulating Network Latency And Packet Loss In Linux

Recently, I had a project requirement to simulate sporadic network latency and packet loss amongst a cluster of server VM instances (Ubuntu 18.04 LTS) on GCP (Google Cloud Platform).  But the challenge here was that this being GCP, there was no underlying network infrastructure that I had access to whereby I could simulate latency through tweaking QoS parameters or generate packet loss using policers.  So, the problem statement boiled down to this fundamental question:

How can I quickly and easily simulate network latency and packet loss directly on a Linux (Ubuntu) server?

I came across a very useful and versatile utility called "NetEm" (which is short for "Network Emulation").  Much more details about this utility can be found here: https://wiki.linuxfoundation.org/networking/netem.   In this blog post, we are going to explore some of the basics.

What Is NetEm?

In a nutshell, NetEm allows a user to add delay, packet loss and other network characteristics (eg. packet duplication, re-ordering and corruption) to outgoing packets (ie. egress packets) from a specific network interface.  Note that if you're running a version of Ubuntu that uses Linux kernel 2.6 or higher, then NetEm is already enabled in the kernel.

The NetEm utility is controlled by the Linux command line utility called "tc" (which is short for "Traffic Control"), which comes bundled with "iproute2", which is a collection of utilities for controlling TCP/IP network traffic in Linux.  The good news is that if your Linux kernel is at version 2.6 or higher, then iproute2 should be included.  If not, then you can always install it by issuing the following command: sudo apt-get install iproute2.

Some Examples Of Using NetEm

Let's run through a few simple examples of using NetEm and tc.  The generic command syntax for a Linux traffic control rule looks like the following:

sudo tc qdisc <OP_TYPE> dev <INTF_NAME> root netem <PROPERTY> <DELAY>

• sudo: be sure to run this command as root.
• tc: invoke the Traffic Control utility.
• qdisc: qdisc is a scheduler (also called a "queuing discipline") which arranges packets for output. The default behaviour is FIFO (first-in first-out).
• <OP_TYPE>: this is the operation type to be performed. Supported values are: add, del or change.
• dev <INTF_NAME>: INTF_NAME is the interface name to which the tc rule will apply to (eg. ens4).
• root: the rule will be applied to the egress scheduler (aka. "root qdisc"), as opposed to the ingress scheduler (ake. "ingress qdisc").
• netem: use NetEm to emulate a network property.
• PROPERTY: several network properties can be emulated, including:
  • delay: adds a fixed amount of delay (in ms) to egress packets on a specified interface. Optional parameters here can be used to add a delay variation (jitter).
  • loss: adds random packet loss (in %) to egress packets on a specified interface.
  • Note: There are several other configurable properties, but this blog post will focus on delay and loss, for brevity.

Once again, please note that Linux traffic control rules (using tc and NetEm) are applied to the egress scheduler of an interface.

1. Simulating Network Latency

In this first example, we are going to simulate network latency by adding a fixed amount of delay (50ms) to all egress packets on a particular network interface (in all of our examples, the interface is "ens4").  The command to do this looks like the following:

sudo tc qdisc add dev ens4 root netem delay 10ms

First, let's take a look at the network latency prior to making this change by pinging a remote destination IP address, as shown in the screenshot below.  In the example below, we are pinging between Ubuntu VMs in a GCP cluster, so the latency (highlighted below) is very small, and averages at around 0.6ms.

Next, let's verify which interface is being used to route packets to destination 10.128.0.20.  To do this, we run the netstat -nr command.  As shown in the screenshot below, we can see that interface "ens4" is being used to route packets to the destination.

Now that we have identified the interface being used (ens4), let's now add 50ms latency to all egress packets on this interface using the command from above.  As soon as we apply this command, we can see from the screenshot below that an extra 50ms of delay has indeed been added to all egress traffic on the interface ... we now see an average of 50.68ms of delay.

Finally, to remove the added delay and restore the network latency to its true value, we simply remove the rule using the following command:

sudo tc qdisc del dev ens4 root netem

As we see in the screenshot below, once we remove the traffic control rule, the network latency returns back to its normal average value of around 0.7ms.

2. Simulating Packet Loss

In this second example, we are going to simulate packet loss by adding a fixed loss probability of 10% to all egress packets on interface "ens4".  The command to do this looks like the following:

sudo tc qdisc add dev ens4 root netem loss 10%

As shown in the screenshot below, we issue the command and attempt a large number of pings to a remote IP address.  At the bottom of the ping results (highlighted in the screenshot below), we can see a packet loss of around 9%, which is very close to the loss probability we specified in the command above.  Running a larger number of ping samples will get us even closer to this value of 10%.

Note that it is not necessary to delete a traffic control rule first and re-adding it in order to change its value.  For this, we can use the "change" op_type.  For example, let's say we wanted to change the packet loss probability from 10% to 5%, we can issue the following command:

sudo tc qdisc change dev ens4 root netem loss 5%

Finally, to remove the packet loss rule entirely and return back to normal state, we can use the following command:

sudo tc qdisc del dev ens4 root netem

-- Jag --