Linux Tuning for Network Latency Reduction

Table of Contents

1. Understanding Network Latency in Linux
   • 1.1 What is Network Latency?
   • 1.2 Key Sources of Latency in Linux
2. Kernel Parameters for Latency Reduction
   • 2.1 Core Network Buffer Tuning
   • 2.2 TCP/UDP Stack Optimization
   • 2.3 Interrupt and Scheduling Tuning
3. Network Stack Tuning
   • 3.1 Disabling Unnecessary Protocols
   • 3.2 MTU and Fragmentation
   • 3.3 Packet Scheduling Algorithms
4. Interrupt Handling Optimization
   • 4.1 Interrupt Coalescing
   • 4.2 Receive Side Scaling (RSS)
   • 4.3 CPU Affinity for Interrupts
5. TCP Optimization for Low Latency
   • 5.1 Congestion Control Algorithms
   • 5.2 TCP Timestamps and SACK
   • 5.3 Fast Open and Early Retransmit
6. Hardware and Driver Tuning
   • 6.1 Updating Network Drivers
   • 6.2 Offloading Features (TSO, GRO, Checksum)
   • 6.3 SR-IOV and Direct Hardware Access
7. Monitoring and Benchmarking Latency
   • 7.1 Essential Tools for Latency Measurement
   • 7.2 Benchmarking Workflows
8. Best Practices for Sustained Low Latency
9. Conclusion
10. References

1. Understanding Network Latency in Linux

1.1 What is Network Latency?

Network latency is the total time taken for a packet to travel from a source to a destination and back. It comprises four components:

• Propagation Delay: Time for a signal to travel across the physical medium (e.g., fiber, copper).
• Transmission Delay: Time to push a packet onto the network (dependent on packet size and link speed).
• Processing Delay: Time spent by network devices (routers, switches, or the OS) to inspect and forward packets.
• Queuing Delay: Time a packet waits in buffers before being processed (a major target for Linux tuning).

In Linux, processing and queuing delays are the most controllable.

1.2 Key Sources of Latency in Linux

Linux introduces latency through several mechanisms:

• Kernel Network Stack Overhead: The Linux kernel’s network stack (e.g., sk_buff packet processing, protocol parsing) adds processing delay.
• Interrupt Handling: Legacy interrupt models (e.g., shared IRQs) can cause delays if CPU cores are overloaded.
• Buffer Bloat: Large kernel buffers (e.g., net.core.rmem_max) cause packets to queue, increasing queuing delay.
• TCP Congestion Control: Default TCP algorithms (e.g., Cubic) prioritize throughput over low latency.
• Inefficient Driver/Hardware: Outdated drivers or unoptimized hardware offloading (e.g., TSO/GRO) can bottleneck performance.

2. Kernel Parameters for Latency Reduction

The Linux kernel exposes hundreds of tunable parameters via /proc/sys/net/ (configurable with sysctl). Below are critical parameters to reduce latency.

2.1 Core Network Buffer Tuning

Kernel buffers (receive/transmit queues) balance throughput and latency. Large buffers reduce packet loss but increase queuing delay; small buffers minimize delay but risk loss under load.

Example: Set these persistently by adding to /etc/sysctl.conf:

net.core.rmem_max = 131072  
net.core.wmem_max = 131072  
net.core.netdev_budget = 300  

Apply changes with sysctl -p.

2.2 TCP/UDP Stack Optimization

TCP and UDP parameters directly impact latency:

Why? tcp_low_latency=1 tells the kernel to process packets immediately instead of batching, reducing queuing delay.

2.3 Interrupt and Scheduling Tuning

Kernel scheduling can starve network processing. Tune these parameters to prioritize network tasks:

3. Network Stack Tuning

3.1 Disabling Unnecessary Protocols

Linux enables protocols like IPv6, IPX, or AppleTalk by default, even if unused. These add processing overhead.

• Disable IPv6 (if unused):
  Add to /etc/sysctl.conf:

  net.ipv6.conf.all.disable_ipv6 = 1  
  net.ipv6.conf.default.disable_ipv6 = 1  

• Unload Unused Modules:
  Remove modules like ipx, appletalk, or netrom via rmmod or blacklist them in /etc/modprobe.d/blacklist.conf.

3.2 MTU and Fragmentation

The Maximum Transmission Unit (MTU) determines packet size. Larger MTUs reduce fragmentation (good for throughput) but increase transmission delay. For low latency:

• Use MTU 1500 (Ethernet default) to avoid jumbo frames (which add processing delay).
• Disable fragmentation for UDP (critical for real-time apps like VoIP):

  net.ipv4.ip_no_pmtu_disc = 1  # Disable Path MTU Discovery (PMTUD)  

3.3 Packet Scheduling Algorithms

The Linux kernel uses packet schedulers to order packets in transmit queues. For low latency, avoid schedulers that prioritize fairness over speed:

4. Interrupt Handling Optimization

Network interface cards (NICs) generate interrupts to signal packet arrival. Poorly managed interrupts cause CPU bottlenecks.

4.1 Interrupt Coalescing

NICs often batch interrupts to reduce CPU load (interrupt coalescing), but this increases latency. Use ethtool to tune coalescing:

# Reduce coalescing (lower latency, higher CPU)  
ethtool -C eth0 rx-usecs 10 rx-frames 1 tx-usecs 10 tx-frames 1  

# Verify settings  
ethtool -c eth0  

• rx-usecs 10: Interrupt after 10µs of idle time.
• rx-frames 1: Interrupt after 1 packet (minimal batching).

4.2 Receive Side Scaling (RSS)

RSS distributes incoming packets across CPU cores, preventing a single core from being overwhelmed. Enable RSS via ethtool:

# Check RSS support  
ethtool -k eth0 | grep rx-vlan-filter  

# Enable RSS (if supported)  
ethtool -K eth0 rx-vlan-filter on  

4.3 CPU Affinity for Interrupts

Bind NIC interrupts to dedicated CPU cores to avoid context switching. Identify interrupts with:

grep eth0 /proc/interrupts  
# Example output:  123:  100000   0   0   0  IR-PCI-MSI 1048576-edge  eth0  

Bind interrupt 123 to CPU core 2:

echo 2 > /proc/irq/123/smp_affinity_list  

5. TCP Optimization for Low Latency

TCP, the workhorse of reliable networking, introduces latency via congestion control, retransmissions, and handshakes.

5.1 Congestion Control Algorithms

Default TCP algorithms like Cubic prioritize throughput. For low latency, use:

5.2 TCP Timestamps and SACK

• TCP Timestamps: Enable to improve RTT estimation (reduces retransmit delay):

  net.ipv4.tcp_timestamps = 1  

• Selective Acknowledgment (SACK): Allows retransmitting only lost packets (not entire windows):

  net.ipv4.tcp_sack = 1  

5.3 Fast Open and Early Retransmit

• TCP Fast Open (TFO): Reduces handshake latency by sending data in the SYN packet:

  net.ipv4.tcp_fastopen = 3  # Enable for client and server  

• Early Retransmit: Retransmit lost packets faster:

  net.ipv4.tcp_early_retrans = 1  

6. Hardware and Driver Tuning

6.1 Updating Network Drivers

Outdated NIC drivers often have latency bugs. Use the latest drivers from the vendor (e.g., Intel’s ixgbe, Mellanox’s mlx5):

# Check current driver  
ethtool -i eth0  

# Update via package manager (e.g., Debian/Ubuntu)  
apt install linux-modules-extra-$(uname -r)  

6.2 Offloading Features

NICs can offload tasks (checksum, TSO, GRO) from the CPU, but misconfigured offloading increases latency:

Enable/disable with ethtool:

ethtool -K eth0 tso off gro off tx-checksum-ipv4 on  

6.3 SR-IOV for Virtualization

In virtualized environments (e.g., KVM, VMware), Single Root I/O Virtualization (SR-IOV) bypasses the hypervisor, reducing latency by giving VMs direct NIC access. Enable SR-IOV in the BIOS and configure via ip link:

# Create 4 virtual functions (VFs) on eth0  
echo 4 > /sys/class/net/eth0/device/sriov_numvfs  

# Assign VF to VM (via libvirt or virt-manager)  

7. Monitoring and Benchmarking Latency

Tuning is ineffective without measuring baseline and post-tuning latency.

7.1 Essential Tools

7.2 Benchmarking Workflow

1. Baseline Measurement: Run ping, iperf3, and tcptrace to establish baseline latency (e.g., average RTT, 99th percentile).
2. Apply Tuning: Modify one parameter at a time (e.g., tcp_low_latency).
3. Re-test: Re-run benchmarks to isolate the impact of each change.
4. Iterate: Roll back changes that increase latency or CPU usage.

8. Best Practices for Sustained Low Latency

• Incremental Changes: Tune one parameter at a time to avoid cascading failures.
• Staging Environment: Test changes in staging before production (latency tuning can break throughput).
• Monitor Post-Deployment: Use tools like Prometheus + Grafana to track latency metrics (e.g., node_network_transmit_latency).
• Document Everything: Log all changes (e.g., sysctl values, ethtool settings) for rollback.
• Use Real-Time Kernels: For ultra-low latency (e.g., <1ms), use a PREEMPT_RT kernel (e.g., linux-image-rt-amd64 on Debian).

Conclusion

Reducing network latency in Linux requires a holistic approach, combining kernel parameter tuning, interrupt management, TCP optimization, and hardware offloading. By prioritizing minimal queuing delay, optimizing interrupt handling, and leveraging modern TCP algorithms, you can achieve sub-millisecond latency for even the most demanding applications.

Remember: latency tuning is iterative. Always measure, test, and validate changes to avoid trading latency for stability or throughput.

References

1. Linux Kernel Documentation: Networking Guide
2. Red Hat: Tuning TCP for Low Latency
3. Intel: Ethernet Controller Tuning Guide
4. Linux Foundation: PREEMPT_RT Patch Set
5. ethtool Man Page: ethtool(8)
6. TCP Congestion Control: BBR vs Cubic