Insider Tips for Tuning the Linux Kernel

Table of Contents

1. Understanding Your Workload: The First Step
2. Kernel Parameters: The Basics
   • Key Subsystems to Tune
   • Tools for Modifying Parameters
3. Compiling a Custom Kernel: When and How
   • Why a Custom Kernel?
   • Step-by-Step Compilation Guide
4. Tuning for Specific Workloads
   • Web Servers (Nginx/Apache)
   • Databases (PostgreSQL/MySQL)
   • HPC and Scientific Computing
5. Stability, Testing, and Rollback Strategies
   • Testing Tools
   • Monitoring Post-Tuning
   • Rollback Plans
6. Advanced Tips: Beyond Basic Tuning
   • Kernel Live Patching
   • BPF: Dynamic Tracing and Tuning
7. References

1. Understanding Your Workload: The First Step

Before tweaking a single parameter, you must measure your system’s current behavior and identify bottlenecks. Without this, you’re guessing—and guessing often leads to worse performance.

Key Metrics to Measure

• CPU: Usage, load average, context switches, cache misses.
• Memory: Usage, swap activity, page faults, OOM (Out-of-Memory) events.
• I/O: Disk throughput, latency, queue depth, read/write patterns.
• Network: Bandwidth, latency, TCP/UDP connection counts, packet loss.

Tools for Profiling

• top/htop: Real-time CPU/memory usage (great for quick overviews).
• vmstat/iostat: System-wide memory, I/O, and CPU statistics.
  Example: iostat -x 5 (disk I/O details every 5 seconds).
• perf: Linux’s built-in performance analysis tool for tracing kernel/process activity.
  Example: perf record -g -p <pid> (record call graphs for a process).
• sar (System Activity Reporter): Collects and reports long-term system metrics.
• nmon: Interactive tool for monitoring CPU, memory, I/O, and network in real time.
• bpftrace: Advanced tracing for deep insights (e.g., bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); }' to track syscalls).

Tip: Always establish a “baseline” of metrics before making changes. Compare post-tuning results to this baseline to validate improvements.

2. Kernel Parameters: The Basics

Most kernel tuning starts with adjusting runtime parameters stored in /proc/sys/ (virtual filesystem) or managed via sysctl. These parameters control behavior for subsystems like memory, networking, and I/O.

2.1 Key Subsystems to Tune

Memory Management

The kernel’s memory subsystem is critical for performance. Here are essential parameters:

Networking

For systems handling high network traffic (e.g., web servers, proxies):

I/O Scheduling

The I/O scheduler determines how the kernel orders disk requests. Choose the right scheduler for your workload:

To change the scheduler for a disk (e.g., /dev/sda):

echo mq-deadline > /sys/block/sda/queue/scheduler  

2.2 Tools for Modifying Parameters

• Temporary Changes (lost on reboot): Use sysctl -w or write directly to /proc/sys/.
  Example: sysctl -w vm.swappiness=10

• Permanent Changes: Edit /etc/sysctl.conf or drop files in /etc/sysctl.d/ (e.g., /etc/sysctl.d/99-custom.conf).
  Example entry:

  vm.swappiness = 10  
  net.core.somaxconn = 1024  

  Apply with sysctl -p (or reboot).

3. Compiling a Custom Kernel: When and How

Most users won’t need a custom kernel, but for specialized use cases (e.g., embedded systems, HPC, or removing bloat), compiling your own can yield benefits like:

• Smaller kernel size (faster boot, less memory usage).
• Support for custom hardware/drivers.
• Optimizations for specific CPU architectures (e.g., CONFIG_MK8 for AMD K8).

3.1 Why a Custom Kernel?

• Remove Unused Modules: Distro kernels include drivers for hundreds of devices (e.g., sound cards, printers) you may not need. Removing them reduces memory overhead.
• Enable Experimental Features: Access cutting-edge kernel features (e.g., new file systems like btrfs improvements).
• Optimize for Workload: Enable CONFIG_PREEMPT_RT for real-time systems, or CONFIG_HZ=1000 for lower latency (vs. default 250/300Hz).

3.2 Step-by-Step Compilation Guide

Prerequisites

• Kernel source code (from kernel.org or your distro’s repo).
• Build tools: gcc, make, libncurses-dev (for menuconfig), bc, flex, bison.

Step 1: Obtain the Source

# Download from kernel.org (e.g., 6.5.0)  
wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.0.tar.xz  
tar -xf linux-6.5.0.tar.xz  
cd linux-6.5.0  

Step 2: Configure the Kernel

Use a baseline config and customize:

# Start with your current kernel's config  
cp /boot/config-$(uname -r) .config  

# Simplify: Remove modules not needed by your system (insider tip!)  
make localmodconfig  # Automatically disables unused modules  

# Edit config interactively (optional)  
make menuconfig  

In menuconfig, key options to consider:

• General Setup > Local version: Add a custom suffix (e.g., -custom) to identify your kernel.
• Processor type and features > Processor family: Select your CPU (e.g., “Intel Core2/Newer Xeon”).
• Device Drivers: Disable unused drivers (e.g., “Sound card support” if headless).

Step 3: Compile and Install

# Compile (use -jN where N = number of CPU cores + 1)  
make -j$(nproc)  

# Install modules  
sudo make modules_install  

# Install kernel image and headers  
sudo make install  

# Update bootloader (GRUB example)  
sudo update-grub  

Step 4: Reboot and Test

Reboot and select your custom kernel from the GRUB menu. Verify with uname -r.

Warning: Always keep a backup kernel (e.g., the distro’s default) in case your custom kernel fails to boot.

4. Tuning for Specific Workloads

Kernel tuning varies drastically based on your workload. Below are targeted tips for common use cases.

4.1 Web Servers (Nginx/Apache)

Web servers thrive on low latency and high connection throughput. Focus on:

• TCP Tuning:

  • net.ipv4.tcp_tw_reuse = 1: Reuse TIME_WAIT sockets to handle more connections.
  • net.ipv4.tcp_max_tw_buckets = 5000: Limit TIME_WAIT sockets to prevent resource exhaustion.
  • net.ipv4.tcp_keepalive_time = 300: Reduce keepalive interval to detect dead connections faster.

• File Descriptors:
  Increase the maximum open file descriptors (web servers open many sockets/files). Edit /etc/security/limits.conf:

  * soft nofile 100000  
  * hard nofile 100000  

• I/O Scheduler: Use mq-deadline or noop for SSDs to minimize latency.

4.2 Databases (PostgreSQL/MySQL)

Databases are memory and I/O intensive. Prioritize:

• Memory:

  • vm.swappiness = 10: Minimize swapping (databases prefer in-memory data).
  • Huge Pages: Enable vm.nr_hugepages to reduce TLB (Translation Lookaside Buffer) misses. For PostgreSQL:

    echo 4096 > /proc/sys/vm/nr_hugepages  # 4096 * 2MB = 8GB hugepages  

    Update postgresql.conf: huge_pages = on.

• I/O:

  • vm.dirty_ratio = 30 and vm.dirty_background_ratio = 15: Allow more dirty pages before flushing to disk (reduces I/O spikes).
  • Use deadline scheduler for rotational disks (minimizes read latency).

4.3 HPC and Scientific Computing

HPC workloads (e.g., simulations, machine learning) demand CPU/memory bandwidth and low latency.

• CPU Scheduler: Disable CONFIG_PREEMPT (unless real-time is needed) to reduce context-switch overhead.
• Memory: Enable CONFIG_TRANSPARENT_HUGEPAGES=always to reduce TLB misses for large datasets.
• Network: Use TCP BBR congestion control (instead of CUBIC) for high-bandwidth, long-distance links:

  echo bbr > /proc/sys/net/ipv4/tcp_congestion_control  

5. Stability, Testing, and Rollback Strategies

Tuning can break things. Always test changes in a staging environment before production, and have a rollback plan.

5.1 Testing Tools

• stress-ng: Simulate CPU, memory, I/O, or network load.
  Example: stress-ng --cpu 8 --memory 4G --io 4 --timeout 300s (8 CPU cores, 4GB memory, 4 I/O workers for 5 minutes).
• sysbench: Benchmark CPU, memory, I/O, and databases.
  Example: sysbench memory --memory-block-size=1M --memory-total-size=10G run (test memory throughput).
• fio: Advanced I/O benchmarking (e.g., simulate database workloads).

5.2 Monitoring Post-Tuning

Use tools like Prometheus + Grafana or Zabbix to track long-term metrics. Key metrics to watch:

• Latency (CPU, disk, network).
• Error rates (e.g., dmesg for kernel errors, journalctl -k -p err).
• Resource utilization (CPU, memory, I/O saturation).

5.3 Rollback Plans

• Backup /etc/sysctl.conf and /boot/ before modifying kernels/parameters.
• For kernel upgrades: Keep the old kernel in grub.cfg (never run update-grub with only the new kernel).
• For parameters: Use sysctl -p /etc/sysctl.conf.bak to revert to a saved config.

6. Advanced Tips: Beyond Basic Tuning

6.1 Kernel Live Patching

Live patching allows updating the kernel without rebooting, critical for systems requiring 24/7 uptime. Tools include:

• kpatch (Red Hat/CentOS): Applies patches via kernel modules.
• kgraft (SUSE): Similar to kpatch, integrated with SUSE Linux Enterprise.
• Canonical Livepatch (Ubuntu): Free for personal use, commercial for enterprise.

Example (Ubuntu):

sudo snap install canonical-livepatch  
sudo canonical-livepatch enable <token>  # Get token from https://ubuntu.com/livepatch  

6.2 BPF: Dynamic Tracing and Tuning

BPF (Berkeley Packet Filter) is a powerful framework for tracing kernel/process behavior and even modifying runtime behavior (e.g., with tc for traffic control).

Insider Tip: Use bpftrace to identify bottlenecks before tuning. For example, trace disk I/O latency:

bpftrace -e 'tracepoint:block:block_rq_complete { @us[args->rwbs] = hist(args->latency / 1000); }'  

This shows a histogram of I/O latency in microseconds, helping you decide if I/O scheduler tweaks are needed.

7. References

• Linux Kernel Documentation
• sysctl.conf(5) Man Page
• Performance Tuning Guide (Red Hat)
• Linux Kernel in a Nutshell (book by Greg Kroah-Hartman)
• BPF Trace Examples
• Kernel Compilation Guide (Ubuntu)

By following these tips, you’ll be able to tune the Linux kernel to squeeze out maximum performance for your specific workload—all while maintaining stability. Remember: measure first, tweak second, and test always. Happy tuning! 🐧