funwithlinux guide

Step-by-Step Guide to Linux System Tuning

Linux is renowned for its flexibility, stability, and performance, but out-of-the-box configurations are often generalized to work across diverse hardware and workloads. **System tuning** is the process of optimizing these default settings to align with your specific use case—whether you’re running a high-traffic web server, a database, a desktop workstation, or an embedded device. By fine-tuning components like the CPU, memory, disk I/O, and network, you can unlock significant gains in speed, efficiency, and reliability. This guide will walk you through a structured, step-by-step approach to Linux system tuning. We’ll start with assessing your current system performance, then dive into tuning critical subsystems (CPU, memory, disk, network), and finally cover monitoring and maintenance to keep your system optimized long-term.

Table of Contents

  1. Prerequisites
  2. Step 1: Assess Current System Performance
  3. Step 2: CPU Tuning
  4. Step 3: Memory Tuning
  5. Step 4: Disk I/O Tuning
  6. Step 5: Network Tuning
  7. Step 6: Kernel Parameter Tuning
  8. Step 7: Services and Daemons Optimization
  9. Step 8: Security Considerations
  10. Step 9: Monitoring and Maintenance
  11. Conclusion
  12. References

Prerequisites

Before starting, ensure you have:

  • A Linux system (tested on Ubuntu 22.04 LTS, CentOS Stream 9, and Debian 12; most steps work across distros).
  • Root access (via sudo or direct root login) to modify system settings.
  • Basic familiarity with Linux command-line tools (e.g., htop, sysctl, systemctl).
  • A backup of critical data (e.g., using rsync or tar) and configuration files (e.g., /etc/sysctl.conf, /etc/fstab)—tuning can cause instability if misconfigured.
  • Optional: A non-production environment to test changes before applying them to live systems.

Step 1: Assess Current System Performance

Before tuning, you need to identify bottlenecks. Use these tools to measure baseline performance:

Key Metrics to Monitor

  • CPU usage: User vs. system CPU, load average, and idle time.
  • Memory usage: Total/used/free RAM, swap usage, and cache behavior.
  • Disk I/O: Read/write throughput, latency, and I/O wait percentage.
  • Network: Bandwidth usage, packet loss, and TCP/UDP connection stats.

Tools for Assessment

1. CPU and Memory: htop

htop is an interactive process viewer that shows real-time CPU, memory, and swap usage.

  • Install: sudo apt install htop (Debian/Ubuntu) or sudo dnf install htop (RHEL/CentOS).
  • Run: htop.
  • Key metrics:
    • %CPU: User (us), system (sy), idle (id), and I/O wait (wa) percentages. High wa indicates disk I/O bottlenecks.
    • Load average: 1/5/15-minute averages (ideally < number of CPU cores).
    • Mem/Swap: Total, used, and free memory/swap.

2. Disk I/O: iostat

iostat (from the sysstat package) reports disk I/O statistics.

  • Install: sudo apt install sysstat or sudo dnf install sysstat.
  • Run: iostat -x 5 (5-second intervals, -x for extended stats).
  • Key metrics:
    • %iowait: Time CPU spends waiting for I/O (high values = disk bottleneck).
    • r/s, w/s: Reads/writes per second.
    • rkB/s, wkB/s: Read/write throughput in KB/s.
    • avgqu-sz: Average I/O queue length (should be < 1 per disk).

3. Network: iftop and ss

  • iftop: Real-time network bandwidth usage per interface.
    • Install: sudo apt install iftop or sudo dnf install iftop.
    • Run: sudo iftop -i eth0 (replace eth0 with your interface).
  • ss: Shows active network connections (faster than netstat).
    • Run: ss -tuln (TCP/UDP listeners) or ss -s (summary stats).

4. System-Wide: systemd-analyze

For boot-time performance: systemd-analyze blame shows services slowing down boot.

Action: Record baseline metrics (e.g., CPU %wa, memory usage, disk throughput) to compare after tuning.

Step 2: CPU Tuning

CPU tuning focuses on optimizing core utilization, frequency scaling, and process scheduling.

1. CPU Governor

Linux uses “governors” to adjust CPU frequency dynamically. Common governors:

  • performance: Runs CPU at maximum frequency (best for servers).
  • powersave: Prioritizes energy efficiency (default for laptops/desktops).
  • ondemand: Scales frequency based on load (balance of performance/power).

Check current governor:

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Set to performance (server workloads):

# Temporarily (resets on reboot)
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Permanently (Debian/Ubuntu): Edit /etc/default/cpufrequtils
sudo nano /etc/default/cpufrequtils
# Add: GOVERNOR="performance"
sudo systemctl restart cpufrequtils

2. CPU Affinity

Pin processes to specific CPU cores to reduce cache misses. Use taskset for running processes or numactl for NUMA systems.

  • Example: Pin process ID 1234 to cores 0 and 1: 
    taskset -cp 0,1 1234

3. Hyper-Threading

Hyper-threading (HT) allows 2 threads per CPU core but may reduce performance for CPU-bound workloads (e.g., databases).

  • Check if HT is enabled: grep -E 'siblings|cpu cores' /proc/cpuinfo (siblings > cpu cores = HT on).
  • Disable HT (temporarily): 
    echo 0 | sudo tee /sys/devices/system/cpu/cpu*/online  # Replace * with HT cores (e.g., 1,3,5...)
    Note: Permanently disable via BIOS/UEFI for production.

Step 3: Memory Tuning

Optimize memory usage to reduce swapping and improve cache efficiency.

1. Swappiness

vm.swappiness controls how aggressively the kernel swaps memory to disk (0 = swap only when out of memory; 100 = swap frequently).

  • Default: 60 (too high for servers with ample RAM).
  • Recommended: 10-20 for servers, 60 for desktops.

Check current value:

sysctl vm.swappiness

Set temporarily:

sudo sysctl vm.swappiness=10

Set permanently: Edit /etc/sysctl.conf:

sudo nano /etc/sysctl.conf
# Add: vm.swappiness=10
sudo sysctl -p  # Apply changes

2. Dirty Memory Ratios

vm.dirty_ratio and vm.dirty_background_ratio control when the kernel flushes “dirty” (unwritten) memory to disk.

  • dirty_background_ratio: Percentage of memory that triggers background writeback (default: 10).
  • dirty_ratio: Percentage that triggers synchronous writeback (default: 20).

Tune for high-throughput workloads (e.g., file servers):

sudo sysctl vm.dirty_background_ratio=5
sudo sysctl vm.dirty_ratio=10
# Add to /etc/sysctl.conf to persist:
# vm.dirty_background_ratio=5
# vm.dirty_ratio=10

3. Huge Pages

Huge pages (2MB/1GB) reduce TLB (Translation Lookaside Buffer) misses for memory-intensive apps (e.g., databases like PostgreSQL, Redis).

  • Check available huge pages: grep HugePages_Total /proc/meminfo.
  • Allocate 1024 huge pages (2MB each = 2GB total): 
    sudo sysctl vm.nr_hugepages=1024
# Persist: Add vm.nr_hugepages=1024 to /etc/sysctl.conf

Step 4: Disk I/O Tuning

Disk I/O is often the slowest subsystem; optimize with filesystem choices, mount options, and I/O schedulers.

1. Filesystem Choice

  • Ext4: Stable, default for most systems. Good for general use.
  • XFS: Better for large files (e.g., media servers) and high-throughput workloads.
  • Btrfs: Supports snapshots and RAID but has higher overhead.

Tip: Use mkfs.xfs instead of mkfs.ext4 for new disks in high-I/O environments.

2. Mount Options

Tweak /etc/fstab to optimize read/write performance.

  • noatime: Disable access time logging (reduces writes).
  • nodiratime: Disable directory access time logging.
  • discard: Enable TRIM for SSDs (automatically free unused blocks).

Example /etc/fstab entry for an SSD:

UUID=abc123 /mnt/data xfs defaults,noatime,nodiratime,discard 0 0
  • Apply changes: sudo mount -o remount /mnt/data.

3. I/O Schedulers

The I/O scheduler orders disk requests to minimize latency. Choose based on storage type:

  • SSD/NVMe: mq-deadline (multi-queue, low latency).
  • HDD: bfq (fair queueing for multiple processes) or deadline.

Check current scheduler:

cat /sys/block/sda/queue/scheduler  # Replace sda with your disk

Set scheduler (temporarily):

echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

Set permanently (systemd systems):

sudo nano /etc/udev/rules.d/60-ioscheduler.rules
# Add for SSD:
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
# For HDD: ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"
sudo udevadm trigger

Step 5: Network Tuning

Optimize TCP/IP settings for faster connections and higher throughput.

1. TCP Buffer Sizes

Larger buffers improve throughput for high-latency networks (e.g., WAN links).

  • Check current values: 
    sysctl net.ipv4.tcp_rmem  # Read buffer
sysctl net.ipv4.tcp_wmem  # Write buffer
  • Tune (add to /etc/sysctl.conf): 
    net.ipv4.tcp_rmem = 4096 87380 67108864  # min, default, max (bytes)
net.ipv4.tcp_wmem = 4096 65536 67108864
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864

2. Congestion Control Algorithm

bbr (Bottleneck Bandwidth and RTT) often outperforms the default cubic for high-throughput networks.

  • Check available algorithms: sysctl net.ipv4.tcp_available_congestion_control.
  • Set bbr: 
    sudo sysctl net.ipv4.tcp_congestion_control=bbr
# Persist: Add to /etc/sysctl.conf

3. Disable Unused Protocols

Turn off IPv6 if not needed to reduce overhead:

sudo sysctl net.ipv6.conf.all.disable_ipv6=1
sudo sysctl net.ipv6.conf.default.disable_ipv6=1

Step 6: Kernel Parameter Tuning

Use sysctl to modify kernel parameters at runtime and persist changes in /etc/sysctl.conf.

Critical Parameters

ParameterPurposeRecommended Value
vm.swappinessSwap aggressiveness10-20 (servers)
fs.file-maxMax open file descriptors1000000 (high-concurrency)
net.ipv4.tcp_syncookiesMitigate SYN floods1
net.core.somaxconnMax pending TCP connections1024 (web servers)

Apply changes:

sudo sysctl -p  # Load /etc/sysctl.conf

Step 7: Services and Daemons Optimization

Disable unused services to free CPU/memory and reduce attack surface.

1. List and Disable Services

Use systemctl to manage services:

  • List enabled services: systemctl list-unit-files --type=service --state=enabled.
  • Disable unused services (e.g., Bluetooth, printing, Avahi): 
    sudo systemctl disable --now bluetooth cups avahi-daemon

2. Optimize Active Services

Tune critical services like Nginx or PostgreSQL:

  • Nginx: Adjust worker_processes (set to number of CPU cores) and worker_connections in /etc/nginx/nginx.conf: 
    worker_processes auto;  # Uses all CPU cores
events {
    worker_connections 10240;  # Increase for high traffic
}

Step 8: Security Considerations

Tuning for performance shouldn’t compromise security.

  • Limit user processes: Use ulimit to prevent resource exhaustion. Edit /etc/security/limits.conf:

    * hard nproc 1000  # Max processes per user
* hard nofile 65535  # Max open files per user

  • Secure kernel parameters: Disable sysrq (emergency commands) and restrict core dumps:

    sysctl kernel.sysrq=0
sysctl fs.suid_dumpable=0

Step 9: Monitoring and Maintenance

After tuning, monitor performance to validate changes and catch regressions.

Tools for Ongoing Monitoring

  • Prometheus + Grafana: Open-source stack for metrics collection and visualization.
  • atop: Logs historical system activity (run atop -r /var/log/atop/atop_20240101 to review past data).
  • journalctl: Check system logs for errors after tuning: 
    journalctl -p err -b  # Errors since last boot

Maintenance Tips

  • Test changes incrementally: Tune one subsystem at a time and measure impact.
  • Backup configurations: Save copies of /etc/sysctl.conf, /etc/fstab, and service files.
  • Update the kernel: New kernels often include performance improvements (e.g., better I/O schedulers).

Conclusion

Linux system tuning is a iterative process that balances performance, stability, and security. By following this guide—starting with assessment, then optimizing CPU, memory, disk, and network subsystems—you can tailor your system to your workload. Always test changes in a non-production environment, monitor results, and back up critical data. With careful tuning, even aging hardware can deliver impressive performance gains.

References