The Ultimate Linux Performance Tuning Checklist

Table of Contents

1. Baseline Measurement: Know Your Starting Point
2. CPU Performance Tuning
3. Memory (RAM) Optimization
4. Storage I/O Tuning
5. Network Performance Tuning
6. Kernel and System-Level Tuning
7. Application-Specific Tuning
8. Monitoring and Maintenance
9. Best Practices and Common Pitfalls
10. Conclusion
11. References

1. Baseline Measurement: Know Your Starting Point

Before tuning, establish a performance baseline to identify bottlenecks. Without this, you won’t know if changes improve or degrade performance.

Key Metrics to Measure:

• CPU: Usage (user, system, idle), load average, context switches, interrupts.
• Memory: Total/used/free RAM, swap usage, cache/buffer utilization, page faults.
• Storage: Disk I/O (read/write throughput, IOPS), latency, queue length.
• Network: Bandwidth usage, packet loss, latency, TCP connections.

Essential Tools:

• top/htop: Real-time CPU/memory usage (htop adds color and interactivity).
• vmstat: Virtual memory statistics (procs, memory, swap, IO, system, CPU).
  Example: vmstat 5 (refresh every 5 seconds).
• iostat: Disk I/O statistics (utilization, throughput, latency).
  Example: iostat -x 5 (extended stats, 5-second intervals).
• sar: System activity reporter (historical data; install via sysstat package).
  Example: sar -u 5 10 (CPU usage, 5s intervals, 10 samples).
• perf: Advanced CPU profiling (identify hot functions, context switches).
  Example: perf top (real-time CPU usage by function).
• nload/iftop: Network bandwidth monitoring (nload for overall, iftop for per-connection).
• free -h: Human-readable memory usage (RAM + swap).

Checklist:
✅ Run baseline tests during peak and off-peak hours.
✅ Record metrics for CPU, memory, storage, and network.
✅ Identify bottlenecks (e.g., “90% CPU usage by nginx” or “high swap usage”).

2. CPU Performance Tuning

CPU bottlenecks often manifest as high load averages, slow response times, or applications starved for cycles.

Key Optimizations:

a. CPU Scheduling and Governors

• Governors: Control CPU frequency scaling (balance performance/power).

  • performance: Max frequency (ideal for servers).
  • powersave: Min frequency (laptops/embedded).
  • ondemand: Scale based on load (default for desktops).
    Check/set: cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
    Set for all CPUs: echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor (persist via cpupower tool).

• Nice Values and renice: Adjust process priority (lower = higher priority, range: -20 to 19).
  Example: renice -5 -p 1234 (increase priority of PID 1234).

b. Interrupt Handling

• IRQ Balance: Distribute hardware interrupts across CPUs to avoid bottlenecks.
  Install/enable irqbalance service: sudo systemctl enable --now irqbalance.

• Isolate CPUs for Critical Workloads: Use isolcpus kernel parameter to reserve cores for real-time apps (e.g., databases, audio processing).
  Edit grub config: isolcpus=2,3 (reserve cores 2 and 3), then update grub and reboot.

c. Hyper-Threading and NUMA

• Hyper-Threading (HT): Enable for workloads with high parallelism (e.g., web servers); disable if apps struggle with thread contention (check via lscpu | grep 'Thread(s) per core').
• NUMA (Non-Uniform Memory Access): On multi-socket systems, bind processes to local CPU/memory nodes with numactl to reduce latency.
  Example: numactl --cpunodebind=0 --membind=0 ./app (run app on node 0 CPU/memory).

d. Avoid CPU Steal (Virtual Environments)

• CPU Steal: Time spent waiting for the hypervisor to schedule CPU (common in VMs).
  Check with vmstat 1 (column st). Mitigate by allocating dedicated cores or reducing overcommitment.

Checklist:
✅ Set CPU governor to performance for servers.
✅ Enable irqbalance on multi-core systems.
✅ Use renice or chrt (real-time scheduling) for critical apps.
✅ Check for CPU steal in VMs and adjust hypervisor settings.

3. Memory (RAM) Optimization

Insufficient memory leads to swapping (disk-based “virtual memory”), slowing the system to a crawl. Optimize RAM usage to avoid this.

Key Optimizations:

a. Swap Configuration

• Swappiness: Controls how aggressively the kernel swaps (0 = avoid swap, 100 = swap early).

  • Default: 60 (balanced).
  • Servers: Set to 10-20 (prioritize RAM).
  • Embedded: Set to 0 (disable swap if no disk).
    Persist: Add vm.swappiness=10 to /etc/sysctl.conf, then sysctl -p.

• Zswap/Zram: Compress swap in RAM (reduces disk I/O).
  Enable zram: sudo modprobe zram num_devices=1 && echo 2G > /sys/block/zram0/disksize && mkswap /dev/zram0 && swapon /dev/zram0.

b. File System Cache Tuning

The kernel uses free RAM for caching files (pagecache), improving read performance. Adjust how aggressively cache is written to disk:

• vm.dirty_ratio: Max % of RAM that can be dirty (unwritten) before flushing (default 20).
• vm.dirty_background_ratio: % of RAM that triggers background flushing (default 10).
  For write-heavy workloads (e.g., databases), lower these values to reduce latency:
  vm.dirty_ratio=15 and vm.dirty_background_ratio=5 in /etc/sysctl.conf.

c. Huge Pages

Reduce memory overhead for large workloads (databases, virtualization) by using 2MB/1GB pages instead of 4KB default.

• Check: grep HugePages_Total /proc/meminfo.
• Enable: Add default_hugepagesz=2M hugepagesz=2M hugepages=1024 to grub cmdline (reserves 2GB).

d. Detect Memory Leaks

Use smem -s pss (proportional set size) to find apps hoarding memory, or valgrind --leak-check=full ./app for debugging.

Checklist:
✅ Set vm.swappiness=10 for servers.
✅ Enable zswap/zram if swap is needed.
✅ Tune dirty_ratio for write-heavy apps.
✅ Use huge pages for databases (e.g., MySQL innodb_use_hugepages=ON).

4. Storage I/O Tuning

Slow storage (HDDs, misconfigured SSDs) is a common bottleneck. Focus on reducing latency and increasing throughput.

Key Optimizations:

a. I/O Schedulers

Choose a scheduler matching your workload:

• noop: No-op (best for SSDs/RAID; minimal overhead).
• deadline: Prioritizes read requests (good for databases).
• cfq: Completely Fair Queueing (default for HDDs; fair share per process).
  Set for a device: echo deadline | sudo tee /sys/block/sda/queue/scheduler (persist via udev rules).

b. Filesystem Optimization

• Ext4: Disable access time logging (noatime mount option) to reduce writes:
  Edit /etc/fstab: UUID=... / ext4 defaults,noatime 0 1.
• XFS: Enable delayed allocation (allocsize=16m) and use xfs_fsr for defragmentation.
• Btrfs: Enable compression (compress=zstd) for read-heavy, compressible data.

c. SSD Optimization

• Enable TRIM (reclaim deleted blocks): fstrim -av (run weekly via cron).
• Avoid partition alignment issues (use GPT instead of MBR).
• Disable swap on SSDs (use zram instead to reduce wear).

d. RAID and LVM

• RAID Level: RAID 10 for performance+redundancy; RAID 5/6 for capacity (slower writes).
• LVM: Use lvcreate --stripes 4 (striping across 4 disks) for better I/O.
• Avoid nested LVM/RAID (adds overhead).

e. Benchmark and Test

Use dd if=/dev/zero of=/tmp/test bs=1G count=1 oflag=direct (direct I/O, bypass cache) to test write speed, or fio for advanced workload simulation.

Checklist:
✅ Use noop or deadline scheduler for SSDs.
✅ Add noatime to /etc/fstab for all filesystems.
✅ Enable TRIM on SSDs with fstrim.
✅ Use RAID 10 for databases or high-I/O apps.

5. Network Performance Tuning

Poor network performance (latency, packet loss, slow TCP) cripples distributed systems. Optimize TCP/IP stack and hardware offloading.

Key Optimizations:

a. TCP/IP Tuning

• Window Scaling: Increase TCP window size for high-latency links (net.ipv4.tcp_window_scaling=1).
• TIME_WAIT Recycling: Reuse closed connections to reduce overhead:
  net.ipv4.tcp_tw_reuse=1 (reuse) and net.ipv4.tcp_tw_recycle=0 (avoid issues with NAT).
• Congestion Control: Use bbr (Bottleneck Bandwidth and RTT) for high-throughput links (default: cubic).
  Set: echo bbr | sudo tee /proc/sys/net/ipv4/tcp_congestion_control.

b. Buffer Sizes

Increase TCP send/receive buffers to handle high bandwidth:
net.core.rmem_max=16777216 (16MB) and net.core.wmem_max=16777216 in /etc/sysctl.conf.

c. Offloading

Enable hardware offloading to reduce CPU usage:

• Check: ethtool -k eth0 (look for rx-checksumming, tso, gso).
• Enable: ethtool -K eth0 rx on tx on tso on gso on.

d. Interrupt Coalescing

Reduce CPU interrupts from network cards by batching packets:
ethtool -C eth0 adaptive-rx on adaptive-tx on (auto-adjusts based on load).

Checklist:
✅ Use bbr congestion control for high-speed networks.
✅ Increase TCP buffer sizes (rmem_max, wmem_max).
✅ Enable hardware offloading (checksum, TSO).
✅ Tune interrupt coalescing for high-throughput links.

6. Kernel and System Tuning

Fine-tune the kernel and system services to reduce overhead and free resources.

Key Optimizations:

a. Kernel Parameters (sysctl.conf)

Add these to /etc/sysctl.conf for better performance:

# Network
net.ipv4.tcp_syncookies=1          # Prevent SYN floods
net.core.somaxconn=65535           # Max pending connections (e.g., Nginx)
net.ipv4.ip_local_port_range=1024 65535  # More ephemeral ports

# File descriptors
fs.file-max=1000000                # Increase system-wide file handles

# Security (optional)
kernel.randomize_va_space=2        # ASLR (address space layout randomization)

b. Disable Unnecessary Services

Stop services wasting CPU/memory (e.g., bluetooth, cups, avahi-daemon):

• List services: systemctl list-unit-files --type=service --state=enabled.
• Disable: sudo systemctl disable --now bluetooth.

c. File Descriptors (ulimit)

Increase per-process file descriptors (critical for web servers/databases):

• Temporarily: ulimit -n 65535.
• Permanently: Add * soft nofile 65535 and * hard nofile 65535 to /etc/security/limits.conf.

d. Use tmpfs for Temporary Files

Mount /tmp and /var/tmp as tmpfs (RAM disk) to speed up reads/writes:
Add to /etc/fstab: tmpfs /tmp tmpfs defaults,size=50% 0 0 (uses 50% of RAM).

Checklist:
✅ Tune sysctl.conf for network/file descriptors.
✅ Disable unused services (e.g., Bluetooth, print servers).
✅ Increase ulimit for high-concurrency apps.
✅ Mount /tmp as tmpfs.

7. Application-Specific Tuning

Tune apps to align with system resources (e.g., Nginx worker processes = CPU cores).

Key Examples:

a. Web Servers (Nginx/Apache)

• Nginx:
  • worker_processes auto (uses all CPU cores).
  • worker_connections 10240 (max connections per worker).
  • Enable gzip compression and proxy_cache for static assets.
• Apache:
  • Use mpm_event (event-driven) instead of mpm_prefork (process-per-connection).
  • Set MaxRequestWorkers 1000 (adjust based on RAM).

b. Databases (MySQL/PostgreSQL)

• MySQL:
  • innodb_buffer_pool_size=70% of RAM (caches table data/indexes).
  • query_cache_type=0 (disable query cache; deprecated in 8.0).
  • max_connections=500 (avoid too many open connections).
• PostgreSQL:
  • shared_buffers=25% of RAM (OS cache handles the rest).
  • work_mem=64MB (memory per sort/join operation).

c. Java Applications

• Tune JVM heap: -Xms2G -Xmx2G (initial/max heap; avoid overcommitting).
• Use G1GC for large heaps: -XX:+UseG1GC -XX:MaxGCPauseMillis=200.

d. Containers (Docker/Kubernetes)

• Set resource limits: docker run --memory=2G --cpus=1 app (prevents resource starvation).
• Use --mount type=tmpfs for container /tmp to avoid disk I/O.
• Prefer containerd over Docker for lower overhead in Kubernetes.

Checklist:
✅ Set Nginx worker processes to CPU cores.
✅ Allocate 70% RAM to MySQL innodb_buffer_pool_size.
✅ Limit JVM heap to avoid swapping.
✅ Set Docker resource limits to prevent node instability.

8. Monitoring and Maintenance

Tuning isn’t a one-time task—continuous monitoring ensures performance stays optimal.

Key Practices:

a. Set Up Monitoring

• Prometheus + Grafana: Metrics collection and visualization (CPU, memory, I/O dashboards).
• Nagios/Zabbix: Alert on thresholds (e.g., “CPU > 90% for 5 minutes”).
• journalctl: Monitor system logs for errors (journalctl -u nginx --since "1 hour ago").

b. Automated Tuning Tools

• tuned: Profile-based tuning (e.g., tuned-adm profile throughput-performance for servers).
• sysctl.d: Load kernel parameters at boot (place .conf files in /etc/sysctl.d/).

c. Regular Maintenance

• Update the kernel and apps (security/performance fixes).
• Rotate logs with logrotate to avoid disk bloat.
• Benchmark after changes (e.g., ab -n 1000 -c 100 http://localhost/ for web servers).

Checklist:
✅ Deploy Prometheus/Grafana for real-time metrics.
✅ Use tuned to apply pre-built profiles.
✅ Schedule log rotation and updates.
✅ Benchmark after tuning to validate improvements.

9. Best Practices and Pitfalls

• Test in Staging: Never tune production directly; replicate workloads in staging first.
• Incremental Changes: Tune one parameter at a time to isolate impact.
• Document Everything: Note before/after metrics and config changes.
• Avoid Over-Tuning: Disabling swap or setting swappiness=0 can crash apps if RAM fills up.
• Monitor Post-Deployment: Performance may degrade over time (e.g., memory leaks, traffic spikes).

Conclusion

Linux performance tuning is a iterative process that combines baseline measurement, targeted optimizations, and ongoing monitoring. This checklist provides a structured approach to identifying bottlenecks and applying fixes—from CPU and memory to storage, network, and applications. Remember: there’s no “one-size-fits-all” solution, so test rigorously and adapt to your specific workload.

References

• Linux Kernel Documentation
• Red Hat Performance Tuning Guide
• Nginx Optimization
• MySQL Performance Tuning
• sysctl Parameters
• Tuned Project
• Perf Wiki