DIY Linux Performance Tuning: A Step-by-Step Tutorial

Table of Contents

1. Assessing Baseline Performance
2. CPU Tuning
3. Memory Tuning
4. Disk I/O Tuning
5. Network Tuning
6. Kernel Tuning with sysctl
7. Monitoring and Maintaining Performance
8. References

1. Assessing Baseline Performance

Before tuning, you need to understand your system’s current behavior. Without a baseline, you can’t measure the impact of changes. We’ll use built-in and open-source tools to gather metrics on CPU, memory, disk, and network usage.

Key Tools for Baseline Assessment

Step 1: Gather Real-Time Metrics with htop

htop is an enhanced version of top with a user-friendly interface. Install it first (e.g., sudo apt install htop on Debian/Ubuntu, sudo dnf install htop on RHEL/CentOS):

htop  

What to look for:

• CPU usage: %us (user), %sy (system), %id (idle). High %sy may indicate kernel-level bottlenecks.
• Load average: 1/5/15-minute averages. Values > number of CPU cores suggest saturation.
• Memory: Total, used, free, and cached memory. Linux uses free memory for caching, so “available” (not just “free”) is key.
• Processes: Sort by CPU (F6) or memory (F6) to identify resource hogs.

Step 2: Measure Disk I/O with iostat

iostat (part of the sysstat package) tracks disk read/write rates and latency. Install sysstat first:

sudo apt install sysstat  # Debian/Ubuntu  
sudo dnf install sysstat  # RHEL/CentOS  

Run iostat with 2-second intervals to monitor disks:

iostat -x 2  

Key metrics:

• %iowait: Percentage of time CPU is idle waiting for disk I/O. >20% suggests disk bottlenecks.
• r/s/w/s: Reads/writes per second.
• avgrq-sz: Average request size (larger = better for throughput).
• avgqu-sz: Average queue length (should be <1 per disk).

Step 3: Capture Historical Data with sar

sar (also in sysstat) logs system activity over time. Enable it by editing /etc/default/sysstat (set ENABLED="true"), then restart the service:

sudo systemctl restart sysstat  

View CPU usage from the last 24 hours:

sar -u  # -u for CPU, -r for memory, -d for disk  

Historical data helps identify patterns (e.g., peak load times).

Takeaway: Record baseline metrics (CPU usage, load average, disk I/O, memory usage) in a spreadsheet or note file. You’ll compare these after tuning!

2. CPU Tuning

The CPU is the “brain” of the system. Bottlenecks here often manifest as slow response times or high load averages.

Step 1: Identify CPU Bottlenecks

Use mpstat (from sysstat) to check per-core usage. For a 4-core system:

mpstat -P ALL 2  # -P ALL: show all cores, 2: interval in seconds  

If one core is consistently >90% busy while others are idle, your workload may be single-threaded (e.g., some databases). If all cores are saturated, you may need more CPU resources or to optimize multi-threaded performance.

Step 2: Prioritize Critical Processes with nice/renice

Linux uses “niceness” values (-20 to 19) to prioritize processes. Lower values = higher priority.

• Start a process with custom priority:

  nice -n 5 ./my_script.sh  # Starts with niceness 5 (lower priority than default 0)  

• Adjust an existing process (use htop to find the PID):

  renice -n -5 1234  # Sets PID 1234 to niceness -5 (higher priority)  

Step 3: Set CPU Affinity (Pin Processes to Cores)

Some workloads (e.g., databases, real-time apps) perform better when pinned to specific CPU cores, reducing cache misses. Use taskset:

• Pin PID 1234 to cores 0 and 1:

  taskset -cp 0,1 1234  

• Start a process on core 2:

  taskset -c 2 ./my_app  

Step 4: Disable Hyper-Threading (If Needed)

Hyper-threading (HT) can boost throughput for multi-threaded workloads but may hurt latency-sensitive apps (e.g., gaming, real-time systems). Check if HT is enabled:

grep -E 'siblings|cpu cores' /proc/cpuinfo  

If siblings > cpu cores, HT is enabled. Disable it temporarily (persist via BIOS for long-term):

echo 0 | sudo tee /sys/devices/system/cpu/cpuX/online  # Replace X with HT core IDs (e.g., 1,3,5...)  

3. Memory Tuning

Linux manages memory aggressively, using free RAM for caching (files, disk blocks) to speed up access. However, misconfigured memory settings can lead to swapping (using disk as RAM), which cripples performance.

Step 1: Understand Memory Usage with free and vmstat

Check memory stats with free -h (human-readable):

free -h  
              total        used        free      shared  buff/cache   available  
Mem:           15Gi       2.3Gi       8.5Gi       345Mi       4.7Gi        12Gi  
Swap:          0B          0B          0B  

• available: Estimated memory available for new apps (includes free + reclaimable cache).
• buff/cache: Data cached from disk (safe to reclaim if needed).

Use vmstat 2 to monitor swapping:

vmstat 2  
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----  
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st  
 1  0      0 8934560 123456 4987652    0    0     0     5  890 1230  5  2 93  0  0  

• si/so: Swap in/out (kB/s). Non-zero values indicate swapping (bad for performance).

Step 2: Tune Swappiness

swappiness (0-100) controls how aggressively the kernel swaps. Lower values = prefer using RAM over swap. Default is 60, but for desktops/servers with ample RAM, reduce it:

• Temporarily set swappiness to 10:

  sudo sysctl vm.swappiness=10  

• Persist the change (reboot-safe):

  echo "vm.swappiness=10" | sudo tee /etc/sysctl.d/99-swappiness.conf  
  sudo sysctl -p /etc/sysctl.d/99-swappiness.conf  # Apply immediately  

Note: Set to 0 for systems with >16GB RAM (avoid swap unless critical).

Step 3: Clear Cached Memory (Temporarily)

If cached memory is hogging RAM needed for apps, clear it (use sparingly—caching improves performance!):

sudo sysctl -w vm.drop_caches=3  # 1=pagecache, 2=dentries/inodes, 3=both  

Step 4: Optimize for Memory Leaks

Apps with memory leaks slowly consume RAM, leading to swapping. Identify leaks with smem (memory usage per process, including shared memory):

sudo apt install smem  
smem -s pss -r  # Sort by PSS (Proportional Set Size, more accurate than RSS)  

Fix leaks by updating the app or limiting its memory with systemd (e.g., MemoryMax=2G in the service file).

4. Disk I/O Tuning

Disk I/O is often the biggest bottleneck in Linux systems. Mechanical HDDs are slow, but even SSDs can be optimized with proper settings.

Step 1: Identify Slow Disks with iotop

iotop shows real-time disk I/O usage per process:

sudo iotop -o  # -o: show only processes doing I/O  

Look for processes with high DISK READ/WRITE rates.

Step 2: Optimize Filesystem Mount Options

Tweak mount options in /etc/fstab to reduce unnecessary disk writes. Common options:

• noatime/nodiratime: Disable access time logging (huge boost for read-heavy workloads).
• discard: Enable TRIM for SSDs (reclaims unused blocks).
• data=writeback: For ext4, improves write performance (riskier for data integrity; use data=ordered for safety).

Example /etc/fstab entry for an SSD:

UUID=abc123 /mnt/ssd ext4 defaults,noatime,nodiratime,discard 0 2  

Remount to apply changes without rebooting:

sudo mount -o remount /mnt/ssd  

Step 3: Use hdparm to Tune HDD/SSD Performance

hdparm adjusts disk parameters. For SSDs, enable TRIM (if not using discard in fstab):

sudo hdparm -I /dev/sda | grep TRIM  # Check if TRIM is supported  
sudo fstrim -av  # Manual TRIM (run weekly via cron for SSDs)  

For HDDs, enable DMA (direct memory access) for faster transfers:

sudo hdparm -d1 /dev/sda  # Enable DMA  

Step 4: Choose the Right Filesystem

• ext4: Default for most systems (stable, good all-around).
• XFS: Better for large files (e.g., video editing, databases).
• Btrfs: Supports snapshots and RAID (experimental for critical data).

Format with optimal parameters (e.g., XFS with large block size for big files):

mkfs.xfs -b size=4096 /dev/sdb1  # 4KB block size (default for most)  

Step 5: Avoid Swap (If Possible)

Swap is a last resort. If you must use it, place swap on an SSD and limit its size (1-2x RAM for hibernation, 0-512MB otherwise). Disable swap temporarily:

sudo swapoff -a  

To persist, comment out the swap line in /etc/fstab.

5. Network Tuning

Poor network performance can stem from misconfigured TCP/IP settings, slow DNS, or unnecessary services.

Step 1: Monitor Network Usage with iftop and ss

iftop shows bandwidth usage per connection:

sudo iftop -i eth0  # Monitor interface eth0  

ss (socket statistics) replaces netstat for checking open ports/connections:

ss -tuln  # List TCP/UDP ports (-t: TCP, -u: UDP, -l: listening, -n: numeric)  

Step 2: Tune TCP/IP Buffers

TCP buffer sizes control how much data can be in transit. Small buffers cause frequent retransmissions; large buffers improve throughput for high-latency links (e.g., WAN).

Temporarily increase TCP buffers:

sudo sysctl -w net.core.rmem_max=16777216  # Max receive buffer (16MB)  
sudo sysctl -w net.core.wmem_max=16777216  # Max send buffer (16MB)  
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"  # Min/default/max receive  
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"  # Min/default/max send  

Persist these in /etc/sysctl.d/99-network.conf.

Step 3: Enable TCP Timestamps and Window Scaling

TCP timestamps (tcp_timestamps=1) and window scaling (tcp_window_scaling=1) improve performance over high-latency networks. They’re enabled by default, but verify:

sysctl net.ipv4.tcp_timestamps  
sysctl net.ipv4.tcp_window_scaling  

Step 4: Optimize DNS Caching

Slow DNS lookups delay web/app access. Use systemd-resolved or dnsmasq to cache DNS queries. Enable systemd-resolved:

sudo systemctl enable --now systemd-resolved  
sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf  

6. Kernel Tuning with sysctl

The Linux kernel exposes hundreds of tunable parameters via sysctl. Adjusting these can optimize for specific workloads (e.g., servers vs. desktops).

Key sysctl Parameters to Tune

Persist Changes with sysctl.d

Create a custom config file (e.g., /etc/sysctl.d/99-tuning.conf) and add parameters:

vm.swappiness=10  
net.core.somaxconn=1024  
kernel.sched_min_granularity_ns=1000000  

Apply changes:

sudo sysctl --system  

7. Monitoring and Maintaining Performance

Tuning isn’t a one-time task—systems evolve, and workloads change. Set up monitoring to track performance over time.

Step 1: Long-Term Monitoring with Prometheus + Grafana

For advanced monitoring, use Prometheus (metrics collection) and Grafana (visualization). Install via Docker:

docker run -d -p 9090:9090 prom/prometheus  
docker run -d -p 3000:3000 grafana/grafana  

Add the node_exporter to Prometheus to scrape Linux metrics, then build dashboards in Grafana to track CPU, memory, disk, and network trends.

Step 2: Automate Checks with sar

sar logs data to /var/log/sysstat/saXX (XX = day of month). Generate reports for past days:

sar -f /var/log/sysstat/sa15  # Report for the 15th  

Step 3: Create a Tuning Workflow

1. Assess: Capture baseline metrics with htop, iostat, sar.
2. Tune: Adjust one parameter at a time (e.g., swappiness, mount options).
3. Monitor: Use sar or Grafana to check if performance improved.
4. Iterate: Revert changes that hurt performance; double down on those that help.

8. References

• Linux Performance Tuning Guide (Red Hat)
• man pages: top(1), sysctl(8), fstab(5)
• SSD Optimization Guide (Arch Linux Wiki)
• TCP Tuning Guide (Cyberciti)
• nmon Performance Monitor

By following these steps, you’ll transform your Linux system into a lean, mean performance machine. Remember: measure first, tune second, and always test changes in a non-production environment! Happy tuning! 🚀