funwithlinux guide

Troubleshooting Common Linux Performance Issues

Performance issues in Linux often manifest as slow response times, unresponsive applications, or high resource utilization. The root cause typically lies in one or more system components: **CPU**, **memory (RAM)**, **disk I/O**, **network**, or misconfigurations. Troubleshooting Linux performance requires a systematic approach: observe symptoms, isolate the bottleneck, and apply targeted fixes. This blog focuses on actionable techniques, leveraging built-in Linux tools to diagnose issues without requiring specialized software.

Linux is renowned for its stability and efficiency, but even the most robust systems can suffer from performance degradation over time. Whether you’re a system administrator, developer, or DevOps engineer, understanding how to diagnose and resolve common Linux performance issues is critical to maintaining a responsive, reliable environment. This blog will guide you through identifying bottlenecks in CPU, memory, disk I/O, network, and system configuration, with practical tools and step-by-step troubleshooting workflows.

Table of Contents

  1. Introduction to Linux Performance Troubleshooting
  2. CPU Bottlenecks: Identifying and Resolving High CPU Usage
  3. Memory Issues: Leaks, Swapping, and Exhaustion
  4. Disk I/O Problems: Slow Reads, Writes, and Latency
  5. Network Performance: Bandwidth, Latency, and Packet Loss
  6. Resource Contention: When Processes Fight for Resources
  7. System Configuration Missteps
  8. General Troubleshooting Methodology
  9. Conclusion
  10. References

1. CPU Bottlenecks: Identifying and Resolving High CPU Usage

The CPU is the “brain” of the system, and bottlenecks here can cripple performance. Common causes include:

  • A single process consuming excessive CPU (e.g., a misbehaving application).
  • High context switching (frequent process/thread switches).
  • CPU throttling (due to overheating or power management).

How to Identify CPU Issues

Use these tools to diagnose CPU problems:

1.1 top/htop: Real-Time CPU Monitoring

top is a classic tool for viewing process CPU usage. htop (an enhanced version) adds color-coding and interactivity.

Example Output Interpretation:

top - 14:30:00 up 2 days,  4:15,  2 users,  load average: 2.85, 2.40, 2.10
Tasks: 287 total,   1 running, 286 sleeping,   0 stopped,   0 zombie
%Cpu(s): 75.0 us, 15.0 sy,  0.0 ni,  5.0 id,  0.0 wa,  0.0 hi,  5.0 si,  0.0 st
MiB Mem :  15988.3 total,   2345.1 free,   8762.5 used,   4880.7 buff/cache
MiB Swap:   2048.0 total,   1980.2 free,     67.8 used.   6540.3 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1234 appuser   20   0  234560 123456  78900 R  95.0   7.7   4:30.12 java
  • Load Average: The 1/5/15-minute averages (2.85, 2.40, 2.10 here). A load > number of CPU cores indicates saturation.
  • %CPU: us (user space), sy (system/kernel), ni (nice), id (idle). High us suggests user processes; high sy may indicate kernel inefficiencies.
  • Running Process: The java process uses 95% CPU (R = running).

1.2 mpstat: Per-Core CPU Usage

Identify if CPU load is spread across cores or concentrated on one:

mpstat -P ALL 5  # Monitor all cores every 5 seconds

Output shows %usr, %sys, %idle per core. A single core at 100% may indicate a single-threaded bottleneck.

1.3 pidstat: Per-Process/Thread CPU Details

Drill into a specific process or its threads:

pidstat -p 1234 5  # Monitor process 1234 every 5s
pidstat -t -p 1234 5  # Monitor threads of process 1234

1.4 perf: Advanced CPU Profiling

Identify which functions consume CPU:

perf top -p 1234  # Show hot functions in process 1234

Troubleshooting Steps

  1. Kill/Restart the Offending Process: If a misbehaving app (e.g., java above) is the culprit, restart it or kill it with kill -9 1234.
  2. Optimize the Application: If the process is critical, profile it with perf to fix inefficient code.
  3. Reduce Context Switching: Use vmstat 1 to check cs (context switches). High cs (e.g., >10k/s) may require reducing thread count or optimizing I/O.
  4. Check for Throttling: Use cpufreq-info (Debian) or sensors to verify CPU frequency isn’t capped due to heat.

2. Memory Issues: Leaks, Swapping, and Exhaustion

Memory issues arise when the system runs out of physical RAM, leading to swapping (using disk as “virtual memory”) or crashes. Common causes:

  • Memory leaks (applications not releasing unused memory).
  • Excessive cache/buffer usage (rarely an issue, but misconfigured apps may hoard memory).
  • High swap usage (slow disk-based memory).

How to Identify Memory Issues

2.1 free -h: Total/Used Memory

free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi       8.5Gi       2.3Gi       1.2Gi       4.8Gi       6.3Gi
Swap:          2.0Gi        67Mi       1.9Gi
  • available: Estimated memory available for new apps (includes free + reclaimable cache).
  • Swap Used: If Swap: used is high, the system is swapping heavily.

2.2 vmstat: Swap and Paging Activity

vmstat 5  # Report every 5 seconds
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  1    67M  2345M  1234M  3646M    5   10   120   240  500 1500 75 15  5  0  0
  • si/so: Swap in/out (MB/s). High si/so indicates active swapping (bad for performance).

2.3 slabtop: Kernel Memory Usage

Check if the kernel is hoarding memory (e.g., due to leaks in drivers):

slabtop -o  # One-shot output

2.4 pmap: Per-Process Memory Breakdown

Analyze memory usage of a process:

pmap -x 1234  # Detailed memory map for PID 1234

Troubleshooting Steps

  1. Check for Leaks: Use valgrind --leak-check=full ./app to detect leaks in custom apps. For production, use gdb or strace.
  2. Adjust Swappiness: Reduce vm.swappiness (default 60) to prioritize RAM over swap: 
    sysctl vm.swappiness=10  # Temporary
echo "vm.swappiness=10" >> /etc/sysctl.conf  # Permanent
  3. Clear Caches (Cautious!): Free pagecache, dentries, and inodes (only if necessary): 
    sync; echo 3 > /proc/sys/vm/drop_caches

3. Disk I/O Problems: Slow Reads, Writes, and Latency

Disk I/O is often the slowest system component. Issues include:

  • High I/O latency (slow response to read/write requests).
  • I/O saturation (disk can’t keep up with requests).
  • Failing disks (bad sectors, mechanical issues).

How to Identify Disk I/O Issues

3.1 iostat: Disk Throughput and Latency

iostat -x 5  # Extended stats every 5 seconds

Key metrics:

  • r/s/w/s: Reads/writes per second.
  • rMB/s/wMB/s: Throughput (MB/s).
  • avgqu-sz: Average request queue size (high = saturation).
  • await: Average time (ms) for I/O to complete (includes queueing). >20ms = slow.

3.2 iotop: Per-Process I/O Usage

Identify which process is hogging disk I/O:

iotop -o  # Show only processes doing I/O (--only)

3.3 smartctl: Disk Health Check

Test for hardware failures:

smartctl -a /dev/sda  # Check disk /dev/sda

Look for Reallocated_Sector_Ct (bad sectors) or Temperature_Celsius.

Troubleshooting Steps

  1. Prioritize Critical I/O: Use ionice to set I/O priority (e.g., ionice -c 1 -n 0 -p 1234 for real-time priority).
  2. Optimize Filesystems: Use ext4 or xfs with proper alignment. Avoid noatime (disable access time logging) in /etc/fstab: 
    /dev/sda1 / ext4 defaults,noatime 0 1
  3. Replace Failing Disks: If smartctl shows errors, replace the disk immediately.

4. Network Performance: Bandwidth, Latency, and Packet Loss

Network issues manifest as slow transfers, timeouts, or dropped connections. Common causes:

  • Bandwidth saturation (too much traffic).
  • High latency (delays in packet delivery).
  • Packet loss (due to congestion or faulty hardware).

How to Identify Network Issues

4.1 iftop: Real-Time Bandwidth Usage

Monitor bandwidth per connection:

iftop -i eth0  # Monitor interface eth0

4.2 ss/netstat: Connection Statistics

Check for excessive connections (e.g., a DoS attack):

ss -s  # Summary of connections
ss -tulpn  # List TCP/UDP ports and processes

4.3 ping/traceroute: Latency and Packet Loss

Test connectivity to a remote host:

ping -c 10 example.com  # Check packet loss and latency
traceroute example.com  # Identify hops with high latency

Troubleshooting Steps

  1. Limit Bandwidth for Non-Critical Apps: Use tc (traffic control) or trickle to throttle bandwidth hogs.
  2. Fix DNS Issues: Slow DNS can cause delays. Test with dig example.com and switch to faster DNS servers (e.g., 1.1.1.1).
  3. Check Firewalls: Ensure iptables/ufw isn’t dropping packets accidentally: 
    iptables -L -v  # List firewall rules

5. Resource Contention: When Processes Fight for Resources

Resource contention occurs when multiple processes compete for CPU, memory, or I/O, leading to degraded performance for all.

How to Identify Contention

  • sar: System Activity Reporter (logs historical data): 
    sar -u 5 3  # CPU usage every 5s, 3 times
sar -r 5 3  # Memory usage
  • atop: Advanced system monitor showing resource trends over time.

Troubleshooting Steps

  1. Use cgroups: Limit resources for non-critical processes (e.g., in Kubernetes or systemd): 
    systemctl set-property myservice.service CPUShares=512 MemoryLimit=1G
  2. Prioritize Critical Workloads: Use nice (CPU priority) or ionice (I/O priority) for key apps.

6. System Configuration Missteps

Even healthy hardware/software can underperform due to misconfigurations. Common culprits:

Key Configurations to Check

  • Swappiness: As discussed earlier, vm.swappiness controls swap usage.
  • I/O Scheduler: Use cfq (fair for rotational disks) or deadline/none (better for SSDs): 
    cat /sys/block/sda/queue/scheduler  # Current scheduler
echo deadline > /sys/block/sda/queue/scheduler  # Temporary change
  • CPU Governor: Set to performance (max frequency) instead of powersave for servers: 
    cpupower frequency-set -g performance

7. General Troubleshooting Methodology

Follow this workflow for any performance issue:

  1. Observe: Use tools like top, iostat, and iftop to collect data.
  2. Identify: Narrow down the bottleneck (CPU, memory, disk, or network).
  3. Isolate: Use per-process tools (e.g., pidstat, iotop) to find the root cause.
  4. Resolve: Apply fixes (kill processes, adjust configs, optimize apps).
  5. Verify: Recheck with monitoring tools to ensure the issue is resolved.

Conclusion

Linux performance troubleshooting requires a mix of tool expertise and systematic analysis. By mastering tools like top, iostat, and ss, and following the methodology outlined here, you can quickly diagnose and resolve most common issues. Remember: proactive monitoring (e.g., with Prometheus + Grafana) is key to preventing problems before they impact users.

References