How to Conduct a Linux Performance Audit

Table of Contents

1. Understanding the Scope and Objectives
2. Pre-Audit Preparation
3. CPU Performance Analysis
4. Memory Utilization Audit
5. Disk I/O Performance Analysis
6. Network Performance Assessment
7. Process and Thread Analysis
8. System-Level Metrics and Kernel Tuning
9. Post-Audit: Analysis and Recommendations
10. Automating Performance Audits
11. Conclusion
12. References

1. Understanding the Scope and Objectives

Before diving into tools and metrics, clarify the purpose of your audit. Without clear goals, you’ll collect irrelevant data and waste time. Ask:

Key Questions to Define Scope:

• What’s the problem? (e.g., “Application X is slow,” “Server crashes under load,” “High disk latency”)
• Which services are critical? (e.g., web server, database, message queue)
• What are the performance baselines? (e.g., “Normal CPU usage is 40%,” “Database response time < 200ms”)
• What’s the time frame? (e.g., “Audit during peak hours” or “24-hour continuous monitoring”)

Example Objectives:

• Identify why a PostgreSQL database has slow query times.
• Determine if a web server can handle 10x traffic during a sale.
• Diagnose intermittent “out of memory” errors.

2. Pre-Audit Preparation

To ensure a smooth audit, gather tools, access, and context upfront.

2.1 Define Metrics to Track

Focus on metrics relevant to your objectives. Common metrics include:

• CPU: Usage, load average, context switches.
• Memory: Used/free/cached memory, swap usage.
• Disk I/O: Throughput, latency, queue length.
• Network: Bandwidth, packet loss, TCP connections.

2.2 Gather Tools

Linux offers a rich ecosystem of built-in and third-party tools. Install these before the audit (some may require root):

2.3 Secure Access

• Root Privileges: Many tools (e.g., perf, iotop) require root to access low-level metrics.
• Log Access: Check /var/log/syslog, /var/log/messages, and application logs (e.g., Apache’s access.log).
• Service Details: Document running services (systemctl list-units --type=service), hardware specs (lscpu, lsmem, lsblk).

3. CPU Performance Analysis

The CPU is often the first suspect in performance issues. A saturated CPU can slow down all system processes.

3.1 Key CPU Metrics

• Usage (%): Breakdown by user (%user), system (%sys), idle (%idle), and steal time (%steal for VMs).
• Load Average: 1/5/15-minute average of processes waiting for CPU (runnable + uninterruptible sleep).
• Context Switches: Rate at which the kernel switches between processes/threads (high values = overhead).
• CPU Saturation: When %idle < 10% and load average > number of CPU cores.

3.2 Tools to Analyze CPU

top/htop: Real-Time CPU Usage

• top (default): Shows processes sorted by CPU usage. Press 1 to view per-core data.
• htop (enhanced): Color-coded, interactive, and easier to read (install with apt install htop or yum install htop).

What to look for:

• High %user (application-level load) vs. %sys (kernel-level load).
• Processes with %CPU > 80% (potential bottlenecks).

mpstat (Multi-Processor Statistics)

Part of the sysstat package, mpstat shows per-core CPU usage:

mpstat -P ALL 5 10  # Check all cores (-P ALL) every 5s, 10 times  

Interpretation:

• Cores with %idle < 10% are saturated.
• Imbalanced core usage (e.g., one core at 100%, others idle) may indicate poor application threading.

sar (System Activity Reporter)

sar (also in sysstat) collects historical CPU data (if sysstat is configured to log). Use it to compare current vs. past performance:

sar -u 5 10  # CPU usage every 5s, 10 times  
sar -q  # Load average  

perf: Deep Dive into CPU Usage

For application-level CPU bottlenecks, perf traces function calls and CPU cycles:

perf top  # Real-time CPU usage by function  
perf record -p <PID>  # Record CPU activity for a process  
perf report  # Analyze recorded data  

3.3 Common CPU Issues & Fixes

• Saturation: Reduce load (kill unnecessary processes, optimize applications) or add CPU cores.
• High Context Switches: Reduce process/thread count (e.g., use connection pooling instead of spawning new threads).

4. Memory Utilization Audit

Insufficient memory can lead to slowdowns (due to swapping) or crashes (OOM killer).

4.1 Key Memory Metrics

• Total/Used/Free Memory: free -h shows physical memory usage.
• Cached/Buffered Memory: Linux caches disk data in memory to speed up access (this is not wasted memory).
• Swap Usage: If physical memory is full, Linux swaps data to disk (slow!).
• Slab Allocation: Kernel-level memory usage (e.g., inodes, file handles) via slabtop.

4.2 Tools to Analyze Memory

free: Quick Memory Snapshot

free -h  
# Example output:  
#              total        used        free      shared  buff/cache   available  
# Mem:           15G        8.5G        1.2G        300M        5.3G        6.0G  
# Swap:          4.0G        500M        3.5G  

• available (not free) is the best indicator of usable memory (includes cached/buffered memory that can be freed).

vmstat: Memory and Swap Activity

vmstat 5  # Report every 5s  
# Example output:  
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----  
#  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st  
#  2  0  512000 122880  65536 5452544    0    0     0     0  100  200  5  2 93  0  0  

• si (swap in) and so (swap out): Non-zero values indicate swapping (bad!).

htop: Per-Process Memory Usage

Sort processes by memory (F6 → %MEM) to identify memory hogs. Look for processes with RES (resident set size, physical memory used) > 10% of total RAM.

OOM Killer Logs

If the system ran out of memory, the OOM killer may have terminated a process. Check logs:

grep -i "out of memory" /var/log/syslog  

4.3 Common Memory Issues & Fixes

• High Swap Usage: Add physical memory or reduce memory usage (e.g., tune application heap size).
• Memory Leaks: Use valgrind (for C/C++ apps) or py-spy (Python) to identify leaks.
• Low Available Memory: Check for unnecessary processes (e.g., systemctl disable unused-service).

5. Disk I/O Performance Analysis

Slow disk I/O is often the culprit for laggy databases, file servers, or backup jobs.

5.1 Key Disk Metrics

• Throughput: MB/s read/written (rkB/s, wkB/s in iostat).
• IOPS (I/O Operations Per Second): Number of read/write requests.
• Latency:
  • await: Average time per I/O (includes queueing + service time).
  • svctm: Service time (time the disk spends processing the I/O).
• Queue Length: Number of pending I/O requests (avgqu-sz in iostat).

5.2 Tools to Analyze Disk I/O

iostat: Disk Throughput and Latency

iostat -x 5  # -x for extended stats, every 5s  
# Example output for sda:  
# Device            r/s     w/s     rkB/s     wkB/s   avgrq-sz  avgqu-sz     await     svctm     %util  
# sda              100     50      40960     20480     409.60     10.00      66.67      3.33     50.00  

• %util > 80%: Disk is saturated.
• await > 20ms: High latency (mechanical disks: ~10-20ms; SSDs: <5ms).
• avgqu-sz > 2-3: I/O queue is too long (disk can’t keep up).

iotop: Per-Process Disk Usage

Identify which process is causing I/O:

iotop -o  # Show only processes doing I/O  

blktrace: Low-Level Disk Traces

For deep dives (e.g., diagnosing sporadic I/O spikes):

blktrace -d /dev/sda -o - | blkparse -i -  # Trace sda and parse output  

5.3 Common Disk I/O Issues & Fixes

• High Latency: Upgrade to faster disks (SSD/NVMe), use RAID, or optimize I/O (e.g., batch writes).
• I/O Saturation: Reduce load (e.g., move backups to off-peak hours) or add disks.
• Fragmentation: Use e4defrag (ext4) or xfs_fsr (XFS) to defragment file systems.

6. Network Performance Assessment

Poor network performance can manifest as slow remote access, timeouts, or failed connections.

6.1 Key Network Metrics

• Bandwidth Usage: rx/tx (receive/transmit) MB/s.
• Packet Loss: Dropped packets (via ping, mtr).
• Latency: Round-trip time (RTT) via ping or traceroute.
• TCP Connections: Established, listening, or TIME_WAIT connections.

6.2 Tools to Analyze Network

iftop: Real-Time Bandwidth Usage

iftop -i eth0  # Monitor interface eth0  

Identifies which IPs/ports are consuming bandwidth.

ss: TCP Connection Details

ss -tuln  # List listening TCP/UDP ports  
ss -s     # Summary of TCP connections  
# Example:  
# Total: 1000 (kernel 1200)  
# TCP:   200 (estab 150, closed 30, orphaned 5, synrecv 0, timewait 25)  

• Too many TIME_WAIT connections? Tune net.ipv4.tcp_tw_recycle (caution: may break NAT).

tcpdump: Packet Capture

For debugging specific issues (e.g., malformed packets):

tcpdump -i eth0 port 80  # Capture HTTP traffic on eth0  

mtr: Combined Ping + Traceroute

mtr google.com  # Shows latency and packet loss per hop  

6.3 Common Network Issues & Fixes

• Bandwidth Saturation: Upgrade network links or limit bandwidth for non-critical services (e.g., tc).
• Packet Loss: Check for faulty cables, misconfigured firewalls, or overloaded switches.
• High Latency: Optimize routing (e.g., use a CDN for web traffic) or reduce distance to servers.

7. Process and Thread Analysis

Even if system-level metrics look normal, a single misbehaving process can cripple performance.

7.1 Key Process Metrics

• CPU/Memory Usage: Per-process %CPU and %MEM (via top/htop).
• Thread Count: Too many threads cause context-switch overhead.
• File Handles: Processes may leak file handles (lsof -p <PID> to check).

7.2 Tools to Analyze Processes

ps: Process Snapshot

ps aux --sort=-%cpu | head  # Top CPU-consuming processes  
ps aux --sort=-%mem | head  # Top memory-consuming processes  

pstree: Thread and Process Hierarchy

pstree -p <PID>  # Show threads (if process uses threading)  

lsof: Open Files and Network Connections

Identify resource leaks (e.g., a process opening thousands of files):

lsof -p <PID> | wc -l  # Count open files for a process  

7.3 Fixes for Process Issues

• Resource Hogs: Restart or optimize the process (e.g., tune JVM heap size for Java apps).
• Leaky Processes: Update the application or patch the leak (use valgrind or strace for debugging).

8. System-Level Metrics and Kernel Tuning

Kernel parameters and system-wide settings can impact performance.

8.1 Kernel Tuning with sysctl

Check/modify kernel parameters in /etc/sysctl.conf (persistent) or via sysctl -w (temporary):

• vm.swappiness: Controls swap aggressiveness (0 = avoid swap, 100 = swap early; default 60).
• net.core.somaxconn: Maximum pending TCP connections (increase for high-traffic web servers).
• fs.file-max: Maximum open file handles (increase for databases).

8.2 System Logs

Check for kernel warnings/errors:

dmesg | grep -i error  
journalctl -k -p err  # Kernel errors from systemd journal  

9. Post-Audit: Analysis and Recommendations

Collecting data is useless without action. Follow these steps:

9.1 Correlate Metrics

Look for patterns across subsystems. For example:

• High disk latency + slow database queries → Database is I/O-bound.
• High CPU usage + many TIME_WAIT connections → Web server is handling too many short-lived TCP connections.

9.2 Prioritize Issues

Rank issues by severity:

• Critical: Causes outages (e.g., OOM kills, 100% CPU saturation).
• High: Degrades user experience (e.g., slow queries, 50% packet loss).
• Low: Minor inefficiencies (e.g., unused services running).

9.3 Document and Act

Create a report with:

• Findings (e.g., “PostgreSQL uses 90% of disk I/O due to unindexed queries”).
• Recommendations (e.g., “Add indexes to the users table,” “Upgrade from HDD to SSD”).
• Action items with owners and deadlines.

10. Automating Performance Audits

Manual audits are great for deep dives, but automation ensures ongoing performance monitoring.

10.1 Tools for Automation

• Prometheus + Grafana: Collect metrics, store them, and visualize with dashboards (e.g., CPU, memory, disk I/O over time).
• Nagios/Zabbix: Monitor services and send alerts (e.g., “Disk usage > 90%”).
• collectd: Lightweight daemon to collect system metrics (integrates with Prometheus).
• Custom Scripts: Use bash/Python to log metrics to a file or database:

  # Example: Log CPU usage every 5s  
  while true; do mpstat | awk '/all/ {print $4}' >> cpu_usage.log; sleep 5; done  

10.2 Benefits of Automation

• Trend Analysis: Identify slow degradation (e.g., memory leaks growing over weeks).
• Proactive Alerts: Catch issues before they impact users.
• Baseline Comparison: Easily spot deviations from “normal” performance.

11. Conclusion

A Linux performance audit is a systematic way to diagnose bottlenecks and optimize system health. By combining manual deep dives with automated monitoring, you can ensure your Linux systems remain fast, reliable, and scalable. Remember: performance is a journey, not a one-time task—regular audits and ongoing monitoring are key to long-term success.

12. References

• Brendan Gregg’s Linux Performance Tools (industry-standard guide).
• Linux sysstat Documentation (for sar, iostat, mpstat).
• Kernel Tuning Guide.
• Prometheus Documentation.
• The Linux Performance Wiki.