funwithlinux guide

Real World Examples of Linux Server Performance Tuning

Linux servers power everything from small business websites to global cloud infrastructures. While Linux is renowned for its stability and efficiency, even well-configured servers can hit performance bottlenecks as workloads grow, user traffic spikes, or applications evolve. Performance tuning isn’t just about making servers “faster”—it’s about optimizing resource usage, reducing latency, improving reliability, and ensuring scalability under real-world demands. In this blog, we’ll dive into **practical, real-world examples** of Linux server performance tuning. Each example will walk through a common scenario (e.g., a slow web server, unresponsive database, or resource-starved container environment), explain how the bottleneck was diagnosed, detail the specific tuning steps taken, and highlight the measurable results. By the end, you’ll have actionable insights to apply to your own Linux servers.

Table of Contents

  1. Understanding Performance Bottlenecks
  2. Real-World Tuning Examples
  3. General Best Practices for Linux Performance Tuning
  4. References

1. Understanding Performance Bottlenecks

Before diving into examples, it’s critical to recognize common performance bottlenecks. Linux server performance is typically constrained by one or more of the “big four” resources:

  • CPU: High usage (e.g., >80% sustained) can cause slow response times, especially for CPU-bound workloads (e.g., video encoding, complex calculations).
  • Memory (RAM): Insufficient memory leads to swapping (using disk as “virtual memory”), which drastically slows down the system (measured via swapon -s or vmstat).
  • Disk I/O: Slow read/write speeds (e.g., high %iowait in iostat) bottleneck databases, file servers, or applications with heavy disk access.
  • Network: Limited bandwidth, high latency, or connection mismanagement (e.g., too many TIME_WAIT sockets) can cripple web servers or distributed systems.

Tuning starts with diagnosis: using tools like htop, iostat, netstat, or application-specific metrics (e.g., Nginx access logs) to identify which resource is constrained.

2. Real-World Tuning Examples

Let’s explore four common scenarios where Linux server performance was degraded—and how targeted tuning resolved the issues.

Example 1: Web Server (Nginx) – Fixing Latency During Traffic Spikes

Scenario

A small e-commerce website running on Nginx (serving static assets and proxying to a Node.js backend) began experiencing slow page loads (3–5 seconds) and intermittent 502/504 errors during peak traffic (e.g., sales events). The server had 4 CPU cores and 8GB RAM—resources that should have handled the load.

Problem Identification

  • Symptoms: Slow page loads, high 5xx error rates (15% of requests), and user complaints.
  • Tools Used: top, netstat, nginx -V, and Nginx’s built-in stub_status module.
  • Data Collected:
    • top showed Nginx worker processes consuming 90%+ CPU.
    • netstat -nat | grep TIME_WAIT | wc -l revealed 10,000+ TIME_WAIT connections (idle TCP connections tying up resources).
    • stub_status showed: Active connections: 5000, Requests per second: 200 (low for the traffic volume).

Diagnosis

The bottlenecks were:

  1. Too many Nginx worker processes: Default Nginx config used worker_processes auto; (4 workers, matching CPU cores), but each worker was overloaded with 1,250+ connections.
  2. No keepalive connections: Each request opened a new TCP connection, leading to excessive TIME_WAIT sockets.
  3. Missing caching: Static assets (images, CSS) were being re-fetched on every request, increasing backend load.

Tuning Steps

  1. Optimize Nginx Worker Processes/Connections:
    Nginx uses worker_processes (number of CPU cores) and worker_connections (max connections per worker). The formula worker_processes * worker_connections should exceed expected concurrent connections.
    Updated /etc/nginx/nginx.conf:

    worker_processes auto;  # Use all 4 CPU cores (no change here)
events {
    worker_connections 1024;  # Increased from default 1024 to 2048 (tested stability)
    multi_accept on;  # Accept multiple connections at once
}

  2. Enable Keepalive Connections:
    Reduce TCP handshake overhead by reusing connections. Added to the http block:

    keepalive_timeout 65;  # Keep connections alive for 65s (default was 0)
keepalive_requests 100;  # Allow 100 requests per connection (default 100)

  3. Cache Static Assets:
    Cache images, CSS, and JS to reduce backend requests. Added to the server block for static files:

    location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
    proxy_cache my_cache;  # Define cache zone in http block
    proxy_cache_valid 200 302 1h;  # Cache 200/302 responses for 1 hour
    proxy_cache_valid 404 1m;  # Cache 404s for 1 minute
    expires 7d;  # Add "Cache-Control: max-age=604800" header
}

# Define cache zone in http block:
http {
    proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m use_temp_path=off;
}

Results

  • CPU Usage: Dropped from 90% to 40% during peak traffic.
  • Response Time: Page load time decreased from 3–5s to 200–300ms.
  • Error Rate: 5xx errors fell from 15% to 0.1%.
  • Connections: TIME_WAIT sockets reduced from 10,000+ to <500.

Example 2: Database Server (MySQL) – Speeding Up Slow Queries & Connections

Scenario

A SaaS platform using MySQL 8.0 (on a server with 8 CPU cores and 32GB RAM) faced slow query responses (5–10 seconds for common user reports) and frequent “Too many connections” errors during business hours. The database had ~500k daily queries.

Problem Identification

  • Symptoms: Slow report generation, “Too many connections” errors, and backend timeouts.
  • Tools Used: mysqltuner, show processlist;, slow_query_log, and pg_stat_statements (for query analytics).
  • Data Collected:
    • mysqltuner report showed:
      • innodb_buffer_pool_size = 1G (only 3% of available RAM).
      • max_connections = 151 (default, but 200+ concurrent connections were being requested).
    • slow_query_log revealed 20+ queries with full table scans (no indexes) taking 5–10s each.

Diagnosis

The bottlenecks were:

  1. Underprovisioned InnoDB Buffer Pool: MySQL was reading from disk instead of caching frequently accessed data in RAM.
  2. Insufficient Connections: max_connections was too low, causing connection failures.
  3. Unoptimized Queries: Full table scans on large tables (10M+ rows) hogged CPU and I/O.

Tuning Steps

  1. Increase InnoDB Buffer Pool Size:
    The InnoDB buffer pool caches table data and indexes. For a dedicated database server, allocate 50–70% of RAM to it. With 32GB RAM, we set:

    # In /etc/mysql/my.cnf:
[mysqld]
innodb_buffer_pool_size = 20G  # 62.5% of 32GB RAM
innodb_buffer_pool_instances = 4  # 1 instance per 4–8GB of buffer pool (avoids contention)

  2. Raise Connection Limits:
    Increase max_connections and add a connection pool (via PgBouncer) to reuse idle connections:

    [mysqld]
max_connections = 500  # Higher than peak concurrent requests (300)
wait_timeout = 60  # Close idle connections after 60s (default 28800s)

    Deployed PgBouncer to pool connections, reducing the number of new TCP connections to MySQL.

  3. Optimize Slow Queries:

    • Added indexes to columns used in WHERE, JOIN, and ORDER BY clauses (e.g., CREATE INDEX idx_user_id ON orders(user_id);).
    • Rewrote a slow report query from: 
      SELECT * FROM orders WHERE created_at > '2023-01-01' AND total > 1000;  # Full table scan
      To: 
      SELECT id, user_id, total FROM orders WHERE created_at > '2023-01-01' AND total > 1000;  # Uses covering index on (created_at, total)

Results

  • Query Latency: Slow queries (5–10s) reduced to 100–300ms.
  • Connections: “Too many connections” errors eliminated; PgBouncer reduced active MySQL connections from 300+ to 50–60.
  • I/O Usage: Disk reads (measured via iostat) dropped by 70% (thanks to the buffer pool).

Example 3: File Server (NFS) – Accelerating Large File Transfers

Scenario

A media production team used an NFS server (running on Ubuntu 22.04) to share large video files (10–50GB) between workstations. Transfers were slow (10–15 MB/s), despite a 1Gbps network. The server had a 4TB HDD (7200 RPM) and 16GB RAM.

Problem Identification

  • Symptoms: Slow file transfers, long wait times for video exports, and workstation timeouts.
  • Tools Used: iostat, sar, nfsstat, and mount.
  • Data Collected:
    • iostat -x 5 showed %iowait (disk I/O wait time) at 85% during transfers.
    • mount revealed the NFS export was mounted with default settings: rsize=4096,wsize=4096 (small read/write blocks).
    • sar -d 5 showed disk read/write speeds of only 15–20 MB/s (well below HDD limits of ~100 MB/s).

Diagnosis

The bottlenecks were:

  1. Small NFS Block Sizes: Default rsize/wsize (4KB) caused excessive round-trips between client and server.
  2. Unoptimized Filesystem Mount: The underlying ext4 filesystem was mounted with atime (updates access time on every read), adding unnecessary disk writes.

Tuning Steps

  1. Increase NFS Block Sizes:
    Updated the NFS export and client mounts to use larger block sizes (1MB) for better throughput:

    # On NFS server: Edit /etc/exports to set rsize/wsize:
/media/share 192.168.1.0/24(rw,sync,rsize=1048576,wsize=1048576,no_subtree_check)

# On NFS clients: Remount with new sizes:
mount -o remount,rsize=1048576,wsize=1048576 192.168.1.100:/media/share /mnt/share

  2. Optimize Filesystem Mount:
    Disabled atime (no need to track access times for media files) and enabled barrier=0 (reduces write overhead for non-critical data):

    # Edit /etc/fstab on the NFS server:
UUID=abc123 /media/share ext4 defaults,noatime,barrier=0 0 2

# Remount the filesystem:
mount -o remount /media/share

  3. Enable Read Caching:
    Increased Linux’s vm.dirty_background_ratio to allow more data to be cached in RAM before writing to disk:

    # Edit /etc/sysctl.conf:
vm.dirty_background_ratio = 10  # Start writing cached data to disk when 10% of RAM is dirty
vm.dirty_ratio = 20  # Force write when 20% of RAM is dirty

# Apply changes:
sysctl -p

Results

  • Transfer Speed: Increased from 10–15 MB/s to 80–90 MB/s (near the 1Gbps network limit).
  • %iowait: Dropped from 85% to 15% during transfers.
  • User Productivity: Video exports that took 2 hours now completed in 20–30 minutes.

Example 4: Containerized Environment (Docker/Kubernetes) – Resolving Resource Contention

Scenario

A DevOps team ran 50+ Docker containers (microservices, databases, and monitoring tools) on a single Kubernetes node (8 CPU cores, 32GB RAM). Containers frequently froze, and dmesg showed Out Of Memory (OOM) kills.

Problem Identification

  • Symptoms: Container crashes, slow response from microservices, and kubectl top pods showed some pods using 200%+ of their requested CPU.
  • Tools Used: docker stats, kubectl top pods, dmesg, and cAdvisor.
  • Data Collected:
    • docker stats revealed 3 containers (a Redis instance, a log processor, and a CI runner) using 8–10GB RAM each (no limits set).
    • kubectl top pods showed CPU throttling (pods hitting cpu.cfs.throttled_seconds).
    • dmesg | grep -i oom confirmed OOM kills for the log processor pod.

Diagnosis

The bottlenecks were:

  1. Unlimited Resources: Containers had no --memory or --cpu limits, leading to resource hoarding.
  2. Poor CPU/Memory Allocation: Critical services (e.g., Redis) competed with non-critical ones (e.g., CI runners) for resources.
  3. Inefficient Docker Storage Driver: The server used devicemapper (slow for containers); overlay2 is recommended for better performance.

Tuning Steps

  1. Set Resource Limits:
    Defined CPU/memory limits in Docker Compose/Kubernetes manifests to prevent hoarding:

    # Kubernetes Pod example (log processor):
apiVersion: v1
kind: Pod
metadata:
  name: log-processor
spec:
  containers:
  - name: log-processor
    image: my-log-processor:latest
    resources:
      requests:
        cpu: 100m  # 0.1 CPU cores
        memory: 512Mi
      limits:
        cpu: 1000m  # 1 CPU core
        memory: 2Gi  # Hard limit of 2GB RAM

  2. Prioritize Critical Services:
    Assigned higher CPU shares to critical pods (e.g., Redis) to ensure they get resources during contention:

    # Kubernetes Pod for Redis:
resources:
  limits:
    cpu: 2000m
    memory: 4Gi
  requests:
    cpu: 1000m
    memory: 2Gi

  3. Switch to Overlay2 Storage Driver:
    Reconfigured Docker to use overlay2 (faster, more efficient than devicemapper):

    # Stop Docker:
systemctl stop docker

# Edit /etc/docker/daemon.json:
{
  "storage-driver": "overlay2"
}

# Start Docker and verify:
systemctl start docker
docker info | grep "Storage Driver"  # Should show "overlay2"

Results

  • OOM Kills: Eliminated entirely (no more dmesg OOM entries).
  • CPU Throttling: Reduced by 90% (critical pods now get priority).
  • Container Start Time: Decreased from 30–60s to 5–10s (thanks to overlay2).

3. General Best Practices for Linux Performance Tuning

Beyond the examples above, these practices will help you avoid bottlenecks and maintain performance:

  • Monitor Continuously: Use tools like Prometheus+Grafana, Nagios, or sar to track baseline metrics (CPU, memory, I/O) and detect anomalies early.
  • Test Incrementally: Change one setting at a time, then measure impact. This avoids breaking things and isolates what works.
  • Document Changes: Log every tuning step, config file, and result (e.g., “Increased Nginx worker_connections to 2048; CPU usage dropped by 50%”).
  • Leverage Automation: Use Ansible/Chef to apply tuning consistently across servers, and tools like tuned-adm (RHEL) to apply prebuilt profiles (e.g., throughput-performance).

4. References

By applying these examples and best practices, you can transform underperforming Linux servers into efficient, scalable systems that handle real-world workloads with ease.