Mastering Linux Performance Tuning: A Comprehensive Guide

Table of Contents

1. Understanding Performance Bottlenecks

   • CPU Limitations
   • Memory Constraints
   • Disk I/O Bottlenecks
   • Network Latency/Throughput Issues

2. Essential Performance Monitoring Tools

   • Real-Time System Monitors (top, htop)
   • Resource-Specific Tools (vmstat, iostat, sar)
   • Advanced Tracing Tools (perf, strace)
   • Network Analysis Tools (tcpdump, ss, netstat)

3. CPU Performance Tuning

   • Identifying CPU Bottlenecks
   • CPU Scheduling and Prioritization (nice, cgroups)
   • Interrupt Handling (irqbalance)
   • NUMA Awareness and Optimization

4. Memory Performance Tuning

   • Key Memory Metrics (Free, Buffers/Cache, Swap)
   • Page Cache and Swap Management
   • OOM Killer Configuration
   • Huge Pages and Memory Optimization

5. Disk I/O Performance Tuning

   • Storage Types (HDD, SSD, NVMe) and I/O Characteristics
   • I/O Schedulers (CFQ, Deadline, NOOP)
   • Filesystem Optimization (ext4, XFS, Btrfs)
   • RAID and Storage Performance

6. Network Performance Tuning

   • Key Network Metrics (Bandwidth, Latency, Retransmissions)
   • TCP Tuning (Buffers, Congestion Algorithms like BBR)
   • UDP Optimization
   • Network Interface and Offloading

7. Kernel Tuning

   • sysctl Parameters (vm, net, kernel.*)
   • Kernel Version and Module Management
   • Boot-Time Kernel Parameters

8. Application-Level Tuning

   • Database Tuning (MySQL, PostgreSQL)
   • Web Server Tuning (Nginx, Apache)
   • Java Application Tuning (JVM Heap, GC)
   • Container Optimization (Docker, Kubernetes)

9. Continuous Monitoring and Alerting

   • Tools (Prometheus + Grafana, Nagios, Zabbix)
   • Setting Up Dashboards and Alerts

10. Best Practices for Linux Performance Tuning

    • Baselining and Incremental Changes
    • Documentation and Rollback Plans
    • Security vs. Performance Tradeoffs

11. Conclusion

12. References

1. Understanding Performance Bottlenecks

Before tuning, you must identify what is slowing down the system. Performance bottlenecks typically manifest in one of four subsystems: CPU, memory, disk I/O, or network. These are often called the “four horsemen” of system performance.

CPU Limitations

• Symptoms: High %us (user CPU) or %sy (system CPU) in top, processes stuck in R (running) state, slow response times for CPU-bound tasks (e.g., video encoding, scientific computing).
• Causes: Too many concurrent processes, inefficient code (e.g., unoptimized loops), high interrupt load, or misconfigured CPU scheduling.

Memory Constraints

• Symptoms: High swap usage (si/so in vmstat), OOM (Out-of-Memory) killer terminating processes, slow disk I/O due to swapping (thrashing), and high %wa (I/O wait) in top.
• Causes: Insufficient physical memory, memory leaks, excessive page cache usage, or aggressive swapping.

Disk I/O Bottlenecks

• Symptoms: High %iowait in top, slow file operations, low throughput, or high latency (long await in iostat).
• Causes: Slow storage (HDD vs. SSD), misconfigured I/O schedulers, inefficient filesystem settings, or unoptimized database queries.

Network Latency/Throughput Issues

• Symptoms: Slow network transfers, timeouts, high packet loss, or TCP retransmissions (visible in tcpdump or ss -ti).
• Causes: Bandwidth saturation, misconfigured TCP buffers, latency from long distances, or inefficient network protocols.

2. Essential Performance Monitoring Tools

To diagnose bottlenecks, you need tools to measure subsystem health. Below are critical tools for monitoring Linux performance:

Real-Time System Monitors

• top/htop: Real-time overview of CPU, memory, and process activity. htop (enhanced top) adds color, mouse support, and easier process management.

  top -o %CPU  # Sort by CPU usage  
  htop         # Interactive view with memory/CPU breakdown  

• glances: A modern alternative to htop with disk I/O and network metrics.

Resource-Specific Tools

Advanced Tracing Tools

• perf: Kernel-level performance analysis (CPU usage, function calls, interrupts).

  perf top        # Real-time CPU usage by function  
  perf record -g  # Profile a process and generate a call graph  

• strace: Trace system calls of a process (debug slow I/O or network issues).

  strace -p 1234  # Trace process ID 1234  

• dstat: Combines vmstat, iostat, and netstat into a single view.

Network Analysis Tools

• ss/netstat: Show network connections, ports, and socket stats. ss is faster than netstat.

  ss -tuln  # List TCP/UDP ports in use  
  ss -ti    # TCP connection details (retransmissions, buffer sizes)  

• tcpdump: Capture and analyze network packets (for debugging latency/loss).

  tcpdump -i eth0 port 80  # Capture HTTP traffic on eth0  

• iftop: Real-time network bandwidth usage per connection.

3. CPU Performance Tuning

CPU bottlenecks often stem from inefficient process scheduling, high interrupts, or NUMA (Non-Uniform Memory Access) misconfiguration.

Identifying CPU Bottlenecks

Use top/htop to check %us (user CPU), %sy (system CPU), and %id (idle). High %sy may indicate kernel inefficiencies (e.g., excessive context switches).

• pidstat: Drill into per-process CPU usage:

  pidstat -u 5  # CPU usage per process every 5 seconds  

CPU Scheduling and Prioritization

• nice/renice: Adjust process priority (range: -20 [highest] to 19 [lowest]).

  nice -n 10 ./myapp  # Start process with low priority  
  renice -5 -p 1234   # Increase priority of PID 1234  

• cgroups (Control Groups): Limit CPU usage for containers or processes (e.g., isolate a database to 2 cores):

  # Create a cgroup with 2 CPU cores  
  sudo cgcreate -g cpu:/db-limit  
  sudo cgset -r cpu.cfs_quota_us=200000 db-limit  # 2 cores (1 core = 100000 us)  
  sudo cgexec -g cpu:/db-limit ./database-process  

Interrupt Handling

• irqbalance: Distribute hardware interrupts across CPUs to avoid bottlenecks on a single core. Install and enable it:

  sudo apt install irqbalance  
  sudo systemctl enable --now irqbalance  

• Isolate CPUs for critical workloads: Use isolcpus kernel parameter (e.g., isolcpus=2,3 in grub to reserve cores 2 and 3 for high-priority tasks).

NUMA Awareness

Multi-socket systems use NUMA, where memory access is faster from local CPU sockets. Use numactl to bind processes to CPUs/memory:

numactl --cpunodebind=0 --membind=0 ./myapp  # Bind to node 0 (CPU and memory)  

4. Memory Performance Tuning

Memory tuning focuses on reducing swapping, optimizing page cache, and preventing OOM events.

Key Memory Metrics

• Page Cache: Linux caches frequently accessed files in memory (buff/cache in free -h). This is normal and frees up when applications need memory.
• Swap: Disk-based “extension” of memory. Avoid excessive swapping (thrashing) as it kills performance.

Page Cache and Swap Management

• vm.swappiness: Controls how aggressively the kernel swaps. Lower values (0-10) reduce swapping; higher values (60 default) prioritize freeing memory.

  sysctl vm.swappiness=10  # Temporary change  
  echo "vm.swappiness=10" >> /etc/sysctl.conf  # Persistent  

• Dirty Page Ratio: vm.dirty_ratio (max % of memory with dirty pages before syncing to disk) and vm.dirty_background_ratio (background sync threshold). For write-heavy workloads:

  sysctl vm.dirty_ratio=40  
  sysctl vm.dirty_background_ratio=10  

OOM Killer Configuration

The OOM killer terminates processes when memory is exhausted. To prioritize or exclude processes:

• Edit /proc/<pid>/oom_score_adj (range: -1000 [never kill] to 1000 [first to kill]):

  echo -1000 > /proc/1234/oom_score_adj  # Protect PID 1234  

Huge Pages

Large memory pages (2MB/1GB) reduce TLB (Translation Lookaside Buffer) pressure for memory-intensive apps (e.g., databases, VMs). Enable with:

sysctl vm.nr_hugepages=1024  # Allocate 1024 x 2MB huge pages  

5. Disk I/O Performance Tuning

Disk I/O is often the slowest subsystem. Optimize by aligning storage, I/O schedulers, and filesystem settings.

Storage Types and I/O Schedulers

• HDD: Rotational disks benefit from schedulers that reorder I/O (e.g., deadline or cfq).
• SSD/NVMe: No seek time; use noop (pass-through) or mq-deadline (multi-queue) for lower latency.

Check/set the I/O scheduler for a disk (e.g., /dev/sda):

cat /sys/block/sda/queue/scheduler  # View current scheduler  
echo "mq-deadline" > /sys/block/sda/queue/scheduler  # Temporary  

For persistence, add elevator=mq-deadline to the kernel command line in grub.

Filesystem Optimization

• ext4: Default for many systems. Use noatime (disable access time logging) and discard (TRIM for SSDs) in /etc/fstab:

  UUID=... / ext4 defaults,noatime,discard 0 1  

• XFS: Better for large filesystems/databases. Enable inode64 (use 64-bit inodes) and allocsize=16m (preallocate space):

  mount -t xfs -o inode64,allocsize=16m /dev/sdb1 /data  

• Btrfs: Good for snapshots, but avoid for high-performance databases.

RAID Configuration

• RAID 0: Striping (high throughput, no redundancy) – ideal for scratch disks.
• RAID 10: Mirroring + striping (high read/write performance, redundancy) – best for databases.
• RAID 5/6: Parity-based (lower write performance due to parity calculations) – use only for read-heavy workloads.

6. Network Performance Tuning

Network tuning focuses on maximizing throughput, reducing latency, and minimizing packet loss.

TCP Tuning

• TCP Congestion Algorithms: Modern algorithms like BBR (Bottleneck Bandwidth and RTT) outperform legacy ones (CUBIC) on high-latency networks (e.g., cloud, WAN). Enable BBR:

  sysctl net.ipv4.tcp_congestion_control=bbr  

• TCP Buffers: Increase send/receive buffers to handle high bandwidth:

  sysctl net.core.rmem_max=268435456  # Max receive buffer (256MB)  
  sysctl net.core.wmem_max=268435456  # Max send buffer  
  sysctl net.ipv4.tcp_rmem="4096 87380 268435456"  # Min/default/max receive  

UDP Optimization

UDP lacks TCP’s reliability but is faster for real-time apps (e.g., video streaming). Tune for low latency:

sysctl net.core.rmem_default=1048576  # Larger buffers for high UDP throughput  

Network Interface Optimization

Use ethtool to enable hardware offloading (checksum, TCP segmentation) and set speed/duplex:

ethtool -K eth0 tx-checksum-ipv4 on  # Enable TX checksum offloading  
ethtool -s eth0 speed 1000 duplex full  # Set 1Gbps full-duplex  

7. Kernel Tuning

The Linux kernel has hundreds of tunable parameters via sysctl and boot options.

Key sysctl Parameters

Apply changes persistently by adding to /etc/sysctl.conf and running sysctl -p.

Kernel Boot Parameters

Add these to GRUB_CMDLINE_LINUX in /etc/default/grub (then run update-grub):

• intel_idle.max_cstate=1: Reduce CPU power saving (lower latency for real-time apps).
• transparent_hugepage=never: Disable THP (avoids latency spikes in databases like MongoDB).

8. Application-Level Tuning

Even well-tuned systems suffer if applications are misconfigured.

Database Tuning (MySQL)

• Query Optimization: Use EXPLAIN to fix slow queries, add indexes, and avoid full table scans.
• Connection Pools: Limit concurrent connections (max_connections=500) to avoid resource exhaustion.
• InnoDB Buffers: Set innodb_buffer_pool_size to 50-70% of available memory (e.g., 4G on an 8GB server).

Web Server Tuning (Nginx)

• Worker Processes: Set to number of CPU cores (worker_processes auto).
• Worker Connections: Increase with worker_connections 10240 (adjust based on memory).
• Keepalive: Enable keepalive_timeout 65 to reuse TCP connections.

Java Application Tuning (JVM)

• Heap Size: Set Xms (initial) and Xmx (max) to avoid resizing (e.g., -Xms4G -Xmx4G).
• Garbage Collection: Use G1GC for large heaps (-XX:+UseG1GC) or ZGC/Shenandoah for low-latency apps.

9. Continuous Monitoring and Alerting

Tuning isn’t a one-time task—monitor performance over time to catch regressions.

Tools

• Prometheus + Grafana: Open-source monitoring with custom dashboards. Use node_exporter to collect Linux metrics.
• Nagios/Zabbix: Enterprise-grade monitoring with alerting (email, Slack).
• Datadog/New Relic: Cloud-based tools with prebuilt Linux performance dashboards.

Example Grafana Dashboard

Create a dashboard with panels for:

• CPU usage (per core, %us/%sy).
• Memory: Used, free, swap.
• Disk I/O: %iowait, throughput, latency.
• Network: Bandwidth, TCP retransmissions.

10. Best Practices for Linux Performance Tuning

1. Baseline First: Measure current performance before tuning to quantify improvements.
2. Tune Incrementally: Change one variable at a time and test results.
3. Monitor After Changes: Use tools like sar to compare pre/post-tuning metrics.
4. Document Everything: Log changes, rationale, and results for future reference.
5. Balance Security and Performance: Avoid disabling security features (e.g., SELinux) for minor gains.

11. Conclusion

Linux performance tuning is a iterative process that requires understanding system behavior, diagnosing bottlenecks, and applying targeted optimizations. By mastering tools like perf, iostat, and sysctl, and following best practices for CPU, memory, disk, and network tuning, you can unlock significant performance gains. Remember: the goal isn’t to maximize raw speed, but to align system behavior with application requirements—whether that’s low latency for real-time apps or high throughput for data processing.

12. References

• Brendan Gregg’s Linux Performance – Authoritative guides and tools.
• Linux Kernel Documentation – Official kernel tuning docs.
• Red Hat Performance Tuning Guide – In-depth RHEL-specific tuning.
• Nginx Performance Tuning – Best practices for Nginx.
• TCP BBR Congestion Control – Cloudflare’s guide to BBR.
• Systems Performance: Enterprise and the Cloud by Brendan Gregg (book).

Happy tuning! 🚀<|FCResponseEnd|># Mastering Linux Performance Tuning: A Comprehensive Guide

Introduction

Linux is the backbone of modern computing, powering everything from embedded devices and personal laptops to enterprise servers, cloud infrastructure, and supercomputers. Its flexibility, stability, and open-source nature make it a top choice for critical workloads. However, even the most robust Linux systems can underperform without proper tuning. Whether you’re managing a high-traffic web server, a database cluster, or a real-time application, optimizing Linux performance is key to reducing latency, improving throughput, and lowering operational costs.

This guide demystifies Linux performance tuning, taking you from identifying bottlenecks to implementing advanced optimizations. We’ll cover core subsystems (CPU, memory, disk I/O, network), essential monitoring tools, kernel tweaks, and application-level best practices. By the end, you’ll have the skills to diagnose performance issues and fine-tune your Linux environment for maximum efficiency.

Table of Contents

1. Understanding Performance Bottlenecks

   • CPU Limitations
   • Memory Constraints
   • Disk I/O Bottlenecks
   • Network Latency/Throughput Issues

2. Essential Performance Monitoring Tools

   • Real-Time System Monitors (top, htop)
   • Resource-Specific Tools (vmstat, iostat, sar)
   • Advanced Tracing Tools (perf, strace)
   • Network Analysis Tools (tcpdump, ss, netstat)

3. CPU Performance Tuning

   • Identifying CPU Bottlenecks
   • CPU Scheduling and Prioritization (nice, cgroups)
   • Interrupt Handling (irqbalance)
   • NUMA Awareness and Optimization

4. Memory Performance Tuning

   • Key Memory Metrics (Free, Buffers/Cache, Swap)
   • Page Cache and Swap Management
   • OOM Killer Configuration
   • Huge Pages and Memory Optimization

5. Disk I/O Performance Tuning

   • Storage Types (HDD, SSD, NVMe) and I/O Characteristics
   • I/O Schedulers (CFQ, Deadline, NOOP)
   • Filesystem Optimization (ext4, XFS, Btrfs)
   • RAID and Storage Performance

6. Network Performance Tuning

   • Key Network Metrics (Bandwidth, Latency, Retransmissions)
   • TCP Tuning (Buffers, Congestion Algorithms like BBR)
   • UDP Optimization
   • Network Interface and Offloading

7. Kernel Tuning

   • sysctl Parameters (vm, net, kernel.*)
   • Kernel Version and Module Management
   • Boot-Time Kernel Parameters

8. Application-Level Tuning

   • Database Tuning (MySQL, PostgreSQL)
   • Web Server Tuning (Nginx, Apache)
   • Java Application Tuning (JVM Heap, GC)
   • Container Optimization (Docker, Kubernetes)

9. Continuous Monitoring and Alerting

   • Tools (Prometheus + Grafana, Nagios, Zabbix)
   • Setting Up Dashboards and Alerts

10. Best Practices for Linux Performance Tuning

    • Baselining and Incremental Changes
    • Documentation and Rollback Plans

11. Conclusion

12. References

1. Understanding Performance Bottlenecks

Before tuning, you must identify what is slowing down the system. Performance bottlenecks typically manifest in one of four subsystems: CPU, memory, disk I/O, or network. These are often called the “four horsemen” of system performance.

CPU Limitations

• Symptoms: High %us (user CPU) or %sy (system CPU) in top, processes stuck in R (running) state, slow response times for CPU-bound tasks (e.g., video encoding, scientific computing).
• Causes: Too many concurrent processes, inefficient code (e.g., unoptimized loops), high interrupt load, or misconfigured CPU scheduling.

Memory Constraints

• Symptoms: High swap usage (si/so in vmstat), OOM (Out-of-Memory) killer terminating processes, slow disk I/O due to swapping (thrashing), and high %wa (I/O wait) in top.
• Causes: Insufficient physical memory, memory leaks, excessive page cache usage, or aggressive swapping.

Disk I/O Bottlenecks

• Symptoms: High %iowait in top, slow file operations, low throughput, or high latency (long await in iostat).
• Causes: Slow storage (HDD vs. SSD), misconfigured I/O schedulers, inefficient filesystem settings, or unoptimized database queries.

Network Latency/Throughput Issues

• Symptoms: Slow network transfers, timeouts, high packet loss, or TCP retransmissions (visible in tcpdump or ss -ti).
• Causes: Bandwidth saturation, misconfigured TCP buffers, latency from long distances, or inefficient network protocols.

2. Essential Performance Monitoring Tools

To diagnose bottlenecks, you need tools to measure subsystem health. Below are critical tools for monitoring Linux performance:

Real-Time System Monitors

• top/htop: Real-time overview of CPU, memory, and process activity. htop (enhanced top) adds color, mouse support, and easier process management.

  top -o %CPU  # Sort by CPU usage  
  htop         # Interactive view with memory/CPU breakdown  

• glances: A modern alternative to htop with disk I/O and network metrics.

Resource-Specific Tools

Advanced Tracing Tools

• perf: Kernel-level performance analysis (CPU usage, function calls, interrupts).

  perf top        # Real-time CPU usage by function  
  perf record -g  # Profile a process and generate a call graph  

• strace: Trace system calls of a process (debug slow I/O or network issues).

  strace -p 1234  # Trace process ID 1234  

• dstat: Combines vmstat, iostat, and netstat into a single view.

Network Analysis Tools

• ss/netstat: Show network connections, ports, and socket stats. ss is faster than netstat.

  ss -tuln  # List TCP/UDP ports in use  
  ss -ti    # TCP connection details (retransmissions, buffer sizes)  

• tcpdump: Capture and analyze network packets (for debugging latency/loss).

  tcpdump -i eth0 port 80  # Capture HTTP traffic on eth0  

• iftop: Real-time network bandwidth usage per connection.

3. CPU Performance Tuning

CPU bottlenecks often stem from inefficient process scheduling, high interrupts, or NUMA (Non-Uniform Memory Access) misconfiguration.

Identifying CPU Bottlenecks

Use top/htop to check %us (user CPU), %sy (system CPU), and %id (idle). High %sy may indicate kernel inefficiencies (e.g., excessive context switches).

• pidstat: Drill into per-process CPU usage:

  pidstat -u 5  # CPU usage per process every 5 seconds  

CPU Scheduling and Prioritization

• nice/renice: Adjust process priority (range: -20 [highest] to 19 [lowest]).

  nice -n 10 ./myapp  # Start process with low priority  
  renice -5 -p 1234   # Increase priority of PID 1234  

• cgroups (Control Groups): Limit CPU usage for containers or processes. Example: Isolate cores 2-3 for a database:

  sudo cgcreate -g cpu:/db-limit  
  sudo cgset -r cpu.cfs_quota_us=200000 db-limit  # 2 cores (1 core = 100000 us)  
  sudo cgexec -g cpu:/db-limit ./database-process  

Interrupt Handling

• irqbalance: Distribute hardware interrupts across CPUs to avoid bottlenecks on a single core. Install and enable it:

  sudo apt install irqbalance  
  sudo systemctl enable --now irqbalance  

NUMA Awareness

Multi-socket systems use NUMA, where memory access is faster from local CPU sockets. Use numactl to bind processes to CPUs/memory:

numactl --cpunodebind=0 --membind=0 ./myapp  # Bind to node 0 (CPU and memory)  

4. Memory Performance Tuning

Memory tuning focuses on reducing swapping, optimizing page cache, and preventing OOM events.

Key Memory Metrics

• Page Cache: Linux caches frequently accessed files in memory (buff/cache in free -h). This is normal and frees up when applications need memory.
• Swap: Disk-based “extension” of memory. Avoid excessive swapping (thrashing).

Page Cache and Swap Management

• vm.swappiness: Controls how aggressively the kernel swaps. Lower values (0-10) reduce swapping; higher values (60 default) prioritize freeing memory.

  sysctl vm.swappiness=10  # Temporary change  
  echo "vm.swappiness=10" >> /etc/sysctl.conf  # Persistent  

• Dirty Page Ratio: vm.dirty_ratio (max % of memory with dirty pages before syncing to disk) and vm.dirty_background_ratio (background sync threshold). For write-heavy workloads:

  sysctl vm.dirty_ratio=40  
  sysctl vm.dirty_background_ratio=10  

OOM Killer Configuration

The OOM killer terminates processes when memory is exhausted. To prioritize or exclude processes:

• Edit /proc/<pid>/oom_score_adj (range: -1000 [never kill] to 1000 [first to kill]):

  echo -1000 > /proc/1234/oom_score_adj  # Protect PID 1234  

Huge Pages

Large memory pages (2MB/1GB) reduce TLB (Translation Lookaside Buffer) pressure for memory-intensive apps (e.g., databases, VMs). Enable with:

sysctl vm.nr_hugepages=1024  # Allocate 1024 x 2MB huge pages  

5. Disk I/O Performance Tuning

Disk I/O is often the slowest subsystem. Optimize by aligning storage, I/O schedulers, and filesystem settings.

Storage Types and I/O Schedulers

• HDD: Rotational disks benefit from schedulers that reorder I/O (e.g., deadline or cfq).
• SSD/NVMe: No seek time; use noop (pass-through) or mq-deadline (multi-queue) for lower latency.

Check/set the I/O scheduler for a disk (e.g., /dev/sda):

cat /sys/block/sda/queue/scheduler  # View current scheduler  
echo "mq-deadline" > /sys/block/sda/queue/scheduler  # Temporary  

For persistence, add elevator=mq-deadline to the kernel command line in grub.

Filesystem Optimization

• ext4: Default for many systems. Use noatime (disable access time logging) and discard (TRIM for SSDs) in /etc/fstab:

  UUID=... / ext4 defaults,noatime,discard 0 1  

• XFS: Better for large filesystems/databases. Enable inode64 (use 64-bit inodes) and allocsize=16m (preallocate space):

  mount -t xfs -o inode64,allocsize=16m /dev/sdb1 /data  

RAID Configuration

• RAID 0: Striping (high throughput, no redundancy) – ideal for scratch disks.
• RAID 10: Mirroring + striping (high read/write performance, redundancy) – best for databases.

6. Network Performance Tuning

Network tuning focuses on maximizing throughput, reducing latency, and minimizing packet loss.

TCP Tuning

• TCP Congestion Algorithms: Modern algorithms like BBR (Bottleneck Bandwidth and RTT) outperform legacy ones (CUBIC) on high-latency networks. Enable BBR:

  sysctl net.ipv4.tcp_congestion_control=bbr  

  Verify with sysctl net.ipv4.tcp_congestion_control.

• TCP Buffers: Increase send/receive buffers to handle high bandwidth:

  sysctl net.core.rmem_max=268435456  # Max receive buffer (256MB)  
  sysctl net.core.wmem_max=268435456  # Max send buffer  

Network Interface Optimization

Use ethtool to enable hardware offloading (checksum, TCP segmentation) and set speed/duplex:

ethtool -K eth0 tx-checksum-ipv4 on  # Enable TX checksum offloading  
ethtool -s eth0 speed 1000 duplex full  # Set 1Gbps full-duplex  

7. Kernel Tuning

The Linux kernel has hundreds of tunable parameters via sysctl and boot options.

Key sysctl Parameters

Apply changes persistently by adding to /etc/sysctl.conf and running sysctl -p.

Kernel Boot Parameters

Add these to GRUB_CMDLINE_LINUX in /etc/default/grub (then run update-grub):

• intel_idle.max_cstate=1: Reduce CPU power saving (lower latency for real-time apps).
• transparent_hugepage=never: Disable THP (avoids latency spikes in MongoDB).

8. Application-Level Tuning

Even well-tuned systems suffer if applications are misconfigured.

Database Tuning (MySQL)

• Query Optimization: Use EXPLAIN to fix slow queries, add indexes, and avoid full table scans.
• InnoDB Buffers: Set innodb_buffer_pool_size to 50-70% of available memory (e.g., 4G on an 8GB server).

Web Server Tuning (Nginx)

• Worker Processes: Set to number of CPU cores (worker_processes auto).
• Worker Connections: Increase with worker_connections 10240 (adjust based on memory).

Java Application Tuning (JVM)

• Heap Size: Set Xms (initial) and Xmx (max) to avoid resizing (e.g., -Xms4G -Xmx4G).
• Garbage Collection: Use G1GC for large heaps (-XX:+UseG1GC).

9. Continuous Monitoring and Alerting

Tuning isn’t a one-time task—monitor performance over time to catch regressions.

Tools

• Prometheus + Grafana: Open-source monitoring with custom dashboards. Use node_exporter to collect Linux metrics.
• Nagios/Zabbix: Enterprise-grade monitoring with alerting (email, Slack).

Example Grafana Dashboard

Create a dashboard with panels for:

• CPU usage (per core, %us/%sy).
• Memory: Used, free, swap.
• Disk I/O: %iowait, throughput, latency.

10. Best Practices for Linux Performance Tuning

1. Baseline First: Measure current performance before tuning to quantify improvements.
2. Tune Incrementally: Change one variable at a time and test results.
3. Monitor After Changes: Use sar to compare pre/post-tuning metrics.
4. Document Everything: Log changes, rationale, and results for future reference.

11. Conclusion

Linux performance tuning is an iterative process that requires understanding system behavior, diagnosing bottlenecks, and applying targeted optimizations. By mastering tools like perf, iostat, and sysctl, and following best practices for CPU, memory, disk, and network tuning, you can unlock significant performance gains. Remember: the goal isn’t to maximize raw speed, but to align system behavior with application requirements.

12. References

• Brendan Gregg’s Linux Performance – Authoritative guides and tools.
• Linux Kernel Documentation – Official kernel tuning docs.
• Red Hat Performance Tuning Guide
• TCP BBR Congestion Control
• Systems Performance: Enterprise and the Cloud by Brendan Gregg.

Happy tuning! 🚀