funwithlinux guide

Tuning Linux Systems for HPC Environments

High-Performance Computing (HPC) environments are designed to solve complex computational problems by leveraging parallel processing, massive data throughput, and low-latency communication. From climate modeling and drug discovery to computational fluid dynamics, HPC systems demand极致 (extreme) efficiency, reliability, and performance. Linux, with its open-source flexibility, robust kernel, and extensive tooling, has emerged as the operating system (OS) of choice for HPC. However, out-of-the-box Linux configurations are often optimized for general-purpose use, not the specialized demands of HPC workloads—such as high CPU utilization, low memory latency, fast I/O, and low-latency network communication. Tuning a Linux system for HPC involves optimizing hardware, kernel parameters, file systems, networks, and resource management to align with the unique needs of parallel applications. The goal is to minimize bottlenecks, reduce latency, maximize throughput, and ensure efficient utilization of every hardware component. This blog provides a comprehensive guide to tuning Linux for HPC, covering key areas from hardware considerations to monitoring and benchmarking.

Table of Contents

  1. Hardware Considerations for HPC

    • CPU Architecture
    • Memory (RAM) Optimization
    • Storage Subsystems
    • Network Interconnects

  2. Kernel Tuning

    • Kernel Version and Configuration
    • Key Sysctl Parameters
    • CPU Scheduler Tuning
    • Interrupt Handling

  3. File System Optimization

    • Local vs. Parallel File Systems
    • Mount Options and I/O Schedulers
    • Stripe Sizing and Layout

  4. Network Tuning

    • TCP/IP Optimization
    • RDMA and Low-Latency Interconnects
    • Jumbo Frames and MTU

  5. Resource Management

    • Job Schedulers (Slurm, PBS)
    • Process Pinning and NUMA Awareness
    • Cgroups and Resource Isolation

  6. Power and Thermal Management

    • CPU Frequency Scaling
    • C-States and P-States
    • Thermal Throttling Mitigation

  7. Monitoring and Benchmarking

    • Tools for Real-Time Monitoring
    • Benchmarking Workloads

  8. Conclusion

  9. References

1. Hardware Considerations for HPC

Before diving into software tuning, it’s critical to align hardware configuration with HPC workload requirements. HPC systems are typically composed of multi-socket servers with high-core-count CPUs, large memory pools, fast storage, and low-latency networks.

CPU Architecture

  • Multi-Core and Hyper-Threading: HPC workloads often scale with core count, but hyper-threading (HT) can introduce overhead for latency-sensitive applications. Disable HT if benchmarks show no benefit (e.g., echo off > /sys/devices/system/cpu/smt/control).
  • Cache Size: Larger L3 caches reduce memory access latency. Prioritize CPUs with high cache-per-core ratios (e.g., Intel Xeon Scalable, AMD EPYC).
  • SIMD Support: Ensure CPUs support advanced vector extensions (AVX-512, AVX2) for parallel floating-point operations, critical for scientific computing.

Memory (RAM) Optimization

  • Capacity and Bandwidth: HPC workloads (e.g., finite element analysis) require large, bandwidth-optimized memory. Use multi-channel DIMMs (e.g., 8-channel per socket for AMD EPYC) and populate memory slots evenly to maximize bandwidth.
  • NUMA Awareness: Non-Uniform Memory Access (NUMA) architectures mean memory near a CPU socket is faster to access. Use numactl --hardware to map CPU sockets to memory nodes and pin processes to local NUMA nodes (see Resource Management).
  • Huge Pages: Enable transparent huge pages (THP) or explicit huge pages to reduce TLB (Translation Lookaside Buffer) misses, which slow memory access. For THP: 
    echo always > /sys/kernel/mm/transparent_hugepage/enabled

Storage Subsystems

  • Parallel File Systems: For shared storage, use parallel file systems like Lustre, IBM Spectrum Scale (GPFS), or BeeGFS, which distribute I/O across multiple servers.
  • Local Storage: Use NVMe SSDs for node-local scratch space (low latency) and avoid HDDs for performance-critical workloads.
  • RAID Configuration: For redundant local storage, use RAID 0 (striping) for maximum throughput or RAID 10 (mirroring+striping) for a balance of speed and redundancy.

Network Interconnects

  • Low-Latency Fabrics: InfiniBand (IB) or Omni-Path are preferred over Ethernet for HPC due to lower latency (IB FDR: ~0.8µs, 100G Ethernet: ~2µs).
  • RDMA Support: Enable Remote Direct Memory Access (RDMA) to bypass the OS kernel for data transfers, reducing overhead. Use ibv_devinfo to verify RDMA-capable hardware.

2. Kernel Tuning

The Linux kernel is highly configurable, and tuning parameters can significantly impact HPC performance. Use a recent, stable kernel (e.g., 5.15+ LTS) with HPC-specific optimizations (e.g., CONFIG_HIGH_RES_TIMERS, CONFIG_PREEMPT_NONE for low latency).

Key Sysctl Parameters

Modify /etc/sysctl.conf or use sysctl -w to adjust runtime parameters:

ParameterPurposeRecommended Value
vm.swappinessControls swap usage. HPC avoids swapping.0 (disable swap if possible)
vm.dirty_ratio% of memory filled with dirty pages before writeback.40 (reduce for I/O-bound workloads)
vm.dirty_background_ratio% of memory triggering background writeback.10
net.core.rmem_max/wmem_maxMax TCP receive/send buffer size.268435456 (256MB) for large transfers
kernel.sched_min_granularity_nsMinimum time a task runs before preemption.10000000 (10ms) for batch workloads

CPU Scheduler Tuning

The default Completely Fair Scheduler (CFS) is suitable for most HPC workloads, but tuning can improve batch performance:

  • Batch Scheduling: Set sched_batch for non-interactive jobs to reduce preemption: 
    chrt -b -p 0 <pid>  # Set PID to batch priority
  • Latency vs. Throughput: For latency-sensitive apps, reduce sched_wakeup_granularity_ns (e.g., 5000000 ns) to prioritize wakeup speed.

Interrupt Handling

Network and storage I/O generate interrupts that can disrupt application performance. Isolate interrupts to dedicated CPU cores:

  • Use irqbalance to distribute interrupts, or manually pin them with smp_affinity: 
    # Pin interrupt 42 to CPU 8 (hex: 100)  
echo 100 > /proc/irq/42/smp_affinity

3. File System Optimization

HPC workloads rely heavily on file I/O, making file system tuning critical. Optimize for throughput (parallel writes) and low latency (small, random reads).

Local vs. Parallel File Systems

  • Local File Systems: Use XFS or ext4 for node-local storage. XFS is preferred for large files and high throughput: 
    mkfs.xfs -d su=64k,sw=8 /dev/nvme0n1  # Stripe unit=64k, 8 stripes
  • Parallel File Systems: For shared storage, configure Lustre with:
    • OST Striping: Stripe files across Object Storage Targets (OSTs) to parallelize I/O (e.g., lfs setstripe -c 8 /lustre/data/file for 8-way striping).
    • Client Cache Size: Increase llite.max_cached_mb to reduce OST round-trips.

Mount Options and I/O Schedulers

  • Mount Options: Disable access time tracking and enable writeback for XFS: 
    mount -t xfs -o noatime,nodiratime,logbsize=256k,swalloc /dev/nvme0n1 /scratch
  • I/O Schedulers: Use mq-deadline for rotational disks or none (noop) for NVMe (no mechanical seek time): 
    echo mq-deadline > /sys/block/sda/queue/scheduler

4. Network Tuning

HPC clusters depend on fast, reliable inter-node communication. Optimize TCP/IP and RDMA parameters for low latency and high bandwidth.

TCP/IP Optimization

  • Buffer Sizes: Increase TCP send/receive buffers to handle large transfers: 
    sysctl -w net.core.rmem_max=268435456  
sysctl -w net.core.wmem_max=268435456  
sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"  
sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"
  • Disable Offloading: For RDMA, disable TCP checksum offloading (handled by hardware): 
    ethtool -K eth0 tx-checksum-ip-generic off

RDMA and Low-Latency Interconnects

  • InfiniBand Configuration: Use ibsysctl to set maximum message size: 
    ibsysctl -w net.ib.rxe.max_msg_size=2147483647  # 2GB
  • Jumbo Frames: Enable 9000-byte MTU on InfiniBand/Ethernet to reduce packet overhead: 
    ethtool -s eth0 mtu 9000

5. Resource Management

Efficiently managing CPU, memory, and I/O resources ensures fair allocation and avoids contention between jobs.

Job Schedulers (Slurm, PBS)

  • Slurm Configuration:
    • Enable SelectType=select/cons_res for NUMA-aware job allocation.
    • Use --cpu-bind=cores to pin tasks to specific cores: 
      srun --cpu-bind=cores --nodes=2 --ntasks-per-node=8 ./my_hpc_app
    • Configure Gres (Generic Resources) to manage GPUs or specialized hardware.

Process Pinning and NUMA Awareness

  • numactl: Pin processes to NUMA nodes to avoid remote memory access: 
    numactl --cpunodebind=0 --membind=0 ./app  # Run on NUMA node 0
  • cgroups: Limit memory/cpu per job with Slurm’s cgroup integration (enable CgroupAutomount=yes in slurm.conf).

6. Power and Thermal Management

CPU frequency scaling and power-saving states can introduce latency. Disable them for HPC to maintain consistent performance.

CPU Frequency Scaling

  • Performance Governor: Lock CPUs to maximum frequency: 
    cpupower frequency-set --governor performance
  • Turbo Boost: Enable Turbo Boost (if supported) for short bursts of higher performance: 
    echo 1 > /sys/devices/system/cpu/cpufreq/boost

C-States and P-States

  • C-States: Disable deep idle states (C3, C6) to reduce wake-up latency: 
    echo 1 > /sys/module/intel_idle/parameters/max_cstate  # Intel CPUs
  • P-States: Avoid dynamic voltage/frequency scaling (DVFS) by setting a fixed P-state.

Thermal Throttling Mitigation

  • Monitor temperatures with sensors and ensure adequate cooling. Use ipmitool to adjust fan speeds: 
    ipmitool raw 0x30 0x30 0x01 0x01  # Set fans to full speed

7. Monitoring and Benchmarking

Tuning is iterative—use monitoring tools to identify bottlenecks and benchmarks to validate improvements.

Tools for Real-Time Monitoring

  • CPU/Memory: mpstat, vmstat, numastat (NUMA usage), perf top (CPU hotspots).
  • I/O: iostat -x 1 (disk throughput), dstat (I/O stats), lfs df (Lustre OST usage).
  • Network: ibstat (InfiniBand status), osu_latency (RDMA latency), iftop (bandwidth).

Benchmarking Workloads

  • CPU/Memory:
    • STREAM (memory bandwidth: ./stream_c.exe).
    • LINPACK (floating-point performance: mpirun -np 16 xhpl).
  • Storage:
    • IOR (parallel I/O: mpirun -np 32 ior -t 4k -b 1G -o /lustre/testfile).
  • Network:
    • OSU Micro-Benchmarks (osu_bw, osu_latency for RDMA bandwidth/latency).

8. Conclusion

Tuning Linux for HPC requires a holistic approach, combining hardware optimization, kernel tweaks, efficient resource management, and rigorous monitoring. By aligning the OS with the demands of parallel workloads—low latency, high throughput, and NUMA awareness—you can unlock the full potential of your HPC cluster. Always validate changes with benchmarks and prioritize stability alongside performance.

9. References