How to Use cgroups and Systemd for Linux Performance Tuning

Table of Contents

1. Understanding Control Groups (cgroups)
   • 1.1 What Are cgroups?
   • 1.2 Key Components of cgroups
   • 1.3 cgroups v1 vs. v2: Which Should You Use?
2. Systemd and cgroups: A Powerful Combination
   • 2.1 How systemd Manages cgroups
   • 2.2 The systemd cgroup Hierarchy
3. Getting Started: Prerequisites and Tools
   • 3.1 Checking Your cgroup Version
   • 3.2 Essential Tools for cgroup Management
4. Performance Tuning with cgroups and Systemd
   • 4.1 CPU Resource Management
   • 4.2 Memory Resource Control
   • 4.3 Block I/O (Disk) Throttling
   • 4.4 Network Bandwidth Control (Advanced)
5. Advanced Configuration: Custom systemd Services
   • 5.1 Creating a Service File with Resource Limits
   • 5.2 Applying and Testing the Service
6. Monitoring and Troubleshooting
   • 6.1 Using systemd-cgtop and systemctl
   • 6.2 Analyzing Metrics with cgroupfs
   • 6.3 Common Pitfalls and Solutions
7. Best Practices for cgroups and systemd Tuning
8. Conclusion
9. References

1. Understanding Control Groups (cgroups)

1.1 What Are cgroups?

Control groups (cgroups) are a Linux kernel feature that allows you to limit, account for, and isolate the resource usage (CPU, memory, disk I/O, network, etc.) of process groups. Introduced in 2008, cgroups provide a flexible framework for resource management, making them indispensable for use cases like containerization (e.g., Docker, Kubernetes), server virtualization, and general system tuning.

At their core, cgroups enable you to:

• Limit resources (e.g., “restrict this app to 2 CPU cores”).
• Account for resource usage (e.g., “track how much memory this service consumed”).
• Isolate workloads (e.g., “prevent a noisy process from starving others”).

1.2 Key Components of cgroups

To use cgroups effectively, it helps to understand their core components:

• Hierarchies: A tree-like structure of cgroups. Each hierarchy is associated with one or more resource controllers (subsystems) and manages a subset of processes.
• Resource Controllers (Subsystems): Kernel modules that regulate specific resources. Examples include:
  • cpu: Limits CPU time and enforces priorities.
  • memory: Controls memory (RAM and swap) usage.
  • blkio: Throttles block device (disk) I/O.
  • cpuset: Assigns processes to specific CPU cores and NUMA nodes.
  • pids: Limits the number of processes in a cgroup.
• cgroups: Nodes in a hierarchy that group processes. Each cgroup inherits properties from its parent but can override them. Processes can belong to multiple cgroups (one per hierarchy).

1.3 cgroups v1 vs. v2: Which Should You Use?

There are two versions of cgroups: v1 (legacy) and v2 (unified).

Why v2? cgroups v2 addresses v1’s limitations (e.g., conflicting hierarchies) and adds features like better memory isolation and unified resource management. Systemd, the dominant init system, uses v2 by default on modern Linux distributions. For this blog, we’ll focus on v2.

2. Systemd and cgroups: A Powerful Combination

Systemd, the init system used by most Linux distributions (Debian, Ubuntu, RHEL, Fedora, etc.), uses cgroups to manage all system processes. Every systemd unit (service, scope, slice) is a cgroup, making it easy to apply resource limits to services and workloads.

2.1 How systemd Manages cgroups

Systemd automatically assigns processes to cgroups, with each unit type corresponding to a cgroup role:

• Services (*.service): Managed long-running processes (e.g., nginx.service, sshd.service).
• Scopes (*.scope): Transient processes started by users or systemd (e.g., session-123.scope for user sessions).
• Slices (*.slice): Group units into hierarchical categories (e.g., system.slice for system services, user.slice for user processes).

2.2 The systemd cgroup Hierarchy

Systemd organizes cgroups into a tree structure under the unified hierarchy (/sys/fs/cgroup for v2). A typical hierarchy looks like:

/sys/fs/cgroup  
├── init.scope          # systemd itself  
├── system.slice        # System services (e.g., nginx, sshd)  
│   ├── nginx.service  
│   └── sshd.service  
├── user.slice          # User processes  
│   ├── user-1000.slice  
│   │   └── session-1.scope  
└── machine.slice       # Virtual machines/containers  

• Slices are the top-level organizers. For example, system.slice contains all system services, while user.slice contains user sessions.
• Services live under slices (e.g., system.slice/nginx.service).
• Scopes are for short-lived or user-started processes (e.g., a terminal session).

To inspect the hierarchy, run:

systemctl list-units --type=slice  

3. Getting Started: Prerequisites and Tools

3.1 Checking Your cgroup Version

To confirm if your system uses cgroups v2:

stat -fc %T /sys/fs/cgroup  

• Output cgroup2fs → v2 (good!).
• Output tmpfs → v1 (check your distro; you may need to enable v2 via kernel boot parameters).

3.2 Essential Tools for cgroup Management

• systemctl: Manage systemd units and their cgroups (e.g., start/stop services, edit resource limits).
• systemd-cgtop: Real-time monitor for cgroup resource usage (like top but for cgroups).
• cgroupfs: The kernel’s cgroup filesystem (mounted at /sys/fs/cgroup), used to read/write cgroup settings and metrics.
• journalctl: View logs for services, including cgroup-related events (e.g., OOM kills).

4. Performance Tuning with cgroups and Systemd

Let’s dive into practical tuning for key resources using systemd and cgroups v2.

4.1 CPU Resource Management

CPU tuning ensures critical services get enough CPU time while limiting resource hogs. Systemd provides several directives to control CPU usage in service files:

1. CPU Shares (Relative Weight)

CPUShares= assigns a relative priority (weight) to a service. Default: 1024. Higher values mean more CPU time when resources are contested.

Example: Give nginx.service twice the CPU priority of default services:

[Service]  
CPUShares=2048  

2. CPU Quota (Absolute Limit)

CPUQuota= sets an absolute limit on CPU usage (e.g., 50% = 50% of one core, 200% = 2 cores on a 4-core system).

Syntax: CPUQuota=<percentage>% (e.g., 50%, 150%).

Example: Limit a batch job service to 1 core (100%):

[Service]  
CPUQuota=100%  

3. CPU Set (Pin to Cores)

CPUSetCpus= restricts a service to specific CPU cores (e.g., 0-1 for cores 0 and 1). Useful for NUMA systems or isolating latency-sensitive workloads.

Example: Pin redis.service to cores 2 and 3:

[Service]  
CPUSetCpus=2-3  

4.2 Memory Resource Control

Memory tuning prevents services from consuming excessive RAM and triggering OOM (Out-of-Memory) kills. Key directives:

1. Memory Limit (Hard Cap)

MemoryLimit= sets a hard limit on RAM usage (e.g., 512M, 2G). If exceeded, the kernel may kill processes in the cgroup.

Example: Limit mysql.service to 2GB RAM:

[Service]  
MemoryLimit=2G  

2. Memory Low/High (Soft Limits)

• MemoryLow= (v2 only): Minimum memory guarantee. The kernel prioritizes this cgroup when memory is low.
• MemoryHigh= (v2 only): Soft limit. The kernel throttles the cgroup if usage exceeds this, but doesn’t kill processes.

Example: Guarantee 1GB RAM to nginx.service and soft-limit at 1.5GB:

[Service]  
MemoryLow=1G  
MemoryHigh=1.5G  
MemoryLimit=2G  # Hard limit  

3. Swap Control

MemorySwapMax= limits total memory (RAM + swap) usage. Set to 0 to disable swap for the service.

Example: Allow myapp.service 512MB RAM + 256MB swap:

[Service]  
MemoryLimit=512M  
MemorySwapMax=768M  # 512M RAM + 256M swap  

4.3 Block I/O (Disk) Throttling

Use blkio directives to limit disk read/write speeds, preventing a single service from saturating storage.

1. Bandwidth Limits

• BlockIOReadBandwidth=<device> <speed>: Limit read speed (e.g., /dev/sda 100M).
• BlockIOWriteBandwidth=<device> <speed>: Limit write speed (e.g., /dev/nvme0n1 50M).

Example: Throttle backup.service to 100MB/s reads and 50MB/s writes on /dev/sda:

[Service]  
BlockIOReadBandwidth=/dev/sda 100M  
BlockIOWriteBandwidth=/dev/sda 50M  

2. I/O Weight (Relative Priority)

BlockIOWeight= sets relative I/O priority (default: 1000). Higher values get more I/O time when the disk is busy.

Example: Give database.service higher I/O priority than backups:

[Service]  
BlockIOWeight=2000  # Twice the default  

4.4 Network Bandwidth Control (Advanced)

cgroups v2 includes a net_cls controller, but network tuning is less mature than CPU/memory. For simple cases, use tc (traffic control) with cgroups, or leverage systemd’s experimental NetworkBandwidth= directive (available in systemd ≥251).

Example (experimental): Limit download.service to 100Mbps:

[Service]  
NetworkBandwidth=100M  

For production, consider tools like nftables or tc with cgroup matching.

5. Advanced Configuration: Custom systemd Services

Let’s create a custom service with resource limits from scratch. We’ll use a hypothetical data-processor.service that needs CPU, memory, and I/O constraints.

5.1 Creating a Service File with Resource Limits

Create /etc/systemd/system/data-processor.service:

[Unit]  
Description=Data Processing Service  
After=network.target  

[Service]  
Type=simple  
ExecStart=/opt/data-processor/bin/processor  
Restart=on-failure  

# CPU Limits  
CPUQuota=200%  # 2 cores  
CPUShares=1536  # Higher priority than default  

# Memory Limits  
MemoryLow=1G    # Guarantee 1GB RAM  
MemoryHigh=1.5G # Soft limit  
MemoryLimit=2G  # Hard limit (kill if exceeded)  
MemorySwapMax=0 # No swap  

# Disk I/O Limits  
BlockIOReadBandwidth=/dev/sda 200M  
BlockIOWriteBandwidth=/dev/sda 100M  
BlockIOWeight=1500  # Higher I/O priority  

[Install]  
WantedBy=multi-user.target  

5.2 Applying and Testing the Service

1. Reload systemd to detect the new service:

   sudo systemctl daemon-reload  

2. Start the service:

   sudo systemctl start data-processor.service  

3. Verify resource limits with systemctl show:

   sudo systemctl show data-processor.service | grep -E 'CPUQuota|MemoryLimit|BlockIO'  

4. Monitor in real time with systemd-cgtop:

   sudo systemd-cgtop /system.slice/data-processor.service  

6. Monitoring and Troubleshooting

6.1 Using systemd-cgtop and systemctl

• systemd-cgtop: View live resource usage (CPU, memory, I/O) for cgroups. Filter by slice/service:

  sudo systemd-cgtop /system.slice  # All system services  
  sudo systemd-cgtop /user.slice    # User processes  

• systemctl status: Check if a service is throttled or killed:

  sudo systemctl status data-processor.service  

6.2 Analyzing cgroup Metrics with cgroupfs

cgroup metrics are exposed in /sys/fs/cgroup/<cgroup-path>/. For data-processor.service:

• CPU usage: /sys/fs/cgroup/system.slice/data-processor.service/cpu.stat
• Memory usage: /sys/fs/cgroup/system.slice/data-processor.service/memory.current
• I/O stats: /sys/fs/cgroup/system.slice/data-processor.service/io.stat

Example: Check current memory usage:

cat /sys/fs/cgroup/system.slice/data-processor.service/memory.current  

6.3 Common Pitfalls and Solutions

7. Best Practices for cgroups and systemd Tuning

1. Monitor First, Tune Later: Use systemd-cgtop and journalctl to identify resource hogs before setting limits.
2. Start with Soft Limits: Use MemoryHigh= or CPUQuota= before hard limits like MemoryLimit=.
3. Use Slices for Grouping: Organize related services into custom slices (e.g., analytics.slice for all data services).
4. Avoid Over-Limiting: Set limits 10-20% higher than observed peak usage to avoid throttling under load.
5. Document Changes: Track service file modifications in version control (e.g., /etc/systemd/system/).
6. Test Limits Under Load: Simulate traffic with tools like stress-ng to validate limits don’t break functionality.

8. Conclusion

cgroups and systemd are powerful tools for Linux performance tuning, enabling precise control over CPU, memory, disk I/O, and more. By leveraging systemd’s integration with cgroups v2, you can ensure services get the resources they need while preventing resource starvation.

Start small: Identify critical services, apply basic limits, and monitor results. Over time, refine your configuration using the best practices outlined here. With cgroups and systemd, you’ll transform a chaotic system into a well-tuned machine.

9. References

• Linux Kernel cgroups v2 Documentation
• systemd.resource-control(5) Manual
• systemd-cgtop(1) Manual
• Red Hat: Managing Resources with systemd and cgroups
• Ubuntu Server Guide: cgroups
• cgroups v2 vs. v1 Comparison