Exploring Systemd's cgroups for Advanced Resource Management

Table of Contents

1. Understanding cgroups: What Are Control Groups?
   • cgroups v1 vs. v2: Key Differences
2. Systemd and cgroups: A Symbiotic Relationship
   • How Systemd Manages cgroups
   • Systemd’s cgroup Hierarchy: Slices, Scopes, and Services
3. Key Concepts in Systemd’s cgroup Implementation
   • Slices: Organizing Workloads
   • Scopes: Managing External Processes
   • Services: Systemd’s Core Units
   • cgroup.procs and Thread Ownership
4. Practical Guide: Managing Resources with Systemd
   • Viewing cgroup Configurations
   • Setting CPU Limits
   • Controlling Memory Usage
   • Managing Block I/O
   • Limiting Process IDs (PIDs)
5. Advanced Use Cases
   • Custom Slice Hierarchies for Multi-Tenant Environments
   • Dynamic Resource Adjustment
   • Integration with Container Runtimes
   • Monitoring cgroup Metrics
6. Troubleshooting Common cgroup Issues
7. Best Practices for Systemd cgroup Management
8. Conclusion
9. References

1. Understanding cgroups: What Are Control Groups?

Cgroups (short for “control groups”) are a Linux kernel feature introduced in 2006 that allow administrators to isolate, limit, and account for system resources consumed by processes or groups of processes. Resources controlled by cgroups include:

• CPU (usage, shares, quotas)
• Memory (usage limits, swap limits)
• Block I/O (bandwidth, priority)
• Network I/O (via extensions like net_cls or tc)
• Process IDs (maximum number of PIDs per group)

Cgroups act as a “fence” around processes, ensuring no single workload hogs resources, and enabling fair distribution across users, applications, or containers.

cgroups v1 vs. v2: Key Differences

Cgroups have evolved through two major versions, and understanding their differences is critical for working with systemd:

Systemd fully embraces cgroups v2, leveraging its unified hierarchy for cleaner, more reliable resource management. Most modern Linux distributions now default to v2, so we’ll focus on v2 in this guide.

2. Systemd and cgroups: A Symbiotic Relationship

Systemd isn’t just a service manager—it’s a cgroup manager. Unlike traditional tools (e.g., cgmanager), systemd tightly couples service management with cgroup control, making it the de facto interface for cgroup configuration on most Linux systems.

How Systemd Manages cgroups

Systemd automatically assigns every process to a cgroup, using its own unit system (services, scopes, slices) to organize the cgroup hierarchy. This integration means:

• Every systemd unit (e.g., nginx.service, [email protected]) maps to a cgroup.
• Resource limits defined in unit files are enforced via cgroups.
• Systemd tools like systemctl and journalctl provide visibility into cgroup metrics.

Systemd’s cgroup Hierarchy: Slices, Scopes, and Services

Systemd organizes cgroups into a tree-like hierarchy with three core unit types: slices, scopes, and services. Here’s how they fit together:

/- (root cgroup)  
├─ system.slice/ (system services)  
│  ├─ nginx.service/  
│  ├─ docker.service/  
│  └─ ...  
├─ user.slice/ (user sessions)  
│  ├─ [email protected]/  
│  │  └─ gnome-terminal.scope/  
│  └─ ...  
└─ machine.slice/ (containers VMs)  
   └─ libvirt-qemu.slice/  

• Slices: High-level containers for grouping related units (e.g., system.slice for system services).
• Scopes: Manage processes started outside systemd (e.g., user sessions, containers).
• Services: Systemd’s primary unit for managing long-running processes (e.g., sshd.service).

3. Key Concepts in Systemd’s cgroup Implementation

Slices: Organizing Workloads

Slices are the “folders” of the cgroup hierarchy. They group related units to enforce resource limits at a macro level. Systemd defines several default slices:

• -.slice: The root slice (parent of all other slices).
• system.slice: Contains system services (e.g., apache2.service, sshd.service).
• user.slice: Manages user sessions (e.g., [email protected] for UID 1000).
• machine.slice: For virtual machines and containers (e.g., libvirt VMs, LXC containers).

Example: A custom slice like webapps.slice could group frontend.service and backend.service, allowing you to limit CPU/memory for the entire web stack.

Scopes: Managing External Processes

Scopes handle processes not started by systemd itself (e.g., user-launched apps, container runtimes like Docker). They are created dynamically when a process is spawned (e.g., via systemd-run or logind for user sessions).

Example: When you open a terminal, systemd-logind creates a scope (e.g., gnome-terminal.scope) to track its processes.

Services: Systemd’s Core Units

Services are the most common systemd units and map directly to long-running processes. Every service runs in its own cgroup, making it easy to isolate and limit resources for individual applications.

Example: nginx.service runs in /sys/fs/cgroup/system.slice/nginx.service/, with its own CPU, memory, and I/O limits.

cgroup.procs and Thread Ownership

Each cgroup contains a cgroup.procs file listing the PIDs of processes in that cgroup. Importantly, threads (lightweight processes) are not tracked here—only main PIDs. This ensures resource limits apply to entire processes, not individual threads.

4. Practical Guide: Managing Resources with Systemd

Let’s dive into hands-on resource management using systemd and cgroups.

Viewing cgroup Configurations

Before setting limits, inspect existing cgroup setups with these tools:

• systemctl status <unit>: Shows cgroup path and basic resource metrics for a unit.

  systemctl status nginx.service  
  # Output includes: CGroup: /system.slice/nginx.service  

• systemd-cgls: Lists the entire cgroup hierarchy.

  systemd-cgls /system.slice/nginx.service  # Inspect nginx’s cgroup  

• systemd-cgtop: Real-time monitor for cgroup resource usage (CPU, memory, I/O).

  systemd-cgtop  # Similar to top, but for cgroups  

• systemctl show <unit>: Dump all properties (including cgroup limits) for a unit.

  systemctl show nginx.service | grep -i cpu  # Check CPU limits  

Setting CPU Limits

To limit CPU usage, enable CPU accounting and define a quota or shares.

Step 1: Enable CPU Accounting

In the service unit file (e.g., /etc/systemd/system/nginx.service), add:

[Service]  
CPUAccounting=yes  # Required to track/enforce CPU limits  

Step 2: Define Limits

• CPUQuota=<percentage>: Restrict CPU usage to a percentage of a core (e.g., 50% = 0.5 cores).

  CPUQuota=50%  # Nginx can use up to 50% of one core  

• CPUShares=<weight>: Prioritize CPU access (default: 1024; higher = more priority).

  CPUShares=512  # Lower priority than default (1024)  

Reload and restart the service for changes to take effect:

sudo systemctl daemon-reload  
sudo systemctl restart nginx.service  

Controlling Memory Usage

Limit memory (RAM + swap) to prevent applications from consuming excessive resources.

Step 1: Enable Memory Accounting

[Service]  
MemoryAccounting=yes  # Required for memory limits  

Step 2: Define Limits

• MemoryLimit=<size>: Hard limit (e.g., 512M, 2G). Processes exceeding this are killed.

  MemoryLimit=1G  # Max 1GB RAM  

• MemoryHigh=<size>: Soft limit (processes are throttled but not killed if exceeded).

  MemoryHigh=512M  # Throttle when using >512M  

Example Unit File:

[Unit]  
Description=Example Service with Memory Limits  

[Service]  
ExecStart=/usr/bin/myapp  
MemoryAccounting=yes  
MemoryHigh=512M  
MemoryLimit=1G  

[Install]  
WantedBy=multi-user.target  

Managing Block I/O

Control disk I/O bandwidth and priority for services to prevent I/O starvation.

Step 1: Enable Block I/O Accounting

[Service]  
BlockIOAccounting=yes  # Required for I/O limits  

Step 2: Define Limits

• BlockIOWeight=<weight>: I/O priority (1-1000; default 500). Higher = more I/O time.

  BlockIOWeight=750  # Higher I/O priority than default  

• BlockIOReadBandwidth=<device>=<speed>: Limit read speed (e.g., /dev/sda 100M).

  BlockIOReadBandwidth=/dev/sda 100M  # Max 100MB/s reads on /dev/sda  

Limiting Process IDs (PIDs)

Prevent fork bombs or runaway processes by limiting the number of PIDs a cgroup can spawn.

Step 1: Enable PID Accounting

[Service]  
TasksAccounting=yes  # Required for PID limits  

Step 2: Set PID Limit

TasksMax=100  # Allow max 100 PIDs in this cgroup  

5. Advanced Use Cases

Custom Slice Hierarchies for Multi-Tenant Environments

Create custom slices to isolate workloads (e.g., tenants, teams, or project groups).

Example: tenant-a.slice

1. Create a slice unit file at /etc/systemd/system/tenant-a.slice:

   [Unit]  
   Description=Slice for Tenant A Workloads  
   Before=slices.target  
   
   [Slice]  
   CPUAccounting=yes  
   CPUQuota=200%  # Allow 2 CPU cores  
   MemoryAccounting=yes  
   MemoryLimit=4G  # Max 4GB RAM for all Tenant A services  

2. Assign services to the slice by adding Slice=tenant-a.slice to their unit files:

   [Service]  
   Slice=tenant-a.slice  
   ExecStart=/usr/bin/tenant-app  

3. Reload systemd and start the slice:

   sudo systemctl daemon-reload  
   sudo systemctl start tenant-a.slice  

Dynamic Resource Adjustment

Temporarily adjust resource limits without editing unit files using systemctl set-property:

# Temporarily set Nginx CPU quota to 75%  
sudo systemctl set-property nginx.service CPUQuota=75%  

# Reset to unit file default (after testing)  
sudo systemctl set-property nginx.service CPUQuota=  # Empty value = reset  

Integration with Container Runtimes

Tools like Docker and Podman use systemd cgroups to manage container resources. For example:

• Docker can be configured to use the systemd cgroup driver (instead of cgroupfs) for better integration.
• Containers are automatically placed in machine.slice (e.g., docker-<container-id>.scope).

Configure Docker for systemd cgroups:
Edit /etc/docker/daemon.json:

{  
  "exec-opts": ["native.cgroupdriver=systemd"]  
}  

Restart Docker:

sudo systemctl restart docker.service  

Monitoring cgroup Metrics

• systemd-cgtop: Real-time dashboard for cgroup CPU, memory, and I/O usage.
• Prometheus + Node Exporter: Collect cgroup metrics from /sys/fs/cgroup/ (e.g., node_cgroup_memory_usage_bytes).
• Journalctl: Track resource-related events (e.g., OOM kills):

  journalctl -u nginx.service | grep -i "out of memory"  

6. Troubleshooting Common cgroup Issues

• Resource Limits Not Applying:

  • Ensure accounting is enabled (e.g., CPUAccounting=yes).
  • Verify cgroup version: Use stat -fc %T /sys/fs/cgroup/—output cgroup2fs confirms v2.

• cgroup Version Conflicts:

  • If using cgroups v1, some systemd features (e.g., unified hierarchy) won’t work. Upgrade to a kernel supporting v2 (5.2+).

• OOM Events:

  • Check journalctl -k | grep -i "out of memory" to identify processes killed by the OOM killer.
  • Increase MemoryLimit or adjust MemoryHigh for the affected service.

7. Best Practices for Systemd cgroup Management

1. Enable Accounting Selectively: Only enable CPUAccounting, MemoryAccounting, etc., for units that need limits (avoids performance overhead).
2. Use Slices for Logical Grouping: Group related services into custom slices to simplify resource management.
3. Avoid Over-Limiting Critical Services: Leave buffer space for peak loads (e.g., don’t set MemoryLimit to 100% of available RAM).
4. Test Limits in Staging: Validate resource limits under load to prevent outages in production.
5. Monitor Continuously: Use systemd-cgtop and Prometheus to track cgroup usage and adjust limits proactively.

8. Conclusion

Systemd’s integration with cgroups transforms resource management from a low-level kernel task into a user-friendly, service-oriented process. By mastering slices, scopes, and services, you can enforce granular limits, prevent resource starvation, and ensure stability in complex environments—whether managing a single server or a multi-tenant cloud platform.

As Linux continues to dominate container and cloud workloads, systemd’s cgroup tools will remain essential for efficient, scalable resource control. Start small (e.g., limiting a non-critical service) and gradually adopt advanced patterns like custom slices and dynamic adjustments to unlock the full potential of cgroups.

9. References

• Systemd Documentation: CGroup Documentation
• Linux Kernel cgroups v2 Documentation
• Red Hat: Managing System Resources with Systemd
• Ubuntu Wiki: Systemd cgroups
• Docker Documentation: Cgroup Drivers