How to Debug Systemd: A Guide for System Administrators

Table of Contents

1. Understanding Systemd Basics
2. Essential Systemd Debugging Tools
3. Checking Service Status and Logs
4. Debugging Failed Services
5. Boot-Time Debugging
6. Unit File Validation and Debugging
7. Advanced Debugging Techniques
8. Best Practices for Systemd Debugging
9. References

1. Understanding Systemd Basics

Before diving into debugging, it’s critical to grasp core systemd concepts:

• Units: The fundamental building blocks of systemd, representing services, sockets, devices, mounts, targets, etc. Units are defined in .service, .socket, .mount, or .target files.
• Targets: Groups of units that define system states (e.g., multi-user.target for a multi-user command-line system, graphical.target for a GUI).
• Journald: Systemd’s logging daemon, which collects and stores logs from services, the kernel, and other system components.
• Control Groups (cgroups): Systemd uses cgroups to manage resource allocation (CPU, memory) for processes and units.

2. Essential Systemd Debugging Tools

Systemd provides a suite of built-in tools to diagnose issues. Familiarize yourself with these:

systemctl

The primary command for managing systemd units. Use it to start/stop services, check statuses, and modify unit behavior.

journalctl

Queries the journald log database to retrieve logs for units, boot sessions, or specific time ranges.

systemd-analyze

Analyzes boot performance, validates unit files, and generates dependency graphs.

systemd-cgls / systemd-cgtop

Inspect control groups (cgroups) to monitor resource usage by units and processes.

coredumpctl

Retrieves and analyzes coredump files for crashed processes (useful for debugging service crashes).

3. Checking Service Status and Logs

When a service misbehaves, start with the basics: checking its status and logs.

Step 1: Check Service Status with systemctl status

Use systemctl status <unit> to get a high-level overview of a service:

systemctl status sshd.service

Sample Output:

● sshd.service - OpenSSH server daemon
     Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-03-12 10:00:00 UTC; 2h ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 1234 (sshd)
      Tasks: 1 (limit: 4915)
     Memory: 5.2M
        CPU: 1.2s
     CGroup: /system.slice/sshd.service
             └─1234 /usr/sbin/sshd -D

Mar 12 10:00:00 server systemd[1]: Started OpenSSH server daemon.
Mar 12 10:05:00 server sshd[1234]: Accepted publickey for user from 192.168.1.1 port 54321 ssh2: RSA SHA256:...

Key Fields:

• Loaded: Whether the unit is loaded, enabled (starts on boot), or masked.
• Active: Current state (e.g., active (running), failed, inactive).
• Main PID: The process ID of the primary service process.
• CGroup: The control group path for resource management.
• Recent logs: The last few lines of the service’s journal.

Step 2: Inspect Logs with journalctl

If the status shows failed, use journalctl to dig deeper. Filter logs by unit, time, or priority:

• Show all logs for a unit:

  journalctl -u sshd.service

• Follow real-time logs (like tail -f):

  journalctl -u sshd.service -f

• Filter by priority (e.g., only errors):

  journalctl -u sshd.service -p err  # -p 3 (err), 2 (crit), 1 (alert), 0 (emerg)

• Logs from previous boots:

  journalctl -u sshd.service -b -1  # -b -1 = previous boot, -b 0 = current boot

4. Debugging Failed Services

A failed status in systemctl status indicates the service couldn’t start. Here’s how to diagnose:

Common Failure Causes & Fixes

1. Permission Denied Errors

If logs show Permission denied, the service user (e.g., sshd, nginx) lacks access to files/directories.

Example Log:

sshd[1234]: error: Bind to port 22 on 0.0.0.0 failed: Permission denied

Fix: Ensure the service user has read/execute permissions on the binary (e.g., /usr/sbin/sshd) and write access to log directories.

2. Invalid ExecStart Path

If ExecStart in the unit file points to a non-existent binary, the service will fail.

Example Unit File Error:

[Service]
ExecStart=/usr/bin/sshd  # Incorrect path; should be /usr/sbin/sshd

Fix: Verify the binary path with which sshd, then update ExecStart.

3. Dependency Failures

Services often depend on other units (e.g., network.target for network access). Use systemctl list-dependencies to check:

systemctl list-dependencies sshd.service --reverse  # Show what depends on sshd
systemctl list-dependencies sshd.service           # Show what sshd depends on

Fix: Ensure dependencies (e.g., network-online.target) are active before the service starts. Adjust After= or Requires= in the unit file:

[Unit]
After=network-online.target  # Start sshd only after network is online
Requires=network-online.target

4. Configuration File Errors

Malformed config files (e.g., /etc/ssh/sshd_config) can cause services to fail. Check logs for syntax errors:

journalctl -u sshd.service | grep "configuration error"

Fix: Validate config files with tools like sshd -t (for OpenSSH) or nginx -t (for Nginx).

5. Boot-Time Debugging

If the system is slow to boot or fails to start, use these tools to diagnose boot issues.

Identify Slow Boot Services with systemd-analyze blame

systemd-analyze blame lists units by their boot time, highlighting bottlenecks:

systemd-analyze blame

Sample Output:

12.345s NetworkManager-wait-online.service
 5.678s docker.service
 2.345s udisks2.service

Here, NetworkManager-wait-online.service is delaying boot by 12 seconds. Fixes may include disabling NetworkManager-wait-online (if network isn’t critical for early boot) or optimizing network configuration.

Critical Boot Path with systemd-analyze critical-chain

The “critical chain” shows units that directly impact boot time (e.g., systemd-udevd, sysinit.target):

systemd-analyze critical-chain

Sample Output:

The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

multi-user.target @15.000s
└─docker.service @9.325s +5.675s
  └─network-online.target @9.324s
    └─NetworkManager-wait-online.service @-3.020s +12.344s
      └─NetworkManager.service @-3.100s +0.076s
        └─dbus.service @-3.200s
          └─basic.target @-3.250s
            └─sockets.target @-3.250s
              └─docker.socket @-3.250s +0.000s
                └─sysinit.target @-3.260s

Emergency and Rescue Modes

If the system won’t boot, use emergency mode (minimal shell with read-only root) or rescue mode (basic functional system):

• Emergency Mode: Add emergency to the kernel command line (via GRUB) to boot into a shell with root mounted read-only.
• Rescue Mode: Add rescue to the kernel command line for a more functional environment.

Once in emergency/rescue mode, check logs with journalctl -b and fix broken units or config files.

6. Unit File Validation and Debugging

Malformed unit files are a common source of issues. Validate and debug them with these steps.

Validate Unit Files with systemd-analyze verify

Check for syntax errors in unit files:

systemd-analyze verify sshd.service

Example Error:

sshd.service:23: Unknown key name 'ExecStartt' in section 'Service'

Here, a typo (ExecStartt instead of ExecStart) is causing the error.

Override Units Safely with systemctl edit

Avoid editing vendor-provided unit files directly (they may be overwritten by updates). Instead, create an override file:

systemctl edit sshd.service  # Opens an editor for overrides

This creates /etc/systemd/system/sshd.service.d/override.conf, which takes precedence over the original unit file.

Check Unit Dependencies and Conflicts

Use systemctl show to inspect unit properties like Requires, After, or Conflicts:

systemctl show sshd.service --property=Requires,After,Conflicts

Sample Output:

Requires=basic.target system.slice
After=network.target auditd.service systemd-journald.socket basic.target system.slice
Conflicts=shutdown.target

7. Advanced Debugging Techniques

For stubborn issues, use these advanced tools:

Trace System Calls with strace

strace monitors system calls (e.g., open(), read()) made by a service, revealing file access issues or missing dependencies:

• Trace a running service:

  strace -p $(systemctl show -p MainPID sshd.service --value)

• Trace on startup (use systemd-run for transient units):

  systemd-run --user --pty --service-type=exec strace -f /usr/sbin/sshd -D

Enable Systemd Debug Logging

To debug systemd itself (e.g., if the init process is misbehaving), enable debug logging by adding systemd.log_level=debug to the kernel command line (via GRUB). Reboot, then check logs with:

journalctl -u systemd -p debug

Coredump Analysis with coredumpctl

If a service crashes, use coredumpctl to retrieve and analyze the coredump:

coredumpctl list sshd  # List coredumps for sshd
coredumpctl debug sshd  # Open the latest coredump in gdb

8. Best Practices for Systemd Debugging

• Document Changes: Track modifications to unit files or systemd configs (e.g., in a wiki or version control).
• Test in Staging: Validate new unit files or service changes in a non-production environment first.
• Version Control Unit Files: Store custom unit files in Git to revert changes if debugging breaks things.
• Monitor Systemd Metrics: Use tools like Prometheus + Node Exporter to track unit status, boot time, and resource usage over time.

9. References

• systemd(1) Man Page
• systemctl(1) Man Page
• journalctl(1) Man Page
• systemd-analyze(1) Man Page
• Systemd Documentation (freedesktop.org)
• Red Hat Systemd Debugging Guide