funwithlinux guide

Troubleshooting Common Systemd Issues in Linux

Systemd has become the de facto init system for most modern Linux distributions, replacing traditional SysVinit. It manages system startup, service lifecycle, logging, network configuration, and more. While powerful, systemd’s complexity can lead to confusing issues—from failed services to boot problems. This guide demystifies common systemd pitfalls, providing step-by-step troubleshooting workflows and solutions.

Table of Contents

2. Service Fails to Start

Symptoms

  • systemctl start <service> returns an error (e.g., “Failed to start .service”).
  • systemctl status <service> shows “inactive (dead)” or “failed” with a red status.
  • The service does not load on boot despite being enabled.

Diagnosis Steps

  1. Check Service Status:

    systemctl status <service> -l  # -l shows full logs in the status output

    Look for error messages like “Unit .service not found” or “Permission denied”.

  2. Inspect Logs with Journald:
    Journald captures detailed service logs. Use:

    journalctl -u <service>  # Show logs for the service  
journalctl -u <service> -f  # "Follow" real-time logs (useful for debugging on start)

    Common errors here include missing binaries, invalid configuration files, or failed pre-start scripts.

  3. Verify Unit File Syntax:
    Systemd unit files (.service, .timer, etc.) define how services run. A syntax error will prevent startup:

    systemd-analyze verify <service>.service

    Unit files are typically stored in:

    • /etc/systemd/system/ (user-defined units)
    • /usr/lib/systemd/system/ (distro-provided units)

Solutions

  • Unit Not Found: The service package may not be installed. Install it (e.g., sudo apt install <package> for Debian/Ubuntu).
  • Permission Issues: Ensure the service’s executable or configuration files have correct permissions (e.g., chmod 644 /etc/<service>/config).
  • Invalid Unit Syntax: Fix typos in the unit file (e.g., missing = in ExecStart=/path/to/binary). Reload systemd after edits: 
    systemctl daemon-reload
  • Dependency Failures: If the service depends on another unit (e.g., After=network.target), ensure the dependency is active. Use systemctl list-dependencies <service> to check.

3. High CPU/Memory Usage by systemd or Its Services

Symptoms

  • System lag or unresponsiveness.
  • top/htop shows systemd, systemd-journald, or a specific service (e.g., systemd-resolved) using >50% CPU/memory.

Diagnosis Steps

  1. Identify the Culprit:
    Use htop to sort by CPU/memory (press P for CPU, M for memory). Note the process name (e.g., systemd-journald).

  2. Check Service-Specific Logs:
    For a misbehaving service (e.g., nginx), check its logs with journalctl -u nginx. Spamming logs (e.g., “error connecting to database”) can cause high CPU in systemd-journald.

  3. Analyze Boot Services:
    Use systemd-analyze blame to identify slow-starting services that may be stuck in loops:

    systemd-analyze blame  # Lists services by boot time (slowest first)

Solutions

  • Log Spam: If systemd-journald is using high CPU, check for services spamming logs (e.g., a script with infinite error messages). Fix the root cause (e.g., repair the script) and clear logs: 
    journalctl --vacuum-size=100M  # Reduce journal size to 100MB
  • Stuck Services: Restart the problematic service: 
    systemctl restart <service>
    If it persists, mask the service temporarily (prevents auto-start): 
    systemctl mask <service>
  • Memory Leaks: Update the service or systemd (distro updates often fix leaks). For example: 
    sudo apt update && sudo apt upgrade systemd

4. Boot Problems (e.g., Stuck at Boot, Emergency Mode)

Symptoms

  • Boot hangs at “Loading initial ramdisk…” or “Reached target Basic System”.
  • System drops to “Emergency Mode” (root shell with limited tools).

Diagnosis Steps

  1. Check Boot Logs:
    After rebooting, access logs from the last boot with:

    journalctl -b -1  # -b -1 = previous boot; omit for current boot

    Look for errors like “Failed to mount /mnt/external” (fstab issue) or “filesystem check failed”.

  2. Use Emergency/Rescue Mode:
    If the system won’t boot normally, boot into Emergency Mode (select from GRUB menu) or Rescue Mode (run systemctl rescue from a live CD).

Solutions

  • Fstab Errors: A misconfigured /etc/fstab (e.g., invalid UUID, missing mount point) is a common cause. In Emergency Mode, edit fstab with nano /etc/fstab and comment out the problematic line (add # at the start).

  • Filesystem Corruption: Run a filesystem check (fsck) on the root partition. In Emergency Mode:

    fsck /dev/sda1  # Replace /dev/sda1 with your root partition (check with `lsblk`)

  • Failed Units: Mask or disable the unit causing the boot failure. For example, if bad-service.service is failing:

    systemctl mask bad-service.service

5. Unit Dependency Issues (e.g., Service Starts Before Dependencies)

Symptoms

  • A service starts but fails because its dependency (e.g., a database) isn’t ready.
  • systemctl status <service> shows “Failed to start because dependency failed”.

Diagnosis Steps

  1. Check Unit Dependencies:
    List all dependencies for the service:

    systemctl list-dependencies <service> --reverse  # Shows what depends on this service  
systemctl list-dependencies <service>  # Shows what this service depends on

  2. Inspect the Unit File:
    Look for After=, Requires=, or Wants= directives. For example:

    [Unit]  
Description=My Service  
After=network.target mysql.service  # Should start after network and mysql  
Requires=mysql.service  # Fail if mysql isn't running

Solutions

  • Add Missing Dependencies: Edit the unit file to include After= and Requires= for critical dependencies. For example, ensure a web app starts after its database:

    After=mysql.service  
Requires=mysql.service

    Reload systemd: systemctl daemon-reload.

  • Use Wants= Instead of Requires=: If the dependency is optional (service can run without it), replace Requires= with Wants=. Wants= is a “soft” dependency and won’t fail the service if the dependency is missing.

  • Delay Service Start: For race conditions (e.g., network not ready), add a short delay with ExecStartPre=/bin/sleep 5 in the [Service] section.

6. Journald Logging Issues (e.g., Logs Not Persisted, High Disk Usage)

Symptoms

  • journalctl shows “No entries” for past reboots.
  • Disk fills up due to large journal logs (check with df -h).

Diagnosis Steps

  1. Check Journald Configuration:
    The main config file is /etc/systemd/journald.conf. Key settings:

    • Storage=: volatile (logs in RAM, lost on reboot), persistent (saved to /var/log/journal/), or auto (use persistent if /var/log/journal exists).

  2. Verify Journal Health:

    journalctl --verify  # Checks for corrupted log files

Solutions

  • Enable Persistent Logs: Edit /etc/systemd/journald.conf and set:

    Storage=persistent

    Create the journal directory and restart journald:

    sudo mkdir -p /var/log/journal  
sudo systemctl restart systemd-journald

  • Limit Journal Size: Prevent disk bloat by setting size limits in journald.conf:

    SystemMaxUse=500M  # Max total size for all logs  
MaxFileSize=100M   # Max size per log file

    Reload and vacuum old logs:

    systemctl restart systemd-journald  
journalctl --vacuum-size=500M

Symptoms

  • DNS resolution fails (ping google.com returns “unknown host”).
  • Network interfaces (e.g., eth0) don’t get an IP address on boot.

Diagnosis Steps

  1. Check systemd-resolved (DNS):

    systemctl status systemd-resolved  # Is the service active?  
resolvectl status  # Show DNS servers and domains

    If /etc/resolv.conf is not symlinked to systemd-resolved’s stub resolver, DNS may fail:

    ls -l /etc/resolv.conf  # Should point to ../run/systemd/resolve/stub-resolv.conf

  2. Check systemd-networkd (Network Config):
    For systems using systemd-networkd (instead of NetworkManager), check .network files in /etc/systemd/network/:

    cat /etc/systemd/network/20-wired.network  # Example config  
systemctl status systemd-networkd

Solutions

  • Fix systemd-resolved DNS:

    • Ensure the service is enabled: systemctl enable --now systemd-resolved.
    • Add custom DNS servers (e.g., Cloudflare) in /etc/systemd/resolved.conf: 
      [Resolve]  
DNS=1.1.1.1 1.0.0.1
    • Restart resolved: systemctl restart systemd-resolved.

  • Fix systemd-networkd IP Assignment:
    Ensure the .network file has correct settings (e.g., DHCP):

    [Match]  
Name=eth0  # Match interface name (check with `ip link`)  

[Network]  
DHCP=yes  # Use DHCP for IP

    Reload and restart:

    systemctl daemon-reload  
systemctl restart systemd-networkd

8. Timedatectl and Time Synchronization Issues

Symptoms

  • date shows incorrect time/timezone.
  • SSL errors (“certificate expired”) due to clock skew.
  • timedatectl shows “NTP service: inactive”.

Diagnosis Steps

  1. Check Current Time/Timezone:

    timedatectl  # Shows local time, UTC, timezone, and NTP status

  2. Verify systemd-timesyncd (NTP Client):

    systemctl status systemd-timesyncd  
journalctl -u systemd-timesyncd  # Check for NTP sync errors (e.g., "Connection refused")

Solutions

  • Set Timezone:

    timedatectl list-timezones | grep "America/New_York"  # Find your timezone  
sudo timedatectl set-timezone America/New_York

  • Enable NTP Sync:
    Ensure systemd-timesyncd is active:

    sudo systemctl enable --now systemd-timesyncd

    If sync fails, add reliable NTP servers in /etc/systemd/timesyncd.conf:

    [Time]  
NTP=pool.ntp.org time.nist.gov

    Restart the service: systemctl restart systemd-timesyncd.

9. Conclusion

Systemd is a powerful but complex init system, and troubleshooting its issues requires familiarity with tools like systemctl, journalctl, and unit files. By methodically diagnosing logs, verifying configurations, and leveraging systemd’s built-in debugging utilities, most common problems can be resolved quickly. Remember: logs are your best friend—journalctl and systemctl status should be your first stops.

10. References