funwithlinux guide

Advanced Systemd: Custom Unit Files and Beyond

Systemd has become the de facto init system for most Linux distributions, replacing traditional SysVinit and Upstart. Beyond its role as a process manager, systemd orchestrates system services, handles device management, manages network sockets, and even schedules tasks. While many users interact with systemd through basic commands like `systemctl start` or `systemctl enable`, its true power lies in **custom unit files**—flexible configurations that let you manage applications, scripts, or services not covered by default packages. In this blog, we’ll dive deep into systemd’s advanced capabilities, starting with crafting custom unit files, exploring dependencies, templates, and targets, and even venturing beyond unit files into timers and environment management. Whether you’re a system administrator automating workflows or a developer deploying a custom application, mastering these concepts will give you granular control over your system.

Table of Contents

  1. Understanding Systemd Units
  2. Anatomy of a Custom Unit File
  3. Advanced Unit File Features
  4. Beyond Unit Files: Targets, Timers, and Environment
  5. Debugging and Best Practices
  6. Conclusion
  7. References

1. Understanding Systemd Units

At its core, systemd manages units—resources that represent system components. Units are defined by plain-text configuration files (.service, .timer, .target, etc.) and control how services start, stop, and interact with the system.

Types of Units

Systemd supports dozens of unit types, but the most common are:

  • .service: Manages processes (e.g., nginx.service, ssh.service).
  • .timer: Schedules tasks (replaces cron for systemd-aware workflows).
  • .target: Groups units (similar to SysVinit runlevels, e.g., multi-user.target).
  • .socket: Enables socket activation (starts services only when a connection is received).
  • .mount/.automount: Controls filesystem mounting.

Where Units Live

Systemd loads units from three primary directories (in order of priority, highest first):

  1. /etc/systemd/system/: Local administrator-defined units (custom units go here).
  2. /run/systemd/system/: Runtime-generated units (temporary, e.g., from systemctl edit).
  3. /usr/lib/systemd/system/: Distribution-provided units (default services like sshd).

2. Anatomy of a Custom Unit File

A custom unit file (e.g., myapp.service) is a plain-text file with sections (enclosed in [ ]) and directives (key-value pairs). Let’s break down the structure with a practical example: a service to run a custom Python script.

Example: A Simple Custom Service

Suppose we have a Python script at /opt/myapp/app.py that runs a web server. We’ll create a myapp.service unit to manage it. Here’s the full file:

[Unit]
Description=My Custom Python Web App
Documentation=https://example.com/myapp/docs
After=network.target  # Start only after the network is up
Requires=network.target  # Fail if network.target isn't available

[Service]
Type=simple  # Process runs in foreground (no forking)
User=ubuntu  # Run as non-root user "ubuntu"
Group=ubuntu
WorkingDirectory=/opt/myapp  # Set working directory
ExecStart=/usr/bin/python3 /opt/myapp/app.py  # Command to start the service
ExecStop=/bin/kill -TERM $MAINPID  # Graceful stop (systemd sets $MAINPID)
Restart=on-failure  # Restart if the service exits with non-zero status
RestartSec=5s  # Wait 5 seconds before restarting
Environment="LOG_LEVEL=info"  # Set environment variable

[Install]
WantedBy=multi-user.target  # Start when the system reaches multi-user mode
Alias=myapp  # Allow starting with `systemctl start myapp` (instead of myapp.service)

Key Sections Explained

[Unit]: Metadata and Dependencies

This section defines the unit’s purpose, documentation, and relationships with other units.

  • Description: Human-readable name (appears in systemctl status).
  • Documentation: URLs or man pages for further info.
  • After=X: Start after unit X has started (ordering, not a hard dependency).
  • Requires=X: Declare a hard dependency—if X fails, this unit fails too.
  • Wants=X: Soft dependency—best-effort start of X, but this unit proceeds even if X fails.
  • Conflicts=X: Stop this unit if X starts, and vice versa.

[Service]: Service Behavior

This section configures how the service runs, stops, and restarts.

  • Type: Defines how systemd manages the process:

    • simple (default): Process runs in the foreground; systemd tracks it directly.
    • forking: Process forks a child and exits (e.g., traditional daemons like httpd).
    • oneshot: Process runs once and exits (e.g., a setup script).
    • dbus: Process acquires a D-Bus name; systemd waits for this.
    • notify: Process sends a signal to systemd when ready (uses sd_notify()).

  • ExecStart: The command to start the service (absolute paths required).

  • ExecStop: Command to stop the service (optional; systemd sends SIGTERM by default).

  • ExecReload: Command to reload configuration (e.g., nginx -s reload).

  • User/Group: Run the service as a specific user/group (critical for security—avoid root!).

  • WorkingDirectory: Set the process’s working directory.

  • Restart: When to restart the service:

    • no (default): Never restart.
    • on-failure: Restart if exit code is non-zero, signal, or timeout.
    • always: Restart even if the service exits successfully.
    • on-abnormal: Restart only on crashes or timeouts (not clean exits).

  • Environment/EnvironmentFile: Set environment variables (use EnvironmentFile=/path/to/env for files).

[Install]: Activation on Boot

This section defines how the unit is “installed” (i.e., enabled to start on boot).

  • WantedBy=X: When enabled, create a symlink in X.wants/ (e.g., multi-user.target.wants/).
  • RequiredBy=X: Harder dependency—symlink in X.requires/; X fails if this unit is missing.
  • Alias: Short name for the unit (e.g., Alias=myapp lets you use systemctl start myapp).

3. Advanced Unit File Features

Dependencies: Fine-Grained Control

Systemd’s dependency system is powerful but nuanced. Beyond Requires and Wants, use these directives for precision:

  • Before=X: Start before unit X (reverse of After=X).
  • BindsTo=X: If X stops, this unit stops too (stronger than Requires).
  • PartOf=X: If X is stopped/restarted, this unit is too (but not the reverse).

Example: A database-dependent app:

[Unit]
Description=App That Depends on PostgreSQL
After=postgresql.service
BindsTo=postgresql.service  # App stops if PostgreSQL stops

Template Units: Reusable Configurations

For managing multiple instances of the same service (e.g., multiple Node.js apps), use template units with @ in the filename (e.g., [email protected]). Templates use %i to reference the instance name.

Example: [email protected] (template):

[Unit]
Description=My App Instance %i
After=network.target

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/opt/myapp/%i  # Instance-specific directory
ExecStart=/usr/bin/node /opt/myapp/%i/server.js
Restart=on-failure

[Install]
WantedBy=multi-user.target

To start an instance named api:

systemctl start [email protected]

Systemd replaces %i with api, so the service runs from /opt/myapp/api/.

Instantiation and Special Variables

Template units support special variables to make configurations dynamic:

  • %i: Instance name (from @instance in the unit name).
  • %N: Full unit name (e.g., [email protected]).
  • %p: Unit prefix (e.g., myapp from [email protected]).
  • %m: Machine ID (from /etc/machine-id).

Example: Use %h (home directory of the user) in a user unit:

[Service]
WorkingDirectory=%h/myapp  # Equivalent to /home/ubuntu/myapp for user "ubuntu"

Socket Activation (Advanced)

Socket activation starts a service only when a connection is received (reduces idle resource usage). It requires a .socket unit and a .service unit.

Example: myapp.socket (listens on port 8080):

[Unit]
Description=Socket for My App

[Socket]
ListenStream=8080  # TCP port 8080
Accept=false  # Single connection; use "true" for multiple

[Install]
WantedBy=sockets.target

myapp.service (starts on connection):

[Unit]
Description=My App (Socket-Activated)

[Service]
Type=simple
ExecStart=/opt/myapp/server.py
StandardInput=socket  # Read from the socket

Enable with systemctl enable --now myapp.socket. The service starts only when port 8080 is accessed!

4. Beyond Unit Files: Targets, Timers, and Environment

Targets: System States (Runlevels 2.0)

Targets group units to define system states (e.g., “multi-user” or “graphical”). They replace traditional runlevels but are more flexible.

Common targets:

  • multi-user.target: Text-based multi-user mode (default for servers).
  • graphical.target: Multi-user mode with GUI (depends on multi-user.target).
  • rescue.target: Single-user mode for recovery.
  • poweroff.target: Shutdown the system.

To set the default target (e.g., boot to text mode):

sudo systemctl set-default multi-user.target

To create a custom target (e.g., myapp.target for grouping app units):

[Unit]
Description=My App Target
Requires=network.target mydb.service
After=network.target mydb.service
AllowIsolate=yes  # Let users switch to this target with `systemctl isolate`

Timers: Scheduling Tasks (Cron Alternative)

Systemd timers (*.timer) schedule units to run at specific times, offering more features than cron (e.g., calendar events, monotonic time, and dependency management).

Example: A weekly backup timer (backup.timer):

[Unit]
Description=Weekly Backup Timer

[Timer]
OnCalendar=weekly  # Run every Sunday at 00:00 (default)
Persistent=true  # Run missed jobs when system boots (if offline during schedule)
AccuracySec=1min  # Allow 1-minute delay to avoid CPU spikes

[Install]
WantedBy=timers.target  # Start timer when system boots

Pair with a backup.service (the actual task):

[Unit]
Description=Weekly Backup Service

[Service]
Type=oneshot  # Run once and exit
ExecStart=/opt/backup/run.sh

Enable and start the timer:

sudo systemctl enable --now backup.timer

Check pending timers with:

systemctl list-timers --all

Environment Management

Systemd lets you set environment variables globally or per-unit:

  • Per-unit: Use EnvironmentFile=/path/to/env in [Service] (file format: KEY=VALUE).
  • Global: Edit /etc/environment (system-wide) or ~/.config/environment.d/*.conf (user-specific).
  • Runtime variables: Use systemctl set-environment KEY=VALUE (persists until reboot).

5. Debugging and Best Practices

Debugging Units

Systemd provides powerful tools to troubleshoot misbehaving units:

  • Reload systemd: After editing a unit file, run sudo systemctl daemon-reload.
  • Check status: systemctl status myapp.service (shows logs, PID, and recent activity).
  • View logs: journalctl -u myapp.service (add -f to follow live logs, -n 50 for last 50 lines).
  • Verify syntax: systemd-analyze verify myapp.service (catches typos or invalid directives).
  • Check dependencies: systemctl list-dependencies myapp.service (shows all required units).

Best Practices

  • Use absolute paths: Always specify full paths in ExecStart, WorkingDirectory, etc.
  • Run as non-root: Use User/Group to avoid unnecessary privileges.
  • Limit dependencies: Only Require/Want critical units to speed up boot.
  • Test with oneshot: For scripts, use Type=oneshot and test with systemctl start.
  • Document units: Add Description and Documentation directives for clarity.
  • Avoid Restart=always: Use on-failure unless you need the service to restart even on success.

6. Conclusion

Systemd’s custom unit files are the cornerstone of modern Linux system management, offering unparalleled control over services, scheduling, and system states. By mastering unit file structure, dependencies, templates, and timers, you can automate workflows, secure services, and optimize system behavior.

While systemd’s complexity can feel overwhelming, the payoff is a more robust, flexible system. Start with simple services, experiment with timers, and gradually explore advanced features like socket activation. Your future self (and your servers) will thank you.

7. References