Monitoring and Logging Systemd Services Like a Pro

Table of Contents

1. Understanding Systemd Services
2. Logging with Journald: The Basics
3. Monitoring Service Status and Health
4. Advanced Log Analysis with Journalctl
5. Integrating with External Monitoring Tools
6. Alerting on Service Issues
7. Best Practices for Systemd Monitoring and Logging
8. Conclusion
9. References

1. Understanding Systemd Services

Before diving into monitoring and logging, let’s recap what systemd services are. A systemd service is defined by a .service unit file, which describes how to start, stop, or manage a process. These files are stored in standard locations:

• /usr/lib/systemd/system/ (vendor-provided services)
• /etc/systemd/system/ (user-customized services)
• /run/systemd/system/ (runtime-generated services)

Key Components of a .service File

A typical .service file has three main sections:

• [Unit]: Metadata (description, dependencies, documentation).
• [Service]: Service behavior (executable path, restart policies, user/group, logging settings).
• [Install]: Installation targets (e.g., multi-user.target for boot-time activation).

Example Service File (/etc/systemd/system/myapp.service):

[Unit]
Description=My Custom Application
After=network.target mysql.service  # Start after network and MySQL

[Service]
User=appuser
Group=appgroup
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yaml
Restart=always  # Restart on failure
RestartSec=5    # Wait 5s before restarting
StandardOutput=journal  # Send stdout to journald
StandardError=journal   # Send stderr to journald

[Install]
WantedBy=multi-user.target  # Enable on boot for multi-user mode

Understanding this structure helps you configure services for better observability (e.g., StandardOutput=journal ensures logs flow to journald).

2. Logging with Journald: The Basics

Systemd includes journald, a centralized logging daemon that collects and stores logs from services, the kernel, and user processes. Unlike traditional text-based logs (e.g., /var/log/syslog), journald stores logs in a binary format for efficient querying, compression, and metadata enrichment.

Key Journald Concepts

• Persistent vs. Volatile Storage: By default, logs are stored in /run/log/journal/ (volatile, lost on reboot). To enable persistence, create /var/log/journal/ and set proper permissions:

  sudo mkdir -p /var/log/journal
  sudo systemd-tmpfiles --create --prefix /var/log/journal
  sudo systemctl restart systemd-journald

• Journalctl: The primary tool to query journald logs.

Essential journalctl Commands

Example: View Nginx Errors from the Last 30 Minutes

journalctl -u nginx.service --since "30 minutes ago" -p err

Configuring Journald

Customize journald behavior via /etc/systemd/journald.conf. Key settings:

• Storage=persistent: Enforce persistent logs (default: auto).
• SystemMaxUse=5G: Limit total journal size to 5GB.
• MaxRetentionSec=7days: Keep logs for 7 days.
• Compress=yes: Compress logs (default: yes).

After editing, restart journald:

sudo systemctl restart systemd-journald

3. Monitoring Service Status and Health

Beyond logs, monitoring a service’s runtime status (e.g., active, failed, inactive) is critical. Systemd provides systemctl, the service management tool, for this.

Basic Service Status Checks

Example: Check Nginx Status

systemctl status nginx.service

Sample output:

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-03-12 10:00:00 UTC; 2h 30min ago
       Docs: man:nginx(8)
   Main PID: 1234 (nginx)
      Tasks: 2 (limit: 4915)
     Memory: 3.5M
        CPU: 1.2s
     CGroup: /system.slice/nginx.service
             ├─1234 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             └─1235 nginx: worker process

Monitoring Resource Usage

To track CPU, memory, or disk usage of a service, combine systemctl with tools like ps, top, or systemd-cgtop (which shows resource usage by systemd control groups):

# Show resource usage of all services
systemd-cgtop --unit

# Check memory usage of nginx
ps -p $(systemctl show -p MainPID nginx.service --value) -o %mem,rss,cmd

4. Advanced Log Analysis with Journalctl

For pro-level troubleshooting, journalctl offers powerful filtering and output options to slice through large log volumes.

Advanced Filters

• Match specific messages: Use -g (grep) to match patterns in log messages:

  journalctl -u myapp.service -g "database connection failed"

• Exclude logs: Use --no-pager with grep -v to exclude noise:

  journalctl -u myapp.service --no-pager | grep -v "debug: "

• Filter by user/UID:

  journalctl _UID=1000  # Logs from user with UID 1000

• Output formats: Use -o to export logs in JSON, CSV, or verbose format (useful for automation):

  # Export nginx logs to JSON for analysis
  journalctl -u nginx.service --since "1 day ago" -o json > nginx-logs.json

Following and Aggregating Logs

• Follow multiple services: Monitor logs from nginx and mysql simultaneously:

  journalctl -u nginx.service -u mysql.service -f

• Aggregate by boot: Each system boot has a unique ID; list boots with journalctl --list-boots, then filter by boot ID:

  journalctl --boot=1  # Logs from the previous boot (0 = current)

5. Integrating with External Monitoring Tools

For large-scale or distributed systems, systemd’s built-in tools may not suffice. Integrate with external tools for centralized monitoring and log aggregation.

Systemd Exporter for Prometheus

Prometheus + Grafana is a popular stack for metrics-based monitoring. Use systemd-exporter to expose systemd service metrics (e.g., systemd_service_state, systemd_service_restarts_total).

Setup Steps:

1. Install systemd-exporter (from GitHub):

   wget https://github.com/prometheus-community/systemd-exporter/releases/download/v0.5.0/systemd-exporter-0.5.0.linux-amd64.tar.gz
   tar -xzf systemd-exporter-0.5.0.linux-amd64.tar.gz
   sudo cp systemd-exporter-0.5.0.linux-amd64/systemd-exporter /usr/local/bin/

2. Create a systemd service for the exporter:

   [Unit]
   Description=Systemd Exporter for Prometheus
   After=network.target
   
   [Service]
   User=prometheus
   ExecStart=/usr/local/bin/systemd-exporter --collector.services --collector.timers
   
   [Install]
   WantedBy=multi-user.target

3. Start the exporter and configure Prometheus to scrape it (add to prometheus.yml):

   scrape_configs:
     - job_name: 'systemd'
       static_configs:
         - targets: ['localhost:9558']  # Default systemd-exporter port

4. Build Grafana dashboards using metrics like systemd_service_state{state="active"} to track service health.

Centralized Logging with ELK Stack

For log aggregation, use the ELK Stack (Elasticsearch, Logstash, Kibana) with journalbeat (a lightweight shipper for journald logs).

Setup Steps:

1. Install Journalbeat (from Elastic):

   sudo apt install journalbeat  # Debian/Ubuntu

2. Configure journalbeat.yml to ship logs to Elasticsearch or Logstash:

   output.elasticsearch:
     hosts: ["localhost:9200"]
   journalbeat.inputs:
     - paths: []  # Defaults to journald logs
       include_matches:
         - "_SYSTEMD_UNIT=nginx.service"  # Ship only nginx logs

3. Start Journalbeat and visualize logs in Kibana (create dashboards for error rates, service uptime, etc.).

6. Alerting on Service Issues

Proactive monitoring requires alerts when services fail or degrade. Here are two approaches:

Systemd OnFailure Triggers

Use the OnFailure= directive in .service files to run a script when a service fails. For example, send an email alert:

1. Create an alert script (/usr/local/bin/alert.sh):

   #!/bin/bash
   echo "Service $1 failed at $(date)" | mail -s "ALERT: $1 Failed" [email protected]

2. Make it executable:

   sudo chmod +x /usr/local/bin/alert.sh

3. Update your service file:

   [Service]
   OnFailure=alert@%n.service  # %n = service name (e.g., myapp.service)

4. Create a template service for alerts (/etc/systemd/system/[email protected]):

   [Unit]
   Description=Alert on failure of %i
   
   [Service]
   Type=oneshot
   ExecStart=/usr/local/bin/alert.sh %i

Prometheus Alertmanager

For metric-based alerts (e.g., “Nginx restarted 5 times in 10 minutes”), use Prometheus Alertmanager with rules based on systemd-exporter metrics.

Example Alert Rule (prometheus/rules/systemd.rules.yml):

groups:
- name: systemd_alerts
  rules:
  - alert: ServiceRestartLoop
    expr: increase(systemd_service_restarts_total{state="active"}[10m]) > 5
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Service {{ $labels.name }} is restarting too often"
      description: "{{ $labels.name }} restarted {{ $value }} times in 10 minutes."

7. Best Practices for Systemd Monitoring and Logging

To maintain a robust setup:

1. Enable Persistent Journals: Avoid data loss by storing logs in /var/log/journal/.
2. Limit Journal Size: Use SystemMaxUse in journald.conf to prevent disk exhaustion.
3. Secure Logs: Restrict journal access with sudo chmod 600 /var/log/journal/* (only root can read).
4. Document Services: Add Documentation= in .service files to link runbooks for troubleshooting.
5. Use Drop-In Files: Customize vendor services with drop-in files (/etc/systemd/system/<service>.service.d/override.conf) instead of editing original files.
6. Audit Regularly: Run systemctl list-units --failed weekly to check for silent failures.

8. Conclusion

Monitoring and logging systemd services is not just about reacting to failures—it’s about proactively ensuring reliability. By mastering journalctl for logs, systemctl for status checks, and integrating with tools like Prometheus or ELK, you can gain deep visibility into your services. Combine this with alerts and best practices like persistent logging and documentation, and you’ll manage systemd services like a pro.

9. References

• Systemd Documentation
• Journalctl Man Page
• Systemd Exporter
• Journalbeat Guide
• Prometheus Alerting