How to Use SystemTap for Linux Performance Analysis

What is SystemTap?
Installation and Setup
Core Concepts
Basic SystemTap Workflow
Practical Use Cases
Advanced SystemTap Techniques
Troubleshooting Common Issues
Best Practices
Conclusion
References

What is SystemTap?

SystemTap is an open-source tool for dynamic tracing on Linux systems. It enables users to write small scripts (called “SystemTap scripts”) that hook into kernel or user-space functions, collect data, and generate reports—all without rebooting the system or modifying kernel code.

Key Features:

Dynamic Instrumentation: Probes are inserted and removed at runtime, no kernel recompilation needed.
Flexibility: Scripts can target specific kernel functions, system calls, user-space applications, or hardware events.
Rich Data Collection: Captures metrics like function execution time, process IDs, file descriptors, network packets, and more.
Low Overhead: Designed to minimize performance impact when used correctly (though excessive probing can cause overhead).

SystemTap works by translating scripts into C code, compiling it into a kernel module, loading the module, and then executing the probes. The results are printed to the console or saved to a file for analysis.

Installation and Setup

Before using SystemTap, you’ll need to install it along with dependencies like kernel headers and debug symbols. The exact steps vary by Linux distribution.

Prerequisites

Kernel Headers: Required to compile SystemTap modules. Match your running kernel version (e.g., linux-headers-$(uname -r)).
Debug Symbols: Optional but highly recommended for detailed tracing (e.g., kernel debuginfo packages).
SystemTap Package: The core tool itself.

Installing SystemTap on Major Distributions

Ubuntu/Debian:

# Install SystemTap and kernel headers
sudo apt update
sudo apt install systemtap linux-headers-$(uname -r)

# Install kernel debug symbols (optional but useful for deep tracing)
sudo apt install linux-image-$(uname -r)-dbgsym  # May require enabling the debug repo

Fedora/RHEL/CentOS:

# Install SystemTap and kernel headers
sudo dnf install systemtap kernel-devel-$(uname -r)  # Fedora
# OR
sudo yum install systemtap kernel-devel-$(uname -r)  # RHEL/CentOS

# Install kernel debug symbols (Fedora)
sudo dnf debuginfo-install kernel-$(uname -r)

# For RHEL/CentOS, enable the debuginfo repo first (e.g., via subscription-manager)

Arch Linux:

sudo pacman -S systemtap linux-headers
# Kernel debug symbols: Install `linux-debug` from the AUR

Verifying the Installation

Run a simple “hello world” script to confirm SystemTap is working:

Create a file hello.stp with:

probe begin {
  println("Hello, SystemTap!")
  exit()  # Exit immediately after running
}

Execute it with:
```
sudo stap hello.stp
```

If successful, you’ll see:

Hello, SystemTap!

Core Concepts

To write effective SystemTap scripts, you need to understand its core components:

Probes: The Building Blocks

A probe is a point in the system where SystemTap collects data. Probes are defined with the probe keyword followed by a probe point (the location to instrument).

Common Probe Points:

begin: Runs when the script starts.
end: Runs when the script exits (e.g., via exit() or Ctrl+C).
syscall.*: Taps into system calls (e.g., syscall.open, syscall.write).
kernel.function("function_name"): Hooks into a specific kernel function (e.g., kernel.function("vfs_read")).
timer.ms(interval): Runs periodically (e.g., timer.ms(1000) for every 1 second).
process("/path/to/app").function("function_name"): Traces user-space application functions.

Tapsets: Reusable Libraries

Tapsets are pre-written SystemTap libraries that simplify common tasks. They provide helper functions and probe aliases to avoid rewriting boilerplate code. For example:

syscall tapset: Wraps system call probes (e.g., syscall.open instead of kernel.function("sys_open")).
process tapset: Simplifies user-space tracing (e.g., process.exec to track process starts).
timer tapset: Provides timing probes (e.g., timer.s(5) for 5-second intervals).

List all available tapsets with:

stap -l 'tapset::*'

Variables and Functions

SystemTap scripts use:

Global Variables: Persist across probes (e.g., global count).
Local Variables: Scoped to a single probe (e.g., pid = pid()).
Aggregation Functions: For summarizing data (e.g., @count, @sum, @avg, @hist_log for histograms).
Helper Functions: Built-in or from tapsets (e.g., pid(), execname(), uid()).

Basic SystemTap Workflow

Write a Script: Define probes, collect data, and output results.
Validate the Script: Check for syntax errors with stap -p4 script.stp (preprocess, parse, and translate to C without running).
Run the Script: Execute with sudo stap script.stp.
Analyze Output: Interpret the collected data to diagnose issues.

Practical Use Cases

Let’s explore real-world scenarios where SystemTap shines.

1. CPU Usage Profiling

Identify which kernel or user-space functions are consuming the most CPU.

Script: `cpu_profiler.stp`

# Profile kernel functions and user-space apps every 2 seconds
global func_counts

probe kernel.function("*").call, process("*").function("*").call {
  func_counts[probefunc(), execname()]++
}

probe timer.s(2) {
  println("Top CPU Functions (Last 2 Seconds):")
  foreach ([func, app] in func_counts- limit 10) {
    printf("  %-40s %-20s %d\n", func, app, func_counts[func, app])
  }
  delete func_counts  # Reset for next interval
}

probe end {
  println("Profiling stopped.")
}

How to Run:

sudo stap cpu_profiler.stp

Output Explanation:
Shows the top 10 functions (kernel or user-space) by call count over 2-second intervals. Useful for identifying CPU hogs.

2. Disk I/O Tracing

Track which processes are reading/writing to disk and how much data they transfer.

Script: `disk_io_tracing.stp`

# Trace read/write syscalls and log process, size, and file
global io_stats

probe syscall.write, syscall.pwrite64 {
  io_stats[execname(), pid(), "write"] += count  # 'count' is bytes written
}

probe syscall.read, syscall.pread64 {
  io_stats[execname(), pid(), "read"] += count  # 'count' is bytes read
}

probe timer.s(5) {
  println("Disk I/O Summary (Last 5 Seconds):")
  foreach ([app, pid, op] in io_stats) {
    printf("  App: %-15s PID: %-6d Op: %-5s Bytes: %d\n", app, pid, op, io_stats[app, pid, op])
  }
  delete io_stats
}

Output:
Shows per-process read/write bytes over 5-second windows, helping identify I/O-heavy applications.

3. Network Activity Monitoring

Trace TCP/UDP traffic to see which processes are sending/receiving data.

Script: `network_monitor.stp`

# Track TCP send/receive bytes per process
global tcp_bytes

probe kernel.function("tcp_sendmsg").call {
  bytes = $skb->len  # Length of the socket buffer
  tcp_bytes[execname(), pid(), "send"] += bytes
}

probe kernel.function("tcp_recvmsg").call {
  bytes = $skb->len
  tcp_bytes[execname(), pid(), "recv"] += bytes
}

probe timer.s(5) {
  println("TCP Traffic Summary (Last 5 Seconds):")
  foreach ([app, pid, dir] in tcp_bytes) {
    printf("  App: %-15s PID: %-6d Direction: %-5s Bytes: %d\n", app, pid, dir, tcp_bytes[app, pid, dir])
  }
  delete tcp_bytes
}

Note: Requires kernel debug symbols for tcp_sendmsg/tcp_recvmsg access.

4. User-Space Application Tracing

Debug a specific application (e.g., nginx) by tracing its functions.

Script: `app_tracing.stp`

# Trace nginx's request handling functions
probe process("/usr/sbin/nginx").function("ngx_http_handle_request").call {
  printf("Nginx PID %d handling request from %s\n", pid(), ipaddr($c->client->sockaddr))
}

probe process("/usr/sbin/nginx").function("ngx_http_finalize_request").return {
  duration = gettimeofday_us() - @entry(gettimeofday_us())  # Time taken to process request
  printf("Nginx PID %d finished request in %d us\n", pid(), duration)
}

Requirement: Install nginx debug symbols (e.g., nginx-dbg on Ubuntu) for full function names.

5. Memory Leak Detection

Track kernel memory allocations/frees to identify leaks.

Script: `memory_leak.stp`

# Track kmalloc (kernel malloc) and kfree calls
global allocations, frees

probe kernel.function("kmalloc").call {
  ptr = $ptr  # Allocated memory pointer
  size = $size  # Allocation size
  allocations[ptr] = size
}

probe kernel.function("kfree").call {
  ptr = $ptr  # Freed memory pointer
  if (ptr in allocations) {
    frees[ptr] = allocations[ptr]
    delete allocations[ptr]
  }
}

probe end {
  println("Unfreed Kernel Allocations (Potential Leaks):")
  foreach (ptr in allocations) {
    printf("  Pointer: %p, Size: %d bytes\n", ptr, allocations[ptr])
  }
}

Run: Let the script run for a while, then stop with Ctrl+C to see unfreed allocations.

Advanced SystemTap Techniques

Writing Custom Tapsets

For repeated tasks, create reusable tapsets. For example, a my_tapset.stp tapset:

// my_tapset.stp
@__private__ function log_io(app: string, pid: long, bytes: long) {
  printf("[%s] PID %d transferred %d bytes\n", app, pid, bytes)
}

Use it in scripts with:

%{
#include "my_tapset.stp"
%}

probe syscall.write {
  log_io(execname(), pid(), count)
}

Optimizing Script Performance

Limit Probes: Avoid kernel.function("*")—target specific functions.
Batch Data: Use aggregation functions (e.g., @count) instead of printing in every probe.
Short-Circuit Probes: Skip unnecessary work with if (condition) next;.
Use stap -v: Check for slow probes (e.g., kernel.function("*") is expensive).

Aggregating and Visualizing Data

SystemTap can output to files (e.g., printf("%d %s\n", count, app) > "output.txt"), which you can parse with tools like gnuplot or Python for graphs. For example:

sudo stap cpu_profiler.stp > cpu_data.txt
gnuplot -e "plot 'cpu_data.txt' using 3:xtic(1) with bars"  # Bar chart of counts

Troubleshooting Common Issues

“Missing Kernel Headers”: Install linux-headers-$(uname -r).
“Debug Symbols Not Found”: Install kernel/user-space debuginfo packages.
“Permission Denied”: Run with sudo.
“Script Runs Slow”: Reduce the number of probes or use aggregation.
“Module Compilation Failed”: Check for syntax errors with stap -p4 script.stp.

Best Practices

Start Small: Begin with simple scripts and iterate.
Use Tapsets: Leverage existing tapsets to avoid reinventing the wheel.
Test in Staging: SystemTap can have overhead—test scripts in non-production first.
Limit Overhead: Avoid excessive probing (e.g., kernel.function("*")).
Document Scripts: Add comments to explain probes and logic.

Conclusion

SystemTap is a versatile tool for Linux performance analysis, enabling deep insights into kernel and user-space behavior. By mastering its probes, tapsets, and scripting capabilities, you can diagnose complex performance issues, debug applications, and optimize system behavior. Start with simple scripts, experiment with the use cases above, and gradually explore advanced techniques to unlock SystemTap’s full potential.

Table of Contents