funwithlinux guide

How to Use Perf and Other Profiling Tools for Linux

In the world of Linux development and system administration, performance is often the difference between a seamless user experience and a frustrating one. Whether you’re optimizing a critical application, debugging a slow server, or simply curious about where your system’s resources are being spent, **profiling tools** are indispensable. These tools help identify bottlenecks—such as CPU hogs, memory leaks, inefficient system calls, or cache misses—by collecting and analyzing runtime data. Among the most powerful profiling tools for Linux is `perf`, a built-in utility that traces CPU usage, memory, and more. But `perf` is just the tip of the iceberg. This blog will guide you through `perf` in depth, along with other essential tools like Valgrind, gprof, strace, and BPF-based tools (e.g., bpftrace). By the end, you’ll have the knowledge to diagnose performance issues like a pro.

Table of Contents

  1. Understanding Profiling: Types and Use Cases
  2. Perf: The Swiss Army Knife of Linux Profiling
  3. Valgrind: Memory and Cache Debugging
  4. gprof: GNU Profiler for CPU Bound Workloads
  5. strace: Tracing System Calls
  6. BPF Tools (bpftrace): Advanced System Tracing
  7. Best Practices for Effective Profiling
  8. Conclusion
  9. References

1. Understanding Profiling: Types and Use Cases

Profiling tools collect data about an application or system during execution to identify inefficiencies. Common types of profiling include:

  • CPU Profiling: Identifies functions or code paths consuming the most CPU time.
  • Memory Profiling: Detects leaks, excessive allocations, or inefficient memory usage.
  • I/O Profiling: Analyzes disk/network bottlenecks (e.g., slow file reads, blocked syscalls).
  • Cache Profiling: Uncovers poor cache utilization (e.g., cache misses slowing down CPU).
  • System Call Tracing: Tracks interactions with the kernel (e.g., open(), read(), fork()).

Choose tools based on your hypothesis: Use perf for CPU/memory, Valgrind for memory leaks, strace for syscalls, and BPF tools for system-wide issues.

2. Perf: The Swiss Army Knife of Linux Profiling

perf (Performance Event Collector) is a Linux-specific tool built into the kernel. It leverages hardware performance counters (e.g., CPU cycles, cache misses) and software events (e.g., page faults, context switches) to profile applications and the kernel.

2.1 Installing Perf

perf is included with the Linux kernel source but may require separate installation:

  • Ubuntu/Debian:

    sudo apt install linux-tools-common linux-tools-$(uname -r)

    (The $(uname -r) ensures compatibility with your kernel version.)

  • Fedora/RHEL:

    sudo dnf install perf

  • Arch Linux:

    sudo pacman -S perf

Verify installation with perf --version.

2.2 Basic Perf Commands

perf stat: Measure Event Counts

perf stat runs a command and reports hardware/software event statistics (e.g., CPU cycles, instructions, cache misses).

Example: Profile a simple C program (myapp):

perf stat ./myapp

Output includes:

  • cycles: Total CPU cycles used.
  • instructions: Number of instructions executed.
  • cache-misses: L1/L2/L3 cache misses (critical for CPU-bound apps).
  • seconds time elapsed: Wall-clock time.

Filter events with -e:

perf stat -e cycles,instructions,cache-misses ./myapp

perf top: Real-Time CPU Usage

perf top shows a live, interactive view of functions consuming the most CPU (like top but for functions).

perf top

Key columns:

  • Overhead: Percentage of CPU time spent in the function.
  • Command: Process name.
  • Shared Object: Binary/library (e.g., ./myapp, libc.so).
  • Symbol: Function name (e.g., main, malloc).

perf record + perf report: Detailed Call Analysis

perf record captures profiling data to a file (perf.data), and perf report visualizes it.

Workflow:

  1. Record data:

    perf record -g ./myapp  # -g enables call graphs (stack traces)

    The -g flag is critical for understanding why a function is called (e.g., which parent function triggers it).

  2. Analyze with perf report:

    perf report

    Navigate using arrow keys. Focus on high-overhead functions and their call graphs to identify bottlenecks.

2.3 Advanced Perf Techniques

Flame Graphs: Visualizing Call Stacks

Flame graphs (invented by Brendan Gregg) are a powerful way to visualize call stacks. They show which functions are active over time, with taller bars indicating more CPU time.

Steps to generate a flame graph:

  1. Install the FlameGraph tools:

    git clone https://github.com/brendangregg/FlameGraph.git  
cd FlameGraph

  2. Record data with perf script:

    perf record -g -F 99 ./myapp  # -F 99 = 99 samples/sec (avoids overhead)  
perf script > out.perf

  3. Generate the flame graph:

    ./stackcollapse-perf.pl out.perf > out.folded  
./flamegraph.pl out.folded > myapp_flamegraph.svg

Open myapp_flamegraph.svg in a browser. Wider/taller bars indicate hot paths (e.g., a process_data() function taking 40% of CPU).

Kernel Profiling

perf can profile kernel code (e.g., drivers, system calls) with --kernel-callchains:

perf record -g --kernel-callchains -p <PID>  # Profile PID with kernel stacks

2.4 Real-World Example: Profiling a C Application

Let’s profile a simple program (cpu_hog.c) that wastes CPU:

#include <stdio.h>  

void waste_time() {  
    int x = 0;  
    for (long i = 0; i < 1e9; i++) {  
        x += i;  // Pointless computation  
    }  
}  

int main() {  
    printf("Starting...\n");  
    waste_time();  
    printf("Done.\n");  
    return 0;  
}

  1. Compile: gcc -O0 -o cpu_hog cpu_hog.c ( -O0 disables optimizations for clarity).

  2. Run perf stat:

    perf stat ./cpu_hog

    Output shows high cycles and instructions (expected for a CPU hog).

  3. Generate a flame graph:

    perf record -g -F 99 ./cpu_hog  
perf script | ~/FlameGraph/stackcollapse-perf.pl > out.folded  
~/FlameGraph/flamegraph.pl out.folded > cpu_hog_flame.svg

    The SVG will show waste_time() as the tallest bar, confirming it’s the bottleneck.

3. Valgrind: Memory and Cache Debugging

Valgrind is a framework for debugging and profiling. Its most popular tools are Memcheck (memory leaks/errors) and Cachegrind (cache usage).

3.1 Installing Valgrind

# Ubuntu/Debian  
sudo apt install valgrind  

# Fedora/RHEL  
sudo dnf install valgrind  

# Arch  
sudo pacman -S valgrind

3.2 Memcheck: Detecting Memory Leaks

Memcheck identifies:

  • Use of uninitialized memory.
  • Invalid memory access (e.g., out-of-bounds array access).
  • Memory leaks (unfreed allocations).

Example: Profile a leaky program (leaky.c):

#include <stdlib.h>  

void leak() {  
    int* data = malloc(1024);  // Never freed!  
}  

int main() {  
    leak();  
    return 0;  
}

Compile: gcc -o leaky leaky.c

Run Memcheck:

valgrind --leak-check=full ./leaky

Output highlights:

  • definitely lost: 1,024 bytes in 1 block: Confirmed leak in leak().
  • Stack trace pointing to malloc in leak().

3.3 Cachegrind: Analyzing Cache Usage

Cachegrind simulates CPU caches to show cache misses, which often slow down CPU-bound apps.

Example: Profile cpu_hog (from Section 2.4):

valgrind --tool=cachegrind ./cpu_hog

Output includes:

  • I refs: Instruction references.
  • D1 misses: L1 data cache misses (critical for performance).
  • LLd misses: Last-level (L3) data cache misses.

Use cg_annotate to drill into source code:

cg_annotate cachegrind.out.<PID>

4. gprof: GNU Profiler for CPU Bound Workloads

gprof is a classic tool for profiling CPU usage in C/C++ programs. It requires recompiling the app with profiling flags.

4.1 Compiling for gprof

Add -pg to compile and link flags:

gcc -pg -o myapp myapp.c

4.2 Generating and Interpreting Reports

  1. Run the app to generate gmon.out:

    ./myapp

  2. Generate report:

    gprof ./myapp gmon.out > report.txt

Report sections:

  • Flat profile: Functions sorted by CPU time (e.g., waste_time 99.9% of time).
  • Call graph: Parent/child function relationships (e.g., main calls waste_time).

5. strace: Tracing System Calls

strace traces system calls (e.g., open(), read(), write()) and signals, making it ideal for debugging I/O issues or permission errors.

5.1 Basic strace Usage

Trace a program:

strace ./myapp

Output shows every syscall, arguments, and return values (e.g., open("file.txt", O_RDONLY) = -1 ENOENT (No such file or directory)).

5.2 Filtering and Advanced Features

Filter syscalls with -e:

strace -e open,read ./myapp  # Only show open/read calls

Trace a running process with -p:

strace -p 1234  # Trace PID 1234

Count syscalls with -c:

strace -c ./myapp

Output summarizes syscall frequency (e.g., read called 100 times, write 5 times).

6. BPF Tools (bpftrace): Advanced System Tracing

BPF (Berkeley Packet Filter) is a kernel-level virtual machine for safe, efficient tracing. bpftrace is a high-level language for BPF, enabling system-wide profiling (e.g., tracing all processes, kernel functions).

6.1 Introduction to BPF

BPF is used for:

  • Network monitoring (e.g., tcpdump uses BPF).
  • System profiling (e.g., tracing execve to log new processes).
  • Security (e.g., detecting suspicious syscalls).

6.2 Simple bpftrace Examples

Install bpftrace (Ubuntu/Debian):

sudo apt install bpftrace

Trace execve (process launches):

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s executed %s\n", comm, str(args->filename)); }'

Count CPU scheduling events:

sudo bpftrace -e 'tracepoint:sched:sched_switch { @count[prev_comm] = count(); }'

7. Best Practices for Effective Profiling

  • Profile in production-like environments: Test on hardware/load similar to production.
  • Minimize overhead: Use sampling (perf record -F 99) instead of full tracing for long runs.
  • Start broad, then drill down: Use perf top or strace -c to find hotspots, then perf report or Memcheck for details.
  • Compare before/after fixes: Validate optimizations by re-profiling.

8. Conclusion

Profiling is a critical skill for optimizing Linux applications. perf is your go-to for CPU and call-stack analysis, Valgrind for memory issues, strace for syscalls, and BPF tools for system-wide tracing. By combining these tools, you can diagnose even the trickiest performance bottlenecks.

Remember: Profiling is iterative. Start with hypotheses, measure, optimize, and repeat.

9. References