funwithlinux guide

Essential Kernel Debugging Techniques for Developers

Kernel debugging is a critical skill for developers working on operating systems, device drivers, or system-level software. Unlike user-space debugging, kernel debugging operates in a constrained environment where traditional tools like `gdb` or `printf` behave differently, and a single mistake can crash the entire system. The Linux kernel, for example, runs with full privileges, manages hardware directly, and lacks the safety nets of user-space (e.g., memory protection for kernel code). Debugging kernel issues—such as null pointer dereferences, race conditions, or memory leaks—requires specialized techniques and tools tailored to this low-level environment. This blog explores essential kernel debugging techniques, from basic logging to advanced tracing and post-mortem analysis. Whether you’re a seasoned kernel developer or just starting, these methods will help you diagnose bugs efficiently and build more reliable system software.

Table of Contents

  1. Introduction to Kernel Debugging
  2. Understanding the Kernel Debugging Environment
  3. Essential Static Debugging Techniques
  4. Dynamic Tracing and Profiling
  5. Interactive Debugging with kgdb
  6. Post-Mortem Debugging with kdump and crash
  7. Advanced Techniques: eBPF for Kernel Tracing
  8. Common Kernel Bugs and How to Diagnose Them
  9. Best Practices for Effective Kernel Debugging
  10. Conclusion
  11. References

Understanding the Kernel Debugging Environment

Kernel debugging differs fundamentally from user-space debugging due to the kernel’s role as the core of the operating system. Key differences include:

  • Privilege Level: The kernel runs in ring 0 (x86) or EL1 (ARM), with unrestricted access to hardware and memory. A bug here can corrupt the entire system, leading to panics or data loss.
  • No User-Space Safety Nets: There’s no signal handling for errors like segmentation faults. Instead, the kernel triggers an “oops” (non-fatal error) or “panic” (fatal error) and may reboot.
  • Limited Tooling: Traditional user-space tools (e.g., valgrind) don’t work in kernel-space. Specialized tools like kgdb, ftrace, or kdump are required.
  • Kernel Symbols and Artifacts: Debugging relies on kernel symbols (stored in vmlinux, the uncompressed kernel image), module symbols, and kallsyms (a runtime symbol table).

Key Concepts:

  • vmlinux: The kernel’s uncompressed, executable image with debug symbols (if compiled with -g).
  • Modules: Loadable kernel modules (.ko files) with their own symbols.
  • kallsyms: /proc/kallsyms provides runtime access to kernel symbol addresses, critical for debugging oops logs.
  • Oops/Panic Logs: Kernel error reports containing registers, stack traces, and faulty instruction pointers (RIP).

Essential Static Debugging Techniques

Static debugging involves embedding debug logic directly into kernel code or using compile-time tools to inspect behavior. These techniques are simple but powerful for initial diagnostics.

printk and Kernel Logging

printk is the kernel’s equivalent of printf and the most basic debugging tool. It logs messages to the kernel ring buffer, accessible via dmesg or /var/log/kern.log.

Log Levels:

printk supports log levels to prioritize messages. Levels range from KERN_EMERG (highest priority, system unusable) to KERN_DEBUG (lowest, verbose debugging):

printk(KERN_EMERG "System halted!\n");    // Level 0
printk(KERN_ALERT "Immediate action needed!\n"); // Level 1
printk(KERN_CRIT "Critical condition!\n");       // Level 2
printk(KERN_ERR "Error occurred!\n");            // Level 3
printk(KERN_WARNING "Warning!\n");               // Level 4
printk(KERN_NOTICE "Notice (normal but significant)\n"); // Level 5
printk(KERN_INFO "Informational message\n");     // Level 6
printk(KERN_DEBUG "Debugging message\n");        // Level 7

Controlling Log Output:

  • Use dmesg to view the ring buffer: dmesg | grep "my debug message".
  • Adjust verbosity with /proc/sys/kernel/printk, which sets the default log level (e.g., echo 8 > /proc/sys/kernel/printk enables all levels).

Pitfalls:

  • Buffer Size: The ring buffer is limited (default ~16KB). Avoid large messages or spamming printk in hot paths (e.g., interrupt handlers).
  • Performance Impact: Frequent printk calls slow down the kernel. Use them sparingly.

debugfs and procfs for Debug Data

debugfs and procfs allow kernel code to expose debug data via virtual files in /sys/kernel/debug (debugfs) or /proc (procfs). debugfs is preferred for debugging due to its flexibility and isolation from production interfaces.

Example: Creating a debugfs File

To expose a variable for read/write via debugfs:

#include <linux/debugfs.h>
#include <linux/module.h>

static struct dentry *debug_dir; // Handle for debugfs directory
static u32 debug_counter = 0;    // Variable to expose

static int __init my_debug_init(void) {
    // Create a directory under /sys/kernel/debug
    debug_dir = debugfs_create_dir("my_driver", NULL);
    if (!debug_dir)
        return -ENOMEM;

    // Expose debug_counter as a read/write file
    debugfs_create_u32("counter", 0644, debug_dir, &debug_counter);
    return 0;
}

static void __exit my_debug_exit(void) {
    debugfs_remove_recursive(debug_dir); // Cleanup
}

module_init(my_debug_init);
module_exit(my_debug_exit);
MODULE_LICENSE("GPL");

After loading the module, read/write the counter via:

cat /sys/kernel/debug/my_driver/counter
echo 42 > /sys/kernel/debug/my_driver/counter

Compiler Debug Flags

Compiling the kernel with debug flags enables richer debugging information:

  • -g: Includes debug symbols in vmlinux and modules, required for tools like gdb or crash.
  • -O0: Disables optimizations, making variables and control flow easier to inspect (use cautiously—slows the kernel).
  • -Og: Balances debuggability and performance (recommended for most debugging).

To enable these, configure the kernel with make menuconfig:

  • Set Kernel hacking → Compile-time checks and compiler options → Debug information to Reliable or Full.
  • Disable Kernel hacking → Compile the kernel with frame pointers (optional but helps with stack traces).

Dynamic Tracing and Profiling

Dynamic tracing tools capture runtime behavior without modifying kernel code, ideal for diagnosing performance issues or intermittent bugs.

Ftrace: Function Tracing and Beyond

ftrace is a built-in kernel tracer that supports function calls, events, and custom tracepoints. It’s accessible via /sys/kernel/debug/tracing.

Basic Function Tracing:

  1. Enable the function tracer: 
    echo function > /sys/kernel/debug/tracing/current_tracer
  2. View traces: 
    cat /sys/kernel/debug/tracing/trace
  3. Filter by function or module: 
    echo "sys_open" > /sys/kernel/debug/tracing/set_ftrace_filter

Advanced Tracers:

  • Function Graph Tracer: Shows call graphs with latency: 
    echo function_graph > /sys/kernel/debug/tracing/current_tracer
  • Event Tracer: Traces kernel events (e.g., scheduler, memory allocations): 
    ls /sys/kernel/debug/tracing/events # List events
echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable # Enable event

perf: Performance Analysis and Event Tracing

perf is a versatile tool for profiling and tracing kernel/user-space events. It uses hardware performance counters and kernel tracepoints.

Common Use Cases:

  • Trace syscalls: perf trace shows syscalls in real time.
  • Sample function calls: perf record -g -p <pid> profiles a process, and perf report visualizes the call graph.
  • Trace kernel events: perf list shows available events (e.g., sched:sched_switch), and perf record -e sched:sched_switch captures them.

Example: Diagnosing Latency

To find functions causing high latency:

perf record -g -a sleep 10  # Profile all CPUs for 10 seconds
perf report --sort=dso,function  # Show latency by function

Interactive Debugging with kgdb

kgdb extends gdb to debug the kernel interactively, supporting breakpoints, watchpoints, and stack inspection. It requires two machines (or a VM) connected via serial, Ethernet, or USB.

Setting Up kgdb

1. Configure the Kernel:

Enable kgdb in make menuconfig:

  • Kernel hacking → KGDB: kernel debugger
  • Kernel hacking → KGDB: use kgdb over serial console (or Ethernet with kgdboc).

2. Boot the Target Kernel with kgdb Options:

Add to the bootloader (e.g., GRUB) command line:

kgdboc=ttyS0,115200 kgdbwait
  • kgdboc=ttyS0,115200: Use serial port ttyS0 at 115200 baud.
  • kgdbwait: Pause at boot to wait for a debugger connection.

3. Connect from the Host:

On the host machine, run:

gdb vmlinux
(gdb) target remote /dev/ttyS0  # Replace with your serial port

Breakpoints, Watchpoints, and Inspecting State

Once connected, use standard gdb commands:

  • break sys_open: Break when sys_open is called.
  • watch my_variable: Pause when my_variable is modified.
  • bt: Print a stack trace.
  • info registers: Inspect CPU registers.
  • c: Continue execution.

Post-Mortem Debugging with kdump and crash

When the kernel panics, kdump captures a memory dump (vmcore) for offline analysis. The crash utility then inspects the vmcore using vmlinux symbols.

Configuring kdump

1. Reserve Memory for the Crash Kernel:

Add crashkernel=128M to the bootloader command line (adjust size based on RAM).

2. Install kdump Tools:

On Debian/Ubuntu:

sudo apt install kdump-tools

On RHEL/CentOS:

sudo yum install kexec-tools

3. Trigger a Panic to Test:

Force a panic (for testing only!):

echo c > /proc/sysrq-trigger

The system will reboot, and kdump will save vmcore to /var/crash/.

Analyzing vmcores with crash Utility

The crash tool parses vmcore to inspect the kernel’s state at panic time:

crash vmlinux /var/crash/202401011234/vmcore

Key crash Commands:

  • bt: Show the panic stack trace.
  • ps: List running processes.
  • lsmod: List loaded modules.
  • kmem: Inspect kernel memory usage.
  • struct task_struct 0xdeadbeef: Dump a task_struct (process descriptor).

Advanced Techniques: eBPF for Kernel Tracing

eBPF (extended Berkeley Packet Filter) is a revolutionary technology for writing sandboxed kernel programs without recompiling the kernel. It’s ideal for tracing, monitoring, and debugging.

Introduction to eBPF

eBPF programs run in a restricted VM in the kernel, triggered by events (e.g., syscalls, function calls, network packets). They can collect data and send it to user-space via maps.

Using bcc and bpftrace for Debugging

BCC (BPF Compiler Collection):

BCC provides Python/JavaScript frontends to write eBPF programs. Example: Trace sys_open calls with filenames:

from bcc import BPF

# eBPF program to trace sys_open
bpf_code = """
#include <vmlinux.h>
#include <bcc/proto.h>

int trace_sys_open(struct pt_regs *ctx, const char __user *filename) {
    bpf_trace_printk("Opening file: %s\\n", filename);
    return 0;
}
"""

# Load and attach to sys_open
b = BPF(text=bpf_code)
b.attach_kprobe(event="sys_open", fn_name="trace_sys_open")

# Print output
b.trace_print()

bpftrace:

bpftrace is a high-level scripting language for eBPF. Example: Count sys_open calls per process:

bpftrace -e 'kprobe:sys_open { @count[pid, comm] = count(); }'

Common Kernel Bugs and How to Diagnose Them

Null Pointer Dereferences

Symptom: Oops log with NULL pointer dereference and RIP (instruction pointer).
Diagnosis:

  1. Use addr2line to map RIP to a function/line: 
    addr2line -e vmlinux 0xffffffff81234567
  2. Check the function for uninitialized pointers (e.g., struct my_struct *ptr; ptr->field = 42;).

Race Conditions and Deadlocks

Symptom: System hang, high CPU usage, or lockdep warnings.
Diagnosis:

  • Use ftrace to trace lock acquisition order: 
    echo lockdep > /sys/kernel/debug/tracing/current_tracer
  • Use perf lock to detect contended locks: 
    perf lock record -p <pid>  # Record lock events
perf lock report           # Analyze contention

Memory Leaks

Symptom: Increasing kernel memory usage (free -m shows shrinking MemFree).
Diagnosis:

  • Use kmemleak: Enable with kmemleak=on in the boot command line, then: 
    echo scan > /sys/kernel/debug/kmemleak
cat /sys/kernel/debug/kmemleak  # List unreferenced objects

Best Practices for Effective Kernel Debugging

  1. Reproduce the Bug Consistently: Debugging intermittent bugs is nearly impossible. Isolate triggers (e.g., specific workloads, hardware).
  2. Use a Test Environment: Never debug on production systems—use VMs (QEMU/KVM) or dedicated test machines.
  3. Leverage Multiple Tools: Combine printk, ftrace, and kgdb for layered insights.
  4. Enable Debug Symbols: Always compile the kernel with -g to get meaningful stack traces.
  5. Document Everything: Log steps, oops logs, and tool outputs for later analysis.
  6. Start Simple: Use dmesg and printk first before moving to complex tools like eBPF.

Conclusion

Kernel debugging is a challenging but rewarding skill. By mastering techniques like printk, ftrace, kgdb, and eBPF, developers can diagnose even the most elusive kernel bugs. Remember to combine static and dynamic tools, leverage post-mortem analysis, and follow best practices to minimize downtime and system risk. With persistence and the right tools, you’ll be able to build more robust and reliable kernel software.

References

  1. Linux Kernel Documentation: Debugging
  2. Ftrace User Guide
  3. kgdb Documentation
  4. kdump and crash Utility
  5. eBPF.io
  6. BPF Compiler Collection (BCC)
  7. bpftrace Reference Guide
  8. Linux Kernel Development by Robert Love (3rd Edition)
  9. Kernel Memory Leak Debugging