The Linux Kernel/Debugging

Performance
There are many factors that can affect the performance of the Linux kernel, including hardware configurations, software configurations, and workload characteristics.

In this context, performance optimization of the Linux kernel involves identifying and addressing performance in the system. This can involve tuning kernel parameters, optimizing system resources, and identifying and fixing bugs and other issues that may be impacting performance.

Given the complexity of the Linux kernel and the wide range of factors that can affect performance, performance optimization can be a challenging task. However, with the right tools and techniques, it is possible to significantly improve the performance and reliability of Linux-based systems.

, short for performance events, is a powerful interface that provides detailed insights into the performance characteristics of software running on a system. By analyzing the data collected by perf_events, developers can identify performance bottlenecks and optimize software to improve performance and reduce resource utilization. Perf_events is designed to be a lightweight, low-overhead monitoring solution that has minimal impact on system performance.

🔧 TODO

⚲ Interfaces
 * – performance analysis tools


 * Basic commands:
 * – display help information about perf
 * – System profiling tool.
 * – Run a command and record its profile into perf.data
 * – Read perf.data (created by perf record) and display th...
 * Other commands:
 * – Read perf.data (created by perf record) and display an...
 * – Create archive with object files with build-ids found ...
 * – Support for Arm Statistical Profiling Extension within...
 * – General framework for benchmark suites
 * – Manage build-id cache.
 * – List the buildids in a perf.data file
 * – Shared Data C2C/HITM Analyzer.
 * – Get and set variables in a configuration file.
 * – Run record sessions on background
 * – Data file related processing
 * – Read perf.data files and display the differential profile
 * – Filter sample events using a dynamically loaded shared...
 * – List the event names in a perf.data file
 * – simple wrapper for kernel's ftrace functionality
 * – Filter to augment the events stream with additional in...
 * – Support for Intel Processor Trace within perf tools
 * – Show I/O performance metrics
 * – Searches running kernel for symbols
 * – Tool to trace/measure kernel memory properties
 * – Tool to trace/measure kvm guest os
 * – Tool to trace/measure kernel work properties (latencies)
 * – List all symbolic event types
 * – Analyze lock events
 * – Profile memory accesses
 * – Define new dynamic tracepoints
 * – Tool to trace/measure scheduler properties (latencies)
 * – Read perf.data (created by perf record) and display tr...
 * – Process trace data with a Perl script
 * – Process trace data with a Python script
 * – Run a command and gather performance counter statistics
 * – Runs sanity tests.
 * – Tool to visualize total system behavior during a workload
 * – strace inspired tool
 * – display the version of perf binary


 * – sets up performance monitoring

🛠️ Utilities
 * , https://pcp.io/ – Performance Co-Pilot
 * , https://prometheus.io/
 * https://github.com/redhat-nfvpe/container-perf-tools
 * https://github.com/brendangregg/perf-tools – performance analysis tools based on Linux perf_events (aka perf) and ftrace
 * readprofile – a tool to read kernel profiling information

⚙️ Internals
 * – simple profiling
 * – simple profiling
 * – simple profiling

📚 References
 * stress-ng – exercises various kernel interfaces
 * – instruments CPU performance counters, tracepoints, kprobes, and uprobes
 * https://perf.wiki.kernel.org/
 * http://trac.gateworks.com/wiki/linux/profiling
 * Analyzing application performance in RHEL 9
 * Monitoring and managing system status and performance in RHEL 9
 * The Unofficial Linux Perf Events Web-Page
 * Real-time Linux

User space debug interfaces
⚲ Interfaces
 * – prints or control the kernel ring buffer
 * – system call, which is used to control the kernel printk buffer
 * – system calls and signals tracing tool
 * – process trace system call
 * /sys/kernel/debug/ –
 * dmesg --console-level
 * gdb /usr/src/linux/vmlinux /proc/kcore
 * /proc/self/stack
 * gdb /usr/src/linux/vmlinux /proc/kcore
 * /proc/self/stack


 * debug
 * ⌨️ hands-on:
 * echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control

⚙️ Internals

📚 References

Tracing and logging
⚲ API:

User-space interface:
 * – prints or control the kernel ring buffer
 * – system call, which is used to control the kernel printk buffer
 * /proc/kmsg

Most common functions


 * - conditional debug-level message
 * - conditional debug-level or message
 * ⌨️ hands-on:
 * echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control
 * Log messages with other levels:
 * Log messages with other levels:

⚙️ Internals

📚 References:
 * Debugging by printing
 * – systemtap script translator/driver
 * – trace system calls and signals
 * – function, latency and event tracing
 * – trace system calls and signals
 * – function, latency and event tracing
 * – function, latency and event tracing
 * – function, latency and event tracing
 * – function, latency and event tracing
 * – function, latency and event tracing
 * – function, latency and event tracing

'''🔧 TODO. 🚀 advanced features'''
 * – memory leak detector
 * - continues a previous log message in the same line
 * https://git.kernel.org/pub/scm/libs/libtrace/
 * https://git.kernel.org/pub/scm/libs/libtrace/
 * https://git.kernel.org/pub/scm/libs/libtrace/
 * https://git.kernel.org/pub/scm/libs/libtrace/
 * https://git.kernel.org/pub/scm/libs/libtrace/
 * https://git.kernel.org/pub/scm/libs/libtrace/
 * https://git.kernel.org/pub/scm/libs/libtrace/

kgdb and kdb
⚲ Interfaces

⚙️ Internals

📚 References
 * – Analyze Linux crash dump data or a live system
 * – Analyze Linux crash dump data or a live system
 * – Analyze Linux crash dump data or a live system
 * – Analyze Linux crash dump data or a live system

⚲ API:

📖 References

📚 Further reading
 * Linux Extended BPF (eBPF) Tracing Tools
 * bpftrace – High-level tracing language for Linux eBPF
 * BCC – Tools for BPF-based Linux IO analysis, networking, monitoring, and more
 * Example of trace.py
 * eBPF Programming for Linux Kernel Tracing
 * eBPF Programming for Linux Kernel Tracing
 * eBPF Programming for Linux Kernel Tracing

📖 References for debugging

📚 Further reading
 * https://drgn.readthedocs.io/ – programmable debugger
 * https://crash-utility.github.io/
 * https://wiki.ubuntu.com/Kernel/Debugging
 * Linux Applications Debugging Techniques