eBPF-based Linux profiler — CPU, off-CPU, and PMU, system-wide or per-PID, pprof output.
One binary, runs locally, no backend or telemetry.
- Quickstart
- What you can do with perf-agent
- Requirements
- Usage
- Flags
- Output
- Library usage
- Architecture
- Building
- Testing
- Contributing
- Security
- License
# Build (one-time, see BUILDING.md for full toolchain setup)
make build
# Grant capabilities once so subsequent runs don't need sudo
sudo setcap cap_sys_admin,cap_bpf,cap_perfmon,cap_sys_ptrace,cap_checkpoint_restore+ep ./perf-agent
# Capture a 30-second CPU profile of one process — output is pprof
./perf-agent --profile --pid <PID> --duration 30s
# Inspect
go tool pprof <output>.pb.gzHot-attach to a running process — no restart, no preinstalled agent. The agent never modifies the process it measures.
Find why a service is "slow but not CPU-busy." --offcpu hooks sched_switch and accumulates blocking time per call site — lock waits, syscall blocks, channel reads, mutex contention.
One profile, multiple runtimes. Native (DWARF + ELF) symbolizes alongside Node.js (--perf-basic-prof), Go, and any runtime that writes a /tmp/perf-<pid>.map. The hybrid FP+DWARF unwinder handles release-built C++/Rust without -fno-omit-frame-pointer.
Python frames: not currently supported. The
--inject-pythonflag and its perf-trampoline injector have been removed. It mutated the process it measured, required CPython 3.12+, and requiredCAP_SYS_PTRACEto ptrace into the target. Its replacement — walking the interpreter's frame chain from BPF, which needs no injection and covers CPython 3.6+ — is tracked in #83 and is not built yet.Until #83 lands, perf-agent has no Python-level frames of its own. Python processes still profile fine, but the stacks show C interpreter frames (
_PyEval_EvalFrameDefault, …) instead of Python qualnames. If you control how the interpreter starts, launching it yourself withpython -X perf(3.12+) makes CPython write/tmp/perf-<pid>.map, and perf-agent will read and decode those entries — that path is unchanged.
--pmu summarizes IPC, cache miss rate, runqueue latency (P50/P99), and context-switch reasons (preempted vs voluntary vs I/O wait). Combine with --per-pid in system-wide mode to see which processes dominate the node's wait time.
Run as a DaemonSet on the host PID namespace (recommended): perf-agent
sees every node process and tags each sample with pod_uid,
container_id, and cgroup_path parsed from /proc/<pid>/cgroup — no
kubelet API, no client-go.
For single-tenant pods, sidecar mode also works with
shareProcessNamespace: true (which exposes every container's processes
to every other container — fine when the agent and target are co-deployed
by the same operator, a security regression otherwise). Downward-API
env vars then add pod_name / namespace / container_name labels.
--pid <N> accepts in-pod PIDs and translates them to host PIDs automatically.
Production builds usually strip debug info. Point perf-agent at a
debuginfod-protocol server with --debuginfod-url=URL. A per-mapping
classifier routes each binary in the target:
- Has local DWARF or resolvable
.gnu_debuglink→ blazesym's process-mode (system libs from distro debuginfo land here for free). - Stripped, build-id only (Rust/Go release builds) → file-mode against the
cached
.debug, fetched on demand and content-addressed by build-id. - Deleted-but-still-mapped binary (sidecar / mount-namespace case) →
same flow, opened via
/proc/<pid>/map_files.
Cache layout, dispatcher details, and the address-normalization math: see docs/debuginfod-symbolization.md.
High-fidelity pprof: every Mapping carries the absolute path, GNU build-id, and file offsets; every Location is address-stable across runs. Feeds go tool pprof -diff_base and Go's native -pgo=... flag.
For toolchains that don't speak pprof, add --perf-data-output app.perf.data to emit a kernel-format perf.data alongside the pprof output. Same capture, two formats:
- AutoFDO PGO for Rust (
rustc -Cllvm-args=-sample-profile-file=...) and C++ (clang -fprofile-sample-use=...) via Google'screate_llvm_prof. End-to-end demo:examples/rust-pgo,examples/cpp-pgo. - FlameGraph —
perf script | stackcollapse-perf.pl | flamegraph.plproduces an SVG. Demo:examples/flamegraph.
See docs/perf-data-output.md for the per-tool walkthrough.
--flamegraph-output auto writes an interactive flame graph beside the pprof file — one HTML file, no server, no CDN, no external script or font, correct opened straight off disk. Click a frame to zoom, / to search, Esc to clear then reset.
sudo ./perf-agent --pid 1234 --profile --flamegraph-output auto --duration 30sTo render a profile written earlier, or one from any other pprof producer:
go run ./cmd/flamegraph -o profile.html profile.pb.gz
go run ./cmd/flamegraph -folded profile.pb.gz # the a;b;c 123 text formColour encodes domain — application, libc/startup, GPU runtime, unsymbolized, perf-agent's own shim, the [gpu:launch] CPU→GPU boundary, GPU kernel — not a hash of the frame name, and the page carries a legend saying so.
The page also states what it is not showing: the count of frames with no symbol, the share of GPU time sitting under [gpu:launch unsampled] with no CPU caller, the launch sampling period and what it does to the widths, and every per-sample label that is deliberately kept out of the tree. A profile with no samples renders a page that says so rather than an empty rectangle.
- Linux kernel 5.8+ (BTF + CO-RE).
- Root, OR
setcap cap_sys_admin,cap_bpf,cap_perfmon,cap_sys_ptrace,cap_checkpoint_restore+ep ./perf-agent.
What each capability is for, and when cap_sys_admin can be dropped
| Capability | Why it is needed |
|---|---|
cap_bpf |
Load eBPF programs and create maps |
cap_perfmon |
perf_event_open, stack traces, tracing attachment |
cap_sys_ptrace |
Read /proc/<pid>/maps and /proc/<pid>/mem of the target |
cap_checkpoint_restore |
Follow /proc/<pid>/map_files/ symlinks during symbolization |
cap_sys_admin |
Only on kernels older than 5.8/5.9 — see below |
cap_checkpoint_restore (or cap_sys_admin) is required, not optional.
blazesym reaches the file behind every mapping through
/proc/<pid>/map_files/, and the kernel refuses to follow those magic
symlinks without one of the two — so without it every user-space frame in a
profile is a bare hex address. perf-agent checks this at startup and refuses
to run rather than write a profile that looks like a result and is not.
cap_sys_admin is kept for backward compatibility. Two capabilities were added
to the kernel to carve out the roles perf-agent used it for — but they did not
arrive in the same release:
CAP_PERFMON(kernel 5.8) coversperf_event_open, includingpid=-1for system-wide profiling.CAP_CHECKPOINT_RESTORE(kernel 5.9) covers/proc/<pid>/map_files.
perf-agent's documented floor is kernel 5.8, and on exactly 5.8
cap_checkpoint_restore does not exist — so dropping cap_sys_admin there would
break symbolization. That single kernel minor version is why the full set is
still the default.
On kernel 5.9 or newer the minimal set is:
sudo setcap cap_bpf,cap_perfmon,cap_sys_ptrace,cap_checkpoint_restore+ep ./perf-agentIf you run 6.x — as most deployments now do — this is the set to use. It matters
most for per-pod and sidecar deployments, where cap_sys_admin is the near-root
capability that gets a workload rejected by admission policy.
# CPU profiling — DWARF/hybrid walker is the default
./perf-agent --profile --pid <PID>
# Force frame-pointer-only walker (cheaper startup, may truncate on FP-less binaries)
./perf-agent --profile --unwind fp --pid <PID>
# Force DWARF walker (eager CFI compile + per-frame hybrid)
./perf-agent --profile --unwind dwarf --pid <PID>
# Off-CPU profiling
./perf-agent --offcpu --pid <PID>
# Combined on-CPU + off-CPU
./perf-agent --profile --offcpu --pid <PID>
# PMU only (hardware counters)
./perf-agent --pmu --pid <PID>
# System-wide
./perf-agent --profile -a --duration 30s
# All features with metadata tags
./perf-agent --profile --offcpu --pmu --pid <PID> --duration 30s \
--tag env=production \
--tag version=1.2.3 \
--tag service=apiProfiling a Python workload? See the note under Cross-language flame graphs — Python-level frames are unavailable until #83 lands.
| Flag | Description | Default |
|---|---|---|
--profile |
Enable CPU profiling with stack traces | false |
--offcpu |
Enable off-CPU profiling with stack traces | false |
--pmu |
Enable PMU hardware counters | false |
--pid <PID> |
Target process ID | - |
-a, --all |
System-wide (all processes) | false |
--per-pid |
Per-PID breakdown (only with -a --pmu) |
false |
--duration |
Collection duration | 10s |
--sample-rate |
CPU profile sample rate (Hz) | 99 |
--unwind |
Stack unwinding strategy: fp | dwarf | auto (auto routes to dwarf; the hybrid walker covers FP-safe code via the FP path) |
auto |
--profile-output |
Output path for CPU profile | auto-named |
--offcpu-output |
Output path for off-CPU profile | auto-named |
--pmu-output |
Output path for PMU metrics (auto for auto-named) |
stdout |
--flamegraph-output |
Also write a self-contained interactive HTML flame graph of the profile (auto for auto-named). Requires --profile or --offcpu. |
- |
--perf-data-output |
Also emit a Linux kernel-format perf.data (consumable by perf script, FlameGraph, hotspot, AutoFDO create_llvm_prof, …). Requires --profile. |
- |
--tag key=value |
Add tag to profile (repeatable) | - |
--debuginfod-url=URL |
Add a debuginfod-protocol server (repeatable). Falls back to DEBUGINFOD_URLS env. Unset → off. |
- |
--symbol-cache-dir=DIR |
Local directory for fetched artifacts. | /tmp/perf-agent-debuginfod |
--symbol-cache-max=BYTES |
LRU cap for the symbol cache. | 2147483648 (2 GiB) |
--symbol-fetch-timeout=DUR |
Per-artifact HTTP fetch timeout. | 30s |
--symbol-fail-closed |
(M2 stub) Refuse to symbolize a mapping whose fetch failed. | false |
Either --pid or -a/--all is required. At least one of --profile, --offcpu, or --pmu must be specified.
Output files are auto-named by process name + timestamp + profile type:
| Mode | Per-PID example | System-wide example |
|---|---|---|
--profile |
myapp-202604021430-on-cpu.pb.gz |
202604021430-on-cpu.pb.gz |
--offcpu |
myapp-202604021430-off-cpu.pb.gz |
202604021430-off-cpu.pb.gz |
--pmu-output auto |
myapp-202604021430-pmu.txt |
202604021430-pmu.txt |
--flamegraph-output auto |
myapp-202604021430-on-cpu.html |
202604021430-on-cpu.html |
Process name comes from /proc/<pid>/comm. Override with --profile-output / --offcpu-output.
CPU and off-CPU profiles are full-fidelity pprof: every Mapping carries the absolute path, GNU build-id, and file offsets; every Location is keyed by file offset (not symbol name) so cross-run diffing and sample-PGO converters work. [kernel] and [jit] sentinels handle the special cases. Tags from --tag key=value land as profile-level comments; k8s identity labels (when running in a pod) attach per-sample.
go tool pprof myapp-202604021430-on-cpu.pb.gzWith --debuginfod-url configured, pprof comes back fully symbolized —
function names + source :line — even when debug info isn't present
locally. See docs/debuginfod-symbolization.md.
On-CPU time, runqueue latency, context-switch reasons, hardware counters (cycles, instructions, cache misses), and derived metrics (IPC, cache miss rate).
Example:
=== PMU Metrics (PID: 84228) ===
Samples: 26358
On-CPU Time (time slice per context switch):
Min: 0.003 ms
P50: 0.071 ms
P99: 9.183 ms
Runqueue Latency (time waiting for CPU):
Min: 0.001 ms
P50: 0.012 ms
P99: 0.850 ms
Context Switch Reasons:
Preempted (running): 45.2% (11912 times)
Voluntary (sleep/mutex): 42.1% (11095 times)
I/O Wait (D state): 12.7% (3351 times)
Hardware Counters:
IPC (Instr/Cycle): 2.342
Cache Misses/1K: 0.022
perf-agent is also a Go library via the perfagent package:
agent, _ := perfagent.New(
perfagent.WithPID(12345),
perfagent.WithCPUProfile("profile.pb.gz"),
perfagent.WithPMU(),
)
defer agent.Close()
agent.Start(ctx); time.Sleep(10*time.Second); agent.Stop(ctx)See the perfagent package docs for in-memory output, custom label enrichers, and metrics exporters.
┌──────────────────────────────────────────────────────────────────────────┐
│ USER SPACE (Go) │
│ │
│ ┌──────────┐ │
│ │ main.go │ │
│ └────┬─────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ perfagent.Agent │ lifecycle + --unwind dispatch│
│ └─────┬────────────┘ │
│ ┌─────────────────────┼─────────────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌───────────────┐ ┌──────────────────────┐ ┌──────────────┐ │
│ │ CPU Profiler │ │ DWARF CPU/Off-CPU │ │ PMU Monitor │ │
│ │ (FP path) │ │ Profiler │ │ │ │
│ │ profile/ │ │ unwind/dwarfagent/ │ │ cpu/ │ │
│ │ offcpu/ │ │ (hybrid walker) │ │ │ │
│ └───────┬───────┘ └──────────┬───────────┘ └──────┬───────┘ │
│ │ │ │ │
│ │ ┌───────────────┴───────────────┐ │ │
│ │ ▼ ▼ │ │
│ │ ┌─────────────────┐ ┌──────────────────────┐ │
│ │ │ unwind/ehcompile│ │ unwind/ehmaps │ │
│ │ │ .eh_frame → CFI │ │ per-PID map lifecyle│ │
│ │ └─────────────────┘ │ + MMAP2 watcher │ │
│ │ └──────────┬───────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ unwind/procmap (Resolver) │ │
│ │ /proc/<pid>/maps + .note.gnu.build-id, lazy per-PID cache │ │
│ └────────────────────┬─────────────────────────────────────────┘ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ pprof/ ProfileBuilder │ │
│ │ address-keyed Locations + per-binary Mapping (build-id, │ │
│ │ file offsets) + kernel/[jit] sentinels + name-based │ │
│ │ fallback when resolver misses │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ Symbolization: blazesym (DWARF + ELF + perf-maps for JIT runtimes) │
└─────────────┬──────────────────┬──────────────────┬──────────────────────┘
│ │ │
══════════════╪══════════════════╪══════════════════╪═══════════════════════
│ eBPF load │ │
▼ ▼ ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ KERNEL SPACE (eBPF) │
│ │
│ ┌──────────────┐ ┌────────────────┐ ┌────────────────┐ ┌──────────┐ │
│ │ perf.bpf.c │ │ perf_dwarf.bpf │ │ offcpu.bpf.c │ │ cpu.bpf.c│ │
│ │ (FP only) │ │ (hybrid: FP │ │ + offcpu_dwarf │ │ HW ctrs │ │
│ │ stackmap │ │ fast path, │ │ sched_switch │ │ rq lat │ │
│ │ aggregated │ │ DWARF for │ │ blocking-ns │ │ ctx swch │ │
│ │ counts │ │ FP-less PCs) │ │ │ │ │ │
│ └──────┬───────┘ └────────┬───────┘ └────────┬───────┘ └────┬─────┘ │
│ │ │ │ │ │
│ │ CFI tables, classification, pid_mappings │ │
│ │ via HASH_OF_MAPS keyed by build-id │ │
│ │ │ │ │
│ └────────┬──────────┴──────────────┬────────────────────┘ │
│ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────────┐ │
│ │ stack ringbuf│ │ aggregated maps │ │
│ │ (DWARF path) │ │ (FP path) │ │
│ └─────────────┘ └─────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ OUTPUT │
│ │
│ *-on-cpu.pb.gz *-off-cpu.pb.gz │
│ PMU: console / file │
└──────────────────────────────────────┘
Two stack-walker paths: --unwind fp (cheap, kernel-side aggregation; truncates on FP-less code) and --unwind dwarf / auto (default — FP fast path with .eh_frame-derived CFI fallback for release C++/Rust without frame pointers).
Sample addresses resolve through procmap.Resolver (lazy /proc/<pid>/maps + build-id), so each pprof Mapping carries real per-binary identity and each Location is keyed by (mapping_id, file_offset) — what go tool pprof -diff_base and sample-based PGO converters need to round-trip.
Requires Go 1.26+, Clang/LLVM, Linux headers, and blazesym (Rust C library for symbolization).
Regenerating the eBPF objects needs Clang 18 specifically. The *_bpfel.o
files are committed build artifacts, so their bytes depend on the exact
compiler that produced them — a different Clang rewrites objects in packages
you did not touch, and the resulting diff is noise that will drift straight
back for the next person. CI pins Clang 18 (Ubuntu 24.04's default) and fails
if regeneration changes a committed object; make generate-check runs the same
check locally and prints your Clang version when it disagrees.
You only need this to run make generate. Building and testing use the
committed objects and work with any toolchain.
make buildThe Makefile defaults to GOTOOLCHAIN=auto, so Go fetches the pinned toolchain automatically if your system Go is older. Override with GOTOOLCHAIN=local make build to enforce the locally-installed toolchain.
See BUILDING.md for the full toolchain setup.
Unit tests run without root; integration tests require root or a setcap'd binary.
# Build + cap the binary once, then run tests as a normal user
make build
sudo setcap cap_sys_admin,cap_bpf,cap_perfmon,cap_sys_ptrace,cap_checkpoint_restore+ep ./perf-agent
# Unit tests (no root)
make test-unit
# Integration tests — auto-skip when neither root nor caps are available
make test-integrationTest gates honor file capabilities on the perf-agent binary: a setcap'd perf-agent lets the test runner exec it without sudo. For tests that load BPF in-process (library tests), the test binary itself needs caps — setcap it after go test -c.
For detailed testing documentation see TESTING.md.
PRs welcome. Read CONTRIBUTING.md before opening one — it covers build/test conventions, the commit-message style, and what's in-scope vs. deferred. By participating you agree to the Code of Conduct.
If you find a security issue, please do not open a public issue. See SECURITY.md for the reporting channel and threat model. perf-agent runs with elevated kernel capabilities; we take privilege-escalation and kernel-DoS reports seriously.
Apache License 2.0 — see LICENSE.