Building a Profiler That Tells You When It's Lying
Lost events, attach overhead, and a report that carries its own error bars.
Every profiler we tried had the same flaw: it reported a percentage without saying how many events it dropped to produce it.
We built a small in-house profiler for one workload — read latency on a storage service — with a stricter contract: every report must include the sample count, the lost-event rate, and the measured overhead of the probe itself.
The design constraints
- Aggregate in the kernel. A histogram per latency bucket, not one event per call. Exporting 200k events/s is how you lose 8% of them.
- Per-CPU maps. No cross-CPU contention on the hot path.
- Two paths. Aggregated histograms for the distribution; a small ring buffer for a bounded number of outliers, with an explicit cap.
- Error bars in the report. The tool prints lost events and an A/B overhead measurement; a report without them is not accepted by our incident reviews.
Kernel-side aggregation
b = BPF(text='''
BPF_HISTOGRAM(lat, u64, 64);
int trace_exit(struct pt_regs *ctx) {
u64 start = start_ns.lookup(&pid);
if (!start) return 0;
u64 delta = bpf_ktime_get_ns() - *start;
lat.increment(bpf_log2l(delta / 1000)); // microseconds
start_ns.delete(&pid);
return 0;
}
''')The userspace side reads one histogram every second. The event volume at the boundary between kernel and userspace is 1/s, not 200k/s, which is the entire trick.
The overhead harness
We run every probe configuration against a fixed workload for five minutes, with the probe attached and detached, and report the delta:
throughput: 41.2k req/s (probe off) -> 40.1k req/s (probe on) -2.7%
p99: 18ms -> 18.4ms +2.2%A 3% overhead is acceptable for a diagnostic run and unacceptable for a permanent agent. We ship the profiler as an on-demand tool for exactly this reason.
What loss looks like, and how we fixed it
Baseline configuration: 64 perf-buffer pages, one event per call. At 200k events/s we lost 8% of events, concentrated in bursts.
| Config | Lost | Overhead | Notes |
|---|---|---|---|
| 64 pages, per-event export | 8.0% | 3.1% | unusable for burst analysis |
| 256 pages, per-event export | 1.4% | 3.0% | better, still bursty |
| 256 pages, kernel histogram | 0.2% | 2.7% | our default |
| 256 pages + outlier ring | 0.2% | 2.9% | outliers retained |
The last row is what we run: histograms for the distribution, a 4,096-entry ring buffer for calls over 10ms, with a counter for dropped outliers. The report says "12 outliers dropped", which is honest.
Why this matters
A tracepoint that loses 8% of events in bursts does not have a sampling error; it has a bias toward whatever happened outside the burst. In our case the burst was the incident. The tool reported a calm system because it discarded the evidence.
The same logic applies to sampling profilers. A 99Hz CPU profile over 30 seconds is 2,970 samples; a function that takes 5ms may never appear. That does not mean it is not the problem.
The report format
window 2025-07-30T14:02Z .. 14:07Z
events 61.2M reads
samples lost 0.2% (122k)
outliers kept 4,096 / dropped 12
probe overhead +2.7% throughput, +2.2% p99
p50 0.8ms
p99 31ms
p99.9 240msNothing exotic, and every line is defensible. If a number cannot be defended, it does not go in the report.
A profiler that hides its own error is worse than no profiler. The first feature we built was the one that says "I dropped 8% of the evidence".