eBPF Is Not Free: Sampling, Loss, and the Bias You Can't See

"Low overhead" is a relative statement. A kprobe on malloc at 100,000 calls per second is not low overhead; it is a tax on the exact path you are trying to understand.

We learned this while chasing a 2ms read path that appeared in production latency but never in our traces.

Two tools, often confused

Sampling and tracing answer different questions:

Tool	Question	Bias
perf sampling at 99Hz	where is time spent?	misses short or rare paths; statistical error
eBPF tracepoint	did this happen, and how often?	observes every event; can drop events; perturbs timing
uprobe on a hot function	what are the arguments?	per-call overhead on the hot path
off-CPU profiling	why is it waiting?	scheduling events; sampling again

We had been using a tracepoint on sys_enter_read to find slow disk reads, filtered to one container. At 200,000 events per second, the perf ring buffer dropped about 8% of them — and the drops were not random. They concentrated in bursts, which is exactly when the slow reads happened.

The drop is a result, not an error

Every eBPF program that writes to a perf or ring buffer can lose events. bcc reports it; bpftrace prints a lost-events line. If you do not look at it, you are reporting a sample of the burst as if it were the burst.

bpftrace -e '
tracepoint:syscalls:sys_enter_read /pid == 4412/ {
  @start[tid] = nsecs;
}
tracepoint:syscalls:sys_exit_read /pid == 4412 && @start[tid]/ {
  $d = nsecs - @start[tid];
  if ($d > 1000000) { @slow = hist($d / 1000000); }
  delete(@start[tid]);
}'

This filters in the kernel and records only slow calls. The event rate fell from 200k/s to a few hundred per minute, which is the point: filter before you copy to user space.

The bias that remains

Filtering fixes volume, not perturbation. A uprobe on a function that runs 100k times per second adds instructions and cache pressure on the hot path, and it can change the timing you are measuring. We measured a 6–14% throughput regression on a proxy with a per-request uprobe, depending on the function.

The honest way to report an eBPF measurement:

1. Run an A/B with the probe detached.
2. Record lost events and attach time as part of the result.
3. Prefer tracepoints over kprobes where both exist.
4. Prefer filters and aggregation in the kernel over exporting raw events.
5. For rare paths use targeted tracing; for hot paths use sampling.

What found the 2ms read

Not the tracepoint. We ran an off-CPU profile on the container and saw a single syscall wait on a network filesystem, in a path our tracepoint filter had excluded because it came from a sidecar PID namespace.

The lesson was not "eBPF is bad". It was that our filter encoded an assumption — "the slow reads come from this PID" — and the assumption was wrong. The tracepoint was faithful; the question was not.

A short checklist before you attach anything

• Is the question "where" or "did"? Choose the tool accordingly.
• What is the event rate, and what is the buffer size?
• What is the lost-event count during the window?
• What is the overhead with the probe on versus off?
• Which PIDs and namespaces are excluded, and could the answer be there?

---

eBPF shows you exactly what you asked for, at a price you pay on the hot path. Ask carefully.