A Decision Tree for OOMKilled

A pod with a 4GiB limit died at 3.9GiB. The first suggestion in the incident channel was "raise the limit to 6". That would have moved the symptom three weeks into the future and hidden a goroutine leak growing at 40MiB per hour.

OOMKilled is one of the few failures where the evidence is complete, if you know where to look. This is the decision tree I use.

Question 1: Was it really OOM, and whose memory?

Exit code 137 means SIGKILL, which is not necessarily the kernel OOM killer:

kubectl get pod $POD -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

You want reason: OOMKilled for container limits, or reason: Evicted with a node MemoryPressure condition for node-level pressure. They have different fixes: one is a limit, the other is capacity or scheduling.

Question 2: Limit or node?

A container can be killed by its cgroup limit while the node has plenty of memory, and a node can evict pods that are under their limits when it runs out.

• kubectl describe node → MemoryPressure condition, eviction events.
• kubectl describe pod → Last State: Terminated, Reason: OOMKilled.
• If several unrelated pods die together, suspect the node. If one pod dies repeatedly, suspect the limit or the workload.

Question 3: Which kind of memory?

On cgroup v2, read the container's own accounting:

cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.peak
cat /sys/fs/cgroup/memory.stat
cat /sys/fs/cgroup/memory.events

memory.stat separates anon (heap, stacks, mmap) from file (page cache). A file-heavy workload can be killed with a small heap because page cache counts against the cgroup limit, and the fix is different from a heap leak.

memory.events gives you oom and oom_kill counters, which is how you separate a one-off spike from a recurring pattern.

Question 4: Leak, spike, or misconfiguration?

Plot the container's memory over 24 hours and look at the shape:

Shape	Likely cause	Evidence to collect
Sawtooth, high peaks	allocation burst, lazy GC	heap profile at peak, GC logs
Monotonic growth	leak	heap diff over 30 min, goroutine count
Step change after deploy	new allocation path	profiles before/after
Flat anon, growing file	page cache pressure	memory.stat file
Killed but RSS low	page cache or sidecar	per-container memory.stat

For Go, GOMEMLIMIT plus a heap profile diff is usually enough. For Java, the heap is only part of it: metaspace, direct buffers, and thread stacks do not appear in -Xmx.

What we found

Goroutine count had grown from 400 to 11,000 over four days. A channel send in a retry path had no receiver when the request was cancelled, and each abandoned request left one goroutine parked forever.

The heap profile at death showed 2.1GiB around runtime scheduler frames: the memory was in goroutine stacks, not live objects, which is why the heap graph looked benign.

The fix was a select with ctx.Done() and a GOMEMLIMIT as a backstop. Peak memory fell from 3.9GiB to 700MiB. We kept the 4GiB limit, because the limit was never the problem.

The checklist

1. Confirm OOMKilled vs Evicted.
2. Container limit vs node pressure.
3. anon vs file vs stacks, from memory.stat.
4. Shape over time: leak, burst, or misconfiguration.
5. Fix the cause; change the limit last.

---

Raising a memory limit is a valid fix exactly once: when you have proven the workload is legitimate and the limit was wrong. Every other time, it is a delay.