Page Cache, Eviction, and the cgroup Limit

A log-processing pod with a 4GiB limit was killed while its Go heap was 600MiB. The team raised the limit to 6GiB; it died at 5.8GiB two days later. The heap had not moved.

The missing term was page cache. In cgroup v2, memory.current includes file cache, and memory.max is enforced against it. A workload that reads files quickly can be killed with a tiny heap.

What actually counts

cat /sys/fs/cgroup/memory.stat | grep -E '^(anon|file|shmem|slab|sock) '

Line	Contents	Reclaimable?
anon	heap, stacks, anonymous mmap	no (swap aside)
file	page cache from reads and writes	yes, but not instantly
shmem	tmpfs, shared memory	no (swap aside)
slab	kernel objects, dentries	partly
sock	socket buffers	no

File cache is reclaimable, which is why "it will get reclaimed" is technically true and practically wrong. Reclaim happens under pressure, and pressure is exactly the moment you needed headroom.

The three ways it goes wrong

Dirty pages. Writes count as file cache until writeback completes. A burst of writes against a slow disk can pin 2GiB of dirty pages, and the cgroup cannot evict them until they are flushed.

No throttling before the cliff. If memory.high is unset, allocation goes from comfortable to OOM with no intermediate signal. memory.high triggers reclaim and throttling instead of a kill.

Sampling misses the peak. memory.current read every 30s can show 2.8GiB while the container died at 4GiB. memory.peak preserves the high-water mark; use it in incident review.

What our workload did

Twenty files of 2GiB, read sequentially, parsed, discarded. Heap stayed at 600MiB. Page cache climbed to 3.4GiB in about 90 seconds, dirty pages from the output writes pushed it over 4GiB, and the kernel killed the process.

Three fixes, with measured trade-offs:

Fix	Result
memory.high = 3.4GiB + fadvise(DONTNEED)	OOM gone; p99 processing 1.1s → 1.15s
O_DIRECT reads	OOM gone; throughput −18%
Read buffer 4MiB → 512KiB	OOM delayed to 6 minutes; not a fix

We kept memory.high and explicit posix_fadvise(POSIX_FADV_DONTNEED) after each chunk. That is a deliberate trade: 4% slower processing in exchange for bounded memory. O_DIRECT was too expensive for our access pattern.

Eviction is not OOM

Signal	Meaning	Fix
Reason: OOMKilled, exit 137	cgroup limit exceeded	size the limit, reduce cache, set high
Reason: Evicted, node MemoryPressure	kubelet reclaiming node memory	requests/limits, QoS, node capacity
memory.events oom_kill rising	repeated cgroup kills	same as row one
memory.events high rising	throttling, not killing	expected; watch latency

Node eviction is decided by kubelet thresholds and QoS class; container OOM is decided by the cgroup. They look similar in kubectl get pods and have completely different owners.

The checklist

1. Read memory.stat once: anon vs file vs shmem.
2. Read memory.peak, not just the current sample.
3. Look at dirty pages: is writeback the bottleneck?
4. Set memory.high below memory.max so pressure throttles before it kills.
5. Only then change the limit, and change requests with it.

---

The limit is not a memory budget for your heap. It is a budget for everything the kernel is holding on your behalf, including the file cache you thought was free.