The Day an Agent Ate the Workstation: OOM Forensics of a 178 GiB CLI Memory Leak
The Freeze
Mid-afternoon, a production Threadripper-class workstation went dark. Console frozen, SSH dead on every interface, nothing but a hard reset on the table. It was the first-ever freeze on this particular platform — a recently rebuilt machine — and the instinctive first hypothesis was the obvious one: new board, novel failure, blame the silicon.
The first rule of incident response is that instinct is not evidence. Before forming any hypothesis, read the previous boot. Persistent journald means the machine's last minutes are sitting on disk waiting to be read, and they almost always tell you more than the console did.
What the Previous Boot Said
The journal from the crashed boot was unambiguous. At 13:19:29 the kernel invoked the OOM killer, and over the next sixteen seconds it logged one of the more lopsided memory pictures we have seen on any machine: a single process — an agentic coding CLI, driving a routine multi-node package upgrade over SSH — sitting at 195.7 GB of virtual memory and 178.5 GiB of anonymous resident set. On a machine with a quarter terabyte of RAM and, at the time, no swap.
The final kill line reads like a confession:
Out of memory: Killed process 212498 (agy) total-vm:195714000kB, anon-rss:178545396kB
Thirteen seconds later the oom_reaper confirmed the reclaim: 178 GiB of plain anonymous heap, released instantly the moment the process died. Not page cache, not shared memory, not a kernel leak — ordinary userspace allocation that had grown monotonically through a long multi-step agent run and was never freed. The CLI in question was Google's Antigravity agent (the process name in the kill line, agy, is its binary), current version, auto-updated that same morning.
Just as important is what the journal did not contain: no machine-check exceptions, no memory-controller errors, no panic, no lockup traces. Across the entire boot the hardware behaved impeccably. The platform was exonerated in ten minutes of log reading. The freeze was a software event wearing a hardware costume.
The Kernel Killed Fourteen Innocents First
Here is the part that generalises beyond one leaky process. Between the first OOM invocation and the final kill, the kernel terminated roughly fourteen other processes — log shippers, API servers, worker processes, monitoring agents — while the actual elephant kept allocating. The journal shows its resident set growing by another ~3 GiB during the kill storm itself.
This is not a kernel bug. It is oom_score_adj arithmetic doing exactly what it was told. Kubernetes assigns burstable pods adjustment scores near +1000, which makes them maximally attractive OOM victims by design. An interactive process launched from a desktop terminal session inherits a modest +200. When the kernel ranks victims, a tiny log shipper at adjustment +1000 outranks a 178 GiB monster at +200. So the machine ate its own services, one by one, in badness order, while the process actually responsible was near the back of the queue.
If you run agentic workloads alongside orchestrated services on the same host, this ordering is lying in wait for you. The kernel will sacrifice your infrastructure to protect your runaway agent.
Why a Leak Became a Freeze
An OOM kill, on its own, is a non-event. Process dies, memory returns, services restart, life goes on. The outage here was not the kill — it was the ten-plus minutes before the kill, when the machine was fully unresponsive.
That window is direct reclaim. With no swap configured, the kernel had no cushion: as free memory approached zero, every allocation on the machine — every process, every subsystem — stalled inside memory reclaim that could not actually reclaim anything meaningful, because the pressure was one giant unswappable anonymous heap. The console was not crashed; it was waiting its turn for pages that were never coming. A swapless machine converts a slow leak into a hard freeze, skipping the degraded-but-observable phase entirely.
The Four-Layer Guard
The forensics took about half an hour. The guards went in the same afternoon, in four layers, each independently sufficient to have prevented the freeze:
1. Cgroup caps on every service that runs an agent. Any systemd service that invokes an agentic CLI on a loop now carries MemoryHigh (throttle-and-reclaim) and MemoryMax (hard kill) limits. An unattended agent can no longer take more than its share.
2. Transient scopes for interactive agent sessions. Interactive invocations are wrapped in systemd-run user scopes with their own memory ceiling. One subtlety: the CLI self-updates by rewriting its own binary in place, so wrapping the binary is futile — we wrap the invocation instead, via a shell function, which survives updates.
3. earlyoom, with the victim ordering inverted. A userspace OOM daemon now acts at 5% available memory — before reclaim thrash sets in, while the machine is still responsive — with an explicit preference list that targets agent CLIs and browsers first and shields the control plane, container runtime, remote access, and model serving. What the kernel got backwards, policy now gets right.
4. Swap, added deliberately. A modest swapfile with low swappiness converts any future runaway from a freeze into a visible slowdown that monitoring can catch and a human (or an agent) can act on. On a Kubernetes node this has a strict ordering requirement: the kubelet must be made swap-tolerant first, or the next service restart fails. Pods still never touch swap; it exists purely as a cushion for the moment something else misbehaves.
Report Upstream, Not on the Timeline
A bug this severe, in a tool this new, deserved better than a screenshot thread. It turned out the vendor's issue tracker already had an open report of the same signature — monotonic memory growth during long agentic tasks — at roughly one-fifteenth the scale, unconfirmed and uncommented. Our incident was the strongest evidence yet that the bug class is real, severe, and present in the current release.
So we wrote it up properly: full kernel timeline, the victim-ordering analysis, environment details, and the complete mitigation set as a workaround for other Linux users, posted directly on the tracker issue, with an offer to reproduce under an instrumented cgroup. And a short note to the team that builds the product, pointing at the data.
The investigation, the guards, and the upstream report were executed end to end by HAL, our autonomous infrastructure agent, with operator review at the decision points. An agent diagnosing a runaway agent, then deploying the controls that keep its own kind honest, has a certain symmetry we appreciate.
The Lessons
Agentic CLIs are a genuinely new resource-hazard class. They run for hours, accumulate context and tool output across hundreds of steps, and — in at least one current mainstream implementation — never let go of any of it. Treat them the way you treat any untrusted workload: cap them in cgroups, prefer them as OOM victims, and never run them uncushioned on a swapless machine.
The kernel's OOM victim ordering was designed for a world where orchestrated pods are expendable and interactive processes are precious. On an AI workstation, that assumption is now exactly backwards.
And when you find the bug, send the vendor your forensics, not your frustration. It is better engineering, and it is a better introduction.