An Immune System for a GPU Cluster: Building a Reflex Arc for Autonomous Self-Healing

Brains With No Hands

The brief was to design a self-healing v2. Live investigation found four generations of self-healing already stacked in the cluster, and almost none of it closed the loop. Five primitive deterministic watchdogs were doing the only real automatic remediation. Above them sat three successively cleverer "brains," each built to supersede the last, none ever wired to a set of hands.

The picture was stark. One generation's triage queue held over a million unprocessed events with a delivery status of "never" — a firehose of log data pouring into a consumer that had delivered nothing, feeding an armed executor that had received zero remediation plans in 27 days. The next generation was a healthy LLM triage brain, green for weeks, but advisory-only by hard policy: its own prompt forbade it from claiming a state change, and its outcome enum did not even contain "fixed." The newest was a budget-capped agent, connected to the bus and idle.

Maximum architectural sophistication; near-zero closed-loop coverage. The diagnosis wrote itself: do not build a fifth brain. Name the shape that connects a brain to a hand.

The Reflex Arc

Biology already had the shape, and the cluster was already using its vocabulary (it spoke of macrophages and antibodies). A reflex arc has three parts and a feedback loop: afferent sensing, an interneuron decision, and efferent action, with verification feeding learning.

• Afferent. Metrics and alerts, a log pipeline, and the existing probe suite all normalised onto one event bus.
• Interneuron. The decision tier emits a single typed verdict: a class (reflex / deliberative / page), a confidence, an antibody identifier, a blast radius, and a proposed action.
• Efferent. *One* executor with two tiers. A deterministic reflex tier runs signed, pre-validated playbooks with no LLM in the loop, in milliseconds to seconds, for known failure classes. A deliberative tier hands genuinely novel, low-confidence, or no-antibody incidents to the budget-capped agent.

The only genuinely new code is the connective tissue — an "efferent bridge" — plus one safety envelope made uniform across both tiers: allowlist (not blocklist) for auto-action; idempotency and preconditions; rollback that has been dry-run tested; rate limits and circuit breakers; blast-radius caps; a shadow → suggest → canary → auto promotion ladder so that no remediator is ever *born* at "auto"; one reversible, attributable audit trail; and a hard carve-out so that the most critical placement stays in advisory mode until proven for weeks. The envelope was built as pure, unit-testable functions, green and lint-clean, with an explicit test proving a critical-tier target *pages* a human instead of acting.

The Honest Reversal

A first pass concluded the missing piece was the efferent bridge. A deeper second-pass review overturned that. The blocker was never the architecture. It was signal quality — and the evidence was two bugs that would have quietly defeated the safety envelope.

The confidence gate was calibrating on a constant. The safety envelope auto-executes only above a confidence threshold. But 97% of incident rows — 7,588 of 7,825 — carried a NULL runtime confidence (legacy rows from before a schema migration), and the shadow harness was silently substituting a literal 0.5 for every one of them. The "calibrated" gate was, in practice, comparing against a hardcoded number.

Deduplication was leaking roughly sevenfold — through characters you cannot see. 112,974 raw sightings collapsed to 7,825 incidents but produced only 1,056 truly distinct signatures, 86.5% redundant. Part of the cause was a dedup window that was too short. The vivid part: the model that generated incident signatures was injecting *zero-width and invisible Unicode* inside its tokens, which fragmented a single image-pull fault across more than seventy distinct "signatures." One fault, seventy fingerprints, none of them matching, because the difference between them was glyphs with no width.

The fixes were unglamorous and decisive. NULL confidence was remapped to 0.0 with a separate "calibrated" flag — never the silent 0.5 — so a strict less-than gate means an uncalibrated row can rise to "suggest" but can never auto-execute. Signatures are now Unicode-normalised with invisible characters stripped, guarded by an anti-merge test that proves two genuinely distinct faults never collapse into one. The dedup window was widened.

The Disciplined Landing

We shipped a live but tightly gated "suggest" loop for the two *non-mutating* diagnostic classes only; every mutating reflex stayed in shadow. Suggest mode uses recording fakes for the executor and asserts they stay empty, so it cannot dispatch a real plan or spend a cent — the live run reported zero dispatch attempts. An adversarial multi-agent review across three dimensions raised eleven findings, confirmed and fixed the three real ones, and independently verified each of the six critical safety invariants, finding zero violations.

The closing lesson came from the data. The *only* mutating actions the system ever wanted to take across the entire shadow corpus were all the same false positive: flagging a node as "offline 26 hours" when that node had been deliberately cordoned for maintenance. An immune system that does not know what is *intentionally* switched off will attack healthy tissue. Teaching it the difference between "down" and "off by design" is the prerequisite for the first mutating canary — and that, not the architecture, is the next piece of work.