Building One Operator Cockpit for a Sovereign GPU Fleet

One Pane, Many Signals

Running bare-metal GPU infrastructure means watching a lot of things that don't naturally live in one place: GPU box health and thermals, a Kubernetes control plane and its workloads, a distributed storage cluster, a metrics stack, and out-of-band power through baseboard controllers. Operators were context-switching across half a dozen tools to answer a single question. So we built a unified operator cockpit — a Rust service backend serving a modern reactive web UI — that pulls all of those signals into one view.

The interesting engineering isn't the dashboard itself. It's what it took to harden that cockpit to the point where it could be exposed without handing anyone the keys to the fleet. (Note: the artefact described here is a point-in-time engineering snapshot, not a current product offering.)

Open-Sourcing as a Forcing Function

Preparing an internal tool for a wider audience is a discipline that surfaces every lazy assumption you baked in. Three pieces of work did the heavy lifting:

Config-driven topology. The original tool had a hard-coded node map and embedded operational assumptions about the specific fleet. Every one of those had to become configuration: the infrastructure topology is generated from a config file, the network-diagnostics plan is loaded from config, the host thermal monitor reads its identity from the environment rather than a baked-in hostname. A clean export was shipped as a single fresh commit rather than republishing internal history, precisely because the tracked config carried real operational topology that had no business being public.

Read-only by default, with gates around every dangerous capability. Nine distinct dangerous capabilities — a remote terminal, container exec, cluster writes, host service writes, power control, process termination, network diagnostics, tool integration, and alert silencing — were each put behind an explicit feature gate, *off by default*. A fresh clone is safe to run and observe; a single destructive action requires deliberately opting in. This is the right posture for any operations tool that might run in front of more than one set of eyes.

Secret-scanning in CI. Release verification runs the full test suite locked, a frontend type-check, a production build, a public-safety scan, a dedicated secret-scanner (clean), and a dependency audit — so the moment an operational secret tries to slip into the public surface, the build fails. (One honest detail from the hardening: the first refactor broke two API tests because their fixtures still expected the real node count; they were replaced with generic fixtures — a small reminder that "make it config-driven" includes the test data.)

Adding Power With a Safety Net

Later iterations added per-node power controls — on, off, reboot — driven through baseboard controllers with an SSH fallback. Crucially, these sit behind the power feature flag, behind session authentication, and behind a *storage-aware* confirmation dialog: powering a storage-tier node escalates to an explicit confirm that understands the redundancy implications, while non-storage nodes get a plain confirmation. The control was extracted into a single shared component used in two places, so the two call sites can never drift apart in their safety behaviour. Surfacing a destructive capability is easy; the engineering is in making it *hard to do by accident*.

Making the Cluster Legible

A recurring problem for any operator new to a cluster is that the workload names are opaque and the scheduling is invisible. We added a service catalogue describing the fleet's workloads in plain language, and a placement panel on each pod that explains, in availability-aware terms, *why* a pod sits where it does. The panel encodes a real Kubernetes truth most dashboards gloss over: soft (preferred) affinity is ignored during execution, so a pod that landed on the wrong tier will not migrate back on its own. Rather than colour a border amber and leave the operator guessing, the UI says so explicitly and points to restart or evict. A good operations tool doesn't just show state; it teaches the system's actual semantics.

Shrinking to the Phone Without Breaking the Desktop

The final chapter was a responsive overhaul, and the discipline there was a Playwright visual-regression harness across five device profiles (desktop down to a 360px Android). The work grew the test suite from 11 to 25 and held the desktop snapshot diff at exactly zero — the whole point of visual-regression testing is to change the small screen without silently regressing the large one. The headline proof: a control-plane matrix that was over 2,400 pixels wide, an unreadable horizontal crush, became usable on a 390px phone by pinning its first column sticky. And the mobile pass flushed out a latent bug that had nothing to do with screen size: the embedded terminal had never sent its own dimensions to the backend pseudo-terminal, so line-wrapping was silently wrong at any non-default size — found only because someone finally drove it at phone viewports. The cluster ends up installable as a phone app; the desktop never noticed.

The through-line across all of it: an operations cockpit for serious infrastructure earns trust by being read-only until you ask otherwise, by teaching the system's real behaviour instead of hiding it, and by proving — with snapshots and with power-offs — that it does what it claims.