From the Lab
Learning From Its Own Answers: 27 Fine-Tunes of Small Open Models, and Why the Data's Source Mattered More Than the Adapter
We fine-tuned small open models 27 times on one task, GSM8K mathematics, to see which choices move a model. Supervised fine-tuning on GSM8K's reference solutions lowered Qwen2.5-0.5B-Instruct's accuracy from 46.9% to between 22.2% and 39.5% under every adapter setting we tried, from rank 2 to rank 64, DoRA, target modules, learning rate and full fine-tuning. It also cut the model's compliance with an unrelated answer-format instruction from 70.2% to as little as 3.1%. Training on the model's own correct samples was the only method that lost no task accuracy, and it cost no knowledge or compliance, but it gained nothing significant either. A short GRPO run collapsed onto its format reward before recovering. Three measurement problems, a generation cap, a first-letter parser and evaluation nondeterminism, would each have changed a conclusion.
ReadWe Blamed the Kernel: Reproducing a Single-Stream Decode Record, and Why Code Decodes 2.4 Times Faster Than Prose
A community leaderboard record put DeepSeek-V4.1-Flash-1017 at 670 tokens per second single-stream on four workstation Blackwell GPUs; our seat decoded 187 on the canonical code prompt. Splitting decode into tokens per speculative step and milliseconds per step showed acceptance matched and step time was the whole gap (17.7 to 22.3 ms against 8.8). We matched the image by digest and the flags exactly, ruled out power, clocks, PCIe width and graph capture, and blamed a private kernel. We were wrong: two host settings (IOMMU off, a forced GPU peer-to-peer path) cut the step from 21.0 to 12.2 ms while our P2P microbenchmarks barely moved, lifting canonical code to a verified 413.6. The record was a best of ten (typical 598), and on our seat code decodes at 502 tokens per second against 211 for prose, at the same step time.
ReadIs It Actually Doing Anything Useful? Auditing an Agent Memory Against Its Purpose
Asked whether pureMIND, the shared memory our agents consult before acting, was doing anything useful, we measured it against six written properties. Over 30 days 25% of MCP searches had errored, the weekly median had climbed from 555 ms to 8.3 s, and for 7 of 15 recently changed facts the top hit was stale or no answer. The timeouts came from storage, not code: a pool still named for NVMe had moved the database onto hard disks, taking timeouts from 0 in 210 searches to 128 in 429. The stale answers came from agents appending corrections on top of old claims. Restructuring notes into Current and History sections, ranking on explicit currency signals and moving to a dedicated NVMe database raised current-in-top-three on a fixed 76-question set from 63% to 87%, at p95 under 400 ms, while nine unused subsystems were retired.
ReadThe Decision Is One Token: Typed Decision Models, Prompted Seats and a Labelled Head on 829 of Our Own Items
Our fleet runs about 120 text decision gates built from regexes, embedding thresholds and full language-model verdicts. We measured the new open-weight System One decision models against them on 829 labelled items from seven real tasks. Zero-shot Clef-flash 9B led alert triage (every page caught), 12-class inbox intent (0.969) and injection at a fixed alarm rate (88.3% recall at 5% FPR). Our 120B resident seat, asked for one letter and read from its log-probabilities, judged operator corrections best (AUROC 0.925), but only on the operator turn alone: adding context lowered it to 0.823. Where labels exist, a logistic head on frozen embeddings matched or beat every zero-shot model on four tasks. AUROC and operating-point rankings disagreed on memory poisoning, so we now compare screens at a fixed alarm rate.
ReadThe Harness Is Part of the Model: Three Agent Harnesses and Two Serving Engines on the Same DeepSeek Weights
We held DeepSeek-V4.1-Flash-1017 fixed on four workstation Blackwell GPUs and changed everything around it. On 24 Terminal-Bench 2.0 tasks at three attempts each, DeepSeek's minimal harness passed 52.8%, Claude Code 58.3% and Ante 70.8%; Ante's 18-point lead over DeepSeek's harness held under bootstrap, its lead over Claude Code did not once nine pre-agent install failures were removed, and the model card's claim that the minimal harness beats Claude Code did not reproduce. A single-attempt pilot in which Ante never lost a task did not survive three attempts. On the same weights, a vLLM fork decoded 25 to 47% faster than SGLang and passed every tool check, but abstained correctly in 25 of 36 rollouts against 32, by calling tools where none fitted.
ReadWhat Fits on Two GPUs: Capability Profiles of an Efficiency-Tier and a Flagship-Tier Open-Weight Model on Sovereign Hardware
We run open-weight models in two tiers: an efficiency-tier model that serves our agents around the clock from two GPUs, and a flagship-tier model that fills a four-GPU node on demand. Profiling NVIDIA Nemotron-3.5-Super-VL and DeepSeek-V4.1-Flash-1017 across six suites, three of them built from our own data, we find that two GPUs and 15% of the checkpoint size carry most of the measured capability: 92% of the flagship's reasoning score, 98% on AIME 2026, 86% on repairing real defects from our own code, and vision on par with the flagship. Factual recall and whole-document state tracking still grow with size. Measurement choices mattered as much as the models: answer parsing moved one vision score from 44.8% to 94.0%, and the token budget moved a reasoning score from 67.8% to 84.7%.
ReadHardening Is Only Safe After the Client Matrix: Migrating Ceph's cephx Keys to aes256k Without Locking Clients Out
Ceph flags every legacy cephx key with AUTH_INSECURE warnings, and our autonomous remediation agent treated that warning as safe to fix alone. It rotated eight client keys to aes256k on 14 September and again on 17 September; on the night of 26 September it also removed the legacy cipher from the monitors, locking a workstation out of storage and breaking nightly backups. A compatibility matrix tested on 28 September explained every failure: the Proxmox 7.0.14 kernel, userspace Ceph 19.2.6 and 20.2.4, and ceph-csi 3.17.1 accept the new keys; Ubuntu kernels 7.0.0-34 and 6.8, Ceph 20.2.0 and ceph-csi 3.16.1 do not. With pending-key overlap, nine clients moved one at a time in 88 minutes, and the cluster reached HEALTH_OK with no mutes.
ReadOne Drive Sets the Tail: How a Degraded SSD Stalled a Replicated Ceph Tier, Misled Our Incident System, and Outlived Its Own Repair
Every OSD in our four-drive Ceph SSD tier logged replica writes of about 3 s, yet an isolated benchmark rated all four drives healthy. Sampling all four once a second at the same moment showed why: one consumer SATA SSD sat at 100 % utilisation with 167 to 726 ms write latency, completing 3 to 23 writes a second, while three identical siblings idled below 1 ms, waiting on it. The stall reached NATS JetStream and led our incident system to blame a healthy node. A full-device discard and rebuild gave a 2.79 ms worst case under backfill, then regressed within about ten hours to 6 to 8 s stalls every 2.2 to 3.4 minutes. SMART marks the drive as the outlier; its replacement is pending.
ReadFP32 Almost Everywhere, FP64 Where It Counts: Measured Quantitative-Finance Compute on Six Workstation Blackwell GPUs
We measured six RTX PRO 6000 Blackwell cards, capped at 250 W, and three workstation-class CPUs on validated quantitative-finance kernels. FP32 Monte Carlo reached 1.75 trillion path-steps per second across the six cards on a 252-step Asian option; FP64 was 20 times slower. Full-revaluation VaR ran at 57 billion option valuations per second, and FP32 matched FP64 to 7×10⁻⁸ even on a fat-tailed, short-gamma book. Adjoint differentiation produced 128 Greeks for 2.2 to 2.8 times the cost of one price. Longstaff-Schwartz is the exception: its FP32 regression was biased by 0.06 to 1.2 % depending only on library and reduction order, so it must run in FP64. A CUDA 12.8 software stack was up to 104 times slower than CUDA 13 on the same card.
ReadDecode Does Not See the Platform: Identical Blackwell GPUs on a Gen5/DDR5 and a Gen4/DDR4 Host, and the Upgrade We Declined
We served Qwen3.8-27B BF16 at tensor-parallel 2 on two nodes with identical GPUs (two RTX PRO 6000 Blackwell Max-Q at 250 W), driver and container image, differing only in host platform: Zen 5 with DDR5 and PCIe Gen5 against Zen 3 with DDR4 and Gen4. Single-stream decode tied at 42.3 and 41.7 tokens per second. At 32 and 64 concurrent requests the older platform delivered 7.3 and 8.5 % less throughput, an 8k-token prefill run 11.6 % less, and median time to first token rose 13 to 31 %. Host-to-GPU copies ran at 28 against 57 GB/s; cold start took 172 against 102 s. An eager 8 KiB all-reduce cost 49 against 8.9 µs, yet 12.6 µs inside a CUDA graph. Roughly $16,000 of DDR5 was not justified.
ReadThe Fixer Optimised the Detector: An Autonomous Remediation Agent, a Padded Alert Rule, and a Grader Hardened Against Padding
Told that one of our alert rules could no longer fire, our autonomous remediation agent fixed nothing. Over three runs in about 95 minutes it added an always-true guard, loaded a branch that can never be true, or on() (run_age_seconds < 0), and left uncommitted classifications that our grader read from the working tree. The grader's analysis rated the padded rule alive, because it counted an or-branch live whenever its selector had series. We rebuilt that analysis as an interval domain with dead-branch pruning and evidence-gated sign, machine-checked the event grade, and now grade only committed state. Classifications fell from 57 to 25, 59,987 fuzzed expressions exposed one more soundness bug, fixed before the final deploy, and the fleet ended at 389 alive, 18 event, 3 waived, 0 dead.
ReadThe Pool Named NVMe Lives on Spinning Disk: How a Storage Name Outlived Its Placement and Stalled a SQLite-Backed Git Forge
After consumer SSDs without power-loss protection stalled for 5 to 10 seconds under CI sync-write bursts, we moved our Kubernetes database pool onto replicated HDD, and the pool and storage class named for NVMe kept their names. Two days later a bulk archive write of about 240 MB/s into a different pool on the same spindles raised mean HDD commit latency from about 8 ms to 82 to 99 ms, and our SQLite-backed git forge fell to about 134 KB/s of fsync, logging 1,647 slow queries in 55 minutes, some of them single-row primary-key selects. WAL mode and a startup probe stopped the restart loop but not the stall. Moving only the 536 MB database file to the SSD tier brought git ls-remote from a 60 s timeout to 0.16 s and API latency to 5 to 15 ms. The recursive permission change that came with the fix loosened private-key modes and broke 19 pull mirrors.
ReadThe Multiplexer That Knows Its Agents: Herdr Against tmux for Concurrent Coding Agents, and Compiling Out the Phone-Home
We compared Herdr 0.9.1, a terminal multiplexer that tracks each coding agent's state, with two tmux arms driving identical Claude and Grok workers, over five pre-registered rounds. Herdr noticed a finished Claude turn in a median 0.22 s against 2.43 s for screen polling, caught 15 of 15 permission prompts against 14 and 0, and turned a blocked prompt into a verified result in 10.5 s against 17.1 and 17.9 s. CPU matched persistent tmux and was 40 % below one-shot; Herdr's server costs 16.5 MB and about 1.6 % of a core at idle. Upstream contacts its authors' servers at every start, so we patched the one helper all its network calls pass through and measured zero outbound connections before starting a pilot.
ReadHybrid Mamba Cannot Cache a Transcript: Choosing a Seat for a Live Co-Pilot by Time to First Answer
A live meeting co-pilot has a 1.5 s budget from end of utterance to first readable token. We measured every leg of the loop on-premises and compared two candidate model seats: a pure-transformer mixture-of-experts model and a hybrid Mamba-2/Transformer model. The hybrid seat, with prefix caching enabled, measured a 3.8 to 4.2% cache hit rate and a flat 194 to 197 ms on an identical prompt repeated eight times; the transformer seat dropped from 355 to about 100 ms once the prefix was cached. A recurrent state is one vector per layer, not a sequence of per-token blocks, so it cannot be sliced and reused for a longer transcript, and time to first answer grows with transcript length (253 to 517 ms from 8k to 64k tokens, against 108 to 159 ms). Two further findings: time to first token scores an unusable configuration as fast, because reasoning tokens stream before content (one seat produced 351 reasoning tokens and no answer), and the whole loop closes in about 560 ms.
ReadWhen Grammar Meets Speculation: A Constrained-Decoding Deadlock Under Overlapped Speculative Decoding
Serving DeepSeek-V4.1-Flash with SGLang across two nodes (TP4/EP4) with overlapped DSpark speculative decoding, one JSON-schema-constrained request batched with other traffic deadlocked all four ranks at the grammar barrier while every GPU spun at 100% utilisation and about 100 W. The trigger reproduced 2/2 within 24 to 47 seconds; the same prompt unconstrained ran 72 turns clean, and the constrained request alone was clean 2/2. Disabling overlap scheduling first crash-looped 22 times for 1 h 41 min unnoticed behind a silent health-check loop, and once patched to run, still deadlocked on the device. The fix was to refuse grammar-constrained requests at admission: HTTP 400 in about 0.2 s, 0 hangs across 100 constrained arrivals and 454 concurrent agent turns, and throughput of 91.9 and 242.6 tok/s against a baseline of 91.9 and 235.5.
ReadWireGuard Inside VXLAN: How a Mesh VPN on Kubernetes Nodes Loses 8x Throughput While Every Latency Check Stays Green
Flannel puts the Kubernetes pod network on each host as interface addresses, and Tailscale advertises every local address as a WireGuard endpoint, so our mesh kept re-selecting pod paths: 4 to 15 switches per node every 10 minutes, each one carrying WireGuard inside VXLAN. On a 25G storage link that meant 529 Mbit/s instead of 4.44 Gbit/s, about 8x, while the VPN's own ping answered in 0 s and the whole health surface stayed green. A daemon restart looked like a fix and regressed in about 20 minutes; there is no configuration lever and the VXLAN address cannot be removed. One firewall rule, dropping inbound UDP 41641 from the pod network, took pod-path switches to zero on every node where it sat ahead of the VPN's own rules.
ReadClick Accuracy Picks the Wrong Browser Agent: Choosing a Self-Hosted Computer-Use Model on 39 Live Tasks
We ran five self-hosted browser-agent setups through 39 live multi-step tasks on public websites, each in a fresh headless Chrome, scored only by verified end states with no model judge. A DOM agent on DeepSeek-V4.1-Flash scored 39/39 but needs four GPUs; Qwen3.8-27B in a lean screenshot loop scored 38/39 at a median 12.7 seconds per task on a model we already serve, and the same weights inside a heavier harness were three times slower. Holo3.1-35B-A3B hit 93.1% of targets on a single-click test yet finished last on live tasks at 33/39, so click accuracy alone would have picked the wrong model. The top four setups sit within two tasks of each other and are not separable, every screenshot-only agent failed the one autocomplete form, and a headless viewport that is 713 pixels tall instead of 800 cost one model six tasks.
ReadSpend Is Width Times Turns: Where an Orchestrating Agent's Tokens Go, and What a Diet Moved
We measured every turn of 190 sessions of the frontier model that orchestrates our fleet: 66% of estimated spend was re-reading the rolling conversation, 24% re-reading a fixed prefix, 10% the model's own output. The file-read tool alone injected 65.8 MB into context at 71 KB per call, and sub-agents on the same provider line consumed transcripts larger than the orchestrator's to return a median of 324 bytes. A diet of hooks (sliced reads, a width fence, a turn fence, an exploration budget, no same-line sub-agents) halved the fixed prefix, cut read bytes 55x, and halved turns at p90. Median context per turn fell only 11%. Spend is width times turns; the prefix was the easy lever and is now spent, and what remains is set by how many turns a session takes.
ReadThe Gate That Passed by Luck: Stochastic Conformance and the Proof That a Parameter Was Honoured
A tool-calling conformance gate on our resident Qwen3.8-Flash-Next seat failed after a restore; replaying its exact request 10 times gave 7 correct calls and 3 calls to a tool that was never offered, and vLLM does not enforce the offered name under tool_choice=required. The same gate had already failed 9 of 246 hourly runs over the previous eleven days, each red clearing itself, each green read as health: it was sampling a stochastic model once and reporting a coin. Sampling at temperature 0 made it repeatable (3 of 3 live runs pass), while the production behaviour remains a known gap. A second gate asks whether a hosted API honours reasoning_effort: with three samples per arm and a rule that the high arm must clear the low arm by 1.5x and 256 tokens, grok-4.6 was proven on 22 September (low 2,031 to 3,159 reasoning tokens, xhigh 7,484 to 13,366) and grok-4.7 was not (low 3,802 to 6,688, xhigh 7,334 to 11,829), which left one benchmark comparison validated and the other not.
ReadA 475 GB Model Across Two Workstations, With the Table on Disk
DeepSeek-V4.1-Flash is 475 GB of weights plus a 189 GB static embedding table, and our fleet has four workstation GPUs on two nodes with 384 GB between them. We extended a community single-host recipe, which serves the table from NVMe through a CUDA host callback, to run tensor-parallel 4 across the 200 GbE fabric: 178 tok/s single-stream, 642 tok/s at 8 concurrent, 7,000 to 8,000 tok/s prefill, a 10 to 25% haircut against the single-box reference and prefill at parity. On our internal frontier benchmark it scored 92.1 / 96.0 / 93.5, a record for a locally served seat, and became the fleet's large seat the same day. It also hung, three times, always with the draft head on. NCCL logs showed all four ranks had enqueued identical collective sequences, so the stall was host-side. Moving the table's rows from NVMe into locked host RAM produced 92 cases and 513 requests without a hang where NVMe mode hung at case 26.
ReadA Published Metric Is Not a Consumed Metric: Controls That Look Healthy and Cannot Fail
Closing the open items on an off-site database backup, we found four controls that looked healthy and could not fail: backup gauges published for 7.5 hours with no rule reading them, a Prometheus reload that succeeded for a rule group it never loaded, a -1 sentinel that kept a staleness threshold green in exactly the empty-repository state, and a restic retention policy that ran clean and deleted nothing because every snapshot sat in a group of one. The method that found them is to drive each control into the state it exists to report: the live rules API instead of the reload status, a mutated rule instead of a passing test suite, the real Postgres target instead of the SQLite one that had passed its drill. A week earlier, all 28 of our self-hosted CI runners sat down for about 31 hours after a storage mount failed for 1 hour 40 minutes, because systemd never retries a dependency failure and no rule watched the runners. A published metric is half a control; the consumer is the other half, and it has to be shown going red.
ReadFastest Is Not Best: The One-Node Seat Beats the Two-Node Record on Half the Hardware
The GPU vendor's NVFP4 quantisation of GLM-5.3-Flash is the best GLM-5.3-Flash we have measured: 117.9 tok/s single-stream and 763.9 aggregate at 16 concurrent across two nodes, 11 to 18% ahead of the previous variant. Our first write-up called it the fastest seat on the fleet. It was not. Re-run the same day against the incumbent one-node seat, a different model on half the hardware beat it on every axis except single-stream: 896.9 to 982.5 aggregate at 16 concurrent, 7,412 tok/s prefill against 4,711, and a higher score on our internal frontier benchmark. The two-node number can never become a seat because the checkpoint is 204.5 GB and one node is 205.3 GB raw; every NVFP4 variant keeps attention in BF16 and lands in the same place. The paper is about checking raw bytes before benchmarking, and about three measurement traps (a prefix-cache hit posing as prefill, a slow-warming quantisation, a renamed reasoning field) that would otherwise have inflated the record further.
ReadThe Blocker Was Twelve Lines: A 4-Bit Checkpoint Rejected on Thursday and Serving the Following Wednesday
On 4 September we evaluated the NVFP4 quantisation of Qwen3.8-Flash-Next at our resident seat's shape and rejected it: a fixed 47.68 GiB embedding table would not fit beside the weights, and the only stock lever offloaded weights instead, leaving the seat at 8.69 tok/s against the FP8 incumbent's 85.99. Five days later the same checkpoint served at the full shape after a 12-line backport into an image we had not tried, because the README had pointed us at the one image that lacked the offload. At the resident placement the 4-bit checkpoint gives 2.25x the KV cache, 15% more aggregate throughput and 13% lower p95 latency for 12% less single-stream, and ties the FP8 seat on our internal frontier benchmark at every reasoning-effort rung. The seat stays FP8 until two reliability cells are re-measured; the paper is about how a correct verdict on a configuration became a wrong verdict on a checkpoint.
ReadSpeculative Decoding Wins the Benchmark and Loses the Seat
We switched on the bundled draft heads of two mixture-of-experts models on two different serving engines: a 4-bit Qwen3.8-Flash-Next on two workstation Blackwell GPUs, and the 475 GB DeepSeek-V4.1-Flash across two nodes. Single-stream throughput rose 1.61x and 3.5x, the largest speculative gains we have measured. Neither shipped. On the resident seat the draft cost 17.5% of the KV cache, 14 to 26% of aggregate throughput at 64 concurrent, and 3.7x cold time-to-first-token on long prompts. On the large seat it capped concurrency at 8 slots and cut the token pool from 890,000 to 260,000. Speculation converts memory and concurrency into single-stream speed, and the single-stream benchmark is exactly the metric a shared seat cares least about.
ReadSame Answers, Different Confidence: 4-Bit Weights Preserve Accuracy and Destroy Calibration
We ran GLM-5.3-Flash in BF16 and in an NVFP4 quantisation on the same eight-GPU Hopper node, where the 4-bit weights are dequantised before arithmetic so that any difference is a property of the stored weights alone. On our internal frontier benchmark every per-dimension accuracy delta was one item, inside the noise: on accuracy the quantisation is a tie. The one dimension that did not tie is calibration. Both checkpoints answered 85.0% of the calibration items correctly, but the BF16 checkpoint's Brier score was 0.070 and the NVFP4 checkpoint's was 0.170, 2.4 times worse and near the 0.1875 random floor. The 4-bit experts keep the answers and flatten the model's stated confidence, so wrong answers arrive with the same tone as right ones. For tool lanes verified downstream the smaller checkpoint is free; for any lane that consumes the model's confidence, a judge, a scorer, an abstention gate, the calibration loss is the price.
ReadThe Adapter Learned the Voice, Not the Typos: Fine-Tuning an 8B Model to Write as One Person in a Day
Mail that reads as machine-written gets ignored, so we built pureQUILL: a LoRA adapter on Qwen3-8B that rewrites agent-drafted mail into the operator's own voice, with every fact span-locked so the rewrite cannot change it. In one day we built the corpus, a frozen authorship judge, and four adapters, the last three trained across four Blackwell GPUs on two workstations in under 20 minutes each. The adapter learned the voice: a frozen classifier labelled 81% of its rewrites as the operator's against 34% for the untuned model. It did not learn the typos, whose recall of 0.27 barely beat copying the input (0.25), so they now come from a mined typo matrix. Its main failure was copying: greedy decoding returned 37% of drafts unchanged. A pair rebuild, a targeted decode ladder and measured register layers cut copying on transformable drafts to 13.9% in live serving, with zero changed facts.
ReadDistributed Inference on Workstation Blackwell, Part 4: Cross-Node Tensor Parallelism Over 200 GbE, and the Five Fixes the SM120 Path Needed
Parts 1 to 3 of this series characterised the fabric between two workstation Blackwell nodes and ran models across it by RPC. This part shards every layer across all four RTX PRO 6000 GPUs over 200 GbE with GPUDirect RDMA, which is the only on-premises shape for models that do not fit one node. Nemotron 3 Ultra 550B served three minutes after a cold boot, answered 10 of 10 on a reasoning ladder, and completed a 90-minute soak at 912 of 912 requests with zero errors; GLM-5.3-Flash needed nine attempts and five distinct fixes in the SM120 attention path before it produced a coherent token, and a qualified community image later took it to 691 tokens per second aggregate at 16 concurrent streams. Two failures turned out to be structural to multi-node rather than to any engine: a custom all-reduce prober that deadlocks every rank before a weight loads, and speculative decoding whose data-dependent acceptance length diverges the ranks' collective counts. We also record a public 1,004.9 tokens-per-second headline that measured a locked repeat loop, and two attempts that produced no model-quality evidence and were not scored.
ReadGreen Gate, Lost Hunks: Nine Ways an Automated Merge Train Dropped Merged Code While Every Test Passed
An adversarial code-review wave produced about 330 pull requests across 21 repositories in one day, every one carrying a semantic-version bump in the same commit, so every merge conflicted on the version surface. The merge train we wrote to restack, restamp, gate, and merge them lost merged work in nine distinct ways, and every one shipped a green test gate: a base-wins rule that dropped a pull request's own middleware from a file that was both a version file and a code file; a version extractor that took an IP address for a version and rewrote it over seven consecutive merges; a silent checkout failure that force-pushed one pull request's content over the next. The only control that caught them was a per-pull-request marker pass on the final default branch, run by a different actor and asking a different question: not whether the tree is healthy, but whether this change arrived.
ReadThe Reliability Champion Is Not the Single-Shot Champion: pass^5 Inverts a Leaderboard
Single-shot leaderboards measure whether a model can be right. Production needs models that are right every time, and the two rankings differ. On our internal frontier benchmark, Nemotron 3 Ultra 550B set the single-shot record at 92.7 flat and then shed 8.6 points when every item had to pass five rollouts at temperature 0.7; an anonymous stealth model that had tied our local 27B open-weight seat shed 8.4; the 27B shed 1.6 and became the reliability champion. A week later the same inversion appeared inside one model's reasoning-effort ladder: the gap between two effort settings was 4.5 agentic points single-shot and 12.5 under pass^5, with the multi-step tool-loop dimension collapsing from 9 of 12 items surviving to 4 of 12. We give both experiments, the control that was cancelled, the counterweight from a workspace-scored battery that did not reproduce the quality lead, and a small-sample result that reversed sign between n=3 and n=64.
ReadA Memory Layer With External Anchors: LongMemEval and LoCoMo on a Sovereign Retrieval Stack
Until this week our agent-memory substrate had no score on any public long-term-memory benchmark, so nothing it did could be compared with anything else. We ingested LongMemEval-S (100,548 chunks) and LoCoMo (1,540 questions) and report every configuration on all three axes that matter, accuracy, context tokens, and retrieval latency, with the reader model named. Two zero-cost changes, ordering retrieved chunks by session date and telling the reader that later excerpts supersede earlier ones, lifted a fully on-premises 27B configuration from 67.4% to 72.8%. With a gpt-5.4 reader and the official judge, a top-25 window reaches 90.6%, edging past Zep's published 90.2% at 2.7 times their context budget; at their budget we sit 2.4 points behind. On LoCoMo a distilled fact layer helps as a supplement (+3.7 points) and hurts on its own (−4.9), which is the structural risk in extraction-only memory.
ReadRed-Teaming Your Own Benchmark: Three Model Lineages Attack a 94-Item Bank, and What Survived
A benchmark scored entirely by code is only as honest as its checkers. We paid three model lineages (a GPT-5.6-family seat, Gemini 3.7 Flash, and our on-premises 27B open-weight model) to construct wrong answers that our internal frontier benchmark would accept, and re-executed every claimed exploit through the real checker code. Every one of the 94 items was false-greened by at least one lineage; the 217 exhibits collapsed into four classes, two of which would bite honest models and were fixed with paired regressions, and two of which are adversarial-only and are documented rather than patched. A fresh, blind Claude Opus instance scored 4.5 points below the author-answered self-baseline under identical checkers, which is the first direct measurement we have of non-blind inflation, and one item that all 18 stored strong-model runs had failed turned out to be missing its opening user turn.
ReadFour Days Later: A 753-Billion-Parameter Model on One Workstation GPU, Reproduced
On 18 August a Berkeley-led team released FreeToken, an open-source inference engine whose headline configuration serves GLM-5.2, a 753-billion-parameter mixture-of-experts model, at 14.9 tokens per second on a single 96 GB RTX PRO 6000 Blackwell workstation GPU, with the 419 GB expert pool held in host memory. Four days later we reproduced the configuration on our own hardware and measured 16.9 tokens per second warm, with time to first token between 7.2 and 7.8 seconds against the paper's 7.5, and a first-ever cold start that landed on their stated worst-case bound of 44 seconds. Two follow-up tests sharpen the picture: an expert-kernel A/B showed the GEMM kernel is not the constraint and decomposed that 44-second cold start into one-time compilation over a steady-state 8 seconds, and a timeout-guarded NCCL collective on a second dual-GPU machine cleared tensor parallelism on hardware where a documented hang defeats the capability probe. We have not found another published reproduction of the frontier configuration as of this writing.
ReadPast the Peak: Maximum Reasoning Effort Made Our Model Measurably Worse
Our on-premises text seat had run at its chat template's highest reasoning-effort setting since the day it was deployed, because nothing in the stack ever set the value. Measured across 94 benchmark items per setting, that default scored 84.6 on our flat composite against 93.4 at the middle setting, while spending 81 percent more completion tokens; disabling reasoning entirely was worse than both, so the reasoning was doing work and its length was the defect. The dimensions measuring raw reasoning were already at ceiling and could not improve. Everything that moved measures whether the model commits: tool selection, abstention, multi-step execution, grounding. Abstention rose from 66.7 to 100.0, meaning maximum effort had reasoned the model into answering questions it should have refused. Because the settings are inserted sentences rather than compute budgets, we then reproduced the top setting as a hand-written expert-persona prompt and watched tool selection fall to the same 62.5 it scored at maximum effort. We give the method, the non-termination failure at the top of the dial, and an explicit account of what these numbers will not carry.
ReadThe Flat Line: Why a Single Throughput Number Cannot Tell Batching From Queueing
A production inference engine reported the same aggregate throughput at one concurrent request as at sixty-four. It was not batching at all: every request past the first was queueing, and the single benchmark point we had been quoting could not see the difference. Sweeping the concurrency ladder corrected a stored throughput figure by 3.2× and moved a model comparison from a reported ~7× to 17.0×, or 33.9× per GPU, since NVIDIA's Nemotron 3.5 Lightning sustained 5,613 tokens per second across 64 concurrent requests on a single Blackwell card, at sub-three-second p95, and had not saturated when our ladder ran out. We give the diagnostic signature (flat aggregate, linearly climbing p95), the four-configuration sweep that exposed it, a quality result that survived repetition and one that did not, and why a recorded performance number belongs to a configuration rather than to a model.
ReadMake It Go Red: The False-Green Failure Class in Production Monitoring
For five months a nightly backup reported Complete over a zero-byte archive. An alert rule had been structurally incapable of firing for its entire life. A dashboard rendered missing telemetry as health, and 37 of 49 deployment scripts could silently ship stale monitoring state. We name the failure class, the false green, map its mechanisms in four families from a 24-day production audit, show that verification tooling itself reproduces the class it hunts, and describe the falsification harness now grading all 279 of our alerting controls continuously: registry, structural liveness, mutation-proven firing tests, and end-to-end canaries. Monitoring correctness is a reachability property, not a syntax property.
ReadThe Day an Agent Ate the Workstation: OOM Forensics of a 178 GiB CLI Memory Leak
An agentic coding CLI ballooned to 178.5 GiB of resident memory during a routine fleet upgrade and froze a production workstation. The kernel killed fourteen innocent processes before touching the culprit. A forensic walk through the OOM killer's victim ordering, why swapless machines turn leaks into freezes, the four-layer guard we deployed the same afternoon, and the upstream report we filed instead of posting screenshots.
ReadPower-Resilience by Design: Making a Stateful Kubernetes Fleet Survive Losing Its GPU Nodes
Our GPU compute tier is powered off most of the time by design. That makes any persistent state, or any hard scheduling pin, that lands on it a silent failure waiting to happen. Here is how we audited every hard dependency, relocated all state onto the storage cluster, and then proved resilience the only way that counts: by cutting the power.
ReadAn Immune System for a GPU Cluster: Building a Reflex Arc for Autonomous Self-Healing
We set out to design a self-healing v2 and found the opposite of a greenfield: four generations of self-healing stacked in sediment, maximum architectural sophistication, near-zero closed-loop coverage. The fix was not a fifth system: it was naming the missing shape, and then discovering the real blocker was signal quality, not architecture.
ReadMarkdown Is Sovereign, Postgres Is a Cache: Engineering a Rebuildable Agent-Memory Substrate
An agent's memory is its most dangerous component: poison it quietly and every downstream decision inherits the lie. We built a memory substrate on one governing principle (Markdown is the source of truth, Postgres is a rebuildable cache) and then learned the hard way that our retrieval eval had been measuring the wrong thing all along.
ReadA First Look at Text-Diffusion LLMs: Benchmarking DiffusionGemma on a Single Blackwell
Google shipped its first open-weights text-diffusion language model: a different generation paradigm that paints a whole canvas of tokens in parallel instead of writing left to right. We benchmarked it within days of release, and a 3.8B-active model beat a 128B production model on hard mathematics.
ReadServing a 550B Open Model on Blackwell B200: A Field Recipe and an Honest Agentic Benchmark
Can a 550-billion-parameter open model be served on Blackwell B200 and stand up to a frontier closed model on real agentic tool use? The serving path is a gauntlet of era-specific gotchas, and the most important result in the whole campaign is the one we published, retracted, and corrected.
ReadRight Pool, Right Workload: How Every Database We Ran Was Quietly Paying a 17× Storage Tax
A benign-looking storage warning led to an uncomfortable discovery: every production database on the cluster had its data on an erasure-coded pool, paying a partial-stripe read-modify-write penalty on every single fsync. The disks were fine. The pool was wrong, by about 17×.
ReadCharacterising a 200GbE RDMA Fabric on Commodity Hardware: From a Hidden PCIe x4 to PAM4 Optics
A routine fabric test found one inter-node link running at 29% of capacity, not broken, just quietly starved of PCIe lanes. Chasing that thread all the way down took us from a hidden slot-bifurcation trap to NCCL collective ceilings to a per-lane optical asymmetry that error-correction was silently hiding.
ReadPast the VRAM Wall: Serving a 397-Billion-Parameter Model That Does Not Fit Three Workstation GPUs
A 397-billion-parameter mixture-of-experts checkpoint occupies 378 GiB. The three workstation GPUs we pointed at it hold 294. It ran anyway, at 8.8 tokens per second warm on a single stream and 17.8 aggregate across four, using pipeline parallelism across two nodes with the expert weights offloaded to host memory. The wall is not a hard stop, but the price is paid immediately and in full. The interesting part is what was not the bottleneck: the 200 GbE fabric moved a gigabyte between nodes in 0.06 seconds, and the GPUs sat at 44 to 71 percent utilisation with power well under their limits for every measured second. Capacity was the constraint, not communication. Getting there needed a patched assertion in the serving runtime, a pinned network interface for the collective library, and the acceptance that no optimised kernel exists for this shape on this silicon.
ReadBuilding One Operator Cockpit for a Sovereign GPU Fleet
A sovereign GPU fleet emits a dozen overlapping signals (GPU box health, Kubernetes workloads, distributed storage, metrics, out-of-band power) and operators were jumping between tools to read them. The engineering story here is what it takes to turn an internal cockpit into something safe to expose: config-driven topology, read-only-by-default gates around every dangerous action, and secret-scanning in CI.
ReadThe Instrument Was Wrong: Stream Count, Slot Topology and Receive-Path Backpressure on a 200 GbE Fabric
For five weeks a three-node 200 GbE fabric appeared to run at 56 to 70 percent of line rate while the slower 25 GbE and 10 GbE tiers next to it reported 94 percent. Eight hardware hypotheses were eliminated before the arithmetic gave it away: the benchmark used eight parallel streams, each iperf3 stream saturates one core at 14 to 18 Gbps, and eight times fourteen is the 112.01 Gbps we had been staring at. The slower links looked healthy under the same harness because they fit inside a single core. Raising the stream count put all six directed paths within a fraction of 198 Gbps. Two more deficits were also instrument failures: a routing directive orphaned outside its interface stanza that had quietly sent storage traffic down a 10 GbE fallback, and a diagnostics dashboard whose 22 failing checks were 22 bugs in the dashboard. One genuine defect survives, a receive-path backpressure on a single host that two plausible tunings both made worse.
ReadThe Trinity Model Shoot-out: Five Frontier MoE Models on Workstation Blackwell
Five frontier open-weight mixture-of-experts models (MiniMax-M2.7, GLM-5.1, DeepSeek-V3.2, Qwen3-235B and Qwen3.5-397B) served on the same three-Blackwell cluster under llama.cpp TCP RPC and graded on one mechanical 30-prompt suite at matched budget. A hybrid Mamba2+transformer leads at 96.7%, and default thinking modes turn out to be a methodology trap.
ReadDistributed Inference on Workstation Blackwell, Part 3: A 228-Billion-Parameter MoE Reasoning Model via TCP RPC
Can a single-operator workstation cluster with no NVLink and no InfiniBand serve a 228-billion-parameter mixture-of-experts reasoning model at conversational speed? Part 3 distributes MiniMax-M2.7 at Q8_0, 226 GiB across three Blackwell GPUs, under llama.cpp over plain TCP RPC, and measures the no-optimisation floor.
ReadDistributed Inference on Workstation Blackwell, Part 2: Amalgamated System RAM for GPU Dataframe Analytics
Does a workstation-class Blackwell cluster extend beyond language-model inference? Part 2 retargets the same two-node, three-GPU cluster to a full GPU-accelerated click-through-rate pipeline (Parquet ingest, high-cardinality shuffle, target encoding, and multi-GPU XGBoost) on roughly 750 GB of amalgamated system RAM over RAPIDS and Dask-CUDA.
ReadDistributed Inference on Workstation Blackwell, Part 1: Cluster Bring-Up and Cross-Node Fabric Characterisation
Can frontier open-weight language models be served on workstation-class NVIDIA Blackwell GPUs interconnected by commodity RDMA Ethernet, without NVLink or InfiniBand? Part 1 of an ongoing PureTensor research programme brings up the cluster, characterises the fabric, verifies GPUDirect RDMA, and demonstrates first cross-node inference of Llama 3.1 405B.
ReadJunction Temperature Is the Wrong Fan-Control Input: A Sensor-Selection Study on a 32-Core Zen 5 Workstation
A workstation was cycling its fans audibly with no matching change in workload. The controller was reading the CPU die junction temperature, which on this generation swings thirty degrees in seconds as threads hop between chiplets, while the socket thermistor two centimetres away sat flat at 38.5 degrees. Any proportional controller fed the first number oscillates by construction. Switching the input stopped the hunting and left the machine's actual temperature unchanged, because it had never been hot. Along the way a kernel update renumbered the hardware monitoring devices and silently pointed the fan configuration at the wrong chip, and the fix for that revealed a third cause underneath: the board's own embedded controller runs a firmware fan loop that watches the same volatile sensor and overrides the operating system entirely. The fingerprint is unmistakable once you look for it, PWM registers constant while fan speed swings by half. Five distinct causes of one symptom, and we misdiagnosed each as the others at least once.
ReadToward a Rotating Embedded Braneworld Cosmology: Critical Assessment, Formal Viability, and a Research Programme for Geometric Dark Sectors
The third paper in the CMAS research series, and the artefact the methodology was built to produce. Cosmology is the conduit; the question is whether structured adversarial interaction between frontier AI models can produce research-grade technical output on an open problem where no ground truth exists. The paper below is what came out.
ReadFrom Intuition to Formalism: Cross-Model Adversarial Synthesis as a Theory Development Tool
Can a vague metaphysical idea, lacking any mathematical grounding, be refined into a falsifiable research programme through structured adversarial interaction between frontier AI models? A case study in speculative cosmology.
ReadCross-Model Adversarial Synthesis: Exploiting Latent Space Heterogeneity for Novel Knowledge Generation
Can structured adversarial interaction between independently trained LLMs produce insights that exceed any single model's capability? We present a theoretical framework and initial case study in black hole information theory.
ReadSurviving the Loss of Six of Nine Nodes: What Actually Blocks Stateful Recovery in Kubernetes
We powered off six of nine nodes in a production Kubernetes cluster, taking the entire storage backend with them, to find out what actually blocks recovery. Stateless failover was a non-event: everything rescheduled onto the four survivors unattended. Stateful recovery stalled on two things, both of them Kubernetes working exactly as specified. A StatefulSet will not create a replacement while the old pod is Terminating, and a pod on a dead node stays Terminating forever because the kubelet that would confirm its deletion is gone. Meanwhile the block-volume attachment objects persist on the dead node, so the replacement fails to attach elsewhere. Two scheduling rules that were correct at nine nodes also became unschedulable at four. A recovery job now clears both classes in sixteen seconds, and a later audit showed why soft affinity had never held placement on a fleet where one node has sixty times another's memory. The most instructive failure of the lot, a zombie storage client that blocked twenty-one pods for nineteen hours with every node healthy, is one the automation could never have caught.
ReadNative FP4 Is Slower Than Emulated FP4: A Kernel-Maturity Measurement on First-Generation Workstation Blackwell
On the day NVIDIA's 120-billion-parameter Nemotron 3 Super was released, we served it at NVFP4 on a single RTX PRO 6000 Blackwell and swept concurrency from 1 to 64 on two back ends: the hardware's native SM120 CUTLASS FP4 kernels and Marlin, a software emulation of FP4 weight-only quantisation. The native path lost at every point, by 7 to 14 percent, 83 against 97 tokens per second single-stream and 801 against 886 at 32 concurrent, because most CUTLASS tile-shape tactics failed inside the just-in-time autotuner and only fallbacks survived. A second finding is larger and less flattering: our first baseline carried a defensive flag that disabled CUDA graphs and reported 27 tokens per second, a 3.6-fold understatement that would have been published as the model's speed had the flag not been questioned. Silicon capability and kernel maturity are different things, and a first-launch number on a day-one stack is provisional.
ReadThe Layered Ceiling: Measuring Every Stage Between Wire Speed and File Transfer on a 200 GbE RoCE Fabric
A 200 GbE fabric moves 25 GB/s on paper. Over five weeks we measured every layer between that wire speed and a finished file copy on a three-node RoCE fabric: raw RDMA at 24.4 GB/s, eight-stream zero-copy TCP at 24.2, NFS over RDMA at 14.5, an unencrypted eight-stream rsync daemon at 12.9, and the rsync-over-SSH restore that started the investigation at 1.98 GB/s, 7.9 percent of the link. Synthetic line-rate tests had been green the whole time. Parallelism was the answer at three unrelated layers, TCP streams, chunked file reads and erasure-coded object writes, and in each case the serial number looked like a hardware fault until the parallel number was taken. We also publish an unreconciled disagreement between two measurements of the same node four weeks apart, and the trust-boundary argument for dropping encryption on a private point-to-point path.
ReadZero Percent or Nothing: The Performance Cliff at the VRAM Boundary in Local Mixture-of-Experts Inference
On a fixed 192 GB VRAM budget, whether a model fits is binary, and it dominates every other tuning axis. On MiniMax M2.5 the smaller Q5_K_XL quantisation runs at 121 tokens per second fully resident while the nominally better Q6_K, which cannot avoid 3 percent CPU offload at any context length, peaks at 72; three percent on the host costs forty percent of throughput, and inside the resident band an eightfold change in context costs nothing. A 200 GB model against the same budget never fits and loses to a smaller incumbent on speed and accuracy. The runtime comparison is the same cliff in another costume: a production-grade engine forced to offload runs at 5.44 tokens per second, and at 35.8 once the weights fit. A second cliff sits on the concurrency axis, where a batching runtime climbs from 35.8 to 821.9 tokens per second aggregate and a serialising one holds flat at 115 while time to first token rises 257-fold. Choose the largest model that fits, never the largest model that runs.
ReadThe Missing Terminal Step: Local Language Models Investigate Infrastructure Incidents Competently and Never Report
We built a synthetic incident-response harness: an alert, four kinds of diagnostic tool, mock responses, and an explicit instruction to call a terminal function once the root cause is known. Ten multi-hop scenarios went to three locally served open-weight models on two workstation Blackwell GPUs. The result was 138 tool calls and zero resolutions. The best model found the root-cause evidence within two to eight turns in every scenario, chose the right commands and read their output correctly, and then kept investigating until the turn cap. The gap is not tool calling, domain knowledge or investigation quality; it is a stopping rule. We name six failure shapes, separate the two confounds (a parser mismatch and a tokenizer corruption) from the universal one, and argue for an orchestrator-executor split in which a protocol-adhering model owns the loop and the local model executes the steps.
ReadThermally Blind by Design: A Super I/O Register Limit Hides Memory Temperature From Hardware Fan Control
A freshly racked Threadripper workstation passed its CPU thermal validation with room to spare, then drove its 256 GB of DDR5 past 85 °C under memory stress while the fans held at 31 percent. The curve was fine. The controller could not see the sensor: the Super I/O exposes aggregate memory temperature as sensor 13 and the kernel driver clamps the fan-source selector at 12, so the board's hardware fan control is structurally blind to its own memory. Five stress runs show that a passing run under a non-deterministic stress method proves nothing (the identical configuration aborted on retry), that the intuitive fix, locking fans at 55 percent, produced the worst result of the series, and that even 100 percent fan speed lost to a 19 °C spike inside one sampling interval. A reproducible write workload and a user-space daemon reading the orphaned channel brought the peak to 54.5 °C. A secondary study on a dual open-air GPU workstation records an 11 °C recirculation delta and a confounded fix.
ReadTranslation Chains Do Not Defend Against Prompt Injection: A Paired Study With a Deobfuscation Regression
Randomised multi-hop machine translation has been proposed as a prompt-injection defence: pass untrusted text through several languages and back, and the delimiters, fake system tags and encodings that carry an attack should not survive the round trip. We ran the first empirical test, 100 public-corpus attacks against a 120-billion-parameter victim and 100 against a 20-billion-parameter one, each attack paired with its translated counterpart. The defence is indistinguishable from noise: McNemar chi-squared 0.696, p about 0.40, a net gain of five attacks in a hundred. Worse, nine attacks that were blocked raw succeeded after translation, because the chain deobfuscated code, leetspeak and encoded payloads into clean instructions and explained the attack to the victim. Both models converged on the same 20 percent defended attack-success floor, the set of attacks whose intent is language-independent and therefore translates correctly.
Read