Serving a 550B Open Model on Blackwell B200: A Field Recipe and an Honest Agentic Benchmark

The Goal

Take a 550-billion-parameter open model — Nemotron-3-Ultra, with 55B active parameters in NVFP4 quantisation, roughly 330–350 GB across 113 shards — stand it up on rented Blackwell B200 hardware under a tensor-parallel-8 serving configuration, and then honestly benchmark it against a frontier closed model on *both* reasoning and agentic tool use. The reasoning result is interesting. The agentic result is a lesson in how easy it is to measure your own harness instead of the model.

The Blackwell Serving Gauntlet

Serving a new-architecture model on brand-new datacentre silicon is a cascade of failures, each masking the next. The ones worth hardening into a recipe:

• CUDA-13-only wheels force a new driver. The current inference-server nightlies ship CUDA-13 wheels and require driver ≥ 580. The provisioned box shipped an older driver, and the *proprietary* 580 driver reported "No devices found" on a chassis with eight B200s physically present. Blackwell requires the open kernel modules — only the open-driver variant brought all eight GPUs back, after a reboot.
• Desktop driver on NVSwitch hardware → CUDA Error 802. On the multi-GPU box, the host had a *desktop* driver where a *datacentre* driver belongs. The tell is maddening: the GPU listing tool cheerfully shows all eight GPUs while every CUDA context fails with "system not yet initialized." The fix is a matched datacentre driver-plus-fabric-manager pair, where the fabric manager build must match the driver *exactly*.
• JIT/runtime-compiler crash. A just-in-time kernel path crashed on a missing runtime-compiler library; the fix was installing the matching CUDA-13 runtime-compiler package and pointing the loader at it, plus a build tool for one of the attention backends.
• Serving choices to dodge the JIT entirely. Forcing a weight-only MoE backend and eager execution sidestepped both a fragile JIT path and a compiler-version mismatch, and the *built-in* reasoning parser had to be used because the model's bundled custom parser failed to register.

None of these are interesting once solved — and that is exactly why writing them down is the contribution.

Throughput

On the prior generation (Hopper), the model has no native FP4 and runs via weight-only emulation: ~9.5 tokens/sec single-stream. But throughput scales almost linearly with batching — 9.3 → 36.7 → 73.4 → 141.9 → 282.6 tok/s at concurrency 1 / 4 / 8 / 16 / 32. The win on shared hardware is in continuous batching, not single-stream latency. On B200 the model runs NVFP4 natively, removing the emulation tax entirely.

Capability vs a Frontier Closed Model

A contamination-controlled, adversarially-verified battery of 63 items with deterministic graders, against a leading frontier closed model. On the validly-measured items the closed model led 94% to 88% overall — a real but modest gap — with a revealing per-dimension breakdown:

The open model *ties* on math and graduate science, and actually *beats* the frontier model on instruction-following. Its one clear weakness is code. A near-frontier open model you can self-host is a genuinely different proposition from a frontier API.

The Honesty Beat

The first agentic write-up was a disaster dressed as a finding. It declared the model scored 0 out of 13 on tool-calling and agentic tasks, and reached for a tidy narrative: that NVFP4 quantisation had broken the model's tool use. It made a good story.

It was wrong. Every one of those 13 items had failed with an HTTP 400 Bad Request, because the serving process had been started *without the tool-calling flags enabled*. The model never received a single one of those requests. We had measured our own misconfiguration and blamed the model's quantisation.

So we retracted the claim. The corrected serving config renders tools to the model through its native chat template, and the model emits clean structured tool calls parsed client-side. Re-run on native B200, the tool-calling sanity check went from 0/8 to 8/8; a 20-scenario function-calling suite scored 97%; and a deeper benchmark that grades on final environment state through a real reason-act loop — not single-call string matching — scored 100% across all categories, against the closed model's own 100% on the same scenarios. A standout was refusal behaviour: the model correctly declined to call a tool when none applied, and asked for missing parameters instead of hallucinating them — the single most common enterprise agentic failure mode.

The number that matters here is not 97% or 100%. It is that a "0%" result flipped to "100%" with a one-line server-config change. The point of publishing the retraction is that an agentic score measures the harness at least as much as the model, and a lab that won't retract its own headline isn't measuring — it's marketing.