Distributed Inference on Workstation Blackwell, Part 3: A 228-Billion-Parameter MoE Reasoning Model via TCP RPC

Abstract

Part 3 returns to language-model inference with three deliberate departures from Part 1: a sparse mixture-of-experts model rather than a dense transformer, the llama.cpp framework rather than vLLM, and a plain-TCP remote-procedure-call (RPC) transport rather than NCCL over RDMA. The model is MiniMax-M2.7 — 228.69 billion total parameters, 256 experts with 8 active per token (~10 B active), quantised to Q8_0 for a 226 GiB on-disk footprint across six shards. It is distributed across the same three Blackwell GPUs from Parts 1 and 2 with a 1:1:1 tensor split. The result is a 235.7 GiB aggregate residency, load-balanced to within 3 GiB of perfect symmetry, serving a frontier-scale sparse reasoning model at a sustained 82–83 tokens per second — conversational pace for a single user — with no hyperscaler fabric, no NVLink, and no DGX-class chassis.

1. A Complementary Regime

NCCL over RDMA, the transport characterised in Part 1, is the right tool for dense collective inference where activations are large and the parallelism pattern is tensor- or pipeline-parallel. Sparse MoE inference is a different regime: with 8 active experts per token and ~10 B active parameters, per-token compute is roughly 4.4% of a dense 228 B model's, routing is per-token and per-layer, and the activations crossing GPU boundaries are small. That makes sparse MoE the natural first workload for a lower-bandwidth transport — and llama.cpp's RPC-over-TCP backend the right baseline for establishing the cluster's no-optimisation floor on this model class.

2. Configuration

The hardware is unchanged from Parts 1 and 2: two nodes (Node A with two Blackwell Workstation Edition GPUs, Node B with one Blackwell Max-Q GPU), three GPUs at roughly 96 GB VRAM each, on the same 200 Gbps RoCE fabric. llama.cpp is built from source (build 8070) against CUDA 13.2 with the CUDA and RPC backends enabled and the native Blackwell compute-capability flag set. The cross-node distribution path is deliberately minimal: the remote node's toolchain matches the local one, so the locally built binaries are copied directly into its build tree rather than rebuilt. An RPC server on Node B advertises its single Blackwell Max-Q over TCP; the server on Node A sees three devices total — two local Blackwells and one remote — and splits the model's layers evenly across them.

Before loading, GPU-resident auxiliary services on both nodes are drained to expose VRAM to the loader; the live speech-recognition channel is deliberately retained, with roughly 5–10 GB per node reserved for it. The drain is reversible, and the original service set is restored after the run.

3. Load and Residency

Cold-start load from the six Q8_0 shards completes without error, with all 62 transformer blocks plus the output projection offloaded to GPU (63 of 63 layers) and a 623 MiB embedding table retained as a host-mapped buffer. The split is balanced: two devices carry 78.5 GiB of weights each and the third carries 75.7 GiB, because the layer count is not an exact multiple of three. Steady-state residency observed externally is 80.0, 75.7, and 80.0 GiB across the three GPUs — 235.7 GiB aggregate — leaving roughly 50 GB of headroom for context growth.

This residency figure is the load-bearing claim of the paper: no single Blackwell, and no two Blackwells in the same node, can hold this model at Q8_0. Only the three-Blackwell aggregate with a cross-node transport makes the deployment possible at all.

4. Throughput and the Utilisation Fingerprint

Prompt evaluation (prefill) runs at approximately 337 tokens per second; sustained generation sits between 82 and 83 tokens per second with under 1% run-to-run variance — roughly 12 ms per generated token through 62 MoE layers.

Per-GPU utilisation, sampled during the long generation, is bursty and spiky rather than saturated: peaks of 31% on the local Blackwells and 28% on the remote Max-Q, a median in the 18–22% range, and visible zero-utilisation gaps between spikes. This is the expected fingerprint of llama.cpp's sequential RPC dispatch — only one device executes any given layer while the other two wait, with no pipeline-parallel overlap. The operational reading is twofold: the 82–83 tok/s figure is a single-stream, non-pipelined throughput, and the fabric is plainly not the bottleneck — the devices are most often idle waiting on sequential dispatch, not on the 200 Gbps transport. A pipelined or expert-parallel framework such as vLLM on the same hardware would lift both peak throughput and utilisation; this paper deliberately reports the floor, not the ceiling.

5. Reasoning-Model Behaviour

MiniMax-M2.7 is a reasoning model: its chat template returns a separate internal reasoning channel and a final-answer channel. A generation budget too small for the thinking phase to complete terminates with an empty answer field — a property of reasoning-trained models, not of the transport. The remedy is either a budget large enough to cover thinking plus answer, or an explicit instruction to suppress the reasoning channel. Any OpenAI-compatible client that displays only the answer field and ignores the reasoning field can otherwise observe an apparently empty response. (This budget-versus-thinking dynamic recurs, quantified as a 33-point swing, in the companion Trinity Model Shoot-out.)

6. Position in the Programme

Part 1 set the dense-inference ceiling (405 B dense via vLLM and NCCL); Part 2 set the dataframe-analytics point (Criteo via Dask-CUDA and UCXX); Part 3 adds sparse-MoE inference at 228 B under a minimal framework and transport — the no-optimisation floor. Together the three describe a usable volume in the framework–fabric–model space rather than a single point in it. Subsequent parts cover a 400 Gbps fabric upgrade and a throughput-optimised, expert-parallel ceiling measurement on the same model.

The full paper — GGUF metadata, per-device residency tables, the service-drain protocol, a reproduced reasoning sample, and the roadmap — is available in the PDF above.