The Trinity Model Shoot-out: Five Frontier MoE Models on Workstation Blackwell
Download PDFAbstract
A companion to Part 3 of the Workstation Blackwell programme, this paper grades five frontier open-weight mixture-of-experts models on identical hardware, transport, and serving configuration: MiniMax-M2.7 (Q8_0), GLM-5.1 (Q2_K_XL), DeepSeek-V3.2 (Q2_K_XL), Qwen3-235B-A22B (Q4_K_M), and Qwen3.5-397B-A17B (Q4_K_XL). All five run on the same two-node, three-GPU Blackwell cluster under llama.cpp TCP RPC, at a 16,384-token context and a matched 8,192-token per-request budget, against a single mechanical 30-prompt suite (five categories, three difficulty tiers, suite maximum 90). Qwen3.5-397B — a hybrid Mamba2+transformer MoE with the lowest active-parameter count of any reasoning entry — leads the field at 87 of 90 (96.7%), more than 25 points clear of last place. Two findings only become visible at five models: a hybrid state-space architecture out-scoring every pure transformer at matched footprint, and default thinking modes acting as a matched-budget methodology trap.
1. The Field
| Model | Total / active | Architecture | Quant | Score | tok/s |
|---|---|---|---|---|---|
| Qwen3.5-397B-A17B | 397 B / ~17 B | Hybrid Mamba2 + transformer, 512 experts / 10 active | Q4_K_XL | 87/90 — 96.7% | 38.1 |
| Qwen3-235B-A22B | 235 B / ~22 B | Pure-transformer MoE, 128 experts / 8 active | Q4_K_M | 82/90 — 91.1% | 59.6 |
| GLM-5.1 | 355 B / ~32 B | MoE | Q2_K_XL | 77/90 — 85.6% | 37.0 |
| DeepSeek-V3.2 | 671 B / ~37 B | MoE (MLA + DSA) | Q2_K_XL | 73/90 — 81.1% | 41.1 |
| MiniMax-M2.7 | 228.7 B / ~10 B | MoE | Q8_0 | 58/90 — 64.4% | 76.2 |
The exercise is matched on every non-architecture axis: same cluster, same llama.cpp build, same transport, same context and budget, and the same deterministic grader running offline against saved JSON artefacts. Scoring is up to three points per prompt (two for correctness, one for format) across mathematics, code, graduate-knowledge recall, reasoning traps, and structured output, at floor, discriminator, and ceiling difficulty.
2. What Five Models Reveal That Three Did Not
A hybrid state-space architecture wins at matched footprint. Qwen3.5-397B is the largest model in the matrix by total parameters but carries the lowest active-parameter count of any reasoning entry (~17 B), with attention on every fourth block and Mamba2 state-space blocks elsewhere. Under the same budget and transport it out-scores every pure-transformer model — a perfect floor tier, a perfect discriminator tier, and only three losses, all at ceiling.
The mathematics gap is about model quality, not reasoning traces. Earlier three-model revisions read a reasoner-versus-non-reasoner gap into the mathematics category. The five-model matrix dissolves that framing: MiniMax scores 0 of 18 on mathematics while Qwen3.5 scores 18 of 18 — both without emitted reasoning traces. A strong non-thinking model simply beats every thinking model in the matrix on short-answer mathematics at matched footprint.
Block count predicts latency better than active parameters. Aggregate generation rates track transformer-block count more closely than active-parameter count across the pure-transformer entries. Qwen3.5's hybrid is the instructive exception — its Mamba2 blocks carry higher per-token work than a matched transformer block at this fabric, consistent with the experimental state-space kernel coverage in this llama.cpp build, and worth re-measuring as that path matures.
3. Default Thinking Modes Are a Matched-Budget Trap
The single sharpest methodology finding: run identically except for one switch, and Qwen3.5 scores 96.7% with thinking off and only 63.3% with thinking on. At the shared 8,192-token budget the thinking-on run consumes its entire budget inside the chain-of-thought before the grader's required final-answer marker is emitted — saved traces show the reasoning stream ending mid-sentence at the token boundary, the model about to switch to its visible answer exactly as the budget expires. Appending a no-think instruction to the user turn was silently ignored by this template; the canonical fix is to pass the thinking flag through the chat-template arguments, verified by probing the rendered prompt.
The lesson generalises. Any matched-budget comparison of models whose chat template defaults to thinking-on must either raise the budget until the chain-of-thought reliably terminates, or pin the thinking flag explicitly. The headline numbers here take the second option to keep the budget matched. This is the same mechanism flagged qualitatively in Part 3, here measured as a 33-point swing.
4. Threats to Validity
The suite is n=30 with six prompts per category and no bootstrap: a single-prompt shift moves the suite total by roughly 1.1 points, so results are reliable to the nearest percent and not to the tenth. The grader is deterministic but not infallible — a correct answer in a form it does not accept scores zero. Throughput figures are aggregate rates over the full suite including prompt processing, not steady-state single-request decode. Qwen3.5's state-space kernel path is experimental in this build, so its absolute tokens-per-second should be re-measured as kernel work matures. All five models were run at temperature 0.2. With the top of the suite now near-saturated, a harder v3 instrument — an expanded ceiling tier, research-grade multi-step prompts, and a temperature sweep — is on the roadmap, alongside adding Nemotron Ultra at NVFP4 once weights are published.
The full paper — the complete five-way scorecard by category and difficulty tier, per-model throughput and visible-token accounting, the thinking-mode trace analysis, and the full prompt list — is available in the PDF above.