A First Look at Text-Diffusion LLMs: Benchmarking DiffusionGemma on a Single Blackwell

A Different Way to Generate

Almost every language model in production generates left to right, one token conditioned on all the tokens before it. A text-diffusion model does something else: it starts from a canvas of masked positions and *denoises* the whole block in parallel over a fixed number of steps, refining all positions at once. DiffusionGemma — Google's first open-weights model of this class, released under a permissive licence — makes that paradigm available to benchmark on real hardware, and we wanted a genuine first look within days of release.

It is a 26B-parameter mixture-of-experts model with 3.8B active parameters. Generation uses a fixed 256-token denoising canvas per block, up to 48 denoising steps, a descending temperature schedule, and an entropy-bound sampler. It also exposes a separate "thought" channel in its chat template.

Standing It Up Is Half the Story

The architecture is new enough that the standard transformers install cannot even load it — the model class was four days old, and the fleet-wide library version didn't know it existed. It needed an isolated runtime with a bleeding-edge transformers build. And because no production inference server supports the diffusion architecture, the model runs on raw transformers behind a small OpenAI-compatible shim, co-resident on a single Blackwell GPU with a live production serving process. Warm load was ~6 seconds; peak memory ~60 GB alongside the existing ~31 GB service, which made a batch of four the safe ceiling on the shared card before OOM.

Diffusion Breaks the Tokens-Per-Second Metric

The first thing diffusion forces you to do is rethink throughput. Because the model fills the entire 256-token canvas every block regardless of how long the answer actually is — then emits an end-of-sequence marker plus padding — there are two different "tokens per second," and conflating them is a reporting error:

• canvas-tok/s (~414 peak) is the true parallel-generation rate: how fast the model paints the full canvas.
• content-tok/s (~223 best, ~147 steady-state) is the rate of *emitted answer* tokens, which badly understates the parallel work being done.

Per-block latency ran 0.6–1.4 seconds per 256-token canvas. On a shared card in stock eager mode, end-to-end throughput sat well below the vendor's published figures, which assume a dedicated GPU and an optimised runtime — a fair gap to flag rather than hide.

The Surprise: A 3.8B-Active Model Leading on Math

We ran a zero-shot intelligence battery against a 128B-parameter FP8 production model. The 128B model won 7 of 8 benchmarks overall — but the headline is where the small diffusion model *won*:

A model with roughly 34× fewer active parameters leading on hard mathematics, and within striking distance on grade-school math, is a genuine punch above its weight.

That result comes with honest measurement caveats, which matter as much as the number. MATH is parsing-sensitive: DiffusionGemma reliably boxes its final answer, while the larger model often replies in prose and gets under-scored by a strict grader — so part of the gap is presentation, not capability. A separate benchmark showed a ~50-point gap that turned out to be a generative letter-format penalty, not a real commonsense difference. Reporting the caveat is the difference between a benchmark and a press release.

It Speaks Its Own Tool Dialect

On an agentic tool-use probe the model scored essentially perfectly — valid tool calls, correct tool selection, correct arguments, the discipline to *not* call a tool when none applied, and multi-step chaining. But it does not emit OpenAI-style JSON tool calls; it emits its own native dialect, a compact call-with-braces syntax. Production use needed a parser written specifically for that format. Tellingly, all three bugs we hit during the evaluation were in *our* harness, not the model: a generation wrapper whose output object needed the right field sliced (a naive access returned zero tokens), a benchmark that needed the target function name pinned in-prompt, and the tool parser reading the wrong format. The model was correct each time.

Why It Matters

Text diffusion is an early, under-explored architecture class, and the easy move is to dismiss a 3.8B-active model against 128B incumbents. The data argues for curiosity instead: parallel denoising appears to spend its limited active-parameter budget unusually well on structured reasoning, and the right way to measure it isn't the autoregressive tokens-per-second habit. We expect to revisit this as runtimes mature and the architecture gets first-class serving support.