A Speed Record Is Only as Good as Its Notes Field: Attempting to Reproduce a 1,004.9 tok/s Community Recipe

Abstract

A community speed leaderboard for local inference carried an entry for GLM-5.3-Flash, a mixture-of-experts model, reporting 1,004.9 tokens per second at a single stream on two RTX PRO 6000 Blackwell GPUs (everything here about that entry is as we noted it on the day we read the page; we did not archive the page). The entry's notes field disclosed its own condition: "acceptance 64/1.00 on locked attractor". We read that as every drafted token being accepted because generation had settled into a repeating loop that a speculative proposer could replay. The entry disclosed this; the headline and the caveat simply sit in different fields.

We tried to adopt the recipe as a seat on our own two-GPU node on 2 September 2026. Across 16 logged launch attempts in one afternoon, the published flags did not compose with the only checkpoint we could fit, the default attention kernel asked for 104,448 bytes of shared memory on a part that allows about 99 KiB, and the pool sizer failed until we set memory flags by hand. After two source patches, one environment setting and those flags, the server reported ready and answered its warm-up request with HTTP 200. The first real request then crashed it with a NaN probability tensor. With CUDA graphs off it stayed up and, as reported in our notes, answered with garbage; our notes report NaN from the first 4-bit expert layer in later runs. A newer FlashInfer build did not change that, as reported in our notes. At four-way tensor parallelism, the entry's own checkpoint failed identically under two different kernel runners, inside the weight loader.

We therefore measured no throughput, and we do not claim the 1,004.9 figure is unreachable on the submitter's rig. We claim two narrower things. On our hardware and build, no configuration we tried produced output we could confirm as valid for this model. And the headline number is conditional on a state in which every drafted token is accepted; it is 2.57 to 2.69 times the 373 to 391 tokens per second that the entry itself gives as a steady state.

1. A Number With a Caveat in the Next Column

A throughput figure for a speculative decoder is the product of two things: how fast the verifier runs, and how many drafted tokens it accepts per step. Speculative decoding means a cheap proposer guesses several upcoming tokens and the full model checks them in one pass. When the text is a copy of the last few dozen tokens, a proposer that replays recent text is always right.

The entry we looked at reported exactly that ceiling. Its metadata, as we noted it on the day, named SGLang (an open-source serving engine), a 4-bit NVFP4 checkpoint, tensor parallel 2 (the model split across two GPUs), sparse attention with TileLang kernels, FlashInfer CUTLASS as the mixture-of-experts runner, a block-draft speculative method with 64 draft tokens, and an n-gram proposer. Batch size was 1, context 8k, clocks locked. Its notes field read "acceptance 64/1.00 on locked attractor".

The page was read on the day and not archived, so every figure about the entry here is as we noted it. The phrase "locked attractor" is quoted; the interpretation (a repeat loop) is ours, and the entry did not use that word. We also tested a different checkpoint from the submitter's and a different power configuration, so nothing here is a statement about whether the recipe works on its original build.

2. Setup

The node has two RTX PRO 6000 Blackwell workstation GPUs, compute capability 12.0 (SM120), about 94 GB usable per card: the allocator reported 93.96 GB available when loading began, from 102,014,189,568 bytes in total. This is the same GPU family as the entry. Our cards were power-capped at 325 W each, where the entry reports 450 W; we did not verify the submitter's card variant.

The software was the project's public container image for this model, an SGLang development build, with PyTorch 2.13 on CUDA 13.0, FlashInfer 0.6.17 and TileLang 0.1.12. For one run we built a derived image with a FlashInfer 0.6.18 development build (tagged 19 August 2026).

The checkpoint was a community compressed-tensors NVFP4 build of about 185 GB in 11 shards, in which only the routed experts are 4 bit and the attention and dense layers stay in BF16. It was the checkpoint we could get that fits two cards. The submitter's own ModelOpt-format checkpoint, a different quantisation container, was tried later.

The procedure was 16 logged launch attempts (run numbers 1 to 14, 16 and 17; there is no log for run 15), each a fresh container: runs 1 to 14 on one node, runs 16 and 17 across two nodes. They fell between 14:50 and 17:14 UTC in two bursts (14:50 to 15:51 and 17:08 to 17:14). Each launch was followed by a small completion request (about 24 tokens, temperature 0) and a read of the server log. We changed one thing between runs where we could; run 4 set three memory flags together. The seat that normally runs on the node was held from 14:49 to 15:56 UTC and restored afterwards.

The common flags were tensor parallel 2, the sparse-attention backend with TileLang prefill and decode, a bfloat16 KV cache, language-only mode, and the mixture-of-experts runner on automatic, which resolved to FlashInfer CUTLASS for this checkpoint.

3. Sixteen Logged Launches

Table 1 lists the runs that moved the problem. Where the only evidence for an outcome is our session notes, the last column says so.

Table 1. The runs that changed the failure. Runs 11, 12, 14, 16 and 17 involve the other checkpoint and appear in Table 3 and Section 6.

The first two failures are plain incompatibilities: the entry's flags require a ModelOpt-quantised checkpoint, and ours was compressed-tensors, whose loader raised on untargeted attention layers.

The third and fourth failures are about memory. With weights at 88.17 GB on a card with 93.96 GB available, 5.79 GB remained. The model has a hybrid of full attention and a linear-attention (state-space style) component, and the latter keeps per-request state in what the engine calls a mamba cache. The pool sizer computed a maximum of minus one slots, with 0.05 GB remaining and 70.39 MB needed per request. We set the memory fraction to 0.985, six mamba slots and 16,384 tokens. Table 2 shows where the memory went.

Table 2. Memory on each GPU through the successful boot in run 5.

The pools that fit were small: 0.19 GB of KV cache for 16,384 tokens, 0.02 GB of mamba conv state and 0.46 GB of mamba state. Six slots at five per request cap the server at one running request. The entry reports 8k context at batch 1, the same order, but this is not a seat: our production seat needs far more context and concurrency than one 16,384-token request slot. Even a fixed kernel would leave it unable to replace the seat, and the binding constraint is the mamba state, not the KV cache.

4. Two Source Patches, One Environment Setting, and Memory Flags

Our session notes called the changes needed to boot "three patches"; the retained files show a different count.

The first source patch is the 12-line change in the quantisation loader: the compressed-tensors scheme lookup returns "unquantised" for layers that are neither targeted nor ignored, instead of raising. The second is in the TileLang sparse-attention kernel. On a device whose opt-in shared memory per block is under 120 KB, it selects a block size of 32, 128 threads and 2 stages. The default block of 64 asked for 104,448 bytes (102.0 KiB) at the failure in run 4. The per-block opt-in ceiling on SM120 is about 99 KiB; our notes give 101,376 bytes from a probe whose output we did not keep, and the code comment in the patch says roughly 99 KB.

The environment setting disables a DeepGEMM prenorm optimisation. The memory flags (fraction 0.985, mamba cache 6, 16,384 total tokens) are not patches, but the pool sizer needed them.

On this architecture the default kernel exceeds the per-block shared-memory ceiling; we make no statement about the recipe on the hardware it was written for. As far as our notes show, the build we ran did not check the device's opt-in limit.

5. Ready, HTTP 200, and NaN

After the tiling patch, run 5 looked healthy. The server reported ready at 15:08:53 UTC after a scheduler end-to-end start of 75.6 s, and the startup warm-up request returned HTTP 200. The first real request then failed a device-side assertion on both ranks, "probability tensor contains either inf, nan or element < 0", and the scheduler died; the process had aborted by 15:12:26 UTC. That is the only hard log evidence of NaN we hold.

With CUDA graphs off (run 6), the process did not crash. It returned HTTP 200 and, according to our notes, the text was garbage ("!!!!") with null prompt log-probabilities. The logs show only the 200. Our notes report NaN in runs 7 and 8, also with graphs off. Run 13 had graphs on, and its retained log shows a 200 and no crash, so its NaN status rests on notes alone and the graph switch is not a clean explanation. The visible symptom differed between runs 5 and 6: in run 5 (graphs on) the stack crashed at the sampler, and in run 6 (graphs off) it answered quietly. A deployment check that waits for "ready" and a 200 on a warm-up would have passed every one of these runs.

Run 9 dumped tensors layer by layer for the first three passes. Our notes report that layers 0 to 2 (dense BF16) were clean, that the router and top-k inputs of layer 3 were clean, and that the output of the layer 3 expert block was 100 per cent NaN. We could not re-run that analysis: the scan output was not kept, and we did not re-load the dump tensors afterwards. We therefore treat the layer attribution as reported.

Runs 6 to 8 and 13 are reported results, so what they rule out is provisional. On our notes it was not CUDA graphs (run 6), not the sparse attention path (run 8, forced), not the runner flag (run 7, where the setting was accepted and then not used), and not the FlashInfer version we started with (run 13). In this build the compressed-tensors NVFP4 mixture-of-experts scheme offers two runners, FlashInfer CUTLASS and FlashInfer TRT-LLM, according to our source inspection. TRT-LLM has no SM120 kernels (run 10), so CUTLASS is what ran.

Public issue reports we read that day describe FlashInfer's CUTLASS FP4 grouped GEMM as written for SM100 and returning zeros or NaN on SM120; our observation fits, but we did not verify them, and ours is one build and one checkpoint format. Whether the fault lies in the kernel numerics or in how compressed-tensors weights are prepared was undetermined at the end of the two-GPU trials, because no ModelOpt W4A4 checkpoint fit on two cards to separate them.

6. The Capacity Wall and a Loader Failure at Four-Way Parallelism

The submitter's checkpoint is ModelOpt-mixed NVFP4, about 198 GB. Table 3 shows what happened on two cards.

Table 3. The two checkpoints we could try on two GPUs.

Run 11 failed earlier on a missing "quantization" key in the config reader. Run 12 used a wrapper that, per our notes, nests the quantisation config and rewrites the weight-name prefix; it failed a 4.5 GiB allocation (4,848,615,424 bytes, with 2,756,182,016 free), which our notes attribute to experts being allocated unquantised. After the prefix fix (run 14) the failing allocation was 77,594,624 bytes (74 MiB) with 27,787,264 free. The entry reports 187.9 GB peak across two GPUs with presharded loading, which would fit by a small margin. We could not reproduce that load, because our notes say the per-rank presharded files are created on first launch and we ran out of memory before then. So we do not know whether their run fits so tightly because of the loader or the checkpoint.

To remove the capacity excuse, we ran the submitter's checkpoint at tensor parallel 4 across our two nodes, with the server arguments logging a memory fraction of 0.72, and tried two runners.

Table 4. The cross-node attempt, 17:08 to 17:14 UTC.

Two different runners stopped on the same assertion, so the fault sits in weight loading, upstream of both, and is independent of the kernel. The 256 against 512 mismatch is a factor of two in scale width, and matches an open upstream issue for another model of the family, as we read it that day. This is a second defect on a different checkpoint format. It does not separate kernel from weight preparation for the checkpoint that gave NaN, because here no kernel ran. The summary is two different failures on two different formats; it is not evidence that a kernel fault and a weight-preparation fault both exist independently. No SGLang configuration we tried on this hardware produced a single valid token for this model.

7. What the Headline Measures

The rest of this section uses only the entry's own notes, as we recorded them.

Six runs from one submitter on one box, with batch 1, 8k context and a 403-token prompt, formed a ladder in draft width.

Table 5. The reported draft-width ladder and the reported steady state. All values as noted from the unarchived page.

Throughput climbs with draft width, and the 64-token rung is the headline. The entry's own notes give 373 to 391 tokens per second as the steady state.

Acceptance of "64/1.00" means every drafted token was accepted. At 1,004.9 tokens per second with 64 tokens accepted per verification step, the verifier ran about 15.7 steps per second (1,004.9 divided by 64), or roughly 64 ms per step. That is replay throughput: the draft tokens cost almost nothing when the continuation is a copy of the preceding 32 to 64 tokens. The ratio of the headline to the entry's own steady state is 2.57 to 2.69 (1,004.9 divided by 391 and by 373).

We did not measure any of this on our hardware. The entry's n-gram proposer, described in its notes as a 32-gram cyclic tiling, does not appear in the command line it lists, per our notes, so it would be an environment setting or an unpublished patch. As we noted it, the submitter named no image or commit.

None of this suggests the entry was misleading; the condition was written down. A leaderboard column holds the number and the notes column holds its meaning, and a reader who copies the first and skips the second is quoting something else.

8. What We Can and Cannot Say About a Page We Did Not Archive

Every run is a single attempt (n = 1), on one node, with one SGLang build and, for the NaN finding, one checkpoint. No run was repeated, and there are no confidence intervals because there is nothing to put them on. NaN was reported in runs 6 to 9 and 13 (notes) and logged in run 5; only run 5 has a hard log signature.

The per-layer attribution to the first 4-bit expert block could not be re-run, and we have the dump files but not the scan output. Our checkpoint was not the entry's. The NaN is about this build, this checkpoint format and SM120. The entry used ModelOpt, and our only run of that checkpoint stopped in the loader at four-way parallelism. The community page was not archived, so every figure about the entry, including the quoted phrase, the ladder and the steady state, is as we noted it and cannot be checked from retained material. We measured no throughput on the recipe, so we make no claim that 1,004.9 is unreachable on its original rig.

Claims that survive: the recipe as listed did not boot on our node without two source patches, one environment setting and memory flags; the first real sample then crashed it with a NaN probability tensor; and at four-way parallelism two runners failed identically in the loader. The specific numbers about the entry, the layer-level attribution and the garbage-output reports do not survive without the original artefacts.

9. What We Changed

We stopped working on the SGLang path for this model on this hardware. It would need a weight-loader fix for the mixture-of-experts down projection and a different 4-bit kernel path, which at the time was an unmerged upstream proposal.

We added an intake rule for community speed numbers. Read the notes field and the per-run table first, and take the lowest steady-state figure as the working number until we have measured it ourselves on non-repeating prompts. A new seat is also judged on a real sampled request with its output checked, not on a warm-up and a 200.

10. Methods and Artefacts

Engine: an SGLang development build in a public container; PyTorch 2.13 with CUDA 13.0; FlashInfer 0.6.17, and a derived image with a FlashInfer 0.6.18 nightly, CUTLASS DSL 4.6.2 and NCCL 2.30.7; TileLang 0.1.12. Method: sequential single-change launch attempts, one container each, server log capture, small health and completion requests, a per-layer tensor dump on the first three passes, a source inspection of the image for the available mixture-of-experts runners, and a final cross-node tensor-parallel 4 pair. Hardware: one node with two RTX PRO 6000 Blackwell workstation GPUs at 325 W each, and for the last pair a second node. Statistics: none; every run is n = 1.

We retain all 16 launch logs, the launch scripts, both patched files with their originals, the dump files for three passes, the probe and tensor-scan scripts, and the build log for the derived image. We do not retain the community page, the probe output, the tensor-scan output or the text of any response.

PureTensor operates a sovereign AI infrastructure fleet (on-premises GPU compute, storage, and serving) run day-to-day by autonomous agents under human direction. This paper is PT-R-2026-050. The measurements were taken on 2 September 2026, between 14:50 and 17:14 UTC; launch logs, patches and tensor dumps are retained.