STREAM Triad Under-Reported Our Memory Bandwidth by a Third: Measure the Kernel Your Workload Runs

Abstract

Our estimates of how fast a very large mixture-of-experts model can decode from host memory rested on one formula: tokens per second is roughly the realised memory bandwidth divided by the bytes of active weights read per token. The bandwidth term came from the STREAM Triad benchmark, run on three hosts on 6 August. A day later we re-measured the same hosts with two further kernels, a Triad written with non-temporal (cache-bypassing) stores and a pure sequential read, and the Triad figures turned out to be the wrong quantity for the job.

Ordinary-store Triad counts three words per element and moves four, because each store to a line not in cache first reads that line (read for ownership). On two hosts the non-temporal Triad ran at 1.351 and 1.336 times the ordinary figure, near the 4/3 the mechanism predicts. Across three hosts the old figures summed to 348 GB/s; the pure-read kernel summed to 495.1 GB/s, so the old sum was 29.7% below the read capability, and per host 27% to 35% below. (True traffic is a third larger than a Triad reports, so the reported figure is a quarter low; against a pure-read kernel the old sum was about 30 percent low.) Against each host's own pin rate, the old method put all three hosts at 52 to 53%; pure read put them at 72.6% to 81.5%.

The error made the four-channel host, in fact the most efficient of the three, look misconfigured and prompted a search for a free doubling. It also supported an argument that a target decode rate needed "about 450 GB/s, more than the sum of all three hosts (348)". We withdrew both the suspicion and the argument.

An overnight ladder of very large models on two workstations followed. In two runs, one per model, the ordering followed bytes of active weights read per token and not parameter count: a 2.8-trillion-parameter model at 2-bit decoded at 4.2 tokens per second (warm runs 3.5 to 4.7) against 3.6 (3.4 to 3.7) for a 753-billion-parameter model at 8-bit, so the ranges overlap. We also record the cost of a cold page cache; the largest swing we saw rests on a log we did not keep. Section 8 says which claims survive.

1. The Formula and the Number Feeding It

When a model is too large for GPU memory, a mixture-of-experts (MoE) design lets most of it sit in host RAM, because each token activates only a few experts. Decode speed then depends on how quickly those active weights stream from RAM. Our working estimate was tokens per second equal to realised bandwidth divided by bytes of active weights per token.

The bandwidth term came from STREAM Triad, the standard memory-bandwidth benchmark, which computes a[i] = b[i] + s*c[i] over arrays far larger than cache. We ran it on three hosts on 6 August.

Two things in that work looked wrong afterwards. The four-channel host's Triad result of 88 GB/s had been set against a "published" figure of about 170 GB/s for that processor class, and the gap suggested a firmware or NUMA fault, perhaps a free doubling. And we had argued that a target decode rate needed about 450 GB/s of realised bandwidth, more than the sum of our three hosts at 348.

We asked two questions. Is Triad the right measurement of what a weight-streaming workload can pull from host memory, and what changes once the right kernel is used? And, from an overnight run on 22 and 23 August, does bytes per token rather than parameter count explain what the largest configurations achieved, and what else moves throughput at that scale?

2. Three Hosts and Three Kernels

Pin rate is the theoretical peak: transfers per second, times 8 bytes, times channels.

Table 1. The three hosts. The first host's modules are rated 5600 MT/s and run at 5200, the supported speed for four dual-rank modules on that platform. The three pin rates sum to 661.3 GB/s.

The probe is a single C program using OpenMP, compiled with gcc at -O3 for the native architecture. It allocates three arrays of 128 Mi doubles, 1 GiB each, a 3 GiB working set against 128 MiB of L3 on the Threadripper-class hosts (the third host's L3 size was not recorded, but no plausible L3 approaches 3 GiB), so it measures DRAM and not cache. It runs three kernels. The first is ordinary-store Triad, reported by STREAM's convention as 3 words (24 bytes) per element over elapsed time. The second is the same Triad written with 256-bit non-temporal stores, which bypass the cache and avoid fetching the target line; every worker thread issues a store fence before the timer stops. The third is a pure read, a summed sequential read of one array at 1 word per element. The pure read is the closest of the three to reading expert tensors out of RAM.

Runs use 16 threads across physical cores, best of five. A compile-time gate confirms that the binary contains the non-temporal opcode and a fence, and after each run every array element and the reduction results are checked analytically (relative error under 1e-9).

That validation exists because of a mistake: a first attempt reported 687 TB/s on one host, because the third array was initialised to zero, nothing consumed the result, and the compiler deleted the loop.

3. What the Three Kernels Measured

Table 2 gives the figures. "Old" is the 6 August ordinary-store Triad. The bandwidth tables survive only in two records written on the day; the raw run output was not retained.

Table 2. Bandwidth in GB/s by kernel and host. The sums are of three independent hosts. Raw output not retained.

Table 3. The old Triad figure against the other two kernels. The old sum is 29.7% below the pure-read sum; per host the shortfall is 27% to 35%.

Pure read is not the highest figure on every host. On the DDR4 server the non-temporal Triad (144.5) exceeds pure read (136.3) by 6.0%, while on the other two hosts pure read is higher, by 3.8% and 2.4%. One internal note had called pure read "the highest number of all", and the table does not support that. The DDR4 server may have been carrying cluster workloads at the time; we have no idle re-measurement.

The second detail is run-to-run spread. The four-channel host gave 88 GB/s ordinary Triad on 6 August and 96.7 on 7 August, a rise of 9.9% for the same kernel. We do not know the conditions of the first run, so the range for ordinary Triad on that host is 88 to 96.7. The eight-channel DDR5-4800 host was steadier: 161 on the first day and 163.1 on the second, 1.3% apart.

4. Why Triad Reports Three and Moves Four

Triad reads two arrays and writes one, so STREAM's documented convention counts three words per element. On hardware with write-allocate caches, a store to a line not in cache first reads the line. This read for ownership adds a fourth word of traffic that the formula does not count; non-temporal stores skip it.

If that is the whole effect, the ordinary figure should be three quarters of the true traffic rate, and the non-temporal figure four thirds of the ordinary one. We have a same-session pair for two hosts.

Table 4. Ordinary against non-temporal Triad, measured on 7 August, in GB/s. The mechanism predicts a ratio of 1.333 (4/3). No re-measured ordinary figure is recorded for the third host.

Applying 4/3 to the ordinary figures gives 128.9 against 130.6 measured on the first host and 217.5 against 217.9 on the second. That agreement is our evidence for the mechanism. We read no memory-controller traffic counters, so the claim that four words moved is inferred from the ratio and was not observed.

The claim "by a third" is exact only for traffic against the reported figure: true traffic is a third larger than the figure an ordinary Triad reports (4/3), so the reported figure is a quarter below the traffic (3/4). The old three-host sum against the pure-read sum is a different comparison, and the gap there is about 30 percent.

STREAM is not wrong; its number is a valid measure of requested-byte throughput. The mistake was ours: streaming expert weights from RAM is read-only, so an ordinary Triad is a poor proxy for it, both for the uncounted read traffic and for its mix of two reads to one write.

5. The Misdiagnosis and the Withdrawn Argument

The four-channel host's 88 GB/s had been compared with about 170 GB/s, which is the pin-rate arithmetic for that platform (166.4), not an achieved result. The other two hosts had been judged against their own much larger pin rates, and so the host that was in fact the most efficient of the three looked like an outlier.

Table 5. Each kernel as a share of the host's own configured-speed pin rate. The old-Triad shares are recomputed from Table 2 (88/166.4, 161/307.2, 99/187.7, 348/661.3).

Under the old method all three hosts sat at 52 to 53 percent of pin rate; the apparent outlier came from comparing one host's Triad against a published figure. That the four-channel host was the best of the three holds only under pure read (81.5% against about 72.6%).

The suspected BIOS or NUMA fault was ruled out: all four channels were populated, the modules ran at the platform-supported 5200 MT/s (forcing the rated 5600 would be an out-of-specification overclock for about 8% theoretical gain), there was a single NUMA node, and a firmware revision one release behind showed no evidence of any bandwidth effect. We retracted the free doubling on 7 August.

The 450 GB/s argument also fell: the corrected sum is about 495 GB/s, so we withdrew it. The broader conclusion stayed, on other grounds. GPU memory at about 1.8 TB/s per GPU (a quoted figure, not measured here) is roughly 3.6 times the 495.1 GB/s sum (1800/495.1 = 3.64) and roughly 8 times the fastest single host (1800/223.2 = 8.1).

The 495.1 GB/s figure assumes all three hosts stream at once with no compute, PCIe, network or pipeline cost, so it is an upper bound and not a serving result. A real expert-gather workload (random selection, many small tensors, on-the-fly dequantisation) will sit below it.

6. The Ladder: Bytes per Token

With the bandwidth term repaired, we tested the bytes-per-token idea against what two large models did. We served Kimi K3 (about 2.8 trillion parameters) and GLM-5.2 (753 billion) from GGUF files with a mid-August build of llama.cpp (CUDA 13.2, Blackwell target) on two nodes: one with two RTX PRO 6000 Blackwell GPUs and 503 GB of host memory, one with two further GPUs of that class and 251 GB, joined by llama.cpp's RPC backend over a 200 Gb/s RDMA fabric. Layers were split across GPUs with memory-mapped weights. Each test was one cold request and three warm requests of 512 streamed tokens, a fixed mathematics prompt, temperature 0.6, batch size one; "warm mean" is the mean decode rate of the three warm runs.

Table 6. The ladder. Decode rates in tokens per second; TTFT is time to first token. Warm TTFT ranged from 0.70 to 1.04 s across all four.

Spreading the 1-bit model across both nodes raised the warm mean from 7.6 to 8.9 (+17%), with no measurable cross-node cost in warm TTFT. The first one-node launch's warm runs (5.2, 6.9, 7.7) came as the 554 GB map paged in; the 7.6 is from the second launch. The 2-bit warm runs rose from 3.5 to 4.7, so 4.2 is the mean of a ramp and 4.7 the better steady-state indicator. Server-side eval timings match the client (3.47, 4.42 and 4.72).

The comparison that matters is between the last two rows. GLM-5.2 at Q8_0 has about 40 billion active parameters per token, and K3 about 100 billion; both figures are from secondary descriptions and not verified independently. At 8 bits per weight the GLM model reads about 40 GB per token. K3 at 2.2 bits per weight, the figure in our earlier working note, reads about 28 GB; at the 2.46 bits that its file size over parameter count gives, about 31 GB.

Table 7. Bytes per token against observed speed. Bits per weight are file size over parameter count; byte figures are active parameters times bit width.

The byte ratio is 40/28 = 1.43 on nominal bits, or 42/31 = 1.35 on file-derived bits; the speed ratio is 4.2/3.6 = 1.17, or 1.31 using the 4.7 trend. The model with 2.5 times as many active parameters is the faster one, which is consistent with its moving fewer bytes per token, though paging and VRAM share also differ and the observed ratio is below the predicted one. Within K3, 1-bit (8.9) against 2-bit (4.2) is a 2.1-fold speed ratio against a 1.56-fold ratio in bits per weight, confounded by placement (the 1-bit run kept about 68% of its weights in VRAM).

We derive no effective host-RAM bandwidth for any ladder run: 3.6 tokens per second times about 42 GB is about 150 GB/s, but that draws on VRAM and RAM together.

7. What a Cold Page Cache Costs

Table 6 shows what first touch costs: 117.9 s to first token on the 861 GB map, 73.6 s on the 801 GB one, and for the one-node 1-bit run 39.9 s on the first launch against 0.77 s on the second, with the cache populated. Memory-mapped weights live in the operating system's page cache; anything that evicts them turns tokens into disk reads.

Notes written that night also record a much larger slowdown on the 861 GB configuration while a concurrent 801 GB model download ran on the same host, with recovery to 4.2 in the later quiet run. The slowdown was not logged, so we give no figure for it. Eviction of the mapped weights by the download is a diagnosis inferred from the coincidence and the recovery, with no controlled comparison; it is consistent with the verifiable cold-start rows above.

We also tried to extend the memory pool. VRAM-only serving held about 360 GB of weights, and adding the second-largest host's RAM took that to about 830 GB, stable. Adding a third host's RAM (about 1.0 TB; our notes give both 1.01 and 1.03) was blocked: exporting its CPU memory as an RPC device crashed during tensor load, reproducibly across two attempts, while a GPU-only export was stable. We did not establish whether the cause lies in the RPC backend of the llama.cpp build we used or in our configuration. The 6 to 8 tokens per second projected for a fully resident 861 GB model is not a measurement.

8. What These Numbers Will Not Carry

Bandwidth. Each host was measured in one session, best of five at 16 threads, with no variance reported and no raw output retained. The 88 against 96.7 repeat on the four-channel host shows that spread can reach about 10%, and we cannot say where it came from. The DDR4 host may not have been idle. A different kernel would give a different ceiling, and no hardware counters were read.

Ladder. Every configuration was one run, single stream at batch size one, one prompt, three warm runs, with no confidence intervals; paging and VRAM share differed between configurations. The K3 active-parameter figure comes from a secondary document.

Claims that survive: the 4/3 ratio on two hosts; the roughly 30 percent understatement of the old three-host sum; the measured ladder rates; and the ordering of GLM Q8 and K3 2-bit by bytes per token, for these two models and one run each. Claims that do not: that 163.1 "exactly" reproduced 161 (it was 1.3% higher), that pure read is highest on every host, any fitted law relating bytes per token to speed, or any size for the unlogged slowdown.

Still unknown: whether the three-quarters ratio is entirely read for ownership; the idle DDR4 result; the cause of the 88 and 96.7 difference; and the 2-bit steady state.

9. What We Changed

We replaced the single Triad measurement with a three-kernel probe (ordinary, non-temporal, pure read), reported as separate quantities, with element-wise validation and the compile-time opcode check. It shipped on 7 August. We now quote bandwidth as a share of each host's own pin rate before comparing hosts or concluding that one is misconfigured, and we use pure read as a working upper estimate for weight streaming (non-temporal Triad exceeded it on one host), before compute, PCIe and scheduling.

For frontier-scale offload we compare configurations by bytes per token, and we keep bulk disk traffic off any node serving a memory-mapped model. The next experiment is the controlled one: the same model with and without a competing download, logs kept.

10. Methods and Artefacts

The bandwidth probe is a purpose-written C and OpenMP program (gcc -O3 -march=native; three 1 GiB arrays; 16 threads; best of five) with analytical validation of every element. The ladder used a mid-August llama.cpp master build with CUDA 13.2, a Blackwell architecture target and the RPC backend, with client-side timing of streamed tokens and server-side eval timings. Statistics are best-of-N for bandwidth and the mean of three warm runs for the ladder. We retain the probe source and its test script, the per-run result files and server logs of the ladder, and contemporaneous working notes; the raw bandwidth run output and the log of the slow page-cache run were not retained.

PureTensor operates a sovereign AI infrastructure fleet (on-premises GPU compute, storage, and serving) run day-to-day by autonomous agents under human direction. This paper is PT-R-2026-049. The bandwidth measurements were taken on 6 and 7 August 2026 and the model ladder ran overnight on 22 and 23 August 2026; the probe source and the ladder's result files and server logs are retained, while the raw bandwidth output is not.