Characterising a 200GbE RDMA Fabric on Commodity Hardware: From a Hidden PCIe x4 to PAM4 Optics

The Smoking Gun

A 200GbE RDMA fabric is supposed to be boring. Ours wasn't, and the way it failed is instructive at four different layers of the stack. A routine throughput test found one inter-node leg pinned at 58 Gbps outbound while its twin ran at 198 Gbps — a 3.4× gap that pointed away from the network and straight at the host.

The NIC had silently negotiated PCIe 4.0 x4 instead of x16. It wasn't broken. It was sitting in a chipset-connected slot, hardwired to four lanes, sharing bandwidth with the onboard secondary NICs, Wi-Fi, USB, and SATA. The link came up, the cables were fine, the switch was fine — and the card was running at 29% of its capacity. On this class of workstation board, only two of the five physical x16 slots are electrically x16 and CPU-direct; the rest hang off the chipset. The fix was geometry, not money: relocate the NIC to a CPU-direct x16 slot and flip a BIOS bifurcation setting. The leg jumped from 58 Gbps to 198 Gbps — a 3.4× improvement with zero new hardware.

The lesson generalises to any commodity multi-GPU build: a link being *up* tells you nothing about its lane width. Read the negotiated PCIe link state directly, because a starved card looks identical to a healthy one until you measure it.

The Real Ceiling: NCCL Over RoCE vs Local PCIe

With the fabric healthy, the point was to characterise what collective GPU workloads actually see. Inter-node single-direction GPU-to-GPU transfers reached 22.4–23.5 GB/s; two-rank all-reduce reached ~21–23 GB/s. For contrast, *local* intra-node GPU-to-GPU peer-to-peer over PCIe ran at ~50 GB/s — roughly 2× the inter-node ceiling. That ratio is the real subject of the exercise: a 200 Gbps RoCE fabric delivers something close to half of local PCIe peer-to-peer for collectives, which is the number that should drive parallelism decisions (and it framed an estimate that a future 400 Gbps direct-attach upgrade would land around 42–47 GB/s, with two 400G rails preferable to a single 800G link on this PCIe generation).

iperf3 verification across every leg confirmed line rate on most paths but surfaced a second, subtler ceiling: one node-to-node leg capped at ~113 Gbit/s (57% of line rate) while every host-facing path ran at 99%. That isolates the remaining bottleneck as a *host-to-host* condition — NUMA/IRQ affinity, MTU, or ring-buffer asymmetry — rather than a spine-switch fault. Naming where a ceiling *isn't* is as valuable as finding where it is.

Descending to the Physical Layer

A firmware-and-optics audit closed the loop. Firmware was already current (the OEM build actually led the public catalogue), so there was nothing to flash — but the optical diagnostics were the interesting part. The 200GBASE-SR4 link carries four lanes of 50G PAM4. Reading the optical module's per-lane diagnostics and correlating them against the link's forward-error-correction counters told a precise story:

The two weakest lanes received about 0.4–0.5 dB less light — and absorbed dramatically more corrected bits. Lane 4, getting half a dB less light than lane 2, ate roughly 150× more corrected bits (212 million vs 1.4 million). This is PAM4's signature: with four amplitude levels instead of two, a fraction of a dB of optical margin multiplies the raw bit-error rate by close to an order of magnitude.

And yet the link held a flawless full 200 Gb/s — zero link-down events, zero thermal events, a handful of thousand receive drops across hundreds of gigabytes transferred (on the order of 1e-8, operationally invisible). RS-FEC silently absorbed the entire lane asymmetry. The transmit side was uniform; the imbalance was purely receive-side optical.

Why This Matters

That last point is the real lesson, and it cuts against intuition: a link with one lane bleeding 212 million corrected bits ran perfectly, because forward error correction is doing exactly its job — masking marginal physical-layer behaviour completely. The fault is invisible at every layer above the optics. You only see it if you read the per-lane counters and line them up against optical power. On commodity hardware, fabric characterisation isn't one measurement; it's a descent from PCIe topology through NIC firmware to the optical physical layer, and a problem can hide cleanly at any one of those layers while everything above it reports green.