Distributed Inference on Workstation Blackwell, Part 2: Amalgamated System RAM for GPU Dataframe Analytics

Abstract

Part 1 of this programme brought up a two-node, three-GPU workstation-class NVIDIA Blackwell cluster on a 200 Gbps RoCE fabric and demonstrated cross-node pipeline-parallel inference of a 405-billion-parameter language model. Part 2 asks whether the same topology is a viable substrate for the rest of the production machine-learning lifecycle, not just inference. The cluster is retargeted from a vLLM serving stack to a RAPIDS and Dask-CUDA distributed-dataframe stack and exercised against a realistic click-through-rate (CTR) pipeline on the public Criteo click-logs corpus: distributed Parquet ingest, a high-cardinality groupby, global target-encoding of all 26 categorical features, and multi-GPU gradient-boosted-tree training. The pipeline runs end to end across roughly 750 GB of amalgamated system RAM — of which approximately 558 GB is exposed to Dask-CUDA as worker memory — without a single worker failure, OOM, or stall. The amalgamated pool is load-bearing: neither node's RAM is sufficient alone.

1. The Question, Restated

Real production ML is rarely inference alone. A representative CTR pipeline ingests a large tabular corpus, aggregates and encodes high-cardinality categorical features, and trains a gradient-boosted ensemble across multiple GPUs. Each stage has a different resource profile from language-model inference — ingest is storage-bound, groupby and target-encoding are shuffle-bound, tree training is compute-bound — and none of them depend on NVLink or InfiniBand. What they depend on is pooled system RAM and a cross-node transport that does not stall under shuffle pressure. Part 2 measures whether the workstation-class cluster delivers both.

2. System and Corpus

The hardware is unchanged from Part 1: two nodes (Node A with two Blackwell Workstation Edition GPUs, Node B with one Blackwell Max-Q GPU), three GPUs at roughly 96 GB VRAM each (~288 GB aggregate), on the same 200 Gbps RoCE fabric. The stack above CUDA is different: RAPIDS cuDF and dask-cudf, dask-cuda, and XGBoost 3.2 with its Dask integration, with Unified Communication X (UCXX) over TCP as the transport in place of NCCL.

The corpus is a subsample of the public Criteo click logs (redistributed on Hugging Face): approximately 450 million banner-impression rows with 13 numerical features, 26 hashed categorical features, and a binary click label — about 110 GB uncompressed, narrowed to roughly 29 GB of Snappy-compressed Parquet after type-narrowing. Three Dask-CUDA workers, one per GPU, are each given an 80 GB RMM pool and a 200 GB host-memory limit, for an aggregate worker budget of approximately 558 GB.

3. The Benchmark Suite

Four benchmarks exercise the stack along the axes that matter for a real CTR pipeline — each a genuine pipeline step rather than a microbenchmark.

A deliberate reporting discipline runs through the suite. B1 is a streaming scan, not a persist-into-memory operation: partitions are released as soon as their reductions retire, and throughput is reported as a column-pruned effective rate (on-disk corpus size over scan wall-time) rather than a raw disk rate. Parquet metadata-only enumeration is timed separately at 0.85 s — the ratio between the two is the correct signal that the full scan does real work. A metadata-only shape computation masquerading as an ingest number is a common error in distributed-dataframe benchmarks, and is explicitly avoided here.

4. What the Measurements Show

Three findings stand out. First, the three-Blackwell configuration absorbs the entire corpus into a single distributed GPU dataframe and supports end-to-end groupby, target-encoding, and multi-GPU XGBoost training without manual dataset partitioning or any CPU fallback. Second, the fabric is nowhere near the bottleneck: at sub-GB/s sustained cross-node shuffle rates, the 200 Gbps link has very large headroom — the training time is dominated by the quantile-sketch build, not the transport. Third, the UCXX-over-TCP configuration is a conservative baseline; the InfiniBand-capable transport provider, once available to the UCXX wheel chain, is expected to reduce shuffle overhead further without changing the correctness of any result here.

The AUC of 0.7055 is deliberately not the headline. With only the 13 numerical features and no categorical encoding fed into training, it sits below the full-feature Criteo baseline of approximately 0.78; folding in the target encodings lifts it but changes nothing about the fabric or framework story. The headline is that a two-node, Ethernet-connected workstation cluster completes 100 rounds of multi-GPU gradient-boosted-tree training in single-digit seconds, on a workload normally targeted at DGX-class hardware.

Two operational findings were preserved from bring-up. Dask-CUDA workers must have accumulated partition state evicted before XGBoost's quantile-DMatrix builder runs, or GPU-memory exhaustion is the likely failure mode — the scheduler's worker-restart hook is a clean eviction pattern for multi-stage harnesses. And the Dask integration module is no longer auto-imported with the top-level package in XGBoost 3.2, so harnesses carried forward from earlier versions need an explicit import.

5. Position in the Programme

Part 1 established the dense-inference operating point; Part 2 establishes the dataframe-analytics operating point on the same iron. Subsequent parts characterise a 400 Gbps fabric upgrade, UCXX-over-InfiniBand against this TCP baseline, and a full-feature Criteo AUC comparable to published Merlin numbers. All benchmark code, launcher scripts, and raw result JSON are released with the programme's harness repository, so third parties on comparable hardware produce directly comparable result files.

The full paper — methodology, partitioning strategy, per-feature shuffle timings, related work, and acknowledgements — is available in the PDF above.