AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-46.html

Cluster the Damn Boxes

(Spine Ch. 46.)

“Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.” Andrew S. Tanenbaum and David J. Wetherall, Computer Networks, 5th edition (2011)

Three days in the MLX profiler. I read kernel timings until my eyes went dry, certain there was a scheduling bug in the distributed backend, because the four-box run was measurably slower than the one-box run and that is not how adding hardware is supposed to work. Two of my five machines sit on a Thunderbolt 5 bridge I had personally measured at 43.8 Gbit/s. The other three do not. I never once checked which layers had landed on which link, because checking would have meant admitting the profiler was the wrong tool. The thing that fixed it cost less than the coffee I drank finding it.

I have five Apple Silicon machines on a desk and a shelf: two M3 Ultras, an M1 Ultra, an M4 Max laptop and an M3 Max, 36GB to 96GB apiece. For about a week I told people I had a cluster. What I actually had was five computers on the same subnet and a very nice feeling about myself. The tell came when I asked a simple question: which of these boxes is holding layers 40 through 60 right now, and how long does the activation tensor spend on the wire getting there? I could not answer it.

Bottom line: Putting one model across many machines is a bandwidth-and-latency problem wearing a compute costume. The unified memory adds up (this is the real reason to do it, you get to load a model that does not fit on one box). The tokens per second usually do not, at least not for a single request. Pipeline parallel buys you throughput on many concurrent requests, not speed on one. Tensor parallel buys you speed on one request and eats your interconnect alive. And most of what looks clustered (a Swarm, a Kubernetes deployment, a Ray cluster, a load balancer with four backends) is not one model on many machines at all. It is many copies of one model, which is a much easier idea, and a better one, if it fits.

When it bites

The pattern

There are exactly two ways to split one model across machines, and everything else is packaging.

Pipeline parallel. Cut the model by layer: box A holds layers 0 to 20, box B holds 21 to 40, and so on. Each token walks the chain. The wire carries one activation tensor per hop. The bad news is that while box A is working, boxes B, C and D are idle, so a single request runs at roughly the speed of one box plus the hop cost. That is why Exo’s own numbers show single-request performance falling as you add devices, and multi-request throughput rising almost linearly: 49.3 TPS on one, 95.7 on two, 108.8 on three, a 2.2x speedup for Llama 3.2 3B (Exo Labs, 2025).

Tensor parallel. Cut each layer by weight. Every box does a slice of the same matmul, then they all-reduce the result. Now nobody is idle and a single request actually gets faster. Exo documents 1.8x on two devices and 3.2x on four (github.com/exo-explore/exo). The bill arrives as interconnect: an all-reduce per layer, every layer, every token. This is why vLLM’s tensor-parallel path (Megatron-LM’s algorithm, docs.vllm.ai) assumes NVLink inside a node and pipeline parallelism between nodes.

That split is why Apple’s RDMA work matters more than the raw Thunderbolt number. Bandwidth was rarely the thing killing me; latency was. Apple’s TN3205 puts RDMA over Thunderbolt at under 50 microseconds of memory access latency against roughly 300 before (developer.apple.com, March 2026), shipped as an opt-in you enable from recovery mode with rdma_ctl enable on macOS 26.2. A 6x latency cut on a per-layer all-reduce is the difference between tensor parallel being a benchmark and being a deployment.

My own boxes: iperf3 -P 4 measured 43.8 Gbit/s at 0.58 ms RTT over a Thunderbolt 5 bridge between two of the five hosts, against 9.42 Gbit/s on the 10 GbE the other three use. STREAM triad across all five ran 177.6 to 351.2 GB/s, and per unit of capacity it is worse: bandwidth per gigabyte varies 2.05x, and my two 36GB boxes differ 1.49x while the allocator hands them identical layer counts, so the slower one paces the pipeline. Naive layer placement is wrong on both axes at once. So I modeled placement proportional to the bandwidth available at each boundary and got a 1.20x improvement. Say the next part out loud before anyone quotes you: that 1.20x is a model of stage times, not an end-to-end measurement. It is arithmetic over measured link speeds and measured memory bandwidth. Ch. 7 is the whole argument for why those two things get different words.

All of that went up on 29 July 2026 as a comment on the exo founder’s own open issue, exo-explore/exo #957. A comment. With tables in it. I have caught myself saying “I contributed to exo” in a room, and what I actually did was leave a very long comment on somebody else’s issue, which is the open source equivalent of yelling directions at a truck that is already moving.

The third thing, and the one most people actually need: many copies, not one split model. Kubernetes, Docker Swarm, a Ray Serve deployment, llm-d on top of vLLM. llm-d is a CNCF Sandbox project started by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA, checked 9 September 2026, Kubernetes-native distributed inference wrapped around vLLM. It will happily run your model in ten places at once and never once ask which layer goes where. They replicate and route, they do not split. If your model fits on one box, this is the correct answer and the parallelism chapter you are reading is a distraction. You get linear throughput, independent failure domains, and a rollout story. Everyone wants to brag about how much they can load; almost nobody checks first whether it goes in on one box. That’s what she said, and she was also right about the memory ceiling.

One worked example

Jeff Geerling, December 2025. Four M3 Ultra Mac Studios, 1.5 TB of aggregate unified memory, connected with Thunderbolt 5 and RDMA enabled. He ran things that have no business running outside a datacenter: Qwen3 235B at 8-bit, DeepSeek v3.1 671B at 8-bit, and Kimi K2 Thinking, a trillion-parameter model.

The numbers: 32 tokens/sec on Qwen3 235B and about 30 tokens/sec on Kimi K2 Thinking across the four nodes. Not fast by API standards, but extremely fast for a model that fits on approximately zero consumer machines.

HPL crashed over Thunderbolt without RDMA, and was stable with RDMA on. The interconnect was not “a bit slower” in the broken configuration; it was a source of failures that looked like something else.

Notice whose numbers those are. Geerling’s, not mine. I have measured the wire and I have measured the memory. I have never once measured end-to-end tokens per second on my own five boxes: not one named model, at one named quant, against the single M3 Ultra baseline, written down with a date on it. My recollection is that the placement change helped and the cluster felt quicker, and “felt quicker” is exactly the sentence Ch. 7 exists to beat out of me. Every time I open the terminal to fix that I end up back in the profiler like a man texting an ex.

The quiet failure

The loud failure is the cluster that does not work. You get an error, you fix the error, you move on.

The quiet failure:

You built four replicas and told everyone you built a cluster, and the day a model arrives that does not fit on one box, you find out none of it applies.

Swarm and Kubernetes have no opinion about layer 37. When the 235B lands and you go to shard it, every piece of your “clustering” experience turns out to have been load balancing.

Second quiet failure: you measure the wrong request shape. You benchmark with one request at a time because that is how you use it interactively, you see pipeline parallel make things slower, and you conclude pipeline parallel is broken. It is not broken. It is a throughput technique and you handed it a latency test.

Third: you tune software to fix a wire. Two of my five boxes were on the slow link and holding a third of the layers. I did not find that with a profiler. I found it by drawing five boxes on a napkin.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 7 is where the language rules come from: measured, modeled, and assumed are different claims, and this chapter demonstrates why. Ch. 45 sets up local inference on a single box, the baseline every cluster claim here has to beat. Ch. 47 picks up what clustering does not solve: once the model runs, something still has to decide which model gets the request.

Sources and receipts

Verified:

What I could not verify: