How data center networks changed
Fifteen years ago, most servers in a data center had a 1Gbps network card, and the network between racks was built on a bet that they wouldn't all use it at the same time. Today, a single GPU has an 800Gbps network card of its own, and the network around it is built so that every GPU can send at full speed, all the time. Almost everything in between changed too: how the switches are wired, the protocols on the wire, how many network cards a server has, and who makes the software that runs the switches.
This page follows those changes one piece at a time, starting from what someone coming from general compute already knows: a server, its network card, and the switch at the top of the rack. Each section starts with how things used to work and ends with what an AI cluster does today. It's a companion to my notes on the NVL72, which cover what's inside the rack. This page covers the network around it.
On this page
The pieces
A server talks to the network through its network card, the NIC. A NIC has two sides. On one side, it plugs into the server over PCIe, the same bus that GPUs and SSDs use, which on modern servers goes straight to the CPU. On the other side, it has one or more ports, where the cable to the switch plugs in.
Both sides are made of lanes. A PCIe slot usually has 16, and a 400Gbps port has 4 or 8. The NIC can't go faster than the slower of its two sides. That's why each time NICs doubled in speed, servers needed a new PCIe generation: a 400Gbps NIC needs PCIe 5, and an 800Gbps NIC needs PCIe 6.
Pick a PCIe generation and a port speed, and see which side holds the NIC back. Faster lanes are drawn thicker. The bar shows how much of the other side you actually get.
Between the NIC and the switch is a cable. When I started, it was the same Cat5 you'd find in an office. Today, each port is a metal socket called a cage, and what goes into it is a module. For a long run, the module is a transceiver: it turns the electrical signal into light and sends it over fiber. Inside a rack, it's often still copper: a direct-attach copper cable, or DAC, with the modules built into its ends. It's cheaper, uses almost no power, and has fewer parts that can fail. Fiber isn't faster than copper. It reaches further: each time speeds go up, copper's reach gets shorter, and today it's down to about three meters.
Modules come in a handful of shapes, and their names mostly tell you how many lanes they carry: SFP has one, QSFP (quad) four, and QSFP-DD (double density) and OSFP eight. A number after the name gives the lane speed, rounded up a little: a QSFP28 has four lanes that run at up to 28Gbps and carry 25Gbps of Ethernet each, so 100Gbps.
A NIC can have more than one port, and a server can have more than one NIC. To use two ports as one, you bond them: the operating system presents a single interface with a single address. The bond picks a link for each connection by hashing its addresses and ports, and the connection stays there. Two 200Gbps ports give you 400Gbps in total, but any one connection still tops out at 200Gbps. What bonding gives you is redundancy: if a link dies, its connections move to the other one and keep going.
A port isn't always one port, either. Switch ports have lanes too, and the switch's software can split them: a 400Gbps port with eight lanes can become four 100Gbps ports, through a breakout cable that fans out to four servers. It also works the other way. NVIDIA's Quantum-2 InfiniBand switch has 32 cages, but each one carries two 400Gbps ports, for 64 in total. So when someone tells you how many ports a switch has, ask whether they're counting cages or ports.
All those cables end at a switch at the top of the rack: the top-of-rack switch, or ToR. A typical one has 48 ports for the servers below it, plus a few faster ports going up to the rest of the network. That might be 48 ports of 1Gbps with four 10Gbps uplinks, or, a few generations later, 48 ports of 25Gbps with eight 100Gbps uplinks. Add up each side and they don't match: the servers can send more than the uplinks can carry. That gap has a name, oversubscription, and it gets its own section further down.
Some racks have two ToRs, and each server's two NICs go to a different one, so the rack stays connected when a switch dies or gets upgraded. A second switch in every rack is expensive, though: at Roblox, we looked at adding one, and it cost too much. With a single ToR, when the switch dies or needs an upgrade, every server in the rack goes offline.
In the racks built for AI, which come later on this page, the GPUs' network cards skip the ToR entirely, and connect to switches in racks of their own.
Leaf and spine
For most of my career, the network above the ToR was someone else's problem. My servers plugged into the switch at the top of the rack, and whatever happened past it was the network team's job. As long as the bandwidth was there, I never had to understand it. AI changed that. When thousands of GPUs exchange data all the time, the shape of the network decides how fast a job runs, and I had to learn the layer I'd been ignoring.
Into the early 2010s, many data center networks were built like a tree. ToRs sat at the bottom, a few aggregation switches each connected a group of racks, and a pair of very large, very expensive core switches sat at the top. That shape fit the traffic of the time, which mostly went north/south: a request came in from the internet, a server answered, and the response went back out.
It had three problems. Traffic between racks under different aggregation switches climbed to the core and back, through the most oversubscribed links. Spanning tree, the protocol that keeps Ethernet frames from looping, switched off the redundant links, so you paid for links that sat idle. And the tree grew at the top: more capacity meant a bigger core, and losing the core took out a large part of the data center. Then MapReduce, distributed storage and microservices made servers talk to each other more than to users.
The fix was to stop building a tree. In a leaf and spine network, the ToRs become leaves, and above them sits a row of spine switches, all the same size. Two rules define it: every leaf connects to every spine, and leaves never connect to each other, and neither do spines. Spines don't need each other: every leaf already reaches every spine, so a link between two spines would only make a path longer, and it would take a port that could connect another rack.
Not everyone uses the names the same way. Some teams call the ToR the leaf, and others keep that name for the layer above it, so when someone mentions a leaf, ask which one they mean.
With those two rules, the path between any two racks has the same shape: up from the leaf to a spine, and back down to the other leaf. Every spine is a possible path, and every path is the same length, so it doesn't matter where two servers sit. Every link carries traffic, too. Instead of one large Ethernet network, each link between switches is routed, like a small piece of the internet, often with BGP, the protocol the internet itself runs on. Routing has no trouble using many paths at once, so nothing gets switched off.
To pick a spine, a leaf does what a bond does with its links: it hashes each flow, by its addresses and ports, and sends it up one of the paths. This is called ECMP, for equal-cost multipath. Like bonding, it keeps each flow on one path, so its packets arrive in order. But it also means two large flows can land on the same spine while another one sits idle. That matters a lot for AI, and I'll come back to it.
None of this is new. It's a Clos network, named after Charles Clos, who designed it in 1953 to build large telephone exchanges out of small switches. Data centers adopted it for the same reason: many small, identical, cheaper switches instead of a few big ones. When a spine fails, you lose a slice of capacity, not the data center.
Here are the same eight racks wired both ways. Switch between them and follow the traffic from one rack to another.
This is the part I couldn't work out on my own, and it turns out to be simple arithmetic. Take switches with 64 ports each, and a network where nothing is oversubscribed. Each leaf uses half its ports for servers and half to go up: 32 down, 32 up. Each of those 32 uplinks goes to a different spine, so you need 32 spines. Each spine has 64 ports, one per leaf, so you can have at most 64 leaves. That's 64 racks of 32 servers, or 2,048 servers, connected by 96 switches.
Two numbers decide everything. The uplinks on a leaf set how many spines you need, and the ports on a spine set how many racks you can connect. Accept some oversubscription, say 48 ports down and 16 up, and you need only 16 spines, while the same 64 racks now hold 3,072 servers.
The number of spines also decides how much a failure costs. With 4 spines, losing one takes away a quarter of the capacity between racks. With 32, it's about 3%.
That's the textbook case, with one kind of switch. Real deployments start from the racks and servers, and pick switches to fit. Enter yours and see what the network looks like.
A single layer of spines stops at 64 racks with 64-port switches. To go further, have each spine keep half its ports for going up. That small leaf and spine network is a pod, and pods connect through a third layer of switches, often called super-spines. With the same switches, a pod holds 1,024 servers, and the super-spines connect 64 pods: 65,536 servers with no oversubscription, using 5,120 switches. Traffic between pods crosses five switches instead of three, but every path is still the same length. Facebook described theirs in 2014: pods of 48 racks, connected through four planes of spine switches.
When a spine dies, the leaves stop using their links to it, and ECMP moves its flows to the spines that are left. Those flows lose a few packets, but connections survive. The fabric loses capacity until the spine comes back. A leaf is different: it's the only way in and out for its rack, so when it dies, the rack goes offline, the same single-ToR problem as before. At work, we talk about leaf pairs: two leaves per rack, with every server connected to both, so a failure costs the rack half its bandwidth instead of all of it.
Oversubscription
Most networks are built on a bet: not every server will send at full speed at the same time. Traffic is bursty, and a data center runs many different services, from live traffic to analytics, that rarely peak together. So instead of buying uplinks for the worst case, you buy them for a realistic one, and save the difference. The gap between the two is the oversubscription: a rack whose servers can send 2 Tbps, behind 1.2 Tbps of uplinks, like the one in the calculator above, is oversubscribed 1.67 to 1.
The bet has a name, statistical multiplexing, and it holds as long as the traffic is uncorrelated. It loses when everyone sends at once: a large batch job shuffling its data, a backup window, a deploy that restarts every instance of a service together. AI training is the extreme case. Thousands of GPUs exchange their results after every step, in lockstep, so the bet is guaranteed to lose. That's why the networks between GPUs are built non-blocking, with no oversubscription at all.
When the bet loses, packets get dropped. A switch has a small buffer for each port, and when more traffic arrives than the port can send, the buffer fills and the next packets are thrown away. TCP notices and sends them again, but that takes time. When it has to fall back on its timeout, Linux waits at least 200 milliseconds: for a request that normally takes a millisecond, one dropped packet makes it two hundred times slower.
In my experience, live traffic suffered first, along with anything sensitive to latency, like caches and replication between data centers. Drops turned into timeouts, timeouts into retries, and the retries piled onto the traffic that caused the problem.
The drops often happen while the graphs say everything is fine. Monitoring averages a link's traffic over 30 or 60 seconds, but a buffer can fill and overflow in a few milliseconds. These microbursts are how a link at 30% can still drop packets.
Applications can soften the blow by marking their traffic. Every IP packet has a field, DSCP, that says how important it is, and switches configured to trust it give each class its own queue. Live traffic goes first, while analytics and bulk transfers wait, or get dropped first.
Oversubscription doesn't stop at the rack: a server's NIC is shared by everything running on it. At Twitter, Mesos packed containers from different teams onto the same machines, and when they all pushed at once, everyone on the machine paid for it. So we made the network a resource like CPU and memory. Each container got a share of the NIC in proportion to its cores, enforced with tc and fq_codel. A container that sent more than its share got its packets marked with ECN instead of dropped: TCP slows down as if it had lost a packet, but nothing is lost, so there's no 200 millisecond timeout. Kubernetes still doesn't schedule the network the way it schedules CPU and memory.
North/south traffic has always been squeezed the hardest. The spines connect to a few border leaves, and those connect to the routers that lead out of the building, an exit much smaller than the fabric itself. For a long time, that was fine: a single user request fans out into many calls between servers, so most traffic never left the building. AI clusters change that, and I'll come back to it in North/south catches up.
I used to wonder whether non-blocking and lossless meant the same thing. They don't. Non-blocking is about capacity: the network has enough bandwidth for every server to send at full speed. Lossless is about behavior: what a switch does when a queue fills up anyway. A normal switch drops the next packet and lets TCP sort it out. A lossless switch makes the sender wait instead. InfiniBand does it with credits: a switch only sends a packet when the next one has said it has room for it. Ethernet does it with pause frames, called PFC, which tell the previous switch to stop sending a class of traffic for a moment.
GPU networks are built lossless because RDMA, which GPUs use to move data between each other, copes badly with loss. But waiting has a cost too. A paused switch fills up and pauses the one before it, so the congestion spreads backward, until flows that had nothing to do with it get stuck behind it. ECN keeps that rare: switches mark packets before their buffers fill, senders slow down early, and the pauses almost never fire. RoCE, RDMA over Ethernet, relies on it, the same way we did at Twitter.
And non-blocking doesn't mean free of congestion. Two large flows hashed onto the same spine can still overflow its link. When many servers answer one at the same moment, which is called incast, the bottleneck is the last link, the one to that server, and no number of spines helps. A network can be non-blocking and still drop packets, or oversubscribed and lossless.
Here's one congested link, handled three ways. Three servers send to R1, more than its link can carry, and a fourth sends to R2, whose link is idle.
Speed over time
At Twitter, our servers were still on 1Gbps in 2014. I worked on the messaging team, running Kafka and other event buses, and a handful of cores could saturate a server's network card. So we scaled out, until we hit the limits on how many instances a service could have. The move to 10Gbps came in 2015, late. At first, it cut the number of instances we needed by a lot. Then more data went into the queues, more services read from them, and we saturated the network again. Bandwidth gets used as fast as it arrives.
From 1Gbps to 1.6Tbps is a factor of 1,600. Drawn to scale, the speed I started my career with is a sliver:
The year next to each bar is when the standard was published, and servers took a while to follow: 10Gbps was standardized in 2002, thirteen years before it reached ours at Twitter. Switches move first, and their uplinks usually run a step or two ahead of the servers below them.
Every one of those speeds is made of lanes, and the lanes are where the progress happens. Each lane is driven by a SerDes, short for serializer/deserializer: the circuit on a chip that turns data into a single fast stream of bits, and back. Make the SerDes faster, and every port built from it gets faster too.
That explains the odd steps. 10Gbps was a single 10Gbps lane. 40Gbps was four of them side by side, with four times the parts, so it cost a lot more. Then 25Gbps lanes arrived, and a single one gave servers two and a half times the speed of 10Gbps, for much less than two and a half times the price. Many server fleets skipped 40Gbps and went straight to 25Gbps. From there, the steps follow the lanes: 50, 100 and now 200Gbps per lane, grouped by 1, 2, 4 or 8.
Making a lane faster eventually meant changing how bits are put on the wire. Up to 25Gbps per lane, the signal used two levels, one bit per symbol, called NRZ. At 50Gbps, the industry switched to PAM4: four levels, two bits per symbol, twice the speed at the same frequency. Four levels sit closer together than two, so noise causes more errors, and every PAM4 link relies on forward error correction (FEC) to fix them, at the cost of a little latency. Even so, copper's reach shrinks with each generation, which is why only short links stay on copper.
On the server side, PCIe set the pace. A PCIe 3 slot with 16 lanes carries about 126Gbps, and PCIe 3 was the norm in servers from about 2012 until PCIe 4 started arriving in 2019. For most of a decade, that capped server network cards at 100Gbps, whatever the network could do.
Even with the slot out of the way, something has to move all those bytes. At 400Gbps, the kernel's TCP stack can't keep up: every byte gets copied between the kernel and the application, and one core can't match the link. Servers built for AI use RDMA instead, which Protocols covers next.
Today, the fastest links in a data center belong to GPUs. In an NVL72 rack, each GPU gets its own network card for east/west traffic: 400Gbps on GB200, with ConnectX-7, and 800Gbps on GB300, with ConnectX-8. The next generation, Vera Rubin, moves to 1.6Tbps per GPU with ConnectX-9, built on 200Gbps lanes. For the first time, servers have caught up with the switches: a GPU's network card runs as fast as the switch port it plugs into.
At these speeds, the optics become a problem of their own. Every link needs a transceiver at each end, and at 800Gbps and 1.6Tbps, those modules draw enough power to add up to a large share of the network's budget. The next step is to move the optics into the switch package, right next to the chip: co-packaged optics. NVIDIA announced its first co-packaged switches in 2025, and says they're 3.5 times more power efficient than switches with pluggable modules.
Protocols
For most of my career, the protocol was TCP, with a bit of UDP here and there, and BGP between the switches. I first read about RDMA around 2014, and it sounded like something only companies the size of Google could run. Today, it's how GPUs talk to each other, and it changes what the network means for a server.
A normal network request goes through a lot before it reaches the wire. The application calls send(), and the kernel copies the data into its own buffers. Its TCP stack splits the data into packets and keeps track of what's been acknowledged, IP addresses each packet, and the driver hands it to the network card. On the other side, the same steps run in reverse. At 1Gbps, a CPU had time to spare for all of this. At 400Gbps, it doesn't.
There are two ways around it. The first is to skip the kernel: with a framework like DPDK, a program takes over the network card and polls it directly from userland. That removes the system calls and the copies into the kernel, but the CPU still does all the protocol work. It's mostly used by software that processes packets itself, like load balancers. The second is RDMA, remote direct memory access, which goes further: it skips the kernel and moves the protocol into the network card's hardware. The CPU only says what to send, and the card does the rest.
RDMA works in a few steps. The application registers a buffer with the network card: the operating system pins it in place, and the card hands back a key for it. Then it sets up a queue pair for each connection, queues that the application writes to and the card reads from directly, and the two machines swap their buffer addresses and keys.
From then on, there's no send(). The application posts a request on its queue, something like "write these bytes at this address on the other machine, with this key", and goes back to work. The card on the other side writes the bytes straight into the registered buffer, and the remote CPU isn't involved at all. It's like handing someone the address of a locker and its key: from then on, they can drop things off without asking you.
Almost nobody writes this by hand. The API, called verbs, is low level, and applications use libraries on top of it: MPI in scientific computing, and in AI, NCCL, NVIDIA's library for moving data between GPUs. When a training job exchanges its results after every step, between servers it's NCCL posting RDMA writes underneath.
RDMA is one idea that runs on three different stacks, which is where I used to get lost. InfiniBand is the original, and it's a world of its own: it isn't TCP, it isn't IP, and it isn't even Ethernet. It has its own switches, its own addresses, assigned by a central subnet manager, and its own transport, built into the network card. RoCE, for RDMA over Converged Ethernet, takes InfiniBand's transport and carries it over Ethernet instead. Its second version puts it inside UDP packets, so it can be routed like any other IP traffic, and that's what most Ethernet clusters run. iWARP runs RDMA over TCP. It exists, but it's much less common.
So, to my old question: no, they don't all run over TCP. Only the least common one does.
Why two families? InfiniBand was designed from the start as a fabric for clusters: lossless by construction, with credits, and routed centrally by its subnet manager. It has had adaptive routing for years, and with NVIDIA's SHARP, its switches combine the results of a collective operation as the data passes through, so less of it has to travel. For a long time, it was the usual choice for supercomputers and the first large AI clusters.
Ethernet's strength is that it's everywhere. Many vendors build it, most operators already know its tools, and IP routing scales to the largest networks there are. For RDMA, though, it needed work: PFC and ECN to avoid losses, and something better than ECMP to spread traffic. That's what NVIDIA's Spectrum-X and the Ultra Ethernet Consortium's first specification, from 2025, go after, and I'll come back to how in the next section.
Storage rides on the same ideas. NVMe over Fabrics takes the commands a server would send to a local SSD and sends them over the network instead, so a drive in another rack shows up as if it were plugged in. It runs over RDMA for the lowest latency, or over plain TCP. With GPUDirect Storage, data goes from storage straight into a GPU's memory, without stopping in the CPU's memory first.
Here are the stacks side by side, and what a single write looks like over TCP and over RDMA.
An AI cluster today
At Twitter, everything a server did went over the same network: requests from users, calls between services, replication, logs, deploys. Oversubscription was the bet that made that affordable. A GPU server doesn't work that way. It's plugged into several networks, each built for one kind of traffic, and most of its network cards belong to just one of them.
In a GPU cluster, there are usually four:
- The compute network, also called the back-end or east/west, connects GPUs in different racks. Each GPU gets its own network card, and the network is non-blocking and built for RDMA. It's by far the most expensive network in the building.
- The front-end network connects the servers to everything else: requests coming in, the job scheduler, provisioning, the rest of the data center. It's an ordinary data center network, oversubscribed like the ones above.
- The storage network carries training data, model weights and checkpoints. Some designs give it its own fabric, and others run it over the front-end network.
- The out-of-band network, a small 1Gbps one, reaches each server's BMC, the management chip that keeps working when the server's operating system doesn't.
In an NVL72 rack there's one more, NVLink, which connects the rack's 72 GPUs to each other. I wrote about it in my notes on the NVL72.
Why not one big network? Because the traffic is too different. A training step can't finish until the slowest of the GPUs' exchanges does, so one congested link holds up the whole job. Add a checkpoint, with every GPU writing its state to storage at once, or hundreds of servers loading a model's weights, and on a shared network the least urgent traffic would set the pace for the most expensive hardware in the building. Separate networks also let each one be built for its job: the compute network non-blocking and often lossless, the front-end ordinary and oversubscribed, instead of paying compute-network prices for traffic that doesn't need it.
The number of network cards shows the split. A general compute server has one, maybe two. A GB300 tray in an NVL72 rack has five: a ConnectX-8 at 800Gbps for each of its four GPUs, all on the compute network, and a BlueField-3 DPU at 400Gbps for the front-end and storage. An 8-GPU server like NVIDIA's DGX B200 has ten: eight for the GPUs and two DPUs. Over a whole rack, the difference is hard to picture. A rack of 40 general servers at 25Gbps can send 1Tbps. The 72 GPUs of an NVL72 rack can send 57.6Tbps on the compute network alone, more than fifty times as much, and every one of those links leaves the rack on fiber, to a leaf switch somewhere else.
The compute network is a leaf and spine network, with one twist: the GPUs aren't wired to whichever leaf is closest. In most designs, GPU 0 of every server connects to the same leaf switches, GPU 1 to another set, and so on. Each set is a rail. The communication libraries know this, and arrange their exchanges so that most traffic stays on one rail, a single switch away, instead of climbing to the spines. I explained rails, and the trick NCCL uses to keep traffic on them, in my notes on the NVL72.
That leaves the problem I mentioned in leaf and spine: ECMP. Hashing each flow onto one path works when thousands of small flows are in flight, and randomness evens them out. Training traffic is the opposite: each network card carries a handful of huge, long flows, and they all start at the end of a step. Put 8 flows on 8 paths at random, and the chance that no two share a path is about 0.24%. Two flows on the same link get half of it each, and the whole step waits for them. RoCE makes it worse: every packet goes to the same UDP port, 4791, so one of the fields the hash relies on never changes. When Meta wrote about its RoCE network for training, it said ECMP performed poorly for exactly this reason. One of its first fixes was to give each rack twice the uplink capacity it needed, which means paying for twice the network.
The fix is to stop choosing a path per flow. With packet spraying, the switch chooses a path for every packet, usually whichever link is least busy at that moment. Nothing is copied, and each packet still takes one path, but a single flow can now use every path at once, and two flows can't pile onto the same link. Think of a convoy of trucks. ECMP sends the whole convoy down one lane, chosen at the on-ramp. Spraying lets each truck take the emptiest lane.
The catch is order: each path has its own queues, so packets arrive out of order. Classic RoCE treats a gap as a lost packet and sends everything after it again, which would undo the gain. RDMA gets around it: every packet of a write carries enough information to know where in memory it belongs, so the receiving card writes it straight to its place, whatever order it arrives in. Each truck has its delivery address painted on the side. That's what Spectrum-X does, and what Ultra Ethernet's transport is built around.
Here are the two side by side. Four GPUs in one rack send to four GPUs in another, through four spines, and the row under each receiving GPU is its memory, filling as packets arrive.
North/south catches up
At Roblox, most of the network's traffic was small gRPC requests between services, and the links had room to spare. That's the traffic the squeeze at the edge was built for. In an AI cluster, north/south is everything that isn't GPUs talking to GPUs: requests coming in, management, and storage. Some of it is still small. Some of it is hundreds of gigabytes at a time.
Three kinds of transfers are big. When a new model is deployed for inference, every server that will run it downloads the weights from storage, and weights are large: DeepSeek-V3 is about 700GB. Training saves checkpoints, a copy of the model's state written every so often so that a failure doesn't throw away days of work. A checkpoint is much bigger than the weights, because it includes the optimizer's state: with the usual setup, that's 12 bytes per parameter on top of the weights themselves, so several terabytes for a model with a few hundred billion parameters. And inference servers increasingly move the KV cache, the model's working memory for each conversation, out of the GPU when it doesn't fit. For a model like Llama 3.1 70B, that's about 320KB per token, or 40GB for a single conversation of 128,000 tokens.
What makes these transfers expensive isn't the bytes, it's the waiting. While a server downloads weights or a training job writes a checkpoint, its GPUs sit idle, and that adds up quickly: 7,200 GPUs waiting 20 minutes is 2,400 GPU-hours, or about $25,000 at CoreWeave's on-demand price for a GB200, every time a model is deployed.
The link out of each server is rarely the problem. A GB300 tray has 400Gbps of north/south, or 50GB per second, enough to pull a whole 700GB model in about 14 seconds. The trouble is that every server asks at the same moment. When a thousand trays load the same model, they need hundreds of terabytes between them, and two things set the pace: the oversubscribed links above the leaves, and the storage itself. The storage servers have to read all of it from their drives, often with their CPUs in the path.
Most of the fixes avoid the network rather than make it bigger. Servers cache weights on their local drives, so the next deploy of the same model doesn't go back to storage. Some systems load a model from storage once, then copy it from server to server over the compute network, which has eight times the bandwidth per tray. Some clusters give model weights their own storage cluster, so a deploy doesn't compete with training. And training jobs write checkpoints to local drives or CPU memory first, let the GPUs get back to work, and copy them to remote storage in the background, which PyTorch's asynchronous save does.
The network is changing too. Storage traffic is getting the treatment east/west got a few years earlier: RDMA, even for S3, GPUDirect Storage, and less oversubscription between the GPU racks and the storage than the rest of north/south gets. The KV cache is becoming a storage tier of its own: NVIDIA has announced storage platforms built on BlueField-4 to hold it. North/south is still much smaller than east/west, but for the traffic that keeps GPUs waiting, the gap is closing.
Here's the arithmetic for deploying a model on NVL72 racks, with nothing cached: every tray loads its part of the model at the same moment, over its 400Gbps of north/south.
Switches and software
In the places I've worked, the switches came from several vendors, each as one package: its hardware, its operating system, its command line, its support. Running two vendors meant knowing two of everything. This is the part of the network I know the least, and what surprised me most when I looked into it is how much that package has come apart.
Inside, a switch is two machines in one box. The switch chip, an ASIC built for this one job, forwards every packet: it looks up where the packet goes in its tables and sends it out the right port, at full speed on every port at once. Next to it sits an ordinary small computer running the switch's operating system. That's where BGP runs and routes get worked out, and the results get written into the chip's tables. The traffic never touches that CPU: at 51.2Tbps, no CPU could keep up. It's the same split as RDMA, where software sets things up and hardware moves the bytes. Networking people call the two halves the control plane and the data plane.
Cisco and Juniper long built their switches around chips they designed themselves. Today, most data center switches use chips bought from a chip company, what the industry calls merchant silicon. Broadcom's Tomahawk family is one: the first, in 2014, switched 3.2Tbps, the fifth, in 2022, 51.2Tbps, and the sixth, shipping since 2025, 102.4Tbps. NVIDIA's Spectrum family came with its acquisition of Mellanox in 2020. Spectrum-4 also switches 51.2Tbps, and Spectrum-X pairs Spectrum switches with NVIDIA's SuperNICs for the spraying from An AI cluster today. A chip's total bandwidth has doubled about every two years, and that's what made a 64-port switch at 800Gbps possible: more ports per switch means fewer layers to connect the same racks.
Once the chips were for sale, the software could come apart from the hardware too. The best-known example is SONiC, an operating system for switches that Microsoft built to run Azure's network and open-sourced in 2016. It moved to the Linux Foundation in 2022, and it's now backed by the companies that make the chips and the switches, along with large operators like Google.
Under the hood, SONiC is Debian Linux. The piece that makes it work across vendors is SAI, the Switch Abstraction Interface: one common set of calls for programming a switch chip. Each chip maker implements SAI for its own chips, so the same operating system runs on chips from Broadcom, NVIDIA, Cisco or Marvell. That's what the vendor packages I knew couldn't offer: one operating system, one set of commands and one way to monitor, across hardware bought wherever it's available. And because it's Linux, a switch can be run like any other server.
Glossary
- BGP
- The routing protocol the internet runs on, also used between the switches of a data center.
- Bonding
- Joining two or more ports into one interface with one address. Each flow stays on one link.
- Breakout
- A cable that splits one fast port into several slower ones, like one 400Gbps port into four 100Gbps.
- Cage
- The metal socket on a NIC or switch that a module or cable plugs into.
- Clos network
- A network built from many small, identical switches in layers, named after Charles Clos, who designed it for telephone exchanges in 1953. Leaf and spine is one.
- Control plane, data plane
- In a switch, the software on its CPU that decides where packets go, and the chip that moves them.
- DAC
- Direct-attach copper, a copper cable with the modules built into its ends. Cheap and short, for links inside a rack.
- DPU
- A network card with its own computer on the board. In GPU servers, a BlueField, for north/south traffic and storage.
- ECMP
- Equal-cost multipath. A switch hashes each flow onto one of several equal paths.
- ECN
- Explicit congestion notification. A switch marks packets when its queue starts to grow, and senders slow down when they see the marks.
- FEC
- Forward error correction, extra bits that let the receiver fix errors without a resend. Every PAM4 link uses it.
- Incast
- Many servers answering one at the same moment, overflowing the last link.
- InfiniBand
- A complete network stack built for clusters, with its own links, switches and addresses, lossless by design.
- KV cache
- A model's working memory for a conversation, kept so it isn't recomputed at every token.
- Lane
- One electrical or optical channel. Ports and PCIe slots are made of lanes.
- Leaf, spine
- The two layers of a leaf and spine network. Leaves connect the racks, and every leaf connects to every spine.
- Lossless
- A network that doesn't drop packets when it's congested, and makes senders wait instead.
- Merchant silicon
- Switch chips sold by chip companies to switch makers, like Broadcom's Tomahawk.
- NIC
- Network interface card, how a server connects to the network.
- Non-blocking
- A network with enough capacity for every server to send at full speed at once.
- NRZ, PAM4
- Two ways to put bits on a wire: two signal levels for one bit per symbol, or four levels for two bits.
- NVMe over Fabrics
- NVMe commands sent over the network, so a remote drive looks local.
- Oversubscription
- Giving a layer less capacity going up than coming in, betting that not everyone sends at once. Written as a ratio, like 3:1.
- Packet spraying
- Choosing a path for every packet instead of every flow, so one flow can use all the paths.
- PFC
- Priority flow control, Ethernet pause frames that make the switch upstream wait, per class of traffic.
- Rail
- The leaf switches that the same GPU of every server connects to: GPU 0 to rail 0, and so on.
- RDMA
- Remote direct memory access. The network card reads and writes memory itself, with no CPU, kernel or copies.
- RoCE
- RDMA over Converged Ethernet: InfiniBand's transport carried over Ethernet. RoCEv2 puts it in UDP over IP, so it can be routed.
- SAI
- Switch Abstraction Interface, one common set of calls for programming a switch chip, which each chip maker implements.
- SONiC
- An open-source operating system for switches, started at Microsoft and now hosted by the Linux Foundation.
- Super-spine
- A third layer of switches that connects groups of leaves and spines, called pods.
- ToR
- Top-of-rack switch, the switch every server in a rack plugs into. In leaf and spine, it's the leaf.
- Transceiver
- A module that plugs into a cage and turns electrical signals into light. Also called optics.
Further reading
- A Study of Non-Blocking Switching Networks, by Charles Clos, 1953: where leaf and spine comes from, designed for telephone exchanges.
- A Scalable, Commodity Data Center Network Architecture, by Mohammad Al-Fares, Alexander Loukissas and Amin Vahdat, 2008: the fat-tree paper that brought Clos networks to data centers.
- Introducing data center fabric, from Facebook, 2014: pods, spine planes, and why traffic between machines dwarfs traffic to the internet.
- Jupiter Rising, from Google, 2015: a decade of Clos networks in Google's data centers.
- RFC 7938: using BGP, the internet's routing protocol, inside a data center.
- Congestion Control for Large-Scale RDMA Deployments, from Microsoft, 2015: DCQCN, how ECN and PFC work together for RoCE.
- RoCE networks for distributed AI training at scale, from Meta, 2024: building a network for training, and what went wrong with ECMP.
- Turbocharging generative AI workloads with Spectrum-X, from NVIDIA: adaptive routing and putting sprayed packets back in order.
- The Ultra Ethernet specification: the new transport for AI over Ethernet.
- SONiC architecture: the containers, the database and SAI, from the project itself.
- My notes on the NVL72: the rack this network connects.