Franck Cuny

How an NVL72 rack works

Last updated

For about fifteen years, a compute rack in a data center looked roughly the same. It held 35 to 40 servers, each one rack unit tall, with one or two CPU sockets and one or two network cards. Each server connected to one or two top-of-rack switches, for redundancy, and those switches connected up to the rest of the network. The whole rack drew around 10 kW, and fans cooled it with air.

An NVL72 rack breaks almost every one of those assumptions.

NVL stands for NVLink, and 72 is the number of GPUs in the rack. There are no servers. Instead there are 18 compute trays, each with 2 Grace CPUs and 4 GPUs. Each tray has a network card per GPU, plus one or two DPUs for storage and management, depending on the design.

Nine more trays hold NVLink switches, which connect every GPU to every other GPU in the rack. The switches you'd expect at the top of the rack aren't there. Only a pair of small management switches remains, and the switches for traffic between racks live in racks of their own. The rack draws around 120 kW on GB200 and up to 142 kW on GB300, more than ten times the old one, and it's liquid cooled.

So you stop thinking of the rack as a place where you put servers. The rack is the computer: it's what you schedule work on, what a single failure can disrupt, and what a building has to be designed around.

I came to these racks from general compute, and they were a very different beast. This is the page I wish I'd had while learning them: what's inside, how the GPUs talk to each other and to the rest of the data center, and what happens when something breaks.

On this page

What's in the rack

Here's the rack from the front. Select any part to see what's inside. The order varies a little by vendor, but the shape is the same: the NVLink switch trays in the middle, compute trays above and below them, and power shelves at the top and bottom.

NVIDIA calls each machine a tray. I was used to servers, and sometimes sleds, so the word took a while to stick. It fits, though: a tray isn't a server you can reason about on its own, because its GPUs are wired to every other tray in the rack through the switch trays.

On the dual-socket servers I worked with, the board layout was inconsistent from one model to the next. Sometimes both NICs hung off the same socket, sometimes each socket had its own. Sometimes all the NVMe drives sat on the same socket as the NICs, sometimes not. That made every model hard to tune for performance, and benchmarks hard to reproduce or to compare from one generation to the next. If you dual source your servers, as hyperscalers often do, even two machines of the same generation can perform and behave differently.

The compute tray is clearly designed on purpose. It's symmetric: each Grace has its own two GPUs, each GPU has its own network card, and on GB300 each network card has its own drive. The two Graces are joined by a CLink, and that link is what lets GB300 get away with a single DPU for the whole tray. From what I've seen so far, following NVIDIA's reference design also narrows the gap between vendors. They still differ on details like the number of DPUs or drives, but the GPUs, NVLink and network cards are laid out the same way.

The DPU took me the longest to understand. A DPU, here a BlueField-3, is a network card with a computer on the board: ARM cores, memory, and its own operating system. In this rack it carries the tray's storage and management traffic, the north/south side.

What's easy to miss is that the operating system is there by default. BlueField-3 cards sold as DPUs ship in DPU mode: the ARM side owns the network card, and the host's traffic goes through a virtual switch running on it. You can switch the card to NIC mode, which turns the ARM cores off and makes it behave like a regular network card.

Running that second operating system makes sense when the host isn't yours. A GPU cloud can hand a whole tray to a customer and keep networking, storage and isolation on the DPU, out of the customer's reach. When you own both the hardware and the workload, I find it easier to treat the DPU as a network card. Otherwise there's a second operating system on every tray, and someone has to own it: the network team, or the compute team? I wrote more about that question in Who owns the software?

Three networks

A GPU in this rack sits on three networks, and each one has its own job:

There's also a small fourth one, the 1 Gb/s out-of-band network for the BMCs. Its switches are the only ones inside the rack.

From the software's side, NVLink doesn't feel like a network. GPUs in the rack share one address space: a CUDA kernel can load and store into another GPU's memory directly, the same way a thread on one CPU socket reads memory attached to the other socket. There's no network card in the way.

Each GPU has 18 NVLink links, one to each NVSwitch chip in the rack, for 1.8 TB/s in total, or 900 GB/s in each direction. That's nine times what its network card gets on GB300, and eighteen times on GB200. Anything that can stay inside the rack does.

East/west and rails

The part of east/west that took me longest to understand is the rail. It only makes sense once you look at how GPUs talk to each other during training.

A large training job splits the work across thousands of GPUs, and they constantly exchange results: after each step, every GPU needs what the others computed. These exchanges are called collectives, and they follow a regular pattern. The work inside a rack goes over NVLink. Across racks, each GPU mostly talks to the GPU in the same position elsewhere: GPU 0 of a tray with the GPU 0s of other trays, GPU 1 with the GPU 1s, and so on.

A rail is a network built around that pattern. Rail 0 is every GPU 0, in every tray of every rack, and all their network cards connect to the same leaf switches. Rails 1, 2 and 3 work the same way, so with four GPUs per tray there are four rails. In a small cluster one leaf per rail is enough, and two GPUs on the same rail are a single switch apart. Traffic between different rails has to climb to a spine and come back down.

GPU traffic needs a network that never makes anyone wait. Collectives are synchronous: every GPU waits for the slowest transfer, so a single congested link slows down the whole job. That's why these networks are built non-blocking, with as much bandwidth going up to the spines as comes in from the GPUs. They're also lossless: InfiniBand only sends a packet when the next switch has room for it, so nothing gets dropped.

The spine is where flows from different leaves meet, and where they can collide on the same link. In a lossless network, a collision doesn't drop packets. The congestion backs up instead, and slows down everything behind it. So the less traffic crosses the spine, the better.

That's what PXN (PCI × NVLink) is for. When a GPU has data for a GPU on another rail, NCCL first copies it over NVLink to the GPU in the same tray that sits on the destination's rail, and that GPU sends it through its own network card. The data stays on one rail and never touches a spine. NCCL also bundles the messages that several GPUs send to the same destination into one. Some researchers take the idea further and drop the spine entirely: rail-only networks send all cross-rail traffic over NVLink, for a fraction of the network cost.

Pick a GPU to send from, then one to send to, and compare the two ways the data can leave the rack:

    The diagram shows the smallest version: one leaf per rail, shared by two racks. A bigger cluster keeps the same shape. In NVIDIA's GB200 SuperPOD, built on InfiniBand, a unit is 8 racks, or 576 GPUs, and each of the four rails gets its own group of 8 leaves and 6 spines. Up to 16 of those units connect through a core layer, for 9,216 GPUs.

    On GB300, NVIDIA's design adds one more idea: planes. Each ConnectX-8 port runs at 800 Gb/s and is split into two 400 Gb/s links, and each link goes to a different copy of the rail network, called a plane. It's the same rails, leaves and spines, built twice, and every GPU has a path into both.

    The reasons are failure and scale. With a single link per GPU, a bad cable, port or leaf cuts that GPU off from the network. With two planes, the GPU loses half its bandwidth instead, and the job can keep running, slower. And 400 Gb/s ports mean twice as many ports per switch, so the same switches build a bigger network with fewer tiers.

    North/south

    North/south is everything that isn't GPUs talking to GPUs: reading training data, loading model weights, writing checkpoints, managing the trays, and the requests that come in when the rack serves a model. It goes through the DPU. On GB300, its two 400 Gb/s ports connect to two separate switches, so the tray stays reachable if one fails. In NVIDIA's designs this network is Ethernet, even when east/west is InfiniBand.

    It also gets much less bandwidth than east/west: 800 Gb/s for the whole tray, shared by four GPUs, against 800 Gb/s for each GPU on the compute network.

    Management comes in two kinds. Out-of-band management reaches each tray's BMC through the small 1 Gb/s switches inside the rack, and keeps working when the tray's operating system doesn't. In-band management, like provisioning or the job scheduler, rides north/south with everything else.

    The switches for north/south and for management are still cooled by air, like the power supplies. We're starting to look at liquid cooling for them, but that changes a number of things around power.

    Each step out is much slower

    NVLink comes with the same lesson as NUMA: placement matters, and you want data next to the compute that uses it. The difference is how steep it gets. On a dual-socket CPU server, remote memory gives you something like half to two thirds of the local bandwidth. On a GPU, every step out costs close to an order of magnitude.

    Here's how fast one GPU can pull data in from each place, drawn to scale. Change the amount of data to see what each step costs in time, or race them:

    Another GPU in the rack is nine times slower than the GPU's own memory. A GPU in another rack is 80 times slower on GB300, and 160 times on GB200. Storage and everything outside the cluster come last, at 320 times. That rung didn't move between generations: GB300 doubled east/west, but in NVIDIA's designs north/south is still 800 Gb/s per tray, on both.

    That's why jobs are laid out the way they are. The chattiest work, like tensor parallelism, stays inside the rack on NVLink, and only what has to cross racks goes east/west. It's also why getting data in and out, like loading weights or writing checkpoints, needs planning of its own.

    When something breaks

    In general compute, a failed server was a small event. We happily kept around 10% of the capacity free for maintenance and failures, planned or not, and a dead machine meant one server less for a while. With GPUs, we can't afford that: it's too expensive to have GPUs sitting there doing nothing. So every GPU that goes offline is capacity you don't get back. And in an NVL72 rack, a failure takes more GPUs offline than the one that broke.

    Pick something that breaks, and step through what happens: what fails, what you drain, and what the repair takes down.

    of 72 GPUs offline

    For a GPU, the numbers go 1, 4, 72. One GPU fails, and you find out from an Xid error or a health check. You drain its tray to move the workloads off it: draining a single GPU is possible, but in practice it's not good enough, and not just where I work. The repair usually means a new tray, and that's where the whole rack comes in.

    To replace a tray, NVIDIA's maintenance flow first takes it out of its NVLink partition, a group of GPUs allowed to talk to each other over NVLink. By default the whole rack is one partition, and changing a partition is, in NVIDIA's words, "disruptive to workloads". Split the rack into smaller partitions and only the affected one pauses. The software is improving, too: swapping a tray used to often require restarting the whole NVLink fabric, and the 2.0 release of NVIDIA's GB200 software removed that step. A switch tray is different. Every partition depends on all nine of them, so smaller partitions don't shrink that blast radius.

    What that pause costs depends on what runs on the rack. An inference service runs many small, independent replicas: when a rack drains, its traffic moves to replicas elsewhere, and the service keeps going with less capacity, as long as there's spare capacity somewhere to absorb it. Training is different. A job that spans racks is synchronous, so when one rack leaves, the whole job stops and restarts from its last checkpoint, and every other GPU in it waits. Inference degrades; training stops. I put numbers on what that costs in the training section of my SLO note.

    That's also why you'd rather replace several trays at once. Every repair costs a pause, so you batch them, and the drained trays wait for their turn.

    Planned maintenance has the same problem. Even a kernel upgrade is a challenge: rebooting a tray takes four GPUs offline at once, and the reboot itself is slow, with all the firmware it goes through. In general compute, rolling a new kernel across the fleet was routine. Here, every reboot is expensive, and that's a problem the industry still has to solve.

    The rack is the unit you buy

    NVIDIA publishes reference architectures for building clusters out of these racks, and each one is built around a scaling unit: the block that everything else is sized for. The network switches, the storage, the management nodes and the power are all specified for one unit, and you grow by adding units. In NVIDIA's GB200 SuperPOD, a unit is eight racks, or 576 GPUs, and a cluster can grow to 16 of them. In its GB300 reference architecture for enterprises, a unit is a single rack. As NVIDIA puts it, customers "scale by rack, not by four-node increments."

    It's the intro's idea again, seen from the buyer's side: you don't add servers to a cluster anymore, you add racks.

    On the data center floor, a unit is a row of racks. The NVL72 racks sit in the middle, with storage on one side, general compute for the control plane and CPU workers on the other, and a CDU at each end for the liquid cooling. What threw me off at first is the network: the switches live in racks of their own, and there are a lot of them. NVIDIA's GB300 design calls for two large switches per NVL72 rack for the CPUs and storage, and up to twelve more for the GPU network. The general compute racks I was used to had one or two switches at the top.

    Pick a kind of traffic and follow it along the row. Which side holds which switches varies; here east/west is on the left:

      Each unit also needs the rest of the cluster around it. NVIDIA's GB300 design assumes a control plane of 12 management nodes, in the general compute racks, to run software like Kubernetes, Slurm and NVIDIA's own management tools. Storage is sized per unit too: for one GB200 unit, NVIDIA gives two tiers, 40 or 125 GB/s of read bandwidth, with writes at about half. Spread over 576 GPUs, that's well under 1 GB/s each, against the 25 GB/s of north/south bandwidth each GPU has. When every GPU reads or writes at once, as with a checkpoint, the storage sets the pace, not the network.

      The building has to keep up as well. NVIDIA asks for a data center built to Uptime Institute Tier 3, with concurrent maintainability: any part of the power and cooling can be serviced without shutting the racks down.

      NVIDIA calls its reference architectures best practices: the design its partners build from, and, for the GB300 one, the architecture NVIDIA uses for its own training and research clusters. In practice, staying close to it is also the easiest path, from buying the racks to running them with NVIDIA's software.

      Glossary

      CDU
      Coolant distribution unit: moves heat from the racks' liquid loop to the building's water. One at each end of a row.
      CLink
      The link between the two Grace CPUs in a tray.
      ConnectX (CX-7, CX-8)
      The network card for east/west, one per GPU. The CX-8 is also a PCIe switch.
      DPU
      A network card with its own computer on the board: ARM cores, memory, and an operating system. Here a BlueField-3, for north/south.
      East/west
      Traffic between GPUs in different racks. The scale-out network.
      Grace
      NVIDIA's ARM CPU. Two per tray, each with two GPUs.
      North/south
      Everything that isn't GPU to GPU: storage, management, and requests from users.
      NVLink
      The links between GPUs in a rack, through the switch trays. The scale-up network.
      NVLink-C2C
      The link between a Grace CPU and its two GPUs.
      NVLink partition
      A group of GPUs allowed to talk to each other over NVLink. By default, the whole rack.
      NVSwitch
      The switch chip behind NVLink. Two per switch tray, 18 per rack.
      Plane
      A full copy of the rail network. NVIDIA's GB300 designs build two, and each GPU has a link into both.
      PXN
      NCCL's trick: move data over NVLink to a GPU on the right rail before sending it, so it never touches a spine.
      Rail
      Every GPU in the same position, in every tray, with their network cards on the same leaf switches.
      Scalable unit
      The block a reference architecture sizes everything around: 8 racks in GB200's SuperPOD, 1 rack in NVIDIA's GB300 design.
      Tray
      NVIDIA's word for a machine in the rack: a compute tray or a switch tray.
      Xid
      An error the GPU driver reports. Usually how you find out a GPU failed.

      Further reading