How frontier models decode

Every setup on this page is a mix of these four. They differ in what each chip holds and what has to travel between chips.

The default in almost every open model card. Eight GPUs share a fast switch, and each holds an eighth of every layer. After each layer they add their partial results together.

To fit a model too big for one server, chain servers together, each holding a run of layers. It fits the model but makes no single token faster. The leaked description of GPT-4 (2023) is the only reported layout for any OpenAI model, and it looked like this.

Google's TPUs are wired as a 3D grid. For its PaLM model, Google published a layout in which every weight matrix is cut along two of the grid's axes, which keeps traffic low as chips are added. How Gemini is laid out has not been published.

The big mixture-of-experts models are served on dozens to hundreds of GPUs acting as one. Each GPU handles attention for its own set of users and holds just a few of the experts. In every layer, each token travels to the GPUs holding the experts it needs, and the results travel back.

Reading the prompt (prefill) is one big burst of maths. Writing the answer (decode) is one token at a time. Most labs that describe their stack now give the two jobs separate machines and pass the KV cache from one to the other.

Some labs split decode itself: attention on one set of machines, experts on another, with batches bouncing between them so neither side waits.

The chips that work together on a single copy of a model, with the layout each uses. Blue entries are what model cards recommend; gold entries are deployments that labs have described.

The long-form report has the detail, a citation for every claim, and where the sources disagree.