# Frontier LLM decode deployment: what is publicly known

Frontier LLM decode has split into two regimes. The first is the public "fits on a node" recipe: TP4–TP8 on one 8-GPU server, which is what nearly every model card (Llama, Mistral, Qwen, GLM, MiniMax, Kimi K2.5, Grok-2, gpt-oss) prescribes. The second is the production regime for large MoEs. Where labs have actually disclosed production or showcase deployments, decode runs as **one wide expert-parallel instance of 32–320 accelerators**. Each accelerator is its own data-parallel attention rank (DP-attention), the instance has **no or minimal tensor parallelism**, it sits behind a separate prefill pool (PD disaggregation), and it carries **~30–480 concurrent sequences per accelerator**. **DeepSeek is the canonical example**. Its decode instance went from a 320-H800 design point (V3 report, Dec 2024) to **144 H800 at DP144/EP144** in Feb-2025 production, at ~1,850 tok/s/GPU and 20–22 tok/s/user. It then moved to **one 64–128-GPU NVLink-class "supernode" at DP=EP=64–128** for V4.1 in Oct 2026. Kimi K2 (96 H200 decode), LongCat-Flash (16-GPU minimum, 128-GPU benchmark), Step-3 (32–48 GPUs with attention–FFN disaggregation), ByteDance MegaScale-Infer (~10k-GPU fleet, H20 attention + L40S experts) and Huawei's DeepSeek serving (160 Ascend 910C / EP320) follow the same pattern. The cause is arithmetic. Decode reads all active weights and every sequence's KV cache per step, so it stays memory-bandwidth bound until a batch of roughly **240–280 tokens per weight copy** (dense bf16). For a top-8-of-256 MoE that threshold multiplies by E/k = 32, to **~9k tokens**. That batch is only reachable by pooling many attention ranks behind sharded experts, and the all-to-all traffic that pooling requires is what pushes deployments into rack-scale NVLink domains (GB200/GB300 NVL72), TPU torus slices and Ascend supernodes. The closed labs (OpenAI, Anthropic, Google, xAI, Amazon, Microsoft) **disclose no devices-per-replica, TP/EP degree or batch size for any current model**. They name only systems: GPT-5.5 on GB200 NVL72, Codex-Spark on Cerebras, Claude on TPU/Trainium/NVIDIA, Gemini on TPUs with PD disaggregation, GPT-5.2 on Maia 200. The only detailed closed-model topology in public is the 2023 GPT-4 leak (128 GPUs, TP8×PP16). Published per-GPU DeepSeek-class decode throughput rose about **7x in 18 months**, from ~1,850 tok/s on H800 to 13,386 on GB200 with NVFP4 and ~14k on GB200 at 25 tok/s/user. Throughput falls **10–30x** when the per-user speed target rises from ~25 to ≥200 tok/s, which is why the fastest tiers are moving to SRAM machines (Cerebras, Groq LPX).

*Confidence labels used throughout: **O** = official/first-party (the lab, or a vendor describing its own system); **3P** = third-party measured (framework teams, benchmarkers, hardware vendors running someone else's model); **L** = leak or press report; **E** = estimate (analyst model or arithmetic derived here from cited numbers). Where a closed lab discloses nothing, the tables say "undisclosed" rather than guessing.*

## Decode is bandwidth-bound until MoE makes it network-bound

### Step time is weights plus KV over HBM bandwidth

Google DeepMind's scaling book gives the governing lower bound. A decode step takes at least **(Batch × KV-cache size + Parameter bytes) / total HBM bandwidth**. A fuller form is (B·KV)/BW + max(2·B·Params/FLOPs, Params/BW). The attention term is *always* bandwidth-bound because its arithmetic intensity is ~1 at one new token per sequence, and batching cannot raise it. Only the MLP term can turn compute-bound ([JAX Scaling Book](https://jax-ml.github.io/scaling-book/inference/)). The MLP crossover is the **critical batch B_crit ≈ peak FLOPs / HBM bytes/s**: **~240 tokens on TPU v5e** (1.97e14 / 8.2e11) and **~280 on H100** in bf16. It falls to ~120 with int8 weights and bf16 math, and returns to ~240 with int8 weights and int8 activations ([JAX Scaling Book](https://jax-ml.github.io/scaling-book/inference/)). Newer chips are *more* bandwidth-starved per FLOP. Derived FLOP/byte ratios are ~575 for v6e, ~312 (bf16) or ~623 (fp8) for Ironwood, and ~1,170 for TPU 8i at FP4. Each generation therefore needs a larger batch to escape the memory-bound regime (E, derived from [scaling-book TPU table](https://jax-ml.github.io/scaling-book/tpus/) and [Google TPU 8 deep dive](https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive)).

Three worked floors anchor these bounds against measurements. A dense 70B model in FP8 streams ~70 GB per step, so one 8 TB/s B200 cannot beat **~9 ms/token (~110 tok/s/user) at batch 1**. TP8 raises that ceiling roughly 8x, minus all-reduce latency (E, derived; B200 bandwidth from [getdeploying](https://getdeploying.com/gpus/amd-mi355x-vs-nvidia-b200)). PaLM 540B with int8 weights on 64 TPU v4 has a weight-read floor near 540 GB / (64 × 1.2 TB/s) ≈ 7 ms, yet measured **28.5 ms/token at batch 64**. Low-batch decode at 64-way model parallelism was therefore dominated by communication and latency overhead, consistent with the paper's 14% MFU ([Pope et al.](https://arxiv.org/abs/2211.05102); E for the floor). Meta's Llama 3 405B in FP8 at TP8 on power-capped 2.4 TB/s H100s measured **44.5 ms at batch 1**, roughly double the ~21 ms needed to stream a ~50 GB FP8 shard per GPU. The gap is TP all-reduce and kernel overhead ([Meta CP paper](https://arxiv.org/html/2411.01783v3); E for the decomposition).

### KV-cache bytes per token set the batch a replica can hold

The batch needed to reach B_crit only helps if the KV cache fits, so per-token KV size determines how much batch a device can carry. The table collects the per-token KV figures that appear in the sources.

| Model | Attention type | KV bytes per token (all layers) | Source / label |
|---|---|---|---|
| DeepSeek-V3/R1 | MLA, 576-element latent (512 + 64 RoPE) × 61 layers | **70,272 B** (BF16) | O, [ISCA'25 Insights paper](https://arxiv.org/html/2505.09343) |
| Qwen2.5-72B | GQA | 327,680 B (4.66× DS-V3) | O (cited by DeepSeek), same |
| Llama 3.1 405B | GQA, 8 KV heads, 126 layers | 516,096 B (7.28× DS-V3) | O (cited by DeepSeek), same |
| Llama 3 70B | GQA, 8 KV heads, 80 layers | ~160 kB (int8) | O worked example, [scaling-book applied inference](https://jax-ml.github.io/scaling-book/applied-inference/) |
| gpt-oss-120b | GQA 8 KV heads × d64; 18 full + 18 banded-128 layers | ~36 KiB/token (BF16) for full-attention layers + ~4.5 MiB/seq constant | E from [model card](https://arxiv.org/html/2508.10925) |
| Muse Spark Open 816B | GQA 16 KV heads × d64, 62 layers | ~254 KB (BF16) | E from [HF card](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B) |
| Step-3 | MFA | 512 B per token per layer at 8-bit | O, [Step-3 paper](https://arxiv.org/html/2507.19427v1) |
| DeepSeek V4.1 | compressed sparse attention, head_dim 512 | 528 B (FP8) or 288 B (FP4) per entry | O, [FlashMLA](https://github.com/deepseek-ai/FlashMLA) |
| DeepSeek V4-Pro / V4-Flash at 1M tokens | CSA + HCA + SWA | 10% / 7% of V3.2's KV | O, [V4 report](https://arxiv.org/html/2606.19348) |

At DeepSeek's production mean context of 4,989 tokens, MLA uses ~350 MB per sequence in BF16. With ~88 sequences per decode GPU that is ~31 GB per GPU, which fits on an 80 GB H800 only because EP144 leaves each GPU just 2 routed + 1 shared expert per layer (E, derived from [Day-6 overview](https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md)). The KV size also determines the attention parallelism. **MLA has effectively one shared KV head in its decode (MQA) mode**, so TP-sharding attention replicates the whole latent cache on every TP rank. That is why SGLang, vLLM, Perplexity and DeepSeek itself moved to DP-attention for decode ([LMSYS](https://lmsys.org/blog/2025-05-05-large-scale-ep/); [V3.2 report](https://arxiv.org/html/2512.02556); [DeepInfer Q&A](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/inference_deployment_qa.md)). Under GQA the same logic caps useful attention TP at the number of KV heads ([NVIDIA Helix blog](https://developer.nvidia.com/blog/asking-an-encyclopedia-sized-question-how-to-make-the-world-smarter-with-multi-million-token-real-time-inference)). Google reached the same answer in 2022 for PaLM's multiquery attention. Sharding the KV over *batch* rather than heads raised the maximum context at batch 128 from 660 to **43,000 tokens**, a 32x gain ([Pope et al.](https://arxiv.org/abs/2211.05102)).

### MoE multiplies the critical batch by E/k

For MoE with E experts and k active, each expert sees only B·k/E of the tokens, so **B_crit scales by E/k**. E=16, k=2 gives ~1,920 tokens on v5e ([JAX Scaling Book](https://jax-ml.github.io/scaling-book/inference/)). DeepSeek-R1 (256 routed, top-8, E/k = 32) needs **~9k tokens per expert-parallel group** before expert GEMMs stop being bandwidth-bound (E, derived). Experts are 95–99% of parameters in frontier open MoEs: **97.7% for DeepSeek R1/V3.2 (623 GB FP8), 98.9% for Kimi K2 (948 GB FP8), 96.6% for Qwen3-235B, 95.4% for gpt-oss-120b** ([StorageReview](https://www.storagereview.com/news/nvidia-groq-3-lpx-everything-we-know)). An 8-GPU node can hold neither the weights of the largest models nor a ~9k-sequence KV pool. The solution is to spread experts thin, at 2–8 per GPU, across 32–320 devices and let every device run its own attention batch. The aggregate batch then feeds each expert hundreds of tokens. DeepSeek states this rationale directly: only 8 of 256 experts activate, so cross-node EP raises per-expert batch and cuts per-GPU weight bandwidth ([Day-6](https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md)).

Wide EP turns decode into a networking problem. DeepEP's low-latency kernels on H800 + 400 Gb/s CX7 InfiniBand (128 tokens/GPU, hidden 7168, top-8, FP8 dispatch / BF16 combine) take **77 µs dispatch / 114 µs combine at EP8, 192 / 369 µs at EP128, and 194 / 360 µs at EP256** ([DeepEP README](https://github.com/deepseek-ai/DeepEP/blob/2be0d4f/README.md)). Unoverlapped, ~560 µs per MoE layer over 58 MoE layers is ~32 ms per step. Overlap is therefore mandatory: DeepSeek's two-micro-batch overlap and 5-stage decode pipeline, and zero-SM RDMA hooks. It also explains why ~45–50 ms TPOT was DeepSeek's H800 operating point (E, derived). SemiAnalysis counts **~120 all-to-all collectives per DeepSeek-R1 forward pass**. It finds decode "network-bound on the EP dispatch and combine collectives" between ~75 and 175 tok/s/user, and notes that **NVLink 5 gives 900 GB/s/GPU while 400G RoCE gives ~50 GB/s/GPU, an 18x gap** ([InferenceX, May 2026](https://inferencex.semianalysis.com/blog/gb200-nvl72-vs-b200-disagg-deepseek-r1-fp4-dynamo-trt)). Rack-scale scale-up domains exist to close that gap. NVIDIA's TensorRT-LLM gets **up to 1.8x more output tok/s/GPU at EP32 than at EP8** at 100 tok/s/user on GB200 NVL72 ([NVIDIA Wide-EP blog](https://developer.nvidia.com/blog/scaling-large-moe-models-with-wide-expert-parallelism-on-nvl72-rack-scale-systems)). Two-batch overlap, by contrast, which matters on IB, was *disabled* inside NVL72 because NVLink made communication cheap enough ([LMSYS GB200 Part 1](https://lmsys.org/blog/2025-06-16-gb200-part-1/)).

### Hardware: bandwidth per device and size of the scale-up domain

| Device | HBM / on-chip memory | Mem BW per device | Scale-up domain (decode-relevant) | Label / source |
|---|---|---|---|---|
| H100 (Meta GTI variant, 500 W cap) | 96 GB HBM2e | 2.4 TB/s | 8 GPUs (NVLink); B_crit ~280 | O, [Meta CP paper](https://arxiv.org/html/2411.01783v3); [scaling book](https://jax-ml.github.io/scaling-book/inference/) |
| H200 | 141 GB (capacity per Mistral sizing, E) | 4.8 TB/s | 8 GPUs | 3P, [vLLM](https://vllm.ai/blog/2026-02-03-dsr1-gb200-part1) |
| B200 | 192 GB HBM3e | 8 TB/s | 8 GPUs (HGX) | 3P, [getdeploying](https://getdeploying.com/gpus/amd-mi355x-vs-nvidia-b200) |
| GB200 NVL72 | ~186 GB/GPU | 8 TB/s | 72 GPUs, 130 TB/s NVLink, 900 GB/s/GPU | O, [NVIDIA](https://developer.nvidia.com/blog/scaling-large-moe-models-with-wide-expert-parallelism-on-nvl72-rack-scale-systems) |
| GB300 NVL72 | 20 TB/rack (~278–288 GB/GPU) | 576 TB/s/rack (~8 TB/s/GPU) | 72 GPUs, 130 TB/s | O, [NVIDIA GB300](https://www.nvidia.com/en-us/data-center/gb300-nvl72/) |
| Vera Rubin NVL72 (2H 2026) | 288 GB HBM4 | ~22 TB/s | 72 GPUs, 260 TB/s, 3.6 TB/s/GPU | O prelim., [NVIDIA](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72) |
| AMD MI355X | 288 GB HBM3E | 8 TB/s | 8 GPUs | 3P, [TechRadar](https://www.techradar.com/pro/amd-unveils-puzzling-new-mi355x-ai-gpu-as-it-acknowledges-there-wont-be-any-ai-apu-for-now) |
| AMD Helios (MI455X, 2026) | 432 GB HBM4 | 19.6 TB/s (AMD) vs 23.3 TB/s (StorageReview), conflicting | 72 GPUs, 260 TB/s UALoE | O/3P, [AMD brochure](https://www.amd.com/content/dam/amd/en/documents/solutions/ai/amd-helios-bro.pdf); [StorageReview](https://www.storagereview.com/news/amd-mi455x-and-helios-432gb-hbm4-72-gpu-racks-and-a-real-answer-to-vera-rubin) |
| TPU v5e / v6e | 16 / 32 GiB | 0.82 / 1.6 TB/s | 256-chip 2D torus | O, [scaling book](https://jax-ml.github.io/scaling-book/tpus/) |
| TPU v7 Ironwood | 192 GB | 7.37 TB/s | 64-chip 4×4×4 cube; up to 9,216 via OCS; slices 4–2,048 supported | O, [Google](https://blog.google/products/google-cloud/ironwood-tpu-age-of-inference/); 3P [SemiAnalysis](https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-swing-at-the) |
| TPU 8i (announced Apr 2026) | 288 GB + 384 MB SRAM | 8.6 TB/s | Boardfly, 1,024 active chips, ≤7 hops | O, [Google deep dive](https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive) |
| AWS Trainium2 | 96 GB | ~2.9 TB/s | 64-chip UltraServer (185 TB/s) | O, [AWS Trn2](https://aws.amazon.com/ec2/instance-types/trn2/) |
| AWS Trainium3 | 144 GB HBM3e | 4.9 TB/s | 144-chip UltraServer (706 TB/s) | O, [AWS Trn3](https://aws.amazon.com/ec2/instance-types/trn3/) |
| Microsoft Maia 200 | 216 GB + 272 MB SRAM | 7 TB/s | 4-chip fully-connected tray; Ethernet collectives to 6,144 | O, [Microsoft](https://blogs.microsoft.com/blog/2026/01/26/maia-200-the-ai-accelerator-built-for-inference/) |
| Huawei Ascend 910C | 128 GB (2 dies) | 3.2 TB/s | CloudMatrix384 (UB all-to-all) | 3P, [The Register](https://www.theregister.com/2025/07/29/huawei_rackscale_boogeyman/) |
| Huawei Ascend 950DT (Q4 2026) | 144 GB HiZQ 2.0 | 4 TB/s | Atlas 950 SuperPoD, 8,192 chips | 3P, [Tom's Hardware](https://tomshardware.com/tech-industry/semiconductors/huawei-unveils-ascend-roadmap-backed-by-in-house-hbm) |
| Meta MTIA 400 | n/a (relative specs only) | +51% vs MTIA 300 | 72-device switched backplane | 3P of Meta disclosure, [StorageReview](https://www.storagereview.com/news/metas-mtia-roadmap-four-chip-generations-in-two-years-put-genai-inference-first) |
| Groq 3 LP30 (LPX rack) | 500 MB SRAM/chip; 128 GB per 256-chip rack | 150 TB/s SRAM | 256 LPUs per rack | 3P, [StorageReview](https://www.storagereview.com/news/nvidia-groq-3-lpx-everything-we-know) |
| Cerebras WSE-3 | 44 GB on-wafer SRAM | n/a | per-wafer; multi-wafer pipelines | O, [Cerebras](https://www.cerebras.ai/press-release/cerebras-inference-llama-405b) |

Aggregate HBM bandwidth per scale-up domain is ~38 TB/s for HGX H200, ~64 TB/s for HGX B200, 576 TB/s for GB300 NVL72, ~1,580 TB/s for Vera Rubin NVL72 and 706 TB/s for a Trn3 UltraServer. A ~600 GB FP8 DeepSeek-class MoE fits in one NVL72 with ample KV room, but its FFN alone needs **~5 Groq LPX racks** (E, derived from [StorageReview](https://www.storagereview.com/news/nvidia-groq-3-lpx-everything-we-know) and the specs above). That capacity gap is why NVIDIA pairs LPX with HBM GPUs rather than replacing them.

## DeepSeek publishes the only full production decode lineage

### Five disclosures trace the decode instance from one node to a supernode

DeepSeek is the only frontier lab that has published its production decode topology, fleet size, throughput and economics, and it has done so repeatedly. **DeepSeek-V2** (May 2024) ran on a single 8×H800 node with FP8 parameters and a KV cache quantized to ~6 bits per element. Its KV was 93.3% smaller than DeepSeek 67B's, generation throughput was **>50K tok/s per node**, and max generation throughput was 5.76x higher ([V2 paper](https://arxiv.org/html/2405.04434)). The **V3 technical report** (Dec 2024) then described a fully disaggregated design. Its prefill minimum unit is **4 nodes / 32 GPUs** (attention TP4+SP × DP8, MoE EP32, 32 redundant experts reshuffled ~every 10 minutes). Its **decode minimum unit is 40 nodes / 320 H800**: attention TP4 with sequence parallelism × DP80, MoE **EP320 with one expert per GPU**, and **64 GPUs hosting redundant and shared experts**. Dispatch/combine is point-to-point over IB with IBGDA, and one micro-batch's attention overlaps the other's MoE ([V3 report §3.4](https://arxiv.org/html/2412.19437v2)). The report concedes the "recommended deployment unit is relatively large".

The **Open Source Week Day-6** disclosure (1 Mar 2025) gave the configuration actually in production. **Decode units span 18 nodes / 144 H800 with routed experts at EP144 and MLA plus the shared expert at DP144**, i.e. **no attention TP**. Each GPU holds **2 routed + 1 shared expert**, with 32 redundant experts (144 × 2 = 288 = 256 + 32). **Prefill units span 4 nodes at EP32/DP32**, with 9 routed + 1 shared expert per GPU. Matmuls and dispatch run in FP8, while core MLA and combine run in BF16. Decode splits attention into two steps inside a **5-stage pipeline**, and the decode load balancer evens out **KV-cache usage and request counts per GPU** ([Day-6](https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md)). Over the 24 h window the fleet averaged **226.75 nodes (~1,814 H800)** and peaked at 278. It served **608B input tokens/day (56.3% hit the on-disk KV cache) and 168B output tokens/day** at **20–22 tok/s per user**, with an average KV length of 4,989. Per H800 node that was **~73.7k input tok/s in prefill and ~14.8k output tok/s in decode**, i.e. **~1,850 output tok/s per H800**. Cost was $87,072/day at $2/GPU-hour, against a theoretical revenue of $562,027/day, a 545% cost-profit margin ([Day-6](https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md)). The token volumes reconcile with the node count. 168B output/day ≈ 1.94M tok/s ÷ 14.8k ≈ **~131 decode nodes**, and 608B input ÷ 73.7k ≈ **~95 prefill nodes**, which sum to ~227. That puts the **prefill:decode node ratio at ~0.73:1** (≈7.3 decode units of 18 nodes and ≈24 prefill units of 4 nodes), with **~88 concurrent sequences per decode H800** (E, derived). The companion **profile-data** release shows decode at **EP128, TP1, 4K prompts, 128 requests per GPU** ([profile-data](https://github.com/deepseek-ai/profile-data)). LMSYS's comparison table gives **18,598 tok/s/node** for that profile (16 nodes, KV 4,096), against 14,800 for the production blog figure ([LMSYS](https://lmsys.org/blog/2025-05-05-large-scale-ep/)).

**V3.1** adopted UE8M0 FP8 scales for microscaling compatibility ([V3.1 card](https://huggingface.co/deepseek-ai/DeepSeek-V3.1)). **V3.2** added DeepSeek Sparse Attention: an FP8 ReLU lightning indexer selects **2,048 KV tokens per query**, turning core attention from O(L²) to O(Lk). It runs on MLA's MQA mode, and DeepSeek's cost curves were measured "benchmarking the actual service deployed on H800 GPUs" ([V3.2 report](https://arxiv.org/html/2512.02556)). Attention traffic per decoded token therefore becomes roughly constant beyond 2K context, plus the indexer's FP8 scan (E). DeepSeek disclosed **no V3.2 production parallelism**. Launch came with day-0 Ascend/CANN, Cambricon and Hygon support (3P, [TechNews](https://technews.tw/2025/10/02/deepseek-new-model-debuts-with-support-for-ascend-chips-and-cann/)).

**V4** (Apr 2026) comes in two sizes: Pro at 1.6T total / 49B active and Flash at 284B / 13B, both with 1M context and FP4 experts. At 1M tokens V4 cuts single-token FLOPs to 27% (Pro) or 10% (Flash) of V3.2 and **KV to 10% or 7%**. Its MegaMoE fine-grained-EP mega-kernel gives **1.50–1.73x** for general inference and up to **1.96x** for latency-sensitive serving, validated on NVIDIA and Ascend ([V4 report](https://arxiv.org/html/2606.19348)). On **20 Oct 2026** DeepSeek open-sourced **DeepInfer**, "the framework that powers DeepSeek's large-scale production services", for V4.1-Flash on **NVIDIA SM100 and Huawei Ascend 950**. V4.1-Flash (Sept 2026) is a different size from V4-Flash: **552B total, activating ~8B parameters per token in prefill and ~16B in decode** (O, company-reported; secondary sources add a 196B Engram component and a vision encoder on top, [Let's Data Science](https://letsdatascience.com/news/deepseek-updates-flash-model-with-leaner-serving-1cf96356); [ProPakistani](https://propakistani.pk/2026/09/11/deepseek-v4-1-flash-launches-with-lower-prices-and-native-vision/)). Hopper is unsupported, and mainline FlashMLA dropped Hopper kernels on 30 Sep 2026 ([DeepInfer README](https://github.com/deepseek-ai/DeepInfer); [FlashMLA](https://github.com/deepseek-ai/FlashMLA)). Its deployment Q&A is the most detailed production disclosure of 2026. It describes encoder/prefill/decode (E/P/D) disaggregation. **Prefill runs at EP2–8/DP2–8 on ordinary 8-GPU machines** ("smallest HBM allows"). **Decode runs at EP64–128/DP64–128 on "ordinary 64–128-GPU supernodes"** ("largest available"). Attention DP equals EP, with **no TP and no CP**. The P:D GPU ratio is **~1:1**, and the smallest efficient unit is **64 prefill + 64 decode GPUs**. Decode starts with **1 redundant expert per GPU** (7 experts/GPU at EP64), reassigns every 10,000 steps, and reaches a measured slowest/mean MoE load of **1.090**. A dynamic-length draft (DSpark) proposes **5 tokens** and accepts **~3.2–3.3**. The production operating point is **~60 sequences per rank** at an estimated **14,466 tok/s/rank** (~240 tok/s per sequence, E). Main KV is FP4-style (per-16 e4m3 scales), and indexer KV and routed experts are MXFP4. **Decode does not offload KV**, but KV it generates is written to a **~100 PiB 3FS SSD tier** with 3-day (agent) or 7-day (chat) TTL. Interconnect is budgeted at **~400 Gbps per 8 GPUs**. The Q&A claims **155 trillion tokens/day per cluster** and **$38 revenue per GPU-hour** at valley pricing ([DeepInfer Q&A](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/inference_deployment_qa.md)). The published DeepInfer benchmarks used **4 SM100 GPUs for prefill and 64 for decode**, each within one scale-up domain ([DeepInfer README](https://github.com/deepseek-ai/DeepInfer)). The Q&A does not name the exact production chip (B200/GB200/GB300/Ascend 950).

| Era | Decode instance | Hardware | Attention | MoE (experts/GPU) | Batch/GPU | Output tok/s/GPU | Per-user speed | Label / source |
|---|---|---|---|---|---|---|---|---|
| V2 (May 2024) | 1 node, 8 GPUs | H800 | n/a | n/a | n/a | >50K/node (~6.3k/GPU, E) | n/a | O, [V2](https://arxiv.org/html/2405.04434) |
| V3 report (Dec 2024) | 40 nodes, **320 GPUs** | H800, IB | **TP4+SP × DP80** | EP320 (1/GPU; 64 GPUs redundant+shared) | n/a | n/a | n/a | O, [V3 report](https://arxiv.org/html/2412.19437v2) |
| V3/R1 profile (Feb 2025) | 16 nodes, 128 GPUs | H800 | TP1 | EP128 | **128 req** | ~2,325 (18,598/node) | ~50 ms TPOT | O, [profile-data](https://github.com/deepseek-ai/profile-data); [LMSYS](https://lmsys.org/blog/2025-05-05-large-scale-ep/) |
| V3/R1 production (Feb 2025) | 18 nodes, **144 GPUs** | H800, IB | **DP144, no TP** | EP144 (2 routed + 1 shared) | ~88 seqs (E) | **~1,850** | **20–22 tok/s** | O, [Day-6](https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md) |
| V4.1-Flash + DeepInfer (Oct 2026) | **64–128 GPUs**, one supernode | SM100 / Ascend 950 | DP = EP, no TP/CP | EP64–128 (7 at EP64) | **~60 seqs** | ~14,466/rank (estimate table) | ~240 tok/s/seq (E) | O, [DeepInfer Q&A](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/inference_deployment_qa.md) |

The V3 report and Day-6 disagree, and the disagreement is substantive. **The report specifies attention TP4+SP with DP80 over 320 GPUs. Day-6 specifies pure DP144 with no TP over 144 GPUs.** The 320-GPU figure is best read as the Dec-2024 design point and Day-6 as the steady-state production configuration two months later. The move from TP4 to pure DP removes the 4x duplication of MLA's single-head latent cache. DeepSeek never states that reason in prose; it comes from SGLang and third parties ([LMSYS](https://lmsys.org/blog/2025-05-05-large-scale-ep/)). Neither document gives production TPOT distributions or decode instance counts. The MTP draft module adds **85–90% second-token acceptance and 1.8x TPS** per the V3 report ([V3 §5.4.3](https://arxiv.org/html/2412.19437v2)), widened to 80–90% in the ISCA paper ([arXiv 2505.09343](https://arxiv.org/html/2505.09343)).

### Third parties reproduced DeepSeek decode on every major platform

| Deployment | Decode devices | HW | Parallelism | Batch/device | Decode tok/s/device | Latency point | Label / source |
|---|---|---|---|---|---|---|---|
| SGLang, Atlas Cloud (May 2025) | 9 nodes = **72** | H100, IB | DP-attn, EP72, DP dense FFN + LM head | **256** | **2,785** (22,282/node) at KV 2,000 | ITL ~100 ms; $0.20/M out | 3P, [LMSYS](https://lmsys.org/blog/2025-05-05-large-scale-ep/) |
| same, simulated MTP | 72 | H100 | same | 128 | 2,172 (17,373/node) at KV 4,000 | 55.6 ms | 3P, same; [CloudMatrix comparison](https://arxiv.org/pdf/2506.12708v3) |
| SGLang GB200 Part I (Jun 2025) | 12 trays = **48** | GB200 NVL72 | DP-attn + EP48, FP8 | — | **7,583** at 2K in (2.7x H100) | — | 3P, [LMSYS GB200 P1](https://lmsys.org/blog/2025-06-16-gb200-part-1/) |
| SGLang GB200 Part II (Sep 2025) | **48** | GB200 NVL72 | FP8 attn + NVFP4 MoE | 1,408 total (2K) / 768 (4K), ≈29/rank | **13,386** (9,087 with BF16 attn + FP8 MoE) | — | 3P, [LMSYS GB200 P2](https://lmsys.org/blog/2025-09-25-gb200-part-2/) |
| vLLM wide-EP, CoreWeave (Dec 2025) | multi-node (EP/DP undisclosed) | H200, IB CX7 | DP-attn + EP, DBO, EPLB | — | **~2,200** (was ~1,500) | — | 3P, [vLLM](https://vllm.ai/blog/2025-12-17-large-scale-serving) |
| vLLM GB200 (Feb 2026) | **8** (1×8) | GB200 | DP+EP, NVFP4 dispatch | — | **10,100** at 2K/2K | — | 3P, [vLLM GB200](https://vllm.ai/blog/2026-02-03-dsr1-gb200-part1) |
| TRT-LLM perf table | DEP4 | GB200, FP4 | DP-attn + EP4 | — | ~4,989 (1K/1K), ~4,474 (1K/8K) | — | 3P, [TRT-LLM perf](https://nvidia.github.io/TensorRT-LLM/1.2.0/developer-guide/perf-overview.html) |
| Perplexity (Apr 2025) | 8 (single node) to 128 (16 nodes) | H200 / H100 | EP8 DP8 TP1 → **EP128 DP32 TP4** | ~100 at 5K ctx (8×H200) | EP128 ≈ **5x** single-node at equal speed | >100 tok/s at batch 1 (TP8) | 3P, [Perplexity](https://www.perplexity.ai/hub/blog/lower-latency-and-higher-throughput-with-multi-node-deepseek-deployment) |
| InferenceX Dynamo+TRT (May 2026) | **20–36** | GB200 NVL72, FP4, MTP | EP4 → EP32 along the curve | up to ~800 conc/GPU | **14,125 @25 tok/s/user**; 6,781 @100; 432 @200 | 3.49 ms TPOT best | 3P, [InferenceX](https://inferencex.semianalysis.com/blog/gb200-nvl72-vs-b200-disagg-deepseek-r1-fp4-dynamo-trt) |
| MLPerf v6.0 (Apr 2026) | 288 (4×NVL72) | GB300 | Dynamo disagg, wide-EP, MTP (≤3 drafts) | — | **8,064 server / 9,821 offline** (NVIDIA per-GPU figures) | server SLO | 3P, [NVIDIA MLPerf](https://developer.nvidia.com/blog/nvidia-extreme-co-design-delivers-new-mlperf-inference-records/) |
| Huawei CloudMatrix-Infer | **160 NPUs / 320 dies** | Ascend 910C, INT8 | **DP320 / EP320**, 1 expert per die | 96 (4K/256) | **1,943** | 49.4 ms | O (Huawei), [arXiv 2506.12708](https://arxiv.org/pdf/2506.12708v3) |
| Huawei xDeepServe peak | 144 chips / 288 dies | 910C | DP288 / EP288 | 60/die | **~2,400/chip** | ~50 ms (90% MTP acceptance) | O (Huawei), [arXiv 2508.02520](https://arxiv.org/html/2508.02520v6) |

Several patterns run through these reproductions. Two-batch overlap gained **+25.5% at 256 tokens/device but cost −27% at 32 tokens/device**, and EPLB with 288 slots (256 + 32 redundant) gave **2.54x** in decode ([LMSYS](https://lmsys.org/blog/2025-05-05-large-scale-ep/)). Perplexity found micro-batching cut step time 29% (2,667 → 1,896 µs) at batch 128/GPU but hurt below batch 32 ([Perplexity](https://www.perplexity.ai/hub/blog/lower-latency-and-higher-throughput-with-multi-node-deepseek-deployment)). SGLang sized GB200 decode batches to keep the KV pool full: cutting the 2K-ISL batch from 1,408 to 768 lost ~10% throughput ([LMSYS GB200 P2](https://lmsys.org/blog/2025-09-25-gb200-part-2/)). The MLPerf figures need care. NVIDIA's per-GPU numbers (8,064 server, 9,821 offline) do not equal the 288-GPU aggregates divided evenly: 1,555,110 / 288 ≈ 5,400, and 2,494,310 / 288 ≈ 8,660. The per-GPU comparison likely comes from a different system size (E). In Huawei's "efficiency" metric the 910C leads at **1.29 tok/s/TFLOPS** vs 1.17 for DeepSeek's H800 profile ([CloudMatrix](https://arxiv.org/pdf/2506.12708v3)). Per chip it only matches an H800. Power per token is likely worse given SemiAnalysis's ~600 kW system estimate vs ~120 kW for GB200 NVL72 ([The Register](https://www.theregister.com/2025/07/29/huawei_rackscale_boogeyman/)).

## Chinese labs deploy 16–320 accelerators per decode instance

The Chinese labs fall into two groups. Their model cards publish "minimum fit" configurations, almost always one node at TP4–TP8. Their papers and showcase deployments describe wide-EP or attention–FFN-disaggregated decode instances far larger than any model card implies.

### Moonshot Kimi: 16-GPU minimum, 96-GPU decode instance, INT4 since K2-Thinking

Moonshot's K2 deploy guide is explicit. **"The smallest deployment unit for Kimi-K2 FP8 weights with 128k seqlen on mainstream H200 or H20 platform is a cluster with 16 GPUs"**, with either TP16 or DP16+EP (`--max-num-seqs 256`), or TRT-LLM `--tp_size 16 --ep_size 8 --max_batch_size 128` with attention-DP. The guide's large-scale SGLang example is **"4P12D H200"**. Its **decode is 12 nodes = 96 GPUs at `--tp-size 96 --dp-size 96`**, i.e. DP-attention 96 with EP across all 96, plus `--cuda-graph-bs 480`, `--max-running-requests 46080` and 96 redundant experts. 46,080 = 96 × 480, so the decode batch is **480 sequences per GPU** (O, [K2 deploy guide](https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/docs/deploy_guidance.md)). The LMSYS/Mooncake team ran this on **128 H200 (4 prefill + 12 decode nodes)** with Moonshot staff acknowledged and NVIDIA DGX Cloud GPUs. They measured **decode 288K tok/s (24K/node, ~3,000/GPU)** and prefill 224K tok/s at 2,000 in / 100 out, about **$0.21 per 1M output tokens** at $2.3/H200-hour ([LMSYS K2](https://lmsys.org/blog/2025-07-20-k2-large-scale-ep)). At 480 sequences and ~3,000 tok/s per GPU the implied TPOT is **~160 ms**, a throughput-maximizing point that is not comparable to 50 ms-SLA figures (E). **Conflict:** the claim that K2 runs on "128 H800" has **no source**. The only documented 128-GPU K2 deployment is on **H200**, and it was run by the SGLang/Mooncake team, not described as Moonshot production. Moonshot's own checkpoint-engine benchmarks instead show **TP16 on H20** (16×H20, and 256×H20 as 16 vLLM instances × TP16) as its test instance shape for K2 ([checkpoint-engine](https://github.com/MoonshotAI/checkpoint-engine)). Moonshot has never disclosed its production decode size or chip mix.

From **K2-Thinking** onward Moonshot applies **INT4 weight-only QAT to the MoE experts**. The model card's rationale is that "thinking models exhibit excessive decoding lengths" and post-hoc quantization hurt accuracy, while native INT4 gives a **"lossless 2x speed-up in low-latency mode"** ([K2-Thinking card](https://huggingface.co/moonshotai/Kimi-K2-Thinking)). K2.6 keeps the method ([K2.6 card](https://huggingface.co/moonshotai/Kimi-K2.6)). INT4 shrinks the ~1 TB FP8 expert weights to ~0.6 TB, which moves the minimum footprint from 16 GPUs to one node: the K2.5 vLLM recipe is **TP8 verified on 8×H200** ([vLLM K2.5](https://docs.vllm.ai/projects/recipes/en/latest/moonshotai/Kimi-K2.5.html)). Kimi's serving platform **Mooncake** (FAST'25 best paper) is KVCache-centric, with separate prefill and decode pools and a distributed CPU-DRAM/SSD KV store. It reports "75% more requests" for Kimi under real workloads, while a Tsinghua summary gives 115% / 107% on A800 / H800 clusters. The two figures conflict, likely across paper versions ([USENIX](https://www.usenix.org/conference/fast25/presentation/qin); [Tsinghua](https://www.tsinghua.edu.cn/en/info/1245/14138.htm)). Google reports that **Kimi K3 (2.8T parameters)** serves **>700 tok/s to a single user on Ironwood** via a Pallas megakernel (O for Google's stack, not Moonshot production; [Google Developers Blog](https://developers.googleblog.com/announcing-new-vllm-and-sglang-inference-engines-on-tpu/)).

### Alibaba Qwen, MiniMax, Zhipu GLM and Baidu ERNIE publish node-sized recipes only

For these four labs the sources give only node-sized configurations. **Qwen3-235B-A22B-2507-FP8** runs at **TP4** with 262K context and 128-block FP8 ([HF card](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507-FP8)). **Qwen3-Next-80B-A3B** (Gated DeltaNet hybrid, 512 experts, 2 KV heads in its gated-attention layers) runs at TP4 with NEXTN MTP ([HF card](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct)). **Qwen3.5-397B-A17B** runs at **TP8 on 8×H200 or 8×MI300X/MI355X**, with `-dp 8 --enable-expert-parallel` recommended for throughput, MTP-1 for latency, and NVFP4 on 4 GB200 ([vLLM recipe](https://recipes.vllm.ai/Qwen/Qwen3.5-397B-A17B)). On Ascend it uses TP16 + EP + MTP3 ([vLLM-Ascend](https://docs.vllm.ai/projects/ascend/en/main/tutorials/models/Qwen3.5-397B-A17B.html)). Alibaba's only first-party serving-systems disclosure, **Aegaeon** (SOSP 2025), is about multi-model GPU pooling and cut **1,192 GPUs to 213** for long-tail models, not large-MoE decode ([The Register](https://theregister.com/2025/10/21/alibaba_aegaeon_gpu_scheduling_improvements)). On Ironwood, InferenceX measured **Qwen3.5-397B FP8 (8k/1k) at 9,364 tok/s/chip at 20 tok/s/user** (B200: 8,903) and **$0.181/M at 100 tok/s/user** (B200 $0.222). GB300 NVL72 disaggregated kept a ~30% perf/$ edge mid-curve ([SemiAnalysis](https://newsletter.semianalysis.com/p/tpu-inferencex-full-steam)).

**MiniMax-M2** (230B/10B) needs **4×96 GB GPUs for up to 400K context** (220 GB weights + 60 GB per 1M context tokens). vLLM finds **TP4+EP4 beats TP8+EP8 on H100**, and pure TP8 is unsupported ([MiniMax guide](https://huggingface.co/MiniMaxAI/MiniMax-M2/blob/refs%2Fpr%2F57/docs/vllm_deploy_guide.md); [vLLM recipe](https://docs.vllm.ai/projects/recipes/en/latest/MiniMax/MiniMax-M2.html)). M2.5 measured ~849 output tok/s on 4×H100 at 32 concurrent (8K/1K). **M3** (428B/23B) runs **TP8 on 8×B200** at ~1,628 tok/s for 32 concurrent ([Lambda M2.5](https://lambda.ai/inference-models/minimaxai/minimax-m2.5); [Lambda M3](https://lambda.ai/inference-models/minimaxai/minimax-m3)). **GLM-4.5** (355B/32B) needs **8×H100 in FP8 or 16 in BF16**, doubling to 16 / 32 for full 128K context ([GLM-4.5 card](https://huggingface.co/zai-org/GLM-4.5)). **GLM-5** (744B/40B, with DSA) serves at TP8 with 3-step MTP/EAGLE and has an Ascend xLLM guide ([GLM-5 card](https://huggingface.co/zai-org/GLM-5)). NVIDIA's Dynamo **GLM-5.2** recipe disaggregates **prefill TP8/EP8 and decode TP8/DP8/EP1 on H200 with FP8 KV** at up to 250K context, with an MTP-3 acceptance length of 2.69. On B200 it uses NVFP4 at up to 500K context with HiCache CPU offload ([Dynamo GLM-5.2](https://docs.nvidia.com/dynamo/dev/recipes/glm-5-2)). **Baidu ERNIE-4.5-300B-A47B** ships an official **W4A8C8 TP4** checkpoint, claims lossless 4-bit and 2-bit "convolutional code" quantization, and uses PD disaggregation with dynamic role switching in FastDeploy ([ERNIE README](https://github.com/PaddlePaddle/ERNIE)). **None of these four labs discloses its own production API decode topology, precision or per-GPU throughput.**

### StepFun, Meituan and ByteDance publish full production-grade decode designs

**StepFun Step-3** (321B/38B VLM) introduced **attention–FFN disaggregation (AFD)** in production. Attention (holding the KV) and FFN run on separate **8-GPU Hopper servers**. **2A2F = 32 GPUs** at 4K context with FP8 attention reaches **4,039 peak / 3,910 average tokens per GPU per second** under a **50 ms TPOT SLA**. 3A2F = 40 GPUs with BF16 attention reaches 3,321. 4A2F = 48 GPUs serves 8K, and 16A2F is extrapolated for 32K. The batch is **6,144 = 3 micro-batches × 2,048**, ~192 per GPU over all GPUs, or ~384 per attention GPU (E). The pipeline has three stages (attention / FFN / communication) of **16.6 ms**, a **~272 µs per-layer budget** over 61 layers. FP8 activations go to FFN and BF16 come back, over the zero-SM StepMesh GPUDirect-RDMA library. Step-3 claims a theoretical decode cost of **$0.055/M tokens at 8K** vs $0.068 for DSv3 ([Step-3 paper](https://arxiv.org/html/2507.19427v1)). **Meituan LongCat-Flash** (560B, 18.6–31.3B dynamic active) sets its **minimum deployment unit at 2 nodes / 16 H800 for each of prefill and decode**. It uses DP-attention (MLA), dense FFN at TP2/TP4, and wide EP with a DeepEP modified so zero-computation experts skip communication. Benchmarks ran on **128 Hopper GPUs** at EP128. LongCat published the clearest throughput–latency curve of any Chinese lab: **3,785 TGS at 35 tok/s/user, 2,205 at 68.9, and 804 at 100.5** (BF16), and 4,230 at 26.4 tok/s/user in FP8. Measured TPOT at batch 96 is **~26 ms** against a 16 ms theoretical. MTP acceptance is >90%, and cost is **$0.70/M output at ~100 tok/s/user** ([LongCat-Flash](https://arxiv.org/html/2509.01322)). Its Single Batch Overlap (SBO) hides all-to-all inside one batch via the ScMoE shortcut, unlike DeepSeek's two-batch overlap.

**ByteDance MegaScale-Infer** (Doubao serving) disaggregates experts. Attention nodes are DP replicas, and each **expert node of 1–8 GPUs holds one expert**, with **ping-pong over m ≥ 3 micro-batches**. The heterogeneous mapping puts **H20 on attention and L40S on experts**. The system is **deployed on a cluster of nearly 10,000 GPUs** and cuts cost 1.5–2.0x at equal traffic. Its per-GPU decode throughput is 2.56x vLLM and 1.28x TRT-LLM (homogeneous), and 7.11x / 1.90x on a 317B "Scaled-MoE". The SLO is 150 ms TBT, and A100 needs ≥156 tokens per GPU for full utilization ([MegaScale-Infer](https://arxiv.org/html/2504.02263v4)). The paper does not name which Doubao models it serves.

### Huawei Ascend runs the largest disclosed decode instances

Huawei Cloud's production MaaS serves "DeepSeek, Kimi, GLM, Qwen, and MiniMax" on CloudMatrix384. **CloudMatrix-Infer** uses **one decode instance of 160 Ascend 910C (20 nodes, 320 dies)** at **EP320 / DP320**, with one expert per die: 256 routed + 32 redundant + 32 shared-expert copies. Prefill runs as 6 instances × 16 NPUs at EP32, and the system pools Kunpeng DRAM of all nodes over UB as a KV cache. DeepSeek-R1 is requantized to **INT8 W8A8**, dispatch included. Decode yields **2,733 tok/s/NPU at 46.8 ms (1K/1K, batch 128), 1,943 at 49.4 ms (4K/256, batch 96), 974 at 24.6 ms (batch 24), and 538 at 14.9 ms (batch 8)**. One instance therefore carries ~15,360 concurrent sequences (E) ([CloudMatrix-Infer](https://arxiv.org/pdf/2506.12708v3)). **xDeepServe**, "in production for over a year", peaks at **288 dies (144 chips) with DP288/EP288**, 60 sequences/die and ~2,400 tok/s/chip. It also runs an attention–MoE disaggregated SuperPod configuration of **768 dies: 480 MLA dies + 288 MoE dies**, with A2E 172 µs and E2A 193 µs at a global batch of 46,080. Its **representative production setup is 16 servers: 4 prefill TEs (2 servers, DP8/EP32 each) + 1 decode TE of 8 servers (64 chips, DP128/EP128)**, measured at **34.8 ms average TPOT and 900 ms TTFT** on average 13K-in / 2.1K-out traffic, against **TTFT < 2 s and TPOT 35 ms** SLAs. Prefill runs on 910B and 910C, but **decode runs only on 910C** to use UB ([xDeepServe](https://arxiv.org/html/2508.02520v6)). Huawei's roadmap builds the PD split into silicon. The **950PR** (cheap HiBL HBM) targets prefill, and the **950DT** (144 GB, 4 TB/s) targets decode, in an 8,192-chip Atlas 950 SuperPoD ([Huawei keynote](https://www.huawei.com/en/news/2025/9/hc-xu-keynote-speech); [Tom's Hardware](https://tomshardware.com/tech-industry/semiconductors/huawei-unveils-ascend-roadmap-backed-by-in-house-hbm)).

| Model / system | Arch (total / active) | Devices per decode instance | Hardware | Parallelism | Batch/device | Decode tok/s/device | TPOT or tok/s/user | Label |
|---|---|---|---|---|---|---|---|---|
| Kimi K2 (guide minimum) | 1T / 32B, MLA, 384 experts | **16** | H200 / H20 | TP16 or DP16+EP | ≤256 seqs/inst. | — | — | O, [guide](https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/docs/deploy_guidance.md) |
| Kimi K2 (4P12D) | same, FP8 | **96** | H200 | DP96, EP96, 96 redundant | **480** | ~3,000 | ~160 ms (E) | O config / 3P measured, [LMSYS](https://lmsys.org/blog/2025-07-20-k2-large-scale-ep) |
| Kimi K2.5 / K2.6 | 1T / 32B, INT4 experts | 8 | H200 | TP8 | — | — | 2x low-latency vs FP8 (O) | 3P recipe, [vLLM](https://docs.vllm.ai/projects/recipes/en/latest/moonshotai/Kimi-K2.5.html) |
| Qwen3-235B-A22B | 235B / 22B | 4 | H100/H200 | TP4, FP8 | — | — | — | O, [card](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507-FP8) |
| Qwen3.5-397B-A17B | 397B / 17B | 8 (4 on GB200 NVFP4; 16 Ascend dies) | H200, MI355X, GB200 | TP8 or DP8+EP; MTP | — | 9,364/chip on Ironwood @20 tok/s/user | — | 3P, [vLLM](https://recipes.vllm.ai/Qwen/Qwen3.5-397B-A17B); [SemiAnalysis](https://newsletter.semianalysis.com/p/tpu-inferencex-full-steam) |
| MiniMax-M2 / M2.5 | 230B / 10B | 4 | H100/H200 (96 GB) | TP4+EP4 | 32 conc. tested | ~212 (849 / 4 GPUs) | — | O / 3P, [vLLM](https://docs.vllm.ai/projects/recipes/en/latest/MiniMax/MiniMax-M2.html); [Lambda](https://lambda.ai/inference-models/minimaxai/minimax-m2.5) |
| MiniMax-M3 | 428B / 23B | 8 | B200 | TP8 | 32 conc. tested | ~204 (1,628 / 8) | — | 3P, [Lambda](https://lambda.ai/inference-models/minimaxai/minimax-m3) |
| GLM-4.5 | 355B / 32B | 8 (FP8) / 16 (BF16); 16 FP8 for 128K | H100 | TP8 | ≤8 (sizing assumption) | — | — | O, [card](https://huggingface.co/zai-org/GLM-4.5) |
| GLM-5 / 5.2 | 744B / 40B, DSA | 8 | H200 (FP8), B200 (NVFP4) | decode TP8/DP8/EP1 (Dynamo) | — | — | MTP accept 2.69 | O / 3P, [Dynamo](https://docs.nvidia.com/dynamo/dev/recipes/glm-5-2) |
| ERNIE-4.5-300B-A47B | 300B / 47B | 4 | (Paddle) | TP4, W4A8C8 | — | — | — | O, [ERNIE](https://github.com/PaddlePaddle/ERNIE) |
| Step-3 (2A2F) | 321B / 38B | **32** (16 attn + 16 FFN) | Hopper | AFD, FP8 | ~192 (E) | **3,910–4,039** | **50 ms** | O, [Step-3](https://arxiv.org/html/2507.19427v1) |
| LongCat-Flash | 560B / ~27B | **16 min; 128 benchmark** | H800 | DP-attn + wide EP; FFN TP2/4 | ~8–108 (E) | **804–4,230** | 26–100 tok/s/user | O, [LongCat](https://arxiv.org/html/2509.01322) |
| MegaScale-Infer (Doubao) | undisclosed | attn DP + expert nodes (1–8 GPUs each) | H20 + L40S | disaggregated EP, ping-pong | ≥156 tokens/GPU | 1.28x TRT-LLM | 150 ms TBT SLO | O, [paper](https://arxiv.org/html/2504.02263v4) |
| DeepSeek-R1 on CloudMatrix | 671B / 37B | **160 NPUs (320 dies)** | 910C, INT8 | DP320 / EP320 | 96 | 1,943 | 49.4 ms | O, [paper](https://arxiv.org/pdf/2506.12708v3) |
| xDeepServe production TE | DeepSeek / K2 | **64 chips** | 910C | DP128 / EP128 | — | — | **34.8 ms avg** | O, [paper](https://arxiv.org/html/2508.02520v6) |

## Western open-weight labs ship single-node recipes but serve wider

### Meta: one 8-GPU host per replica, prefill on separate hosts

Meta's Llama 3 paper is the only first-party Western statement of a decode topology for a frontier-scale dense model. **Llama 3.1 405B in BF16 runs on 16 H100s across 2 nodes, with TP8 inside each node and PP2 across nodes**, using 2 micro-batches. **In FP8 it runs on a single 8×H100 node**, which gives up to 50% more prefill throughput and a "substantially better" decode throughput–latency curve ([Llama 3 §6](https://arxiv.org/html/2407.21783v3)). Meta's FP8 recipe quantizes FFN weights and activations only. It leaves attention and the first and last layers in BF16, uses **row-wise scales upper-bounded at 1,200**, and was validated on reward-model score distributions over 100k responses because standard benchmarks missed corruptions ([Llama 3 §6.2](https://arxiv.org/html/2407.21783v3)). Meta's **context-parallelism paper** measured 405B decode on Grand Teton hosts. **TTIT at batch 1 is 44.51 ms at 8K and 46.26 ms at 128K on TP8 (1 node)**, rising to ~65.6 ms on CP2+TP8, while TP16 gives 39.52 ms at 128K. Decode "scales poorly" with both TP and CP across hosts. CP buys KV capacity, not latency. A 1M-token prefill took 77 s on 128 H100s ([CP paper](https://arxiv.org/html/2411.01783v3)). Table 7 of the paper gives 60.23 ms for CP2+TP8 at 128K, which conflicts with 66.63 ms in Table 6.

Meta's production decode is disaggregated. "Prefill/Decode Disaggregation has been enabled in Meta internal inference stack, serving large scale Meta traffic." The published benchmark is **Llama 4 Maverick on 8×H100 hosts at 1P1D over TCP**, with TP on both sides. KV moves layer-by-layer via DRAM over Thrift/TCP with multiple frontend NICs, the **KV block size is 128–256 tokens** (vs vLLM's default 16), prefix caching is disabled on decode, and sticky routing gives **40–50% prefix-cache hits at 90% HBM utilization** ([PyTorch blog](https://pytorch.org/blog/disaggregated-inference-at-scale-with-pytorch-vllm/)). Meta's production EAGLE work reports **Llama 4 Maverick at ~4 ms/token at batch 1 on 8 H100s**, 1.4–2.0x speedups at large batch, and production batch sizes of **1 to >100**. The draft is a 3-layer INT4-FFN head with tree size chosen by batch, acceptance lengths are 2.75–2.94, the model runs as a full CUDA graph, and prefill and decode sit on separate hosts at 94% GPU utilization ([arXiv 2508.08192](https://arxiv.org/html/2508.08192v1)). On hardware, Meta said in Oct 2024 that **AMD MI300X serves "all live traffic" for Llama 405B**: 192 GB/GPU fits 405B in BF16 on one 8-GPU node ([AMD via Yahoo Finance](https://finance.yahoo.com/news/amd-unveils-leadership-ai-solutions-175800080.html)). Its **MTIA 400/450/500** are "primarily" for GenAI inference from 2026–27. MTIA 400 has a **72-device switched scale-up domain**, and MTIA 450 **doubles HBM bandwidth** because Meta identified bandwidth as the inference bottleneck ([Meta newsroom](https://about.fb.com/news/2026/03/expanding-metas-custom-silicon-to-power-our-ai-workloads/); [StorageReview](https://www.storagereview.com/news/metas-mtia-roadmap-four-chip-generations-in-two-years-put-genai-inference-first)). Meta's official sizing guidance for the rest of the line: **Llama 4 Scout** (109B/17B) needs 1 H100 at INT4, 2 at FP8 and 4+ at BF16, and **Maverick** (400B/17B, 128 routed + 1 shared expert, top-1) runs on "a single H100 DGX host" ([Llama 4 blog](https://ai.meta.com/blog/llama-4-multimodal-intelligence/); [llama-models](https://github.com/meta-llama/llama-models)). The newer **Muse Spark Open** (816B/42B, 256 experts top-8, 16 KV heads) needs **≥700 GB in NVFP4 or ≥2 TB in BF16**, and its example commands give no TP/EP flags ([HF card](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B)). How closed Muse Spark is served is undisclosed.

Third parties pushed Maverick to the latency extreme. **One DGX B200 (8 GPUs) reached 1,038 tok/s/user**, verified by Artificial Analysis, using **TP8, EP1, FP8 everywhere, EAGLE-3 with draft length 3 (2.06x), and CUDA-graph max batch 8**. The same server peaks at **72,000 tok/s** in a throughput configuration ([NVIDIA](https://developer.nvidia.com/blog/blackwell-breaks-the-1000-tps-user-barrier-with-metas-llama-4-maverick); [TRT-LLM guide](https://nvidia.github.io/TensorRT-LLM/1.2.0/blogs/tech_blog/blog6_Llama4_maverick_eagle_guide.html)). That 72x ratio between peak throughput and single-user speed on identical hardware sums up the batch–latency trade-off. MLPerf's 405B server scenario allows **TTFT 6 s and TPOT 175 ms** (~5.7 tok/s/user), and Llama-2-70B Interactive requires **TTFT 450 ms and TPOT 40 ms** ([AMD ROCm MLPerf](https://rocm.blogs.amd.com/artificial-intelligence/mlperf-inference5.1-repro/README.html)). MLPerf numbers are therefore large-batch throughput figures. GB300 per-GPU 405B throughput went from 170 to 259 tok/s (server) between v5.1 and v6.0 ([NVIDIA MLPerf](https://developer.nvidia.com/blog/nvidia-extreme-co-design-delivers-new-mlperf-inference-records/)).

### Mistral, gpt-oss and Gemma confirm the "one node" design target

**Mistral Large 2** (123B dense) was "designed for single-node inference" ([Mistral](https://mistral.ai/news/mistral-large-2407)). **Mistral Large 3** (675B total / 41B active, 256K context) fits **one 8×H200 or 8×B200 node in FP8, or one 8×H100/A100 node in NVFP4**. vLLM serves it at `--tensor-parallel-size 8` with an EAGLE draft at 3 speculative tokens ([HF card](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512)). NVIDIA co-optimized it for **GB200 NVL72 wide-EP plus Dynamo PD disaggregation**, claiming up to 10x over H200 and **>5M tok/s/MW at 40 tok/s/user** ([NVIDIA](https://developer.nvidia.com/blog/nvidia-accelerated-mistral-3-open-models-deliver-efficiency-accuracy-at-any-scale/)). Le Chat's **Flash Answers** ran Large 2 on **Cerebras WSE-3 at >1,100 tok/s**, using SRAM plus speculative decoding co-developed with Mistral ([Cerebras](https://www.cerebras.ai/blog/mistral-le-chat)). Mistral Medium 3 self-hosts on "four GPUs and above" ([Mistral](https://mistral.ai/fr/news/mistral-medium-3)). What Le Chat serves Large 3 on today is undisclosed.

OpenAI's **gpt-oss-120b** has 116.8B total / 5.1B active parameters, 36 layers, 128 experts with top-4 routing, and 8 KV heads at d64. It alternates 128-token banded and full attention layers, carries MXFP4 experts in a 60.8 GiB checkpoint, and **fits one 80 GB GPU** ([model card](https://arxiv.org/html/2508.10925)). The vLLM recipe uses **TP1 for max throughput per GPU** (max-num-seqs as high as KV allows), **TP4/TP8 at batch ~8 for min latency**, and TP2 at batch 128 as balanced. AMD examples use TP8 ([vLLM recipe](https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html)). NVIDIA claims **1.5M tok/s per GB200 NVL72 rack (~50k concurrent users)**, and Dynamo disaggregation gives a 4x interactivity gain at 32K ISL ([NVIDIA](https://developer.nvidia.com/blog/delivering-1-5-m-tps-inference-on-nvidia-gb200-nvl72-nvidia-accelerates-openai-gpt-oss-models-from-cloud-to-edge)). InferenceX measured **~17,595 tok/s/GPU** at one GB200 point, about 1.27M per rack ([InferenceX](https://inferencex.semianalysis.com/compare/gptoss-120b-b200-vs-gb200)). With only ~2.7 GB of active MXFP4 weights per token, gpt-oss decode is limited by KV and batch rather than by weight bandwidth (E). **Gemma 3** (≤27B) is "the most capable model you can run on a single GPU or TPU" ([Google](https://blog.google/technology/developers/gemma-3/)). **Gemma 4**'s 26B MoE and 31B dense run on a single 80 GB H100, or a single MI300X at TP1 ([Verdict](https://www.verdict.co.uk/?p=400356); [AMD](https://www.amd.com/en/developer/resources/technical-articles/2026/day-0-support-for-gemma-4-on-amd-processors-and-gpus.html)).

| Model | Arch | Devices per decode replica | Hardware | Parallelism | Batch | Throughput / latency | Label |
|---|---|---|---|---|---|---|---|
| Llama 3.1 405B BF16 | dense 405B, GQA-8 | **16** | H100 | TP8 × PP2, 2 µ-batches | — | — | O, [Llama 3](https://arxiv.org/html/2407.21783v3) |
| Llama 3.1 405B FP8 | same | **8** | H100 | TP8 | 1 | **44.5–46.3 ms TTIT** (8K–128K) | O, [CP paper](https://arxiv.org/html/2411.01783v3) |
| Llama 3.1 405B | same | 8 | MI300X | (TP8) | — | "all live traffic" | O, [AMD](https://finance.yahoo.com/news/amd-unveils-leadership-ai-solutions-175800080.html) |
| Llama 4 Maverick (Meta prod.) | 400B / 17B | **8** (+8 prefill, 1P1D) | H100, TCP | TP8 + PD disagg | 1 to >100 | **~4 ms/token at batch 1** (EAGLE) | O, [PyTorch](https://pytorch.org/blog/disaggregated-inference-at-scale-with-pytorch-vllm/); [arXiv](https://arxiv.org/html/2508.08192v1) |
| Llama 4 Maverick (NVIDIA record) | same | 8 | B200 | TP8, EP1, FP8, EAGLE-3 | ≤8 | **1,038 tok/s/user**; 72k tok/s/server peak | 3P, [NVIDIA](https://developer.nvidia.com/blog/blackwell-breaks-the-1000-tps-user-barrier-with-metas-llama-4-maverick) |
| Llama 4 Scout | 109B / 17B | 1 (INT4) / 2 (FP8) / 4 (BF16) | H100 | TP | — | — | O, [llama-models](https://github.com/meta-llama/llama-models) |
| Muse Spark Open | 816B / 42B | ≥700 GB NVFP4 (one 8-GPU H200/B200 node) | — | undisclosed | — | — | O, [HF](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B) |
| Mistral Large 3 | 675B / 41B | 8 | H200/B200 (FP8); H100/A100 (NVFP4) | TP8 + EAGLE; wide-EP on NVL72 | — | >5M tok/s/MW at 40 tok/s/user (vendor) | O / vendor |
| Mistral Large 2 (Le Chat) | 123B dense | undisclosed | Cerebras WSE-3 | — | — | **>1,100 tok/s** | O, [Cerebras](https://www.cerebras.ai/blog/mistral-le-chat) |
| gpt-oss-120b | 116.8B / 5.1B | **1** (throughput) – 8 (latency) | H100/B200/GB200 | TP1 / TP2 / TP4–8 | KV-limited / 128 / ~8 | ~17.6k tok/s/GPU on GB200 | O recipe / 3P measured |
| Gemma 3 / 4 | ≤31B | 1 | GPU or TPU | none | — | — | O |

## Closed labs disclose systems, never replicas

### OpenAI: the 2023 GPT-4 leak remains the only topology on record

The **SemiAnalysis GPT-4 leak** (July 2023, paywalled, popularized by a deleted Yam Peleg thread) is still the only per-replica decode topology for any closed OpenAI model. It describes a **~1.8T-parameter MoE with 120 layers and 16 experts of ~111B MLP parameters each, top-2 routing, and ~280B active parameters (~560 TFLOP) per token**, served on **128-GPU clusters at 8-way TP × 16-way PP**. Its cost estimate is **$0.0049 per 1k tokens on 128 A100s and $0.0021 on 128 H100s** at 8K context with high batch (L, [The Decoder](https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/); [McGuinness](https://patmcguinness.substack.com/p/gpt-4-details-revealed)). The leak hedges on speculative decoding ("not 100% sure") ([r/LocalLLaMA](https://lr.us.psf.lt/r/LocalLLaMA/comments/14wbmio/gpt4_details_leaked/jrkj14x)). It is internally inconsistent: 120 layers do not divide into 16 stages, and a LinkedIn copy gives 15-way PP as a *training* setup. OpenAI never confirmed any of it. For later models only estimates exist. Microsoft's MEDEC paper lists **GPT-4o ~200B, 4o-mini ~8B, o1-preview ~300B and o1-mini ~100B**, with the authors stating they "cannot vouch" for these figures ([arXiv 2412.19260](https://arxiv.org/pdf/2412.19260)). Epoch puts GPT-5 training compute at ~5e25 FLOP, below GPT-4.5's >1e26 ([Epoch](https://epochai.substack.com/p/notes-on-gpt-5-training-compute)). If 4o is ~200B, an FP8 replica fits in one 8-GPU node, a 16x drop from GPT-4's reported 128 GPUs (E).

OpenAI's official disclosures name hardware only. **GPT-5.5 "runs on NVIDIA GB200 NVL72 rack-scale systems"** (NVIDIA, Apr 2026), with no GPU count, parallelism or tok/s ([NVIDIA](https://blogs.nvidia.com/?p=92703)). Azure's first production **GB300 NVL72 cluster (>4,600 GPUs)** was built for OpenAI ([NVIDIA](https://blogs.nvidia.com/blog/microsoft-azure-worlds-first-gb300-nvl72-supercomputing-cluster-openai)). **GPT-5.3-Codex-Spark** (Feb 2026), "a smaller version of GPT-5.3-Codex", runs on **Cerebras WSE-3 at >1,000 tok/s**. It is OpenAI's first production model off NVIDIA, under a **~750 MW** "dedicated low-latency inference" deal reported at $10B+ ([OpenAI](https://openai.com/index/introducing-gpt-5-3-codex-spark/); [DCD](https://www.datacenterdynamics.com/en/news/openai-signs-10-billion-deal-with-cerebras-with-750mw-of-big-chip-compute/)). **Microsoft Maia 200** serves "the latest GPT-5.2 models from OpenAI" ([Microsoft](https://blogs.microsoft.com/blog/2026/01/26/maia-200-the-ai-accelerator-built-for-inference/)). **Jalapeño** (Broadcom, unveiled 24 Jun 2026) is an inference-only reticle-sized ASIC targeted for servers by end-2026 ([CNBC](https://www.cnbc.com/2026/06/24/openai-and-broadcom-reveal-jalapeno-first-ai-chip-in-partnership.html)). Capacity deals cover AMD (6 GW, MI450 from 2H 2026) and NVIDIA (≥10 GW, Vera Rubin first) ([AMD IR](https://ir.amd.com/news-events/press-releases/detail/1260/amd-and-openai-announce-strategic-partnership-to-deploy-6-gigawatts-of-amd-gpus)). TPU rental was reported in June 2025 (L, [DCD](https://datacenterdynamics.com/en/news/openai-to-use-googles-tpus-report/)), but OpenAI called it early testing. OpenAI's engineering disclosures are product-level. The Codex-Spark WebSocket path cut per-roundtrip overhead 80%, per-token overhead 30% and TTFT 50% ([OpenAI](https://openai.com/index/introducing-gpt-5-3-codex-spark/)), and prompt caching works in 128-token increments ([OpenAI](https://openai.com/index/api-prompt-caching/)). The deal structure reads as a two-tier strategy: bulk reasoning decode on NVL72 racks (and later Jalapeño), and a latency tier on wafer-scale SRAM. **Nothing public gives GPUs per replica, TP/EP degree, batch, speculative decoding or tok/s/GPU for GPT-4 Turbo, 4o, 4.1, 4.5, o1/o3/o4-mini, GPT-5.x or realtime models.**

### Google: Gemini runs disaggregated on TPUs, and that is nearly all

Google's public serving science is extensive; its Gemini specifics are thin. **Pope et al.** served **PaLM 540B on 64 TPU v4** using a 2D weight-stationary FFN layout (E sharded 4-way, F 16-way) and batch-sharded multiquery attention. Low-latency decode ran at batch 64 with int8 weights at **28.5 ms/token (36.9 ms bf16), 14% MFU**. High-throughput decode ran at batch 512 in bf16 at ~94 ms/step and 33% MFU. **PaLM 62B decoded on just 8 chips at batch 512 (37% MFU)**. The rule: "during the generate phase, we select the 2D weight-stationary layout because the batch size in tokens is always small". The paper even proposes "pipelining a batch-1 prefill server into a batch-64 decoding server", an early form of PD disaggregation ([arXiv 2211.05102](https://arxiv.org/abs/2211.05102)). The **scaling book** recommends model parallelism only for decode, with "useful model parallelism typically 8 to 32". Its worked example puts **LLaMA-3-70B int8 on a v5e 4×2 (8 chips) at ~17 ms/step for batch 32 at 8K**, or ~8.5 ms on 4×4, and sizes **~3 prefill servers per decode server** for 8K-prefill / 512-decode ([applied inference](https://jax-ml.github.io/scaling-book/applied-inference/)). Google ran disaggregated Llama2-70B on **16 Trillium chips (8 prefill + 8 decode)** for 7x better TTFT and ~3x better TPOT than interleaving ([Google Cloud](https://cloud.google.com/blog/products/compute/ai-hypercomputer-inference-updates-for-google-cloud-tpu-and-gpu)).

For Gemini itself, Google confirms that 1.5, 2.5 and 3 are **sparse MoE** ([Gemini 1.5](https://arxiv.org/abs/2403.05530); [Gemini 2.5](https://arxiv.org/abs/2507.06261)). Flash was designed for TPU serving latency, with parallel attention/FFN, and Flash/Flash-Lite are distilled to allow "larger batch sizes at the same latency targets". Google's fleet uses **"batching, speculative decoding, disaggregated prefill/decode serving, KV caching"**, AQT quantization, XLA, Pallas and Pathways, plus near-real-time model placement with reserved idle capacity. Its environmental paper reports **0.24 Wh per median Gemini Apps prompt and a 33x energy reduction May 2024 → May 2025** (23x from models, 1.4x from utilization), with **host CPU+DRAM at 25% of per-prompt energy** ([arXiv 2508.15734](https://arxiv.org/html/2508.15734v1)). Speculative decoding is confirmed for AI Overviews ([Google Research](https://research.google/blog/looking-back-at-speculative-decoding/)). SemiAnalysis reports Google "has been running PD disaggregation internally for Gemini serving for ages" (3P, [SemiAnalysis](https://newsletter.semianalysis.com/p/tpu-inferencex-full-steam)). JetStream is "based on the same inference stack used to serve Gemini" ([Google Cloud](https://cloud.google.com/blog/products/compute/ai-hypercomputer-inference-updates-for-google-cloud-tpu-and-gpu)). Gemini 3.5 Flash was claimed at **800 tok/s** (Hassabis), but Artificial Analysis measured **~278**, and the hardware behind the 12x "Antigravity" mode is unresolved ([Hassabis](https://x.com/demishassabis/status/2056904067406860545); [Artificial Analysis](https://artificialanalysis.ai/de/models/gemini-3-5-flash-medium/providers)). Google has not disclosed chips per Gemini replica, TPU generation per model, TP/EP degree, expert counts, or how 1M–10M contexts are sharded. The strongest indirect signal is **TPU 8i's design brief**: Boardfly topology for "all-to-all-heavy MoE" (up to 50% latency improvement), a Collectives Acceleration Engine with 5x lower collective latency, and 384 MB SRAM so "a bigger KV cache stay[s] on silicon". Google built that chip for wide-EP, long-context MoE decode at low latency ([TPU 8 deep dive](https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive)). A 64-chip Ironwood cube (~12.3 TB HBM, ~472 TB/s) is a natural replica size, but Google has not confirmed it (E).

### Anthropic, xAI, Amazon and Microsoft disclose fleets and chips only

**Anthropic** serves Claude "on AWS Trainium, NVIDIA GPUs, and Google TPUs" under "strict equivalence standards". Its Sept 2025 postmortem is the richest serving disclosure from any closed lab ([Anthropic](https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues)). It reveals **separate server pools for the 1M-context window**: up to 16% of Sonnet 4 requests in the worst hour were misrouted to them, and routing is sticky. It confirms **Opus 4/4.1 and Sonnet 4 on first-party TPU servers**, on-accelerator top-k/top-p sampling inside compiled XLA graphs whose behavior varied "for certain batch sizes", **bf16 next-token probabilities**, and top-p thresholds of 0.99/0.999. Fleet facts: **>1M Trainium2 chips** "to train and serve Claude" ([Anthropic–Amazon](https://www.anthropic.com/news/anthropic-amazon-compute)); **up to 1M TPUs, >1 GW in 2026** ([Anthropic](https://anthropic.com/news/expanding-our-use-of-google-cloud-tpus-and-services)) plus ~3.5 GW from 2027 ([TechCrunch](https://techcrunch.com/2026/04/07/anthropic-compute-deal-google-broadcom-tpus/)); up to 1 GW of Grace Blackwell / Vera Rubin via Azure ([Microsoft](https://blogs.microsoft.com/blog/2025/11/18/microsoft-nvidia-and-anthropic-announce-strategic-partnerships/)); and, from May 2026, all of **Colossus 1 (~300 MW, >220k NVIDIA GPUs)** for Pro/Max capacity (L, [CNBC](https://www.cnbc.com/2026/05/06/anthropic-spacex-data-center-capacity.html)). In Sept 2025 SemiAnalysis judged "most of Anthropic's skyrocketing inference needs are served by Google Cloud" (E, [SemiAnalysis](https://newsletter.semianalysis.com/p/amazons-ai-resurgence-aws-anthropics-multi-gigawatt-trainium-expansion)). Two products expose the latency lever without explaining it. **Latency-optimized Claude 3.5 Haiku on Trainium2** is "up to 60% faster" at a higher price ([Anthropic](https://www.anthropic.com/news/trainium2-and-distillation)), and **Opus fast mode** gives "up to 2.5x higher output tokens per second from the same model" ([Claude docs](https://platform.claude.com/docs/en/build-with-claude/fast-mode)). Both match the standard trade of smaller batch, more chips per replica and/or speculation on memory-bound decode (E). **No chips per replica, TP/EP, precision, PD disaggregation or speculative decoding are disclosed for any Claude model.**

**xAI's** only recipes are for open weights. **Grok-1** is 314B with 8 experts and top-2 routing ([grok-1](https://github.com/xai-org/grok-1)). **Grok-2 is "TP=8, so you will need 8 GPUs (each with > 40GB of memory)"**, served via SGLang `--tp 8 --quantization fp8` ([HF grok-2](https://huggingface.co/xai-org/grok-2)), at ~270B total / ~115B active from its config (E). For production, Musk stated in July 2025 that Colossus is for training and **"inference is done by our cloud providers"** ([Musk](https://x.com/elonmusk/status/1947701807389515912)). Oracle says xAI uses OCI "to train and run inference" ([Oracle](https://www.oracle.com/tw/news/announcement/xais-grok-models-are-now-on-oracle-cloud-infrastructure-2025-06-17/)). Nothing is disclosed on Grok 3/4/5 serving. **Amazon** says "the majority of token usage in Amazon Bedrock is already running on Trainium" (Jassy, L, [The New Stack](https://thenewstack.io/openai-bedrock-trainium-silicon/)) and that Bedrock serves production on Trainium3 ([About Amazon](https://www.aboutamazon.com/news/aws/aws-re-invent-2025-ai-news-updates)). Cerebras is coming to Bedrock for Nova and open models ([Cerebras](https://www.cerebras.ai/press-release/awscollaboration)). Nova parameter counts and replica sizes are undisclosed, though AWS's Neuron docs show **Llama 3.1 405B on a single 16-chip trn2.48xlarge** ([Neuron tutorials](https://awsdocs-neuron.readthedocs-hosted.com/en/v2.27.1/libraries/nxd-inference/tutorials/index.html)). **Microsoft** discloses that MAI-1-preview is an MoE trained on ~15k H100 ([Neowin](https://www.neowin.net/news/microsoft-reveals-two-in-house-ai-models-mai-voice-1-and-mai-1-preview/)) and that MAI-Code-1-Flash has 5B active parameters, with a 1.4x efficiency gain from Maia 200 co-design ([Microsoft AI](https://microsoft.ai/news/building-a-hillclimbing-machine-launching-seven-new-mai-models/)). Maia 200's **4-chip fully connected tray (864 GB, 28 TB/s)** is the natural minimum TP group (E). No MAI serving topology is published.

| Lab / model | What is officially disclosed about decode serving | What is not disclosed | Label |
|---|---|---|---|
| OpenAI GPT-4 (2023) | nothing official; leak: 128 GPUs, TP8×PP16, ~1.8T MoE top-2 | everything official | L |
| OpenAI GPT-4o / 4.1 / 4.5 / o-series | none (MEDEC estimates 4o ~200B, o1-preview ~300B) | size, devices, parallelism, batch | E |
| OpenAI GPT-5.x / GPT-5.5 | GPT-5 is a router over main/thinking/mini/nano; **GPT-5.5 runs on GB200 NVL72**; GPT-5.2 on Maia 200 | GPUs per replica, EP/TP, batch, tok/s | O (system only) |
| OpenAI GPT-5.3-Codex-Spark | **Cerebras WSE-3, >1,000 tok/s**, 128K context | wafers per replica, size, precision | O |
| Anthropic Claude (all) | TPU + Trainium + NVIDIA; separate 1M-context pools; on-TPU sampling in bf16; fast mode 2.5x; latency-optimized Haiku 60% faster on Trn2 | chips per replica, parallelism, precision, PD, spec decode | O (platform only) |
| Google Gemini 1.5–3.5 | sparse MoE; TPUs; PD disaggregation, spec decoding, AQT, distillation for larger batches; 0.24 Wh/prompt | chips per replica, TPU generation per model, EP/TP, long-context sharding | O / 3P |
| xAI Grok 3/4/5 | inference on cloud providers (OCI, Azure) | everything else | O statement |
| Amazon Nova | majority of Bedrock tokens on Trainium; Trn3 in production | size, devices, parallelism | O aggregate |
| Microsoft MAI | MAI-1-preview MoE; Maia 200 co-design 1.4x | size, devices, parallelism | O |

## Throughput collapses 10–30x as interactivity rises

Every published decode curve shows the same steep trade between tok/s/GPU and tok/s/user. The InferenceX GB200 NVL72 vs B200 sweep (DeepSeek-R1-0528 FP4, 1K/1K, Dynamo + TRT-LLM, MTP, disaggregated, 22 May 2026) is the best single dataset ([InferenceX](https://inferencex.semianalysis.com/blog/gb200-nvl72-vs-b200-disagg-deepseek-r1-fp4-dynamo-trt)).

| tok/s/user (≈1000/TPOT ms) | 25 | 60 | 100 | 125 | 150 | 200 | 250 | 300 |
|---|---|---|---|---|---|---|---|---|
| GB200 NVL72 tok/s/GPU | **14,125** | 11,017 | 6,781 | 4,130 | 1,922 | 432 | 186 | unreachable |
| B200 (8-GPU HGX) tok/s/GPU | 12,292 | 9,185 | 3,047 | 941 | 583 | 332 | 193 | 67 |

GB200's advantage peaks at **4.39x at 125 tok/s/user ($0.149 vs $0.576 per M tokens)**, in the band where B200's EP spills onto RoCE. **B200 wins at ≥250 tok/s/user**, where tiny batches make NVLink irrelevant. Parallelism changes along the curve. The **lowest-latency GB200 point (3.49 ms TPOT, 286 tok/s/user, concurrency 4) uses decode EP8 on 32 GPUs**, mid-curve (164 tok/s/user) uses EP32, and **max throughput (14.7k tok/s/GPU at 17.8 tok/s/user, 56 ms TPOT, concurrency 16,130) uses only EP4 on 20 decode GPUs**. B200 never exceeds EP8 ([InferenceX](https://inferencex.semianalysis.com/blog/gb200-nvl72-vs-b200-disagg-deepseek-r1-fp4-dynamo-trt)). The cheapest point therefore does not use the widest EP. Max throughput favors whatever minimizes communication per token while filling KV. Wide EP pays off in the middle, where it lets the system hold high per-user speed at large aggregate batch. Lab disclosures show the same shape. LongCat drops from 3,785 to 804 TGS between 35 and 100 tok/s/user ([LongCat](https://arxiv.org/html/2509.01322)). CloudMatrix drops from 1,943 to 538 tok/s/NPU between 49 and 15 ms TPOT ([CloudMatrix](https://arxiv.org/pdf/2506.12708v3)). Maverick on one DGX B200 spans 1,038 tok/s/user to 72,000 tok/s per server ([NVIDIA](https://developer.nvidia.com/blog/blackwell-breaks-the-1000-tps-user-barrier-with-metas-llama-4-maverick)).

Per-GPU DeepSeek-class decode rose roughly **7x from early 2025 to mid 2026**. The steps were ~1,850 (H800 production, Feb 2025), ~2,785 (H100 SGLang, May 2025), ~2,200 (H200 vLLM production-like, Dec 2025), 7,583 (GB200 FP8, Jun 2025), 13,386 (GB200 NVFP4, Sep 2025), 8,064 (GB300 under MLPerf server SLO, Apr 2026) and 14,125 (GB200 at 25 tok/s/user, May 2026). These are not iso-SLO, and the MLPerf per-GPU server figure alone rose **2.77x on identical GB300 hardware in six months** from software ([NVIDIA MLPerf](https://developer.nvidia.com/blog/nvidia-extreme-co-design-delivers-new-mlperf-inference-records/)). InferenceMAX found **GB200 NVL72 FP4 at ~8x the tok/s/MW of a single H200 FP8 node at 30 tok/s/user**, with MTP adding 2–3x throughput/GPU at 70–140 tok/s/user ([InferenceMAX](https://inferencex.semianalysis.com/blog/inferencemax-open-source-inference-benchmarking)). GB300 beats GB200 by ~9% at 93 tok/s/user and ~29% at 162 ([InferenceX GB200 vs GB300](https://inferencex.semianalysis.com/compare/deepseek-r1-gb200-vs-gb300)).

The ≥150 tok/s/user tier costs 5–30x more per token on GPUs, and that cost gap is where SRAM machines fit. **Cerebras** serves Llama 3.1 405B at **969 tok/s** (wafer count undisclosed; ≥~19 WSE-3 for 16-bit weights by arithmetic, E) ([Cerebras](https://cerebras.ai/press-release/cerebras-inference-llama-405b)), gpt-oss-120b at ~3,000 tok/s and Qwen3-235B at ~1,400–1,500 ([Cerebras](https://cerebras.ai/inference)). **SambaNova** puts 405B on 16 SN40L at 129–132 tok/s/user ([SambaNova](https://sambanova.ai/blog/sn40l-chip-best-inference-solution)). **Groq's** gen-1 Llama-2-70B demo needed **576 LPUs** ([arXiv 2503.09650](https://arxiv.org/html/2503.09650v1)). NVIDIA licensed Groq's IP (~$20B, Dec 2025), shelved the prefill-oriented Rubin CPX, and positioned **Groq 3 LPX** racks to run decode FFN/MoE beside Rubin GPUs that hold attention and KV ([Tom's Hardware](https://www.tomshardware.com/tech-industry/semiconductors/nvidia-confirms-20-billion-groq-deal-to-bolster-ai-inference-dominance); [StorageReview](https://www.storagereview.com/news/nvidia-groq-3-lpx-everything-we-know)). Its "35x throughput per MW" claim is unverified.

## Disaggregation, KV tiering, low precision and speculation complete the stack

### Prefill/decode ratios swing from 1:8 to 1:1 with workload

PD disaggregation, introduced by DistServe (7.4x more requests or 12.6x tighter SLOs, [arXiv 2401.09670](https://arxiv.org/abs/2401.09670)) and Splitwise (2.35x throughput at equal cost and power, [arXiv 2311.18677](https://arxiv.org/abs/2311.18677)), is now the default for large-MoE serving. It is confirmed in production at DeepSeek, Meta, Kimi/Mooncake, Huawei, LongCat, ERNIE and (per SemiAnalysis) Gemini. The ratio depends on workload. **DeepSeek production ran ~0.73:1 prefill:decode nodes** with a 56% cache-hit rate (E, derived). **DeepInfer runs ~1:1 GPUs** with 2–8-GPU prefill instances against 64–128-GPU decode instances ([DeepInfer Q&A](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/inference_deployment_qa.md)). SGLang used **4:9 nodes** for DeepSeek on H100 ([LMSYS](https://lmsys.org/blog/2025-05-05-large-scale-ep/)) and Kimi K2 used **4:12** ([LMSYS K2](https://lmsys.org/blog/2025-07-20-k2-large-scale-ep)). xDeepServe's production unit is **8 prefill servers : 8 decode servers** ([xDeepServe](https://arxiv.org/html/2508.02520v6)). InferenceX's GB200 Pareto set moves from **4:32 GPUs at low concurrency to 12:20 at max throughput** ([InferenceX](https://inferencex.semianalysis.com/blog/gb200-nvl72-vs-b200-disagg-deepseek-r1-fp4-dynamo-trt)). Meta's Maverick benchmark used 1P1D ([PyTorch](https://pytorch.org/blog/disaggregated-inference-at-scale-with-pytorch-vllm/)). The scaling book sizes 3 prefill : 1 decode for 8K/512 ([applied inference](https://jax-ml.github.io/scaling-book/applied-inference/)). Prefill instances shrink as decode instances grow. DeepSeek went from 32-GPU prefill to 2–8, and SGLang's GB200 prefill used 2–4 ranks per instance ([LMSYS GB200 P2](https://lmsys.org/blog/2025-09-25-gb200-part-2/)). The two phases want opposite things: prefill is compute-bound and wants the fewest GPUs that hold the weights, while decode wants the largest pool that can be filled with KV. KV moves over RDMA (Mooncake, NIXL), GPUDirect (DeepSeek) or plain TCP via DRAM (Meta), and DeepSeek budgets ~400 Gbps per 8 GPUs, enough for cross-datacenter P/D.

### Attention–FFN disaggregation pays only when the ratio is tuned

AFD goes one step further inside decode. Attention is stateful, KV-bound and grows with context. FFN/MoE is stateless, weight-bound and grows with batch. It is in production at StepFun (Step-3), ByteDance (MegaScale-Infer, ~10k GPUs) and Huawei (xDeepServe, 480 attention dies : 288 MoE dies). NVIDIA's Rubin + Groq LPX pairing is a hardware version of the same split ([StorageReview](https://www.storagereview.com/news/nvidia-groq-3-lpx-everything-we-know)). It is not a free win. vLLM's AFD plugin on Ascend 910C (DeepSeek-V3.2 W8A8) found **48A16F slower than the EP64 baseline (−5.3% at 16K, −10% at 32K) but 64A16F faster (+11.3% / +9.0%)**, i.e. "disaggregation alone does not guarantee a gain" ([vLLM AFD](https://vllm.ai/blog/2026-07-23-vllm-afd-plugin)). Step-3 likewise scales attention servers with context, from 2A2F at 4K to 16A2F at 32K ([Step-3](https://arxiv.org/html/2507.19427v1)). Ping-pong pipelining requires attention and FFN stage times to match ([MegaScale-Infer](https://arxiv.org/html/2504.02263v4)). For million-token contexts NVIDIA's **Helix** shards KV along the sequence (KVP) during attention and reuses the same GPUs as TP/EP for FFN. It is shipped in TensorRT-LLM as generation-phase CP, but its 32x-more-users claim is **simulated** ([arXiv 2507.07120](https://arxiv.org/abs/2507.07120v1); [TRT-LLM Helix](https://nvidia.github.io/TensorRT-LLM/1.3.0rc2/features/helix.html)).

### KV handling: HBM for live sequences, DRAM/SSD for reuse

Decode keeps live KV in HBM and pushes reuse to cheaper tiers. **DeepSeek's decode never offloads KV**. Instead it writes generated KV to a 3FS tier of **~100 PiB SSD (20 × 30.73 TB per server, 10+2 erasure coding, 3–7 day TTL)**, keeps sliding-window KV in host memory, and swaps it in via GPUDirect RDMA ([DeepInfer Q&A](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/inference_deployment_qa.md)). Its decode load balancer routes on **request count and total KV length**, reaching ~80% of perfectly balanced throughput. Meta uses **128–256-token KV blocks** and sticky routing for 40–50% prefix hits ([PyTorch](https://pytorch.org/blog/disaggregated-inference-at-scale-with-pytorch-vllm/)). Google keeps prefix caches in **host DRAM (~450 GiB vs 128 GiB HBM per 8×v5e host)** in an LRU trie with affinity routing ([scaling book](https://jax-ml.github.io/scaling-book/inference/)), and open-sourced TPU-Sync for zero-copy KV transfer and host offload ([SemiAnalysis](https://newsletter.semianalysis.com/p/tpu-inferencex-full-steam)). Huawei pools all Kunpeng DRAM over UB ([CloudMatrix](https://arxiv.org/pdf/2506.12708v3)). llm-d reports **13.9x throughput from hierarchical KV offload** at high concurrency on H100 and 3x from prefix-aware routing on MI300X ([vLLM llm-d](https://docs.vllm.ai/en/stable/deployment/integrations/llm-d/)). KV precision has fallen from BF16 to **~6 bits (DeepSeek-V2), 8-bit (Step-3, CloudMatrix), FP8 (GLM-5.2, SGLang GB200, which made attention up to 1.8x faster) and FP4-style (DeepSeek V4.1)**.

### Quantization tracks hardware generations

Weight precision follows the silicon. On Hopper the standard is **block-wise FP8 (128×128 weights, 1×128 activations)**, used by DeepSeek V3, Qwen, GLM, LongCat and Step-3 ([LongCat](https://arxiv.org/html/2509.01322)). Meta's row-wise FFN-only FP8 is the dense-model variant ([Llama 3](https://arxiv.org/html/2407.21783v3)). Moonshot's **INT4 QAT experts** suit FP4-less Hopper/H20 fleets ([K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)). Ascend serving is **INT8 W8A8 or W4A8**, since the 910 series lacks FP8 ([CloudMatrix](https://arxiv.org/pdf/2506.12708v3); [SGLang Ascend](https://docs.sglang.io/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice)). On Blackwell, **NVFP4/MXFP4** experts are standard: gpt-oss is native MXFP4, DeepSeek V4 and Muse Spark Open ship FP4, and SGLang's NVFP4 GEMMs are up to **1.9x faster than FP8**. Fusing NVFP4 into DeepEP **halves dispatch bytes** ([LMSYS GB200 P2](https://lmsys.org/blog/2025-09-25-gb200-part-2/)), and vLLM's NVFP4 dispatch cuts all-to-all volume 4x vs FP16 ([vLLM GB200](https://vllm.ai/blog/2026-02-03-dsr1-gb200-part1)). Lower precision helps decode twice: it reduces the weight bytes that bound latency at small batch, and it reduces the all-to-all bytes that bound wide EP.

### Speculative decoding helps most where decode is most memory-bound

Every disclosed frontier stack now speculates: DeepSeek MTP (1.8x) and DSpark (5 drafts, ~3.2 accepted), Meta EAGLE (2.75–2.94 accepted), Mistral EAGLE, Qwen/GLM MTP, LongCat MTP (>90%), xDeepServe MTP (70–90%, 2.35 tokens/step with two layers), Google AI Overviews, MLPerf v6.0 (≤3 drafts), and Vertex EAGLE-3 (2–3x, [Google Cloud](https://docs.cloud.google.com/vertex-ai/docs/blog/posts/from-research-to-production-accelerate-oss-llm-with-eagle-3-on-vertex)). The gain shrinks with batch. vLLM EAGLE speedup falls from **1.3x at batch 2 to 0.7x (a slowdown) at batch 48** ([arXiv 2508.08192](https://arxiv.org/pdf/2508.08192v1)). DeepInfer's adaptive draft length gains **−1.5% at 16 sequences/rank but +11% at 96** relative to fixed-5, because it trims verification work rather than adding speculation ([DeepInfer Q&A](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/inference_deployment_qa.md)). Speculation spends spare FLOPs, which exist only below B_crit. It is therefore a latency-tier tool, consistent with the roofline above.

## Cross-model comparison and source conflicts

The table consolidates the decode instance for every lab and model covered. "Devices" means accelerators in one decode replica/instance, not the fleet.

| Lab | Model | Total / active params | Devices per decode instance | Hardware | Parallelism | Batch / device | Decode tok/s/device | TPOT or tok/s/user | Confidence |
|---|---|---|---|---|---|---|---|---|---|
| OpenAI | GPT-4 (2023) | ~1.8T / ~280B (16×111B, top-2) | **128** | A100 / H100 | TP8 × PP16 | "high" | n/a | n/a | **L** |
| OpenAI | GPT-4o / 4o-mini | ~200B / ~8B (est.) | undisclosed | undisclosed | undisclosed | — | — | — | E |
| OpenAI | GPT-5.5 | undisclosed | undisclosed | **GB200 NVL72** | undisclosed | — | — | — | O (system) |
| OpenAI | GPT-5.2 | undisclosed | undisclosed | Maia 200 (among others) | undisclosed | — | — | — | O (system) |
| OpenAI | GPT-5.3-Codex-Spark | undisclosed ("smaller") | undisclosed | **Cerebras WSE-3** | undisclosed | — | — | **>1,000 tok/s** | O |
| OpenAI | gpt-oss-120b | 116.8B / 5.1B | **1–8** | H100 / B200 / GB200 | TP1 (thru) / TP4–8 (latency) | KV-limited / ~8 | ~17.6k (GB200) | — | O / 3P |
| Anthropic | Claude (all) | undisclosed (~175B Sonnet 3.5 est.) | undisclosed | TPU + Trainium + NVIDIA | undisclosed | — | — | fast mode 2.5x | O (platform) / E |
| Google | PaLM 540B | 540B dense | **64** | TPU v4 | 2D weight-stationary (TP64), batch-sharded MQA | 64 / 512 | — | **28.5 ms** (int8, b64) | O |
| Google | Gemini 1.5–3.5 | MoE, undisclosed | undisclosed | TPUs | PD disaggregated, spec decode | — | — | 3.5 Flash: 278 measured vs 800 claimed | O / 3P |
| Google | Gemma 3 / 4 | ≤31B | **1** | GPU / TPU | none | — | — | — | O |
| Meta | Llama 3.1 405B | 405B dense | **8 (FP8) / 16 (BF16)** | H100; MI300X | TP8 / TP8×PP2 | 1 to >100 | — | **44.5 ms at b1** | O |
| Meta | Llama 4 Maverick | 400B / 17B | **8** (+8 prefill) | H100; B200 | TP8 + PD; EAGLE | 1 to >100 | 72k/server (B200 peak) | **~4 ms (Meta, H100)**; 1,038 tok/s/user (B200) | O / 3P |
| Meta | Muse Spark Open | 816B / 42B | ~1 node (≥700 GB NVFP4) | — | undisclosed | — | — | — | O |
| DeepSeek | V3/R1 (Feb 2025) | 671B / 37B | **144** | H800 | **DP144 / EP144** | ~88 (E) | **~1,850** | **20–22 tok/s** | O |
| DeepSeek | V3 report design | same | **320** | H800 | TP4+SP × DP80 / EP320 | — | — | — | O |
| DeepSeek | V4.1-Flash (Oct 2026) | 552B / 8B (prefill) and 16B (decode) | **64–128** | SM100 / Ascend 950 | DP = EP, no TP/CP | ~60 | ~14.5k (est.) | ~240 tok/s (E) | O |
| Moonshot | Kimi K2 | 1T / 32B | **16 min; 96 (4P12D)** | H200 / H20 | DP96 / EP96 | **480** | ~3,000 | ~160 ms (E) | O / 3P |
| Moonshot | Kimi K2.5–K2.6 | 1T / 32B INT4 | **8** | H200 | TP8 | — | — | — | 3P |
| Moonshot | Kimi K3 | 2.8T | undisclosed | Ironwood | Pallas megakernel | 1 | — | **>700 tok/s** single user | O (Google) |
| Alibaba | Qwen3-235B / Qwen3.5-397B | 235B/22B; 397B/17B | **4 / 8** | H200, MI355X, Ironwood | TP4 / TP8 or DP8+EP | — | 9,364/chip (Ironwood @20) | — | O / 3P |
| MiniMax | M2 / M2.5 / M3 | 230B/10B; 428B/23B | **4 / 8** | H100–H200 / B200 | TP4+EP4 / TP8 | 32 tested | ~200 | — | O / 3P |
| Zhipu | GLM-4.5 / GLM-5.x | 355B/32B; 744B/40B | **8** (16 for BF16 or 128K) | H100 / H200 / B200 | TP8; decode TP8/DP8/EP1 | — | — | MTP accept 2.69 | O / 3P |
| StepFun | Step-3 | 321B / 38B | **32–48** | Hopper (H800/H20) | AFD 2A2F–4A2F | ~192 (E) | **3,910–4,039** | **50 ms** | O |
| Meituan | LongCat-Flash | 560B / ~27B | **16 min; 128 bench** | H800 | DP-attn + wide EP | 8–108 (E) | **804–4,230** | 26–100 tok/s | O |
| ByteDance | Doubao (MegaScale-Infer) | undisclosed | attention DP + 1–8-GPU expert nodes | H20 + L40S | disaggregated EP | ≥156 tokens | 1.28x TRT-LLM | 150 ms TBT SLO | O |
| Baidu | ERNIE-4.5-300B | 300B / 47B | **4** | — | TP4, W4A8C8 | — | — | — | O |
| Huawei | DeepSeek-R1 (CloudMatrix) | 671B / 37B | **160 NPUs / 320 dies** | Ascend 910C | DP320 / EP320, INT8 | 96 | 1,943 | 49.4 ms | O (Huawei) |
| Huawei | xDeepServe production | DeepSeek / K2 | **64 chips** | Ascend 910C | DP128 / EP128 | — | — | **34.8 ms** | O (Huawei) |
| xAI | Grok-2 | ~270B / ~115B (E) | **8** | >40 GB GPUs | TP8, FP8 | — | — | — | O recipe |
| xAI | Grok 3 / 4 / 5 | undisclosed | undisclosed | cloud providers (OCI) | undisclosed | — | — | — | O statement |
| Mistral | Large 3 | 675B / 41B | **8** | H200/B200 FP8; H100/A100 NVFP4; NVL72 | TP8 + EAGLE; wide-EP | — | — | 40 tok/s/user at >5M tok/s/MW | O / vendor |
| Mistral | Large 2 (Le Chat) | 123B dense | 1 node; Cerebras for Flash Answers | H100-class; WSE-3 | single node | — | — | **>1,100 tok/s** | O |
| Amazon | Nova | undisclosed | undisclosed | Trainium (majority of Bedrock) | undisclosed | — | — | — | O aggregate |
| Microsoft | MAI-1 / MAI-Code-1-Flash | MoE; 5B active | undisclosed | Maia 200 / Azure | undisclosed | — | — | — | O |

| Conflict | Source A | Source B | Resolution adopted here |
|---|---|---|---|
| DeepSeek V3 decode topology | V3 report: **320 H800, attention TP4+SP × DP80, EP320** ([V3](https://arxiv.org/html/2412.19437v2)) | Day-6: **144 H800, DP144, no TP, EP144** ([Day-6](https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md)); profile-data: EP128 TP1 | Report = Dec-2024 design point; Day-6 = Feb-2025 production; TP dropped to avoid MLA KV duplication |
| Kimi K2 128-GPU deployment hardware | Brief/rumour: "128 H800" (no source found) | LMSYS + Moonshot guide: **128 H200** (4P12D), run by SGLang/Mooncake team ([LMSYS](https://lmsys.org/blog/2025-07-20-k2-large-scale-ep)) | H200; Moonshot production hardware remains undisclosed (its own tests use H20 TP16) |
| Mooncake gains | "75% more requests" ([USENIX](https://www.usenix.org/conference/fast25/presentation/qin)) | 115% / 107% on A800 / H800 ([Tsinghua](https://www.tsinghua.edu.cn/en/info/1245/14138.htm)) | Different paper versions; treat as 75–115% |
| GPT-4 pipeline depth | 16-way PP for inference ([The Decoder](https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/)) | 15-way PP framed as training ([LinkedIn copy](https://il.linkedin.com/in/yampeleg)) | Unverified leak; 120 layers do not divide by 16 |
| Meta CP2+TP8 TTIT at 128K | 66.63 ms (Table 6) | 60.23 ms (Table 7) ([CP paper](https://arxiv.org/html/2411.01783v3)) | Both worse than TP8's 46.26 ms; conclusion unchanged |
| MLPerf v6.0 DeepSeek-R1 per-GPU | 8,064 server / 9,821 offline per GPU (NVIDIA) | 1,555,110 / 2,494,310 on 288 GPUs ≈ 5,400 / 8,660 ([NVIDIA](https://developer.nvidia.com/blog/nvidia-extreme-co-design-delivers-new-mlperf-inference-records/)) | Per-GPU numbers likely from a different system size |
| Gemini 3.5 Flash speed | 800 tok/s, "12x in Antigravity" (Hassabis) | ~278 tok/s (Artificial Analysis) | Claimed mode ≠ default API; hardware unresolved |
| AMD Helios per-GPU BW | 19.6 TB/s (AMD) | 23.3 TB/s (StorageReview) | Unresolved |
| Rubin CPX fate | "shelved" for LPU decode (Ian Buck, [Tom's Hardware](https://www.tomshardware.com/tech-industry/gc-2026-press-q-and-a-transcript)) | "evolved into" LPX ([StorageReview](https://www.storagereview.com/news/nvidia-groq-3-lpx-everything-we-know)) | Either way, NVIDIA's 2026 decode answer is LPX + Rubin |

## Addendum (22 Oct 2026): releases after the research cut

A later review flagged frontier releases from Jul–Oct 2026 that the sections above do not cover. None of them discloses a decode topology from the lab itself, so they all belong in the "undisclosed" rows. The one exception is Kimi K3, which has a third-party recipe.

| Lab | Release | What is public about decode serving | Label |
|---|---|---|---|
| OpenAI | GPT-5.6 Sol (Jul 2026), GPT-6 Astra (3 Sep 2026) | An "Ultrafast" GPT-5.6 Sol tier at ~750 tok/s, which reports variously attribute to Cerebras or to NVIDIA Blackwell. No replica topology. | L / O product only ([Constellation](https://www.constellationr.com/insights/news/openai-launches-gpt-6-astra); [KuCoin](https://www.kucoin.com/news/flash/openai-unveils-gpt-5-6-sol-with-750-tokens-s-speed-on-cerebras-hardware)) |
| Anthropic | Claude Fable 5.1, Claude 5.5 family | Nothing beyond the platform facts above | O |
| Google | Gemini 4 Argon (announced 30 Sep 2026), with 1M-token input and output | Nothing on serving topology | O ([Gemini API docs](https://ai.google.dev/gemini-api/docs/models/gemini-4-argon)) |
| xAI | Grok 4.6 (12 Aug 2026), reported as a ~1.5T-parameter base | Nothing on serving topology | L ([DataNorth](https://datanorth.ai/news/xai-releases-grok-4-6)) |
| Moonshot | Kimi K3 (16 Jul 2026): 2.8T total / 104B active, 896 experts with top-16 + 2 shared, MXFP4 weights trained with QAT | **NVIDIA Dynamo recipe on GB200: 16 GPUs at TP16 over multi-node NVLink. The disaggregated variant is 1P1D with 16 + 16 GPUs, FP8 KV, and KV moved by NIXL.** Moonshot's own card names only vLLM, SGLang and TokenSpeed. | O card / 3P recipe ([HF](https://huggingface.co/moonshotai/Kimi-K3); [Dynamo](https://docs.nvidia.com/dynamo/v1.5.1/recipes/kimi-k3)) |
| Alibaba / Zhipu | Qwen3.8-Max, GLM-5.3-Flash (Aug–Sep 2026) | Aggregator listings only | 3P |
| MiniMax | MiniMax M3.1 Flash (19 Oct 2026), proprietary, 1M context, size undisclosed | ~93 output tok/s on MiniMax's own API (Artificial Analysis); no serving topology | 3P ([Artificial Analysis](https://artificialanalysis.ai/models/minimax-m3-1-flash)) |

A further correction on comparability: tok/s/GPU, tok/s/user, TPOT and server-wide throughput are different quantities. The tables above quote each figure at its own context length, concurrency, precision, speculation setting and latency target, and they should not be read as a ranking.

## Conclusion

The question "how many GPUs serve one decode replica" has stopped being well-posed for frontier MoEs. A model card's "8 GPUs" says what fits on one node. Production says how large a pool can be kept busy, and the answer is set by the scale-up domain rather than the model. DeepSeek's instance followed the domain: from IB-linked H800 nodes (144 GPUs), to a 64–128-GPU NVLink-class supernode, to Ascend supernodes where the same open-source DeepEP now runs at full performance. The closed labs' hardware choices point the same way: GB200 NVL72 for GPT-5.5, TPU 8i's Boardfly built for MoE all-to-all, Maia's 6,144-chip Ethernet collectives, MTIA 400's 72-device domain, Trn3's 144-chip switched UltraServer. Each implies decode instances of tens of accelerators doing DP-attention plus wide EP, even though none of those labs will say so. For a decode-latency benchmark the corollary is uncomfortable. Single-node TP8 at batch 1–64, the configuration most easily reproduced, is the one frontier providers use only for latency tiers. Their bulk traffic runs at 30–480 sequences per accelerator inside 32–320-device instances, where the binding constraints are all-to-all latency (~200–400 µs per dispatch/combine on IB, far less on NVLink), KV capacity, and expert load imbalance (≈1.09 slowest/mean even after EPLB), not HBM bandwidth alone.

Two shifts look durable. First, decode is fragmenting by phase and by sub-layer. PD disaggregation is universal, and AFD (Step-3, MegaScale-Infer, xDeepServe, Rubin+LPX) lets attention and experts run on different silicon at a tunable ratio, so the "replica" becomes a graph of pools with independently chosen parallelism. Second, the latency–throughput curve is being served by different machines at each end: NVL72-class racks for the 20–150 tok/s/user band, where wide EP pays, and SRAM or wafer-scale systems (Cerebras for Codex-Spark and Le Chat, Groq LPX) for ≥300 tok/s/user, where GPUs lose 10–30x of their throughput. A realistic decode-latency benchmark should therefore report the whole Pareto curve under a wide-EP, disaggregated configuration with MTP/EAGLE and FP8/FP4 KV, not a single batch-1 number. It should also treat any closed-model "devices per replica" figure as an estimate unless it traces to the 2023 GPT-4 leak, the only such disclosure on record.