First AA-AgentPerf results for NVIDIA Vera Rubin NVL72

Vera Rubin NVL72 keeps the rack-scale design of GB300 NVL72, with 72 GPUs and 36 CPUs in a single NVLink domain. Each Rubin GPU has the same 288 GB of memory as Blackwell Ultra, but 2.4x the memory bandwidth (from HBM4), 1.7x the NVLink bandwidth and 2.3x the dense NVFP4 compute, and the Vera CPU doubles the NVLink-C2C bandwidth to its GPUs, which is the link that carries any KV cache offloaded to CPU memory.

Each chart below plots total throughput per MW of measured GPU power against per-user output speed. Total throughput counts input tokens (cached and newly prefilled) and output tokens, with ~96% of them cached input, and per-user speed is the 25th percentile (p25) across requests excluding time to first token, which is the speed we quote throughout. Each point is a serving config at a set number of agents, and the arrow marks Vera Rubin’s largest lead over GB300.

• ➤At peak throughput, Vera Rubin NVL72 serves 272M tokens/s per MW on DeepSeek V4 Pro vs 239M for GB300 NVL72 (1.14x), 571M vs 395M on DeepSeek V4.1 Flash (1.45x), 272M vs 201M on GLM-5.2 (1.35x) and 70M vs 56M on Kimi K3 (1.26x)
• ➤Vera Rubin’s lead is largest where GB300’s throughput drops away, reaching ~5.7x on DeepSeek V4 Pro at ~190 tok/s per user, ~5.4x on DeepSeek V4.1 Flash at ~580 tok/s, ~3.0x on GLM-5.2 at ~165 tok/s and ~2.8x on Kimi K3 at ~67 tok/s. Going from ~120 to ~165 tok/s per user on DeepSeek V4 Pro costs GB300 81% of its throughput per MW, compared to 16% for Vera Rubin
• ➤Per GPU, Vera Rubin’s lead is wider again: 2.0x to 2.5x at peak throughput, and up to ~12x on DeepSeek V4 Pro at ~165 tok/s per user, ~18x on DeepSeek V4.1 Flash at ~580 tok/s, ~7x on GLM-5.2 between ~200 and ~310 tok/s and ~7x on Kimi K3 at ~176 tok/s
• ➤Vera Rubin’s fastest configs set the highest per-user speeds on our leaderboard for DeepSeek V4.1 Flash (906 tok/s, the fastest result for any model), GLM-5.2 (408 tok/s) and Kimi K3 (255 tok/s)

Greater power efficiency lowers the operating cost of serving each token, and as AI datacenters are increasingly limited by the power they can draw rather than the number of accelerators they can install, it also allows more agents to be served out of the same datacenter. At ~165 tok/s per user, for example, the same GPU power budget serves ~5x as many DeepSeek V4 Pro tokens on Vera Rubin NVL72 as on GB300 NVL72. Every point on our leaderboard comes with the serving config behind it in our configurations browser, so teams can start from a tuned setup for their own hardware rather than running their own performance study.

DeepSeek V4 Pro has 1.6T total and 49B active parameters, and uses a hybrid of compressed sparse and heavily compressed attention to keep its KV cache small at long context.

• ➤At peak throughput Vera Rubin NVL72 serves 272M tokens/s per MW at 69 tok/s per user, 1.14x GB300 NVL72’s 239M at 67 tok/s, and stays ~1.1x ahead up to ~120 tok/s per user
• ➤The gap opens up past ~130 tok/s per user as GB300’s throughput falls away, with Vera Rubin serving 174M tokens/s per MW at ~165 tok/s per user vs 34M for NVIDIA’s GB300 configs (5.1x), and 132M at ~190 tok/s vs ~23M for GB300 (~5.7x). NVIDIA’s GB300 rack configs top out at ~163 tok/s per user, so beyond that the GB300 line comes from small 4-GPU configs we built
• ➤Vera Rubin’s fastest config runs at 298 tok/s per user, and above ~255 tok/s per user those 4-GPU GB300 configs serve more tokens per MW than Vera Rubin, extending GB300’s results to 340 tok/s
• ➤Vera Rubin’s configs take longer to return the first token (a median of 1.65 s at ~165 tok/s per user, compared to 1.08 s for GB300), and measured against end-to-end speed, which includes that wait, GB300 matches or beats Vera Rubin’s throughput per MW at most speeds

DeepSeek V4.1 Flash has 552B backbone parameters, of which it activates 8B per token during prefill and 16B during decode, plus a 196B-parameter Engram memory that it reads by token lookup. Its sparse attention and FP4 KV cache keep the KV cache to ~890 bytes per token, around a quarter of DeepSeek V4 Flash’s.

• ➤Vera Rubin NVL72 peaks at 571M tokens/s per MW at 68 tok/s per user, the highest throughput per MW on our leaderboard across all four models, ahead of B300’s peak of 433M and GB300 NVL72’s 395M (at 31 and 23 tok/s per user)
• ➤Vera Rubin’s lead over GB300 grows with per-user speed, from ~1.8x at 68 tok/s to ~2.9x at ~165 tok/s and ~3.4x to ~3.6x between ~230 and ~350 tok/s, reaching ~5.4x at ~580 tok/s per user with a 12-GPU config that spreads decode across five workers
• ➤Vera Rubin’s fastest result, 906 tok/s per user from a single 4-GPU server serving one agent, is the fastest result on our leaderboard for any model, ahead of 669 tok/s on B200 and 596 tok/s on GB300 for DeepSeek V4.1 Flash
• ➤Per GPU, Vera Rubin serves 651K tokens/s at peak, 2.3x GB300’s 284K, and ~6.5x the GB300 line at ~300 tok/s per user

NVIDIA’s Vera Rubin runs on DeepSeek V4.1 Flash capped replies at 2,048 output tokens, compared to 16,384 for the other DeepSeek V4.1 Flash results on our leaderboard. About 2% of replies reached the cap and the average reply was about a tenth shorter (275 to 301 tokens vs 319 to 335), which modestly favors Vera Rubin’s throughput.

GLM-5.2 has 753B total and 40B active parameters with DeepSeek-style sparse attention. We plot GLM-5.2 and GLM-5.3 on one leaderboard chart as they share an architecture.

• ➤Vera Rubin NVL72 peaks at 272M tokens/s per MW at 84 tok/s per user, 1.35x GB300 NVL72’s 201M at 68 tok/s per user
• ➤At ~165 tok/s per user Vera Rubin serves 159M tokens/s per MW, ~3.0x the GB300 line at the same speed (~53M), and the gap stays between ~2.7x and ~3x out to ~310 tok/s per user
• ➤Vera Rubin’s fastest result, 408 tok/s per user, is the fastest GLM-5.2 result on our leaderboard, ahead of 333 tok/s on B300 and 330 tok/s on GB300
• ➤Measured against end-to-end speed, which includes time to first token, Vera Rubin still serves 1.1x to 1.8x GB300’s throughput per MW across the range

Kimi K3 is the largest model we feature, with 2.8T total and 104B active parameters, and a hybrid design that uses linear attention in 69 of its 93 layers.

• ➤Every system serves far fewer tokens per MW on Kimi K3 than on the other three models, with Vera Rubin NVL72 peaking at 70M tokens/s per MW at 41 tok/s per user, 1.26x GB300 NVL72’s peak of 56M at 40 tok/s
• ➤At ~67 tok/s per user Vera Rubin serves ~2.8x the GB300 line’s throughput per MW (59M vs ~21M), and its 30-agent config serves ~2.4x the GB300 line at 176 tok/s per user (9.6M vs ~4.0M). NVIDIA’s current Vera Rubin configs leave a gap between ~74 and ~176 tok/s per user
• ➤Vera Rubin’s fastest result, 255 tok/s per user from a single 8-GPU vLLM server, is the fastest Kimi K3 result on our leaderboard, ahead of B300 (235) and GB300 (194)
• ➤Time to first token is long at peak throughput on both systems, with a p95 of 9.3 s for Vera Rubin’s 2,048-agent config and 19 s for GB300’s 768-agent config

NVIDIA shared how it built its Vera Rubin and GB300 configs. The current set of configs is very similar between Vera Rubin and GB300, so most of the difference in performance comes from the upgraded hardware of the newer rack generation, alongside early Rubin-specific kernels and tuning. The techniques include:

• KV caching across memory tiers. With 97% of a request’s input tokens on average repeated from the request before it, prefix caching is essential, and deployments need every memory tier from HBM to host memory (and eventually disk). Most of the Vera Rubin configs offload KV cache to CPU memory over NVLink-C2C, which Vera doubles to 1.8 TB/s per superchip, with the transfers overlapping other GPU work, as long agentic contexts expose GPU-to-CPU bandwidth in a way short synthetic prompts don’t. The DeepSeek V4 Pro prefill workers have 120 to 200 GiB of host memory configured for KV cache. The DeepSeek V4.1 Flash configs instead keep the model’s 196B-parameter Engram memory in CPU memory rather than HBM.
• Session-aware routing with NVIDIA Dynamo. Agent sessions return to the same context turn after turn, so it pays to send each request to the worker already holding its cache. Every Vera Rubin config runs behind NVIDIA Dynamo, NVIDIA’s open-source serving framework, which keeps each agent’s turns on the same prefill worker and, on DeepSeek V4 Pro, GLM-5.2 and Kimi K3, sends each request to the worker holding the most of its prefix in cache. Kimi K3’s linear-attention layers add a complication, as their recurrent state can’t be rolled back to an earlier point like a standard KV cache, so reusing it needs a snapshot saved at the right place.
• Disaggregated serving with wide expert parallelism. A turn can add anywhere from a few tokens to tens of thousands, so in all but the lowest-concurrency configs prefill runs on separate workers, where a large tool output doesn’t hold up decoding for every other agent. NVIDIA sizes each side for the concurrency level, which on DeepSeek V4 Pro means one to six 4-GPU prefill workers as concurrency rises, with the model’s experts spread across up to 32 decode GPUs and each request’s KV cache passed from prefill to decode over NVLink. Kimi K3’s highest-throughput configs also use DWDP (distributed weight data parallelism), a TensorRT-LLM prefill technique described in this paper.
• Speculative decoding tuned to batch size. All four models use speculative decoding (DSpark draft models for DeepSeek V4 Pro and Kimi K3, DeepSeek V4.1 Flash’s built-in DSpark drafting and GLM-5.2’s own multi-token prediction layers), with the draft length tuned for each concurrency level, e.g. from 7 tokens at lower concurrency to 3 at the highest on DeepSeek V4 Pro, as speculation pays off most when batches are small and GPUs have compute to spare. The config behind the ~5.7x result on DeepSeek V4 Pro, at ~190 tok/s per user, drafts 7.
• Rubin-tuned kernels. NVIDIA’s NVFP4 checkpoints run the MoE experts in 4-bit precision on NVFP4 kernels tuned for Rubin, alongside fused kernels for each model’s architecture (such as DeepSeek V4 Pro’s sparse attention and Hyper-Connections). The same attention kernel has to handle inputs anywhere from a handful of tokens to 1M, so autotuning has to cover ranges of shapes instead of specializing on fixed ones, which real-world agentic workloads need and fixed-shape benchmarks don’t exercise.
• Fast host-side request handling. Every request carries the whole conversation so far (up to 132K tokens in our workload), so the serving frontend has to tokenize and forward a long history on every turn. Dynamo’s router is written in Rust to keep host-side latency low, and the DeepSeek V4 Pro and GLM-5.2 configs add a fast tokenizer with a tokenization cache and streamlined request transport, while the configs for the other models pin each worker to the CPU cores and memory nearest its GPUs.

These are first results on Vera Rubin, with early Rubin-specific kernels and tuning, and NVIDIA expects performance to keep improving as further optimizations land.