First results from AA-AgentPerf: the hardware benchmark for the agent era

Every simulated agent in AA-AgentPerf works through a real coding-agent trajectory, generated by agents resolving issues in public code repositories using top open weights models with reasoning enabled. Sessions run up to 200 turns and reach sequence lengths beyond 100K tokens:

• Input lengths range from ~5K to ~131K tokens per request, with a mean of roughly 27K — driven by tool outputs and accumulated history rather than the prompt itself
• Output lengths vary widely across turns: agents mostly emit short tool calls and edits, punctuated by longer stretches of reasoning
• 12+ programming languages are represented, based on the primary language of the source repository
• The test set stays private. Participants receive a representative tuning subset for configuration validation; the full dataset is held out to prevent benchmark-targeted optimization.

This shape is what makes agent serving hard, and what uniform-length synthetic benchmarks miss entirely. Tool results balloon the context, outputs are often only a few hundred tokens, and the same prefix comes back turn after turn. A system’s KV cache behavior, scheduler, and memory hierarchy decide whether it thrives or collapses under this pattern.

AA-AgentPerf is the first inference benchmark to allow the optimizations that labs and inference providers actually run in production: KV cache reuse, speculative decoding, and disaggregated prefill/decode. Benchmarks that switch these off measure a deployment style nobody ships. Allowing them captures what real deployments should look like. It also makes AA-AgentPerf a tool for understanding the impact of each new software and hardware advance, not just each new chip. Accuracy verification controls for quality loss, so an optimization cannot buy capacity at the expense of output quality.

Allowing production optimizations makes serving configuration a first-class part of the benchmark. The solution space for serving a model on a given system is wide — inference framework and version, parallelism strategy, speculative decoding setup, disaggregation topology, and dozens of tuning decisions — and it is moving fast: since DeepSeek V4 Pro’s release we have seen serving configurations for it improve on a near-daily basis.

Results reach the leaderboard through two routes, and every published result discloses which: vendors submit tuned configurations for their own systems, which we run and verify, or Artificial Analysis builds a configuration in-house where no vendor submission is available yet. In the launch results for DeepSeek V4 Pro, the B300 and GB300 configurations were submitted by NVIDIA, while the H200 and MI355X results run configurations built by Artificial Analysis. The full serving configuration behind every number, and who submitted it, is in the configurations browser.

The numbers in this initial release of AA-AgentPerf should be read as a snapshot of a fast-moving frontier, like other live performance benchmarking results reported on Artificial Analysis. These are real results that reflect performance attainable today with the disclosed configurations, but we caution that a variable amount of headroom is potentially available for greater performance on each system, and we expect results to climb as further vendor submissions are made. In particular, that headroom may be greater for the MI355X and H200, as their configurations were submitted by the Artificial Analysis team rather than a vendor team.

Peak throughput is easy to inflate: pile on concurrency until every agent is crawling. AA-AgentPerf instead fixes the level of service and asks how far a system can scale while preserving it. Performance targets are derived from Artificial Analysis serverless API benchmarking data — the tiers of service that actually exist in the market today. Speed and latency are measured per request: P25 output speed and P95 time to first token, computed across all requests in a test phase.

For each tier, an exponential ramp followed by a binary search finds the maximum concurrency the system sustains without violating the targets, with metrics computed over steady-state windows where all agents have been active for at least 30 seconds.

The first wave of results covers NVIDIA and AMD systems running DeepSeek V4 Pro, measured from single accelerators up to full racks, with gpt-oss-120b results to follow. Three findings stand out:

Every result below is normalized per accelerator, per system, and per megawatt, so systems of very different sizes can be compared fairly. The per-megawatt figures are computed on measured accelerator power under load, not rated thermal design power (TDP).

Note that with the relative infancy of kernel optimizations and config design for DeepSeek V4 Pro on AMD systems, we expect significant improvements in AMD performance in the near-term. We are looking forward to working with all vendors on measuring their improvement in performance over time.

The charts below are live views of the AA-AgentPerf leaderboard — the same data as the hardware benchmarks dashboard, updated as new results land. Switch between normalization lenses (per accelerator, per system, per megawatt) for agent capacity, and drill into the capacity search behind every number.

View all serving configurations →

Agent capacity

Concurrent agents and output throughput each system sustains at each output speed target.

Per MWPer Accelerator

Max Concurrent Agents per MW of Accelerator Power

Maximum concurrent agents sustained per megawatt of accelerator power at each service level objective (SLO)

GB300-NVL72B300 x8MI355X x8H200 x8

Concurrent Agents per MW of Accelerator Power vs. Output Speed

Concurrent agents per MW of accelerator power vs. p25 output speed per query (tokens per second)

H200

TP8 / EAGLE

MI355X

TP8 / radix cache / no specdec

B300

Disagg 1p1d / TEP4 ctx+gen / MTP3

GB300

Disagg 6p1d / ctx TP4 / gen DEP16 / MTP3Disagg 6p1d / ctx TP4 / gen DEP8 / MTP3

Search iterationMax passing concurrent agentsTTFT SLO violated

Power Normalization (Per MW)

Values are normalized by GPU power draw in megawatts, based on measured GPU-only power (GPU die plus HBM, excluding CPUs, networking, and cooling overhead).

AA-AgentPerf is a live benchmark. Submissions of configurations for benchmarking are open now, and results are published on a rolling basis as new hardware, software stacks, and model versions become available. Vendors can also submit improved configurations for systems already on the leaderboard; as serving software advances, published results are updated to reflect how real-world performance evolves. The benchmark is designed to evaluate every kind of system fairly, from a single accelerator to a full rack, and from DRAM-only architectures to SRAM-only designs and everything in between.

If you are a hardware vendor or inference provider and want your system benchmarked, reach out to agentperf@artificialanalysis.ai. We will be expanding model coverage, system coverage, and workload breadth on an ongoing basis.

AA-AgentPerf will be upgraded continuously. Planned expansions include:

• Longer context lengths — extending trajectories up to 1M tokens
• Broader model coverage — gpt-oss-120b results across all submitted systems, with additional models to follow
• Broader hardware coverage — results for more NVIDIA and AMD systems, and additional accelerator architectures
• Tool execution performance — testing CPU performance for executing agent tool calls as part of the benchmark
• Results over time — tracking how each system’s numbers improve as serving configurations and frameworks advance
• Deeper analysis of token economics and pricing — including agent capacity per $/hr of rental cost, cost per task, and total cost of ownership