AA-AgentPerf: The Hardware Benchmark for the Agent Era

AA-AgentPerf has been shaped by our work with inference providers and engagement with AI accelerator companies, developers, and enterprise buyers over the past year. It tests inference hardware and software by replaying real coding agent trajectories, measuring how well each system handles long context, tool-calling delays, substantial KV cache reuse opportunities, and other features of agentic work.

AA-AgentPerf is open for submissions, and results will be published on a rolling basis.

Simple throughput, output speed, TTFT on synthetic inference workloads

View all serving configurations →

Concurrent Agents per MW vs. End-to-End Speed

Concurrent agents per megawatt of accelerator power vs. p25 end-to-end speed per query · Output tokens per second over the full request, including time to first token
Serving configuration changes

Output tokens per second across the whole request, from sending it to receiving the final token. Unlike output speed, it counts the wait for the first token, so prefill delay and generation rate fold into a single figure. Charts show the 25th percentile across requests — the speed three quarters of requests exceeded.

Values are normalized by GPU power draw in megawatts, based on measured GPU-only power (GPU die plus HBM, excluding CPUs, networking, and cooling overhead).

Methodology

Read the full methodology

Inside a single agent trajectory

Each simulated agent replays a real multi-turn coding session — reasoning, calling tools, and editing code. Context compounds turn over turn: prior turns are served from KV cache, only new tokens are prefilled, and trajectories grow beyond 100K tokens. This access pattern is what separates agent serving from synthetic benchmarks.