Hardware & Inference Stack Benchmarking

Benchmark

AA-AgentPerf: The Hardware Benchmark for the Agent Era

AA-AgentPerf has been shaped by our work with inference providers and engagement with AI accelerator companies, developers, and enterprise buyers over the past year. It tests inference hardware and software by replaying real coding agent trajectories, measuring how well each system handles long context, tool-calling delays, substantial KV cache reuse opportunities, and other features of agentic work.

AA-AgentPerf is open for submissions, and results will be published on a rolling basis.

Results
Model
View all serving configurations →

Concurrent Agents per MW vs. End-to-End Speed

Concurrent agents per megawatt of accelerator power vs. p25 end-to-end speed per query · Output tokens per second over the full request, including time to first token
Serving configuration changes

Output tokens per second measured across the whole request, from initial request to delivery of the final token. Includes the effect of both time to first token (TTFT), i.e. latency, and output speed. Charts show the p25 (25th percentile) figure, i.e. the speed exceeded by three-quarters of the requests measured.

Values are normalized by GPU power draw in megawatts, based on measured GPU-only power (GPU die plus HBM, excluding CPUs, networking, and cooling overhead).

MethodologyRead the full methodology

Inside a single agent trajectory

Each simulated agent replays a real multi-turn coding session — reasoning, calling tools, and editing code. Context compounds turn over turn: prior turns are served from KV cache, only new tokens are prefilled, and trajectories grow beyond 100K tokens. This access pattern is what separates agent serving from synthetic benchmarks.