AA-AgentPerf Methodology

Overview

AA-AgentPerf is a hardware benchmark that measures the trade-off between an inference deployment's total throughput and the speed each agent receives under realistic agentic workloads.

• Real agentic trajectories: Multi-turn coding sessions with interleaved reasoning, tool calls, and variable context lengths (not synthetic uniform prompts).
• Sustained concurrent load: Every simulated agent stays active for the whole test phase, which stresses KV cache reuse, speculative decoding, and scheduler behavior.
• Live leaderboard: We publish results on a rolling basis, adding hardware as it becomes available and re-measuring systems on new software stacks and model versions.
• Production ready: We test models with realistic optimizations enabled and production-scale deployment topologies.

Dataset

The AA-AgentPerf dataset contains real agentic trajectories covering several use cases, programming languages, and models. We generated the trajectories in the OpenCode agentic harness by prompting three top open-source models with reasoning enabled (DeepSeek V3.2, GLM 4.7, and Kimi K2.5) to resolve issues in real public code repositories. All trajectories include interleaved reasoning and tool calls.

• Input Sequence Length (ISL): Ranges from ~5K to ~131K tokens, with a mean of approximately 27K tokens. Trajectories are truncated to fit within the maximum recommended context length of the models under test.
• Output Sequence Length (OSL): Output lengths vary widely across turns. Some turns produce short outputs such as simple tool calls, while others involve extended reasoning before the model responds or acts.
• Languages: 12+ programming languages are represented, based on the primary language of the source repository. Python repositories are the most frequent source, followed by TypeScript and Go.
• Tool call delays: After each tool call, we simulate processing time with a fixed per-message delay sampled from a distribution of real tool call durations, partitioned by the tool being called. Delays range from under 0.1s up to 5s, with a median of approximately 1 second.

We provide test participants with a representative tuning subset of 500 unique trajectories (18,997 prompts) for configuration validation and performance tuning. The full test dataset is kept private to prevent benchmark-targeted optimizations.

Pareto Frontier

For each system, we publish results as a Pareto frontier of total server throughput against per-request output speed. Each point is one serving configuration at one concurrency level, and the frontier keeps the points that no other configuration on the same system beats on both throughput and speed.

We calculate speed and latency per request. Because agentic workloads contain many small-OSL requests, charts default to P25 output speed, with percentiles computed over all requests sent during a phase.

Test Execution

We measure each deployment at a fixed set of concurrency levels, one phase per level, and calculate metrics from steady-state token timing information. Each phase runs until at least 30 trajectories have completed, every simulated agent has completed at least 3 trajectories, and at least 10 minutes of steady-state measurement time has elapsed. Agent trajectory assignments are deterministic across phases, with dynamically generated prefixes added at the start of each trajectory in each phase to break prefix caching between phases. A max_tokens value of 16K prevents individual requests where a model gets stuck in a repetition loop from skewing results.

We test each model with creator-recommended sampling parameters at the maximum reasoning effort available (max for DeepSeek V4.1 Flash, DeepSeek V4 Pro 0813, Kimi K3, GLM-5.2, and GLM-5.3), with all other parameters set to defaults.

Metrics & Results

We calculate these metrics in each test phase:

• Time to First Token (TTFT): Per-request latency from sending the request to receiving the first output token.
• Output Speed: Per-request output tokens per second, measured after the first token is received.
• Total Server Throughput: Input and output tokens per second across all concurrent agents, as reported by the server. Input includes prompt tokens served from KV cache.

All metrics are filtered to steady-state periods where all agents have been active for at least 30 seconds at target concurrency. Speed metrics use server-side token counts from the usage metadata returned by the inference framework, verified against local tokenization with the candidate model's native tokenizer. We measure power draw from each accelerator during the live test. We normalize the primary result, total server throughput, per MW, per accelerator, and per dollar per hour of accelerator rental. Per-MW figures are based on measured accelerator power under load (GPU die + HBM) rather than rated TDP. Rental prices come from Silicon Data, with our own estimates for chips Silicon Data does not price.

We publish results on the Artificial Analysis leaderboard with the full system configuration each provider discloses. Providers may review their results for factual accuracy before publication.

Submissions

If you are a hardware vendor interested in submitting your system for AA-AgentPerf evaluation and are not already in touch, reach out to agentperf@artificialanalysis.ai.