Changelog

Announcing our new Speech Agent Arena, evaluating Speech to Speech models on real-world scenarios to analyze conversational preference and task success rate

What's changed

• Upgraded Terminal-Bench v2 to Terminal-Bench v2.1, covering the full 89-task set.
• Added reward hacking detection aligned with Terminal-Bench's integrity methodology, scoring reward-hacked trials 0.
• Revised token counting and cost methodology for agents that report reasoning within output tokens.

Benchmark composition and weighting are unchanged from v1.3.

Learn more about who we are and what we do.

The Artificial Analysis Search Index benchmarks how search API providers perform on quality, cost, and speed when used by an agent. We compare different Search API providers across a series of search-related benchmarks using the same agentic setup.

Google has released Gemini 3.7 Flash, improving 4 points over Gemini 3.6 Flash and reaching the Intelligence vs. Time per Task Pareto frontier

Optima is a new platform for benchmarking models on your own workloads. Build a benchmark from your own files, agent traces or coding environment, run it across leading models in a single click, and compare quality alongside cost per task and time per task.

Upstage has released Solar Pro 4

SpaceXAI returns to the intelligence frontier with strengths in agentic performance and cost efficiency

Efficient on-device scale intelligence

Independent analysis and benchmarks of Meta's Muse Glimmer

Our latest patch release, upgrading grader models and 𝜏³-Banking

What's changed

• 𝜏³-Banking moved to upstream v1.0.1: The evaluation now runs the tau2-bench v1.0.1 dataset and grader, replacing the frozen v1.0 dataset.
• Upgraded grader for HLE, AA-LCR and AA-Omniscience: All three are now graded by GPT-5.6 Luna (medium), replacing GPT-4o (Aug '24), Qwen3 235B A22B 2507 Non-Reasoning and Gemini 3 Flash Preview (Reasoning) respectively.

The contributing evaluations and their weights are unchanged from v4.1.

Independent analysis and benchmarks of Meta's Muse Spark 1.2

Announcing the Artificial Analysis Endpoint Accuracy Index, measuring how much of an open weights model's accuracy each serverless API endpoint preserves

DeepSeek releases DeepSeek V4 Flash 0731

Thinking Machines' new Inkling Small scores 40 on the Artificial Analysis Intelligence Index, within a point of its flagship sibling Inkling with less than a third of the total and active parameters

Agnes 2.5 Pro Alpha

Claude Opus 5 is the new leader on our agentic knowledge work benchmark, AA-Briefcase, outperforming Claude Fable 5 by nearly 150 Elo while reducing Cost per Task by 20%

Claude Opus 5 is narrowly the most intelligent model on the Artificial Analysis Intelligence Index, offering comparable intelligence to Fable 5 at 26% lower Cost per Task

Thinking Machines Lab’s Inkling scores an Elo of 836 on on our agentic knowledge work benchmark AA-Briefcase

Kimi K3 is second only to Fable 5 on AA-Briefcase, our agentic knowledge work benchmark, but costs more than Opus 4.8 to run while averaging nearly an hour per task

Google has released Gemini 3.6 Flash and Gemini 3.5 Flash-Lite. Both halve time per task relative to their predecessors and increase token efficiency, Gemini 3.5 Flash-Lite improves by 11 Intelligence Index points while Gemini 3.6 Flash does not improve in intelligence over 3.5 Flash

Yeah. Grok 4.5, GPT-5.6, Muse Spark 1.1, and Kimi K3 all launched within eight days. Six labs now have a model scoring above 50 on the Artificial Analysis Intelligence Index, up from two in early June - and the price of near-frontier intelligence has collapsed

Benchmarks and Analysis of Kimi K3

Thinking Machines has released Inkling, the new leading U.S. open weights model, debuting at 41 on the Artificial Analysis Intelligence Index

GPT-5.6 Sol and Luna are ahead of Terra at every point on the Intelligence vs Cost per Task chart. GPT-5.6 Luna stands out as a particularly cost efficient model

Meta's Muse Spark 1.1 scores 51 on the Artificial Analysis Intelligence Index and is cost and token efficient compared to its peers

GPT-5.6 Sol comes close second to Claude Fable 5 in the Artificial Analysis Intelligence Index at one third of the cost, and leads the Artificial Analysis Coding Agent Index in OpenAI’s Codex harness

Our independent leaderboard for ServiceNow’s EnterpriseOps-Gym

SpaceXAI’s Grok 4.5 scores 54 to place fourth on the Artificial Analysis Intelligence Index following only Fable 5, GPT-5.5, and Opus 4.8. It scores on par with GPT-5.5 in Codex on the Artificial Analysis Coding Agent Index in the Grok Build harness, at much lower cost

Our implementation of Harvey's Legal Agent Benchmark tests language models on 120 private legal tasks spanning 24 practice areas, graded against rubrics of binary criteria.

Our independent leaderboard for Zapier’s AutomationBench

Claude Sonnet 5 achieves 53 on the Artificial Analysis Intelligence Index, but without promotional pricing will cost more per task than Opus 4.8

Agentic knowledge work can take frontier models over 20 minutes per task, as measured in AA-Briefcase, our new benchmark

Announcing the Artificial Analysis Speech to Speech Index, our new synthesis metric for native Speech to Speech model quality, comprising of Big Bench Audio, Full Duplex Bench, and 𝜏-Voice

AA-Briefcase is a new benchmark for testing models on realistic knowledge work tasks in complex projects built by industry experts. Models are evaluated on multi-week knowledge work projects, each with many linked tasks and thousands of input source files, combining rubric and pairwise grading to evaluate verifiable task success, analytical quality, and presentation quality.

Benchmarks and Analysis of GLM-5.2

Announcing Artificial Analysis Intelligence Index v4.1: a shift toward agentic workloads, featuring upgraded benchmarks and new per-task metrics

AA-AgentPerf measures how many concurrent agents an AI system can serve on real coding-agent trajectories while meeting production service-level targets, with Agents per Megawatt as its lead metric. The first results cover NVIDIA and AMD systems, from single accelerators to full racks.

An Artificial Analysis benchmark report for guardrail safety models.

Anthropic is nearly 5 points ahead of any other lab’s best model

Anthropic has released Claude Fable 5, the first publicly available Mythos-class model that ranks #1 in our agentic real-world knowledge work benchmark GDPval-AA

Cohere just released North Mini Code, a small 30B parameter (3B active) open weights coding model that scores 27.6 on the Artificial Analysis Intelligence Index