Changelog

GPT-6 Astra ties leadership with Claude Fable 5.1 in both of our flagship Indices, at lower cost. Astra equals Fable 5.1 in the Intelligence Index at ~40% of the cost, and in the Coding Agent Index at ~60% of the cost.

OpenBMB releases MiniCPM5-2B

We are upgrading Terminal-Bench to v4 and adding AutomationBench-AA, an agentic workflow automation benchmark with a private test set. This is a continuation of our rollout of Intelligence Index v5.

What's changed

• Terminal-Bench moved to v4.0: A harder 66-task set replacing v2.1, with recalibrated compute and time allowances and improved environments and verifiers.
• 𝜏³-Banking replaced by AutomationBench-AA: Our implementation of Zapier's AutomationBench, testing agents on 657 business workflows across simulated applications while following business rules, using Zapier's private question set.

Category weights are unchanged from v4.2: Agents 30%, Coding 20%, General 30%, Scientific Reasoning 20%.

We are accelerating elements of our upcoming v5 release with interim updates to keep pace with the frontier. Index v4.2 has more complex and realistic tasks, and more private test sets to prevent gaming

Independent analysis and benchmarks of Meta's Muse Spark 1.3

Gemini 3.8 Flash scores 59 on the Artificial Analysis Intelligence Index and reaches the Intelligence vs. Cost per Task Pareto frontier

Anthropic's new frontier model scores 66 at max effort and leads on agentic knowledge work, but the 75% cache read price cut only partly offsets higher token usage

Agnes 2.5 Pro Beta

Announcing our new Speech Agent Arena, evaluating Speech to Speech models on real-world scenarios to analyze conversational preference and task success rate

What's changed

• Upgraded Terminal-Bench v2 to Terminal-Bench v2.1, covering the full 89-task set.
• Added reward hacking detection aligned with Terminal-Bench's integrity methodology, scoring reward-hacked trials 0.
• Revised token counting and cost methodology for agents that report reasoning within output tokens.

Benchmark composition and weighting are unchanged from v1.3.

Learn more about who we are and what we do.

The Artificial Analysis Search Index benchmarks how search API providers perform on quality, cost, and speed when used by an agent. We compare different Search API providers across a series of search-related benchmarks using the same agentic setup.

Google has released Gemini 3.7 Flash, improving 4 points over Gemini 3.6 Flash and reaching the Intelligence vs. Time per Task Pareto frontier

Optima is a new platform for benchmarking models on your own workloads. Build a benchmark from your own files, agent traces or coding environment, run it across leading models in a single click, and compare quality alongside cost per task and time per task.

Upstage has released Solar Pro 4

SpaceXAI returns to the intelligence frontier with strengths in agentic performance and cost efficiency

Efficient on-device scale intelligence

Independent analysis and benchmarks of Meta's Muse Glimmer

Our latest patch release, upgrading grader models and 𝜏³-Banking

What's changed

• 𝜏³-Banking moved to upstream v1.0.1: The evaluation now runs the tau2-bench v1.0.1 dataset and grader, replacing the frozen v1.0 dataset.
• Upgraded grader for HLE, AA-LCR and AA-Omniscience: All three are now graded by GPT-5.6 Luna (medium), replacing GPT-4o (Aug '24), Qwen3 235B A22B 2507 Non-Reasoning and Gemini 3 Flash Preview (Reasoning) respectively.

The contributing evaluations and their weights are unchanged from v4.1.

Independent analysis and benchmarks of Meta's Muse Spark 1.2

Announcing the Artificial Analysis Endpoint Accuracy Index, measuring how much of an open weights model's accuracy each serverless API endpoint preserves

DeepSeek releases DeepSeek V4 Flash 0731

Thinking Machines' new Inkling Small scores 40 on the Artificial Analysis Intelligence Index, within a point of its flagship sibling Inkling with less than a third of the total and active parameters

Agnes 2.5 Pro Alpha

Claude Opus 5 is the new leader on our agentic knowledge work benchmark, AA-Briefcase, outperforming Claude Fable 5 by nearly 150 Elo while reducing Cost per Task by 20%

Claude Opus 5 is narrowly the most intelligent model on the Artificial Analysis Intelligence Index, offering comparable intelligence to Fable 5 at 26% lower Cost per Task

Thinking Machines Lab’s Inkling scores an Elo of 836 on on our agentic knowledge work benchmark AA-Briefcase

Kimi K3 is second only to Fable 5 on AA-Briefcase, our agentic knowledge work benchmark, but costs more than Opus 4.8 to run while averaging nearly an hour per task

Google has released Gemini 3.6 Flash and Gemini 3.5 Flash-Lite. Both halve time per task relative to their predecessors and increase token efficiency, Gemini 3.5 Flash-Lite improves by 11 Intelligence Index points while Gemini 3.6 Flash does not improve in intelligence over 3.5 Flash

Yeah. Grok 4.5, GPT-5.6, Muse Spark 1.1, and Kimi K3 all launched within eight days. Six labs now have a model scoring above 50 on the Artificial Analysis Intelligence Index, up from two in early June - and the price of near-frontier intelligence has collapsed

Benchmarks and Analysis of Kimi K3

Thinking Machines has released Inkling, the new leading U.S. open weights model, debuting at 41 on the Artificial Analysis Intelligence Index

GPT-5.6 Sol and Luna are ahead of Terra at every point on the Intelligence vs Cost per Task chart. GPT-5.6 Luna stands out as a particularly cost efficient model

Meta's Muse Spark 1.1 scores 51 on the Artificial Analysis Intelligence Index and is cost and token efficient compared to its peers