Changelog

Korean AI Lab Upstage has released Solar Mini 4 which scores 24 on the Artificial Analysis Intelligence Index, but costs ~5x as much per task as GPT-6 Luna (max) despite similar per-token prices

Google’s new Gemini 4 Argon equals GPT-6 Astra on the Artificial Analysis Intelligence Index at 60% of the Cost per Task with discounted prices. Google is now back to being one of the top three labs in intelligence achieved

Our open-source tool for testing how fast agentic AI runs on laptops and workstations, with launch results for the DGX Spark, Ryzen AI Halo, MacBook Pro (M5 Pro) and RTX 5090 across four open-weights models

GPT-6.1 Sol replaces GPT-6 Sol after just 7 days. It scores 1 point below GPT-6 Astra in the Intelligence Index at less than one quarter of the Cost per Task

Anthropic's new Sonnet model scores 56, just 2 points behind Opus 5.5 (max), but at the highest Output Tokens per Task we have measured

The Artificial Analysis Cyber Index Alliance brings together industry partners to set a new standard for evaluating how AI models perform on enterprise cyber defense tasks. The Alliance launches alongside the Artificial Analysis Cyber Index, which combines three partner-contributed and open benchmarks to evaluate how well agents find and fix vulnerabilities.

GPT-6 Sol and Luna push the cost efficiency frontier by halving cost relative to GPT-5.6 Sol and Luna. Intelligence Index and Coding Agent Index scores remain level with GPT-5.6, with progress in some evaluations and regressions in others

Anthropic's new Opus scores 58 and arrives with a 20% price cut and a larger cache hit discount

Grok 4.7 scores 46 on the Artificial Analysis Intelligence Index to bring SpaceXAI into the top 4 AI labs. Coding Agent Index performance has also improved, overtaking GPT-5.6 Sol

GDPval-AA Elo is now anchored to DeepSeek V4.1 Flash (max) at 1600, with updates to fitting methodology for GDPval-AA and AA-Briefcase to improve Elo stability

Ant Group has released their finance-focused flash model Ling-3.0-flash-Fin

We are adding Agentic Tool Use sourced from AutomationBench-AA, AA-Briefcase to Agentic Knowledge Work, and GDP.pdf to Long-Context. Capability Indices v1.1 tunes each index more closely to the work it covers, combining slices of our core Intelligence Index v4.3 evaluations alongside specialized evaluations.

GPT-6 Astra ties leadership with Claude Fable 5.1 in both of our flagship Indices, at lower cost. Astra equals Fable 5.1 in the Intelligence Index at ~40% of the cost, and in the Coding Agent Index at ~60% of the cost.

OpenBMB releases MiniCPM5-2B

We are upgrading Terminal-Bench to 4.0 and adding AutomationBench-AA, an agentic workflow automation benchmark with a private test set. This is a continuation of our rollout of Intelligence Index v5.

What's changed

• Terminal-Bench moved to v4.0: A harder 66-task set replacing v2.1, with recalibrated compute and time allowances and improved environments and verifiers.
• 𝜏³-Banking replaced by AutomationBench-AA: Our implementation of Zapier's AutomationBench, testing agents on 657 business workflows across simulated applications while following business rules, using Zapier's private question set.

Category weights are unchanged from v4.2: Agents 30%, Coding 20%, General 30%, Scientific Reasoning 20%.

We are accelerating elements of our upcoming v5 release with interim updates to keep pace with the frontier. Index v4.2 has more complex and realistic tasks, and more private test sets to prevent gaming

Independent analysis and benchmarks of Meta's Muse Spark 1.3

Gemini 3.8 Flash scores 59 on the Artificial Analysis Intelligence Index and reaches the Intelligence vs. Cost per Task Pareto frontier

Anthropic's new frontier model scores 66 at max effort and leads on agentic knowledge work, but the 75% cache read price cut only partly offsets higher token usage

Agnes 2.5 Pro Beta

Announcing our new Speech Agent Arena, evaluating Speech to Speech models on real-world scenarios to analyze conversational preference and task success rate

What's changed

• Upgraded Terminal-Bench v2 to Terminal-Bench v2.1, covering the full 89-task set.
• Added reward hacking detection aligned with Terminal-Bench's integrity methodology, scoring reward-hacked trials 0.
• Revised token counting and cost methodology for agents that report reasoning within output tokens.

Benchmark composition and weighting are unchanged from v1.3.

Learn more about who we are and what we do.

The Artificial Analysis Search Index benchmarks how search API providers perform on quality, cost, and speed when used by an agent. We compare different Search API providers across a series of search-related benchmarks using the same agentic setup.