Announcing AA-AnalystAgent: an agentic benchmark for quantitative analysis on real-world spreadsheets and documents

AA-AnalystAgent is our new agentic benchmark for quantitative analysis on real-world spreadsheets and documents. The best model gets only 54% of tasks right on all five attempts — the top three are Claude Opus 5 at 54%, GPT-5.5 at 50% and Claude Fable 5 at 49%.

In real analyst roles, professional judgment and expertise are more important than arithmetic alone. It takes judgment to interpret sources, decide which exceptions and caveats apply, and settle on a methodology that produces a robust number.

When businesses and researchers hand analyst work to an AI agent, they need to be confident it will produce the correct answer every time. That's why AA-AnalystAgent runs every task five times and reports pass^5 as its headline metric. A task counts as solved only when the model answers correctly on all five independent attempts.

AA-AnalystAgent launches as a standalone leaderboard and is not part of the Artificial Analysis Intelligence Index.

Benchmark overview

• 🔢 80 questions across 14 business and scientific domains, including healthcare expenditure reports, trade and commodity statistics, hydrology and weather data, government appropriations, energy cost models, financial models, environmental reporting, and project schedules.
• 🧠 Five functional workflow buckets covering the spread of real analyst work: source lookup and diagnosis, filter and total, ratios/trends/sensitivities, P&L modeling, and cash/balance-sheet/valuation modeling.

Methodology

• 🤖 Agentic harness: each task is solved by an agent running in our open-source Stirrup reference harness, with tools for code execution, web fetch, image viewing, and answer submission.
• 📊 Scoring: each task is run five times per model, and each answer is graded binary correct or incorrect against a held-out reference answer by a numeric equivalence pre-check and an LLM judge. Three metrics: pass^5, pass@1 (average pass rate), pass@5.
• 🔒 Privately-held question set to limit contamination risk. The methodology page documents the agent harness, toolset, and both grading prompts in full.

Reliability separates the top of the AA-AnalystAgent leaderboard more than raw capability. GPT-5.5 (xhigh) has the highest pass@1 at 66%, with Gemini 3.1 Pro Preview and Claude Opus 5 (Adaptive Reasoning, Max Effort) close behind at 64%, but Opus 5 leads on pass^5 because it repeats what it gets right.

Google DeepMind's Gemini 3.1 Pro Preview solves 81% of tasks at least once but only 41% on all five, and finishes ninth. An analyst agent is only useful if its answers hold up without re-checking — an answer that is right at random still has to be verified, which is the work the agent was meant to remove.

We classified 1,567 failing AA-AnalystAgent attempts across ten leading models into seven failure modes, tagging each attempt with every mode it exhibited. The most widespread is anchoring on a wrong early hypothesis, present in 57% of failures. Models commit to a source or interpretation early and defend it for the rest of the trajectory.

The sharpest contrasts are in how models treat their sources. Gemini 3.1 Pro Preview takes sources at their word but fumbles the execution. It sits below the ten-model median on both misreading domain terms and overriding evidence in favor of a generic prior, while posting above-median rates of modeling, scaling and aggregation errors (51% of its failures) and skipped verification (39%). Grok 4.5 (high) is the reverse. It understands the field's language, then substitutes its own assumptions for what the documents say, misreading domain terms on 23% of its failures against a 38% median, while overriding evidence on 54% against a 43% median. Kimi K3 (max) misreads on 48%, 25 points clear of Grok 4.5.