EnterpriseOps-Gym-AA: An Agentic Benchmark for Enterprise Operations

Announcing EnterpriseOps-Gym-AA, our independent leaderboard for ServiceNow's EnterpriseOps-Gym, testing whether AI agents can run real enterprise operations while following business rules and policy.

EnterpriseOps-Gym-AA is our independent leaderboard for ServiceNow's EnterpriseOps-Gym benchmark, run on Artificial Analysis. As enterprises rush to put AI agents in front of live business systems, most benchmarks still stop at single-turn tool calls or read-only questions. EnterpriseOps-Gym-AA tests something much harder: whether an agent can carry out a multi-step task against a live enterprise environment where actions are irreversible and success depends on following policy, not just producing an answer.

EnterpriseOps-Gym-AA aims to be far more complex and challenging than prior enterprise agent benchmarks. That complexity is deliberate: it is built to test whether an agent can hold a real operational workflow together from start to finish, across many interdependent systems and steps, the way a capable employee would.

The benchmark drops agents into live enterprise systems and asks them to complete multi-step, stateful work across 8 business domains including HR, IT Service Management, and Customer Service, graded on the final state of the underlying databases rather than the steps taken. We've adapted the original EnterpriseOps-Gym benchmark to run in our Stirrup agent harness, which allows us to run the benchmark consistently across all models and to continue to add new models as they are released.

As we launch the eval, Claude Fable 5 (max) leads at 51%, ahead of Gemini 3.5 Flash (high, 50%), GPT-5.5 (xhigh, 47%), with GLM-5.2 (max, 43%) as the highest-scoring open weights model.

The full, live leaderboard is on the EnterpriseOps-Gym-AA eval page, which we keep current as new models are released.

Key elements of EnterpriseOps-Gym-AA:

➤ Stateful, multi-step enterprise work: Agents operate live enterprise systems where actions are often irreversible, so they cannot brute-force their way to a solution.

➤ Live tool use across 8 domains: Models act through live MCP servers over 164 database tables and 512 tools, spanning core business systems, collaboration tools, and cross-domain Hybrid tasks.

➤ Outcome-based SQL verification: Each task is graded by executable SQL checks on the final environment state, testing goal completion, state integrity, policy compliance, and unintended side effects.

➤ Built by ServiceNow Research: Developed by ServiceNow Research, with Mila and the Université de Montréal.

Key results:

➤ The frontier just clears 50%: Claude Fable 5 (max) leads at 51%, the only model clearly above half, with Gemini 3.5 Flash (high) close behind at 50%. Even the strongest models complete only about half of the 1,117 oracle-mode tasks.

➤ Models must pass all verification checks to succeed at a task: A single missed constraint often means failure, so while models pass most individual verifier checks (79% for the leader, Claude Fable 5), they complete far fewer whole tasks (51%).

➤ The hardest work is the structured, policy-heavy work: HR (26%) and IT Service Management (28%) are the toughest domains, while lighter collaboration work like Email (58%) and Drive (53%) is markedly easier.

➤ Open weights are within ~8 points: GLM-5.2 (max, 43%) leads the open weights, about eight points behind the overall leader, with DeepSeek V4 Pro (max, 40%) and Kimi K2.7 (40%) close behind.

We would like to thank ServiceNow, Mila, and the Université de Montréal, along with the benchmark authors, for their excellent work building EnterpriseOps-Gym, and appreciate their collaboration in launching it on Artificial Analysis.

Performance by domain

Beyond the overall ranking, models have distinct domain-level personalities. GPT-5.5 (xhigh) ranks 3rd overall and is the single best model at Customer Service, yet it falls to 20th of 28 on Teams, where Claude and Qwen models pull well ahead. Mistral Medium 3.5 is the mirror image: 18th overall, but 4th on Email, while near the bottom on the structured HR and ITSM workflows. A model's headline score can hide sharp per-domain strengths and weaknesses.

Cost per task

Cost per task ranges roughly 90x, from ~$0.01 for DeepSeek V4 Pro (max) to $0.93 for Claude Fable 5 (max), but higher spend doesn't reliably buy a higher score. DeepSeek V4 Pro reaches 40% at just ~$0.01 per task (thanks to DeepSeek's generous first-party API pricing), within ten points of the leader at a fraction of the frontier's cost. Claude Opus 4.8 (max), the second most expensive model at $0.79, reaches 44% - barely ahead of GLM-5.2 (max), which scores 43% for just $0.10.

Effort per task: turns and tokens

Models differ in how much work they put into each task, and it doesn’t always reliably pay off. The leader, Claude Fable 5 (max), is efficient at an average of 6.5 turns per task, while Gemini 3.5 Flash (high) takes 13.5 and Grok 4.5 (high) 17.4. Looking into these numbers, we find that Gemini 3.5 Flash tends to make more attempts on each task before giving up, while Grok 4.5 underperforms as at times it gets stuck in loops where it continually calls the same tools.

Sample task

A representative Hybrid task spans two live systems - Google Drive and IT Service Management - in one request. The task instructions:

Prepare for Monday's external audit: create a copy-protected Drive folder 'Audit 2025 - Financial Documents & Access Logs' and populate it with four documents — financial records showing $47.2m revenue with Q1–Q4 detail, a login-tracking spreadsheet, an audit checklist, and auditor access instructions. Star the folder, share it view-only with the two external auditors with access expiring February 28th, move the 'Disk Space' and 'Core-Switch-HQ Port Flapping' incidents to resolved — but not closed — documented as audit-ready, and delete the obsolete Q4 Budget Report.

Claude Opus 4.8 (max) solved this in 4 turns and 12 tool calls, batching independent operations in parallel: the folder creation, incident lookups, and file search together; then all four documents, both incident resolutions, and the deletion; finally both auditor permissions.

Grading never looks at those steps, but five SQL verifiers run against the final database state to check the each of the following: both incidents are resolved, the folder exists, it contains exactly four files, and an active viewer permission for the auditors is in place. All five must pass - an agent that closed the incidents instead of resolving them, or dropped a fifth file into the folder, scores zero. That all-or-nothing grading is why models pass most individual checks yet complete only around half of whole tasks.

What's next

EnterpriseOps-Gym-AA is a live leaderboard. We will continue to maintain it and add selected new models as they are released, and we will work with ServiceNow to keep it up to date with any updates ServiceNow makes to the benchmark.

Resources

• EnterpriseOps-Gym-AA leaderboard: https://artificialanalysis.ai/evaluations/enterprise-ops-gym-aa
• Methodology: https://artificialanalysis.ai/methodology/intelligence-benchmarking#enterprise-ops-gym-aa
• Official EnterpriseOps-Gym Leaderboard & information page: https://enterpriseops-gym.github.io/
• EnterpriseOps-Gym paper (arXiv 2603.13594): https://arxiv.org/abs/2603.13594
• Artificial Analysis Stirrup agent harness: https://github.com/ArtificialAnalysis/Stirrup