Metodología de los Índices de Capacidad de Artificial Analysis
Resumen
Cada Índice de Capacidad de Artificial Analysis combina un conjunto seleccionado de evaluaciones en una única puntuación compuesta que mide el rendimiento de un modelo en un caso de uso específico, ya sea una capacidad amplia como la programación o un campo profesional como el trabajo legal.
Cada evaluación que compone un índice se ejecuta de forma independiente por Artificial Analysis antes de combinarse en el índice. Como toda métrica de evaluación, los índices de capacidad tienen limitaciones y pueden no aplicarse directamente a cada caso de uso, pero ofrecen una síntesis útil para comparar modelos en el trabajo que importa a un dominio determinado.
Las evaluaciones subyacentes, incluyendo cómo se ejecuta y puntúa cada una, se documentan en la metodología de Benchmarking de Inteligencia.
Desglose de índices
Los índices son basados en habilidades o en industrias. Los índices de habilidades miden una capacidad que se aplica a distintos dominios, como Programación o trabajo Agéntico, y se calculan como un promedio con ponderación equitativa de sus evaluaciones componentes. Los índices de industria se centran en una sola profesión o campo, como Legal, Salud y Medicina, o Finanzas y Contabilidad, y ponderan un conjunto de capacidades según la frecuencia con que cada una aparece en tareas reales de ese campo.
Nuestras ponderaciones de tareas se basan en una taxonomía de actividades laborales al estilo de O*NET. Como toda métrica de evaluación, estos índices tienen limitaciones y pueden no aplicarse directamente a cada caso de uso. La tabla siguiente muestra los componentes y las ponderaciones de cada índice.
| Índice | Tipo | Ponderación | Capacidad | Evaluaciones | Descripción |
|---|---|---|---|---|---|
| Agentic Index | Skill | 50% | Agentic Knowledge Work | GDPval-AA v2 | Tool use, planning, and multi-step execution of real-world knowledge work |
| 50% | Agentic Customer Interaction | 𝜏³-Banking | Multi-turn, tool-using customer-support workflows over a knowledge base | ||
| Coding Index | Skill | 50% | Agentic Terminal Use | Terminal-Bench v2.1 | Operating real terminal environments for builds, scripts, and debugging |
| 50% | Code Generation | SciCode | Generating and executing scientific research code | ||
| Finance & Accounting Index | Industry | 30% | Business Knowledge | AA-Omniscience | Domain recall in accounting, corporate finance, economics, and investments |
| 30% | Agentic Knowledge Work | GDPval-AA v2 | Tool use, planning, and multi-step task execution, such as running spreadsheets, querying ERPs, or coordinating a workflow | ||
| 20% | Reasoning | HLE | Multi-step quantitative and analytic reasoning, used for sensitivity analysis, valuation, and structured problem solving | ||
| 10% | Agentic Customer Interaction | 𝜏³-Banking | Multi-turn, tool-using customer workflows over an unstructured knowledge base | ||
| 5% | Long-Context | LCR | Reading and reasoning across long financial filings, deal documents, and research notes | ||
| 5% | Non-Hallucination | AA-Omniscience | Avoiding fabricated figures or citations | ||
| Strategy & Ops Index | Industry | 30% | Business Knowledge | AA-Omniscience | Working knowledge of business processes, accounting basics, and operational concepts |
| 30% | Agentic Knowledge Work | GDPval-AA v2 | Tool use, planning, and orchestrating multi-step office workflows end-to-end | ||
| 30% | Agentic Customer Interaction | 𝜏³-Banking | Multi-turn dialogue and tool use to resolve customer and stakeholder requests | ||
| 5% | Instruction Following | IFBench | Following exact formats and policy constraints | ||
| 5% | Long-Context | LCR | Holding context across long threads, policies, and records | ||
| Legal Index | Industry | 35% | Legal Knowledge | AA-Omniscience | Recall of statutes, doctrines, and procedure across jurisdictions |
| 25% | Agentic Knowledge Work | GDPval-AA v2 | Running matter-management workflows, drafting pipelines, and tool-augmented research | ||
| 10% | Long-Context | LCR | Reading and synthesizing across contracts, discovery productions, and case-law packets | ||
| 10% | Non-Hallucination | AA-Omniscience | Avoiding fabricated case cites or invented statutes | ||
| 5% | Agentic Customer Interaction | 𝜏³-Banking | Multi-turn, tool-using client intake and support workflows | ||
| 15% | Reasoning | HLE | Multi-step argumentation, statutory interpretation, and weighing conflicting authorities | ||
| Healthcare & Medical Index | Industry | 35% | Medical & Health Knowledge | AA-Omniscience | Clinical knowledge across diagnosis, pharmacology, and care pathways |
| 25% | Agentic Knowledge Work | GDPval-AA v2 | Tool use, planning, and orchestrating EHR/pharmacy workflows end-to-end | ||
| 15% | Non-Hallucination | AA-Omniscience | Avoiding fabricated drug interactions, doses, or guidelines | ||
| 15% | Reasoning | HLE | Multi-step clinical reasoning across biology and medicine | ||
| 10% | Agentic Customer Interaction | 𝜏³-Banking | Multi-turn, tool-using patient and member support workflows | ||
| Engineering Index | Industry | 35% | Engineering Knowledge | AA-Omniscience | Domain recall across civil, electrical, mechanical, and other engineering disciplines |
| 35% | Reasoning | HLE, GPQA Diamond, Crit-Pt | Multi-step quantitative reasoning for derivations, sizing calculations, and design trade-offs | ||
| 25% | Agentic Knowledge Work | GDPval-AA v2 | Tool use, planning, and multi-step execution of engineering deliverables | ||
| 5% | Agentic Terminal Use | Terminal-Bench v2.1 | Operating real terminal environments for builds, scripts, system administration, and debugging | ||
| Economics Index | Industry | 35% | Economics Knowledge | AA-Omniscience | Recall across micro and macroeconomics, public finance, and markets |
| 35% | Reasoning | HLE | Multi-step quantitative and analytic reasoning for modeling, estimation, and inference | ||
| 15% | Agentic Knowledge Work | GDPval-AA v2 | Tool use, planning, and multi-step execution of analytical deliverables | ||
| 15% | Long-Context | LCR | Reading and reasoning across long reports, datasets, and research notes |