Metodologia dos índices de capacidade da Artificial Analysis
Visão geral
Os índices de capacidade da Artificial Analysis medem o desempenho dos modelos em casos de uso profissionais específicos, como direito, saúde ou finanças.
Executamos de forma independente cada benchmark que compõe um índice antes de combinar suas pontuações no índice. Como todas as métricas de avaliação, os índices de capacidade têm limitações e podem não se aplicar diretamente a todos os casos de uso, mas oferecem uma síntese útil para comparar modelos no trabalho relevante para determinado domínio.
Os benchmarks subjacentes, incluindo como cada um é executado e pontuado, estão documentados na metodologia de benchmark de inteligência.
Detalhamento dos índices
Cada índice é voltado a uma única profissão ou área, como Direito, Saúde e Medicina ou Finanças e Contabilidade, e pondera um conjunto de capacidades conforme a frequência com que cada uma aparece em tarefas reais dessa área.
Nossas ponderações de tarefas se baseiam em uma taxonomia de atividades profissionais no estilo da O*NET. A tabela abaixo mostra os componentes e os pesos de cada índice.
| Índice | Peso | Capacidade | Avaliações | Descrição |
|---|---|---|---|---|
| Finance & Accounting Index | 30% | Business Knowledge | AA-Omniscience | Domain recall in accounting, corporate finance, economics, and investments |
| 30% | Agentic Knowledge Work | GDPval-AA v2, AA-Briefcase | Tool use, planning, and multi-step task execution, such as building spreadsheets, running an analysis, or coordinating a workflow | |
| 20% | Reasoning | HLE | Multi-step quantitative and analytic reasoning, used for sensitivity analysis, valuation, and structured problem solving | |
| 10% | Agentic Tool Use | AutomationBench-AA | Completing finance workflows across business apps such as spreadsheets, email, and accounting tools without breaking guardrails | |
| 5% | Long-Context | LCR, GDP.pdf | Reading and reasoning across long financial filings, deal documents, and PDF reports | |
| 5% | Non-Hallucination | AA-Omniscience | Avoiding fabricated figures or citations | |
| Strategy & Ops Index | 30% | Business Knowledge | AA-Omniscience | Working knowledge of business processes, accounting basics, and operational concepts |
| 35% | Agentic Knowledge Work | GDPval-AA v2, AA-Briefcase | Tool use, planning, and orchestrating multi-step office workflows end-to-end | |
| 30% | Agentic Tool Use | AutomationBench-AA | Completing operations, HR, marketing, sales, and support workflows across business apps without breaking guardrails | |
| 5% | Long-Context | LCR, GDP.pdf | Holding context across long threads, policies, records, and PDF documents | |
| Legal Index | 35% | Legal Knowledge | AA-Omniscience | Recall of statutes, doctrines, and procedure across jurisdictions |
| 25% | Agentic Knowledge Work | GDPval-AA v2, AA-Briefcase | Running matter-management workflows, drafting pipelines, and tool-augmented research | |
| 15% | Reasoning | HLE | Multi-step argumentation, statutory interpretation, and weighing conflicting authorities | |
| 10% | Long-Context | LCR, GDP.pdf | Reading and synthesizing across contracts, discovery productions, case-law packets, and PDF filings | |
| 10% | Non-Hallucination | AA-Omniscience | Avoiding fabricated case cites or invented statutes | |
| 5% | Agentic Tool Use | AutomationBench-AA | Completing client support and operations workflows across business apps without breaking guardrails | |
| Healthcare & Medical Index | 30% | Medical & Health Knowledge | AA-Omniscience | Clinical knowledge across diagnosis, pharmacology, and care pathways |
| 25% | Agentic Knowledge Work | GDPval-AA v2, AA-Briefcase | Tool use, planning, and orchestrating EHR/pharmacy workflows end-to-end | |
| 15% | Long-Context Reasoning | MLCR-AA | Synthesising findings across long, fragmented patient records and claims files | |
| 10% | Non-Hallucination | AA-Omniscience | Avoiding fabricated drug interactions, doses, or guidelines | |
| 10% | Reasoning | HLE | Multi-step clinical reasoning across biology and medicine | |
| 10% | Agentic Tool Use | AutomationBench-AA | Completing patient support and operations workflows across business apps without breaking guardrails | |
| Engineering Index | 35% | Engineering Knowledge | AA-Omniscience | Domain recall across civil, electrical, mechanical, and other engineering disciplines |
| 30% | Reasoning | HLE, CritPt | Multi-step quantitative reasoning for derivations, sizing calculations, and design trade-offs | |
| 20% | Agentic Knowledge Work | GDPval-AA v2, AA-Briefcase | Tool use, planning, and multi-step execution of engineering deliverables | |
| 15% | Agentic Terminal Use | Terminal-Bench v4.0 | Operating real terminal environments for builds, scripts, system administration, and debugging | |
| Economics Index | 35% | Economics Knowledge | AA-Omniscience | Recall across micro and macroeconomics, public finance, and markets |
| 35% | Reasoning | HLE | Multi-step quantitative and analytic reasoning for modeling, estimation, and inference | |
| 25% | Agentic Knowledge Work | GDPval-AA v2, AA-Briefcase | Tool use, planning, and multi-step execution of analytical deliverables | |
| 5% | Long-Context Reasoning | LCR | Reading and reasoning across long reports, datasets, and research notes |