Methodik des Artificial Analysis Progress Index v1

Artificial Analysis Progress Index v1

The Artificial Analysis Progress Index v1 estimates the general capability of language models on a single scale that is consistent over time. It is designed for analyzing trends in AI progress, including comparisons between models released years apart.

The Artificial Analysis Intelligence Index is designed to measure current frontier capability. Its composition is updated periodically: evaluations are added as models improve, and evaluations that no longer differentiate leading models are retired. Each version of the Intelligence Index is therefore measured on a different set of evaluations, and scores across versions are not intended for long-run comparison.

The Progress Index v1 addresses this by estimating model capability and task difficulty jointly, using item response theory. Models are linked through the evaluation tasks they have in common, so each model's estimate reflects its performance relative to the difficulty of the tasks it attempted, rather than the particular set of evaluations it was tested on. This allows models to be compared on one scale without requiring results on the same evaluations.

The Intelligence Index remains the primary measure of current model intelligence and is the recommended basis for comparing models for present-day use cases. The Progress Index v1 is intended for analysis of capability over time. We fit each effort variant as a separate model and represent each model by its highest-scoring variant.

Modell und Schätzung

The Progress Index v1 is based on item response theory (IRT), a statistical framework that places the capability of each model and the difficulty of each benchmark task on one shared scale.

Models don't need results on all the same benchmarks. As in other applications of IRT, we estimate each task's difficulty from every model that attempted it, and each model's capability from the tasks it attempted. Benchmarks that share models link everything onto one scale, so models tested on older and newer benchmarks can be compared directly.

• Each model has one capability . Each item (one task within a benchmark, or a whole benchmark where only aggregate scores exist) has a difficulty , the capability at which a model is expected to score 50% after the guessing floor is removed, and a discrimination , how sharply the expected score rises as capability passes the difficulty
• A model's expected score on an item follows a logistic curve:

• We calibrate every item's parameters and estimate every model's capability together, in one fit over all results. The fit minimizes a weighted log loss, where is the model's observed score on the item from 0 to 1, after the input data adjustments, plus two penalties:

• The weight is 1 divided by the number of tasks the model has results on in that benchmark (SciCode weights each problem by its number of steps), so each benchmark counts equally for each model
• The capability penalty, with and the number of fitted parameters, keeps scores finite for models that score 0% or 100% on every item
• The item priors pull each discrimination toward 1 and each difficulty toward , its benchmark's average. and set how strong the priors are, chosen by empirical testing. Each prior is divided by , the number of items in its benchmark
• We scale capabilities and difficulties so that o3-mini (high) scores 220 and GPT-5.5 (xhigh) scores 320. We chose these anchors because both are stable frontier models with a clear performance gap across our evaluations. A fixed gap changes a model's odds of success on a given task by the same factor anywhere on the scale
• Prior work: item response theory has been used to evaluate machine learning models (Martínez-Plumed et al., 2019; Rodriguez et al., 2021; Polo et al., 2024), and related methods place language models on one capability scale (Ruan et al., 2024; Ho et al., 2025, the basis of Epoch AI's Epoch Capabilities Index; Anthropic's Claude Mythos Preview system card). The Progress Index v1 fits individual tasks with log loss, using only evaluations we run ourselves

Eingabedaten

The Progress Index v1 uses each model's available results on public Artificial Analysis benchmarks.

• We use task-level data where it is available across a sufficient range of models, and aggregate benchmark scores otherwise (e.g., legacy evaluations). Where a task was run more than once, we average its repeats. Each benchmark carries equal weight for a model, regardless of how many tasks it has
• A benchmark must have results from at least 4 models, and a model must have results on at least 4 benchmarks to be included in the fit
• Input scores are normalized to range from 0 to 1. Where a model could score by guessing, we rescale scores such that guessing counts as 0, so a 25% score on GPQA Diamond (four options per question) counts as 0
• We impute zero scores where a model's context window is too short for an evaluation: models with a context window under 100,000 tokens score 0 on AA-LCR

Versionsverlauf