AA-3D Benchmarking Methodology
Scope & Background
AA-3D v1.0 measures how well AI models build interactive 3D experiences. Each model receives a written brief and builds a single web page with three.js in an offline sandbox. This page describes how we generate the scenes and how we measure their quality through human preference voting.
Every model receives the same brief, instructions, tools and limits, so the scenes are as comparable as possible. These are listed in Default Generation Settings below.
Quality Elo
Each model's Elo score reflects its relative quality, derived from blind votes cast by our recruited panel in AA-3D Voting. We compute ratings using Bradley-Terry Maximum Likelihood Estimation, with every vote weighted equally, and rescale them to an Elo-like range for readability. A 95% confidence interval from the sandwich estimator is reported alongside each rating, which narrows as a model receives more votes. Our Image, Video and Music Arenas use a different fit: power-law weighted Bradley-Terry with Hessian-based confidence intervals.
Voting
Quality Elo comes entirely from blind, pairwise preference votes cast by our recruited panel. In each comparison, a panelist sees two scenes built for the same task by two different models, interacts with both, and selects the one they prefer. Public votes cast in AA-3D Voting do not enter the rating calculation.
- Viewing requirement: a vote can only be cast after each scene has been viewed for at least 30 seconds, so preferences reflect time spent with each scene and not a first impression.
- Blind: panelists never see model names or any identifying details, even after they vote.
- Quality checks: about one in four comparisons in each session has a strong consensus answer. Sessions that fail these checks are rejected and all their votes removed, and quality-check votes are not counted in the rating.
Scene Eligibility
Before scenes enter AA-3D Voting, we check that each one runs. A scene that fails either step is excluded from voting.
- Automated check: we open each scene in a headless browser with software rendering. It gets up to 60 seconds to load and draw its first frame, and it must keep rendering, show a non-blank image, and run offline without errors.
- Review: we review each scene that passes the check before it is shown to panelists.
Leaderboards
We publish a Global Leaderboard for AA-3D v1.0.
- Global Leaderboard: aggregates every eligible vote from our recruited panel.
Tasks
AA-3D v1.0 has 55 tasks. Each task is a written brief for one interactive scene, such as a spinning black hole, a playable pinball table or the Colosseum. A brief describes what the scene must show and how users interact with it, and leaves the implementation to the model.
Tasks span seven domains: landmarks and real places; machines; natural phenomena; living bodies; games, sports and piloting; space; and stories and fantasy.
Default Generation Settings
We run every model in Stirrup, our open-source agent harness. Each task runs in its own sandbox with no network access.
| Setting | Value |
|---|---|
| Attempts per task | 1 |
| Agent turns | Up to 250 |
| Time limit | 3 hours per task |
| Compute | 4 CPU cores and 8 GB of memory, no GPU (WebGL is rendered in software) |
| Output | One HTML page with its assets, up to 200 MiB, not counting the bundled three.js and Rapier libraries |
| Tools | three.js 0.186.0, Rapier 0.20.0 (optional), Blender 4.5 (optional), Python 3, Node.js 22 and a headless Chromium browser for self-testing |
| Asset library | Optional CC0 lighting, materials, 3D models, characters and animation clips from Poly Haven, ambientCG, Kenney, Quaternius and KayKit |
Settings are held constant for every model across all scenes used in AA-3D Voting.
Model & Provider Inclusion Criteria
Our objective is to analyze and compare popular and high-performing AI models to help users choose between them. As such, we apply tests for industry significance and competitive performance to evaluate the inclusion of new models and providers. We continuously refine these criteria and welcome feedback or suggestions. To suggest models or providers, please contact us via the contact page.
Statement of Independence
Benchmarking is conducted with strict independence and objectivity. No compensation is received from any providers for listing or for favorable outcomes on Artificial Analysis.