Announcing the Artificial Analysis Search Index: Same Agent, Different Search

The Artificial Analysis Search Index is the equal-weighted mean of three benchmark scores: DeepSearchQA, BrowseComp, and AA-Omniscience. An ideal provider result scores high on the Artificial Analysis Search Index while keeping latency and cost low.

As a baseline, we run the same model, GPT-5.6 Luna (medium), on the same tasks in single-shot fashion without search tools. The baseline results can be used to show how much each provider lifts the same model above its internal knowledge.

At launch, the leaderboard covers 11 provider results across 7 search providers, and we will keep expanding coverage.

Each provider result pairs one Search API provider with a fixed candidate model, GPT-5.6 Luna (medium). The model runs inside Stirrup, our open-source agent harness, with two tools: web_search and web_fetch. The web_search tool sends the model's query to the provider and returns the provider's native response payload (with content modes standardized to snippets), with max_results=10 and known contamination sources filtered out.

The model has up to 25 turns for each task. When the model decides it has enough information, it calls finish and submits its final answer. Finally, a benchmark-specific grader scores the answer. If the model uses all 25 turns without calling finish, no answer is submitted and the task scores zero.

Stirrup agent

Benchmark tasks

Candidate model

GPT-5.6 Luna (medium)

turn 0 / 25

 

Search provider

 

web_search  

native payload, up to 10 results

Grading

Grader

avg F1 0.79

 

Search API providers

Constants

Same tasks, same model and grader (GPT-5.6 Luna, medium), same Stirrup harness (web_search · web_fetch · finish, max 25 turns). Only the search provider varies.

Restart0.5×1×2×

The methodology page documents the full workflow, metrics, constants, and grading.

The Artificial Analysis Search Index combines the following three benchmarks:

Total cost per task sums search API cost and candidate model cost. Search API cost covers the provider's search calls at their listing price and candidate model cost represents all model tokens used for the task.

If a provider returns focused search results, this can reduce the model's token use, which reduces overall model cost.

For example, Parallel Search (advanced) has a higher search cost per task than Parallel Search (basic): $0.048 vs. $0.045. However, its total cost per task is lower: $0.084 vs. $0.11.

The advanced tier returns higher quality results, so the model uses ~169,000 tokens per task instead of ~339,000, and the model cost falls from $0.07 to $0.036. The trade is time: 35.9s per task vs. 20.4s.

Firecrawl Search shows a different path to low cost. It scores 73 on the Artificial Analysis Search Index at $0.075 total cost per task, helped by credit-based pricing (~$0.03 in search cost per task) and high quality results. However, it has the slowest time per task on the board (55.0s).

Exa Search (fast) is faster per search query than Exa Search (auto): 0.91s vs. 2.10s. However, it costs more per task: $0.16 vs. $0.13. The fast tier runs more searches per task (11.2 vs. 9.37) and the model uses more tokens (~392,000 vs. ~299,000), so both search cost and model cost rise.

Time per task sums measured search time and model time. In an ideal setup, added search latency remains low. However, a provider that is fast per search call can still add more total time. This occurs when the model must run more searches or produce more tokens to reach an answer.

Keenable Search (realtime) has the fastest average search calls on the board: 0.34s per query. It also has the lowest total time per task (15.1s), and scores 67 on the Artificial Analysis Search Index.

Among Parallel's tiers, turbo has the fastest average search calls: 0.51s per query vs. 1.03s for Parallel Search (basic). But the basic tier still scores higher on quality: 73 vs. 67. The turbo tier also runs more searches per task (13.6 vs. 9.03), so the two land close on total time per task: 18.8s vs. 20.4s.

The model only baseline takes ~13.8s per task with zero search time. The DeepSearchQA and BrowseComp tasks explain this result.

These tasks suit a model plus search combination: search results give the model context for its answer.

For example, on BrowseComp tasks without search tools, the model only baseline writes ~4,100 output tokens per task, compared with ~1,600 to ~2,900 for the provider results.

AA-Omniscience shows the opposite pattern. With search tools, the model reads and reconciles sources, and then writes roughly two to three times as many output tokens as the baseline.