Artificial Analysis Search Index: Search API Benchmarking Methodology
Overview
Search API Bench evaluates various search providers in an agentic search paradigm.
The benchmark is a provider-swap comparison. Every sample uses the same candidate answer model and benchmark task, and varies only the search provider.
We evaluate results using a single answer model which measures the uplift provided by each Search API provider.
Throughout this page, one result is a Search API Provider paired with the fixed candidate answer model. We call this a provider result.
Key Metrics
Artificial Analysis Search Index
The public leaderboard headlines a single blended score, the Artificial Analysis Search Index. It is the equal-weighted mean of each benchmark's primary quality metric:
Notes on the calculation:
- AA-Omniscience contributes accuracy, not its Omniscience Index or hallucination rate. Accuracy is the most directly comparable signal in the Search Provider and candidate model paradigm.
- All inputs use the same constants (discussed below), so the index isolates the search provider.
Index Components
| Evaluation | Field | Tasks | Response Type | Scoring |
|---|---|---|---|---|
| DeepSearchQA | Deep-research QA (single- and set-answer) | 900‡ | Open answer / answer set | LLM-graded F1 over answer items, pass@1 |
| AA-Omniscience | Factual QA (correctness + calibration) | 600† | Open answer (abstention allowed) | LLM-graded accuracy, pass@1 |
| BrowseComp | Hard web-search QA | 200^ | Short exact answer | LLM-graded exact-answer accuracy, pass@1 |
All three benchmarks use GPT-5.6 Luna (medium) for grading with their benchmark-specific rubrics.
‡ DeepSearchQA: the full 900-row public eval split.
† AA-Omniscience: 600 private held-out samples, balanced 100 per domain across 6 domains.
^ BrowseComp: a hard 200-sample subset drawn from the 1,266-sample evaluation pool.
Constants
Every value below is held fixed across all provider results, so the search provider is the only variable in a comparison.
- Candidate answer model: GPT-5.6 Luna (medium)
- Reasoning effort: Medium
- Temperature: 0.6
- Maximum output tokens: 127,999 tokens
- Grader model: GPT-5.6 Luna (medium), shared across all three benchmarks
- Turn budget: 25 agent turns (t25)
- Tool calls per sample: Unlimited within the turn budget
- Search results shown: Up to 10 results per search
- Search tools:
web_searchandweb_fetchweb_search— Sends the candidate model's query to the target Search API provider and returns their native response payload.web_fetch— Fetches one URL from a search result and returns page content (text only).
- Extraction layer: Text-only extractor with a 15-second timeout per page
- Search payload format: Provider-native; the model sees the provider's original response body
- Contamination filtering: Enabled, as described below
- Agent workflow: Either:
model_onlyno-search baseline, or- Stirrup search-agent loop
Workflow
Model-Only Baseline
The model_only sample calls the candidate model directly with the benchmark prompt in one-shot fashion with no search tools. It estimates how much the candidate model can solve from internal knowledge or reasoning alone.
A strong model_only score indicates a benchmark sample is answerable without fresh search information, while a lower model_only score makes search uplift easier to observe.
Search-Agent Loop
Search samples use a fixed agent loop orchestrated by the Artificial Analysis Stirrup harness.
The harness provides two tools: web_search and web_fetch.
A given Search Provider API powers the web_search tool, and the candidate model is given a total of 25 turns with unlimited tool calls to solve the given task.
When the model believes it has gathered enough information, it calls a finish tool to submit its final answer. If the model uses all 25 turns without calling finish, no answer is submitted and the task scores zero.
A benchmark-specific grader scores the answer against hidden reference data.
Error Handling
Recoverable errors return to the model as a failed tool call, and remain in published results. Recoverable errors include:
- Timeouts or HTTP errors on fetched web pages
- The model fetches a URL that
web_searchdid not return
Fatal errors are retried until success, and no results are published with fatal errors. Fatal errors include:
- Search API provider errors (e.g. 429, 5xx and timeouts)
Cost
Costs are reported as total experiment cost and split into:
- Candidate answer model cost, including all input, cached, reasoning and output tokens.
- Search API provider cost.
- Note: The
model_onlybaseline has no Search API provider cost.
- Note: The
Latency
Latency is reported in several ways:
- Model time per task. Average derived end-to-end time of candidate model input/thinking/output tokens per task.
- Search time per task. Average measured time spent in
web_searchper task. - Time per task. Sum of model time per task and search time per task.
- Time per Search Query. Average latency per individual
web_searchrequest across the whole benchmark.
A provider can be fast per call and still contribute more total time, if the model searches more often against it.
Contamination Filtering
Contamination filtering removes potential and known benchmark leaks and sources.
Filtering is applied inside both search tools before the candidate model sees any output: web_search results are screened on URL, title, and snippet, and web_fetch page text is screened after extraction. Flagged entries are dropped from the provider's result list without changing the shape of the response.
For example:
- Known source locations. Original benchmark hosted URLs and datasets.
- Dataset text co-signals. Known canary strings in the source material.
Search Provider API Settings
Each provider uses a fixed max_results=10 (or the provider equivalent) setting. All settings are kept at provider defaults except for those listed below.
| Provider result | Documentation | Benchmark request settings |
|---|---|---|
| Brave | Brave Web Search API | q=agent query, result_filter=web |
| Tavily basic | Tavily Search API | search_depth=basic |
| You.com | You.com Search API | safesearch=moderate |
| Firecrawl | Firecrawl Search API | scrape_format=none |
| Exa fast | Exa Search API | type=fast, contents={"highlights": True} |
| Exa auto | Exa Search API | type=auto, contents={"highlights": True} |
| Parallel turbo | Parallel Search API | mode=turbo |
| Parallel basic | Parallel Search API | mode=basic |
| Parallel advanced | Parallel Search API | mode=advanced |
| Keenable pro | Keenable Search API | mode=pro |
| Keenable realtime | Keenable Search API | mode=realtime |
Benchmarks Scope
DeepSearchQA
A deep-research-style QA benchmark with single-answer and set-answer rows.
- Source: Google DeepSearchQA. Hugging Face. Paper.
- Sample scope: full 900-row public
evalsplit. - Main metric: F1.
BrowseComp
A hard web-search QA benchmark.
- Source: OpenAI BrowseComp, distributed with Simple Evals. Paper.
- Sample scope: the 200-sample
n200decision panel, selected as described in Index Components. - Main metric: Accuracy.
AA-Omniscience
A private, held-out factual QA benchmark focused on correctness and calibration.
- Source: AA-Omniscience. Public reference dataset: AA-Omniscience-Public. Paper.
- Sample scope: 600 private samples, balanced 100 rows per domain across 6 domains.
- Main metric: Accuracy.
We use accuracy rather than Omniscience Index or Hallucination Rate. AA-Omniscience was designed to test parametric knowledge and the ability to abstain without sufficient context.
Given a search tool supplying context, a model tends to always answer, so accuracy provides insights to the question: given search context from a provider, how accurately does the model answer?
Interpreting Results
Search API Bench is a multi-objective comparison:
- Did search improve the answer over
model_only? - Which provider improved quality the most?
- How much did that improvement cost?
- How much latency did it add?
- Are gains robust across benchmarks, or specific to one task family?
The strongest provider for a given task may not be the highest-scoring one. A winning search provider for a specific use case should be evaluated on:
- Quality uplift from baseline.
- Cost increase from baseline.
- Latency increase from baseline.