Endpoint Accuracy Index v1.0 Methodology

Endpoint Accuracy Index v1.0

The Endpoint Accuracy Index measures how the intelligence and capability of a specific model varies across a range of API providers who serve it. The same model served by different providers can behave differently depending on quantization, sampling defaults, context handling, token limits, prompt parsers and other endpoint-side configuration. To capture this, we re-run a fixed evaluation suite against each provider endpoint and report how closely the endpoint reproduces the model accuracy.

We also use a self-hosted model deployment as a baseline, expressing each provider endpoint's overall accuracy score as a percentage of the baseline deployment. The index is designed so that 100% means an endpoint matches the reference, and lower scores indicate accuracy is lower compared to the baseline.

We show the statistical significance on each result. Each endpoint result carries a 95% confidence interval computed from per-repeat variation, and we flag whether an endpoint is statistically within the reference or significantly below it. We also report a note for each endpoint that exhibited certain characteristics or shortcomings that affected the result.

Results are point-in-time snapshots rather than live monitoring. Each result is dated and reports the number of repeats used, so it can be interpreted in context.

Evaluation suite

Each endpoint is evaluated on a fixed suite of three evaluations spanning native tool use, hard reasoning and long-context recall. Repeat counts are chosen per evaluation to keep confidence intervals tight, and the number of repeats behind each result is surfaced alongside it.

Evaluation datasets

Scoring

Each evaluation yields a mean accuracy across its repeats. The composite Endpoint Accuracy Index is the weighted blend of the three evaluation scores, normalised to a 0 to 100 scale:

Each evaluation mean accuracy (0 to 1) is multiplied by its weight and divided by the total weight. The three evaluations are equally weighted (33% / 33% / 33%). The composite is only computed when all three evaluations have a current-version result for the endpoint.

Where a self-hosted reference endpoint exists, scores are expressed as a percentage of that reference, both per evaluation and at the composite level, using the same weighted blend over the per-evaluation ratios:

Each endpoint's score is shown as a percentage of the reference, with a 95% confidence interval that combines its own uncertainty with the reference's. We build that interval from per-repeat variation using a two-sided 95% Student-t test, combine the three evals in quadrature, and compare the result to the reference (100%):

• Within range: the confidence interval includes the reference — not statistically distinguishable from it.
• Significantly outside range: the confidence interval sits entirely outside the reference — a statistically significant difference.

The exact score and confidence interval for each endpoint are shown on the chart.

Raw 0 to 1 scores are retained in the data layer and only multiplied by 100 for display. For each endpoint we use the latest run per evaluation; where multiple runs exist we prefer the most recent, and within the same date the run with the fewest errored tasks.

Inference parameters

Endpoints are evaluated under standardised settings so that differences reflect the endpoint, not the harness:

• Sampling: standardised sampling defaults are applied consistently across endpoints: the temperature recommended by the model lab where one is specified, otherwise 0.6 for reasoning models, consistent with our approach for the Artificial Analysis Intelligence Index.
• Reasoning: the highest reasoning mode the endpoint supports is used.
• Tool calling: the API defaults are used, with tool choice set to auto and strict schema adherence enabled. Where an endpoint does not support one of these, we fall back to the closest supported setting for that endpoint.
• Streaming: our baseline is streaming enabled. Endpoints that do not support streaming are run with streaming disabled.
• Output and context: the maximum output tokens and context window supported by the endpoint are used.

Reference runs and reproducibility

The reference run for each model is a self-hosted deployment of the model, typically using vLLM or SGLang. We configure the server using the settings recommended by the model lab and by the serving framework (e.g. using vLLM recipes or the SGLang cookbook), served at the highest recommended precision settings for model inference. Key configuration details (such as the serving framework & CUDA versions, model weight information and launch arguments) are recorded and surfaced alongside the result, viewable in the notes element of the reference run result and below.

The evaluation suite is pinned to fixed versions so results are comparable over time and across endpoints.

Reference inference setup:
- GPU: 8x NVIDIA B200
- SGLang version: 0.5.12 (Docker image: lmsysorg/sglang:deepseek-v4-blackwell)
- CUDA: 13.0
- Precision: native — W4A8 MoE experts (MegaMoE) + FP8 (e4m3) attention/dense
- Model weights: deepseek-ai/DeepSeek-V4-Pro on Hugging Face

Key SGLang launch args:
- moe-a2a-backend: deepep
- kv-cache-dtype: fp8_e4m3
- reasoning-parser: deepseek-v4
- tool-call-parser: deepseekv4

Reference inference setup:
- GPU: 8x NVIDIA B200
- SGLang version: 0.5.13 (Docker image: lmsysorg/sglang:latest)
- CUDA: 13.0
- Precision: native FP8 with DeepGEMM kernels
- Model weights: zai-org/GLM-5.2-FP8 on Hugging Face

Key SGLang launch args:
- moe-a2a-backend: deepep
- kv-cache-dtype: fp8_e4m3
- reasoning-parser: glm45
- tool-call-parser: glm47

Reference inference setup:
- GPU: 8x NVIDIA B200
- SGLang version: 0.5.13.post1 (Docker image: lmsysorg/sglang:v0.5.13.post1)
- CUDA: 13.0
- Precision: native MXFP4 (FlashInfer MXFP4 MoE kernels) + BF16 attention/dense
- Model weights: openai/gpt-oss-120b on Hugging Face

Key SGLang launch args:
- kv-cache-dtype: bfloat16
- reasoning-parser: gpt-oss
- tool-call-parser: gpt-oss

Point-in-time results

Endpoint accuracy results are dated snapshots, not live monitoring. Endpoints can change over time as providers update their serving stack and endpoint configuration, so each result carries the date it was measured. Re-runs happen on a defined cadence and on provider request; we always show the most recent result and its date.

Model selection

We prioritise models for endpoint-accuracy benchmarking where there is a wide ecosystem of providers serving the same model (most commonly open-weights models served by many inference providers) and where the model is widely used in production. The set of included models is a rotating mix that changes over time as new models and providers are added, so it is not fixed.

The models with current Endpoint Accuracy Index coverage are:

• DeepSeek V4 Pro
• GLM-5.2
• gpt-oss-120b