We are launching the Artificial Analysis Endpoint Accuracy Index v1.1, a minor version upgrade alongside new coverage for GLM-5.3 Flash and Muse Spark Open. Across the 103 serverless endpoints we benchmarked, 24 score statistically below our self-hosted reference.

Providers trade off accuracy to optimize for speed and cost. They quantize weights, write custom kernels, tune their inference stacks and cap output lengths, and sometimes they ship bugs. The Endpoint Accuracy Index measures how much of a model's accuracy each endpoint preserves, so developers can pick providers on accuracy as well as price and speed. v1.1 updates AA-LCR-25 to tighten confidence intervals and reduce grader error, while BFCL-500 and HLE-250 are unchanged.

What's changes in v1.1

• AA-LCR-25 moves to dataset version 1.1. The new version corrects the answer keys for two questions and is scored with an updated equality-checker LLM. Raw AA-LCR-25 scores under v1.1 are not directly comparable with v1.0 scores.
• AA-LCR-25 runs 25 repeats per question, up from 10. Across the 48 endpoints and references we scored under both versions, the average AA-LCR-25 95% confidence interval narrows from ±4.7 to ±2.6 points.
• Coverage expands to six open weights models. We add GLM-5.3 Flash and Muse Spark Open, and DeepSeek V4.1 Flash 1017 replaces DeepSeek V4 Pro. They join GLM-5.3, Kimi K3 and gpt-oss-120b. GLM-5.2 leaves coverage.

Narrower AA-LCR-25 intervals matter because they were the widest of the three evaluations in v1.0 (±4.6 points on average, against ±2.9 for BFCL-500 and ±1.6 for HLE-250). In v1.0, noise on long context recall could mask real differences on tool calling and reasoning. With 25 repeats, the reference-inclusive confidence interval on the composite index narrows from ±5.5 to ±4.3 points on average across the 45 endpoints scored under both versions, and the composite moves by 1.6 points on average.

The suite in v1.1:

Evaluation	What it tests	Tasks	Repeats	Weight
BFCL-500	Tool calling	500	3	1/3
HLE-250	Scientific reasoning	250	10	1/3
AA-LCR-25	Long context recall (~100k input tokens per question)	25	25 (was 10)	1/3

How the index is calculated

We run the same three evaluations against each endpoint and against the reference deployment, score each evaluation as a percentage of the reference result, and average the three with equal weight. Each result carries a 95% confidence interval built from per-repeat variation, combined with the reference's own uncertainty. An endpoint is statistically in line with the reference when that interval includes 100, and below the reference when the whole interval sits under 100.

We self-host each reference on the lab's official weights at the recommended precision, following the lab's and serving framework's recipes, and publish the full serving setup with each result. The v1.1 references run on vLLM or SGLang on NVIDIA B200, B300 or GB200 systems. We run the highest reasoning mode each model supports and each endpoint's highest supported output length and context window.

Results are point-in-time. The v1.1 results in this article were measured between 8 October and 4 November 2026, and each endpoint's result carries its own date.

Results across six models

Model	Endpoints	In line with reference	Below reference	Index range
DeepSeek V4.1 Flash 1017	22	22	0	97.5 to 101.4
GLM-5.3	26	16	10	67.2 to 99.3
GLM-5.3 Flash	22	17	5	78.6 to 104.2
gpt-oss-120b	16	11	5	78.2 to 104.0
Kimi K3	12	10	2	91.0 to 101.5
Muse Spark Open	5	3	2	93.4 to 99.9
Total	103	79	24

The causes we found behind the below-reference results fall into four groups:

• Output token caps. Endpoints that cap output at 16K to 32K tokens cut off long HLE-250 reasoning before the model answers.
• Less reasoning. Some endpoints produce markedly fewer reasoning tokens than the reference at the same requested settings, with no truncation, and answer fewer HLE-250 questions correctly.
• Tool call handling. Endpoints parse and format tool calls differently. Returning arguments with the wrong type, as plain text, or with corrupted names fails BFCL-500 tasks the reference passes.
• Request limits and stream behavior. Fixed request timeouts, and streams that end before the model finishes, remove the longest reasoning answers.

One endpoint scores below the reference on HLE-250 and AA-LCR-25 without output truncation or reduced reasoning token counts that we could observe.

Each endpoint with a known cause carries a note on its provider page describing what we observed.

GLM-5.3 Flash

GLM-5.3 Flash enters coverage with 22 endpoints. 17 are statistically in line with the reference, including one that scores 104.2, and 5 score below it.

• Output token caps limit the model on complex workloads. Two endpoints cap output at about 32K tokens, and about half of their HLE-250 responses reach the cap before an answer. They score 22.3% and 28.5% on HLE-250, against 55.4% for the reference.
• Endpoints that reason less give lower quality answers. Two endpoints produce about 30% fewer reasoning tokens per HLE-250 response than the reference, with no truncation, and score 48.4% and 48.6% against 55.4%.
• Tool argument types affect BFCL-500. Two endpoints return object-typed tool arguments as JSON-encoded text instead of objects, which fails those calls.

The reference runs the native FP8 weights. All three endpoints serving FP4 weights are statistically in line with it, scoring 97.2 to 104.2.

Full results: GLM-5.3 Flash - Endpoint Accuracy Index

Muse Spark Open

Muse Spark Open enters coverage with 5 endpoints, measured against a BF16 reference.

• Three of five endpoints are in line with the reference. They score 99.5 to 99.9, including one serving FP4 weights.
• The other two lose accuracy only on tool calling. Both return tool arguments that the tool defines as text, such as booking IDs, as numbers. This fails many multi-turn tasks, and they score 49.9% and 50.7% on BFCL-500 against 59.1% for the reference. On HLE-250 and AA-LCR-25 they are within 2 points of the reference.

Full results: Muse Spark Open - Endpoint Accuracy Index

GLM-5.3

GLM-5.3 has 26 endpoints in coverage. Scores range from 67.2 to 99.3, with 16 statistically in line with the reference and 10 below it.

• Output caps and reasoning loops explain the three scores below 80. A 16,384-token output cap brings HLE-250 to 7.6% and a 32,000-token cap to 24.8%, against 63.6% for the reference. On a third endpoint, most HLE-250 responses fall into repetitive reasoning loops and reach the output limit before a final answer, for 19.6%.
• Tool call handling separates otherwise similar endpoints. Three endpoints score 50.7% to 51.9% on BFCL-500 against 57.3% for the reference. One forces arguments into the tool schema's order and drops some, one corrupts parameter names, and one returns object arguments as text.
• FP4 serving holds up. Five of the six endpoints serving FP4 weights are statistically in line with the FP8 reference, scoring 97.7 to 99.3. The sixth is the endpoint with reasoning loops described above.

Full results: GLM-5.3 - Endpoint Accuracy Index

DeepSeek V4.1 Flash 1017

All 22 DeepSeek V4.1 Flash 1017 endpoints are statistically in line with the reference, scoring 97.5 to 101.4, a range of 3.9 points. On HLE-250 every endpoint scores between 68.3% and 71.7%, against 71.4% for the reference.

Full results: DeepSeek V4.1 Flash 1017 - Endpoint Accuracy Index

Kimi K3

Kimi K3 has 12 endpoints in coverage, and 10 are statistically in line with the reference, scoring 97.1 to 101.5.

• Long reasoning needs long requests. Where requests are cut off after about 10 minutes, about 9% of HLE-250 requests end before the model answers, and HLE-250 falls to 65.3% against 72.1% for the reference.
• Tool call IDs matter in multi-turn conversations. One endpoint reuses tool call IDs across turns of a conversation, which conflicts with the OpenAI tool calling format. It scores 42.1% on BFCL-500 against 55.2% for the reference, while its HLE-250 and AA-LCR-25 scores are within 2 points of the reference.

Full results: Kimi K3 - Endpoint Accuracy Index

gpt-oss-120b

gpt-oss-120b, re-measured on 16 endpoints, has 11 statistically in line with the reference and 5 below it. Its confidence intervals are wider than the other models' (about ±7.5 points on average). The index expresses each score relative to the reference, and gpt-oss-120b's reference scores are lower (17.0% on HLE-250, 37.1% on BFCL-500), so the same run-to-run variation is a larger share of the score.

• Tool call handling separates endpoints. Two endpoints score 21.3% and 22.3% on BFCL-500 against 37.1% for the reference. One returns function calls as plain text rather than structured tool calls, and the other fills optional arguments with placeholder values such as 0 or -1.
• Output caps cut off reasoning. With output capped at 16,384 tokens, HLE-250 drops to 11.0% against 17.0% for the reference.

Full results: gpt-oss-120b - Endpoint Accuracy Index

Comparing v1.0 and v1.1 results

Under v1.0 we published results for GLM-5.3, gpt-oss-120b and Kimi K3 on the same endpoints. For the 45 endpoints scored under both versions, the composite moves by 1.6 points on average, and 10 endpoints changed classification: 6 moved from in line to below the reference, and 4 moved from below to in line.

The methodology page lists each index version in its version history. We will move to a new minor or major version when the suite changes, and results from different versions are not directly comparable.

Explore the results

• Full results: GLM-5.3 Flash, Muse Spark Open, GLM-5.3, DeepSeek V4.1 Flash 1017, Kimi K3 and gpt-oss-120b
• Methodology and version history

This article shows data as at 9 November 2026. For current results, see each model's providers page.