Artificial Analysis provides a free API to support analysis of AI models and in making informed decisions about which AI models to use.
We offer a free API focused on model benchmarks and a commercial API with more comprehensive data. Commercial API documentation is available to partners separately.
To access our free API, create an account for the Artificial Analysis Insights Platform and generate an API key.
When integrating with our API, we recommend using model and creator IDs as primary identifiers since they remain stable, while slugs and names may change over time.
Attribution is required for all use of our free API. Please provide attribution to https://artificialanalysis.ai/.
If you wish to include our logo in your materials, you can download our brand kit here: Artificial Analysis Brand Kit.
Use of the API is subject to our Terms of Use and Data Platform Terms.
For details on how benchmarks are conducted, see our methodology.
Include your API key in the x-api-key header.
To obtain an API key, create an account in the Artificial Analysis Insights Platform and generate an API key.
The free Data API is documented in the Data API docs.
This API provides the official gateway for evaluating code generation submissions against the CritPt v2 benchmark private evaluation set using the CritPt v2 grader. The endpoint and request and response schemas are unchanged.
x-api-key headerWhen using this API or the broader CritPt evaluation, please provide appropriate attribution to the CritPt project.
Use of this endpoint is subject to our Terms of Use and Data Platform Terms.
Submit a batch of code generation submissions for evaluation against CritPt v2 benchmark problems.
Note: Requests may take substantial time to complete as they are processed by the evaluation system.
| Field | Type | Required | Description |
|---|---|---|---|
submissions | array | Yes | Array of submission objects (see below) - must include *all* problems in the public set to be processed |
batch_metadata | object | Yes | Metadata object for the batch (any additional properties allowed) |
| Field | Type | Required | Description |
|---|---|---|---|
problem_id | string | Yes | Unique identifier for the CritPt problem |
generated_code | string | Yes | The code generated by the model |
model | string | Yes | Name/identifier of the model used |
generation_config | object | Yes | Configuration used for code generation (any properties allowed) |
messages | array | No | Optional array of message objects |
| Field | Type | Description |
|---|---|---|
accuracy | number | Average accuracy from 0-1 across submissions |
timeout_rate | number | Share of submission grading processes that timed out in the upstream grading server |
server_timeout_count | number | Number of submission grading processes that timed out in the upstream grading server |
judge_error_count | number | Number of submission grading processes that failed due to a judge error in the grading server; this could be due to model responses or intermittent and may be retried |
All successful responses include the following headers:
X-RateLimit-Limit: Maximum requests allowed in the time windowX-RateLimit-Remaining: Number of requests remaining in current windowX-RateLimit-Reset: Human-readable date string when the rate limit window resets{
"error": "Rate limit exceeded",
"limit": 10,
"remaining": 0,
"reset": "Wed Jan 01 2025 00:00:00 GMT+0000 (Coordinated Universal Time)",
"retryAfter": 3600
}The response also includes Retry-After header with seconds to wait.
curl -X POST https://artificialanalysis.ai/api/v2/critpt/evaluate \
-H "Content-Type: application/json" \
-H "x-api-key: your_api_key_here" \
-d '{
"submissions": [
{
"problem_id": "Challenge_1_main",
"generated_code": "```python\ndef solution():\n return 42\n```",
"model": "gpt-5",
"generation_config": {
"use_golden_for_prev_steps": false,
"parsing": false,
"multiturn_with_answer": false,
"use_python": false,
"use_web_search": false
}
},
# Submission for *all* problems in the public set are required in each grading request
...
],
"batch_metadata": {}
}'{
"accuracy": 0.0,
"timeout_rate": 0.0,
"server_timeout_count": 0,
"judge_error_count": 0
}