Model & API Providers Analysis | Artificial Analysis

Independent analysis of AI language models and API providers

Understand the AI landscape and choose the best model and API provider for your use-case

AI Builders Survey

Participate & receive our report of results

Highlights

Quality

Quality Index; Higher is better

Speed

Throughput in Tokens per Second; Higher is better

Price

USD per 1M Tokens; Lower is better

Navigation

Language Models Comparison Highlights

Quality Comparison by Ability

Varied metrics by ability categorization; Higher is better

General Ability (Chatbot Arena)

Reasoning & Knowledge (MMLU)

Reasoning & Knowledge (MT Bench)

Coding (HumanEval)

OpenAI's GPT-4 is no longer the clear quality leader with the launch of other models including Anthropic's Opus and Mistral's Large. Models have also been released which rival GPT-3.5 performance including Gemini Pro, Mixtral 8x7B and DBRX.

Median across providers: Figures represent median (P50) across all providers which support the model.

Quality vs. Throughput

Quality: General reasoning index, Throughput: Tokens per Second, Price: USD per 1M Tokens

Most attractive quadrant

Size represents Price (USD per M Tokens)

There is a trade-off between model quality and throughput, with higher quality models typically having lower throughput.

Quality: Index represents normalized average relative performance across Chatbot arena, MMLU & MT-Bench.

Throughput: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API).

Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Median across providers: Figures represent median (P50) across all providers which support the model.

Quality vs. Price

Quality: General reasoning index, Price: USD per 1M Tokens

Most attractive quadrant

Higher quality models are typically more expensive. However, model quality varies significantly and some open source models now achieve very high quality.

Quality: Index represents normalized average relative performance across Chatbot arena, MMLU & MT-Bench.

Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Median across providers: Figures represent median (P50) across all providers which support the model.

Throughput

Output Tokens per Second; Higher is better

Throughput: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API).

Median across providers: Figures represent median (P50) across all providers which support the model.

Pricing: Input and Output Prices

USD per 1M Tokens

Input price

Output price

Prices vary considerably, including between input and output token price. GPT-4 stands out as orders of magnitude higher priced than the cheapest models.

Input price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.

Output price: Price per token generated by the model (received from the API), represented as USD per million Tokens.

Median across providers: Figures represent median (P50) across all providers which support the model.

API Provider Highlights: Mixtral 8x7B Instruct

Charts below show providers for Mixtral 8x7B Instruct

Throughput vs. Price: Mixtral 8x7B Instruct

Throughput: Tokens per Second, Price: USD per 1M Tokens

Most attractive quadrant

Mistral

Amazon Bedrock

Groq

Together.ai

Perplexity

Fireworks

Lepton AI

Deepinfra

Replicate

OctoAI

Smaller, emerging providers are offering high throughput and at competitive prices.

Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Throughput: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API).

Median: Figures represent median (P50) measurement over the past 14 days.

Variance data is present on the model and API provider pages amongst the detailed performance metrics. See 'Compare Models' and 'Compare API Providers' in the navigation menu for further analysis.

Pricing (Input and Output Prices): Mixtral 8x7B Instruct

Price: USD per 1M Tokens; Lower is better

Input price

Output price

Providers typically charge different prices for input and output tokens. The ratio of input / output token price for a certain use-case may significantly impact overall costs.

Input price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.

Output price: Price per token generated by the model (received from the API), represented as USD per million Tokens.

Throughput, Over Time: Mixtral 8x7B Instruct

Output Tokens per Second; Higher is better

Mistral

Amazon Bedrock

Groq

Together.ai

Perplexity

Fireworks

Lepton AI

Deepinfra

Replicate

OctoAI

Smaller, emerging providers offer high throughput, though precise speeds delivered vary day-to-day.

Throughput: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API).

Over time measurement: Median measurement per day, based on 8 measurements each day at different times. Labels represent start of week's measurements.

See more information on any of our supported models

Model Name	Creator	License	Context Window

GPT-4	OpenAI	Proprietary	8k
GPT-4 Turbo	OpenAI	Proprietary	128k
GPT-4 Turbo (Vision)	OpenAI	Proprietary	128k
GPT-3.5 Turbo	OpenAI	Proprietary	16k
GPT-3.5 Turbo Instruct	OpenAI	Proprietary	4k

Llama 3 Instruct (70B)	Meta	Open	8k
Llama 2 Chat (13B)	Meta	Open	4k
Llama 2 Chat (70B)	Meta	Open	4k
Llama 3 Instruct (8B)	Meta	Open	8k
Llama 2 Chat (7B)	Meta	Open	4k
Code Llama Instruct (70B)	Meta	Open	16k

Mistral Large	Mistral	Proprietary	33k
Mistral Medium	Mistral	Proprietary	33k
Mixtral 8x22B Instruct	Mistral	Open	65k
Mixtral 8x7B Instruct	Mistral	Open	33k
Mistral Small	Mistral	Proprietary	33k
Mistral 7B Instruct	Mistral	Open	33k

Gemini 1.5 Pro	Google	Proprietary	1000k
Gemini 1.0 Pro	Google	Proprietary	33k
Gemma 7B Instruct	Google	Open	8k

Claude 3 Opus	Anthropic	Proprietary	200k
Claude 3 Sonnet	Anthropic	Proprietary	200k
Claude 3 Haiku	Anthropic	Proprietary	200k
Claude 2.1	Anthropic	Proprietary	200k
Claude 2.0	Anthropic	Proprietary	100k
Claude Instant	Anthropic	Proprietary	100k

Command-R+	Cohere	Open	128k
Command-R	Cohere	Open	128k
Command	Cohere	Proprietary	4k
Command Light	Cohere	Proprietary	4k

DBRX Instruct	Databricks	Open	33k

OpenChat 3.5 (1210)	OpenChat	Open	8k

PPLX-70B Online	Perplexity	Proprietary	4k
PPLX-7B-Online	Perplexity	Proprietary	4k