LLM API Providers Leaderboard - Comparison of over 100 LLM endpoints

Comparison and ranking of API provider performance for over 100 AI LLM Model endpoints across performance key metrics including price, output speed, latency, context window & others. For more details including relating to our methodology, see our FAQs.

API providers compared: OpenAI, Playground AI, Mistral, Microsoft Azure, Ideogram, Amazon Bedrock, Groq, Together.ai, Anthropic, Perplexity, Google, Fireworks, Cerebras, Cohere, Lepton AI, Speechmatics, Deepinfra, Replicate, Runpod, Rev AI, fal.ai, AssemblyAI, DeepSeek, Reka AI, Deepgram, Gladia, Stability.ai, Baseten, Midjourney, Databricks, ElevenLabs, IBM, SambaNova, OctoAI, Cartesia, LMNT, 01.AI, and AI21 Labs.

	Context	Model Quality	Price	Output tokens/s	Latency
						Further Analysis
o1-preview	128k		$26.25	23.5	43.63
o1-mini	128k		$5.25	73.9	13.98
GPT-4o (Aug 6)	128k	100	$4.38	92.6	0.40
GPT-4o	128k	100	$7.50	110.0	0.36
GPT-4o	128k	100	$7.50	108.1	0.35
GPT-4o mini	128k	88	$0.26	140.1	0.39
Llama 3.1 405B	128k	100	$9.50	18.4	0.27
Llama 3.1 405B	128k	100	$7.99	13.3	1.83
Llama 3.1 405B	128k	100	$4.50	30.0	0.47
Llama 3.1 405B	128k	100	$2.80	27.7	1.00
Llama 3.1 405B	128k	100	$8.00	14.2	0.61
Llama 3.1 405B	128k	100	$3.00	72.7	0.66
Llama 3.1 405B	33k	100	$1.79	22.7	0.46
Llama 3.1 405B	8k	100	$6.25	129.3	1.46
Llama 3.1 405B	128k	100	$7.50	27.6	0.66
Llama 3.1 405B Turbo	8k	100	$5.00	88.0	0.58
Llama 3.1 70B	8k	95	$0.60	481.5	0.25
Llama 3.1 70B	128k	95	$0.99	31.8	0.72
Llama 3.1 70B	128k	95	$0.90	53.6	0.28
Llama 3.1 70B	128k	95	$0.80	55.8	0.63
Llama 3.1 70B	128k	95	$2.90	28.5	0.56
Llama 3.1 70B	128k	95	$0.90	73.1	0.41
Llama 3.1 70B	128k	95	$0.36	27.1	0.30
Llama 3.1 70B	128k	95	$0.64	249.5	0.45
Llama 3.1 70B	8k	95	$0.75	424.8	0.84
Llama 3.1 70B	128k	95	$1.50	45.4	0.58
Llama 3.1 70B	128k	95	$1.00	52.2	0.23
Llama 3.1 70B Turbo	128k	95	$0.88	84.8	0.47
Llama 3.1 8B	8k	66	$0.10	1,929.1	0.28
Llama 3.1 8B	128k	66	$0.22	92.6	0.41
Llama 3.1 8B	128k	66	$0.15	165.1	0.24
Llama 3.1 8B	128k	66	$0.07	204.3	0.43
Llama 3.1 8B	128k	66	$0.38	71.4	0.40
Llama 3.1 8B	128k	66	$0.20	288.9	0.27
Llama 3.1 8B	128k	66	$0.06	86.3	0.21
Llama 3.1 8B	128k	66	$0.06	751.2	0.38
Llama 3.1 8B	8k	66	$0.13	1,095.2	0.44
Llama 3.1 8B	128k	66	$0.20	161.4	0.17
Llama 3.1 8B Turbo	128k	66	$0.18	200.0	0.39
Gemini 1.5 Pro (Vertex)	2m	95	$5.25
Gemini 1.5 Pro (AI Studio)	2m	95	$5.25	63.8	0.89
Gemini 1.5 Flash (Vertex)	1m	84	$0.13
Gemini 1.5 Flash (AI Studio)	1m	84	$0.13	209.8	0.40
Gemma 2 27B	8k	78	$0.80	70.8	0.34
Gemma 2 9B	8k	71	$0.20	133.7	0.32
Gemma 2 9B	8k	71	$0.06	67.3	0.32
Gemma 2 9B	8k	71	$0.20	673.3	0.20
Gemma 2 9B	8k	71	$0.30	120.0	0.40
Claude 3.5 Sonnet	200k	98	$6.00	52.4	0.97
Claude 3.5 Sonnet	200k	98	$6.00	94.9	1.02
Claude 3 Opus	200k	93	$30.00	23.3	1.71
Claude 3 Opus	200k	93	$30.00	27.3	1.85
Claude 3 Haiku	200k	74	$0.50	118.1	0.44
Claude 3 Haiku	200k	74	$0.50	157.7	0.58
Mistral Large 2	128k	91	$4.50	39.4	0.51
Mistral Large 2	128k	91	$4.50	42.1	0.43
Mixtral 8x22B	65k	71	$3.00	66.3	0.47
Mixtral 8x22B	65k	71	$1.20	39.2	0.32
Mixtral 8x22B	65k	71	$1.20	50.6	0.38
Mixtral 8x22B	65k	71	$0.65	55.0	0.26
Mixtral 8x22B	65k	71	$1.20	72.1	0.39
Mistral NeMo	128k	64	$0.30	136.9	0.39
Mistral NeMo	128k	64	$0.20	158.5	0.31
Mistral NeMo	128k	64	$0.13	63.4	0.23
Mistral Small	33k	71	$1.50	49.8	0.42
Mistral Small	33k	71	$1.50	51.7	0.39
Mixtral 8x7B	33k	61	$0.70	87.0	0.41
Mixtral 8x7B	33k	61	$0.47	43.9	0.27
Mixtral 8x7B	33k	61	$0.51	60.8	0.40
Mixtral 8x7B	33k	61	$0.45	77.9	0.29
Mixtral 8x7B	33k	61	$0.50	36.9	0.76
Mixtral 8x7B	33k	61	$0.50	94.9	0.33
Mixtral 8x7B	33k	61	$0.24	46.2	0.31
Mixtral 8x7B	33k	61	$0.24	543.0	0.22
Mixtral 8x7B	33k	61	$0.63	86.6	0.50
Mixtral 8x7B	33k	61	$0.60	109.3	0.36
Codestral-Mamba	256k		$0.25	95.7	0.49
Command-R+ (08-2024)	128k		$6.00	45.7	0.56
Command-R+ (08-2024)	128k		$4.38	65.9	0.31
Command-R (08-2024)	128k		$0.75	103.2	0.39
Command-R (08-2024)	128k		$0.26	113.5	0.22
Command-R+ (04-2024)	128k	75	$6.00	45.5	0.57
Command-R+ (04-2024)	128k	75	$6.00	64.0	0.31
Command-R+ (04-2024)	128k	75	$6.00	66.2	0.52
Command-R (03-2024)	128k	63	$0.75	103.1	0.39
Command-R (03-2024)	128k	63	$0.75	152.5	0.21
Command-R (03-2024)	128k	63	$0.75	103.7	0.47
Sonar Large	33k		$1.00	41.7	0.25
Sonar Small	33k		$0.20	138.8	0.17
Sonar 3.1 Small	131k		$0.20	138.0	0.17
Sonar 3.1 Large	131k		$1.00	67.8	0.21
Phi-3 Medium 14B	128k		$0.75	49.7	0.45
Phi-3 Medium 14B	4k		$0.14	71.7	0.23
DBRX	33k	62	$1.13	85.0	0.49
DBRX	33k	62	$1.20	104.4	0.34
Reka Core	128k	90	$6.00	14.3	1.13
Reka Flash	128k	78	$1.10	29.8	0.96
Reka Edge	64k	60	$0.55	34.1	0.97
Jamba 1.5 Large	256k	86	$3.50	62.4	1.19
Jamba 1.5 Mini	256k	64	$0.25	158.4	1.03
DeepSeek-Coder-V2	128k		$0.17	17.1	1.21
DeepSeek-V2.5	128k		$0.17	16.0	1.21
DeepSeek-V2	128k	82	$0.17	17.7	1.22
Qwen2 72B	33k	83	$0.36	36.4	0.29
Qwen2 72B	33k	83	$0.90	54.6	0.36
Yi-Large	32k	81	$3.00	65.5	0.62
GPT-4 Turbo	128k	94	$15.00	31.4	0.63
GPT-4 Turbo	128k	94	$15.00	49.0	0.53
GPT-3.5 Turbo	16k	59	$0.75	82.1	0.42
GPT-3.5 Turbo	16k	59	$0.75	75.8	0.34
GPT-3.5 Turbo Instruct	4k	60	$1.63	88.9	0.40
GPT-3.5 Turbo Instruct	4k	60	$1.63	137.8	0.60
GPT-4	8k	84	$37.50	20.4	0.66
GPT-4	8k	84	$37.50	41.4	0.54
Llama 3 70B	8k	83	$1.18	47.1	0.27
Llama 3 70B	8k	83	$2.86	51.5	0.46
Llama 3 70B	8k	83	$0.90	63.1	0.33
Llama 3 70B	8k	83	$0.80	29.5	0.89
Llama 3 70B	8k	83	$2.90	18.9	0.74
Llama 3 70B	8k	83	$0.90	108.1	0.36
Llama 3 70B	8k	83	$0.36	18.4	0.32
Llama 3 70B	8k	83	$0.64	317.7	0.22
Llama 3 70B (Reference, FP16)	8k	83	$0.90	154.2	0.41
Llama 3 70B (Turbo, FP8)	8k	83	$0.88	89.5	0.49
Llama 3 8B	8k	64	$0.10	75.8	0.27
Llama 3 8B	8k	64	$0.38	78.2	0.34
Llama 3 8B	8k	64	$0.07	71.6	5.31
Llama 3 8B	8k	64	$0.38	73.0	0.41
Llama 3 8B	8k	64	$0.20	114.0	0.36
Llama 3 8B	8k	64	$0.06	111.3	0.20
Llama 3 8B	8k	64	$0.06	1,201.3	0.31
Llama 3 8B	8k	64	$0.20	300.5	0.42
Llama 2 Chat 70B	4k	57	$1.18	46.7	0.31
Llama 2 Chat 70B	4k	57	$2.10	35.6	0.47
Llama 2 Chat 70B	4k	57	$0.90	174.1	0.23
Llama 2 Chat 70B	4k	57	$1.60	17.2	0.95
Llama 2 Chat 13B	4k	39	$0.81	52.6	0.40
Llama 2 Chat 13B	4k	39	$0.20	172.8	0.23
Llama 2 Chat 13B	4k	39	$0.30	52.5	0.48
Llama 2 Chat 7B	4k	29	$0.10	122.6	0.27
Llama 2 Chat 7B	4k	29	$0.56	68.6	0.54
Gemma 7B	8k	45	$0.07	1,023.3	0.83
Gemini 1.0 Pro (AI Studio)	33k	62	$0.75	99.8	1.15
Claude 3 Sonnet	200k	80	$6.00	52.2	0.81
Claude 3 Sonnet	200k	80	$6.00	59.6	0.94
Claude 2.1	200k	55	$12.00	27.7	1.82
Claude 2.1	200k	55	$12.00	30.9	1.09
Claude 2.0	100k	70	$12.00	31.2	1.10
Claude Instant	100k	63	$1.20	61.7	0.59
Claude Instant	100k	63	$1.20	100.9	0.54
Mistral Large	33k	76	$6.00	40.4	0.51
Mistral Large	33k	76	$6.00
Mistral Large	33k	76	$6.00	22.8	0.38
Mistral 7B	33k	40	$0.25	99.4	0.41
Mistral 7B	33k	40	$0.10	38.5	0.29
Mistral 7B	33k	40	$0.16	75.8	0.36
Mistral 7B	33k	40	$0.15	168.5	0.21
Mistral 7B	33k	40	$0.07	95.6	1.11
Mistral 7B	33k	40	$0.06	107.0	0.21
Mistral 7B	16k	40	$0.20	125.6	0.19
Mistral 7B	8k	40	$0.20	127.9	0.30
Codestral	33k		$1.50	50.0	0.42
Mistral Medium	33k	70	$4.09	38.4	0.75
Command	4k		$1.63	21.8	0.56
Command	4k		$1.25	22.3	0.39
Command Light	4k		$0.38	32.8	0.60
Command Light	4k		$0.38	59.4	0.25
OpenChat 3.5	8k	50	$0.06	70.6	0.32
Jamba Instruct	256k	63	$0.55	80.7	0.91
Jamba Instruct	256k	63	$0.55	61.6	0.32

Key definitions

Back to Navigation

Artificial Analysis Quality Index: Average result across our evaluations covering different dimensions of model intelligence. Currently includes MMLU, GPQA, Math & HumanEval. OpenAI o1 model figures are preliminary and are based on figures stated by OpenAI. See methodology for more details.

Context window: Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).

Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).

Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.

Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Output price: Price per token generated by the model (received from the API), represented as USD per million Tokens.

Input price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.

Time period: Metrics are 'live' and are based on the past 14 days of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.