LLM API Providers Leaderboard - Comparison of over 100 LLM endpoints

Comparison and ranking of API provider performance for over 100 AI LLM Model endpoints across performance key metrics including price, output speed, latency, context window & others. For more details including relating to our methodology, see our FAQs.

API providers compared: OpenAI, Playground AI, Mistral, Ideogram, Microsoft Azure, Amazon Bedrock, Hyperbolic, Groq, Together.ai, FriendliAI, Black Forest Labs, Anthropic, Perplexity, Google, Fireworks, Cerebras, Cohere, Recraft AI, Upstage, Simplismart, Speechmatics, Deepinfra, Replicate, , Genmo, Nebius, Adobe, Runpod, Rev AI, fal.ai, AssemblyAI, DeepSeek, Reka AI, Deepgram, Gladia, Baseten, Stability.ai, Midjourney, Databricks, ElevenLabs, IBM, SambaNova, xAI, Cartesia, LMNT, 01.AI, and AI21 Labs.

	Features		Price	Output tokens/s	Latency
						Further Analysis
o1-preview	128k	85	$26.25	142.6	21.40
o1-mini	128k	82	$5.25	212.8	11.11
GPT-4o (Aug '24)	128k	78	$4.38	74.0	0.43
GPT-4o (Aug '24)	128k	78	$4.38	67.6	0.99
GPT-4o (May '24)	128k	78	$7.50	75.7	0.43
GPT-4o (May '24)	128k	78	$7.50	118.1	0.74
GPT-4o mini	128k	73	$0.26	74.2	0.40
GPT-4o mini	128k	73	$0.26	171.0	0.77
GPT-4o (Nov '24)	128k	73	$4.38	118.9	0.38
GPT-4o (Nov '24)	128k	73	$4.38	43.8	1.07
Llama 3.3 70B	33k	74	$0.94	2,172.7	0.32
Llama 3.3 70B	128k	74	$0.40	26.2	0.53
Llama 3.3 70B	128k	74	$0.71	31.1	0.95
Llama 3.3 70B Fast	128k	74	$0.38	71.8	0.59
Llama 3.3 70B Base	128k	74	$0.20	48.0	0.63
Llama 3.3 70B	128k	74	$0.71	26.2	0.46
Llama 3.3 70B	128k	74	$0.90	105.4	0.56
Llama 3.3 70B (Turbo, FP8)	128k	74	$0.20	31.2	0.29
Llama 3.3 70B	128k	74	$0.27	31.1	0.33
Llama 3.3 70B (Spec decoding)	8k	74	$0.69	1,831.4	0.35
Llama 3.3 70B	128k	74	$0.64	275.4	0.24
Llama 3.3 70B	4k	74	$0.75	390.3	0.47
Llama 3.3 70B	128k	75	$0.88	163.5	0.50
Llama 3.1 405B	128k	74	$9.50	18.9	0.40
Llama 3.1 405B	128k	75	$4.00	11.6	0.79
Llama 3.1 405B Standard	128k	74	$2.40	30.5	1.96
Llama 3.1 405B Latency Optimized	128k	74	$3.00	65.0	0.81
Llama 3.1 405B Base	128k	74	$1.50	34.0	0.72
Llama 3.1 405B Vertex	128k	74	$7.75	29.7	0.42
Llama 3.1 405B	128k	74	$8.00	20.6	0.54
Llama 3.1 405B	128k	73	$3.00	73.8	0.70
Llama 3.1 405B	33k	73	$0.90	21.7	0.42
Llama 3.1 405B	8k	74	$6.25	168.5	0.77
Llama 3.1 405B	128k	72	$7.50	30.1	0.70
Llama 3.1 405B Turbo	128k	74	$3.50	73.7	0.81
Llama 3.1 70B	33k	68	$0.60	2,292.1	0.29
Llama 3.1 70B	128k	69	$0.40	28.7	0.68
Llama 3.1 70B Standard	128k	68	$0.72	31.5	0.71
Llama 3.1 70B Latency Optimized	128k	68	$0.90	133.2	0.42
Llama 3.1 70B Base	128k	66	$0.20	46.0	0.64
Llama 3.1 70B Fast	128k	66	$0.38	72.7	0.58
Llama 3.1 70B Vertex	128k	68	$0.00	71.7	0.28
Llama 3.1 70B	128k	68	$2.90	32.1	0.56
Llama 3.1 70B	128k	67	$0.90	130.3	0.42
Llama 3.1 70B (Turbo, FP8)	128k	66	$0.20	40.7	0.27
Llama 3.1 70B	128k	66	$0.27	38.1	0.29
Llama 3.1 70B	128k	68	$0.60	218.0	0.44
Llama 3.1 70B	128k	54	$0.64	275.3	0.28
Llama 3.1 70B (Spec decoding)	8k	54	$0.69	1,853.5	0.35
Llama 3.1 70B	64k	65	$0.75	398.4	0.46
Llama 3.1 70B	128k	43	$1.50	62.0	0.56
Llama 3.1 70B	128k	68	$1.00	52.7	0.33
Llama 3.1 70B Turbo	128k	68	$0.88	231.1	0.35
Llama 3.1 70B	128k	62	$0.90	125.6	0.35
Llama 3.2 90B (Vision)	128k	67	$0.72	35.7	0.51
Llama 3.2 90B (Vision) Vertex	128k	68	$0.00	34.1	0.20
Llama 3.2 90B (Vision)	128k	66	$0.90	66.7	0.32
Llama 3.2 90B (Vision)	33k	68	$0.36	37.8	0.28
Llama 3.2 90B (Vision)	8k	67	$0.90	267.9	0.38
Llama 3.2 90B (Vision) Turbo	128k	66	$1.20	55.9	0.30
Llama 3.2 11B (Vision)	128k	53	$0.16	132.7	0.36
Llama 3.2 11B (Vision)	128k	54	$0.20	121.5	0.27
Llama 3.2 11B (Vision)	128k	54	$0.06	52.8	0.21
Llama 3.2 11B (Vision)	8k	53	$0.18	750.1	0.28
Llama 3.2 11B (Vision) Turbo	128k	54	$0.18	160.6	0.25
Llama 3.1 8B	33k	54	$0.10	2,183.0	0.29
Llama 3.1 8B	128k	53	$0.10	121.9	0.47
Llama 3.1 8B	128k	54	$0.22	90.6	0.39
Llama 3.1 8B Fast	128k	54	$0.04	185.2	0.49
Llama 3.1 8B Base	128k	54	$0.03	52.1	0.60
Llama 3.1 8B Vertex	128k	54	$0.00	119.7	0.18
Llama 3.1 8B	128k	54	$0.38	163.3	0.31
Llama 3.1 8B	128k	53	$0.20	198.1	0.27
Llama 3.1 8B	128k	54	$0.04	55.9	0.22
Llama 3.1 8B	128k	54	$0.10	516.7	0.42
Llama 3.1 8B	128k	53	$0.06	750.0	0.28
Llama 3.1 8B	16k	52	$0.13	1,001.0	0.30
Llama 3.1 8B	128k	54	$0.20	155.9	0.33
Llama 3.1 8B Turbo	128k	53	$0.18	305.7	0.26
Llama 3.1 8B	128k	52	$0.15	460.7	0.29
Llama 3.2 3B	128k	48	$0.10	199.0	0.44
Llama 3.2 3B	128k	49	$0.15	143.6	0.41
Llama 3.2 3B Base	128k	49	$0.01	121.2	0.51
Llama 3.2 3B	128k	50	$0.10	271.1	0.29
Llama 3.2 3B	128k	49	$0.02	161.4	0.17
Llama 3.2 3B	8k	49	$0.06	1,616.0	0.37
Llama 3.2 3B	4k	49	$0.10	1,347.4	0.23
Llama 3.2 3B Turbo	128k	49	$0.06	45.3	0.65
Llama 3.2 1B	128k	26	$0.10	313.6	0.35
Llama 3.2 1B Base	128k	26	$0.01	255.7	0.51
Llama 3.2 1B	128k	26	$0.01	183.5	0.23
Llama 3.2 1B	8k	26	$0.04	3,355.8	0.50
Llama 3.2 1B	4k	26	$0.05	2,047.5	0.27
Gemini 2.0 Flash (exp) (AI Studio)	1m	82	$0.00	169.0	0.47
Gemini 1.5 Pro (Sep) (Vertex)	2m	80	$2.19	58.2	0.40
Gemini 1.5 Pro (Sep) (AI Studio)	2m	80	$2.19	63.8	0.77
Gemini 1.5 Flash (Sep) (Vertex)	1m	74	$0.13	189.9	0.21
Gemini 1.5 Flash (Sep) (AI Studio)	1m	74	$0.13	182.1	0.41
Gemma 2 27B	8k	61	$0.80	59.9	0.42
Gemma 2 9B Fast	8k	55	$0.04	183.5	0.50
Gemma 2 9B Base	8k	55	$0.03	168.9	0.51
Gemma 2 9B	8k	54	$0.04	56.3	0.33
Gemma 2 9B	8k	55	$0.20	650.5	0.25
Gemma 2 9B	8k	55	$0.30	125.1	0.29
Gemini 1.5 Flash-8B AI Studio	1m	47	$0.07	279.4	0.37
Gemini Experimental (Nov) (AI Studio)	2m		$0.00	54.5	1.25
Gemini 1.5 Flash (May) (Vertex)	1m		$0.13	302.2	0.29
Gemini 1.5 Flash (May) (AI Studio)	1m		$0.13	313.0	0.30
Gemini 1.5 Pro (May) (Vertex)	2m	72	$2.19	65.7	0.42
Gemini 1.5 Pro (May) (AI Studio)	2m	72	$2.19	67.1	0.79
Claude 3.5 Sonnet (Oct)	200k	80	$6.00	42.0	1.12
Claude 3.5 Sonnet (Oct) Vertex	200k	80	$6.00	72.7	0.74
Claude 3.5 Sonnet (Oct)	200k	80	$6.00	85.4	1.33
Claude 3.5 Sonnet (June)	200k	76	$6.00	45.5	1.03
Claude 3.5 Sonnet (June) Vertex	200k	76	$6.00	61.3	0.73
Claude 3.5 Sonnet (June)	200k	76	$6.00	86.4	0.86
Claude 3 Opus	200k	70	$30.00	23.6	1.51
Claude 3 Opus Vertex	200k	70	$30.00	27.7	3.15
Claude 3 Opus	200k	70	$30.00	27.5	2.04
Claude 3.5 Haiku Standard	200k	68	$1.60	56.3	0.75
Claude 3.5 Haiku Latency Optimized	200k	68	$2.00	101.0	0.59
Claude 3.5 Haiku Vertex	200k	68	$1.60	65.1	0.99
Claude 3.5 Haiku	200k	68	$1.60	64.9	0.76
Claude 3 Haiku	200k	55	$0.50	110.5	0.79
Claude 3 Haiku	200k	55	$0.50	135.8	0.43
Pixtral Large	128k	74	$3.00	36.0	0.51
Mistral Large 2 (Jul '24)	128k	74	$3.00	30.4	0.43
Mistral Large 2 (Jul '24)	128k	74	$3.00	34.4	0.47
Mistral Large 2 (Jul '24)	128k	74	$3.00	29.4	0.55
Mistral Large 2 (Nov '24)	128k	74	$3.00	44.1	0.41
Mistral Large 2 (Nov '24)	128k	74	$3.00	36.4	0.54
Mistral Small (Sep '24)	33k	61	$0.30	63.4	0.37
Mixtral 8x22B	65k	62	$3.00	81.9	0.32
Mixtral 8x22B Base	65k	60	$0.60	87.7	0.60
Mixtral 8x22B Fast	65k	60	$1.05	101.2	0.61
Mixtral 8x22B	65k	61	$1.20	74.6	0.34
Mixtral 8x22B	65k	55	$1.20	65.1	0.42
Pixtral 12B	128k	56	$0.15	66.5	0.34
Pixtral 12B	128k	57	$0.10	75.0	0.46
Ministral 8B	128k	56	$0.10	137.2	0.31
Mistral NeMo	128k	53	$0.15	120.6	0.31
Mistral NeMo Fast	128k	53	$0.12	159.2	0.51
Mistral NeMo Base	128k	53	$0.06	54.3	0.59
Mistral NeMo	128k	54	$0.06	73.0	0.23
Ministral 3B	128k	53	$0.04	170.4	0.29
Mixtral 8x7B	33k	42	$0.70	99.7	0.31
Mixtral 8x7B	33k	41	$0.51	71.0	0.35
Mixtral 8x7B Fast	33k	41	$0.23	162.0	0.52
Mixtral 8x7B Base	33k	41	$0.12	135.5	0.50
Mixtral 8x7B	33k	43	$0.50	141.8	0.29
Mixtral 8x7B	33k	40	$0.24	83.8	0.22
Mixtral 8x7B	33k	42	$0.24	552.7	0.28
Mixtral 8x7B	33k	42	$0.63	90.9	0.40
Mixtral 8x7B	33k	35	$0.60	96.2	0.27
Codestral-Mamba	256k	33	$0.25	95.0	0.44
Command-R+	128k	55	$6.00	47.6	0.52
Command-R+	128k	55	$4.38	74.8	0.24
Command-R+ (Apr '24)	128k	45	$6.00	47.3	0.51
Command-R+ (Apr '24)	128k	47	$6.00	73.1	0.27
Command-R+ (Apr '24)	128k	44	$6.00	48.8	0.59
Command-R (Mar '24)	128k	36	$0.75	108.7	0.36
Command-R (Mar '24)	128k	37	$0.75	173.8	0.17
Command-R (Mar '24)	128k	36	$0.75	78.2	0.46
Aya Expanse 8B	8k		$0.75	166.0	0.15
Command-R	128k		$0.75	108.4	0.36
Command-R	128k	51	$0.26	117.4	0.17
Aya Expanse 32B	128k		$0.75	121.0	0.17
Sonar 3.1 Small	127k		$0.20	203.6	0.31
Sonar 3.1 Large	127k		$1.00	56.5	0.31
Grok Beta	128k	72	$7.50	66.9	0.38
Nova Pro	300k	75	$1.40	89.7	0.37
Nova Lite	300k	70	$0.10	145.7	0.33
Nova Micro	130k	66	$0.06	194.8	0.32
Phi-4	16k	77	$0.09	82.4	0.22
Phi-3 Medium 14B	128k		$0.30	49.5	0.43
DBRX	33k	50	$1.13	68.1	0.50
DBRX	33k	44	$1.20	82.9	0.31
Llama 3.1 Nemotron 70B Base	128k	72	$0.20	48.2	0.61
Llama 3.1 Nemotron 70B Fast	128k	72	$0.38	69.9	0.60
Llama 3.1 Nemotron 70B	128k	72	$0.27	30.3	0.30
Jamba 1.5 Large	256k	64	$3.50	52.0	0.56
Jamba 1.5 Large	256k	64	$3.50	50.8	0.71
Jamba 1.5 Mini	256k	46	$0.25	181.7	0.35
Jamba 1.5 Mini	256k		$0.25	82.7	0.49
DeepSeek V3	66k	79	$0.48	57.3	1.07
DeepSeek V3 (FP8)	128k	79	$0.25	13.1	1.04
DeepSeek V3	128k	79	$0.90	22.3	1.04
DeepSeek V3	32k	79	$1.25	10.5	0.60
DeepSeek V3 (FP8)	128k	79	$1.25	21.4	0.76
DeepSeek-V2.5 (Dec '24)	64k	72	$0.17	57.8	1.12
DeepSeek-Coder-V2	128k	71	$0.17	57.3	1.05
DeepSeek-V2.5	128k		$2.00	7.8	0.77
Qwen2.5 72B	131k	77	$0.40	33.7	0.60
Qwen2.5 72B	131k	77	$0.20	45.8	0.61
Qwen2.5 72B Fast	131k	77	$0.38	68.3	0.55
Qwen2.5 72B	131k	77	$0.90	79.5	0.35
Qwen2.5 72B	33k	78	$0.27	36.9	0.29
Qwen2.5 72B	8k	77	$2.50	230.3	0.55
Qwen2.5 72B	131k	77	$1.20	86.9	0.42
Qwen2.5 Coder 32B	131k	72	$0.20	37.5	0.46
Qwen2.5 Coder 32B	33k	72	$0.90	95.1	0.35
Qwen2.5 Coder 32B	33k	71	$0.10	48.7	0.24
Qwen2.5 Coder 32B	8k	72	$1.88	315.0	0.35
Qwen2.5 Coder 32B	131k	72	$0.80	83.2	0.50
Qwen2 72B	33k	69	$0.90	64.1	0.34
QwQ 32B-Preview	33k	46	$0.20	35.7	0.50
QwQ 32B-Preview	33k	46	$0.90	105.1	0.36
QwQ 32B-Preview	33k	46	$0.26	62.7	0.26
QwQ 32B-Preview	8k	46	$0.13	301.8	0.55
QwQ 32B-Preview	33k	46	$1.20	58.7	0.52
Yi-Large	32k	61	$3.00	69.0	0.45
GPT-4 Turbo	128k	75	$15.00	35.1	0.68
GPT-4 Turbo	128k	75	$15.00	47.9	1.48
GPT-4	8k		$37.50	26.9	0.76
Llama 3 70B	8k	47	$1.18	46.3	0.36
Llama 3 70B	8k	62	$0.40	32.8	0.61
Llama 3 70B	8k	47	$2.86	47.1	0.45
Llama 3 70B	8k	46	$0.90	125.9	0.34
Llama 3 70B	8k	48	$0.27	44.5	0.27
Llama 3 70B	8k	48	$0.64	349.2	0.26
Llama 3 70B (Reference, FP16)	8k	48	$0.90	164.1	0.37
Llama 3 70B (Turbo, FP8)	8k	48	$0.88	23.4	0.37
Llama 3 8B	8k	44	$0.10	63.3	0.38
Llama 3 8B	8k	45	$0.38	102.4	0.33
Llama 3 8B	8k	45	$0.38	73.7	0.39
Llama 3 8B	8k	45	$0.20	192.5	0.29
Llama 3 8B	8k	45	$0.04	111.4	0.20
Llama 3 8B	8k	45	$0.06	1,201.0	0.34
Llama 3 8B	8k	46	$0.20	228.1	0.33
Llama 2 Chat 7B	4k		$0.10	123.6	0.37
Gemini 1.0 Pro (AI Studio)	33k		$0.75	102.8	1.25
Claude 3 Sonnet	200k	57	$6.00	52.6	0.78
Claude 3 Sonnet	200k	57	$6.00	84.5	0.79
Claude 2.1	200k		$12.00	29.1	1.80
Claude 2.1	200k		$12.00	13.3	0.82
Claude 2.0	100k		$12.00	29.8	0.81
Mistral Small (Feb '24)	33k	59	$1.50	65.9	0.31
Mistral Small (Feb '24)	33k	59	$1.50	52.6	0.39
Mistral Large (Feb '24)	33k	57	$6.00	37.8	0.39
Mistral Large (Feb '24)	33k	56	$6.00	43.6	0.41
Mistral Large (Feb '24)	33k	55	$6.00	35.4	0.52
Mistral 7B	8k	24	$0.25	131.2	0.30
Mistral 7B	8k	28	$0.16	92.6	0.33
Mistral 7B	8k	28	$0.04	102.9	0.20
Mistral 7B	8k	28	$0.20	157.2	0.23
Mistral Medium	33k		$4.09	43.3	0.38
Codestral (May '24)	33k		$0.30	84.2	0.30
OpenChat 3.5	8k	44	$0.06	72.1	0.29
Jamba Instruct	256k		$0.55	183.9	0.34
Jamba Instruct	256k	28	$0.55	75.8	0.53

Key definitions

Back to Navigation

Artificial Analysis Quality Index: Average result across our evaluations covering different dimensions of model intelligence. Currently includes MMLU, GPQA, Math & HumanEval. OpenAI o1 model figures are preliminary and are based on figures stated by OpenAI. See methodology for more details.

Context window: Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).

Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).

Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.

Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Output Price: Price per token generated by the model (received from the API), represented as USD per million Tokens.

Input Price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.

Time period: Metrics are 'live' and are based on the past 14 days of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.