Artificial Analysis LLM Performance Leaderboard

Independent performance benchmarks & pricing across API providers of LLMs. Definitions are below the table.
For further analysis and methodology, see artificialanalysis.ai.

	Features	Model Intelligence	Price	Output tokens/s	Latency
						Further Analysis
o3-mini	200k	63	$1.93	152.7	15.92
o3-mini	200k	63	$1.93	24.2	125.02
o1	200k	62	$26.25	4.7	565.59
DeepSeek R1	64k	60	$0.96	25.5	60.47
DeepSeek R1	128k	60	$2.00	49.2	38.85
DeepSeek R1 Base	128k	60	$1.20	6.1	276.14
DeepSeek R1 Fast	128k	60	$3.00	76.2	19.01
DeepSeek R1	128k	60	$3.99	70.2	0.68
DeepSeek R1	4k	60	$0.00	19.3	113.86
DeepSeek R1	128k	60	$4.25	90.8	16.41
DeepSeek R1	64k	60	$1.16	8.3	181.48
DeepSeek R1	64k	60	$4.00	14.0	84.31
DeepSeek R1	128k	60	$7.00	95.2	18.45
o1-mini	128k	54	$1.93	186.7	11.24
o1-mini	128k	54	$5.78	201.1	13.97
DeepSeek R1 Distill Qwen 14B	64k	50	$0.15	45.8	16.73
DeepSeek R1 Distill Qwen 14B	128k	50	$1.60	144.7	6.99
Gemini 2.0 Pro Experimental (AI Studio)	2m	49	$0.00	122.6	0.56
Gemini 2.0 Flash Vertex	1m	48	$0.26	0.0	0.11
Gemini 2.0 Flash (AI Studio)	1m	48	$0.17	153.0	0.28
DeepSeek V3	66k	46	$0.48	26.8	4.04
DeepSeek V3 (FP8)	128k	46	$0.25	34.5	2.95
DeepSeek V3	128k	46	$0.75	23.2	0.92
DeepSeek V3	128k	46	$1.31	33.7	0.96
DeepSeek V3	64k	46	$0.59	7.5	1.03
DeepSeek V3	64k	46	$0.89	12.5	0.89
DeepSeek V3 (FP8)	128k	46	$1.25	9.1	1.34
DeepSeek R1 Distill Llama 70B	66k	45	$0.94	1,754.2	0.74
DeepSeek R1 Distill Llama 70B Base	128k	45	$0.38	61.6	15.19
DeepSeek R1 Distill Llama 70B	128k	45	$0.34	24.4	26.63
DeepSeek R1 Distill Llama 70B	32k	45	$0.39	22.0	17.97
DeepSeek R1 Distill Llama 70B	128k	45	$0.81	255.4	3.45
DeepSeek R1 Distill Llama 70B (Spec decoding)	128k	45	$0.81	1,627.2	1.28
DeepSeek R1 Distill Llama 70B	16k	45	$0.88	123.8	14.58
DeepSeek R1 Distill Llama 70B	128k	45	$2.00	95.0	19.01
Qwen2.5 Max	32k	45	$2.80	36.1	1.24
DeepSeek R1 Distill Qwen 32B	128k	45	$0.14	44.1	13.76
DeepSeek R1 Distill Qwen 32B	64k	45	$0.30	20.4	30.99
DeepSeek R1 Distill Qwen 32B	128k	45	$0.69	382.0	2.49
Gemini 1.5 Pro (Sep) (Vertex)	2m	45	$2.19	0.0	0.11
Claude 3.5 Sonnet (Oct)	200k	44	$6.00	53.3	0.92
Claude 3.5 Sonnet (Oct) Vertex	200k	44	$6.00	86.9	0.90
Claude 3.5 Sonnet (Oct)	200k	44	$6.00	74.5	1.49
QwQ 32B-Preview	33k	43	$0.20	33.7	0.62
QwQ 32B-Preview	33k	43	$0.14	80.3	0.61
QwQ 32B-Preview	33k	43	$0.90	120.9	0.55
QwQ 32B-Preview	33k	43	$0.26	58.2	0.33
QwQ 32B-Preview	33k	43	$1.20	65.8	0.64
Gemini 2.0 Flash-Lite (Preview) (AI Studio)	1m	42	$0.13	249.2	0.24
GPT-4o (Nov '24)	128k	41	$4.38	59.9	0.46
GPT-4o (Nov '24)	128k	41	$4.38	107.4	1.17
Llama 3.3 70B	33k	41	$0.94	2,288.4	0.19
Llama 3.3 70B	128k	41	$0.40	25.5	0.53
Llama 3.3 70B	128k	41	$0.71	123.4	0.64
Llama 3.3 70B Fast	128k	41	$0.38	72.5	0.60
Llama 3.3 70B Base	128k	41	$0.20	20.1	1.13
Llama 3.3 70B	128k	41	$0.50	128.4	0.52
Llama 3.3 70B	128k	41	$0.90	94.8	0.64
Llama 3.3 70B (Turbo, FP8)	128k	41	$0.20	31.7	0.29
Llama 3.3 70B	128k	41	$0.27	24.7	0.31
Llama 3.3 70B	128k	41	$0.60	204.1	0.33
Llama 3.3 70B	128k	41	$0.39	50.6	1.04
Llama 3.3 70B (Spec decoding)	8k	41	$0.69	1,513.7	0.41
Llama 3.3 70B	128k	41	$0.64	275.1	0.39
Llama 3.3 70B	4k	41	$0.75	455.2	0.32
Llama 3.3 70B Turbo	128k	41	$0.88	195.0	0.64
GPT-4o (ChatGPT)	128k	41	$7.50	104.8	0.45
GPT-4o (Aug '24)	128k	41	$4.38	65.0	0.41
GPT-4o (Aug '24)	128k	41	$4.38	95.3	0.75
GPT-4o (May '24)	128k	41	$7.50	66.8	0.41
Llama 3.1 405B	128k	40	$9.50	18.9	0.44
Llama 3.1 405B	128k	40	$4.00	8.4	0.82
Llama 3.1 405B Standard	128k	40	$2.40	29.9	1.94
Llama 3.1 405B Latency Optimized	128k	40	$3.00	64.6	0.77
Llama 3.1 405B Base	128k	40	$1.50	33.7	0.74
Llama 3.1 405B Vertex	128k	40	$7.75	0.0	0.08
Llama 3.1 405B	128k	40	$3.00	62.5	0.87
Llama 3.1 405B	33k	40	$0.90	15.0	0.54
Llama 3.1 405B	8k	40	$6.25	170.5	0.58
Llama 3.1 405B	128k	40	$7.50	36.6	0.78
Llama 3.1 405B Turbo	128k	40	$3.50	26.3	0.74
Qwen2.5 72B	131k	40	$0.40	44.6	0.59
Qwen2.5 72B	131k	40	$0.20	35.3	0.67
Qwen2.5 72B Fast	131k	40	$0.38	67.9	0.59
Qwen2.5 72B	131k	40	$0.90	90.3	0.55
Qwen2.5 72B	33k	40	$0.27	45.8	0.29
Qwen2.5 72B	8k	40	$2.50	263.1	0.35
Qwen2.5 72B Turbo	131k	40	$1.20	50.5	0.97
Qwen2.5 72B	131k	40	$0.00	18.5	1.19
Phi-4	16k	40	$0.15	120.2	0.52
Phi-4	16k	40	$0.09	37.2	0.25
Tulu3 405B	16k	40	$6.25	182.7	0.64
MiniMax-Text-01	1m	40	$0.42	41.0	0.89
Mistral Large 2 (Nov '24)	128k	38	$3.00	45.7	0.35
Mistral Large 2 (Nov '24)	128k	38	$3.00	33.4	0.51
Grok Beta	128k	38	$7.50	65.3	0.29
Pixtral Large	128k	37	$3.00	35.4	0.39
Qwen2.5 Instruct 32B Fast	128k	37	$0.00	83.0	0.58
Qwen2.5 Instruct 32B Base	128k	37	$0.00	61.0	0.59
Qwen2.5 Instruct 32B	128k	37	$0.79	391.5	0.35
Llama 3.1 Nemotron 70B Base	128k	37	$0.20	47.2	0.63
Llama 3.1 Nemotron 70B Fast	128k	37	$0.38	70.9	0.60
Llama 3.1 Nemotron 70B	128k	37	$0.27	30.9	0.33
Nova Pro	300k	37	$1.40	87.4	0.41
Mistral Large 2 (Jul '24)	128k	37	$3.00	30.8	0.39
Mistral Large 2 (Jul '24)	128k	37	$3.00	31.9	0.47
Mistral Large 2 (Jul '24)	128k	37	$3.00	34.7	0.55
Qwen2.5 Coder 32B	131k	36	$0.20	36.8	0.55
Qwen2.5 Coder 32B	33k	36	$0.90	110.7	0.52
Qwen2.5 Coder 32B	33k	36	$0.10	48.3	0.26
Qwen2.5 Coder 32B	131k	36	$0.79	384.4	0.36
Qwen2.5 Coder 32B	8k	36	$1.88	380.8	0.28
Qwen2.5 Coder 32B	131k	36	$0.80	81.3	0.64
GPT-4o mini	128k	36	$0.26	74.5	0.43
GPT-4o mini	128k	36	$0.26	168.4	0.81
DeepSeek R1 Distill Llama 8B	32k	35	$0.04	52.6	8.94
Llama 3.1 70B	128k	35	$0.40	26.6	0.60
Llama 3.1 70B Standard	128k	35	$0.72	31.4	0.70
Llama 3.1 70B Latency Optimized	128k	35	$0.90	140.7	0.37
Llama 3.1 70B Base	128k	35	$0.20	40.6	0.66
Llama 3.1 70B Fast	128k	35	$0.38	73.0	0.60
Llama 3.1 70B Vertex	128k	35	$0.00	0.0	0.09
Llama 3.1 70B	128k	35	$2.90	54.9	0.55
Llama 3.1 70B	128k	35	$0.90	140.5	0.55
Llama 3.1 70B (Turbo, FP8)	128k	35	$0.20	40.9	0.26
Llama 3.1 70B	128k	35	$0.27	36.2	0.28
Llama 3.1 70B	128k	35	$0.60	214.2	0.34
Llama 3.1 70B	32k	35	$0.35	80.9	1.00
Llama 3.1 70B	64k	35	$0.75	132.1	0.52
Llama 3.1 70B	128k	35	$1.50	59.9	0.42
Llama 3.1 70B Turbo	128k	35	$0.88	194.8	0.49
Llama 3.1 70B	128k	35	$0.90	126.8	0.57
Mistral Small 3	32k	35	$0.15	119.4	0.30
Mistral Small 3	32k	35	$0.90	91.7	0.56
Mistral Small 3	32k	35	$0.09	83.9	0.23
Mistral Small 3	32k	35	$0.80	96.9	0.25
Claude 3 Opus	200k	35	$30.00	25.1	1.32
Claude 3 Opus Vertex	200k	35	$30.00	25.4	2.33
Claude 3 Opus	200k	35	$30.00	27.8	1.03
Claude 3.5 Haiku Standard	200k	35	$1.60	51.3	0.88
Claude 3.5 Haiku Latency Optimized	200k	35	$2.00	95.8	0.56
Claude 3.5 Haiku Vertex	200k	35	$1.60	64.9	0.77
Claude 3.5 Haiku	200k	35	$1.60	65.2	0.62
Gemini 1.5 Pro (May) (Vertex)	2m	34	$2.19	0.0	0.11
Qwen Turbo	1m	34	$0.09	75.0	1.11
Llama 3.2 90B (Vision)	128k	33	$0.72	53.3	0.39
Llama 3.2 90B (Vision) Vertex	128k	33	$0.00	0.0	0.08
Llama 3.2 90B (Vision)	128k	33	$0.90	78.7	0.50
Llama 3.2 90B (Vision)	33k	33	$0.36	35.3	0.27
Llama 3.2 90B (Vision)	8k	33	$0.90	262.8	0.32
Llama 3.2 90B (Vision) Turbo	128k	33	$1.20	55.0	0.34
Mistral Saba	32k	32	$0.30	96.6	0.31
Jamba 1.5 Large	256k	29	$3.50	47.0	0.50
Jamba 1.5 Large	256k	29	$3.50	51.0	0.79
Gemini 1.5 Flash (May) (Vertex)	1m	28	$0.13	0.0	0.11
Nova Micro	130k	28	$0.06	194.6	0.37
Yi-Large	32k	28	$3.00	69.3	0.43
Codestral (Jan '25)	256k	28	$0.45	208.1	0.28
Codestral (Jan '25) Vertex	128k	28	$0.45	150.8	0.15
Llama 3 70B	8k	27	$1.18	47.4	0.40
Llama 3 70B	8k	27	$0.40	32.4	1.41
Llama 3 70B	8k	27	$2.86	52.9	0.43
Llama 3 70B	8k	27	$2.90	19.0	0.83
Llama 3 70B	8k	27	$0.90	154.2	0.46
Llama 3 70B	8k	27	$0.27	38.8	0.31
Llama 3 70B	8k	27	$0.57	24.8	1.54
Llama 3 70B	8k	27	$0.64	344.5	0.23
Llama 3 70B (Reference, FP16)	8k	27	$0.90	137.3	0.41
Llama 3 70B (Turbo, FP8)	8k	27	$0.88	16.5	0.31
Mistral Small (Sep '24)	33k	27	$0.30	98.3	0.30
Mistral Large (Feb '24)	33k	26	$6.00	36.3	0.39
Mistral Large (Feb '24)	33k	26	$6.00	44.0	0.38
Mistral Large (Feb '24)	33k	26	$6.00	39.8	0.51
Mixtral 8x22B	65k	26	$3.00	68.9	0.29
Mixtral 8x22B Base	65k	26	$0.60	88.2	0.62
Mixtral 8x22B Fast	65k	26	$1.05	103.4	0.64
Mixtral 8x22B	65k	26	$1.20	89.5	0.52
Mixtral 8x22B	65k	26	$1.20	72.2	0.82
Phi-3 Medium 14B	128k	25	$0.30	52.5	0.43
Mistral Medium	33k	24	$4.09	43.7	0.36
Llama 3.1 8B	33k	24	$0.10	2,230.1	0.26
Llama 3.1 8B	128k	24	$0.10	114.6	0.50
Llama 3.1 8B	128k	24	$0.22	89.2	0.37
Llama 3.1 8B Fast	128k	24	$0.04	185.6	0.52
Llama 3.1 8B Base	128k	24	$0.03	65.9	0.57
Llama 3.1 8B Vertex	128k	24	$0.00	0.0	0.09
Llama 3.1 8B	128k	24	$0.20	213.7	0.27
Llama 3.1 8B	128k	24	$0.04	49.2	0.22
Llama 3.1 8B	128k	24	$0.10	483.1	0.31
Llama 3.1 8B	16k	24	$0.05	65.4	0.61
Llama 3.1 8B	128k	24	$0.06	750.7	0.32
Llama 3.1 8B	16k	24	$0.13	992.9	0.26
Llama 3.1 8B Turbo	128k	24	$0.18	224.1	0.28
Llama 3.1 8B	128k	24	$0.15	471.1	0.52
Pixtral 12B	128k	23	$0.15	100.1	0.31
Pixtral 12B	128k	23	$0.10	69.3	0.51
Mistral Small (Feb '24)	33k	23	$1.50	116.5	0.29
Mistral Small (Feb '24)	33k	23	$1.50	53.4	0.40
Ministral 8B	128k	22	$0.10	141.7	0.29
Llama 3.2 11B (Vision)	128k	22	$0.16	145.2	0.34
Llama 3.2 11B (Vision)	128k	22	$0.20	148.8	0.48
Llama 3.2 11B (Vision)	128k	22	$0.06	46.2	0.23
Llama 3.2 11B (Vision)	8k	22	$0.18	749.2	0.33
Llama 3.2 11B (Vision) Turbo	128k	22	$0.18	142.9	0.24
Command-R+	128k	21	$6.00	47.0	0.49
Command-R+	128k	21	$4.38	73.9	0.23
Codestral (May '24)	33k	20	$0.30	84.0	0.30
Aya Expanse 32B	128k	20	$0.75	120.9	0.16
DBRX	33k	20	$1.13	66.4	0.47
DBRX	33k	20	$1.20	84.4	0.47
Ministral 3B	128k	20	$0.04	182.2	0.30
Mistral NeMo	128k	20	$0.15	116.5	0.28
Mistral NeMo Fast	128k	20	$0.12	157.7	0.53
Mistral NeMo Base	128k	20	$0.06	25.4	0.79
Mistral NeMo	128k	20	$0.06	68.1	0.25
DeepSeek R1 Distill Qwen 1.5B	128k	20	$0.18	368.1	7.65
OpenChat 3.5	8k	16	$0.06	81.4	0.30
Command-R (Mar '24)	128k	15	$0.75	108.5	0.35
Command-R (Mar '24)	128k	15	$0.75	170.8	0.16
Command-R (Mar '24)	128k	15	$0.75	79.7	0.46
Codestral-Mamba	256k	14	$0.25	95.6	0.46
Llama 2 Chat 7B	4k	8	$0.10	122.4	0.42
o1-preview	128k		$26.25	107.1	27.44
o1-preview	128k		$28.88	108.4	37.09
Llama 3.2 3B	128k		$0.10	202.6	0.43
Llama 3.2 3B	128k		$0.15	72.2	0.34
Llama 3.2 3B Base	128k		$0.01	122.8	0.53
Llama 3.2 3B	128k		$0.10	329.4	0.45
Llama 3.2 3B	128k		$0.02	138.5	0.18
Llama 3.2 3B	32k		$0.04	121.2	0.65
Llama 3.2 3B	8k		$0.06	1,547.1	0.33
Llama 3.2 3B	4k		$0.10	1,483.0	0.18
Llama 3.2 3B Turbo	128k		$0.06	56.8	0.80
Llama 3.2 1B	128k		$0.10	118.0	0.31
Llama 3.2 1B Base	128k		$0.01	265.3	0.50
Llama 3.2 1B	128k		$0.01	138.1	0.25
Llama 3.2 1B	8k		$0.04	3,271.2	0.49
Llama 3.2 1B	4k		$0.05	2,062.8	0.27
Gemini 2.0 Flash (exp) (AI Studio)	1m		$0.00	154.8	0.26
Gemini 1.5 Flash (Sep) (Vertex)	1m		$0.13	0.0	0.10
Gemma 2 27B	8k		$0.80	81.4	1.99
Gemma 2 9B Fast	8k		$0.04	195.7	0.48
Gemma 2 9B Base	8k		$0.03	165.8	0.52
Gemma 2 9B	8k		$0.04	47.4	0.34
Gemma 2 9B	8k		$0.20	655.0	0.23
Gemma 2 9B	8k		$0.30	135.3	0.20
Claude 3.5 Sonnet (June)	200k		$6.00	44.1	1.01
Claude 3.5 Sonnet (June) Vertex	200k		$6.00	74.5	0.87
Claude 3.5 Sonnet (June)	200k		$6.00	85.9	0.81
Claude 3 Haiku	200k		$0.50	98.0	0.77
Claude 3 Haiku	200k		$0.50	136.9	0.50
Mixtral 8x7B	33k		$0.70	101.8	0.29
Mixtral 8x7B	33k		$0.51	58.4	0.34
Mixtral 8x7B Fast	33k		$0.23	161.7	0.52
Mixtral 8x7B Base	33k		$0.12	133.4	0.51
Mixtral 8x7B	33k		$0.50	176.7	0.45
Mixtral 8x7B	33k		$0.24	102.2	0.21
Mixtral 8x7B	33k		$0.24	568.6	0.24
Mixtral 8x7B	33k		$0.63	91.4	0.45
Mixtral 8x7B	33k		$0.60	106.1	0.47
Nova Lite	300k		$0.10	141.1	0.40
Command-R+ (Apr '24)	128k		$6.00	46.7	0.50
Command-R+ (Apr '24)	128k		$6.00	77.5	0.25
Command-R+ (Apr '24)	128k		$6.00	50.3	0.60
Aya Expanse 8B	8k		$0.75	166.0	0.14
Command-R	128k		$0.75	108.6	0.36
Command-R	128k		$0.26	100.5	0.17
Jamba 1.5 Mini	256k		$0.25	170.6	0.31
Jamba 1.5 Mini	256k		$0.25	83.2	0.50
Qwen2 72B	33k		$0.90	66.8	0.45
GPT-4 Turbo	128k		$15.00	34.0	1.15
GPT-4	8k		$37.50	18.9	1.11
Llama 3 8B	8k		$0.10	69.2	0.41
Llama 3 8B	8k		$0.38	103.1	0.32
Llama 3 8B	8k		$0.20	229.3	0.46
Llama 3 8B	8k		$0.04	104.2	0.19
Llama 3 8B	8k		$0.04	45.7	0.75
Llama 3 8B	8k		$0.06	1,197.2	0.34
Llama 3 8B	8k		$0.20	278.0	0.26
Claude 3 Sonnet	200k		$6.00	45.7	0.76
Claude 3 Sonnet	200k		$6.00	56.9	0.51
Claude 2.1	200k		$12.00	28.9	1.76
Claude 2.1	200k		$12.00	13.4	0.82
Claude 2.0	100k		$12.00	29.5	0.81
Mistral 7B	8k		$0.25	128.9	0.31
Mistral 7B	8k		$0.16	93.0	0.33
Mistral 7B	8k		$0.04	92.6	0.22
Mistral 7B	32k		$0.06	100.1	0.94
Mistral 7B	8k		$0.20	171.6	0.24
Jamba Instruct	256k		$0.55	172.3	0.30
Jamba Instruct	256k		$0.55	75.2	1.08

Key definitions

Back to Navigation

Context window: Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).

Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).

Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.

Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Output Price: Price per token generated by the model (received from the API), represented as USD per million Tokens.

Input Price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.

Time period: Metrics are 'live' and are based on the past 72 hours of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.