Artificial Analysis LLM Performance Leaderboard

Independent performance benchmarks & pricing across API providers of LLMs. Definitions are below the table.
For further analysis and methodology, see artificialanalysis.ai.

	Features		Price	Output tokens/s	Latency
						Further Analysis
o1-preview	128k	85	$26.25	37.3	27.51
o1-mini	128k	82	$5.25	80.2	12.90
GPT-4o (Aug '24)	128k	77	$4.38	81.2	0.41
GPT-4o (Aug '24)	128k	77	$4.38	94.6	0.80
GPT-4o (May '24)	128k	77	$7.50	80.8	0.39
GPT-4o (May '24)	128k	77	$7.50	114.8	0.77
GPT-4o mini	128k	71	$0.26	86.8	0.39
GPT-4o mini	128k	71	$0.26	163.0	0.67
GPT-4o (Nov '24)	128k	71	$4.38	181.5	0.41
Llama 3.1 405B	128k	72	$9.50	18.6	0.89
Llama 3.1 405B	128k	74	$4.00	16.3	0.89
Llama 3.1 405B	128k	72	$7.99	13.0	1.77
Llama 3.1 405B Base	128k	72	$1.50	33.7	0.82
Llama 3.1 405B Vertex	128k	72	$7.75	29.5	0.42
Llama 3.1 405B	128k	72	$8.00	19.2	0.55
Llama 3.1 405B	128k	72	$3.00	78.6	0.55
Llama 3.1 405B	33k	72	$1.79	18.7	0.54
Llama 3.1 405B	8k	74	$6.25	166.4	1.49
Llama 3.1 405B Turbo	128k	72	$3.50	80.0	0.71
Llama 3.2 90B (Vision)	128k	67	$2.00	40.3	0.49
Llama 3.2 90B (Vision) Vertex	128k	67	$0.00	35.6	0.20
Llama 3.2 90B (Vision)	128k	66	$0.90	48.5	0.44
Llama 3.2 90B (Vision)	8k	67	$0.36	32.4	0.32
Llama 3.2 90B (Vision)	8k	67	$0.90	250.3	0.35
Llama 3.2 90B (Vision) Turbo	128k	66	$1.20	54.9	0.33
Llama 3.1 70B	8k	66	$0.60	2,113.0	0.36
Llama 3.1 70B	128k	66	$0.40	30.9	0.68
Llama 3.1 70B	128k	65	$0.99	31.3	0.68
Llama 3.1 70B Base	128k	65	$0.20	43.9	0.69
Llama 3.1 70B Fast	128k	65	$0.38	71.3	0.65
Llama 3.1 70B Vertex	128k	65	$0.00	72.0	0.28
Llama 3.1 70B	128k	65	$2.90	29.0	0.56
Llama 3.1 70B	128k	65	$0.90	138.4	0.32
Llama 3.1 70B (Turbo, FP8)	128k	65	$0.32	35.8	0.28
Llama 3.1 70B	128k	65	$0.36	32.7	0.31
Llama 3.1 70B	128k	66	$0.64	250.2	0.35
Llama 3.1 70B (Spec decoding)	8k	66	$0.69	1,669.2	0.36
Llama 3.1 70B	64k	65	$0.75	454.3	0.54
Llama 3.1 70B	128k	65	$1.00	50.0	0.35
Llama 3.1 70B Turbo	128k	65	$0.88	117.5	0.47
Llama 3.1 70B	128k	64	$0.90	114.0	0.46
Llama 3.2 11B (Vision)	128k	53	$0.35	134.9	0.35
Llama 3.2 11B (Vision)	128k	54	$0.20	118.1	0.29
Llama 3.2 11B (Vision)	128k	53	$0.06	75.8	0.21
Llama 3.2 11B (Vision)	8k	54	$0.18	750.9	0.26
Llama 3.2 11B (Vision) Turbo	128k	54	$0.18	146.7	0.31
Llama 3.1 8B	8k	53	$0.10	2,205.4	0.37
Llama 3.1 8B	128k	54	$0.10	111.0	0.49
Llama 3.1 8B	128k	53	$0.22	90.3	0.37
Llama 3.1 8B Fast	128k	53	$0.04	183.2	0.56
Llama 3.1 8B Base	128k	53	$0.03	65.9	0.60
Llama 3.1 8B Vertex	128k	53	$0.00	119.9	0.19
Llama 3.1 8B	128k	53	$0.38	57.9	0.40
Llama 3.1 8B	128k	53	$0.20	266.4	0.26
Llama 3.1 8B	128k	53	$0.06	78.5	0.21
Llama 3.1 8B	128k	53	$0.06	750.3	0.26
Llama 3.1 8B	16k	52	$0.13	1,117.0	0.38
Llama 3.1 8B	128k	53	$0.20	157.1	0.36
Llama 3.1 8B Turbo	128k	52	$0.18	155.0	0.36
Llama 3.1 8B	128k	52	$0.15	342.2	0.41
Llama 3.2 3B	128k	36	$0.10	202.1	0.50
Llama 3.2 3B	128k	47	$0.15	143.9	0.40
Llama 3.2 3B	128k	47	$0.10	262.6	0.28
Llama 3.2 3B	128k	47	$0.04	94.2	0.27
Llama 3.2 3B	8k	47	$0.06	1,688.0	0.31
Llama 3.2 3B	4k	47	$0.10	1,560.0	0.33
Llama 3.2 3B Turbo	128k	47	$0.06	127.4	0.35
Llama 3.2 1B	128k	29	$0.10	313.9	0.34
Llama 3.2 1B	128k	26	$0.10	554.1	0.31
Llama 3.2 1B	128k	27	$0.01	164.5	0.26
Llama 3.2 1B	8k	27	$0.04	3,314.6	0.48
Llama 3.2 1B	4k	27	$0.05	2,479.0	0.31
Gemini 1.5 Pro (Sep) (Vertex)	2m	80	$2.19	58.2	0.45
Gemini 1.5 Pro (Sep) (AI Studio)	2m	80	$2.19	59.7	0.82
Gemini 1.5 Flash (Sep) (Vertex)	1m	68	$0.13	187.8	0.24
Gemini 1.5 Flash (Sep) (AI Studio)	1m	68	$0.13	189.6	0.40
Gemma 2 27B	8k	61	$0.80	49.9	0.52
Gemma 2 9B Fast	8k	47	$0.04	179.6	0.55
Gemma 2 9B Base	8k	47	$0.03	165.8	0.53
Gemma 2 9B	8k	48	$0.06	49.9	0.35
Gemma 2 9B	8k	45	$0.20	664.9	0.19
Gemma 2 9B	8k	48	$0.30	97.3	0.41
Gemini 1.5 Pro (May) (Vertex)	2m	72	$5.25	64.5	0.43
Gemini 1.5 Pro (May) (AI Studio)	2m	72	$5.25	61.5	0.85
Gemini 1.5 Flash (May) (Vertex)	1m		$0.13	310.4	0.26
Gemini 1.5 Flash (May) (AI Studio)	1m		$0.13	310.9	0.35
Gemini Experimental (Nov) (AI Studio)	33k		$0.00	40.7	2.65
Gemini 1.5 Flash-8B AI Studio	1m		$0.07	284.2	0.35
Claude 3.5 Sonnet (Oct)	200k	80	$6.00	47.6	0.99
Claude 3.5 Sonnet (Oct) Vertex	200k	80	$6.00	60.2	0.75
Claude 3.5 Sonnet (Oct)	200k	80	$6.00	59.6	0.80
Claude 3.5 Sonnet (June)	200k	77	$6.00	48.9	0.96
Claude 3.5 Sonnet (June) Vertex	200k	77	$6.00	60.3	0.74
Claude 3.5 Sonnet (June)	200k	77	$6.00	59.5	1.00
Claude 3 Opus	200k	70	$30.00	27.4	1.61
Claude 3 Opus Vertex	200k	70	$30.00	28.1	3.21
Claude 3 Opus	200k	70	$30.00	26.8	2.59
Claude 3.5 Haiku	200k	69	$2.00	60.0	0.81
Claude 3.5 Haiku Vertex	200k	69	$2.00	68.7	0.99
Claude 3.5 Haiku	200k	69	$2.00	67.0	0.99
Claude 3 Haiku	200k	54	$0.50	118.1	0.47
Claude 3 Haiku	200k	54	$0.50	134.5	0.50
Mistral Large 2 (Nov '24)	128k	74	$3.00	34.3	0.51
Mistral Large 2 (Jul '24)	128k	73	$3.00	32.9	0.48
Mistral Large 2 (Jul '24)	128k	73	$3.00	33.8	0.48
Mistral Large 2 (Jul '24)	128k	73	$3.00	56.2	0.42
Pixtral Large	128k	73	$3.00	35.4	0.63
Mixtral 8x22B	65k	62	$3.00	65.3	0.45
Mixtral 8x22B Base	65k	62	$0.60	86.4	0.70
Mixtral 8x22B Fast	65k	62	$1.05	97.5	0.71
Mixtral 8x22B	65k	61	$1.20	83.1	0.29
Mixtral 8x22B	65k	60	$1.20	60.4	0.81
Mistral Small (Sep '24)	128k	60	$0.30	54.8	0.47
Pixtral 12B	128k	56	$0.15	67.7	0.40
Pixtral 12B	128k	57	$0.10	75.6	0.51
Ministral 8B	128k	53	$0.10	136.1	0.42
Mistral NeMo	128k	52	$0.15	119.7	0.40
Mistral NeMo Fast	128k	53	$0.12	155.5	0.55
Mistral NeMo Base	128k	53	$0.06	52.1	0.64
Mistral NeMo	128k	53	$0.13	53.2	0.25
Ministral 3B	128k	51	$0.04	211.2	0.41
Mixtral 8x7B	33k	44	$0.70	87.3	0.46
Mixtral 8x7B	33k	42	$0.51	75.3	0.35
Mixtral 8x7B Fast	33k	43	$0.23	156.2	0.55
Mixtral 8x7B Base	33k	43	$0.12	130.1	0.54
Mixtral 8x7B	33k	42	$0.50	101.4	0.24
Mixtral 8x7B	33k	43	$0.24	43.5	0.30
Mixtral 8x7B	33k	45	$0.24	552.0	0.24
Mixtral 8x7B	33k	40	$0.60	86.2	0.36
Codestral-Mamba	256k	36	$0.25	94.8	0.57
Command-R+	128k	56	$6.00	43.4	0.52
Command-R+	128k	56	$4.38	71.1	0.29
Command-R	128k	51	$0.75	108.0	0.35
Command-R	128k	51	$0.26	116.3	0.22
Command-R+ (Apr '24)	128k	46	$6.00	43.1	0.52
Command-R+ (Apr '24)	128k	48	$6.00	66.6	0.31
Command-R+ (Apr '24)	128k	44	$6.00	45.7	0.70
Command-R (Mar '24)	128k	36	$0.75	108.1	0.35
Command-R (Mar '24)	128k	36	$0.75	163.6	0.21
Command-R (Mar '24)	128k	36	$0.75	103.5	0.52
Aya Expanse 32B	128k		$0.75	121.4	0.22
Aya Expanse 8B	8k		$0.75	137.7	0.26
Sonar 3.1 Large	131k		$1.00	58.2	0.36
Sonar 3.1 Small	131k		$0.20	142.7	0.35
Grok Beta	8k	70	$7.50	58.3	0.49
Phi-3 Medium 14B	128k		$0.30	45.7	0.44
Solar Pro	4k	61	$0.25	51.7	1.20
Solar Mini	4k	48	$0.15	85.1	1.13
DBRX	33k	48	$1.20	82.8	0.35
Llama 3.1 Nemotron 70B Base	128k	70	$0.20	47.7	0.63
Llama 3.1 Nemotron 70B Fast	128k	70	$0.38	69.7	0.62
Llama 3.1 Nemotron 70B	128k	70	$0.36	23.8	0.33
Reka Flash	128k	58	$0.35	32.8	1.22
Reka Core	128k	57	$2.00	14.9	1.12
Reka Flash (Feb '24)	128k	46	$0.35	31.2	0.89
Reka Edge	64k	30	$0.10	35.2	0.91
Jamba 1.5 Large	256k	64	$3.50	51.0	0.71
Jamba 1.5 Mini	256k	46	$0.25	82.5	0.49
DeepSeek-Coder-V2	128k	67	$0.17	16.4	1.06
DeepSeek-V2	128k	66	$0.17	16.5	1.07
DeepSeek-V2.5	64k	66	$0.17	16.5	1.08
DeepSeek-V2.5	128k	66	$2.00	7.5	0.85
Qwen2.5 72B	131k	75	$0.40	47.0	0.59
Qwen2.5 72B	131k	75	$0.20	46.0	0.63
Qwen2.5 72B Fast	131k	75	$0.38	67.2	0.57
Qwen2.5 72B	131k	75	$0.90	52.6	0.46
Qwen2.5 72B	33k	75	$0.36	21.8	0.40
Qwen2.5 72B	131k	75	$1.20	75.0	0.54
Qwen2.5 Coder 32B	131k	70	$0.20	35.2	0.52
Qwen2.5 Coder 32B	33k	70	$0.90	97.9	0.32
Qwen2.5 Coder 32B	33k	70	$0.18	55.1	0.28
Qwen2.5 Coder 32B	131k	70	$0.80	59.1	0.52
Qwen2 72B	33k	69	$0.36	21.9	0.37
Qwen2 72B	33k	69	$0.90	62.5	0.42
Yi-Large	32k	58	$3.00	67.2	0.44
GPT-4 Turbo	128k	74	$15.00	35.2	0.59
GPT-4 Turbo	128k	74	$15.00	44.7	1.61
GPT-3.5 Turbo	16k	53	$0.75	98.7	0.39
GPT-3.5 Turbo	16k	52	$0.75	120.9	0.68
GPT-3.5 Turbo Instruct	4k		$1.63	108.6	0.66
GPT-4	8k		$37.50	23.9	0.64
Llama 3 70B	8k	62	$1.18	46.4	0.35
Llama 3 70B	8k	62	$0.40	32.6	0.89
Llama 3 70B	8k	62	$2.86	43.9	0.45
Llama 3 70B	8k	61	$2.90	18.3	0.78
Llama 3 70B	8k	62	$0.90	111.8	0.29
Llama 3 70B	8k	62	$0.36	22.6	0.35
Llama 3 70B	8k	62	$0.64	349.8	0.21
Llama 3 70B (Reference, FP16)	8k	62	$0.90	109.5	0.56
Llama 3 70B (Turbo, FP8)	8k	62	$0.88	44.9	0.41
Llama 3 8B	8k	45	$0.10	57.9	0.35
Llama 3 8B	8k	46	$0.38	101.9	0.32
Llama 3 8B	8k	45	$0.38	73.9	0.47
Llama 3 8B	8k	46	$0.20	153.7	0.30
Llama 3 8B	8k	46	$0.06	123.0	0.21
Llama 3 8B	8k	46	$0.06	1,198.9	0.27
Llama 3 8B	8k	46	$0.20	220.2	0.39
Llama 2 Chat 13B	4k	25	$0.30	53.1	0.48
Llama 2 Chat 7B	4k		$0.10	123.9	0.33
Gemini 1.0 Pro (AI Studio)	33k		$0.75	102.4	1.26
Claude 3 Sonnet	200k	57	$6.00	60.0	0.76
Claude 3 Sonnet	200k	57	$6.00	62.9	1.14
Mistral Large (Feb '24)	33k	57	$6.00	33.8	0.47
Mistral Large (Feb '24)	33k	56	$6.00	36.1	0.41
Mistral Large (Feb '24)	33k	55	$6.00	39.4	0.56
Mistral Small (Feb '24)	33k	50	$1.50	59.3	0.42
Mistral Small (Feb '24)	33k	50	$1.50	53.0	0.44
Mistral 7B	33k	22	$0.25	131.4	0.42
Mistral 7B	33k	24	$0.16	93.5	0.33
Mistral 7B	33k	24	$0.06	94.4	0.21
Mistral 7B	8k	24	$0.20	115.0	0.31
Codestral	33k		$0.30	81.1	0.43
Mistral Medium	33k		$4.09	44.5	0.46
OpenChat 3.5	8k	43	$0.06	74.7	0.32
Jamba Instruct	256k	28	$0.55	75.5	0.53

Key definitions

Back to Navigation

Artificial Analysis Quality Index: Average result across our evaluations covering different dimensions of model intelligence. Currently includes MMLU, GPQA, Math & HumanEval. OpenAI o1 model figures are preliminary and are based on figures stated by OpenAI. See methodology for more details.

Context window: Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).

Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).

Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.

Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Output price: Price per token generated by the model (received from the API), represented as USD per million Tokens.

Input price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.

Time period: Metrics are 'live' and are based on the past 14 days of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.