Artificial Analysis LLM Performance Leaderboard

Independent performance benchmarks & pricing across API providers of LLMs. Definitions are below the table.
For further analysis and methodology, see artificialanalysis.ai.

	Features		Price	Output tokens/s	Latency
						Further Analysis
o1-preview	128k	85	$26.25	149.2	23.38
o1-mini	128k	82	$5.25	231.7	9.31
GPT-4o (Aug '24)	128k	78	$4.38	75.5	0.41
GPT-4o (Aug '24)	128k	78	$4.38	85.3	0.81
GPT-4o (May '24)	128k	78	$7.50	78.0	0.39
GPT-4o (May '24)	128k	78	$7.50	103.6	0.78
GPT-4o (Nov '24)	128k	73	$4.38	118.9	0.39
GPT-4o mini	128k	73	$0.26	88.5	0.41
GPT-4o mini	128k	73	$0.26	132.8	0.80
Llama 3.3 70B	128k	74	$0.40	24.0	0.57
Llama 3.3 70B	128k	74	$0.90	199.3	0.34
Llama 3.3 70B	128k	74	$0.27	46.4	0.34
Llama 3.3 70B	128k	74	$0.64	275.5	0.25
Llama 3.3 70B	128k	75	$0.88	148.6	0.50
Llama 3.1 405B	128k	74	$9.50	18.6	0.93
Llama 3.1 405B	128k	75	$4.00	16.3	0.79
Llama 3.1 405B Standard	128k	74	$2.40	13.3	1.78
Llama 3.1 405B Latency Optimized	128k	74	$3.00	62.7	0.84
Llama 3.1 405B Base	128k	74	$1.50	33.9	0.75
Llama 3.1 405B Vertex	128k	74	$7.75	28.9	0.42
Llama 3.1 405B	128k	74	$8.00	21.9	0.52
Llama 3.1 405B	128k	73	$3.00	74.6	0.60
Llama 3.1 405B	33k	73	$1.79	22.7	0.72
Llama 3.1 405B	8k	74	$6.25	162.8	0.74
Llama 3.1 405B	128k	72	$7.50	30.6	0.76
Llama 3.1 405B Turbo	128k	74	$3.50	72.7	0.75
Llama 3.1 70B	16k	68	$0.60	2,111.8	0.38
Llama 3.1 70B	128k	69	$0.40	28.3	0.70
Llama 3.1 70B Standard	128k	68	$0.72	31.3	0.70
Llama 3.1 70B Latency Optimized	128k	68	$0.90	139.9	0.43
Llama 3.1 70B Base	128k	66	$0.20	38.6	0.69
Llama 3.1 70B Fast	128k	66	$0.38	71.1	0.68
Llama 3.1 70B Vertex	128k	68	$0.00	71.9	0.27
Llama 3.1 70B	128k	68	$2.90	28.0	0.55
Llama 3.1 70B	128k	67	$0.90	143.5	0.38
Llama 3.1 70B (Turbo, FP8)	128k	66	$0.20	27.9	0.33
Llama 3.1 70B	128k	66	$0.27	27.2	0.34
Llama 3.1 70B	128k	68	$0.60	106.5	0.36
Llama 3.1 70B	128k	54	$0.64	250.4	0.35
Llama 3.1 70B (Spec decoding)	8k	54	$0.69	1,622.7	0.35
Llama 3.1 70B	64k	65	$0.75	448.9	0.39
Llama 3.1 70B	128k	43	$1.50	62.8	0.59
Llama 3.1 70B	128k	68	$1.00	50.8	0.33
Llama 3.1 70B Turbo	128k	68	$0.88	98.3	0.53
Llama 3.1 70B	128k	62	$0.90	114.0	0.44
Llama 3.2 90B (Vision)	128k	67	$0.72	17.1	0.51
Llama 3.2 90B (Vision) Vertex	128k	68	$0.00	35.5	0.18
Llama 3.2 90B (Vision)	128k	66	$0.90	45.4	0.44
Llama 3.2 90B (Vision)	33k	68	$0.36	26.8	0.37
Llama 3.2 90B (Vision)	8k	67	$0.90	250.2	0.35
Llama 3.2 90B (Vision) Turbo	128k	66	$1.20	53.2	0.41
Llama 3.2 11B (Vision)	128k	53	$0.16	38.6	0.36
Llama 3.2 11B (Vision)	128k	54	$0.20	121.5	0.30
Llama 3.2 11B (Vision)	128k	54	$0.06	81.5	0.22
Llama 3.2 11B (Vision)	8k	53	$0.18	750.8	0.26
Llama 3.2 11B (Vision) Turbo	128k	54	$0.18	160.0	0.31
Llama 3.1 8B	16k	54	$0.10	2,111.4	0.37
Llama 3.1 8B	128k	53	$0.10	117.1	0.48
Llama 3.1 8B	128k	54	$0.22	90.7	0.38
Llama 3.1 8B Fast	128k	54	$0.04	177.7	0.54
Llama 3.1 8B Base	128k	54	$0.03	65.9	0.60
Llama 3.1 8B Vertex	128k	54	$0.00	119.8	0.17
Llama 3.1 8B	128k	54	$0.38	164.4	0.32
Llama 3.1 8B	128k	53	$0.20	257.1	0.26
Llama 3.1 8B	128k	54	$0.04	78.1	0.22
Llama 3.1 8B	128k	54	$0.10	294.8	0.31
Llama 3.1 8B	128k	53	$0.06	751.1	0.25
Llama 3.1 8B	16k	52	$0.13	1,119.2	0.28
Llama 3.1 8B	128k	54	$0.20	156.9	0.34
Llama 3.1 8B Turbo	128k	53	$0.18	100.5	0.64
Llama 3.1 8B	128k	52	$0.15	330.4	0.35
Llama 3.2 3B	128k	48	$0.10	200.0	0.46
Llama 3.2 3B	128k	49	$0.15	143.7	0.40
Llama 3.2 3B	128k	50	$0.10	265.8	0.29
Llama 3.2 3B	128k	49	$0.04	102.3	0.27
Llama 3.2 3B	8k	49	$0.06	1,668.5	0.31
Llama 3.2 3B	4k	49	$0.10	1,557.6	0.22
Llama 3.2 3B Turbo	128k	49	$0.06	61.7	0.33
Llama 3.2 1B	128k	26	$0.10	313.6	0.34
Llama 3.2 1B	128k	26	$0.10	551.5	0.31
Llama 3.2 1B	128k	26	$0.01	161.0	0.25
Llama 3.2 1B	8k	26	$0.04	3,338.1	0.46
Llama 3.2 1B	4k	26	$0.05	2,480.9	0.21
Gemini 1.5 Pro (Sep) (Vertex)	2m	80	$2.19	58.3	0.62
Gemini 1.5 Pro (Sep) (AI Studio)	2m	80	$2.19	60.6	0.86
Gemini 1.5 Flash (Sep) (Vertex)	1m	69	$0.13	188.0	0.23
Gemini 1.5 Flash (Sep) (AI Studio)	1m	69	$0.13	192.1	0.41
Gemma 2 27B	8k	61	$0.80	48.7	0.55
Gemma 2 9B Fast	8k	55	$0.04	182.2	0.51
Gemma 2 9B Base	8k	55	$0.03	165.3	0.52
Gemma 2 9B	8k	54	$0.06	61.9	0.35
Gemma 2 9B	8k	55	$0.20	660.4	0.19
Gemma 2 9B	8k	55	$0.30	111.2	0.39
Gemini 1.5 Flash-8B AI Studio	1m		$0.07	285.2	0.36
Gemini Experimental (Nov) (AI Studio)	33k		$0.00	40.5	2.66
Gemini 1.5 Flash (May) (Vertex)	1m	60	$0.13	303.6	0.27
Gemini 1.5 Flash (May) (AI Studio)	1m	60	$0.13	302.7	0.37
Gemini 1.5 Pro (May) (Vertex)	2m	72	$2.19	65.4	0.43
Gemini 1.5 Pro (May) (AI Studio)	2m	72	$2.19	65.8	0.84
Claude 3.5 Sonnet (Oct)	200k	80	$6.00	42.8	1.01
Claude 3.5 Sonnet (Oct) Vertex	200k	80	$6.00	59.1	0.75
Claude 3.5 Sonnet (Oct)	200k	80	$6.00	61.8	0.97
Claude 3.5 Sonnet (June)	200k	76	$6.00	48.3	1.00
Claude 3.5 Sonnet (June) Vertex	200k	76	$6.00	58.9	0.76
Claude 3.5 Sonnet (June)	200k	76	$6.00	59.5	1.11
Claude 3 Opus	200k	70	$30.00	25.6	1.62
Claude 3 Opus Vertex	200k	70	$30.00	27.7	3.15
Claude 3 Opus	200k	70	$30.00	27.2	1.62
Claude 3.5 Haiku Standard	200k	68	$1.60	57.8	0.70
Claude 3.5 Haiku Latency Optimized	200k	68	$1.60	101.2	0.62
Claude 3.5 Haiku Vertex	200k	68	$1.60	68.0	1.01
Claude 3.5 Haiku	200k	68	$1.60	64.6	0.86
Claude 3 Haiku	200k	55	$0.50	113.0	0.51
Claude 3 Haiku	200k	55	$0.50	134.2	0.43
Pixtral Large	128k	74	$3.00	37.1	0.65
Mistral Large 2 (Jul '24)	128k	74	$3.00	31.5	0.67
Mistral Large 2 (Jul '24)	128k	74	$3.00	34.1	0.47
Mistral Large 2 (Jul '24)	128k	74	$3.00	54.6	0.47
Mistral Large 2 (Nov '24)	128k	74	$3.00	44.1	0.70
Mistral Small (Sep '24)	128k	61	$0.30	64.4	0.59
Mixtral 8x22B	65k	62	$3.00	79.3	0.55
Mixtral 8x22B Base	65k	60	$0.60	86.1	0.68
Mixtral 8x22B Fast	65k	60	$1.05	99.8	0.70
Mixtral 8x22B	65k	61	$1.20	92.9	0.32
Mixtral 8x22B	65k	55	$1.20	64.0	0.61
Pixtral 12B	128k	56	$0.15	68.0	0.51
Pixtral 12B	128k	57	$0.10	79.1	0.46
Ministral 8B	128k	56	$0.10	134.0	0.52
Mistral NeMo	128k	53	$0.15	116.8	0.50
Mistral NeMo Fast	128k	53	$0.12	156.8	0.57
Mistral NeMo Base	128k	53	$0.06	52.3	0.62
Mistral NeMo	128k	54	$0.13	62.2	0.30
Ministral 3B	128k	53	$0.04	166.1	0.55
Mixtral 8x7B	33k	42	$0.70	97.5	0.56
Mixtral 8x7B	33k	41	$0.51	73.8	0.35
Mixtral 8x7B Fast	33k	40	$0.23	154.9	0.55
Mixtral 8x7B Base	33k	40	$0.12	129.8	0.55
Mixtral 8x7B	33k	43	$0.50	157.0	0.25
Mixtral 8x7B	33k	40	$0.24	48.2	0.30
Mixtral 8x7B	33k	38	$0.40	126.8	0.34
Mixtral 8x7B	33k	42	$0.24	549.9	0.25
Mixtral 8x7B	33k	42	$0.63	89.3	0.42
Mixtral 8x7B	33k	35	$0.60	87.3	0.38
Codestral-Mamba	256k	33	$0.25	96.0	0.68
Command-R+	128k	55	$6.00	41.3	0.53
Command-R+	128k	55	$4.38	73.6	0.29
Command-R+ (Apr '24)	128k	45	$6.00	42.0	0.53
Command-R+ (Apr '24)	128k	47	$6.00	71.7	0.32
Command-R+ (Apr '24)	128k	44	$6.00	47.5	0.60
Command-R (Mar '24)	128k	36	$0.75	108.7	0.36
Command-R (Mar '24)	128k	37	$0.75	164.5	0.22
Command-R (Mar '24)	128k	36	$0.75	75.7	0.46
Command-R	128k		$0.75	108.5	0.36
Command-R	128k	51	$0.26	118.2	0.22
Aya Expanse 8B	8k		$0.75	138.7	0.33
Aya Expanse 32B	128k		$0.75	121.3	0.23
Sonar 3.1 Small	131k		$0.20	139.9	0.33
Sonar 3.1 Large	131k		$1.00	54.6	0.33
Grok Beta	8k	72	$7.50	60.6	0.48
Nova Pro	300k	75	$1.40	94.9	0.42
Nova Lite	300k	70	$0.10	149.9	0.38
Nova Micro	130k	66	$0.06	201.8	0.34
Phi-3 Medium 14B	128k		$0.30	49.6	0.43
Solar Mini	4k	47	$0.15	85.3	1.11
DBRX	33k	50	$1.13	69.0	0.50
DBRX	33k	44	$1.20	82.9	0.34
Llama 3.1 Nemotron 70B Base	128k	72	$0.20	47.6	0.66
Llama 3.1 Nemotron 70B Fast	128k	72	$0.38	60.4	0.65
Llama 3.1 Nemotron 70B	128k	72	$0.27	36.3	0.32
Reka Flash	128k	59	$0.35	32.8	1.25
Reka Core	128k	58	$2.00	14.9	1.16
Reka Flash (Feb '24)	128k	46	$0.35	31.3	0.94
Reka Edge	64k	31	$0.10	35.3	0.92
Jamba 1.5 Mini	256k		$0.25	82.5	0.50
Jamba 1.5 Large	256k		$3.50	51.0	0.70
DeepSeek-Coder-V2	128k	71	$0.17	16.4	1.00
DeepSeek-V2.5	64k	66	$0.17	16.4	1.03
DeepSeek-V2.5	128k		$2.00	7.5	0.79
DeepSeek-V2	128k	66	$0.17	16.3	1.01
Qwen2.5 72B	131k	77	$0.40	49.3	0.62
Qwen2.5 72B	131k	77	$0.20	46.1	0.66
Qwen2.5 72B Fast	131k	77	$0.38	67.3	0.59
Qwen2.5 72B	131k	77	$0.90	54.9	0.41
Qwen2.5 72B	33k	78	$0.36	36.5	0.32
Qwen2.5 72B	131k	77	$1.20	93.7	0.49
Qwen2.5 Coder 32B	131k	72	$0.20	33.9	0.52
Qwen2.5 Coder 32B	33k	72	$0.90	99.8	0.33
Qwen2.5 Coder 32B	33k	71	$0.18	52.8	0.26
Qwen2.5 Coder 32B	131k	72	$0.80	56.8	0.53
Qwen2 72B	33k	74	$0.36	37.1	0.33
Qwen2 72B	33k	69	$0.90	64.3	0.42
Yi-Large	32k	61	$3.00	66.2	0.43
GPT-4 Turbo	128k	75	$15.00	36.9	0.62
GPT-4 Turbo	128k	75	$15.00	43.7	1.58
GPT-3.5 Turbo	16k	53	$0.75	123.5	0.32
GPT-3.5 Turbo	16k	53	$0.75	123.6	0.69
GPT-4	8k		$37.50	25.6	0.73
Llama 3 70B	8k	47	$1.18	45.7	0.50
Llama 3 70B	8k	62	$0.40	32.8	0.64
Llama 3 70B	8k	47	$2.86	45.5	0.44
Llama 3 70B	8k	61	$2.90	19.0	0.80
Llama 3 70B	8k	46	$0.90	110.0	0.35
Llama 3 70B	8k	48	$0.36	20.1	0.37
Llama 3 70B	8k	48	$0.64	341.6	0.22
Llama 3 70B (Reference, FP16)	8k	48	$0.90	110.1	0.53
Llama 3 70B (Turbo, FP8)	8k	48	$0.88	18.7	0.53
Llama 3 8B	8k	44	$0.10	58.8	0.51
Llama 3 8B	8k	45	$0.38	102.5	0.33
Llama 3 8B	8k	45	$0.38	73.8	0.38
Llama 3 8B	8k	45	$0.20	145.7	0.28
Llama 3 8B	8k	45	$0.06	121.7	0.22
Llama 3 8B	8k	45	$0.06	1,198.2	0.26
Llama 3 8B	8k	46	$0.20	234.6	0.44
Llama 2 Chat 13B	4k	25	$0.30	42.0	0.45
Llama 2 Chat 7B	4k		$0.10	125.2	0.50
Gemini 1.0 Pro (AI Studio)	33k		$0.75	103.3	1.29
Claude 3 Sonnet	200k	57	$6.00	58.2	0.78
Claude 3 Sonnet	200k	57	$6.00	64.0	0.88
Mistral Small (Feb '24)	33k	59	$1.50	66.9	0.57
Mistral Small (Feb '24)	33k	59	$1.50	52.5	0.39
Mistral Large (Feb '24)	33k	57	$6.00	37.7	0.65
Mistral Large (Feb '24)	33k	56	$6.00	36.3	0.41
Mistral Large (Feb '24)	33k	55	$6.00	39.6	0.52
Mistral 7B	33k	24	$0.25	132.5	0.50
Mistral 7B	33k	28	$0.16	93.5	0.33
Mistral 7B	33k	28	$0.06	94.9	0.21
Mistral 7B	8k	28	$0.20	119.2	0.37
Codestral	33k		$0.30	81.6	0.51
Mistral Medium	33k		$4.09	44.2	0.61
OpenChat 3.5	8k	44	$0.06	77.4	0.32
Jamba Instruct	256k	28	$0.55	75.1	0.53

Key definitions

Back to Navigation

Artificial Analysis Quality Index: Average result across our evaluations covering different dimensions of model intelligence. Currently includes MMLU, GPQA, Math & HumanEval. OpenAI o1 model figures are preliminary and are based on figures stated by OpenAI. See methodology for more details.

Context window: Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).

Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).

Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.

Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Output price: Price per token generated by the model (received from the API), represented as USD per million Tokens.

Input price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.

Time period: Metrics are 'live' and are based on the past 14 days of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.