Artificial Analysis LLM Performance Leaderboard

Independent performance benchmarks & pricing across API providers of LLMs. Definitions are below the table.
For further analysis and methodology, see artificialanalysis.ai.

	Features	Model Quality	Price	Output tokens/s	Latency
						Further Analysis
o1-preview	128k		$26.25	29.7	34.07
o1-mini	128k		$5.25	71.3	14.63
GPT-4o	128k	100	$4.38	115.9	0.39
GPT-4o (May '24)	128k	100	$7.50	109.1	0.38
GPT-4o (May '24)	128k	100	$7.50	110.2	0.37
GPT-4o mini	128k	88	$0.26	114.6	0.41
Llama 3.1 405B	128k	100	$9.50	18.3	1.09
Llama 3.1 405B	128k	100	$4.00	16.2	0.89
Llama 3.1 405B	128k	100	$7.99	13.4	1.77
Llama 3.1 405B	128k	100	$4.50	57.0	0.34
Llama 3.1 405B	128k	100	$2.80	18.6	1.01
Llama 3.1 405B	128k	100	$8.00	13.4	0.63
Llama 3.1 405B	128k	100	$3.00	72.3	0.59
Llama 3.1 405B	33k	100	$1.79	21.8	0.42
Llama 3.1 405B	8k	100	$6.25	128.6	1.48
Llama 3.1 405B	128k	100	$7.50	27.7	0.67
Llama 3.1 405B Turbo	8k	100	$3.50	90.2	0.66
Llama 3.1 70B	8k	95	$0.60	568.1	0.25
Llama 3.1 70B	128k	95	$0.40	28.9	0.66
Llama 3.1 70B	128k	95	$0.99	31.6	0.71
Llama 3.1 70B	128k	95	$0.90	58.4	0.30
Llama 3.1 70B	128k	95	$0.80	57.4	0.56
Llama 3.1 70B	128k	95	$2.90	22.2	0.61
Llama 3.1 70B	128k	95	$0.90	73.8	0.39
Llama 3.1 70B	128k	95	$0.36	30.8	0.29
Llama 3.1 70B	128k	95	$0.64	249.7	0.44
Llama 3.1 70B	8k	95	$0.75	402.8	0.81
Llama 3.1 70B	128k	95	$1.50	46.7	0.59
Llama 3.1 70B	128k	95	$1.00	47.3	0.26
Llama 3.1 70B Turbo	128k	95	$0.88	62.0	0.52
Llama 3.1 8B	8k	66	$0.10	2,018.0	0.26
Llama 3.1 8B	128k	66	$0.10	90.7	0.51
Llama 3.1 8B	128k	66	$0.22	86.8	0.40
Llama 3.1 8B	128k	66	$0.15	170.4	0.23
Llama 3.1 8B	128k	66	$0.07	212.0	0.35
Llama 3.1 8B	128k	66	$0.38	54.4	0.43
Llama 3.1 8B	128k	66	$0.20	265.4	0.26
Llama 3.1 8B	128k	66	$0.06	76.0	0.21
Llama 3.1 8B	128k	66	$0.06	751.3	0.37
Llama 3.1 8B	8k	66	$0.13	967.0	0.40
Llama 3.1 8B	128k	66	$0.20	160.9	0.18
Llama 3.1 8B Turbo	128k	66	$0.18	170.2	0.39
Llama 3.2 90B (Vision)	128k		$0.40	39.0	0.56
Llama 3.2 90B (Vision)	128k		$2.00	18.3	0.55
Llama 3.2 90B (Vision)	128k		$0.90	47.5	0.54
Llama 3.2 90B (Vision)	128k		$0.36	12.6	1.37
Llama 3.2 90B (Vision) Turbo	128k		$1.20	55.2	0.38
Llama 3.2 11B (Vision)	128k		$0.35	41.5	0.40
Llama 3.2 11B (Vision)	128k		$0.20	118.3	0.34
Llama 3.2 11B (Vision)	128k		$0.06	74.5	0.33
Llama 3.2 11B (Vision) Turbo	128k		$0.18	134.0	0.29
Llama 3.2 3B	128k		$0.10	189.5	0.52
Llama 3.2 3B	128k		$0.15	142.0	0.44
Llama 3.2 3B	128k		$0.00
Llama 3.2 3B	128k		$0.10	249.7	0.30
Llama 3.2 3B	128k		$0.04	111.1	0.28
Llama 3.2 3B	8k		$0.00	1,400.0	0.35
Llama 3.2 3B Turbo	128k		$0.06	123.0	0.30
Llama 3.2 1B	128k		$0.10	302.3	0.38
Gemini 1.5 Pro (Sep '24) (Vertex)	2m		$2.19	61.1	0.45
Gemini 1.5 Pro (Sep '24) (AI Studio)	2m		$2.19	62.2	0.81
Llama 3.2 1B	128k		$0.10	503.2	0.35
Llama 3.2 1B	128k		$0.01	169.0	0.24
Llama 3.2 1B	8k		$0.00	2,204.7	0.33
Gemini 1.5 Pro (May '24) (Vertex)	2m	95	$5.25	63.7	0.55
Gemini 1.5 Pro (May '24) (AI Studio)	2m	95	$5.25	65.0	0.87
Gemini 1.5 Flash (Sep '24) (Vertex)	1m		$0.13	207.6	0.26
Gemini 1.5 Flash (Sep '24) (AI Studio)	1m		$0.13	206.5	0.40
Gemini 1.5 Flash (May '24) (Vertex)	1m	84	$0.13	309.9	0.29
Gemini 1.5 Flash (May '24) (AI Studio)	1m	84	$0.13	313.6	0.36
Gemma 2 27B	8k	78	$0.80	50.7	0.46
Gemma 2 9B	8k	71	$0.06	69.6	0.28
Gemma 2 9B	8k	71	$0.20	667.6	0.19
Gemma 2 9B	8k	71	$0.30	107.7	0.44
Claude 3.5 Sonnet	200k	98	$6.00	50.0	0.96
Claude 3.5 Sonnet	200k	98	$6.00	90.9	0.85
Claude 3 Opus	200k	93	$30.00	22.9	1.71
Claude 3 Opus	200k	93	$30.00	27.7	1.88
Claude 3 Haiku	200k	74	$0.50	117.1	0.46
Claude 3 Haiku	200k	74	$0.50	143.0	0.48
Mistral Large 2	128k	91	$3.00	35.8	0.62
Mistral Large 2	128k	91	$4.50	43.0	0.41
Mixtral 8x22B	65k	71	$3.00	64.4	0.53
Mixtral 8x22B	65k	71	$1.20	39.7	0.32
Mixtral 8x22B	65k	71	$1.20	77.2	0.31
Mixtral 8x22B	65k	71	$1.20	70.4	0.38
Mistral Small (Sep '24)	128k		$0.30	80.7	0.44
Pixtral 12B	128k		$0.15	79.9	0.59
Mistral NeMo	128k	64	$0.15	134.9	0.43
Mistral NeMo	128k	64	$0.20	157.3	0.22
Mistral NeMo	128k	64	$0.13	64.1	0.23
Mixtral 8x7B	33k	61	$0.70	85.9	0.46
Mixtral 8x7B	33k	61	$0.47	88.7	0.52
Mixtral 8x7B	33k	61	$0.51	66.0	0.36
Mixtral 8x7B	33k	61	$0.45	82.1	0.28
Mixtral 8x7B	33k	61	$0.50	109.7	0.50
Mixtral 8x7B	33k	61	$0.50	103.3	0.31
Mixtral 8x7B	33k	61	$0.24	40.3	0.28
Mixtral 8x7B	33k	61	$0.24	543.2	0.22
Mixtral 8x7B	33k	61	$0.63	87.4	0.48
Mixtral 8x7B	33k	61	$0.60	104.0	0.39
Codestral-Mamba	256k		$0.25	94.6	0.61
Command-R+	128k		$6.00	45.2	0.57
Command-R+	128k		$4.38	68.3	0.28
Command-R	128k		$0.75	102.3	0.40
Command-R	128k		$0.26	112.6	0.20
Command-R+ (Apr '24)	128k	75	$6.00	45.2	0.57
Command-R+ (Apr '24)	128k	75	$6.00	65.1	0.30
Command-R+ (Apr '24)	128k	75	$6.00	45.5	0.69
Command-R (Mar '24)	128k	63	$0.75	102.1	0.39
Command-R (Mar '24)	128k	63	$0.75	150.7	0.21
Command-R (Mar '24)	128k	63	$0.75	103.8	0.51
Sonar Large	33k		$1.00
Sonar Small	33k		$0.20
Sonar 3.1 Small	131k		$0.20	144.9	0.18
Sonar 3.1 Large	131k		$1.00	62.0	0.22
Phi-3 Medium 14B	128k		$0.33	50.9	0.46
DBRX	33k	62	$1.13	84.8	0.49
DBRX	33k	62	$1.20	103.2	0.35
Reka Core	128k	90	$4.00	14.8	1.14
Reka Flash	128k	78	$1.10	29.8	0.97
Reka Edge	64k	60	$0.55
Jamba 1.5 Large	256k	86	$3.50	61.1	1.02
Jamba 1.5 Mini	256k	64	$0.25	162.5	0.84
DeepSeek-Coder-V2	128k		$0.17	17.0	1.19
DeepSeek-V2	128k	82	$0.17	16.8	1.24
DeepSeek-V2.5	128k		$0.17	16.8	1.21
Qwen2.5 72B	131k		$0.40	39.5	0.61
Qwen2.5 72B	131k		$0.36	22.1	0.42
Qwen2 72B	33k	83	$0.36	30.6	0.42
Qwen2 72B	33k	83	$0.90	59.3	0.51
Yi-Large	32k	81	$3.00	63.5	0.42
GPT-4 Turbo	128k	94	$15.00	33.2	0.63
GPT-4 Turbo	128k	94	$15.00	47.4	0.54
GPT-3.5 Turbo	16k	59	$0.75	89.7	0.41
GPT-3.5 Turbo	16k	59	$0.75	81.6	0.34
GPT-3.5 Turbo Instruct	4k	60	$1.63	92.9	0.40
GPT-3.5 Turbo Instruct	4k	60	$1.63	131.8	0.61
GPT-4	8k	84	$37.50	22.1	0.68
GPT-4	8k	84	$37.50	39.5	0.54
Llama 3 70B	8k	83	$1.18	47.6	0.50
Llama 3 70B	8k	83	$0.40	28.7	1.59
Llama 3 70B	8k	83	$2.86	52.8	0.47
Llama 3 70B	8k	83	$0.90	64.7	0.31
Llama 3 70B	8k	83	$0.80	29.0	0.89
Llama 3 70B	8k	83	$2.90	18.7	0.76
Llama 3 70B	8k	83	$0.90	104.1	0.34
Llama 3 70B	8k	83	$0.36	20.9	0.32
Llama 3 70B	8k	83	$0.64	318.3	0.22
Llama 3 70B (Reference, FP16)	8k	83	$0.90	148.3	0.46
Llama 3 70B (Turbo, FP8)	8k	83	$0.88	82.0	0.48
Llama 3 8B	8k	64	$0.10	70.9	0.53
Llama 3 8B	8k	64	$0.38	78.3	0.34
Llama 3 8B	8k	64	$0.07	91.6	0.91
Llama 3 8B	8k	64	$0.38	73.2	0.43
Llama 3 8B	8k	64	$0.20	106.5	0.35
Llama 3 8B	8k	64	$0.06	107.2	0.20
Llama 3 8B	8k	64	$0.06	1,201.7	0.29
Llama 3 8B	8k	64	$0.20	280.7	0.48
Llama 2 Chat 70B	4k	57	$1.18	48.0	0.64
Llama 2 Chat 70B	4k	57	$2.10	36.3	0.47
Llama 2 Chat 70B	4k	57	$0.90	171.3	0.22
Llama 2 Chat 70B	4k	57	$1.60
Llama 2 Chat 13B	4k	39	$0.81	52.8	0.40
Llama 2 Chat 13B	4k	39	$0.20	170.1	0.23
Llama 2 Chat 13B	4k	39	$0.30	52.4	0.48
Llama 2 Chat 7B	4k	29	$0.10	123.7	0.52
Llama 2 Chat 7B	4k	29	$0.56
Gemma 7B	8k	45	$0.07	947.8	0.90
Gemini 1.0 Pro (AI Studio)	33k	62	$0.75	98.3	1.20
Claude 3 Sonnet	200k	80	$6.00	38.0	0.95
Claude 3 Sonnet	200k	80	$6.00	60.1	0.95
Claude 2.1	200k	55	$12.00	26.9	1.84
Claude 2.1	200k	55	$12.00	30.8	1.07
Claude Instant	100k	63	$1.20	61.7	0.61
Claude Instant	100k	63	$1.20	108.9	0.45
Claude 2.0	100k	70	$12.00	31.4	1.04
Mistral Large	33k	76	$6.00	33.8	0.61
Mistral Large	33k	76	$6.00
Mistral Large	33k	76	$6.00
Mistral Small (Feb '24)	33k	71	$1.50	47.5	0.47
Mistral Small (Feb '24)	33k	71	$1.50
Mistral 7B	33k	40	$0.25	96.5	0.43
Mistral 7B	33k	40	$0.10	50.2	0.61
Mistral 7B	33k	40	$0.16	78.5	0.35
Mistral 7B	33k	40	$0.15	176.0	0.20
Mistral 7B	33k	40	$0.07	100.4	0.96
Mistral 7B	33k	40	$0.06	109.1	0.20
Mistral 7B	16k	40	$0.20	123.3	0.20
Mistral 7B	8k	40	$0.20	126.0	0.26
Codestral	33k		$0.30	48.7	0.44
Mistral Medium	33k	70	$4.09	38.3	0.77
Command	4k		$1.63	21.8	0.55
Command	4k		$1.25	22.3	0.38
Command Light	4k		$0.38	33.1	0.59
Command Light	4k		$0.38	57.8	0.24
OpenChat 3.5	8k	50	$0.06	70.8	0.29
Jamba Instruct	256k	63	$0.55	81.3	0.74
Jamba Instruct	256k	63	$0.55

Key definitions

Back to Navigation

Artificial Analysis Quality Index: Average result across our evaluations covering different dimensions of model intelligence. Currently includes MMLU, GPQA, Math & HumanEval. OpenAI o1 model figures are preliminary and are based on figures stated by OpenAI. See methodology for more details.

Context window: Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).

Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).

Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.

Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Output price: Price per token generated by the model (received from the API), represented as USD per million Tokens.

Input price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.

Time period: Metrics are 'live' and are based on the past 14 days of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.