LLM Leaderboard - Comparison of GPT-4o, Llama 3, Mistral, Gemini and over 30 models

Comparison and ranking the performance of over 30 AI models (LLMs) across key metrics including quality, price, performance and speed (output speed - tokens per second & latency - TTFT), context window & others. For more details including relating to our methodology, see our FAQs.

For comparison of API Providers hosting the models see

HIGHLIGHTS

Quality:

o1-preview and o1-mini logo

o1-mini are the highest quality models, followed by Gemini 1.5 Pro (Sep '24) logo

Gemini 1.5 Pro (Sep '24) & GPT-4o logo

GPT-4o.Output Speed (tokens/s): Llama 3.2 1B logo

Llama 3.2 1B (535 t/s) and Gemini 1.5 Flash (May '24) logo

Gemini 1.5 Flash (May '24) (307 t/s) are the fastest models, followed by Gemini 1.5 Flash-8B logo

Gemini 1.5 Flash-8B & Gemini 1.5 Flash (Sep '24) logo

Gemini 1.5 Flash (Sep '24).Latency (seconds): Sonar 3.1 Small logo

Sonar 3.1 Small (0.20s) and Sonar 3.1 Large logo

Sonar 3.1 Large (0.24s) are the lowest latency models, followed by Gemma 2 9B logo

Gemma 2 9B & OpenChat 3.5 logo

OpenChat 3.5.Price ($ per M tokens): Llama 3.2 1B logo

Llama 3.2 1B ($0.05) and OpenChat 3.5 logo

OpenChat 3.5 ($0.06) are the cheapest models, followed by Gemini 1.5 Flash-8B logo

Gemini 1.5 Flash-8B & Llama 3.2 3B logo

Llama 3.2 3B.Context Window: Gemini 1.5 Pro (Sep '24) logo

Gemini 1.5 Pro (Sep '24) (2m) and Gemini 1.5 Pro (May '24) logo

Gemini 1.5 Pro (May '24) (2m) are the largest context window models, followed by Gemini 1.5 Flash-8B logo

Gemini 1.5 Flash-8B & Gemini 1.5 Flash (Sep '24) logo

Gemini 1.5 Flash (Sep '24).

		Quality	Price	Output tokens/s	Latency
						Further Analysis
o1-preview	128k	85	$26.25	31.7	31.79
o1-mini	128k	82	$5.25	70.1	14.65
GPT-4o	128k	77	$4.38	123.9	0.42
GPT-4o (May '24)	128k	77	$7.50	109.8	0.41
GPT-4o mini	128k	71	$0.26	101.2	0.45
Llama 3.1 405B	128k	72	$4.50	24.9	0.88
Llama 3.2 90B (Vision)	128k	66	$0.90	39.8	0.52
Llama 3.1 70B	128k	65	$0.89	51.3	0.57
Llama 3.2 11B (Vision)	128k	54	$0.19	117.4	0.36
Llama 3.1 8B	128k	53	$0.15	164.4	0.38
Llama 3.2 3B	128k	47	$0.08	181.1	0.35
Llama 3.2 1B	128k	27	$0.05	534.7	0.36
Gemini 1.5 Pro (Sep '24)	2m	80	$2.19	60.6	0.78
Gemini 1.5 Flash-8B	1m		$0.07	283.2	0.35
Gemini 1.5 Flash (Sep '24)	1m	73	$0.13	207.6	0.38
Gemma 2 27B	8k	61	$0.80	66.0	0.48
Gemma 2 9B	8k	46	$0.20	116.8	0.30
Gemini 1.5 Flash (May '24)	1m		$0.13	307.3	0.33
Gemini 1.5 Pro (May '24)	2m		$5.25	65.0	0.76
Claude 3.5 Sonnet	200k	77	$6.00	58.6	0.88
Claude 3 Opus	200k	70	$30.00	25.8	2.22
Claude 3 Haiku	200k	54	$0.50	132.7	0.46
Mistral Large 2	128k	73	$4.50	35.5	0.52
Mixtral 8x22B	65k	61	$1.20	69.3	0.40
Mistral Small (Sep '24)	128k	60	$0.30	74.8	0.50
Pixtral 12B	128k	56	$0.13	80.2	0.53
Mistral NeMo	128k	52	$0.15	120.3	0.34
Mixtral 8x7B	33k	42	$0.50	88.7	0.38
Codestral-Mamba	256k	36	$0.25	94.0	0.68
Command-R+	128k	56	$5.19	47.1	0.52
Command-R	128k	51	$0.51	105.2	0.35
Command-R+ (Apr '24)	128k	46	$6.00	45.3	0.58
Command-R (Mar '24)	128k	36	$0.75	106.1	0.38
Sonar 3.1 Small	131k		$0.20	132.4	0.20
Sonar 3.1 Large	131k		$1.00	58.2	0.24
Phi-3 Medium 14B	128k		$0.30	50.7	0.44
DBRX	33k	49	$1.16	86.2	0.46
Reka Core	128k	57	$3.00	14.9	1.14
Reka Flash	128k	46	$0.35	29.1	1.11
Reka Edge	64k	30	$0.10	35.2	0.96
Jamba 1.5 Large	256k	64	$3.50	52.0	0.89
Jamba 1.5 Mini	256k	46	$0.25	141.2	0.73
DeepSeek-Coder-V2	128k	67	$0.17	16.0	1.14
DeepSeek-V2	128k	66	$0.17	16.4	1.15
DeepSeek-V2.5	128k	66	$1.09	13.7	1.05
Qwen2.5 72B	131k	75	$0.38	35.5	0.54
Qwen2 72B	128k	69	$0.90	56.2	0.44
Yi-Large	32k	58	$3.00	63.0	0.48
GPT-4 Turbo	128k	74	$15.00	36.2	0.62
GPT-3.5 Turbo	16k	52	$0.75	88.1	0.38
GPT-3.5 Turbo Instruct	4k		$1.63	116.0	0.55
GPT-4	8k		$37.50	28.8	0.58
Llama 3 70B	8k	62	$0.90	51.9	0.50
Llama 3 8B	8k	46	$0.15	95.6	0.40
Llama 2 Chat 70B	4k	39	$1.39	48.1	0.46
Llama 2 Chat 13B	4k	36	$0.30	53.8	0.42
Llama 2 Chat 7B	4k		$0.33	124.4	0.55
Gemini 1.0 Pro	33k		$0.75	97.3	1.20
Claude 3 Sonnet	200k	57	$6.00	59.7	0.79
Mistral Large	33k	56	$6.00	35.7	0.63
Mistral Small (Feb '24)	33k	50	$1.50	61.4	0.47
Mistral 7B	33k	24	$0.16	108.3	0.34
Codestral	33k		$0.30	47.5	0.57
Mistral Medium	33k		$4.09	37.8	0.86
OpenChat 3.5	8k	43	$0.06	74.2	0.32
Jamba Instruct	256k	28	$0.55	75.5	0.73

Key definitions

Back to Navigation

Artificial Analysis Quality Index: Average result across our evaluations covering different dimensions of model intelligence. Currently includes MMLU, GPQA, Math & HumanEval. OpenAI o1 model figures are preliminary and are based on figures stated by OpenAI. See methodology for more details.

Context window: Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).

Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).

Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.

Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Output price: Price per token generated by the model (received from the API), represented as USD per million Tokens.

Input price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.

Time period: Metrics are 'live' and are based on the past 14 days of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.

Models compared: OpenAI: GPT-3.5 Turbo, GPT-3.5 Turbo (0125), GPT-3.5 Turbo (1106), GPT-3.5 Turbo Instruct, GPT-4, GPT-4 Turbo, GPT-4 Turbo (0125), GPT-4 Vision, GPT-4o, GPT-4o (May '24), GPT-4o mini, o1-mini, and o1-preview, Meta: Code Llama 70B, Llama 2 Chat 13B, Llama 2 Chat 70B, Llama 2 Chat 7B, Llama 3 70B, Llama 3 8B, Llama 3.1 405B, Llama 3.1 70B, Llama 3.1 8B, Llama 3.2 11B (Vision), Llama 3.2 1B, Llama 3.2 3B, and Llama 3.2 90B (Vision), Google: Gemini 1.0 Pro, Gemini 1.5 Flash (May '24), Gemini 1.5 Flash (Sep '24), Gemini 1.5 Flash-8B, Gemini 1.5 Pro (May '24), Gemini 1.5 Pro (Sep '24), Gemma 2 27B, Gemma 2 9B, and Gemma 7B, Anthropic: Claude 2.0, Claude 2.1, Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet, Claude 3.5 Sonnet, and Claude Instant, Mistral: Codestral, Codestral-Mamba, Mistral 7B, Mistral Large, Mistral Large 2, Mistral Medium, Mistral NeMo, Mistral Small (Feb '24), Mistral Small (Sep '24), Mixtral 8x22B, Mixtral 8x7B, and Pixtral 12B, Cohere: Command, Command Light, Command-R, Command-R (Mar '24), Command-R+ (Apr '24), and Command-R+, Perplexity: PPLX-70B Online, PPLX-7B-Online, Sonar 3.1 Large, Sonar 3.1 Small , Sonar Large, and Sonar Small, xAI: Grok-1, OpenChat: OpenChat 3.5, Microsoft Azure: Phi-3 Medium 14B and Phi-3 Mini, Databricks: DBRX, Reka AI: Reka Core, Reka Edge, and Reka Flash, Other: LLaVA-v1.5-7B, AI21 Labs: Jamba 1.5 Large, Jamba 1.5 Mini, and Jamba Instruct, DeepSeek: DeepSeek-Coder-V2, DeepSeek-V2, and DeepSeek-V2.5, Snowflake: Arctic, Alibaba: Qwen2 72B and Qwen2.5 72B, and 01.AI: Yi-Large.