LLM Leaderboard - Comparison of GPT-4o, Llama 3, Mistral, Gemini and over 30 models

Comparison and ranking the performance of over 30 AI models (LLMs) across key metrics including quality, price, performance and speed (output speed - tokens per second & latency - TTFT), context window & others. For more details including relating to our methodology, see our FAQs.

For comparison of API Providers hosting the models see

HIGHLIGHTS

Intelligence: o4-mini (high) logo

o4-mini (high) and Gemini 2.5 Pro Preview logo

Gemini 2.5 Pro Preview are the highest quality models, followed by o3 logo

o3 &

Grok 3 mini Reasoning (high).Output Speed (tokens/s):

DeepSeek R1 Distill Qwen 1.5B (382 t/s) and Gemini 2.5 Flash Preview logo

Gemini 2.5 Flash Preview (338 t/s) are the fastest models, followed by Nova Micro logo

Nova Micro &

Gemini 2.5 Flash Preview (Reasoning).Latency (seconds): Aya Expanse 8B logo

Aya Expanse 8B (0.12s) and Command-R (Mar '24) logo

Command-R (Mar '24) (0.14s) are the lowest latency models, followed by Aya Expanse 32B logo

Aya Expanse 32B & Command-R logo

Command-R.Price ($ per M tokens): Gemma 3 4B logo

Gemma 3 4B ($0.03) and Qwen2.5 Coder 7B logo

Qwen2.5 Coder 7B ($0.03) are the cheapest models, followed by Llama 3.2 1B logo

Llama 3.2 1B & Ministral 3B logo

Ministral 3B.Context Window: Llama 4 Scout logo

Llama 4 Scout (10m) and MiniMax-Text-01 logo

MiniMax-Text-01 (4m) are the largest context window models, followed by Gemini 2.0 Pro Experimental logo

Gemini 2.0 Pro Experimental & Gemini 1.5 Pro (Sep) logo

Gemini 1.5 Pro (Sep).

		Intelligence	Price	Output tokens/s	Latency
						Further Analysis
o4-mini (high)	200k	70	$1.93	135.5	34.10
Gemini 2.5 Pro Preview	1m	68	$3.44	209.6	29.05
o3	128k	67	$17.50
Grok 3 mini Reasoning (high)	1m	67	$0.35	91.4	0.35
o3-mini (high)	200k	66	$1.93	181.0	35.19
o3-mini	200k	63	$1.93	205.8	12.26
o1	200k	62	$26.25	77.4	37.96
Llama 3.1 Nemotron Ultra 253B Reasoning	128k	61	$0.00
Gemini 2.5 Flash Preview (Reasoning)	1m	60	$0.99	326.2	9.16
DeepSeek R1	128k	60	$0.96	25.8	3.68
QwQ-32B	131k	58	$0.47	91.8	0.51
Claude 3.7 Sonnet Thinking	200k	57	$6.00
o1-mini	128k	54	$1.93	219.2	10.66
DeepSeek V3 (Mar' 25)	128k	53	$0.48	25.7	4.09
GPT-4.1 mini	1m	53	$0.70	77.7	0.44
GPT-4.1	1m	53	$3.50	112.2	0.42
Gemini 2.0 Flash Thinking exp. (Jan '25)	1m	52	$0.00
DeepSeek R1 Distill Qwen 32B	128k	52	$0.22	21.3	0.98
Grok 3	1m	51	$6.00	47.2	0.54
Llama 4 Maverick	1m	51	$0.36	126.4	0.41
GPT-4o (March 2025)	128k	50	$7.50	202.8	0.50
Gemini 2.0 Pro Experimental	2m	49	$0.00	213.7	28.98
DeepSeek R1 Distill Qwen 14B	128k	49	$0.88	73.0	0.64
DeepSeek R1 Distill Llama 70B	128k	48	$0.60	69.5	0.48
Claude 3.7 Sonnet	200k	48	$6.00	78.4	1.11
Gemini 2.0 Flash	1m	48	$0.17	248.9	0.36
Reka Flash 3	128k	47	$0.35	56.8	0.96
Gemini 2.0 Flash (exp)	1m	46	$0.00	236.3	0.29
DeepSeek V3 (Dec '24)	128k	46	$0.48	25.2	4.08
Qwen2.5 Max	32k	45	$2.80	51.8	1.29
Gemini 1.5 Pro (Sep)	2m	45	$2.19	95.7	0.46
Claude 3.5 Sonnet (Oct)	200k	44	$6.00	78.8	1.00
Sonar	127k	43	$1.00	87.9	2.16
Llama 4 Scout	10m	43	$0.26	119.4	0.39
Sonar Pro	200k	43	$6.00	102.2	2.39
QwQ 32B-Preview	33k	43	$0.58	56.0	0.43
GPT-4o (Nov '24)	128k	41	$4.38	152.6	0.60
Gemini 2.0 Flash-Lite (Feb '25)	1m	41	$0.13	204.8	0.29
Llama 3.3 70B	128k	41	$0.60	118.0	0.48
GPT-4.1 nano	1m	41	$0.17	283.5	0.33
GPT-4o (May '24)	128k	41	$7.50	106.3	0.50
Llama 3.1 405B	128k	40	$3.50	31.6	0.79
Qwen2.5 72B	131k	40	$0.00	55.9	1.24
MiniMax-Text-01	4m	40	$0.42	32.9	0.80
Phi-4	16k	40	$0.22	44.6	0.49
Command A	256k	40	$4.38	54.8	0.25
Tulu3 405B	128k	40	$0.00
Llama 3.3 Nemotron Super 49B v1	128k	39	$0.00
Grok 2	131k	39	$0.00
Gemini 1.5 Flash (Sep)	1m	39	$0.13	196.3	0.21
Mistral Large 2 (Nov '24)	128k	38	$3.00	40.6	0.42
Gemma 3 27B	128k	38	$0.00
Grok Beta	128k	38	$7.50	66.3	0.29
Pixtral Large	128k	37	$3.00	25.0	0.51
Qwen2.5 Instruct 32B	128k	37	$0.15
Llama 3.1 Nemotron 70B	128k	37	$0.24	46.5	0.54
Nova Pro	300k	37	$1.75	110.4	0.40
Mistral Large 2 (Jul '24)	128k	37	$3.00	38.8	0.46
Qwen2.5 Coder 32B	131k	36	$0.20	49.0	0.53
GPT-4o mini	128k	36	$0.26	73.8	0.34
Llama 3.1 70B	128k	35	$0.66	50.8	0.56
Mistral Small 3.1	128k	35	$0.15	165.0	0.28
Mistral Small 3	32k	35	$0.15	130.8	0.29
Claude 3 Opus	200k	35	$30.00	28.8	1.08
Claude 3.5 Haiku	200k	35	$1.60	66.3	2.83
DeepSeek R1 Distill Llama 8B	128k	34	$0.04	46.2	0.72
Gemma 3 12B	128k	34	$0.06	26.2	0.59
Gemini 1.5 Pro (May)	2m	34	$2.19	66.4	0.37
Qwen Turbo	1m	34	$0.09	108.9	1.05
Llama 3.2 90B (Vision)	128k	33	$0.81	37.6	0.36
Qwen2 72B	131k	33	$0.00	31.0	1.36
Nova Lite	300k	33	$0.10	280.2	0.34
Gemini 1.5 Flash-8B	1m	31	$0.07	280.6	0.21
DeepHermes 3 - Mistral 24B	32k	30	$0.00
Jamba 1.5 Large	256k	29	$3.50	66.7	0.50
Hermes 3 - Llama-3.1 70B	128k	29	$0.00
Jamba 1.6 Large	256k	29	$3.50	66.0	0.58
Gemini 1.5 Flash (May)	1m	28	$0.13	314.7	0.28
Nova Micro	130k	28	$0.06	326.4	0.32
Yi-Large	32k	28	$3.00	69.3	0.42
Claude 3 Sonnet	200k	28	$6.00	61.2	0.55
Codestral (Jan '25)	256k	28	$0.45	202.2	0.27
Llama 3 70B	8k	27	$0.88	47.5	0.44
Mistral Small (Sep '24)	33k	27	$0.30	67.1	0.33
Phi-4 Multimodal	128k	27	$0.00	21.3	0.34
Qwen2.5 Coder 7B	131k	27	$0.03	208.8	0.49
Mistral Large (Feb '24)	33k	26	$6.00	30.6	0.46
Mixtral 8x22B	65k	26	$3.00	64.2	0.33
Phi-4 Mini	128k	26	$0.00	57.0	0.34
Phi-3 Medium 14B	128k	25	$0.30	51.0	0.43
Gemma 3 4B	128k	24	$0.03	91.5	0.30
Claude 2.1	200k	24	$12.00	14.0	0.84
Llama 3.1 8B	128k	24	$0.10	169.0	0.35
Pixtral 12B	128k	23	$0.15	105.0	0.30
Mistral Small (Feb '24)	33k	23	$1.50	167.6	0.28
Mistral Medium	33k	23	$4.09	42.0	0.37
Ministral 8B	128k	22	$0.10	142.7	0.28
Gemma 2 9B	8k	22	$0.12
Phi-3 Mini	4k	22	$0.00
LFM 40B	32k	22	$0.15	170.0	0.44
Command-R+	128k	21	$4.38	52.0	0.27
Llama 3 8B	8k	21	$0.09	99.3	0.37
Gemini 1.0 Pro	33k	21	$0.75
Codestral (May '24)	33k	20	$0.30	106.9	0.31
Aya Expanse 32B	128k	20	$0.75	121.8	0.16
Llama 2 Chat 13B	4k	20	$0.00
Command-R+ (Apr '24)	128k	20	$6.00	68.9	0.23
DBRX	33k	20	$1.13	69.7	0.52
Ministral 3B	128k	20	$0.04	226.3	0.27
Mistral NeMo	128k	20	$0.15	138.1	0.29
Llama 3.2 3B	128k	20	$0.05	146.3	0.46
DeepSeek R1 Distill Qwen 1.5B	128k	19	$0.18	382.3	0.28
Jamba 1.5 Mini	256k	18	$0.25	168.0	0.31
Jamba 1.6 Mini	256k	18	$0.25	198.5	0.34
Mixtral 8x7B	33k	17	$0.70	89.5	0.30
DeepHermes 3 - Llama-3.1 8B	128k	16	$0.00
Aya Expanse 8B	8k	16	$0.75	166.6	0.12
Command-R	128k	15	$0.26	84.6	0.19
Command-R (Mar '24)	128k	15	$0.75	175.5	0.14
Codestral-Mamba	256k	14	$0.25	94.6	0.45
Mistral 7B	8k	10	$0.25	113.0	0.29
Llama 3.2 1B	128k	10	$0.03	181.6	0.45
Llama 2 Chat 7B	4k	8	$0.10	132.5	0.45
o1-preview	128k		$26.25	152.8	20.52
GPT-4o (Aug '24)	128k		$4.38	110.7	0.50
GPT-4o mini Realtime (Dec '24)	128k		$0.00
GPT-4o Realtime (Dec '24)	128k		$0.00
o1-pro	200k		$262.50
GPT-4.5 (Preview)	128k		$93.75	61.0	1.12
Llama 3.2 11B (Vision)	128k		$0.16	84.9	0.37
Gemma 2 27B	8k		$0.26
Gemini 2.5 Flash Preview	1m		$0.26	338.4	7.07
Gemma 3 1B	32k		$0.00
Gemini 2.0 Flash Thinking exp. (Dec '24)	2m		$0.00
Claude 3.5 Sonnet (June)	200k		$6.00	79.4	0.73
Claude 3 Haiku	200k		$0.50	133.9	0.53
Mistral Saba	32k		$0.30	93.0	0.32
DeepSeek-V2.5 (Dec '24)	128k		$0.17
DeepSeek-Coder-V2	128k		$0.17
DeepSeek LLM 67B (V1)	4k		$0.00
DeepSeek Coder V2 Lite	128k		$0.09	112.7	0.58
Sonar Reasoning	127k		$2.00	79.7	1.65
Sonar Reasoning Pro	127k		$0.00
R1 1776	128k		$3.50
Grok 3 mini Reasoning (low)	1m		$0.35	107.6	0.34
Grok 3 Reasoning Beta	1m		$0.00
Solar Mini	4k		$0.15	49.1	1.10
Reka Flash	128k		$0.35	46.2	0.97
Reka Core	128k		$2.00	27.3	0.94
Reka Flash (Feb '24)	128k		$0.35	45.9	0.96
Reka Edge	128k		$0.10	85.1	0.90
Arctic	4k		$0.00
Qwen Chat 72B	34k		$1.00
Qwen1.5 Chat 110B	32k		$0.00	23.7	1.60
GPT-4 Turbo	128k		$15.00	43.9	0.72
GPT-4o (ChatGPT)	128k		$7.50
GPT-4	8k		$37.50	27.4	0.70
Llama 2 Chat 70B	4k		$0.00
Gemini 2.0 Flash-Lite (Preview)	1m		$0.13	208.8	0.29
Claude 2.0	100k		$12.00	30.9	0.86
DeepSeek-V2	128k		$0.17
DeepSeek-V2.5	128k		$0.17
OpenChat 3.5	8k		$0.06	48.2	0.26
Jamba Instruct	256k		$0.55	166.8	0.33

Key definitions

Back to Navigation

Context window: Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).

Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).

Latency (Time to First Token): Time to first token received, in seconds, after API request sent. For reasoning models which share reasoning tokens, this will be the first reasoning token. For models which do not support streaming, this represents time to receive the completion.

Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Output Price: Price per token generated by the model (received from the API), represented as USD per million Tokens.

Input Price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.

Time period: Metrics are 'live' and are based on the past 72 hours of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.

Models compared: OpenAI: GPT 4o Audio, GPT 4o Realtime, GPT 4o Speech Pipeline, GPT-3.5 Turbo, GPT-3.5 Turbo (0125), GPT-3.5 Turbo (0314), GPT-3.5 Turbo (1106), GPT-3.5 Turbo Instruct, GPT-4, GPT-4 Turbo, GPT-4 Turbo (0125), GPT-4 Turbo (1106), GPT-4 Vision, GPT-4.1, GPT-4.1 mini, GPT-4.1 nano, GPT-4.5 (Preview), GPT-4o (Aug '24), GPT-4o (ChatGPT), GPT-4o (March 2025), GPT-4o (May '24), GPT-4o (Nov '24), GPT-4o Realtime (Dec '24), GPT-4o mini, GPT-4o mini Realtime (Dec '24), o1, o1-mini, o1-preview, o1-pro, o3, o3-mini, o3-mini (high), and o4-mini (high), Meta: Code Llama 70B, Llama 2 Chat 13B, Llama 2 Chat 70B, Llama 2 Chat 7B, Llama 3 70B, Llama 3 8B, Llama 3.1 405B, Llama 3.1 70B, Llama 3.1 8B, Llama 3.2 11B (Vision), Llama 3.2 1B, Llama 3.2 3B, Llama 3.2 90B (Vision), Llama 3.3 70B, Llama 4 Behemoth, Llama 4 Maverick, and Llama 4 Scout, Google: Gemini 1.0 Pro, Gemini 1.0 Ultra, Gemini 1.5 Flash (May), Gemini 1.5 Flash (Sep), Gemini 1.5 Flash-8B, Gemini 1.5 Pro (May), Gemini 1.5 Pro (Sep), Gemini 2.0 Flash, Gemini 2.0 Flash (exp), Gemini 2.0 Flash Thinking exp. (Dec '24), Gemini 2.0 Flash Thinking exp. (Jan '25), Gemini 2.0 Flash-Lite (Feb '25), Gemini 2.0 Flash-Lite (Preview), Gemini 2.0 Pro Experimental, Gemini 2.5 Flash Preview, Gemini 2.5 Flash Preview (Reasoning), Gemini 2.5 Pro Preview, Gemini Experimental (Nov), Gemma 2 27B, Gemma 2 9B, Gemma 3 12B, Gemma 3 1B, Gemma 3 27B, Gemma 3 4B, and Gemma 7B, Anthropic: Claude 2.0, Claude 2.1, Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet, Claude 3.5 Haiku, Claude 3.5 Sonnet (June), Claude 3.5 Sonnet (Oct), Claude 3.7 Sonnet Thinking, Claude 3.7 Sonnet, and Claude Instant, Mistral: Codestral (Jan '25), Codestral (May '24), Codestral-Mamba, Ministral 3B, Ministral 8B, Mistral 7B, Mistral Large (Feb '24), Mistral Large 2 (Jul '24), Mistral Large 2 (Nov '24), Mistral Medium, Mistral NeMo, Mistral Saba, Mistral Small (Feb '24), Mistral Small (Sep '24), Mistral Small 3, Mistral Small 3.1, Mixtral 8x22B, Mixtral 8x7B, Pixtral 12B, and Pixtral Large, DeepSeek: DeepSeek Coder V2 Lite, DeepSeek LLM 67B (V1), DeepSeek R1, DeepSeek R1 (FP4), DeepSeek R1 Distill Llama 70B, DeepSeek R1 Distill Llama 8B, DeepSeek R1 Distill Qwen 1.5B, DeepSeek R1 Distill Qwen 14B, DeepSeek R1 Distill Qwen 32B, DeepSeek V3 (Dec '24), DeepSeek V3 (Mar' 25), DeepSeek-Coder-V2, DeepSeek-V2, DeepSeek-V2.5, DeepSeek-V2.5 (Dec '24), DeepSeek-VL2, and Janus Pro 7B, Perplexity: PPLX-70B Online, PPLX-7B-Online, R1 1776, Sonar, Sonar 3.1 Huge, Sonar 3.1 Large, Sonar 3.1 Small , Sonar Large, Sonar Pro, Sonar Reasoning, Sonar Reasoning Pro, and Sonar Small, xAI: Grok 2, Grok 3, Grok 3 Reasoning Beta, Grok 3 mini, Grok 3 mini Reasoning (low), Grok 3 mini Reasoning (high), Grok Beta, and Grok-1, OpenChat: OpenChat 3.5, Amazon: Nova Lite, Nova Micro, and Nova Pro, Microsoft Azure: Phi-3 Medium 14B, Phi-3 Mini, Phi-4, Phi-4 Mini, and Phi-4 Multimodal, Liquid AI: LFM 1.3B, LFM 3B, and LFM 40B, Upstage: Solar Mini, Solar Pro, and Solar Pro (Nov '24), Databricks: DBRX, MiniMax: MiniMax-Text-01, NVIDIA: Cosmos Nemotron 34B, Llama 3.1 Nemotron 70B, Llama 3.1 Nemotron Nano 8B, Llama 3.3 Nemotron Nano 8B v1 (Reasoning), Llama 3.1 Nemotron Ultra 253B Reasoning, Llama 3.3 Nemotron Super 49B v1, and Llama 3.3 Nemotron Super 49B v1 (Reasoning), IBM: Granite 3.0 2B, OpenVoice: Granite 3.0 8B, Inceptionlabs: Mercury Coder Mini, Mercury Coder Small, and Mercury Instruct, Reka AI: Reka Core, Reka Edge, Reka Flash (Feb '24), Reka Flash, and Reka Flash 3, Other: LLaVA-v1.5-7B, Cohere: Aya Expanse 32B, Aya Expanse 8B, Command, Command A, Command Light, Command R7B, Command-R, Command-R (Mar '24), Command-R+ (Apr '24), and Command-R+, Bytedance: Skylark Lite and Skylark Pro, AI21 Labs: Jamba 1.5 Large, Jamba 1.5 Large (Feb '25), Jamba 1.5 Mini, Jamba 1.5 Mini (Feb 2025), Jamba 1.6 Large, Jamba 1.6 Mini, and Jamba Instruct, Snowflake: Arctic and Snowflake Llama 3.3 70B, Alibaba: QwQ-32B, QwQ 32B-Preview, Qwen Chat 72B, Qwen Plus, Qwen Turbo, Qwen1.5 Chat 110B, Qwen1.5 Chat 14B, Qwen1.5 Chat 32B, Qwen1.5 Chat 72B, Qwen1.5 Chat 7B, Qwen2 72B, Qwen2 Instruct 7B, Qwen2 Instruct A14B 57B, Qwen2-VL 72B, Qwen2.5 Coder 32B, Qwen2.5 Coder 7B , Qwen2.5 Instruct 14B, Qwen2.5 Instruct 32B, Qwen2.5 72B, Qwen2.5 Instruct 7B, Qwen2.5 Max, Qwen2.5 Max 01-29, Qwen2.5 Omni 7B, Qwen2.5 VL 72B, and Qwen2.5 VL 7B, and 01.AI: Yi-Large.