LLM Leaderboard - Comparison of GPT-4o, Llama 3, Mistral, Gemini and over 30 models

Comparison and ranking the performance of over 30 AI models (LLMs) across key metrics including quality, price, performance and speed (output speed - tokens per second & latency - TTFT), context window & others. For more details including relating to our methodology, see our FAQs.

For comparison of API Providers hosting the models see

HIGHLIGHTS

Quality:o1-preview logo o1-preview and o1-mini logo o1-mini are the highest quality models, followed by Gemini 1.5 Pro (Sep '24) logo Gemini 1.5 Pro (Sep '24) & GPT-4o logo GPT-4o.Output Speed (tokens/s):Llama 3.2 1B logo Llama 3.2 1B (556 t/s) and Gemini 1.5 Flash (May '24) logo Gemini 1.5 Flash (May '24) (306 t/s) are the fastest models, followed by Gemini 1.5 Flash-8B logo Gemini 1.5 Flash-8B & Gemini 1.5 Flash (Sep '24) logo Gemini 1.5 Flash (Sep '24).Latency (seconds):Gemma 2 9B logo Gemma 2 9B (0.29s) and  Sonar 3.1 Small  logo Sonar 3.1 Small (0.31s) are the lowest latency models, followed by OpenChat 3.5 logo OpenChat 3.5 & Sonar 3.1 Large logo Sonar 3.1 Large.Price ($ per M tokens):Llama 3.2 1B logo Llama 3.2 1B ($0.05) and OpenChat 3.5 logo OpenChat 3.5 ($0.06) are the cheapest models, followed by Gemini 1.5 Flash-8B logo Gemini 1.5 Flash-8B & Llama 3.2 3B logo Llama 3.2 3B.Context Window:Gemini 1.5 Pro (Sep '24) logo Gemini 1.5 Pro (Sep '24) (2m) and Gemini 1.5 Pro (May '24) logo Gemini 1.5 Pro (May '24) (2m) are the largest context window models, followed by Gemini 1.5 Flash-8B logo Gemini 1.5 Flash-8B & Gemini 1.5 Flash (Sep '24) logo Gemini 1.5 Flash (Sep '24).
Features
Quality
Price
Output tokens/s
Latency
Further
Analysis
o1-preview
OpenAI logo
128k
85
$26.25
30.8
32.56
o1-mini
OpenAI logo
128k
82
$5.25
69.2
14.75
GPT-4o
OpenAI logo
128k
77
$4.38
118.4
0.44
GPT-4o (May '24)
OpenAI logo
128k
77
$7.50
111.0
0.43
GPT-4o mini
OpenAI logo
128k
71
$0.26
97.7
0.46
Llama 3.1 405B
Meta logo
128k
72
$4.50
22.2
0.88
Llama 3.2 90B (Vision)
Meta logo
128k
66
$0.90
39.8
0.51
Llama 3.1 70B
Meta logo
128k
65
$0.89
52.1
0.54
Llama 3.2 11B (Vision)
Meta logo
128k
54
$0.19
117.3
0.35
Llama 3.1 8B
Meta logo
128k
53
$0.15
163.2
0.38
Llama 3.2 3B
Meta logo
128k
47
$0.08
151.2
0.35
Llama 3.2 1B
Meta logo
128k
27
$0.05
555.7
0.35
Gemini 1.5 Pro (Sep '24)
Google logo
2m
80
$2.19
60.3
0.77
Gemini 1.5 Flash-8B
Google logo
1m
$0.07
284.5
0.35
Gemini 1.5 Flash (Sep '24)
Google logo
1m
73
$0.13
209.5
0.37
Gemma 2 27B
Google logo
8k
61
$0.80
68.8
0.34
Gemma 2 9B
Google logo
8k
46
$0.20
119.3
0.29
Gemini 1.5 Flash (May '24)
Google logo
1m
$0.13
306.4
0.33
Gemini 1.5 Pro (May '24)
Google logo
2m
$5.25
65.1
0.77
Claude 3.5 Sonnet
Anthropic logo
200k
77
$6.00
57.8
0.88
Claude 3 Opus
Anthropic logo
200k
70
$30.00
26.5
2.11
Claude 3 Haiku
Anthropic logo
200k
54
$0.50
132.1
0.47
Mistral Large 2
Mistral logo
128k
73
$4.50
36.1
0.49
Mixtral 8x22B
Mistral logo
65k
61
$1.20
68.7
0.40
Mistral Small (Sep '24)
Mistral logo
128k
60
$0.30
61.8
0.48
Pixtral 12B
Mistral logo
128k
56
$0.13
79.8
0.51
Mistral NeMo
Mistral logo
128k
52
$0.15
109.3
0.36
Mixtral 8x7B
Mistral logo
33k
42
$0.50
90.3
0.36
Codestral-Mamba
Mistral logo
256k
36
$0.25
94.8
0.54
Command-R+
Cohere logo
128k
56
$5.19
47.0
0.51
Command-R
Cohere logo
128k
51
$0.51
105.9
0.34
Command-R+ (Apr '24)
Cohere logo
128k
46
$6.00
45.5
0.56
Command-R (Mar '24)
Cohere logo
128k
36
$0.75
106.4
0.37
Sonar 3.1 Small
Perplexity logo
131k
$0.20
139.3
0.31
Sonar 3.1 Large
Perplexity logo
131k
$1.00
56.1
0.33
Phi-3 Medium 14B
Microsoft Azure logo
128k
$0.30
51.2
0.44
Solar Mini
Upstage logo
4k
48
$0.15
89.0
1.07
Solar Pro
Upstage logo
4k
61
$0.25
51.4
1.19
DBRX
Databricks logo
33k
49
$1.16
85.8
0.46
Reka Core
Reka AI logo
128k
57
$3.00
14.8
1.14
Reka Flash
Reka AI logo
128k
46
$0.35
30.0
1.20
Reka Edge
Reka AI logo
64k
30
$0.10
35.2
0.91
Jamba 1.5 Large
AI21 Labs logo
256k
64
$3.50
51.3
0.71
Jamba 1.5 Mini
AI21 Labs logo
256k
46
$0.25
82.7
0.51
DeepSeek-Coder-V2
DeepSeek logo
128k
67
$0.17
16.6
1.16
DeepSeek-V2
DeepSeek logo
128k
66
$0.17
16.8
1.17
DeepSeek-V2.5
DeepSeek logo
128k
66
$1.09
14.5
1.06
Qwen2.5 72B
Alibaba logo
131k
75
$0.38
35.0
0.52
Qwen2 72B
Alibaba logo
128k
69
$0.90
56.1
0.42
Yi-Large
01.AI logo
32k
58
$3.00
63.3
0.45
GPT-4 Turbo
OpenAI logo
128k
74
$15.00
35.9
0.64
GPT-3.5 Turbo
OpenAI logo
16k
52
$0.75
87.7
0.43
GPT-3.5 Turbo Instruct
OpenAI logo
4k
$1.63
114.4
0.58
GPT-4
OpenAI logo
8k
$37.50
26.9
0.61
Llama 3 70B
Meta logo
8k
62
$0.90
52.4
0.49
Llama 3 8B
Meta logo
8k
46
$0.15
102.1
0.40
Llama 2 Chat 70B
Meta logo
4k
39
$1.39
48.4
0.45
Llama 2 Chat 13B
Meta logo
4k
36
$0.30
53.5
0.39
Llama 2 Chat 7B
Meta logo
4k
$0.33
125.8
0.54
Gemini 1.0 Pro
Google logo
33k
$0.75
95.9
1.20
Claude 3 Sonnet
Anthropic logo
200k
57
$6.00
63.0
0.77
Mistral Large
Mistral logo
33k
56
$6.00
36.4
0.57
Mistral Small (Feb '24)
Mistral logo
33k
50
$1.50
53.5
0.45
Mistral 7B
Mistral logo
33k
24
$0.16
106.3
0.35
Codestral
Mistral logo
33k
$0.30
46.8
0.48
Mistral Medium
Mistral logo
33k
$4.09
38.0
0.76
OpenChat 3.5
OpenChat logo
8k
43
$0.06
76.6
0.32
Jamba Instruct
AI21 Labs logo
256k
28
$0.55
74.3
0.58

Key definitions

Artificial Analysis Quality Index: Average result across our evaluations covering different dimensions of model intelligence. Currently includes MMLU, GPQA, Math & HumanEval. OpenAI o1 model figures are preliminary and are based on figures stated by OpenAI. See methodology for more details.
Context window: Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).
Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).
Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.
Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).
Output price: Price per token generated by the model (received from the API), represented as USD per million Tokens.
Input price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.
Time period: Metrics are 'live' and are based on the past 14 days of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.

Models compared: OpenAI: GPT-3.5 Turbo, GPT-3.5 Turbo (0125), GPT-3.5 Turbo (1106), GPT-3.5 Turbo Instruct, GPT-4, GPT-4 Turbo, GPT-4 Turbo (0125), GPT-4 Vision, GPT-4o, GPT-4o (May '24), GPT-4o mini, o1-mini, and o1-preview, Meta: Code Llama 70B, Llama 2 Chat 13B, Llama 2 Chat 70B, Llama 2 Chat 7B, Llama 3 70B, Llama 3 8B, Llama 3.1 405B, Llama 3.1 70B, Llama 3.1 8B, Llama 3.2 11B (Vision), Llama 3.2 1B, Llama 3.2 3B, and Llama 3.2 90B (Vision), Google: Gemini 1.0 Pro, Gemini 1.5 Flash (May '24), Gemini 1.5 Flash (Sep '24), Gemini 1.5 Flash-8B, Gemini 1.5 Pro (May '24), Gemini 1.5 Pro (Sep '24), Gemma 2 27B, Gemma 2 9B, and Gemma 7B, Anthropic: Claude 2.0, Claude 2.1, Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet, Claude 3.5 Sonnet, and Claude Instant, Mistral: Codestral, Codestral-Mamba, Mistral 7B, Mistral Large, Mistral Large 2, Mistral Medium, Mistral NeMo, Mistral Small (Feb '24), Mistral Small (Sep '24), Mixtral 8x22B, Mixtral 8x7B, and Pixtral 12B, Cohere: Command, Command Light, Command-R, Command-R (Mar '24), Command-R+ (Apr '24), and Command-R+, Perplexity: PPLX-70B Online, PPLX-7B-Online, Sonar 3.1 Large, Sonar 3.1 Small , Sonar Large, and Sonar Small, xAI: Grok-1, OpenChat: OpenChat 3.5, Microsoft Azure: Phi-3 Medium 14B and Phi-3 Mini, Upstage: Solar Mini and Solar Pro, Databricks: DBRX, Reka AI: Reka Core, Reka Edge, and Reka Flash, Other: LLaVA-v1.5-7B, AI21 Labs: Jamba 1.5 Large, Jamba 1.5 Mini, and Jamba Instruct, DeepSeek: DeepSeek-Coder-V2, DeepSeek-V2, and DeepSeek-V2.5, Snowflake: Arctic, Alibaba: Qwen2 72B and Qwen2.5 72B, and 01.AI: Yi-Large.