LLM Leaderboard - Comparison of GPT-4o, Llama 3, Mistral, Gemini and over 30 models
Comparison and ranking the performance of over 30 AI models (LLMs) across key metrics including quality, price, performance and speed (output speed - tokens per second & latency - TTFT), context window & others. For more details including relating to our methodology, see our FAQs.
HIGHLIGHTS
Features | Quality | Price | Output tokens/s | Latency | |||
---|---|---|---|---|---|---|---|
Further Analysis | |||||||
o1-preview | 128k | 86 | $26.25 | 142.7 | 25.24 | ||
o1-mini | 128k | 84 | $5.25 | 227.9 | 9.64 | ||
GPT-4o (Aug '24) | 128k | 78 | $4.38 | 82.4 | 0.66 | ||
GPT-4o (May '24) | 128k | 78 | $7.50 | 97.1 | 0.64 | ||
GPT-4o (Nov '24) | 128k | 73 | $4.38 | 112.7 | 0.38 | ||
GPT-4o mini | 128k | 73 | $0.26 | 114.8 | 0.68 | ||
Llama 3.3 70B | 128k | 74 | $0.75 | 171.0 | 0.38 | ||
Llama 3.1 405B | 128k | 74 | $3.50 | 28.9 | 0.72 | ||
Llama 3.1 70B | 128k | 68 | $0.75 | 72.6 | 0.43 | ||
Llama 3.2 90B (Vision) | 128k | 68 | $0.81 | 36.4 | 0.42 | ||
Llama 3.2 11B (Vision) | 128k | 54 | $0.18 | 123.6 | 0.29 | ||
Llama 3.1 8B | 128k | 54 | $0.10 | 172.6 | 0.34 | ||
Llama 3.2 3B | 128k | 49 | $0.08 | 199.4 | 0.32 | ||
Llama 3.2 1B | 128k | 26 | $0.05 | 552.5 | 0.33 | ||
Gemini 1.5 Pro (Sep) | 2m | 81 | $2.19 | 59.3 | 0.84 | ||
Gemini 1.5 Flash (Sep) | 1m | 72 | $0.13 | 188.3 | 0.35 | ||
Gemma 2 27B | 8k | 61 | $0.26 | 48.6 | 0.59 | ||
Gemma 2 9B | 8k | 55 | $0.12 | 165.5 | 0.43 | ||
Gemini Experimental (Nov) | 33k | $0.00 | 40.4 | 2.62 | |||
Gemini 1.5 Pro (May) | 2m | $2.19 | 66.0 | 0.74 | |||
Gemini 1.5 Flash (May) | 1m | $0.13 | 310.7 | 0.30 | |||
Gemini 1.5 Flash-8B | 1m | $0.07 | 282.4 | 0.36 | |||
Claude 3.5 Sonnet (Oct) | 200k | 80 | $6.00 | 55.5 | 0.95 | ||
Claude 3.5 Sonnet (June) | 200k | 76 | $6.00 | 54.5 | 0.91 | ||
Claude 3 Opus | 200k | 70 | $30.00 | 26.5 | 1.78 | ||
Claude 3.5 Haiku | 200k | 68 | $1.60 | 65.1 | 0.79 | ||
Claude 3 Haiku | 200k | 55 | $0.50 | 122.5 | 0.49 | ||
Pixtral Large | 128k | 74 | $3.00 | 37.2 | 0.81 | ||
Mistral Large 2 (Jul '24) | 128k | 74 | $3.00 | 34.1 | 0.50 | ||
Mistral Large 2 (Nov '24) | 128k | 74 | $3.00 | 45.0 | 0.69 | ||
Mistral Small (Sep '24) | 128k | 61 | $0.30 | 63.3 | 0.59 | ||
Mixtral 8x22B | 65k | 61 | $1.20 | 86.0 | 0.62 | ||
Pixtral 12B | 128k | 56 | $0.13 | 70.7 | 0.50 | ||
Ministral 8B | 128k | 56 | $0.10 | 132.3 | 0.54 | ||
Mistral NeMo | 128k | 54 | $0.09 | 115.3 | 0.54 | ||
Ministral 3B | 128k | 53 | $0.04 | 165.4 | 0.57 | ||
Mixtral 8x7B | 33k | 41 | $0.50 | 123.7 | 0.37 | ||
Codestral-Mamba | 256k | 33 | $0.25 | 96.0 | 0.69 | ||
Command-R+ | 128k | 55 | $5.19 | 50.1 | 0.48 | ||
Command-R+ (Apr '24) | 128k | 45 | $6.00 | 48.0 | 0.53 | ||
Command-R (Mar '24) | 128k | 36 | $0.75 | 107.6 | 0.36 | ||
Command-R | 128k | $0.51 | 111.7 | 0.33 | |||
Aya Expanse 32B | 128k | $0.75 | 120.9 | 0.22 | |||
Aya Expanse 8B | 8k | $0.75 | 143.9 | 0.29 | |||
Sonar 3.1 Large | 131k | $1.00 | 54.2 | 0.33 | |||
Sonar 3.1 Small | 131k | $0.20 | 147.5 | 0.33 | |||
Grok Beta | 8k | 72 | $7.50 | 61.5 | 0.46 | ||
Nova Pro | 300k | 75 | $1.40 | 92.7 | 0.42 | ||
Nova Lite | 300k | 70 | $0.10 | 147.7 | 0.38 | ||
Nova Micro | 130k | 66 | $0.06 | 197.0 | 0.36 | ||
Phi-3 Medium 14B | 128k | $0.30 | 50.1 | 0.43 | |||
Solar Mini | 4k | 47 | $0.15 | 85.4 | 1.10 | ||
DBRX | 33k | 46 | $1.16 | 82.7 | 0.47 | ||
Llama 3.1 Nemotron 70B | 128k | 72 | $0.27 | 47.3 | 0.60 | ||
Reka Flash | 128k | 59 | $0.35 | 32.8 | 1.27 | ||
Reka Core | 128k | 58 | $2.00 | 15.0 | 1.16 | ||
Reka Flash (Feb '24) | 128k | 46 | $0.35 | 31.2 | 0.92 | ||
Reka Edge | 64k | 31 | $0.10 | 35.1 | 0.92 | ||
Jamba 1.5 Large | 256k | $3.50 | 51.1 | 0.70 | |||
Jamba 1.5 Mini | 256k | $0.25 | 82.7 | 0.49 | |||
DeepSeek-Coder-V2 | 128k | 71 | $0.17 | 17.0 | 0.95 | ||
DeepSeek-V2 | 128k | $0.17 | 17.1 | 0.91 | |||
DeepSeek-V2.5 | 128k | $1.09 | 14.7 | 0.88 | |||
Qwen2.5 72B | 131k | 77 | $0.40 | 56.4 | 0.56 | ||
Qwen2.5 Coder 32B | 131k | 72 | $0.80 | 58.2 | 0.37 | ||
Qwen2 72B | 128k | 72 | $0.63 | 46.0 | 0.38 | ||
Yi-Large | 32k | 61 | $3.00 | 66.5 | 0.43 | ||
GPT-4 Turbo | 128k | 75 | $15.00 | 39.5 | 1.23 | ||
GPT-3.5 Turbo | 16k | 53 | $0.75 | 121.5 | 0.61 | ||
GPT-4 | 8k | $37.50 | 26.1 | 0.70 | |||
Llama 3 70B | 8k | 47 | $0.89 | 47.1 | 0.45 | ||
Llama 3 8B | 8k | 45 | $0.15 | 116.5 | 0.33 | ||
Llama 2 Chat 13B | 4k | $0.56 | 41.7 | 0.45 | |||
Llama 2 Chat 7B | 4k | $0.33 | 124.4 | 0.49 | |||
Gemini 1.0 Pro | 33k | $0.75 | 103.0 | 1.29 | |||
Claude 3 Sonnet | 200k | 57 | $6.00 | 63.0 | 0.75 | ||
Mistral Small (Feb '24) | 33k | 59 | $1.50 | 54.0 | 0.42 | ||
Mistral Large (Feb '24) | 33k | 56 | $6.00 | 38.0 | 0.50 | ||
Mistral 7B | 33k | 28 | $0.18 | 99.3 | 0.34 | ||
Codestral | 33k | $0.30 | 82.0 | 0.56 | |||
Mistral Medium | 33k | $4.09 | 43.9 | 0.63 | |||
OpenChat 3.5 | 8k | 44 | $0.06 | 77.3 | 0.34 | ||
Jamba Instruct | 256k | $0.55 | 75.4 | 0.53 |
Key definitions
Models compared: OpenAI: GPT 4o Audio, GPT 4o Realtime, GPT 4o Speech Pipeline, GPT-3.5 Turbo, GPT-3.5 Turbo (0125), GPT-3.5 Turbo (1106), GPT-3.5 Turbo Instruct, GPT-4, GPT-4 Turbo, GPT-4 Turbo (0125), GPT-4 Vision, GPT-4o (Aug '24), GPT-4o (May '24), GPT-4o (Nov '24), GPT-4o mini, o1-mini, and o1-preview, Meta: Code Llama 70B, Llama 2 Chat 13B, Llama 2 Chat 70B, Llama 2 Chat 7B, Llama 3 70B, Llama 3 8B, Llama 3.1 405B, Llama 3.1 70B, Llama 3.1 8B, Llama 3.2 11B (Vision), Llama 3.2 1B, Llama 3.2 3B, Llama 3.2 90B (Vision), and Llama 3.3 70B, Google: Gemini 1.0 Pro, Gemini 1.5 Flash (May), Gemini 1.5 Flash (Sep), Gemini 1.5 Flash-8B, Gemini 1.5 Pro (May), Gemini 1.5 Pro (Sep), Gemini 2.0 Flash (Experimental), Gemini Experimental (Nov), Gemma 2 27B, Gemma 2 9B, and Gemma 7B, Anthropic: Claude 2.0, Claude 2.1, Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet, Claude 3.5 Haiku, Claude 3.5 Sonnet (June), Claude 3.5 Sonnet (Oct), and Claude Instant, Mistral: Codestral, Codestral-Mamba, Ministral 3B, Ministral 8B, Mistral 7B, Mistral Large (Feb '24), Mistral Large 2 (Jul '24), Mistral Large 2 (Nov '24), Mistral Medium, Mistral NeMo, Mistral Small (Feb '24), Mistral Small (Sep '24), Mixtral 8x22B, Mixtral 8x7B, Pixtral 12B, and Pixtral Large, Cohere: Aya Expanse 32B, Aya Expanse 8B, Command, Command Light, Command-R, Command-R (Mar '24), Command-R+ (Apr '24), and Command-R+, Perplexity: PPLX-70B Online, PPLX-7B-Online, Sonar 3.1 Huge, Sonar 3.1 Large, Sonar 3.1 Small , Sonar Large, and Sonar Small, xAI: Grok Beta and Grok-1, OpenChat: OpenChat 3.5, Amazon: Nova Lite, Nova Micro, and Nova Pro, Microsoft Azure: Phi-3 Medium 14B and Phi-3 Mini, Upstage: Solar Mini, Solar Pro, and Solar Pro (Nov '24), Databricks: DBRX, NVIDIA: Llama 3.1 Nemotron 70B, IBM: Granite 3.0 2B, OpenVoice: Granite 3.0 8B, Reka AI: Reka Core, Reka Edge, Reka Flash (Feb '24), and Reka Flash, Other: LLaVA-v1.5-7B, AI21 Labs: Jamba 1.5 Large, Jamba 1.5 Mini, and Jamba Instruct, DeepSeek: DeepSeek-Coder-V2, DeepSeek-V2, and DeepSeek-V2.5, Snowflake: Arctic, Alibaba: QwQ 32B-Preview, Qwen2 72B, Qwen2.5 Coder 32B, and Qwen2.5 72B, and 01.AI: Yi-Large.