LLM API Providers Leaderboard - Comparison of over 100 LLM endpoints
Comparison and ranking of API provider performance for over 100 AI LLM Model endpoints across performance key metrics including price, output speed, latency, context window & others. For more details including relating to our methodology, see our FAQs.
API providers compared: OpenAI, Playground AI, Mistral, Microsoft Azure, Ideogram, Amazon Bedrock, Groq, Together.ai, Anthropic, Perplexity, Google, Fireworks, Cerebras, Cohere, Lepton AI, Speechmatics, Deepinfra, Replicate, Runpod, Rev AI, fal.ai, AssemblyAI, DeepSeek, Reka AI, Deepgram, Gladia, Stability.ai, Baseten, Midjourney, Databricks, ElevenLabs, IBM, SambaNova, OctoAI, Cartesia, LMNT, 01.AI, and AI21 Labs.
Context | Model Quality | Price | Output tokens/s | Latency | |||
---|---|---|---|---|---|---|---|
Further Analysis | |||||||
o1-preview | 128k | $26.25 | 23.5 | 43.63 | |||
o1-mini | 128k | $5.25 | 73.9 | 13.98 | |||
GPT-4o (Aug 6) | 128k | 100 | $4.38 | 92.6 | 0.40 | ||
GPT-4o | 128k | 100 | $7.50 | 110.0 | 0.36 | ||
GPT-4o | 128k | 100 | $7.50 | 108.1 | 0.35 | ||
GPT-4o mini | 128k | 88 | $0.26 | 140.1 | 0.39 | ||
Llama 3.1 405B | 128k | 100 | $9.50 | 18.4 | 0.27 | ||
Llama 3.1 405B | 128k | 100 | $7.99 | 13.3 | 1.83 | ||
Llama 3.1 405B | 128k | 100 | $4.50 | 30.0 | 0.47 | ||
Llama 3.1 405B | 128k | 100 | $2.80 | 27.7 | 1.00 | ||
Llama 3.1 405B | 128k | 100 | $8.00 | 14.2 | 0.61 | ||
Llama 3.1 405B | 128k | 100 | $3.00 | 72.7 | 0.66 | ||
Llama 3.1 405B | 33k | 100 | $1.79 | 22.7 | 0.46 | ||
Llama 3.1 405B | 8k | 100 | $6.25 | 129.3 | 1.46 | ||
Llama 3.1 405B | 128k | 100 | $7.50 | 27.6 | 0.66 | ||
Llama 3.1 405B Turbo | 8k | 100 | $5.00 | 88.0 | 0.58 | ||
Llama 3.1 70B | 8k | 95 | $0.60 | 481.5 | 0.25 | ||
Llama 3.1 70B | 128k | 95 | $0.99 | 31.8 | 0.72 | ||
Llama 3.1 70B | 128k | 95 | $0.90 | 53.6 | 0.28 | ||
Llama 3.1 70B | 128k | 95 | $0.80 | 55.8 | 0.63 | ||
Llama 3.1 70B | 128k | 95 | $2.90 | 28.5 | 0.56 | ||
Llama 3.1 70B | 128k | 95 | $0.90 | 73.1 | 0.41 | ||
Llama 3.1 70B | 128k | 95 | $0.36 | 27.1 | 0.30 | ||
Llama 3.1 70B | 128k | 95 | $0.64 | 249.5 | 0.45 | ||
Llama 3.1 70B | 8k | 95 | $0.75 | 424.8 | 0.84 | ||
Llama 3.1 70B | 128k | 95 | $1.50 | 45.4 | 0.58 | ||
Llama 3.1 70B | 128k | 95 | $1.00 | 52.2 | 0.23 | ||
Llama 3.1 70B Turbo | 128k | 95 | $0.88 | 84.8 | 0.47 | ||
Llama 3.1 8B | 8k | 66 | $0.10 | 1,929.1 | 0.28 | ||
Llama 3.1 8B | 128k | 66 | $0.22 | 92.6 | 0.41 | ||
Llama 3.1 8B | 128k | 66 | $0.15 | 165.1 | 0.24 | ||
Llama 3.1 8B | 128k | 66 | $0.07 | 204.3 | 0.43 | ||
Llama 3.1 8B | 128k | 66 | $0.38 | 71.4 | 0.40 | ||
Llama 3.1 8B | 128k | 66 | $0.20 | 288.9 | 0.27 | ||
Llama 3.1 8B | 128k | 66 | $0.06 | 86.3 | 0.21 | ||
Llama 3.1 8B | 128k | 66 | $0.06 | 751.2 | 0.38 | ||
Llama 3.1 8B | 8k | 66 | $0.13 | 1,095.2 | 0.44 | ||
Llama 3.1 8B | 128k | 66 | $0.20 | 161.4 | 0.17 | ||
Llama 3.1 8B Turbo | 128k | 66 | $0.18 | 200.0 | 0.39 | ||
Gemini 1.5 Pro (Vertex) | 2m | 95 | $5.25 | ||||
Gemini 1.5 Pro (AI Studio) | 2m | 95 | $5.25 | 63.8 | 0.89 | ||
Gemini 1.5 Flash (Vertex) | 1m | 84 | $0.13 | ||||
Gemini 1.5 Flash (AI Studio) | 1m | 84 | $0.13 | 209.8 | 0.40 | ||
Gemma 2 27B | 8k | 78 | $0.80 | 70.8 | 0.34 | ||
Gemma 2 9B | 8k | 71 | $0.20 | 133.7 | 0.32 | ||
Gemma 2 9B | 8k | 71 | $0.06 | 67.3 | 0.32 | ||
Gemma 2 9B | 8k | 71 | $0.20 | 673.3 | 0.20 | ||
Gemma 2 9B | 8k | 71 | $0.30 | 120.0 | 0.40 | ||
Claude 3.5 Sonnet | 200k | 98 | $6.00 | 52.4 | 0.97 | ||
Claude 3.5 Sonnet | 200k | 98 | $6.00 | 94.9 | 1.02 | ||
Claude 3 Opus | 200k | 93 | $30.00 | 23.3 | 1.71 | ||
Claude 3 Opus | 200k | 93 | $30.00 | 27.3 | 1.85 | ||
Claude 3 Haiku | 200k | 74 | $0.50 | 118.1 | 0.44 | ||
Claude 3 Haiku | 200k | 74 | $0.50 | 157.7 | 0.58 | ||
Mistral Large 2 | 128k | 91 | $4.50 | 39.4 | 0.51 | ||
Mistral Large 2 | 128k | 91 | $4.50 | 42.1 | 0.43 | ||
Mixtral 8x22B | 65k | 71 | $3.00 | 66.3 | 0.47 | ||
Mixtral 8x22B | 65k | 71 | $1.20 | 39.2 | 0.32 | ||
Mixtral 8x22B | 65k | 71 | $1.20 | 50.6 | 0.38 | ||
Mixtral 8x22B | 65k | 71 | $0.65 | 55.0 | 0.26 | ||
Mixtral 8x22B | 65k | 71 | $1.20 | 72.1 | 0.39 | ||
Mistral NeMo | 128k | 64 | $0.30 | 136.9 | 0.39 | ||
Mistral NeMo | 128k | 64 | $0.20 | 158.5 | 0.31 | ||
Mistral NeMo | 128k | 64 | $0.13 | 63.4 | 0.23 | ||
Mistral Small | 33k | 71 | $1.50 | 49.8 | 0.42 | ||
Mistral Small | 33k | 71 | $1.50 | 51.7 | 0.39 | ||
Mixtral 8x7B | 33k | 61 | $0.70 | 87.0 | 0.41 | ||
Mixtral 8x7B | 33k | 61 | $0.47 | 43.9 | 0.27 | ||
Mixtral 8x7B | 33k | 61 | $0.51 | 60.8 | 0.40 | ||
Mixtral 8x7B | 33k | 61 | $0.45 | 77.9 | 0.29 | ||
Mixtral 8x7B | 33k | 61 | $0.50 | 36.9 | 0.76 | ||
Mixtral 8x7B | 33k | 61 | $0.50 | 94.9 | 0.33 | ||
Mixtral 8x7B | 33k | 61 | $0.24 | 46.2 | 0.31 | ||
Mixtral 8x7B | 33k | 61 | $0.24 | 543.0 | 0.22 | ||
Mixtral 8x7B | 33k | 61 | $0.63 | 86.6 | 0.50 | ||
Mixtral 8x7B | 33k | 61 | $0.60 | 109.3 | 0.36 | ||
Codestral-Mamba | 256k | $0.25 | 95.7 | 0.49 | |||
Command-R+ (08-2024) | 128k | $6.00 | 45.7 | 0.56 | |||
Command-R+ (08-2024) | 128k | $4.38 | 65.9 | 0.31 | |||
Command-R (08-2024) | 128k | $0.75 | 103.2 | 0.39 | |||
Command-R (08-2024) | 128k | $0.26 | 113.5 | 0.22 | |||
Command-R+ (04-2024) | 128k | 75 | $6.00 | 45.5 | 0.57 | ||
Command-R+ (04-2024) | 128k | 75 | $6.00 | 64.0 | 0.31 | ||
Command-R+ (04-2024) | 128k | 75 | $6.00 | 66.2 | 0.52 | ||
Command-R (03-2024) | 128k | 63 | $0.75 | 103.1 | 0.39 | ||
Command-R (03-2024) | 128k | 63 | $0.75 | 152.5 | 0.21 | ||
Command-R (03-2024) | 128k | 63 | $0.75 | 103.7 | 0.47 | ||
Sonar Large | 33k | $1.00 | 41.7 | 0.25 | |||
Sonar Small | 33k | $0.20 | 138.8 | 0.17 | |||
Sonar 3.1 Small | 131k | $0.20 | 138.0 | 0.17 | |||
Sonar 3.1 Large | 131k | $1.00 | 67.8 | 0.21 | |||
Phi-3 Medium 14B | 128k | $0.75 | 49.7 | 0.45 | |||
Phi-3 Medium 14B | 4k | $0.14 | 71.7 | 0.23 | |||
DBRX | 33k | 62 | $1.13 | 85.0 | 0.49 | ||
DBRX | 33k | 62 | $1.20 | 104.4 | 0.34 | ||
Reka Core | 128k | 90 | $6.00 | 14.3 | 1.13 | ||
Reka Flash | 128k | 78 | $1.10 | 29.8 | 0.96 | ||
Reka Edge | 64k | 60 | $0.55 | 34.1 | 0.97 | ||
Jamba 1.5 Large | 256k | 86 | $3.50 | 62.4 | 1.19 | ||
Jamba 1.5 Mini | 256k | 64 | $0.25 | 158.4 | 1.03 | ||
DeepSeek-Coder-V2 | 128k | $0.17 | 17.1 | 1.21 | |||
DeepSeek-V2.5 | 128k | $0.17 | 16.0 | 1.21 | |||
DeepSeek-V2 | 128k | 82 | $0.17 | 17.7 | 1.22 | ||
Qwen2 72B | 33k | 83 | $0.36 | 36.4 | 0.29 | ||
Qwen2 72B | 33k | 83 | $0.90 | 54.6 | 0.36 | ||
Yi-Large | 32k | 81 | $3.00 | 65.5 | 0.62 | ||
GPT-4 Turbo | 128k | 94 | $15.00 | 31.4 | 0.63 | ||
GPT-4 Turbo | 128k | 94 | $15.00 | 49.0 | 0.53 | ||
GPT-3.5 Turbo | 16k | 59 | $0.75 | 82.1 | 0.42 | ||
GPT-3.5 Turbo | 16k | 59 | $0.75 | 75.8 | 0.34 | ||
GPT-3.5 Turbo Instruct | 4k | 60 | $1.63 | 88.9 | 0.40 | ||
GPT-3.5 Turbo Instruct | 4k | 60 | $1.63 | 137.8 | 0.60 | ||
GPT-4 | 8k | 84 | $37.50 | 20.4 | 0.66 | ||
GPT-4 | 8k | 84 | $37.50 | 41.4 | 0.54 | ||
Llama 3 70B | 8k | 83 | $1.18 | 47.1 | 0.27 | ||
Llama 3 70B | 8k | 83 | $2.86 | 51.5 | 0.46 | ||
Llama 3 70B | 8k | 83 | $0.90 | 63.1 | 0.33 | ||
Llama 3 70B | 8k | 83 | $0.80 | 29.5 | 0.89 | ||
Llama 3 70B | 8k | 83 | $2.90 | 18.9 | 0.74 | ||
Llama 3 70B | 8k | 83 | $0.90 | 108.1 | 0.36 | ||
Llama 3 70B | 8k | 83 | $0.36 | 18.4 | 0.32 | ||
Llama 3 70B | 8k | 83 | $0.64 | 317.7 | 0.22 | ||
Llama 3 70B (Reference, FP16) | 8k | 83 | $0.90 | 154.2 | 0.41 | ||
Llama 3 70B (Turbo, FP8) | 8k | 83 | $0.88 | 89.5 | 0.49 | ||
Llama 3 8B | 8k | 64 | $0.10 | 75.8 | 0.27 | ||
Llama 3 8B | 8k | 64 | $0.38 | 78.2 | 0.34 | ||
Llama 3 8B | 8k | 64 | $0.07 | 71.6 | 5.31 | ||
Llama 3 8B | 8k | 64 | $0.38 | 73.0 | 0.41 | ||
Llama 3 8B | 8k | 64 | $0.20 | 114.0 | 0.36 | ||
Llama 3 8B | 8k | 64 | $0.06 | 111.3 | 0.20 | ||
Llama 3 8B | 8k | 64 | $0.06 | 1,201.3 | 0.31 | ||
Llama 3 8B | 8k | 64 | $0.20 | 300.5 | 0.42 | ||
Llama 2 Chat 70B | 4k | 57 | $1.18 | 46.7 | 0.31 | ||
Llama 2 Chat 70B | 4k | 57 | $2.10 | 35.6 | 0.47 | ||
Llama 2 Chat 70B | 4k | 57 | $0.90 | 174.1 | 0.23 | ||
Llama 2 Chat 70B | 4k | 57 | $1.60 | 17.2 | 0.95 | ||
Llama 2 Chat 13B | 4k | 39 | $0.81 | 52.6 | 0.40 | ||
Llama 2 Chat 13B | 4k | 39 | $0.20 | 172.8 | 0.23 | ||
Llama 2 Chat 13B | 4k | 39 | $0.30 | 52.5 | 0.48 | ||
Llama 2 Chat 7B | 4k | 29 | $0.10 | 122.6 | 0.27 | ||
Llama 2 Chat 7B | 4k | 29 | $0.56 | 68.6 | 0.54 | ||
Gemma 7B | 8k | 45 | $0.07 | 1,023.3 | 0.83 | ||
Gemini 1.0 Pro (AI Studio) | 33k | 62 | $0.75 | 99.8 | 1.15 | ||
Claude 3 Sonnet | 200k | 80 | $6.00 | 52.2 | 0.81 | ||
Claude 3 Sonnet | 200k | 80 | $6.00 | 59.6 | 0.94 | ||
Claude 2.1 | 200k | 55 | $12.00 | 27.7 | 1.82 | ||
Claude 2.1 | 200k | 55 | $12.00 | 30.9 | 1.09 | ||
Claude 2.0 | 100k | 70 | $12.00 | 31.2 | 1.10 | ||
Claude Instant | 100k | 63 | $1.20 | 61.7 | 0.59 | ||
Claude Instant | 100k | 63 | $1.20 | 100.9 | 0.54 | ||
Mistral Large | 33k | 76 | $6.00 | 40.4 | 0.51 | ||
Mistral Large | 33k | 76 | $6.00 | ||||
Mistral Large | 33k | 76 | $6.00 | 22.8 | 0.38 | ||
Mistral 7B | 33k | 40 | $0.25 | 99.4 | 0.41 | ||
Mistral 7B | 33k | 40 | $0.10 | 38.5 | 0.29 | ||
Mistral 7B | 33k | 40 | $0.16 | 75.8 | 0.36 | ||
Mistral 7B | 33k | 40 | $0.15 | 168.5 | 0.21 | ||
Mistral 7B | 33k | 40 | $0.07 | 95.6 | 1.11 | ||
Mistral 7B | 33k | 40 | $0.06 | 107.0 | 0.21 | ||
Mistral 7B | 16k | 40 | $0.20 | 125.6 | 0.19 | ||
Mistral 7B | 8k | 40 | $0.20 | 127.9 | 0.30 | ||
Codestral | 33k | $1.50 | 50.0 | 0.42 | |||
Mistral Medium | 33k | 70 | $4.09 | 38.4 | 0.75 | ||
Command | 4k | $1.63 | 21.8 | 0.56 | |||
Command | 4k | $1.25 | 22.3 | 0.39 | |||
Command Light | 4k | $0.38 | 32.8 | 0.60 | |||
Command Light | 4k | $0.38 | 59.4 | 0.25 | |||
OpenChat 3.5 | 8k | 50 | $0.06 | 70.6 | 0.32 | ||
Jamba Instruct | 256k | 63 | $0.55 | 80.7 | 0.91 | ||
Jamba Instruct | 256k | 63 | $0.55 | 61.6 | 0.32 |
Key definitions
Artificial Analysis Quality Index: Average result across our evaluations covering different dimensions of model intelligence. Currently includes MMLU, GPQA, Math & HumanEval. OpenAI o1 model figures are preliminary and are based on figures stated by OpenAI. See methodology for more details.
Context window: Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).
Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).
Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.
Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).
Output price: Price per token generated by the model (received from the API), represented as USD per million Tokens.
Input price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.
Time period: Metrics are 'live' and are based on the past 14 days of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.