Independent analysis of AI language models and API providers
Understand the AI landscape and choose the best model and API provider for your use-case
Highlights
Quality
Quality Index; Higher is better
Speed
Throughput in Tokens per Second; Higher is better
Price
USD per 1M Tokens; Lower is better
Quality Comparison by Ability
Varied metrics by ability categorization; Higher is better
General Ability (Chatbot Arena)
Reasoning & Knowledge (MMLU)
Reasoning & Knowledge (MT Bench)
Coding (HumanEval)
Median across providers: Figures represent median (P50) across all providers which support the model.
Quality vs. Throughput
Quality: General reasoning index, Throughput: Tokens per Second, Price: USD per 1M Tokens
Most attractive quadrant
Size represents Price (USD per M Tokens)
Quality: Index represents normalized average relative performance across Chatbot arena, MMLU & MT-Bench.
Throughput: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API).
Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).
Median across providers: Figures represent median (P50) across all providers which support the model.
Quality vs. Price
Quality: General reasoning index, Price: USD per 1M Tokens
Most attractive quadrant
Quality: Index represents normalized average relative performance across Chatbot arena, MMLU & MT-Bench.
Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).
Median across providers: Figures represent median (P50) across all providers which support the model.
Throughput
Output Tokens per Second; Higher is better
Throughput: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API).
Median across providers: Figures represent median (P50) across all providers which support the model.
Pricing: Input and Output Prices
USD per 1M Tokens
Input price
Output price
Input price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.
Output price: Price per token generated by the model (received from the API), represented as USD per million Tokens.
Median across providers: Figures represent median (P50) across all providers which support the model.
API Provider Highlights: Llama 3 Instruct (70B)
Throughput vs. Price: Llama 3 Instruct (70B)
Throughput: Tokens per Second, Price: USD per 1M Tokens
Most attractive quadrant
Microsoft Azure
Amazon Bedrock
Groq
Together.ai
Perplexity
Fireworks
Deepinfra
Replicate
OctoAI
Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).
Throughput: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API).
Median: Figures represent median (P50) measurement over the past 14 days.
Variance data is present on the model and API provider pages amongst the detailed performance metrics. See 'Compare Models' and 'Compare API Providers' in the navigation menu for further analysis.
Pricing (Input and Output Prices): Llama 3 Instruct (70B)
Price: USD per 1M Tokens; Lower is better
Input price
Output price
Input price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.
Output price: Price per token generated by the model (received from the API), represented as USD per million Tokens.
Throughput, Over Time: Llama 3 Instruct (70B)
Output Tokens per Second; Higher is better
Throughput: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API).
Over time measurement: Median measurement per day, based on 8 measurements each day at different times. Labels represent start of week's measurements.
See more information on any of our supported models
GPT-4
Llama 3 Instruct (70B)
Mistral Large
Gemini 1.5 Pro
Claude 3 Opus
Command-R+
DBRX Instruct
OpenChat 3.5 (1210)
PPLX-70B Online
Arctic Instruct
DeepSeek-V2-Chat