Llama 3.1 405B: API Provider Benchmarking & Analysis
Analysis of API providers for Llama 3.1 Instruct 405B across performance metrics including latency (time to first token), output speed (output tokens per second), price and others. API providers benchmarked include Microsoft Azure, Amazon Bedrock, Hyperbolic, Together.ai, Google, Fireworks, Lambda, Replicate, Nebius, Databricks, and SambaNova.
Comparison Summary
Output Speed (tokens/s):
SambaNova (141 t/s) and 
Together.ai Turbo (83 t/s) are the fastest providers of Llama 3.1 405B, followed by 
Hyperbolic (79 t/s), 
Amazon Latency Optimized (78 t/s) & 
Fireworks (76 t/s).Latency (TTFT):
Google Vertex (0.41s) and Amazon Latency Optimized (0.46s) have the lowest latency for Llama 3.1 405B, followed by 
Azure (0.47s), Fireworks (0.55s) & Together.ai Turbo (0.58s).Blended Price ($/M tokens):
Nebius Base ($1.50) and Amazon Standard ($2.40) are the most cost-effective providers for Llama 3.1 405B, followed by Amazon Latency Optimized ($3.00), Fireworks ($3.00) & Together.ai Turbo ($3.50).Input Token Price: Nebius Base ($1.00) and Amazon Standard ($2.40) offer the lowest input token prices for Llama 3.1 405B, followed by Amazon Latency Optimized ($3.00), Fireworks ($3.00) & Together.ai Turbo ($3.50).Output Token Price: Amazon Standard ($2.40) and Amazon Latency Optimized ($3.00) offer the lowest output token prices for Llama 3.1 405B, followed by Nebius Base ($3.00), Fireworks ($3.00) & Together.ai Turbo ($3.50).
SambaNova (141 t/s) and Together.ai Turbo (83 t/s) are the fastest providers of Llama 3.1 405B, followed by Hyperbolic (79 t/s), Amazon Latency Optimized (78 t/s) & Fireworks (76 t/s).Latency (TTFT): Google Vertex (0.41s) and Amazon Latency Optimized (0.46s) have the lowest latency for Llama 3.1 405B, followed by Azure (0.47s), Fireworks (0.55s) & Together.ai Turbo (0.58s).Blended Price ($/M tokens): Nebius Base ($1.50) and Amazon Standard ($2.40) are the most cost-effective providers for Llama 3.1 405B, followed by Amazon Latency Optimized ($3.00), Fireworks ($3.00) & Together.ai Turbo ($3.50).Input Token Price: Nebius Base ($1.00) and Amazon Standard ($2.40) offer the lowest input token prices for Llama 3.1 405B, followed by Amazon Latency Optimized ($3.00), Fireworks ($3.00) & Together.ai Turbo ($3.50).Output Token Price: Amazon Standard ($2.40) and Amazon Latency Optimized ($3.00) offer the lowest output token prices for Llama 3.1 405B, followed by Nebius Base ($3.00), Fireworks ($3.00) & Together.ai Turbo ($3.50).Highlights
Intelligence
Artificial Analysis Intelligence Index; Higher is better
Replicate
Databricks