AI Hardware Benchmarking & Performance Analysis

We measure real-world performance of AI accelerator systems during language model inference.

For language model intelligence benchmarks, or API performance benchmarks, see language model comparisons.

System Load Test (AA-SLT)

Original Benchmark

Our original hardware benchmark, covering a wide range of systems. Read the methodology

Highlights

Peak System Output Throughput, Llama 3.3 70B

Total system output tokens per second · Higher is better

Peak Output Speed per Query, Llama 3.3 70B

Output tokens per second per query · Higher is better

Rental Price (ON-DEMAND)

Minimum rental price per GPU per hour (USD) · Lower is better

Throughput

System Output Throughput at 100 tokens/s Per Query Output Speed

gpt-oss-120b (high) | System output throughput (tokens per second) at 100 tokens/s output speed

The total number of tokens that can be processed per second across all concurrent requests. This metric measures the overall system capacity and efficiency, taking into account both the per-query output speed and the system's ability to handle multiple concurrent requests.

  • Max Throughput: Optimized for the highest sustained request volume.
  • Minimum Latency: Tuned to deliver the fastest response times.
  • Optimal: Combines max throughput and minimum latency configurations by selecting the configuration with higher system throughput at a given concurrency level.

Output Speed

Peak Output Speed per Query

gpt-oss-120b (high) | Peak output speed per query (tokens per second)

Tokens per second received by each individual query after the first chunk has been received. Represented as the median result of all queries in each concurrency phase.

  • Max Throughput: Optimized for the highest sustained request volume.
  • Minimum Latency: Tuned to deliver the fastest response times.
  • Optimal: Combines max throughput and minimum latency configurations by selecting the configuration with higher system throughput at a given concurrency level.

Throughput vs Speed

System Output Throughput vs. Output Speed per Query

gpt-oss-120b (high) | System output throughput (tokens per second) vs output speed per query (tokens per second)
8xB200 - TensorRT-LLM - Optimal
8xH100 - vLLM
8xMI300X - vLLM
8xH200 - vLLM

The total number of tokens that can be processed per second across all concurrent requests. This metric measures the overall system capacity and efficiency, taking into account both the per-query output speed and the system's ability to handle multiple concurrent requests.

Tokens per second received by each individual query after the first chunk has been received. Represented as the median result of all queries in each concurrency phase.

  • Max Throughput: Optimized for the highest sustained request volume.
  • Minimum Latency: Tuned to deliver the fastest response times.
  • Optimal: Combines max throughput and minimum latency configurations by selecting the configuration with higher system throughput at a given concurrency level.

System Output Throughput & Output Speed per Query vs. Concurrency

gpt-oss-120b (high) | System output throughput (tokens per second) & output speed per query (tokens per second)
8xB200 - TensorRT-LLM - Optimal
8xH100 - vLLM
8xMI300X - vLLM
8xH200 - vLLM
Throughput
Speed

Tokens per second received by each individual query after the first chunk has been received. Represented as the median result of all queries in each concurrency phase.

The total number of tokens that can be processed per second across all concurrent requests. This metric measures the overall system capacity and efficiency, taking into account both the per-query output speed and the system's ability to handle multiple concurrent requests.

The number of simultaneous requests that can be processed by the API at any given time. Higher concurrency enables better handling of parallel requests, which is essential for applications requiring high throughput and scalability.

  • Max Throughput: Optimized for the highest sustained request volume.
  • Minimum Latency: Tuned to deliver the fastest response times.
  • Optimal: Combines max throughput and minimum latency configurations by selecting the configuration with higher system throughput at a given concurrency level.

Cost

Cost per Million Input and Output Tokens at 100 tokens/s Per Query Output Speed

gpt-oss-120b (high) | Cost per one million input and one million output tokens (USD) at 100 tokens/s output speed

The cost per million input and output tokens is calculated using the average price per GPU per hour and the system output throughput, assuming 1k input and 1k output tokens per request. The formula is:

average price per GPU per hour×number of GPUs×1,000,000system output throughput×3600\frac{\text{average price per GPU per hour} \times \text{number of GPUs} \times 1,000,000}{\text{system output throughput} \times 3600}

Note: Prices used are the lowest 1-year committed prices available on the market.

Prices shown are updated at the start of each month. While prices may fluctuate daily, we maintain a monthly update schedule to provide consistent comparisons.

  • Max Throughput: Optimized for the highest sustained request volume.
  • Minimum Latency: Tuned to deliver the fastest response times.
  • Optimal: Combines max throughput and minimum latency configurations by selecting the configuration with higher system throughput at a given concurrency level.

Concurrency

End-to-End Latency vs. Concurrency

gpt-oss-120b (high) | End-to-end latency (s) vs. concurrency
8xB200 - TensorRT-LLM - Optimal
8xH100 - vLLM
8xMI300X - vLLM
8xH200 - vLLM

The time it takes for a request to be processed and the response to be returned.

The number of simultaneous requests that can be processed by the API at any given time. Higher concurrency enables better handling of parallel requests, which is essential for applications requiring high throughput and scalability.

  • Max Throughput: Optimized for the highest sustained request volume.
  • Minimum Latency: Tuned to deliver the fastest response times.
  • Optimal: Combines max throughput and minimum latency configurations by selecting the configuration with higher system throughput at a given concurrency level.

Pricing

Price per GPU Hour (On-Demand)

Leading cloud hyperscaler endpoints · Price in USD
Runpod
Crusoe Cloud
Digitalocean
Nebius
Amazon Web Services
Google Cloud
Microsoft Azure
Coreweave
Lambda

Instance configurations may differ slightly by provider, even when using the same GPU model. Factors like memory size, interconnect bandwidth, and system architecture can impact pricing.

Prices are based on the US Central region, or the closest available equivalent if not directly listed (e.g., US East - Ohio - for AWS).

Pricing reflects on-demand and 1-year committed hourly rates as listed by each cloud provider.

Prices shown are updated at the start of each month. While prices may fluctuate daily, we maintain a monthly update schedule to provide consistent comparisons.

Frequently Asked Questions

For the current Artificial Analysis System Load Test (AA-SLT), NVIDIA's B200 is the most performant accelerator for LLM inference. It leads on peak throughput and output speed per query, though the right choice can still vary by model, deployment goal and budget.

NVIDIA's B200 currently powers the highest-throughput result in the current Artificial Analysis System Load Test (AA-SLT). The top benchmark is 8xB200 (SXM) serving gpt-oss-120b (high), reaching 92,909 output tokens per second at peak throughput.

NVIDIA's B200 currently powers the fastest single-query result in the current Artificial Analysis System Load Test (AA-SLT). The top benchmark is 8xB200 (SXM) serving gpt-oss-120b (high), reaching 403 output tokens per second per query.

8xB200 (SXM) for gpt-oss-120b (high) currently has the best cost efficiency in the current Artificial Analysis System Load Test (AA-SLT) at $0.19. Artificial Analysis compares systems using cost per one million input and one million output tokens at a model-specific reference speed, so the most cost-efficient hardware depends on both the model and the target output speed.

In the current Artificial Analysis System Load Test (AA-SLT), DeepSeek R1 0528 (May '25) works best on 8xB200 (SXM) with NVIDIA's B200, reaching 45,677 output tokens per second at peak throughput, Llama 4 Maverick works best on 8xB200 (SXM) with NVIDIA's B200, reaching 48,198 output tokens per second at peak throughput, and gpt-oss-120b (high) works best on 8xB200 (SXM) with NVIDIA's B200, reaching 92,909 output tokens per second at peak throughput.