Whisper: API Provider Benchmarking & Analysis
Analysis of Whisper API providers across performance metrics including Artificial Analysis Word Error Rate Index, speed, and price.
Highlights
Artificial Analysis Word Error Rate
% of words transcribed incorrectly; Lower is better
Speed Factor
Input audio seconds transcribed per second; Higher is better
Price
USD per 1000 minutes of audio; Lower is better
Summary Analysis
Artificial Analysis Word Error Rate (AA-WER) Index
% of words transcribed incorrectly, Lower is better
Note: Models that do not support transcription of audio longer than 10 minutes were evaluated on 9-minute chunks of the test set (applies to GPT-4o Transcribe; GPT-4o Mini Transcribe; Voxtral Mini; Voxtral Mini, Deepinfra; Gemini 2.5 Flash Lite). For models with even shorter time limits, all files are split into 30-second chunks (applies to Granite Speech 3.3 8B, IBM; Qwen3 ASR Flash).
Artificial Analysis Word Error Rate (AA-WER) Index: Measures transcription accuracy across 3 datasets to evaluate models in real-world speech with diverse accents, domain-specific language, and challenging channel & acoustic conditions.
AA-WER is calculated as an audio-duration-weighted average of WER across ~2 hours from three datasets: VoxPopuli, Earnings-22, and AMI-SDM. See methodology for more detail.
Artificial Analysis Word Error Rate (AA-WER) Index by Individual Dataset
% of words transcribed incorrectly, Lower is better
VoxPopuli (Parliamentary proceedings)
Earnings 22 Full (Corporate earnings calls)
AMI SDM (Meeting transcripts)
Note: Models that do not support transcription of audio longer than 10 minutes were evaluated on 9-minute chunks of the test set (applies to GPT-4o Transcribe; GPT-4o Mini Transcribe; Voxtral Mini; Voxtral Mini, Deepinfra; Gemini 2.5 Flash Lite). For models with even shorter time limits, all files are split into 30-second chunks (applies to Granite Speech 3.3 8B, IBM; Qwen3 ASR Flash).
Artificial Analysis Word Error Rate (AA-WER) Index: Measures transcription accuracy across 3 datasets to evaluate models in real-world speech with diverse accents, domain-specific language, and challenging channel & acoustic conditions.
AA-WER is calculated as an audio-duration-weighted average of WER across ~2 hours from three datasets: VoxPopuli, Earnings-22, and AMI-SDM. See methodology for more detail.
Artificial Analysis Word Error Rate Index vs. Price
% of words transcribed incorrectly, USD per 1000 minutes of audio
Most attractive quadrant
Size represents Input audio seconds transcribed per second
Amazon Transcribe
Chirp 2, Google
Gladia v2
Incredibly Fast , Replicate
L, v2, Azure
L, v2, Replicate
L, v3, Deepinfra
L, v3, fal.ai
L, v3, Fireworks
L, v3, Groq
L, v3, Replicate
L, v3, SambaNova
L, v3, Turbo, Fireworks
L, v3, Turbo, Groq
Large v2, OpenAI
Large v3, together.ai
Nova-3
Qwen3 ASR Flash
Rev AI
Scribe, ElevenLabs
Speechmatics Enhanced
Universal, AssemblyAI
Voxtral Small
X, Replicate
Artificial Analysis Word Error Rate (AA-WER) Index: Measures transcription accuracy across 3 datasets to evaluate models in real-world speech with diverse accents, domain-specific language, and challenging channel & acoustic conditions.
AA-WER is calculated as an audio-duration-weighted average of WER across ~2 hours from three datasets: VoxPopuli, Earnings-22, and AMI-SDM. See methodology for more detail.
Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
Artificial Analysis Word Error Rate Index vs. Speed Factor
% of words transcribed incorrectly, Input audio seconds transcribed per second
Most attractive quadrant
Size represents USD per 1000 minutes of audio
Amazon Transcribe
Chirp 2, Google
Gladia v2
Incredibly Fast , Replicate
L, v2, Azure
L, v2, Replicate
L, v3, Deepinfra
L, v3, fal.ai
L, v3, Fireworks
L, v3, Groq
L, v3, Replicate
L, v3, SambaNova
L, v3, Turbo, Fireworks
L, v3, Turbo, Groq
Large v2, OpenAI
Large v3, together.ai
Nova-3
Parakeet TDT 0.6B V2, NVIDIA
Scribe, ElevenLabs
Speechmatics Enhanced
Universal, AssemblyAI
Voxtral Small
X, Replicate
Artificial Analysis Word Error Rate (AA-WER) Index: Measures transcription accuracy across 3 datasets to evaluate models in real-world speech with diverse accents, domain-specific language, and challenging channel & acoustic conditions.
AA-WER is calculated as an audio-duration-weighted average of WER across ~2 hours from three datasets: VoxPopuli, Earnings-22, and AMI-SDM. See methodology for more detail.
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Speed Factor vs. Price
Input audio seconds transcribed per second, USD per 1000 minutes of audio
Most attractive quadrant
Amazon Transcribe
Chirp 2, Google
Gladia v2
Incredibly Fast , Replicate
L, v2, Azure
L, v2, Replicate
L, v3, Deepinfra
L, v3, fal.ai
L, v3, Fireworks
L, v3, Groq
L, v3, Replicate
L, v3, SambaNova
L, v3, Turbo, Fireworks
L, v3, Turbo, Groq
Large v2, OpenAI
Large v3, together.ai
Nova-3
Scribe, ElevenLabs
Speechmatics Enhanced
Universal, AssemblyAI
Voxtral Small
X, Replicate
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
Speed
Speed Factor
Input audio seconds transcribed per second, Higher is better
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Speed Factor, Variance
Input audio seconds transcribed per second, Results by percentile, Higher is better
Median; other points represent Min, 25th, 75th percentiles and Max respectively
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Boxplot: Shows variance of measurements
Speed Factor, Over Time
Input audio seconds transcribed per second, Higher is better
Large v2, OpenAI
L, v2, Azure
Speechmatics Enhanced
L, v3, fal.ai
Incredibly Fast , Replicate
L, v2, Replicate
L, v3, Replicate
X, Replicate
Gladia v2
M, Replicate
S, Replicate
L, v3, Groq
Distil-, Groq
L, v3, Deepinfra
L, v3, Turbo, Groq
L, v3, Fireworks
L, v3, Turbo, Fireworks
Universal, AssemblyAI
Amazon Transcribe
Rev AI
Nova-3
Chirp 2, Google
Scribe, ElevenLabs
L, v3, SambaNova
Large v3, together.ai
Voxtral Small
Parakeet TDT 0.6B V2, NVIDIA
Gemini 2.5 Pro
Qwen3 ASR Flash
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Over time measurement: Median measurement per day, based on 8 measurements each day at different times. Labels represent start of week's measurements.
Price
Price
USD per 1000 minutes of audio, Lower is better
Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
For providers which do not price based on audio duration and rather on processing time (incl. Replicate, fal), we have calculated an indicative per minute price based on processing time expected per minute of audio.Further detail present on methodology page.
Note: Groq chargers for a minimum of 10s per request.
Note: Groq chargers for a minimum of 10s per request.
Summary of key metrics & further information
| Provider | Model | Whisper version | Footnotes | Word Error Rate (%) | Median Speed Factor | Price (USD per 1000 minutes) | Further Details |
|---|---|---|---|---|---|---|---|
| OpenAI | Whisper Large v2 | large-v2 | 15.8% | 29.4 | 6.00 | ||
 Microsoft Azure | Whisper Large v2 | large-v2 | 27.2% | 34.0 | 6.00 | ||
 fal.ai | Whisper Large v3 | large-v3 | 16.8% | 312.5 | 0.50 | ||
 Replicate | Incredibly Fast Whisper | large-v3 | 18.2% | 61.0 | 1.49 | ||
| Replicate | Whisper Large v2 | large-v2 | 15.8% | 2.4 | 3.47 | ||
| Replicate | Whisper Large v3 | large-v3 | 24.6% | 3.1 | 4.23 | ||
| Replicate | WhisperX | large-v3 | 16.3% | 7.7 | 1.09 | ||
| Replicate | Whisper (M) | medium | 2.68 | ||||
| Replicate | Whisper (S) | small | 1.37 | ||||
 Groq | Whisper Large v3 | large-v3 | 16.8% | 212.3 | 1.85 | ||
| Groq | Distil-Whisper | 0.33 | |||||
| fal.ai | Whisper Large v3 | large-v3 | 16.8% | 142.0 | 1.15 | ||
 Deepinfra | Whisper Large v3 | large-v3 | 16.8% | 98.0 | 0.45 | ||
| Groq | Whisper Large v3 Turbo | v3 Turbo | 256.6 | 0.67 | |||
 Fireworks | Whisper Large v3 | large-v3 | 416.6 | 1.00 | |||
| Fireworks | Whisper Large v3 Turbo | v3 Turbo | 17.8% | 456.8 | 1.00 | ||
 SambaNova | Whisper-Large-v3 | large-v3 | 16.8% | 169.3 | 1.67 | ||
 Together.ai | Whisper Large v3 | large-v3 | 24.6% | 140.7 | 1.50 | ||
 Speechmatics | Speechmatics Standard | 16.0% | 17.6 | 13.33 | |||
| Speechmatics | Speechmatics Enhanced | 17.6 | 6.70 | ||||
| Microsoft Azure | Azure AI Speech Service | 17.2% | 2.0 | 16.67 | |||
 AssemblyAI | Nano | 16.3% | 85.0 | 2.00 | |||
| AssemblyAI | Universal, AssemblyAI | 14.5% | 85.6 | 6.17 | |||
| AssemblyAI | Slam-1 | 15.2% | 52.5 | 4.50 | |||
 Deepgram | Nova-2 | 17.3% | 162.9 | 4.30 | |||
| Deepgram | Base | 21.9% | 184.2 | 12.50 | |||
| Deepgram | Nova-3 | 18.3% | 179.1 | 4.30 | |||
 Gladia | Gladia v2 | whisper-v2-variant | 16.7% | 50.4 | 10.20 | ||
 Amazon Bedrock | Amazon Transcribe | 14.0% | 17.3 | 24.00 | |||
 Fish Audio | Fish Speech to Text | 0.00 | |||||
 Rev AI | Rev AI | 15.2% | 20.00 | ||||
 Google | Chirp 2, Google | 11.6% | 18.1 | 16.00 | |||
| Google | Chirp | 16.9% | 13.3 | 16.00 | |||
| Google | Chirp 3, Google | 15.0% | 30.2 | 16.00 | |||
 ElevenLabs | Scribe, ElevenLabs | 40.8 | 6.67 | ||||
| Google | Gemini 2.0 Flash | 17.9% | 51.5 | 1.40 | |||
| Google | Gemini 2.0 Flash Lite | 16.6% | 52.9 | 0.19 | |||
| Google | Gemini 2.5 Flash Lite | 16.1% | 101.3 | 0.58 | |||
| Google | Gemini 2.5 Flash | 19.2% | 79.0 | 1.92 | |||
| Google | Gemini 2.5 Pro | 15.0% | 11.7 | 0.00 | |||
| OpenAI | GPT-4o Transcribe | 21.3% | 26.8 | 6.00 | |||
| OpenAI | GPT-4o Mini Transcribe | 20.1% | 38.6 | 3.00 | |||
 IBM | Granite Speech 3.3 8B, IBM | 15.7% | 0.00 | ||||
| Replicate | Parakeet RNNT 1.1B | 6.6 | 1.91 | ||||
 NVIDIA | Parakeet TDT 0.6B V2, NVIDIA | 60.1 | 0.00 | ||||
| Replicate | Canary Qwen 2.5B, NVIDIA | 13.2% | 145.3 | 0.00 | |||
 Mistral | Voxtral Mini | 15.8% | 65.2 | 1.00 | |||
| Mistral | Voxtral Small | 14.7% | 67.8 | 4.00 | |||
| Deepinfra | Voxtral Small | 14.7% | 23.5 | 3.00 | |||
| Deepinfra | Voxtral Mini | 15.8% | 63.9 | 1.00 | |||
 Alibaba Cloud | Qwen3 ASR Flash | 15.0% | 1.92 |
Speech to Text providers compared: OpenAI, Speechmatics, Microsoft Azure, AssemblyAI, fal.ai, Replicate, Deepgram, Gladia, Groq, Deepinfra, Fireworks, Amazon Bedrock, Fish Audio, Rev AI, Google, ElevenLabs, SambaNova, IBM, Together.ai, Mistral, NVIDIA, and Alibaba Cloud.