Whisper: API Provider Benchmarking & Analysis
Analysis of API providers of Whisper across performance metrics including word error rate, speed, and price.
API providers compared include OpenAI, AssemblyAI, Speechmatics, Microsoft Azure, fal.ai, Replicate, Deepgram, Gladia, Groq, Deepinfra, Fireworks, Amazon Bedrock, Fish Audio, Rev AI, Google, ElevenLabs, SambaNova, IBM, Together.ai, and Mistral.
API providers compared include OpenAI, AssemblyAI, Speechmatics, Microsoft Azure, fal.ai, Replicate, Deepgram, Gladia, Groq, Deepinfra, Fireworks, Amazon Bedrock, Fish Audio, Rev AI, Google, ElevenLabs, SambaNova, IBM, Together.ai, and Mistral.
Highlights
Word error rate
Word error rate: % of words transcribed incorrectly, Lower is better
Speed Factor
Speed factor: Input audio seconds transcribed per second, Higher is better
Price
Price: USD per 1000 minutes of audio, Lower is better
Summary Analysis
Word Error Rate vs. Price
Word error rate: % of words transcribed incorrectly, Price: USD per 1000 minutes of audio
Most attractive quadrant
Size represents Speed factor: Input audio seconds transcribed per second
Amazon Transcribe
Azure AI Speech Service
Chirp 2
Distil-Whisper, Groq
Enhanced
GPT-4o Mini Transcribe
GPT-4o Transcribe
Incredibly Fast Whisper, Replicate
Nova-3
Parakeet RNNT 1.1B, Replicate
Scribe
Universal-2
Voxtral Small
Whisper (L, v2), Azure
Whisper (L, v2), OpenAI
Whisper (L, v2), Replicate
Whisper (L, v3, Turbo), Fireworks
Whisper (L, v3, Turbo), Groq
Whisper (L, v3), Deepinfra
Whisper (L, v3), fal.ai
Whisper (L, v3), Fireworks
Whisper (L, v3), Groq
Whisper (L, v3), Replicate
Whisper (L, v3), SambaNova
Whisper (M), Replicate
Whisper (S), Replicate
Whisper Large v3, together.ai
WhisperX, Replicate
Wizper (L, v3), fal.ai
Word Error Rate: Percentage of words incorrect in the transcription. Evaluation updated June 2024 to 5,000 test samples.
Artificial Analysis' independent evaluation is based on Common Voice v16.1, Mozilla's leading open-source speech to text dataset. Further detail present on methodology page.
Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
Word Error Rate vs. Speed Factor
Word error rate: % of words transcribed incorrectly, Speed factor: Input audio seconds transcribed per second
Most attractive quadrant
Size represents Price: USD per 1000 minutes of audio
Amazon Transcribe
Chirp 2
Distil-Whisper, Groq
Enhanced
GPT-4o Mini Transcribe
GPT-4o Transcribe
Incredibly Fast Whisper, Replicate
Nova-3
Parakeet RNNT 1.1B, Replicate
Scribe
Universal-2
Voxtral Small
Whisper (L, v2), Azure
Whisper (L, v2), OpenAI
Whisper (L, v2), Replicate
Whisper (L, v3, Turbo), Fireworks
Whisper (L, v3, Turbo), Groq
Whisper (L, v3), Deepinfra
Whisper (L, v3), fal.ai
Whisper (L, v3), Fireworks
Whisper (L, v3), Groq
Whisper (L, v3), Replicate
Whisper (L, v3), SambaNova
Whisper Large v3, together.ai
WhisperX, Replicate
Wizper (L, v3), fal.ai
Word Error Rate: Percentage of words incorrect in the transcription. Evaluation updated June 2024 to 5,000 test samples.
Artificial Analysis' independent evaluation is based on Common Voice v16.1, Mozilla's leading open-source speech to text dataset. Further detail present on methodology page.
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Speed Factor vs. Price
Speed factor: Input audio seconds transcribed per second, Price: USD per 1000 minutes of audio
Most attractive quadrant
Amazon Transcribe
Chirp 2
Distil-Whisper, Groq
Enhanced
GPT-4o Mini Transcribe
GPT-4o Transcribe
Incredibly Fast Whisper, Replicate
Nova-3
Parakeet RNNT 1.1B, Replicate
Scribe
Universal-2
Voxtral Small
Whisper (L, v2), Azure
Whisper (L, v2), OpenAI
Whisper (L, v2), Replicate
Whisper (L, v3, Turbo), Fireworks
Whisper (L, v3, Turbo), Groq
Whisper (L, v3), Deepinfra
Whisper (L, v3), fal.ai
Whisper (L, v3), Fireworks
Whisper (L, v3), Groq
Whisper (L, v3), Replicate
Whisper (L, v3), SambaNova
Whisper Large v3, together.ai
WhisperX, Replicate
Wizper (L, v3), fal.ai
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
Quality
Word Error Rate
Word error rate: % of words transcribed incorrectly, Lower is better
Word Error Rate: Percentage of words incorrect in the transcription. Evaluation updated June 2024 to 5,000 test samples.
Artificial Analysis' independent evaluation is based on Common Voice v16.1, Mozilla's leading open-source speech to text dataset. Further detail present on methodology page.
Speed
Speed Factor
Speed factor: Input audio seconds transcribed per second, Higher is better
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Speed Factor, Variance
Speed factor: Input audio seconds transcribed per second, Results by percentile, Higher is better
Median; other points represent Min, 25th, 75th percentiles and Max respectively
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Boxplot: Shows variance of measurements
Speed Factor, Over Time
Speed factor: Input audio seconds transcribed per second, Higher is better
Whisper (L, v2), OpenAI
Whisper (L, v2), Azure
Azure AI Speech Service
Enhanced
Wizper (L, v3), fal.ai
Incredibly Fast Whisper, Replicate
Whisper (L, v2), Replicate
Whisper (L, v3), Replicate
WhisperX, Replicate
Whisper (M), Replicate
Whisper (S), Replicate
Whisper (L, v3), Groq
Distil-Whisper, Groq
Whisper (L, v3), Deepinfra
Whisper (L, v3), fal.ai
Whisper (L, v3, Turbo), Groq
Whisper (L, v3), Fireworks
Whisper (L, v3, Turbo), Fireworks
Universal-2
Amazon Transcribe
Nova-3
Chirp 2
Scribe
GPT-4o Transcribe
GPT-4o Mini Transcribe
Whisper (L, v3), SambaNova
Granite 3.3 8B, IBM
Parakeet RNNT 1.1B, Replicate
Whisper Large v3, together.ai
Voxtral Small
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Over time measurement: Median measurement per day, based on 8 measurements each day at different times. Labels represent start of week's measurements.
Price
Price
Price: USD per 1000 minutes of audio, Lower is better
Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
For providers which do not price based on audio duration and rather on processing time (incl. Replicate, fal), we have calculated an indicative per minute price based on processing time expected per minute of audio.Further detail present on methodology page.
Note: Groq chargers for a minimum of 10s per request.
Note: Groq chargers for a minimum of 10s per request.
Summary of key metrics & further information
| Provider | Model | Whisper version | Footnotes | Word Error Rate (%) | Median Speed Factor | Price (USD per 1000 minutes) | Further Details |
|---|---|---|---|---|---|---|---|
| OpenAI | Whisper Large v2 | large-v2 | 10.6% | 31.5 | 6.00 | ||
 Microsoft Azure | Whisper Large v2 | large-v2 | 10.6% | 33.7 | 6.00 | ||
 fal.ai | Whisper Large v3 | large-v3 | 10.3% | 328.1 | 0.50 | ||
 Replicate | Incredibly Fast Whisper | large-v3 | 10.3% | 63.4 | 1.49 | ||
| Replicate | Whisper Large v2 | large-v2 | 11.2% | 2.3 | 3.47 | ||
| Replicate | Whisper Large v3 | large-v3 | 10.3% | 2.7 | 4.23 | ||
| Replicate | WhisperX | large-v3 | 10.9% | 18.5 | 1.09 | ||
| Replicate | Whisper (M) | medium | 12.8% | 2.68 | |||
| Replicate | Whisper (S) | small | 17.0% | 1.37 | |||
 Groq | Whisper Large v3 | large-v3 | 10.3% | 213.7 | 1.85 | ||
| Groq | Distil-Whisper | 13.0% | 247.5 | 0.33 | |||
 Deepinfra | Whisper Large v3 | large-v3 | 10.3% | 96.0 | 0.45 | ||
| fal.ai | Whisper Large v3 | large-v3 | 10.3% | 146.8 | 1.15 | ||
| Groq | Whisper Large v3 Turbo | v3 Turbo | 12.0% | 241.0 | 0.67 | ||
 Fireworks | Whisper Large v3 | large-v3 | 11.2% | 402.9 | 1.00 | ||
| Fireworks | Whisper Large v3 Turbo | v3 Turbo | 13.7% | 471.9 | 1.00 | ||
 SambaNova | Whisper-Large-v3 | large-v3 | 10.8% | 166.4 | 1.67 | ||
 Together.ai | Whisper Large v3 | large-v3 | 10.3% | 180.4 | 1.50 | ||
 AssemblyAI | Universal-1 | 8.7% | 84.7 | 6.17 | |||
| AssemblyAI | Nano | 12.7% | 59.3 | 2.00 | |||
| AssemblyAI | Universal-2 | 8.6% | 84.9 | 6.17 | |||
 Speechmatics | Standard | 12.6% | 17.6 | 13.33 | |||
| Speechmatics | Enhanced | 8.1% | 17.6 | 6.70 | |||
| Microsoft Azure | Azure AI Speech Service | 12.6% | 16.67 | ||||
 Deepgram | Nova-2 | 15.1% | 152.7 | 4.30 | |||
| Deepgram | Base | 26.1% | 159.3 | 12.50 | |||
| Deepgram | Whisper Large v2 | large-v2 | 10.6% | 33.0 | 4.80 | ||
| Deepgram | Nova-3 | 12.8% | 146.6 | 4.30 | |||
 Gladia | Gladia | whisper-v2-variant | 12.9% | 32.8 | 10.20 | ||
 Amazon Bedrock | Amazon Transcribe | 11.2% | 19.8 | 24.00 | |||
 Fish Audio | Fish Speech to Text | 19.1% | 0.00 | ||||
 Rev AI | Rev AI | 0.0% | 20.00 | ||||
 Google | Chirp 2 | 9.8% | 19.8 | 16.00 | |||
| Google | Chirp | 12.4% | 13.3 | 16.00 | |||
 ElevenLabs | Scribe | 7.7% | 40.6 | 6.67 | |||
| Google | Gemini 2.0 Flash | 14.9% | 59.7 | 1.40 | |||
| Google | Gemini 2.0 Flash Lite | 14.5% | 58.9 | 0.19 | |||
| OpenAI | GPT-4o Transcribe | 8.9% | 31.6 | 6.00 | |||
| OpenAI | GPT-4o Mini Transcribe | 13.2% | 37.3 | 3.00 | |||
 IBM | Granite 3.3 8B | 8.4% | 0.00 | ||||
| Replicate | Granite 3.3 8B | 0.0% | 1.20 | ||||
| Replicate | Parakeet RNNT 1.1B | 6.1% | 6.9 | 1.91 | |||
 Mistral | Voxtral Mini | 10.4% | 70.6 | 1.00 | |||
| Mistral | Voxtral Small | 8.8% | 60.2 | 4.00 | |||
| Deepinfra | Voxtral Small | 0.0% | 23.5 | 3.00 | |||
| Deepinfra | Voxtral Mini | 0.0% | 79.6 | 1.00 |