Whisper: API Provider Benchmarking & Analysis
Analysis of API providers of Whisper across performance metrics including word error rate, speed, and price.
API providers compared include OpenAI, AssemblyAI, Speechmatics, Microsoft Azure, fal.ai, Replicate, Deepgram, Gladia, Groq, Deepinfra, Fireworks, Amazon Bedrock, Fish Audio, and Google.
API providers compared include OpenAI, AssemblyAI, Speechmatics, Microsoft Azure, fal.ai, Replicate, Deepgram, Gladia, Groq, Deepinfra, Fireworks, Amazon Bedrock, Fish Audio, and Google.
Highlights
Word error rate
Word error rate: % of words transcribed incorrectly, Lower is better
Speed Factor
Speed factor: Input audio seconds transcribed per second, Higher is better
Price
Price: USD per 1000 minutes of audio, Lower is better
Summary Analysis
Word Error Rate vs. Price
Word error rate: % of words transcribed incorrectly, Price: USD per 1000 minutes of audio
Most attractive quadrant
Size represents Speed factor: Input audio seconds transcribed per second
Whisper (L, v2), OpenAI
Whisper (L, v2), Azure
Enhanced
Wizper (L, v3), fal.ai
Incredibly Fast Whisper, Replicate
Whisper (L, v2), Replicate
Whisper (L, v3), Replicate
WhisperX, Replicate
Whisper (L, v3), Groq
Distil-Whisper, Groq
Whisper (L, v3), fal.ai
Whisper (L, v3), Deepinfra
Whisper (L, v3, Turbo), Groq
Whisper (L, v3), Fireworks
Whisper (L, v3, Turbo), Fireworks
Universal-2
Amazon Transcribe
Nova-3
Chirp 2
GPT-4o Transcribe
GPT-4o Mini Transcribe
Word Error Rate: Percentage of words incorrect in the transcription. Evaluation updated June 2024 to 5,000 test samples.
Artificial Analysis' independent evaluation is based on Common Voice v16.1, Mozilla's leading open-source speech to text dataset. Further detail present on methodology page.
Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
Word Error Rate vs. Speed Factor
Word error rate: % of words transcribed incorrectly, Speed factor: Input audio seconds transcribed per second
Most attractive quadrant
Size represents Price: USD per 1000 minutes of audio
Whisper (L, v2), OpenAI
Whisper (L, v2), Azure
Enhanced
Wizper (L, v3), fal.ai
Incredibly Fast Whisper, Replicate
Whisper (L, v2), Replicate
Whisper (L, v3), Replicate
WhisperX, Replicate
Whisper (L, v3), Groq
Distil-Whisper, Groq
Whisper (L, v3), fal.ai
Whisper (L, v3), Deepinfra
Whisper (L, v3, Turbo), Groq
Whisper (L, v3), Fireworks
Whisper (L, v3, Turbo), Fireworks
Universal-2
Amazon Transcribe
Nova-3
Chirp 2
GPT-4o Transcribe
GPT-4o Mini Transcribe
Word Error Rate: Percentage of words incorrect in the transcription. Evaluation updated June 2024 to 5,000 test samples.
Artificial Analysis' independent evaluation is based on Common Voice v16.1, Mozilla's leading open-source speech to text dataset. Further detail present on methodology page.
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Speed Factor vs. Price
Speed factor: Input audio seconds transcribed per second, Price: USD per 1000 minutes of audio
Most attractive quadrant
Whisper (L, v2), OpenAI
Whisper (L, v2), Azure
Enhanced
Wizper (L, v3), fal.ai
Incredibly Fast Whisper, Replicate
Whisper (L, v2), Replicate
Whisper (L, v3), Replicate
WhisperX, Replicate
Whisper (L, v3), Groq
Distil-Whisper, Groq
Whisper (L, v3), fal.ai
Whisper (L, v3), Deepinfra
Whisper (L, v3, Turbo), Groq
Whisper (L, v3), Fireworks
Whisper (L, v3, Turbo), Fireworks
Universal-2
Amazon Transcribe
Nova-3
Chirp 2
GPT-4o Transcribe
GPT-4o Mini Transcribe
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
Quality
Word Error Rate
Word error rate: % of words transcribed incorrectly, Lower is better
Word Error Rate: Percentage of words incorrect in the transcription. Evaluation updated June 2024 to 5,000 test samples.
Artificial Analysis' independent evaluation is based on Common Voice v16.1, Mozilla's leading open-source speech to text dataset. Further detail present on methodology page.
Speed
Speed Factor
Speed factor: Input audio seconds transcribed per second, Higher is better
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Speed Factor, Variance
Speed factor: Input audio seconds transcribed per second, Results by percentile, Higher is better
Median, Other points represent 5th, 25th, 75th, 95th Percentiles respectively
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Boxplot: Shows variance of measurements
Speed Factor, Over Time
Speed factor: Input audio seconds transcribed per second, Higher is better
Whisper (L, v2), OpenAI
Whisper (L, v2), Azure
Enhanced
Wizper (L, v3), fal.ai
Incredibly Fast Whisper, Replicate
Whisper (L, v2), Replicate
Whisper (L, v3), Replicate
WhisperX, Replicate
Whisper (L, v3), Groq
Distil-Whisper, Groq
Whisper (L, v3), fal.ai
Whisper (L, v3), Deepinfra
Whisper (L, v3, Turbo), Groq
Whisper (L, v3), Fireworks
Whisper (L, v3, Turbo), Fireworks
Universal-2
Amazon Transcribe
Nova-3
Chirp 2
GPT-4o Transcribe
GPT-4o Mini Transcribe
Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).
Over time measurement: Median measurement per day, based on 8 measurements each day at different times. Labels represent start of week's measurements.
Price
Price
Price: USD per 1000 minutes of audio, Lower is better
Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
For providers which do not price based on audio duration and rather on processing time (incl. Replicate, fal), we have calculated an indicative per minute price based on processing time expected per minute of audio.Further detail present on methodology page.
Note: Groq chargers for a minimum of 10s per request.
Note: Groq chargers for a minimum of 10s per request.
Summary of key metrics & further information
| Provider | Model | Whisper version | Footnotes | Word Error Rate (%) | Median Speed Factor | Price (USD per 1000 minutes) | Further Details |
|---|---|---|---|---|---|---|---|
| OpenAI | Whisper Large v2 | large-v2 | 10.6% | 34.1 | 6.00 | ||
 Microsoft Azure | Whisper Large v2 | large-v2 | 10.6% | 34.0 | 6.00 | ||
 fal.ai | Whisper Large v3 | large-v3 | 10.3% | 278.1 | 0.50 | ||
 Replicate | Incredibly Fast Whisper | large-v3 | 10.3% | 64.8 | 1.49 | ||
| Replicate | Whisper Large v2 | large-v2 | 11.2% | 2.7 | 3.47 | ||
| Replicate | Whisper Large v3 | large-v3 | 10.3% | 3.1 | 4.23 | ||
| Replicate | WhisperX | large-v3 | 10.9% | 13.5 | 1.09 | ||
 Groq | Whisper Large v3 | large-v3 | 10.3% | 212.8 | 1.85 | ||
| Groq | Distil-Whisper | 13.0% | 241.9 | 0.33 | |||
| fal.ai | Whisper Large v3 | large-v3 | 10.3% | 111.3 | 1.15 | ||
 Deepinfra | Whisper Large v3 | large-v3 | 10.3% | 101.9 | 0.45 | ||
| Groq | Whisper Large v3 Turbo | v3 Turbo | 12.0% | 236.1 | 0.67 | ||
 Fireworks | Whisper Large v3 | large-v3 | 11.2% | 269.2 | 1.00 | ||
| Fireworks | Whisper Large v3 Turbo | v3 Turbo | 13.7% | 447.4 | 1.00 | ||
 AssemblyAI | Universal-1 | 8.7% | 85.7 | 6.17 | |||
| AssemblyAI | Nano | 12.7% | 84.5 | 2.00 | |||
| AssemblyAI | Universal-2 | 8.6% | 85.2 | 6.17 | |||
 Speechmatics | Standard | 12.6% | 17.7 | 13.33 | |||
| Speechmatics | Enhanced | 8.1% | 9.3 | 6.70 | |||
 Deepgram | Nova-2 | 15.1% | 149.8 | 4.30 | |||
| Deepgram | Base | 26.1% | 166.9 | 12.50 | |||
| Deepgram | Whisper Large v2 | large-v2 | 10.6% | 28.3 | 4.80 | ||
| Deepgram | Nova-3 | 12.8% | 146.5 | 4.30 | |||
 Gladia | Gladia | whisper-v2-variant | 12.9% | 17.8 | 10.20 | ||
 Amazon Bedrock | Amazon Transcribe | 11.2% | 20.0 | 24.00 | |||
 Fish Audio | Fish Speech to Text | 19.1% | 26.5 | 0.00 | |||
 Google | Chirp | 12.4% | 14.1 | 16.00 | |||
| Google | Chirp 2 | 9.8% | 78.5 | 16.00 | |||
| OpenAI | GPT-4o Transcribe | 8.9% | 36.1 | 6.00 | |||
| OpenAI | GPT-4o Mini Transcribe | 13.2% | 38.0 | 3.00 |