Whisper: API Provider Benchmarking & Analysis
Analysis of API providers of Whisper across performance metrics including word error rate, speed, and price.
API providers compared include OpenAI, AssemblyAI, Speechmatics, Microsoft Azure, fal.ai, Replicate, Deepgram, Gladia, Groq, Deepinfra, Fireworks, Amazon Bedrock, Fish Audio, Rev AI, Google, ElevenLabs, SambaNova, IBM, Together.ai, and Mistral.
API providers compared include OpenAI, AssemblyAI, Speechmatics, Microsoft Azure, fal.ai, Replicate, Deepgram, Gladia, Groq, Deepinfra, Fireworks, Amazon Bedrock, Fish Audio, Rev AI, Google, ElevenLabs, SambaNova, IBM, Together.ai, and Mistral.
Highlights
Word error rate
Word error rate: % of words transcribed incorrectly, Lower is better
Speed Factor
Speed factor: Input audio seconds transcribed per second, Higher is better
Price
Price: USD per 1000 minutes of audio, Lower is better
Summary Analysis
Word Error Rate vs. Price
Word error rate: % of words transcribed incorrectly, Price: USD per 1000 minutes of audio
Most attractive quadrant
Size represents Speed factor: Input audio seconds transcribed per second
Amazon Transcribe
Azure AI Speech Service
Chirp 2
Distil-Whisper, Groq
Enhanced
GPT-4o Mini Transcribe
GPT-4o Transcribe
Incredibly Fast Whisper, Replicate
Nova-3
Parakeet RNNT 1.1B
Scribe
Universal-2
Voxtral Small
Whisper (L, v2), Azure
Whisper (L, v2), OpenAI
Whisper (L, v2), Replicate
Whisper (L, v3, Turbo), Fireworks
Whisper (L, v3, Turbo), Groq
Whisper (L, v3), Deepinfra
Whisper (L, v3), fal.ai
Whisper (L, v3), Fireworks
Whisper (L, v3), Groq
Whisper (L, v3), Replicate
Whisper (L, v3), SambaNova
Whisper (M), Replicate
Whisper (S), Replicate
Whisper Large v3, together.ai
WhisperX, Replicate
Wizper (L, v3), fal.ai
Word Error Rate: Percentage of words incorrect in the transcription. Evaluation updated June 2024 to 5,000 test samples.
Artificial Analysis' independent evaluation is based on Common Voice v16.1, Mozilla's leading open-source speech to text dataset. Further detail present on methodology page.
Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
Word Error Rate vs. Speed Factor
Word error rate: % of words transcribed incorrectly, Speed factor: Input audio seconds transcribed per second
Most attractive quadrant
Size represents Price: USD per 1000 minutes of audio
Amazon Transcribe
Chirp 2
Distil-Whisper, Groq
Enhanced
GPT-4o Mini Transcribe
GPT-4o Transcribe
Incredibly Fast Whisper, Replicate
Nova-3
Parakeet RNNT 1.1B
Scribe
Universal-2
Voxtral Small
Whisper (L, v2), Azure
Whisper (L, v2), OpenAI
Whisper (L, v2), Replicate
Whisper (L, v3, Turbo), Fireworks
Whisper (L, v3, Turbo), Groq
Whisper (L, v3), Deepinfra
Whisper (L, v3), fal.ai
Whisper (L, v3), Fireworks
Whisper (L, v3), Groq
Whisper (L, v3), Replicate
Whisper (L, v3), SambaNova
Whisper Large v3, together.ai
WhisperX, Replicate
Wizper (L, v3), fal.ai
Summary of key metrics & further information
| Provider | Model | Whisper version | Footnotes | Word Error Rate (%) | Median Speed Factor | Price (USD per 1000 minutes) | Further Details |
|---|---|---|---|---|---|---|---|
| OpenAI | Whisper Large v2 | large-v2 | 10.6% |
Microsoft Azure
fal.ai
Replicate
Groq
Deepinfra
Fireworks
SambaNova
Together.ai
AssemblyAI
Speechmatics
Deepgram
Gladia
Amazon Bedrock
Fish Audio
Rev AI
Google
ElevenLabs
IBM
Mistral