AssemblyAI: API Provider Benchmarking & Analysis
Highlights
Artificial Analysis Word Error Rate (AA-WER) Index by API
Artificial Analysis Word Error Rate (AA-WER) Index by API
Measures transcription accuracy across 3 datasets to evaluate models in real-world speech with diverse accents, domain-specific language, and challenging channel & acoustic conditions.
AA-WER is calculated as an audio-duration-weighted average of WER across ~2 hours from three datasets: VoxPopuli, Earnings-22, and AMI-SDM. See methodology for more detail.
Artificial Analysis Word Error Rate (AA-WER) Index vs Other Metrics
Artificial Analysis Word Error Rate Index vs. Price
Measures transcription accuracy across 3 datasets to evaluate models in real-world speech with diverse accents, domain-specific language, and challenging channel & acoustic conditions.
AA-WER is calculated as an audio-duration-weighted average of WER across ~2 hours from three datasets: VoxPopuli, Earnings-22, and AMI-SDM. See methodology for more detail.
Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
Speed Factor
Speed Factor
Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.
Artificial Analysis measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particularly for very short durations (under 1 minute).
Price
Price of Transcription
Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.
For providers which do not price based on audio duration and rather on processing time (incl. Replicate, fal), we have calculated an indicative per minute price based on processing time expected per minute of audio.Further detail present on methodology page.
Note: Groq chargers for a minimum of 10s per request.
| Provider | Model | Whisper version | Footnotes | Word Error Rate (%) | Median Speed Factor | Price (USD per 1000 minutes) | Further Details |
|---|---|---|---|---|---|---|---|
 OpenAI | Whisper Large v2 | large-v2 | 15.8% | 26.5 | 6.00 | ||
 Microsoft Azure | Whisper Large v2 | large-v2 | 27.2% | 34.3 | 6.00 | ||
 fal.ai | Wizper Large v3 | large-v3 | 16.8% | 311.9 | 0.50 | ||
 Replicate | Incredibly Fast Whisper | large-v3 | 18.2% | 64.8 | 1.49 | ||
| Replicate | Whisper Large v2 | large-v2 | 15.8% | 2.3 | 3.47 | ||
| Replicate | Whisper Large v3 | large-v3 | 24.6% | 2.9 | 4.23 | ||
| Replicate | WhisperX | large-v3 | 16.3% | 19.0 | 1.09 | ||
 Groq | Whisper Large v3 | large-v3 | 16.8% | 273.8 | 1.85 | ||
 DeepInfra | Whisper Large v3 | large-v3 | 16.8% | 91.9 | 0.45 | ||
| fal.ai | Whisper Large v3 | large-v3 | 16.8% | 135.9 | 1.15 | ||
| Groq | Whisper Large v3 Turbo | v3 Turbo | 372.3 | 0.67 | |||
 Fireworks | Whisper Large v3 | large-v3 | 432.0 | 1.00 | |||
| Fireworks | Whisper Large v3 Turbo | v3 Turbo | 17.8% | 512.7 | 1.00 | ||
 SambaNova | Whisper-Large-v3 | large-v3 | 16.8% | 94.5 | 1.67 | ||
 Together.ai | Whisper Large v3 | large-v3 | 24.6% | 127.3 | 1.50 | ||
 Speechmatics | Speechmatics Standard | 16.0% | 43.8 | 4.00 | |||
| Speechmatics | Speechmatics Enhanced | 14.4% | 24.3 | 6.70 | |||
| Microsoft Azure | Azure Realtime Speech to Text | 17.2% | 2.0 | 16.67 | |||
 Deepgram | Nova-2 | 17.3% | 428.5 | 4.30 | |||
| Deepgram | Base | 21.9% | 406.1 | 12.50 | |||
| Deepgram | Nova-3 | 18.3% | 572.5 | 4.30 | |||
 AssemblyAI | Universal, AssemblyAI | 14.5% | 81.9 | 2.50 | |||
| AssemblyAI | Slam-1 | 15.1% | 57.0 | 4.50 | |||
 Amazon Bedrock | Amazon Transcribe | 14.0% | 18.9 | 24.00 | |||
 Google | Chirp | 14.6 | 16.00 | ||||
| Google | Chirp 2, Google | 11.6% | 18.2 | 16.00 | |||
| Google | Chirp 3, Google | 15.0% | 28.8 | 16.00 | |||
 ElevenLabs | Scribe, ElevenLabs | 41.2 | 6.67 | ||||
| ElevenLabs | Scribe v2 | 14.0% | 35.2 | 6.67 | |||
| Google | Gemini 2.0 Flash | 17.9% | 55.2 | 1.40 | |||
| Google | Gemini 2.0 Flash Lite | 16.6% | 57.0 | 0.19 | |||
| Google | Gemini 2.5 Flash Lite | 16.1% | 82.2 | 0.58 | |||
| Google | Gemini 2.5 Flash | 19.2% | 93.3 | 1.92 | |||
| Google | Gemini 2.5 Pro | 15.0% | 13.4 | 0.00 | |||
| OpenAI | GPT-4o Transcribe | 21.3% | 28.9 | 6.00 | |||
| OpenAI | GPT-4o Mini Transcribe | 20.1% | 43.7 | 3.00 | |||
| Replicate | Parakeet RNNT 1.1B | 6.5 | 1.91 | ||||
 NVIDIA | Parakeet TDT 0.6B V2, NVIDIA | 55.7 | 0.00 | ||||
| Replicate | Canary Qwen 2.5B, NVIDIA | 13.2% | 5.8 | 0.74 | |||
 Hathora | Parakeet TDT 0.6B V3, Hathora | 13.0% | 25.8 | 1.32 | |||
 Mistral | Voxtral Mini | 15.8% | 55.9 | 1.00 | |||
| Mistral | Voxtral Small | 14.7% | 67.4 | 4.00 | |||
| DeepInfra | Voxtral Small | 14.7% | 24.3 | 3.00 | |||
| DeepInfra | Voxtral Mini | 15.8% | 71.8 | 1.00 | |||
 Gladia | Solaria-1, Gladia | 17.4% | 47.5 | 8.33 | |||
| Amazon Bedrock | Nova 2 Omni | 15.9% | 38.9 | 1.85 | |||
| Amazon Bedrock | Nova 2 Pro | 15.8% | 23.4 | 3.10 |
Speech to Text providers compared: OpenAI, Speechmatics, Microsoft Azure, fal.ai, Replicate, Deepgram, Groq, DeepInfra, Fireworks, AssemblyAI, Amazon Bedrock, Google, ElevenLabs, SambaNova, Together.ai, Mistral, NVIDIA, Gladia, and Hathora.