Whisper

Whisper: API Provider Benchmarking & Analysis

Creator:OpenAI

License:Open

Analysis of Whisper API providers across performance metrics including Artificial Analysis Word Error Rate Index, speed, and price.

Highlights

Word Error Rate Index

AA-WER v2 · % of words transcribed incorrectly · Lower is better

Speed Factor

Input audio seconds transcribed per second · Higher is better

Price

USD per 1000 minutes of audio · Lower is better

Artificial Analysis Word Error Rate (AA-WER) Index by API

% of words transcribed incorrectly · Lower is better · AA-WER v2 incorporates 3 datasets: AA-AgentTalk (50%), VoxPopuli-Cleaned-AA (25%), Earnings22-Cleaned-AA (25%)

Note: For Earnings22, if a model cannot reliably handle full-length audio due to time limits, we chunk to ~9 minutes (relevant to: Nova 2 Pro, Amazon; Voxtral Mini Transcribe, Mistral; GPT-4o Transcribe, OpenAI; GPT-4o Mini Transcribe, OpenAI). For models with even shorter time limits, we chunk to ~30 seconds (relevant to: Canary Qwen 2.5B, NVIDIA).

Measures transcription accuracy across 3 datasets to evaluate models in real-world speech with diverse accents, domain-specific language, and challenging channel & acoustic conditions.

AA-WER is calculated as an audio-duration-weighted average of WER across ~8 hours from three datasets: AA-AgentTalk (50%), VoxPopuli-Cleaned-AA (25%), and Earnings22-Cleaned-AA (25%). See methodology for more detail.

API Benchmarks

Artificial Analysis Word Error Rate Index vs. Price

% of words transcribed incorrectly · Lower is better · AA-WER v2 incorporates 3 datasets: AA-AgentTalk (50%), VoxPopuli-Cleaned-AA (25%), Earnings22-Cleaned-AA (25%) · USD per 1000 minutes of audio

Most attractive quadrant

Amazon Transcribe

Canary Qwen 2.5B, Replicate

Enhanced

Gemini 3 Flash High

Gemini 3 Pro High

Gemini 3.1 Pro Preview High

Gemini 3.1 Pro Preview Low

GPT-4o Mini Transcribe

GPT-4o Transcribe

Incredibly Fast , Replicate

L, v3, fal.ai

L, v3, Fireworks

L, v3, Replicate

L, v3, Turbo, Fireworks

L, v3, Turbo, Groq

Large v2

Large v3, together.ai

Nova 2 Pro

Nova-3

Parakeet TDT 0.6B V3, Togetherai

Pulse STT

Rev AI

Scribe v2

Solaria-1

Soniox V4

Universal

Universal-3 Pro

Voxtral Mini Transcribe

Voxtral Mini Transcribe 2

Voxtral Small

Measures transcription accuracy across 3 datasets to evaluate models in real-world speech with diverse accents, domain-specific language, and challenging channel & acoustic conditions.

Estimated cost in USD to transcribe 1,000 minutes of audio, normalized across providers with different billing models, and including billed reasoning tokens where available. Further detail on the methodology page.

Speed Factor

Input audio seconds transcribed per second · Higher is better

Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed. Reported Speed Factor values are medians across benchmark trials from the last 7 days; over-time chart points are daily medians. Artificial Analysis measurements are based on an audio duration of 10 minutes. Speed Factor may vary for other durations, particularly very short durations under 1 minute.

Price

Price of Transcription

USD per 1000 minutes of audio · Lower is better

Summary of Key Metrics & Further Information

Provider	Model	Whisper version	Word Error Rate (%)	Median Speed Factor	Price (USD per 1000 minutes)	Further Details
Alibaba Cloud	Qwen3.5 Omni Flash		13.6%	92.9	0.00	Details
Alibaba Cloud	Qwen3.5 Omni Plus		3.7%	97.5	0.00	Details
Google	Gemini 3.1 Pro Preview (High)		2.9%	6.3	18.15	Details
Google	Gemini 3.1 Pro Preview (Low)		3.8%	6.3	7.72	Details
Google	Gemini 3 Flash (High)		3.1%	15.7	13.70	Details
Google	Gemini 3 Pro (High)		2.9%	6.3	18.40	Details
Google	Gemini 2.5 Flash Lite		5.3%	66.6	6.56	Details
Google	Gemini 2.5 Flash		5.2%	75.7	6.66	Details
Google	Gemini 2.5 Pro		3.0%	12.5	11.39	Details
Google	Gemini 2.0 Flash Lite		3.9%	51.3	0.19	Details
Google	Gemini 2.0 Flash		3.9%	52.8	1.40	Details
Google	Gemini 3.1 Flash-Lite Preview (Minimal)		3.5%	78.7	5.83	Details
Smallest.ai	Pulse STT		4.5%	139.8	5.00	Details
Mistral	Voxtral Mini Transcribe 2		3.7%	76.9	3.00	Details
Mistral	Voxtral Mini Transcribe		3.7%	53.4	1.00	Details
Mistral	Voxtral Small		2.9%	67.7	4.00	Details
DeepInfra	Voxtral Mini		4.0%	80.1	1.00	Details
AssemblyAI	Universal-3 Pro		3.3%	96.3	3.50	Details
AssemblyAI	Universal, AssemblyAI		3.9%	101.8	2.50	Details
Soniox	Soniox V4		4.0%	42.5	1.66	Details
ElevenLabs	Scribe v2		2.2%	30.7	6.67	Details
ElevenLabs	Scribe v1		3.1%	39.5	6.67	Details
Amazon Bedrock	Nova 2 Pro		5.0%	23.6	3.10	Details
Gradium	Gradium Speech-to-Text		8.5%	2.3	13.00	Details
Together.ai	Parakeet TDT 0.6B V3, Togetherai		4.6%	313.7	1.50	Details
Replicate	Canary Qwen 2.5B, NVIDIA		4.3%	5.2	0.74	Details
NVIDIA	Parakeet TDT 0.6B V2, NVIDIA		6.5%	98.1	0.00	Details
Replicate	Parakeet RNNT 1.1B		5.5%	5.6	1.91	Details
Gladia	Solaria-1, Gladia		4.2%	47.6	4.07	Details
OpenAI	GPT-4o Transcribe		4.1%	31.6	6.00	Details
OpenAI	GPT-4o Mini Transcribe		4.6%	51.1	3.00	Details
Deepgram	Nova-3		5.3%	122.0	4.30	Details
Deepgram	Nova-2		5.4%	559.8	4.30	Details
Deepgram	Base		10.8%	658.1	12.50	Details
Groq	Whisper Large v3 Turbo	v3 Turbo	4.8%	266.3	0.67	Details
Fireworks	Whisper Large v3 Turbo	v3 Turbo	4.8%	245.7	1.00	Details
fal.ai	Wizper Large v3	large-v3	4.8%	237.7	0.50	Details
Replicate	Incredibly Fast Whisper	large-v3	5.8%	53.2	1.49	Details
Replicate	Whisper Large v3	large-v3	10.2%	2.6	4.23	Details
fal.ai	Whisper Large v3	large-v3	4.2%	92.5	1.15	Details
Fireworks	Whisper Large v3	large-v3	4.7%	257.0	1.00	Details
Together.ai	Whisper Large v3	large-v3	4.7%	254.7	1.50	Details
OpenAI	Whisper Large v2	large-v2	4.2%	28.6	6.00	Details
Amazon Bedrock	Amazon Transcribe		4.2%	18.3	24.00	Details
Speechmatics	Speechmatics Standard		5.1%	71.7	4.00	Details
Speechmatics	Speechmatics Enhanced		4.1%	68.3	6.70	Details
Rev AI	Rev AI		6.0%	12.4	20.00	Details

Speech to Text providers compared: Alibaba Cloud, Google, Smallest.ai, Mistral, AssemblyAI, Soniox, ElevenLabs, Amazon Bedrock, Gradium, Together.ai, Replicate, DeepInfra, NVIDIA, Gladia, OpenAI, Deepgram, Groq, Fireworks, fal.ai, Speechmatics, Rev AI.