NVIDIA: API Provider Benchmarking & Analysis

Analysis of API providers of NVIDIA across performance metrics including word error rate, speed, and price.
API providers compared include OpenAI, AssemblyAI, Speechmatics, Microsoft Azure, fal.ai, Replicate, Deepgram, Gladia, Groq, Deepinfra, Fireworks, Amazon Bedrock, Fish Audio, Rev AI, Google, ElevenLabs, SambaNova, IBM, Together.ai, and Mistral.

Creator:

NVIDIA

License:

Open Source

Link:

Visit

Highlights

Word error rate

Word error rate: % of words transcribed incorrectly, Lower is better

NVIDIA - Word Error Rate, Speed & Price Analysis | Artificial Analysis

Summary Analysis

Back to Navigation

Word Error Rate vs. Price

Word error rate: % of words transcribed incorrectly, Price: USD per 1000 minutes of audio

Most attractive quadrant

Size represents Speed factor: Input audio seconds transcribed per second

Amazon Transcribe

Azure AI Speech Service

Chirp 2

Enhanced

GPT-4o Mini Transcribe

GPT-4o Transcribe

Nova-3

Parakeet RNNT 1.1B

Scribe

Universal-2

Voxtral Small

Whisper (L, v2), OpenAI

Whisper (L, v3, Turbo), Groq

Whisper (L, v3), Fireworks

Whisper (L, v3), SambaNova

Wizper (L, v3), fal.ai

Word Error Rate: Percentage of words incorrect in the transcription. Evaluation updated June 2024 to 5,000 test samples.

Artificial Analysis' independent evaluation is based on Common Voice v16.1, Mozilla's leading open-source speech to text dataset. Further detail present on methodology page.

Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.

Word Error Rate vs. Speed Factor

Word error rate: % of words transcribed incorrectly, Speed factor: Input audio seconds transcribed per second

Most attractive quadrant

Size represents Price: USD per 1000 minutes of audio

Amazon Transcribe

Chirp 2

Enhanced

GPT-4o Mini Transcribe

GPT-4o Transcribe

Nova-3

Parakeet RNNT 1.1B

Scribe

Universal-2

Voxtral Small

Whisper (L, v2), OpenAI

Whisper (L, v3, Turbo), Groq

Whisper (L, v3), Fireworks

Whisper (L, v3), SambaNova

Wizper (L, v3), fal.ai

Word Error Rate: Percentage of words incorrect in the transcription. Evaluation updated June 2024 to 5,000 test samples.

Artificial Analysis' independent evaluation is based on Common Voice v16.1, Mozilla's leading open-source speech to text dataset. Further detail present on methodology page.

Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.

Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).

Speed Factor vs. Price

Speed factor: Input audio seconds transcribed per second, Price: USD per 1000 minutes of audio

Most attractive quadrant

Amazon Transcribe

Chirp 2

Enhanced

GPT-4o Mini Transcribe

GPT-4o Transcribe

Nova-3

Parakeet RNNT 1.1B

Scribe

Universal-2

Voxtral Small

Whisper (L, v2), OpenAI

Whisper (L, v3, Turbo), Groq

Whisper (L, v3), Fireworks

Whisper (L, v3), SambaNova

Wizper (L, v3), fal.ai

Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.

Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).

Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.

Quality

Back to Navigation

Word Error Rate

Word error rate: % of words transcribed incorrectly, Lower is better

Loading chart...

Word Error Rate: Percentage of words incorrect in the transcription. Evaluation updated June 2024 to 5,000 test samples.

Artificial Analysis' independent evaluation is based on Common Voice v16.1, Mozilla's leading open-source speech to text dataset. Further detail present on methodology page.

Speed

Back to Navigation

Speed Factor

Speed factor: Input audio seconds transcribed per second, Higher is better

Loading chart...

Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.

Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).

Speed Factor, Variance

Speed factor: Input audio seconds transcribed per second, Results by percentile, Higher is better

Median, Other points represent 5th, 25th, 75th, 95th Percentiles respectively

Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.

Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).

Boxplot: Shows variance of measurements

Speed Factor, Over Time

Speed factor: Input audio seconds transcribed per second, Higher is better

Whisper (L, v2), OpenAI

Azure AI Speech Service

Enhanced

Wizper (L, v3), fal.ai

Whisper (L, v3, Turbo), Groq

Whisper (L, v3), Fireworks

Universal-2

Amazon Transcribe

Nova-3

Chirp 2

Scribe

GPT-4o Transcribe

GPT-4o Mini Transcribe

Whisper (L, v3), SambaNova

Granite 3.3 8B, IBM

Parakeet RNNT 1.1B

Voxtral Small

Speed Factor: Audio file seconds transcribed per second of processing time. Higher factor indicates faster transcription speed.

Artificial Analysis' measurements are based on a audio duration of 10 minutes. Speed Factor may vary for other durations, particuarly for very short durations (under 1 minute).

Over time measurement: Median measurement per day, based on 8 measurements each day at different times. Labels represent start of week's measurements.

Price

Back to Navigation

Price

Price: USD per 1000 minutes of audio, Lower is better

Loading chart...

Price: Cost in USD per 1000 minutes of audio transcribed. Reflects the pricing model of the transcription service or software.

For providers which do not price based on audio duration and rather on processing time (incl. Replicate, fal), we have calculated an indicative per minute price based on processing time expected per minute of audio.Further detail present on methodology page.
Note: Groq chargers for a minimum of 10s per request.

Summary of key metrics & further information

Provider	Model	Whisper version	Word Error Rate (%)	Median Speed Factor	Price (USD per 1000 minutes)	Further Details
OpenAI	Whisper Large v2	large-v2	10.6%	32.1	6.00	Details
Microsoft Azure	Whisper Large v2	large-v2	10.6%	33.7	6.00	Details
fal.ai	Whisper Large v3	large-v3	10.3%	298.9	0.50	Details
Replicate	Incredibly Fast Whisper	large-v3	10.3%	64.0	1.49	Details
Replicate	Whisper Large v2	large-v2	11.2%	4.4	3.47	Details
Replicate	Whisper Large v3	large-v3	10.3%	2.2	4.23	Details
Replicate	WhisperX	large-v3	10.9%	7.5	1.09	Details
Replicate	Whisper (M)	medium	12.8%		2.68	Details
Replicate	Whisper (S)	small	17.0%		1.37	Details
Groq	Whisper Large v3	large-v3	10.3%	332.8	1.85	Details
Groq	Distil-Whisper		13.0%	410.3	0.33	Details
fal.ai	Whisper Large v3	large-v3	10.3%	134.8	1.15	Details
Deepinfra	Whisper Large v3	large-v3	10.3%	100.1	0.45	Details
Groq	Whisper Large v3 Turbo	v3 Turbo	12.0%	339.6	0.67	Details
Fireworks	Whisper Large v3	large-v3	11.2%	285.4	1.00	Details
Fireworks	Whisper Large v3 Turbo	v3 Turbo	13.7%	497.7	1.00	Details
SambaNova	Whisper-Large-v3	large-v3	10.8%	40.2	1.67	Details
Together.ai	Whisper Large v3	large-v3	10.3%	182.2	1.50	Details
AssemblyAI	Universal-1		8.7%	86.0	6.17	Details
AssemblyAI	Nano		12.7%	81.1	2.00	Details
Speechmatics	Standard		12.6%	17.7	13.33	Details
Speechmatics	Enhanced		8.1%	17.6	6.70
Microsoft Azure	Azure AI Speech Service		12.6%		16.67	Details
Deepgram	Nova-2		15.1%	163.7	4.30	Details
Deepgram	Base		26.1%	197.6	12.50	Details
Gladia	Gladia	whisper-v2-variant	12.9%	26.2	10.20	Details
Amazon Bedrock	Amazon Transcribe		11.2%	20.4	24.00	Details
Fish Audio	Fish Speech to Text		19.1%		0.00	Details
Rev AI	Rev AI		0.0%		20.00	Details
Google	Chirp 2		9.8%	19.7	16.00	Details
Google	Chirp		12.4%	13.7	16.00	Details
ElevenLabs	Scribe		7.7%	46.2	6.67	Details
Google	Gemini 2.0 Flash		14.9%	54.7	1.40	Details
Google	Gemini 2.0 Flash Lite		14.5%	59.2	0.19
OpenAI	GPT-4o Transcribe		8.9%	36.0	6.00	Details
OpenAI	GPT-4o Mini Transcribe		13.2%	44.3	3.00
IBM	Granite 3.3 8B		8.4%		0.00	Details
Replicate	Parakeet RNNT 1.1B		6.1%	6.5	1.91	Details
Mistral	Voxtral Mini		10.4%	66.3	1.00	Details
Mistral	Voxtral Small		8.8%	55.6	4.00	Details