Text to Speech Benchmarking Methodology

Scope & Background

Artificial Analysis performs benchmarking on Text to Speech models delivered via serverless API endpoints. This page describes our Text to Speech benchmarking methodology, including both our quality benchmarking and performance benchmarking. We consider Text to Speech endpoints to be serverless when customers only pay for usage, not a fixed rate for access.

For both our performance benchmarking and within the Speech Arena, our focus is reflecting the end-user experience of users using the serverless APIs. We focus on benchmarking the time to receive the audio file locally. Where the API response is a URL rather than bytes, we include the time of downloading the file in our response time measurement. Our approach is to use the standard implementation of provider APIs as suggested by each provider's documentation. Where an option on the provider's API, we standardize the sample rate of audio to 22.05 kHz.

Key Metrics

We use the following metrics to track quality, performance and price for Text to Speech models.

Quality Elo

Relative Elo score of the models as determined by responses from users in the Artificial Analysis Text to Speech Arena.

Some models may not be shown due to not yet having enough votes. We use a similar Linear Regression model, similar to how LMSYS calculates Elo scores for Chatbot Arena.

Price per 1M Characters

All TTS models are reported as price per 1M characters of input text. Where providers do not price per character directly, we derive an equivalent:

Per character: Listed rate used directly
Subscription plans: Effective rate from the lowest-cost plan, annual where available, that includes at least 1M characters, assuming 80% utilization
Token-based: Derived using batch pricing, converting characters to input tokens and estimating audio output duration
Output duration: Converted using an assumed speaking rate of 150 words per minute, approximately 825 characters per minute
Per byte: 1 UTF-8 byte is approximately 1 character for English text
Inference time: Estimated from benchmark runs using ~25 texts of ~500 characters
Open weights: No commercial API pricing; listed without price
Speech to Speech models: Derived from the cost to generate speech for 20 fixed TTS arena prompts, based on reported text input, output audio, output text, and separately exposed reasoning/thinking tokens. Output text tokens are included when they are reported as part of the audio response. We exclude caching effects because prompts are run independently as single-turn requests, so their impact on pricing is negligible.

When reporting price, we do not include temporary discounts.

Generation Time

Median time the provider takes to generate a single audio clip with ~500 input characters, calculated over the past 14 days of measurements.

Generation Time includes downloading the audio clip from the provider where a URL is provided rather than an audio response. This is to reflect the end-user latency of receiving a generated audio clip and as URLs can be generated prior to audio completion. Audio clips are generated at batch size of 1 where relevant.

Benchmarking is conducted 4 times daily at random times each day. For each benchmarking evaluation we select a single random voice for each model. A unique prompt of ~500 characters is used for each generation.

Controlled Voices

The Controlled Voice Arena uses voice cloning to standardize the voices used to evaluate each Text to Speech model. By comparing models using the same reference voices, it separates listener preference for a particular voice from broader aspects of model quality, such as speech naturalness, audio quality, pronunciation, pacing, and prosody. This enables a more like-for-like comparison of the underlying Text to Speech models.

The Controlled Voice Arena complements our Provider Voice Arena, where each model is evaluated using a representative set of its own publicly available voices. The reference set consists of 8 professionally recorded voices: 2 US female, 2 US male, 2 UK female, and 2 UK male. Select a controlled voice below to listen to its reference recording.

Provider Voices

The Provider Voice Arena evaluates each Text to Speech model using a representative set of the voices made available by that model's provider. This complements the Controlled Voice Arena, where models are compared using the same cloned reference voices. Provider voices reflect how each model is typically experienced by users through its own public API, product interface, or documentation.

For each model, we select multiple provider-offered voices to make the comparison representative and fair. We select 2 voices for each combination of male and female, and US and UK accents, for 8 voices in total. Where a gender and accent combination is not available, we exclude that combination from evaluation in the Provider Voice Arena.

Voices are selected based on their prominence in provider interfaces and documentation, excluding voices that are not neutral in nature (for example, highly stylized regional accents). Model creators may also request that we use specific voices where many are available. Where provider voices are not provided, as is typically the case for open source models, we use voice clips from professional voice actors as source files for generating speech (see Controlled Voices above for samples).

Select a model creator below to see the provider voices currently used for each of their models.

Model and Provider Inclusion Criteria

Our objective is to analyze and compare popular and high-performing Text to Speech models and providers to support end-users in choosing which to use. As such, we apply an 'industry significance' and competitive performance test to evaluate the inclusion of new models and providers. We are in the process of refining these criteria and welcome any feedback and suggestions. To suggest models or providers, please contact us via the contact page.

Statement of Independence

Benchmarking is conducted with strict independence and objectivity. No compensation is received from any providers for listing or favorable outcomes on Artificial Analysis.