Pronunciation Robustness Overview

The Pronunciation Robustness benchmark measures whether Text to Speech models pronounce challenging text correctly, including words whose reading depends on context, shorthand that must be expanded, exact sequences like codes and emails, and standalone terms like brand and place names. Human reviewers listen to each clip and judge highlighted spans as pronounced correctly or not.

How the Pronunciation Robustness Benchmark Works

Every model reads the same set of sentences. Each sentence carries one or more highlighted spans that are hard to pronounce, with accepted readings defined up front. Human reviewers listen and judge each span.

  1. 01

    A fixed sentence set

    Sentences across four categories, each with highlighted spans and pre-agreed accepted pronunciations. Every model reads the identical text with a pinned voice.

  2. 02

    Every model reads every sentence

    Audio is generated once per model with a consistent voice, then served to reviewers in random order without model names. Sentences are sent as written, with no normalization on our side; each model uses its default text normalization setting where it has one.

  3. 03

    Human reviewers judge span(s) within sentence

    Reviewers hear the clip, see the highlighted span and its accepted readings, and answer whether it was pronounced naturally and correctly with either of Yes, No, or I could not tell. At least three third party reviewers judge over 95% of clips for each published model. A model's score is the share of span judgements answered Yes, out of Yes plus No; "I could not tell" answers are excluded.

Sessions include attention-check clips with known answers. Reviewers who miss any answered check are excluded entirely. We publish results for a model once at least 95% of the samples in the set are covered by three or more reviewers.

What reviewers see

Was the highlighted part pronounced naturally and correctly in this sentence?

Press Ctrl+Z to undo the last edit.

Accepted pronunciations: Ctrl+Z

Count it as correct if it matches any option below.

  • control zee
  • control zed
  • control plus zee
  • control plus zed
YesNoI could not tell

Categories of Pronunciation Challenges Covered

Contextually Appropriate (Contextual Disambiguation)

Words spelled the same but read differently depending on context

e.g. a wound that is bandaged vs. a bandage that is wound

Expanding Shorthand (Text Normalization)

Numbers, dates, units and notation read out naturally

e.g. 6'2", Chapter XVII, or 1 tsp of sugar

Preserving Exact Sequences (Sequence Fidelity)

Codes, paths, emails and identifiers spoken exactly

e.g. .env.local or a.chen@ucsf.edu

Standalone Terms (Term Pronunciation)

Brand, place and technical names pronounced correctly

e.g. Arkansas, façade, or genre

Example Clips by Model

The same sentences, read by every model on the index. Select a model to hear its clips for each category of pronunciation challenge.

Example clips below are a fixed public subset of the benchmark set. The full set is withheld to keep the benchmark uncontaminated.

All example sentences and audio below are for the selected model.

Contextually Appropriate (Contextual Disambiguation)

Words spelled the same but read differently depending on context

St. Mary’s is on Church St.

St.Contextual abbreviation

Accepted:

  • Saint
St.Contextual abbreviation

Accepted:

  • Street

That excuse does not excuse the delay.

excuseFinal-consonant contrast

Accepted:

  • noun ends with an s sound
excuseFinal-consonant contrast

Accepted:

  • verb ends with a z sound

The instructions say to wait 30 sec. before reading sec. 4.

30 sec.Contextual abbreviation

Accepted:

  • thirty seconds
sec. 4Contextual abbreviation

Accepted:

  • section four

Expanding Shorthand (Text Normalization)

Numbers, dates, units and notation read out naturally

Median latency was 12 ms, reported as p50.

12 msMeasurements and units

Accepted:

  • twelve milliseconds
p50Measurements and units

Accepted:

  • P fifty
  • the fiftieth percentile

Take the elevator to the 3rd Fl and turn left.

3rd FlNumbers, ranges and other shorthand

Accepted:

  • third floor

The Class of '09 reunion is on 9/9, of all days.

'09Dates and times

Accepted:

  • oh-nine
  • two thousand nine
  • two thousand and nine
9/9,Dates and times

Accepted:

  • September ninth
  • ninth of September

Preserving Exact Sequences (Sequence Fidelity)

Codes, paths, emails and identifiers spoken exactly

Press Ctrl+Z to undo the last edit.

Ctrl+ZCodes and identifiers

Accepted:

  • control zee
  • control zed
  • control plus zee
  • control plus zed

We're on iOS 26.1.2 here and macOS 15.7.1 on the laptops.

26.1.2Versions and network values

Accepted:

  • two six dot one dot two
  • two six point one point two
  • twenty-six dot one dot two
  • twenty-six point one point two
15.7.1Versions and network values

Accepted:

  • one five dot seven dot one
  • one five point seven point one
  • fifteen dot seven dot one
  • fifteen point seven point one

Dial 999 in the UK, 112 in the EU and 911 here.

999Codes and identifiers

Accepted:

  • nine nine nine
  • triple nine
112Codes and identifiers

Accepted:

  • one one two
911Codes and identifiers

Accepted:

  • nine one one

Standalone Terms (Term Pronunciation)

Brand, place and technical names pronounced correctly

The river continues south through Arkansas.

ArkansasPlace name

Accepted:

  • AR-kan-saw

The paper was typeset in LaTeX.

LaTeXSoftware term

Accepted:

  • LAY-tek
  • LAH-tek

Workers restored the stone façade above the entrance.

façadeBorrowed term

Accepted:

  • fuh-SAHD