语音到语音基准测试方法论
概览
我们当前的语音到语音基准测试针对原生音频模型——即支持原生音频输入与输出的模型——在两个质量维度上进行评测:语音推理与对话动态。
Artificial Analysis Speech to Speech Index
Artificial Analysis Speech to Speech Index 是针对原生音频模型的加权平均分数。它综合了语音推理、对话动态与智能体表现三方面的结果。模型必须在全部三个数据集上都有有效结果,才会出现在该指数中。
当前三个数据集的权重相同:语音推理(Big Bench Audio)33.3%、对话动态(Full Duplex Bench)33.3%、智能体表现(𝜏-Voice)33.3%。
语音推理
概览
我们的语音推理基准测试评估原生音频模型回答推理类问题的能力。
原生音频模型会收到一个输入音频文件,并需要生成一段输出音频。评判模型会获得候选答案、官方答案和原始问题作为上下文,并被提示将候选答案标注为正确或错误。
数据集:Big Bench Audio
原生音频到音频模型的出现,为增强语音智能体的能力、简化工作流程带来了令人振奋的机会。然而,关键在于评估这种简化是否以牺牲模型性能为代价,或者是否引入了其他权衡。
为帮助回答这个问题,我们发布了 Big Bench Audio——一个用于对原生音频模型性能进行基准测试的新数据集。
Big Bench Audio 包含 1,000 个音频文件,对应一系列旨在测试模型智能的问题。这些问题基于 Big Bench Hard 数据集的四个类别(每个类别 250 个问题),并使用 Artificial Analysis 文本转语音竞技场中排名靠前的文本转语音模型的 23 个合成音色生成。
问题类别
形式谬误(250 个问题): 判断一个以非形式方式陈述的论证,能否从给定的上下文中在逻辑上推导出来。
示例:First of all, everyone who is a close friend of Glenna is a close friend of Tamara, too. Next, whoever is neither a half-sister of Deborah nor a workmate of Nila is a close friend of Glenna. Hence, whoever is none of this: a half-sister of Deborah or workmate of Nila, is a close friend of Tamara. Is the argument deductively valid or invalid?
导航(250 个问题): 判断一系列导航步骤是否会让智能体回到起点。
示例:If you follow these instructions, do you return to the starting point? Take 10 steps. Turn around. Take 4 steps. Take 6 steps. Turn around.
物体计数(250 个问题): 在给定的一组物品中,统计某一特定类别物品的数量。
示例:I have three blackberries, two strawberries, an apple, three oranges, a nectarine, a grape, a peach, a banana, and a plum. How many fruits do I have?
谎言之网(250 个问题): 判断以自然语言应用题形式表述的布尔函数的真值。
示例:Leda tells the truth. Alexis says Leda lies. Sal says Alexis lies. Phoebe says Sal tells the truth. Gwenn says Phoebe tells the truth. Does Gwenn tell the truth?
为了能够评估使用原生语音到语音模型所带来的各种权衡,我们在 Big Bench Audio 上测试了多种不同的配置。要进一步了解 Big Bench Audio,请查看这篇文章,或自行下载该数据集。
对话动态
概览
我们的对话动态基准测试评估原生音频模型处理真实对话行为的能力——这些交互在人类对话中自然发生,但语音模型往往难以正确应对。
该基准测试由 Artificial Analysis 基于 Full Duplex Bench v1(Lin et al., 2025)和 Full Duplex Bench v1.5(Lin et al., 2025)的一个子集实现;这两项基准测试系统性地评估全双工口语对话模型的关键交互行为。
指标
来自 Full Duplex Bench v1:
- 停顿处理: 模型在用户自然停顿期间正确地没有打断的样本所占百分比。用于评估模型是否认识到说话者仍然掌握着话语权。
- 话轮转换: 模型在合适时机正确接过对话话轮的样本所占百分比。衡量模型检测话轮边界并及时作出回应的能力。
来自 Full Duplex Bench v1.5:
- 用户打断处理: 模型正确处理用户打断的样本所占百分比——即回应对话中途提出的问题或话题变化。
- 反馈语处理: 当播放 "yeah"、"alright" 或 "mm-hmm" 这类反馈语(backchannel)时,模型正确地继续其回应、而不是将其视为一个新话轮的样本所占百分比。
智能体表现(𝜏-Voice)
概览
我们的智能体表现基准测试评估语音到语音模型端到端完成真实客服任务的能力。该基准测试衡量多轮指令遵循能力、在一次完整交互中协助模拟客户的能力,以及在模拟客服系统上成功调用工具的能力。
该基准测试由 Artificial Analysis 基于 𝜏-Voice(Ray, Dhandhania, Barres & Narasimhan, 2026)实现——这是 Sierra 推出的一项基准测试,用于在真实世界各领域中评估全双工语音智能体在有工具和系统上下文支撑的客服任务上的表现。
指标
- 任务完成率(pass@1): 模型正确解决客户问题的场景所占比例。每个分数为三次独立试验的平均值。被测模型被提示扮演客服智能体,可以使用领域专用工具和一份政策文件。每个场景都有唯一有效的数据库最终状态;评测将最终状态与该参考真值进行比较。
我们使用基础任务集在三个领域进行评测:
- 航空(50 个场景):例如更改航班、在政策限制下改签
- 零售(114 个场景):例如对某笔扣款提出异议、处理退货
- 电信(114 个场景):例如解决账单问题、排查服务故障
语音人设
客户语音使用 ElevenLabs 生成,所依据的提示词改编自 Sierra 公开的 𝜏-Voice 实现。我们使用两个代表标准英语说话者的对照人设,以及五个代表不同口音和说话者特征的常规人设。
对照
Matt Delaney: 来自美国中西部的中年白人男性,冷静而有礼。
提示词: You are a middle-aged white man from the American Midwest. You always behave as if you are speaking out loud in a real-time conversation with a customer service agent. You are calm, clear, and respectful but also human. You sound like someone who's trying to be helpful and polite, even when you're slightly frustrated or in a hurry. You value efficiency but never sound robotic. You sometimes use contractions, informal phrasing, or small filler phrases ("yeah," "okay," "honestly," "no worries") to keep things natural. You sometimes repeat words or self-correct mid-sentence, just like someone thinking aloud. You sometimes ask polite clarifying questions or offer context ("I tried this earlier," "I'm not sure if that helps"). You rarely use formal, business-like or stiff language ("considerable," "retrieve," "representative"). You rarely speak in perfect full sentences unless the situation calls for it. Instead, you speak like a real person having a practical, respectful conversation.
Lisa Brenner: 来自郊区、四十七八岁的白人女性,紧张而不耐烦。
提示词: You are a white woman in your late 40s from a suburban area. You always speak as if you are talking out loud to a customer service agent who is already wasting your time. You're not openly hostile (yet), but you are tense, impatient, and clearly annoyed. You act like this issue should have been resolved the first time, and the fact that you're following up is unacceptable. You often sound clipped, exasperated, or sarcastically polite. You frequently use emphasis ("I already did that"), rhetorical questions ("Why is this still an issue?"), and escalation language ("I'm not doing this again," "I want someone who can actually help"). You expect fast results and get irritated when things are repeated. You often mention how long you've been waiting or how many times you've called. You sometimes threaten escalation but without yelling. You never sound relaxed. You never use slow, reflective speech. You never thank the agent unless something gets resolved.
常规
Mildred Kaplan: 八十出头的白人老年女性,需要他人帮助处理技术问题。
提示词: You are an elderly white woman in your early 80s calling customer service for help with something your grandson or neighbor usually does.
Arjun Roy: 来自达卡、三十五岁上下的孟加拉男性,冷静而直接,带有浓重的孟加拉口音。
提示词: A Bengali man from Dhaka, Bangladesh in his mid-30s calling customer service about a billing issue. His English carries a strong Bengali accent with soft consonants and soft d and r sounds. He speaks in a calm, patient tone but is direct and purposeful, focused on resolving the issue efficiently. His pacing is slow, distracted with a warm yet firm timbre. The speech sounds like it is coming from far away.
Wei Lin: 来自四川、二十七八岁的中国女性,语气轻快、就事论事,带有浓重的四川普通话口音。
提示词: A Chinese woman in her late 20s from Sichuan, calling customer service about a credit card billing issue. She speaks English with a thick Sichuan Mandarin accent. She sounds upbeat, matter-of-fact, and distracted. Her tone is firm but polite, with fast pacing and smooth timbre. Ok audio quality.
Mamadou Diallo: 三十五岁上下的塞内加尔男性,语速匆忙,带有浓重的法语口音。
提示词: A Senegalese man whose first language is French, in his mid-30s, calling customer service about a billing issue. He speaks English with a strong French accent. His tone is hurried, slightly annoyed, and matter-of-fact, as if he's been transferred between agents and just wants the problem fixed.
Priya Patil: 三十出头的马哈拉施特拉女性,专注而直接,带有浓重的马哈拉施特拉口音。
提示词: A woman in her early 30s from Maharashtra, India, calling customer support from her mobile phone. She speaks Indian English with a strong Maharashtrian accent with noticeable regional intonation and rhythm. Her tone is slightly annoyed and hurried, matter-of-fact, and focused on getting the issue resolved quickly. Her voice has medium pitch, firm delivery, short sentences, and faint background room tone typical of a phone call.
价格
- 每小时输入音频价格: 发送给 API 的请求/消息中所包含音频的总成本(美元)。
- 每小时输出音频价格: 由模型生成(从 API 接收)的音频的总成本(美元)。
速度
- 首个音频延迟(TTFA): 生成音频输出的第一个 token 平均所需的秒数,在 Big Bench Audio 问题集上测得。TTFA 是语音智能体应用中感知响应速度的关键指标。
版本历史
Artificial Analysis Speech to Speech Index v1.0
2026 年 6 月至今
- 推出 Artificial Analysis Speech to Speech Index:一项加权平均分数,要求同时具备语音推理、对话动态与智能体表现的结果。
- 三个数据集权重相同:语音推理(Big Bench Audio)33.3%、对话动态(Full Duplex Bench)33.3%、智能体表现(𝜏-Voice)33.3%。
Big Bench Audio (BBA) v1.2
2026 年 5 月至今
- 更新了评判模型与评分框架,使其能更可靠地识别冗长回答中的正确最终答案,包括模型在给出正确答案之前先讨论了错误选项的情况。
Big Bench Audio (BBA) v1.1
2026 年 3 月至 2026 年 5 月
- 准确率按 1,000 个问题中回答正确的数量计算,其中包含模型未作答的问题。
- 使用 Claude Sonnet 4.6 作为评判模型。
Big Bench Audio (BBA) v1.0
2024 年 12 月至 2026 年 3 月
- 准确率按非错误回答中回答正确的比例计算,因此模型不会因未作答的问题而被扣分。
- 使用 Claude Sonnet 3.5 作为评判模型。