语音到语音基准测试方法论
概览
我们当前的语音到语音基准测试针对原生音频模型——即支持原生音频输入与输出的模型——在两个质量维度上进行评测：语音推理与对话动态。
Artificial Analysis Speech to Speech Index
Artificial Analysis Speech to Speech Index 是针对原生音频模型的均等加权分数。它综合了语音推理、智能体表现、Arena 偏好和任务成功率四方面的结果。模型必须在全部四个组件上都有有效结果，才会出现在该指数中。
当前四个组件的权重均等：语音推理（Big Bench Audio）25%、智能体表现（𝜏-Voice）25%、Arena 偏好（Arena Score）25%、任务成功率 25%。
语音推理
概览
我们的语音推理基准测试评估原生音频模型回答推理类问题的能力。
原生音频模型会收到一个输入音频文件，并需要生成一段输出音频。评判模型会获得候选答案、官方答案和原始问题作为上下文，并被提示将候选答案标注为正确或错误。
数据集：Big Bench Audio
原生音频到音频模型的出现，为增强语音智能体的能力、简化工作流程带来了令人振奋的机会。然而，关键在于评估这种简化是否以牺牲模型性能为代价，或者是否引入了其他权衡。
为帮助回答这个问题，我们发布了 Big Bench Audio——一个用于对原生音频模型性能进行基准测试的新数据集。
Big Bench Audio 包含 1,000 个音频文件，对应一系列旨在测试模型智能的问题。这些问题基于 Big Bench Hard 数据集的四个类别（每个类别 250 个问题），并使用 Artificial Analysis 文本转语音竞技场中排名靠前的文本转语音模型的 23 个合成音色生成。
问题类别
形式谬误（250 个问题）： 判断一个以非形式方式陈述的论证，能否从给定的上下文中在逻辑上推导出来。
示例：First of all, everyone who is a close friend of Glenna is a close friend of Tamara, too. Next, whoever is neither a half-sister of Deborah nor a workmate of Nila is a close friend of Glenna. Hence, whoever is none of this: a half-sister of Deborah or workmate of Nila, is a close friend of Tamara. Is the argument deductively valid or invalid?
导航（250 个问题）： 判断一系列导航步骤是否会让智能体回到起点。
示例：If you follow these instructions, do you return to the starting point? Take 10 steps. Turn around. Take 4 steps. Take 6 steps. Turn around.
物体计数（250 个问题）： 在给定的一组物品中，统计某一特定类别物品的数量。
示例：I have three blackberries, two strawberries, an apple, three oranges, a nectarine, a grape, a peach, a banana, and a plum. How many fruits do I have?
谎言之网（250 个问题）： 判断以自然语言应用题形式表述的布尔函数的真值。
示例：Leda tells the truth. Alexis says Leda lies. Sal says Alexis lies. Phoebe says Sal tells the truth. Gwenn says Phoebe tells the truth. Does Gwenn tell the truth?
为了能够评估使用原生语音到语音模型所带来的各种权衡，我们在 Big Bench Audio 上测试了多种不同的配置。要进一步了解 Big Bench Audio，请查看这篇文章，或自行下载该数据集。
对话动态
概览
我们的对话动态基准测试评估原生音频模型处理真实对话行为的能力——这些交互在人类对话中自然发生，但语音模型往往难以正确应对。
该基准测试由 Artificial Analysis 基于 Full Duplex Bench v1（Lin et al., 2025）和 Full Duplex Bench v1.5（Lin et al., 2025）的一个子集实现；这两项基准测试系统性地评估全双工口语对话模型的关键交互行为。
指标
来自 Full Duplex Bench v1：
- 停顿处理： 模型在用户自然停顿期间正确地没有打断的样本所占百分比。用于评估模型是否认识到说话者仍然掌握着话语权。
- 话轮转换： 模型在合适时机正确接过对话话轮的样本所占百分比。衡量模型检测话轮边界并及时作出回应的能力。
来自 Full Duplex Bench v1.5：
- 用户打断处理： 模型正确处理用户打断的样本所占百分比——即回应对话中途提出的问题或话题变化。
- 反馈语处理： 当播放 "yeah"、"alright" 或 "mm-hmm" 这类反馈语（backchannel）时，模型正确地继续其回应、而不是将其视为一个新话轮的样本所占百分比。
智能体表现（𝜏-Voice）
概览
我们的智能体表现基准测试评估语音到语音模型端到端完成真实客服任务的能力。该基准测试衡量多轮指令遵循能力、在一次完整交互中协助模拟客户的能力，以及在模拟客服系统上成功调用工具的能力。
该基准测试由 Artificial Analysis 基于 𝜏-Voice（Ray, Dhandhania, Barres & Narasimhan, 2026）实现——这是 Sierra 推出的一项基准测试，用于在真实世界各领域中评估全双工语音智能体在有工具和系统上下文支撑的客服任务上的表现。
指标
- 任务完成率（pass@1）： 模型正确解决客户问题的场景所占比例。每个分数为三次独立试验的平均值。被测模型被提示扮演客服智能体，可以使用领域专用工具和一份政策文件。每个场景都有唯一有效的数据库最终状态；评测将最终状态与该参考真值进行比较。
我们使用基础任务集在三个领域进行评测：
- 航空（50 个场景）：例如更改航班、在政策限制下改签
- 零售（114 个场景）：例如对某笔扣款提出异议、处理退货
- 电信（114 个场景）：例如解决账单问题、排查服务故障
语音人设
客户语音使用 ElevenLabs 生成，所依据的提示词改编自 Sierra 公开的 𝜏-Voice 实现。我们使用两个代表标准英语说话者的对照人设，以及五个代表不同口音和说话者特征的常规人设。
对照
Matt Delaney： 来自美国中西部的中年白人男性，冷静而有礼。
提示词： You are a middle-aged white man from the American Midwest. You always behave as if you are speaking out loud in a real-time conversation with a customer service agent. You are calm, clear, and respectful but also human. You sound like someone who's trying to be helpful and polite, even when you're slightly frustrated or in a hurry. You value efficiency but never sound robotic. You sometimes use contractions, informal phrasing, or small filler phrases ("yeah," "okay," "honestly," "no worries") to keep things natural. You sometimes repeat words or self-correct mid-sentence, just like someone thinking aloud. You sometimes ask polite clarifying questions or offer context ("I tried this earlier," "I'm not sure if that helps"). You rarely use formal, business-like or stiff language ("considerable," "retrieve," "representative"). You rarely speak in perfect full sentences unless the situation calls for it. Instead, you speak like a real person having a practical, respectful conversation.
Lisa Brenner： 来自郊区、四十七八岁的白人女性，紧张而不耐烦。
提示词： You are a white woman in your late 40s from a suburban area. You always speak as if you are talking out loud to a customer service agent who is already wasting your time. You're not openly hostile (yet), but you are tense, impatient, and clearly annoyed. You act like this issue should have been resolved the first time, and the fact that you're following up is unacceptable. You often sound clipped, exasperated, or sarcastically polite. You frequently use emphasis ("I already did that"), rhetorical questions ("Why is this still an issue?"), and escalation language ("I'm not doing this again," "I want someone who can actually help"). You expect fast results and get irritated when things are repeated. You often mention how long you've been waiting or how many times you've called. You sometimes threaten escalation but without yelling. You never sound relaxed. You never use slow, reflective speech. You never thank the agent unless something gets resolved.
常规
Mildred Kaplan： 八十出头的白人老年女性，需要他人帮助处理技术问题。
提示词： You are an elderly white woman in your early 80s calling customer service for help with something your grandson or neighbor usually does.
Arjun Roy： 来自达卡、三十五岁上下的孟加拉男性，冷静而直接，带有浓重的孟加拉口音。
提示词： A Bengali man from Dhaka, Bangladesh in his mid-30s calling customer service about a billing issue. His English carries a strong Bengali accent with soft consonants and soft d and r sounds. He speaks in a calm, patient tone but is direct and purposeful, focused on resolving the issue efficiently. His pacing is slow, distracted with a warm yet firm timbre. The speech sounds like it is coming from far away.
Wei Lin： 来自四川、二十七八岁的中国女性，语气轻快、就事论事，带有浓重的四川普通话口音。
提示词： A Chinese woman in her late 20s from Sichuan, calling customer service about a credit card billing issue. She speaks English with a thick Sichuan Mandarin accent. She sounds upbeat, matter-of-fact, and distracted. Her tone is firm but polite, with fast pacing and smooth timbre. Ok audio quality.
Mamadou Diallo： 三十五岁上下的塞内加尔男性，语速匆忙，带有浓重的法语口音。
提示词： A Senegalese man whose first language is French, in his mid-30s, calling customer service about a billing issue. He speaks English with a strong French accent. His tone is hurried, slightly annoyed, and matter-of-fact, as if he's been transferred between agents and just wants the problem fixed.
Priya Patil： 三十出头的马哈拉施特拉女性，专注而直接，带有浓重的马哈拉施特拉口音。
提示词： A woman in her early 30s from Maharashtra, India, calling customer support from her mobile phone. She speaks Indian English with a strong Maharashtrian accent with noticeable regional intonation and rhythm. Her tone is slightly annoyed and hurried, matter-of-fact, and focused on getting the issue resolved quickly. Her voice has medium pitch, firm delivery, short sentences, and faint background room tone typical of a phone call.
Speech Agent Arena
概览
Speech Agent Arena 是一项隐藏模型身份的偏好基准，评估参与者在实时语音对话中更喜欢哪个原生音频模型。它通过让真人参与者完成真实场景任务，衡量端到端的对话体验，补充我们的自动化基准。
每一轮中，参与者会收到一个场景，分别与两个身份隐藏的模型完成任务，并在两次通话结束后必须选择整体上更喜欢的模型。参与者还会回答诊断性问题，我们将这些回答与整体偏好投票分开记录。
指标
- 偏好 Elo：偏好 Elo 使用 Bradley–Terry 最大似然估计，分别在全部场景、智能体场景和非智能体场景上计算。近似 95% 置信区间采用与 TTS Arena 相同的基于海森矩阵的方法，并以 GPT Realtime 1.5 为锚点，将其评分固定为 1000 Elo。
- 任务成功率：任务成功率是指在符合评估条件的对话中，模型正确执行完成任务所需的最终工具调用（一次或多次）的对话百分比。符合评估条件的对话数等于成功数加上模型失败数；参与者偏离任务的情况及无法核实的案例会在计算前排除。
数据集 / 评估设置
Arena 包含 35 个场景：15 个使用工具调用的智能体场景，以及 20 个不使用工具的非智能体场景。
- 智能体场景（15 个）：模型会获得与完成指定任务相关的工具。
示例：
- 为一位新患者预约周二上午 9:30 的牙科检查；如果该时段已满，则预约最早可用的上午时段。
- 订购两种不同的披萨和一份配菜并配送上门，含配送费的总价须低于 45 美元。
- 非智能体场景（20 个）：模型不使用工具，而是依据系统提示词提供的信息完成对话。
示例：
- 询问游泳池周日的开放时间、家庭门票价格以及是否提供毛巾。
- 询问瑜伽初学者课程、课程和会员价格，以及需要携带的物品。
请参阅 Speech Agent Arena 概览，了解一轮完整的 Arena 示例、模型输入、工具模式和对话示例。
任务成功率
步骤 1：参与者是否符合评估条件。评审模型 1 接收指定场景、转录文本和所需最终调用的定义。它会检查参与者是否尝试完成指定任务、是否基本保持在任务范围内、是否表达了可评估的最终请求或接受了可评估的结果，以及是否给了模型合理的行动机会。只有符合评估条件的对话才会进入步骤 2。
You assess whether a participant-side conversation is eligible for a Speech to Speech Task Completion Tool Call evaluation.
This evaluation measures whether the model made the correct final task-completing tool call or calls for the participant's actual final request. It does not measure whether the participant mechanically completed every conversational instruction on the scenario card.
You receive:
- the participant's assigned scenario and objectives;
- the full conversation transcript, with participant and model turns identified;
- the required final task-completing tool call or calls, their required counts, and schemas.
Classify the conversation as:
- eligible: The participant attempted the assigned task, remained materially within it, communicated a final request or accepted outcome that can be assessed against the required final call, and gave the model a reasonable opportunity to act. Natural paraphrases, clarifications, changes of mind, implicit but clear authorization, and accepted alternatives are allowed.
- participant_deviation: The participant abandoned or materially replaced the assigned task; contradicted a scenario requirement in a way that materially changes the required final call, its count, or its arguments; clearly refused information needed for the final call after the model reasonably requested it; or ended without ever requesting, accepting, or authorizing the task-completing action.
- unverifiable: The transcript is missing, corrupted, contradictory, or ends at a point where the participant's final requested action or the model's reasonable opportunity to act cannot be determined. Use this only when the evidence cannot support either eligible or participant_deviation.
Use final-tool relevance as the decision boundary:
- An objective is eligibility-relevant only when following or violating it changes which final tool should be called, how many final calls are required, or a material final-call argument such as the action, item, quantity, identity, address, date, time, or accepted resolution. The mandatory transcription exception below governs unspelled participant names and addresses; do not use this general material-argument rule to override that exception.
- Do not make the participant ineligible for omitting a menu, price, availability, delivery-time, explanation, supporting lookup, read-back, or confirmation request when that omission does not change the required final call or its material arguments.
- Details used only by supporting tools are not eligibility requirements unless they also determine a material final-call argument.
- If the model failed to collect an execution detail needed to carry out a task-completing action the participant did request, misunderstood the participant, omitted a supporting step, or ended the call early after having a reasonable opportunity, keep the participant eligible. Model failures are assessed separately. This does not excuse a participant who never requested a scenario-required final item or action, or explicitly requested a materially conflicting one; those omissions or conflicts change the final call and are participant_deviation.
- If the model reasonably requested information required for the final call and the participant clearly refused or replaced it with contradictory information, classify participant_deviation.
- Do not require the participant to repeat information, correct the model's recap, request a read-back, or use the scenario card's exact wording.
- Mandatory precedence rule: an unspelled participant-name or address mismatch visible only by comparing the transcript with private scenario details must not cause participant_deviation or unverifiable. When the participant did not spell, correct, reject, or deliberately replace the assigned value, treat the mismatch as transcription uncertainty and keep the participant eligible. This rule overrides the general requirement that material final-call arguments match. Require exactness for explicitly spelled or corrected values, structured identifiers other than unspelled names or addresses, quantities, money, dates, and times, or when the conversation clearly shows that the participant deliberately changed the requested outcome. If another potentially material difference is genuinely impossible to resolve from the transcript, use unverifiable rather than assuming participant deviation.
- Treat natural changes of mind as eligible only when the final request still satisfies the scenario's explicit constraints on the task-completing action. A clear final request that contradicts such a constraint and changes a final-call argument is participant_deviation. Do not invent a stricter boundary for relative language such as "morning," "before lunch," or "later" when the scenario does not define one.
- Clear delegation is valid authorization. A participant may ask the model to choose the exact option, date, or time within stated constraints. Do not use unverifiable merely because a schema argument is absent or expressed as an inferable range such as "when it is dry"; if the model could reasonably infer it or should have clarified it, keep the participant eligible and leave the model's final-call handling to the next assessment.
- A scenario's explicit participant-facing conditional instruction is always eligibility-relevant when its trigger occurs, even if a required final tool call was already made or the participant's response would not change that call's schema arguments. When the model offers a partial or alternative resolution that materially affects the task outcome, the participant must accept, reject, push back, or otherwise respond as the scenario requires so their final accepted resolution is determinable. If the participant had a reasonable opportunity but ends or remains silent before doing so, classify participant_deviation. Do not mark the conversation eligible merely because the model already escalated the original request. Use unverifiable only when missing or cut-off evidence makes the participant's opportunity to respond unclear.
Assess participant conduct only. Do not decide whether the model actually called a tool correctly, and do not use the recorded tool result as the eligibility verdict.
Return exactly one JSON object with no markdown or additional keys:
{
"participant_eligibility": "eligible | participant_deviation | unverifiable",
"participant_eligibility_rationale": "One concise sentence identifying the decisive final-tool-relevant participant conduct."
}步骤 2a：所需的最终调用。对于符合评估条件的对话，评审模型 2 接收所需的最终调用和按时间顺序排列的完整工具调用记录。它会检查是否使用正确的工具、按正确的调用次数执行了所有所需的最终调用，且没有非预期的最终操作。辅助调用和 end_call 不计为任务完成。
步骤 2b：最终调用的参数。同一次评审模型 2 评估还会检查是否提供了模式要求的所有参数，以及这些参数是否符合参与者最终口头请求和对话上下文。允许操作效果等价的措辞以及不影响结果的转录差异。失败后得到纠正的尝试可以通过；缺少调用、参数错误或额外的最终操作则判为失败。
You assess task-completing tool calls for a Speech to Speech evaluation.
Decide whether the recorded tool trace contains the supplied required task-completing call or complete set of calls for what the participant finally requested.
You receive:
- the assigned scenario and objectives;
- the full conversation transcript;
- the required final tool call or calls, required call counts, and schemas;
- all recorded tool calls in chronological order, including arguments and results.
A result is success only when all of the following are true:
1. The model made every required final call, including the required number of calls when stated.
2. Each final call used the correct tool and supplied every schema-required argument.
3. The arguments matched the participant's final confirmed request and conversation context, including quantities, identity, constraints, corrections, changes of mind, and accepted alternatives.
Judge the arguments that were actually submitted. Do not repair, generalize, or silently infer missing content from the transcript. A material omission or distortion is model_failure even when the surrounding conversation was correct. For escalate_to_supervisor, both issue and customer_request must faithfully and completely represent the participant's actual problem and requested resolution; quantities and singular-versus-multiple requests matter.
Judge operational equivalence rather than character-perfect transcription. Allow minor spelling, capitalization, punctuation, and plausible transcription variations when they preserve the intended entity or meaning and do not change the tool outcome. Require exactness between the participant's final spoken request and the submitted call for explicitly spelled or corrected values, structured identifiers, numbers, dates, times, quantities, and addresses, or whenever a mismatch causes rejection or the wrong action. Do not require a transcript or submitted call to reproduce the private scenario card's spelling. If the transcript and submitted argument differ by a plausible transcription variation and the conversation provides no exact ground truth, do not treat that difference alone as model_failure; use unverifiable only when it is decisive and cannot be resolved. When separate tool evidence already proves model_failure, cite that decisive evidence rather than adding an uncertain spelling allegation.
The participant's final spoken request is the source of truth for final-call arguments. Compare addresses and other values between the conversation and the submitted final call, not against private scenario-card details. Do not penalize an error from a supporting call when the required final call or calls were made correctly. A supporting-call error affects the verdict only when it proves that a required final call used the wrong action or arguments, or prevented a required final call from being made.
When a required final call succeeds, an unspelled phonetic or orthographic variation in a participant or company name is not model_failure if the transcript and conversation make the intended entity clear and the variation does not cause the required final call to reject or act on the wrong entity. Do not use the scenario card's private spelling alone to turn such a variation into failure.
Use tool results as evidence, not as the verdict. A call that returns successfully with wrong arguments is a model_failure. A failed attempt followed by a correct successful call may still be success. A missing final call, wrong final tool, incomplete required call set, schema-invalid arguments, or request-mismatched arguments is model_failure.
When a required call count is stated, count distinct successful task outcomes rather than raw attempts. Failed calls do not count. A repeated call that the result identifies as an idempotent amendment or update to the same outcome does not create an additional outcome. Extra distinct successful final actions that the participant did not request are model_failure.
Use unverifiable only when the recorded transcript or tool trace is missing or contradictory enough that success versus model_failure cannot be determined. Do not invent backend state or assume an unrecorded call occurred.
Return exactly one JSON object with no markdown or additional keys:
{
"task_completion_tool_call_status": "success | model_failure | unverifiable",
"task_completion_tool_call_rationale": "One concise sentence identifying the decisive call evidence."
}发布规则
- 模型至少出场 100 次，且 95% 置信区间的半宽不超过 75 时，才会发布其整体结果。
- 我们不公开参与者的录音和转录文本。只有在确认获得录音同意，并审查和遮蔽所有个人信息后，才会展示示例。
我们使用 Wilson 得分区间报告各模型任务成功率的 95% 置信区间。参与者偏离任务的情况及无法核实的对话会在计算前排除。
上下界
lower = max(0, (centre − spread) / denominator)
upper = min(1, (centre + spread) / denominator)| 项 | 计算 |
|---|---|
| p | p = 成功数 / n |
| n | n = 成功数 + 模型失败数 |
| z | z = 1.96 |
| 分母 | denominator = 1 + z² / n |
| 中心项 | centre = p + z² / (2n) |
| 偏差项 | spread = z · √((p(1 − p) + z² / (4n)) / n) |
纳入 Speech to Speech Index
仅用于 Artificial Analysis Speech to Speech Index：
Arena Score：Arena Score 使用模型获得发布资格时冻结的 Elo，将其换算为相对于 800 Elo 基准线的预期偏好度。
Arena Score = 100 / (1 + 10^((800 − frozen Elo) / 400))
价格
- 每小时输入音频价格： 发送给 API 的请求／消息中所包含音频的总成本（美元）。
- 每小时输出音频价格： 由模型生成（从 API 接收）的音频的总成本（美元）。
速度
- 首个音频延迟（TTFA）： 生成音频输出的第一个 token 平均所需的秒数，在 Big Bench Audio 问题集上测得。TTFA 是语音智能体应用中感知响应速度的关键指标。
版本历史
Artificial Analysis Speech to Speech Index v2.0
2026 年 8 月至今
- 在指数中将对话动态（Full Duplex Bench）替换为任务成功率。
- 四个组件均等加权：语音推理（Big Bench Audio）25%、智能体表现（𝜏-Voice）25%、Arena 偏好（Arena Score）25%、任务成功率 25%。
Artificial Analysis Speech to Speech Index v1.1
2026 年 8 月
- 将 Speech Agent Arena 添加至 Artificial Analysis Speech to Speech Index。
- 四个数据集均等加权：语音推理（Big Bench Audio）25%、对话动态（Full Duplex Bench）25%、智能体表现（𝜏-Voice）25%、Speech Agent Arena 25%。
Artificial Analysis Speech to Speech Index v1.0
2026 年 6 月至今
- 推出 Artificial Analysis Speech to Speech Index：一项加权平均分数，要求同时具备语音推理、对话动态与智能体表现的结果。
- 三个数据集权重相同：语音推理（Big Bench Audio）33.3%、对话动态（Full Duplex Bench）33.3%、智能体表现（𝜏-Voice）33.3%。
Big Bench Audio (BBA) v1.2
2026 年 5 月至今
- 更新了评判模型与评分框架，使其能更可靠地识别冗长回答中的正确最终答案，包括模型在给出正确答案之前先讨论了错误选项的情况。
Big Bench Audio (BBA) v1.1
2026 年 3 月至 2026 年 5 月
- 准确率按 1,000 个问题中回答正确的数量计算，其中包含模型未作答的问题。
- 使用 Claude Sonnet 4.6 作为评判模型。
Big Bench Audio (BBA) v1.0
2024 年 12 月至 2026 年 3 月
- 准确率按非错误回答中回答正确的比例计算，因此模型不会因未作答的问题而被扣分。
- 使用 Claude Sonnet 3.5 作为评判模型。