Artificial Analysis Decision Indexのベンチマーク方法論

Artificial Analysis Decision Index

The Artificial Analysis Decision Index measures the capabilities and performance of decision models across classification, tool calling, knowledge, hallucination, and reasoning. To do this, it combines four evaluations: AA-Omniscience-Decision, a private dataset adapted from the knowledge and hallucination benchmark AA-Omniscience; HLE-MultiChoice, the text-only multiple-choice questions from Humanity's Last Exam; When2Call, NVIDIA's tool-calling evaluation; and WildGuardTest, a safety classification evaluation.

Decision Index evaluation suite

The Decision Index is a weighted average of its four evaluations.

評価能力指標重み
AA-Omniscience-Decision知識とハルシネーション正答率（12.5%）+ 非ハルシネーション率（12.5%）25%
HLE-MultiChoice推論正答率25%
When2Callツール呼び出し正答率25%
WildGuardTest分類平均F125%

タスクと回答

Each task is posed as a state, the text to decide about, and one or more questions about it. A question is either a choice among named options or a yes/no question. A model answers all of a task's questions in a single call.

意思決定モデル

We send the state and questions to the model's decision API. For a choice, the model returns the option it picks and a probability for every option. We score the option it picks, which is the option it gives the highest probability. For a yes/no question, the model returns the probability of yes, and we take its answer as yes when that probability is 0.5 or more.

言語モデル

We send a fixed system prompt and the state and questions as text, and ask for a JSON reply that gives each question one of its allowed answers. We enforce this with structured outputs.

Artificial Analysis Decision Indexの評価

The four evaluations in the Artificial Analysis Decision Index.

AA-Omniscience-Decision

  • Description: 500 private AA-Omniscience questions, each a choice among five options: four answers, one of them correct, and "I don't know". Included in the Artificial Analysis Decision Index at 25% weighting: accuracy and non-hallucination rate at 12.5% each.
  • Original dataset: AA-Omniscience
  • Scoring: Accuracy is the share of questions answered correctly. Non-hallucination rate is 100 × (1 − hallucination rate), where hallucination rate is the share of questions not answered correctly that were answered wrongly rather than with "I don't know".

HLE-MultiChoice

  • Description: The 513 text-only multiple-choice questions from Humanity's Last Exam, each posed as one choice among its lettered answers in their original order. Included in the Artificial Analysis Decision Index at 25% weighting.
  • Dataset: cais/hle. We use its 513 multiple-choice questions without an image, out of 2,500 in all. The questions we use are listed below.

  • Scoring: Accuracy, an exact match on the answer letter.

When2Call

  • Description: The multiple-choice test set of NVIDIA's When2Call. Each task is a conversation with an assistant that has tools, posed as one choice among four candidate replies: answer directly, call a tool, ask for more information, or say it cannot answer. The order of the replies is shuffled per task. Included in the Artificial Analysis Decision Index at 25% weighting.
  • Original dataset: nvidia/When2Call
  • Dataset: We use a filtered set of 3,423 tasks, down from 3,652 in the original test set. We filter out 229 “ask for more information” tasks where there is ambiguity about whether more information is required. The filtered-out tasks are listed below.

  • Scoring: Accuracy, the share of tasks answered with the reply of the correct category.

WildGuardTest

  • Description: The test set of Ai2's WildGuardMix. We ask up to three yes/no questions about each row: is the user's request harmful, is the assistant's reply a refusal, and is that reply harmful. Included in the Artificial Analysis Decision Index at 25% weighting.
  • Original dataset: allenai/wildguardmix (ODC-BY)
  • Dataset: We use 835 of the original 1,725 test rows. We keep rows where all three human annotators agreed on all three labels (prompt harm, response refusal, and response harm), based on the annotator agreement fields on Hugging Face, and exclude questions that consistently trigger model refusals to better isolate model capability.
  • Scoring: The average of three F1 scores: prompt harm, response harm and refusal.

バージョン履歴

Version 1.0

October 2026 to current

  • Launched the Artificial Analysis Decision Index with four evaluations: AA-Omniscience-Decision, HLE-MultiChoice, When2Call and WildGuardTest