Decider v1.3: Release, Decision Index Score, Performance & Price

Accuracy less the Brier score of the probability the model gives its chosen answer, from 0 to 100. Each question scores 50 × (1 + c − (p − c)²), where p is that probability and c is 1 if the answer is right: from 50 to 100 when right and from 50 down to 0 when wrong, more so the more confident the model. It is the payoff of answering when p is above a threshold and routing the question elsewhere otherwise, averaged over every threshold. Probability on "I don't know" lowers p. AA-Paperclip scores each document as one decision.

Average cost per task at current token prices, multiplied by 1 million; for the index, each evaluation's cost per task averaged with the index's weights, multiplied by 1 million. Decision models are billed for input tokens only, and language models for input and output tokens.

The mean squared gap between the probability a model gives its chosen answer and whether that answer is right, from 0 to 1; 0 is certain and always right. For AA-Paperclip, each document's confidence that all its answers are right against whether they are. For the index, the evaluations' Brier scores averaged with the index's weights, rescaled to sum to 1.

Median latency per task, in milliseconds; for the index, each evaluation's median averaged with the index's weights.

Average cost per task at current token prices, multiplied by 1 million; for the index, each evaluation's cost per task averaged with the index's weights, multiplied by 1 million. Decision models are billed for input tokens only, and language models for input and output tokens.