Announcing Optima: create a custom benchmark for your use case

Building and running benchmarks is difficult. We have distilled Artificial Analysis' research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency.

Optima lets you find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task.

We have applied the benchmarking approaches and infrastructure we use at Artificial Analysis across each part of Optima.

• Build benchmarks from your own data and workflows. There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize, Braintrust and Langfuse. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you.
• Run across the latest models. Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released.
• Bring Artificial Analysis grading to your own benchmark. Evaluate responses against objective rubric criteria, or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set.
• Compare performance, cost and time efficiency. Optima benchmarks more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, so you can compare the tradeoffs between models for your specific use case.

Ahead of launch, we gave a group of pre-release testers access to Optima. Examples of benchmarks they created include:

• Which model can save me 10x the cost without a meaningful decrease in quality for my finance and accounting agent?
• Which model best matches the writing style of lawyers for my legal agent?
• Which model can best identify different elements in my custom image dataset?