Artificial Analysis FAQ

What is Artificial Analysis?

Artificial Analysis provides objective benchmarks & information to support developers, customers, researchers, and other users of AI models to make informed decisions in choosing:

1. Which AI model to use for a given task, and
2. Which hosting provider to use to access the model.

What APIs does Artificial Analysis test and show?

We support APIs from the following providers: OpenAI, Microsoft Azure, Together.ai, Mistral, Google, Anthropic, Amazon Bedrock, Perplexity, Fireworks, Lepton, and Deepinfra.

If you’d like to request coverage of a model or hosting provider not currently supported, please contact us.

What do ‘quality’ and ‘speed’ mean on Artificial Analysis?

• Quality means the accuracy and completeness of a model’s responses, assessed by benchmarks like MMLU and Chatbot Arena Elo.
• Speed means the performance of the model, as hosted by a hosting provider. Speed can be divided into latency and throughput, and Artificial Analysis measures speed metrics with our own custom benchmark.

Quality and speed generally have to be traded-off against each other as larger models generally achieve better quality but require more compute to run inference on.

I think some of your data is wrong - what should I do?

We’d love to hear from you - we want Artificial Analysis to be a resource for the AI community that simplifies decision making. Things change quickly and sometimes our data will be out of date.

If there’s something we should update, please fill out our contact form and we’ll be in touch!

I’d like to request that Artificial Analysis supports a new model or a new hosting provider.

Fill out our contact form and we’ll be in touch.

Can you guarantee the benchmark data is accurate?

Our objective is to provide high-quality benchmarks useful in decision making, but please note that we provide no guarentees or warranties and performance can vary for specific use cases. Performance can differ based on individual needs, so we advise further evaluation for the best fit.

Can I see your underlying data?

Artificial Analysis presents highlights from an extensive database of measurements. We only share raw data with select partners.

If you’re interested in discussing access, please get in touch.