Using Artificial Analysis

Artificial Analysis provides benchmarking and related information to support people & organizations in choosing the right model for their use-case and which provider to use for that model.

Different trade offs exist between different models and providers and as such, it is important to start with how you intend to use the model as the first step. Trade-offs exist currently between model quality, price, output speed, latency, context window & other dimensions.

Start with your use-case

Starting with your use-case starts means thinking about which of these are the most important to optimize for in your decision. For example, a use-case of a consumer website which needs to keep people engaged but also has low ARPU (revenue per user) is likely best choosing a model which has higher output speed, lower latency,
and a lower price. This would optimize for keeping users engaged with faster response times and also maintain lower cost as they are limited in how much money they can spend per user.

Choose a model before choosing an API provider

Related page: artificialanalysis.ai/models

Models vary significantly between their technical performance, features and their quality. Artificial Analysis's intelligence index represents a generalist perspective and is useful as an indicator of the quality of responses. It is generally advisable to develop a short-list of models which best suit your use-case for custimized use-case specific testing.

It is typically best to choose which model you prefer to use for your use cases before choosing the API provider as API providers vary in which models they they host and also vary in their performance between models (though typically to a lesser degree). However, to caveat, circumstances may exist which restrict choice of providers (such as organization approved providers, etc)

Choose an API Provider

Related page (Mixtral 8x7B used to illustrate): artificialanalysis.ai/models/mixtral-8x7b-instruct/providers When choosing between providers, start with their price & technical performance metrics (thoughput, latency) to understand which combination is best for your use case. Also look in-detail (accessible at the bottom of each provider in table form) the details of each provider's offering including OpenAI compatibility, rate limits, differing context windows, etc.

Then you're ready to sign up to provider(s) you wish to use and start using the AI & API provider best suited to your use-case.