Follow us on Twitter or LinkedIn to stay up to date with future analysis

Menu

logo
Artificial Analysis
HOME

Multilingual Reasoning: Comparison of Leading AI Models by Language

Multilingual Reasoning Comparison and analysis of AI models reasoning, performance, and price across different languages. Click on any model to see detailed metrics. For more details including relating to our methodology, see our FAQs.

Ranking of top 5 models by language

English 🇬🇧:
Claude 3.5 Sonnet (Oct '24)Claude 3.5 Sonnet (Oct '24)
(94)
o1-previewo1-preview
(93)
GPT-4o (Aug '24)GPT-4o (Aug '24)
(93)
DeepSeek V3DeepSeek V3
(92)
Qwen2.5 Instruct 72BQwen2.5 Instruct 72B
(91)
French 🇫🇷:
Claude 3.5 Sonnet (Oct '24)Claude 3.5 Sonnet (Oct '24)
(89)
Llama 3.3 Instruct 70BLlama 3.3 Instruct 70B
(86)
DeepSeek V3DeepSeek V3
(86)
o1-minio1-mini
(85)
Qwen2.5 Instruct 72BQwen2.5 Instruct 72B
(85)
Spanish 🇪🇸:
Claude 3.5 Sonnet (Oct '24)Claude 3.5 Sonnet (Oct '24)
(91)
DeepSeek V3DeepSeek V3
(90)
o1-previewo1-preview
(89)
GPT-4o (Aug '24)GPT-4o (Aug '24)
(88)
Qwen2.5 Instruct 72BQwen2.5 Instruct 72B
(88)
Chinese 🇨🇳:
o1-previewo1-preview
(89)
Claude 3.5 Sonnet (Oct '24)Claude 3.5 Sonnet (Oct '24)
(88)
GPT-4o (Aug '24)GPT-4o (Aug '24)
(87)
o1-minio1-mini
(87)
DeepSeek V3DeepSeek V3
(87)
Japanese 🇯🇵:
o1-previewo1-preview
(88)
Claude 3.5 Sonnet (Oct '24)Claude 3.5 Sonnet (Oct '24)
(87)
o1-minio1-mini
(86)
DeepSeek V3DeepSeek V3
(86)
GPT-4o (Aug '24)GPT-4o (Aug '24)
(85)

Multilingual Index by Language

+ Add model from specific provider
Artificial Analysis Multilingual Index; Higher is better
English
Spanish
French
German
Swahili
Bengali
Chinese
Japanese
Artificial Analysis Multilingual Quality Index: An average result across evaluations assessing multilingual performance in various dimensions of model intelligence. Includes MMLU (general reasoning) and MGSM (mathematical reasoning), with results computed across multiple languages. See methodology for more details.

Language options

Quality & Context Window

Back to Navigation

Multilingual Index (Average Across Languages)

+ Add model from specific provider
Artificial Analysis Multilingual Index; Higher is better
Artificial Analysis Multilingual Quality Index: An average result across evaluations assessing multilingual performance in various dimensions of model intelligence. Includes MMLU (general reasoning) and MGSM (mathematical reasoning), with results computed across multiple languages. See methodology for more details.

Multilingual Index vs. Output Speed

+ Add model from specific provider
Artificial Analysis Multilingual Index; Output Speed: Output Tokens per Second
Most attractive quadrant
o1-preview
Claude 3.5 Sonnet (Oct)
o1-mini
DeepSeek V3
GPT-4o (Aug '24)
Gemini 1.5 Pro (Sep)
Mistral Large 2 (Nov '24)
Qwen2.5 72B
Nova Pro
Llama 3.3 70B
Gemini 1.5 Flash (Sep)
GPT-4o mini
Claude 3.5 Haiku
Nova Lite
Nova Micro
Aya Expanse 32B
Llama 3.1 8B
There is a trade-off between model quality and output speed, with higher quality models typically having lower output speed.
Artificial Analysis Multilingual Quality Index: An average result across evaluations assessing multilingual performance in various dimensions of model intelligence. Includes MMLU (general reasoning) and MGSM (mathematical reasoning), with results computed across multiple languages. See methodology for more details.
Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).

Multilingual Index vs. Price

+ Add model from specific provider
Artificial Analysis Multilingual Index; Price: USD per 1M Tokens
Most attractive quadrant
o1-preview
Claude 3.5 Sonnet (Oct)
o1-mini
DeepSeek V3
GPT-4o (Aug '24)
Gemini 1.5 Pro (Sep)
Mistral Large 2 (Nov '24)
Qwen2.5 72B
Nova Pro
Llama 3.3 70B
Gemini 1.5 Flash (Sep)
GPT-4o mini
Claude 3.5 Haiku
Nova Lite
Nova Micro
Aya Expanse 32B
Llama 3.1 8B
While higher quality models are typically more expensive, they do not all follow the same price-quality curve.
Artificial Analysis Multilingual Quality Index: An average result across evaluations assessing multilingual performance in various dimensions of model intelligence. Includes MMLU (general reasoning) and MGSM (mathematical reasoning), with results computed across multiple languages. See methodology for more details.
Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).

Multilingual MMLU, Multilingual GSM

+ Add model from specific provider
Multilingual MMLU , Higher is better; Multilingual GSM , Higher is better
MMMLU
MGSM
Multilingual MMLU: Massive Multitask Language Understanding, evaluated across multiple languages. Tests general knowledge and reasoning ability in areas like science, humanities, mathematics and more.
Multilingual GSM: Multilingual Grade School Math, evaluates mathematical reasoning ability across different languages using grade school level word problems.

Pricing

Back to Navigation

Pricing: Input and Output Prices

+ Add model from specific provider
Price: USD per 1M Tokens; Lower is better
Input price
Output price
The relative importance of input vs. output token prices varies by use-case. E.g. Generation tasks are typically more input token weighted while document-focused tasks (e.g. RAG) are more output token weighted.
Input Price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.
Output Price: Price per token generated by the model (received from the API), represented as USD per million Tokens.
Median across providers: Figures represent median (P50) across all providers which support the model.

Speed

Back to Navigation

Output Speed

+ Add model from specific provider
Output Tokens per Second; Higher is better
Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).
Median across providers: Figures represent median (P50) across all providers which support the model.

Latency

+ Add model from specific provider
Seconds to First Token Received; Lower is better
Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.
Median across providers: Figures represent median (P50) across all providers which support the model.

Total Response Time

+ Add model from specific provider
End-to-End Seconds to Output 100 Tokens; Lower is better
The speed difference between the fastest and slowest models is >3X. There is not always a correlation between parameter size and speed, or between price and speed.
Total Response Time: Time to receive a 100 token response. Calculated based on Latency (time to receive first token) and Output Speed (output tokens per second).
Median across providers: Figures represent median (P50) across all providers which support the model.

Models compared: OpenAI: GPT 4o Audio, GPT 4o Realtime, GPT 4o Speech Pipeline, GPT-3.5 Turbo, GPT-3.5 Turbo (0125), GPT-3.5 Turbo (1106), GPT-3.5 Turbo Instruct, GPT-4, GPT-4 Turbo, GPT-4 Turbo (0125), GPT-4 Vision, GPT-4o (Aug '24), GPT-4o (May '24), GPT-4o (Nov '24), GPT-4o Realtime (Dec '24), GPT-4o mini, GPT-4o mini Realtime (Dec '24), o1, o1-mini, and o1-preview, Meta: Code Llama 70B, Llama 2 Chat 13B, Llama 2 Chat 70B, Llama 2 Chat 7B, Llama 3 70B, Llama 3 8B, Llama 3.1 405B, Llama 3.1 70B, Llama 3.1 8B, Llama 3.2 11B (Vision), Llama 3.2 1B, Llama 3.2 3B, Llama 3.2 90B (Vision), and Llama 3.3 70B, Google: Gemini 1.0 Pro, Gemini 1.5 Flash (May), Gemini 1.5 Flash (Sep), Gemini 1.5 Flash-8B, Gemini 1.5 Pro (May), Gemini 1.5 Pro (Sep), Gemini 2.0 Flash (exp), Gemini 2.0 Flash Thinking (exp), Gemini Experimental (Nov), Gemma 2 27B, Gemma 2 9B, and Gemma 7B, Anthropic: Claude 2.0, Claude 2.1, Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet, Claude 3.5 Haiku, Claude 3.5 Sonnet (June), Claude 3.5 Sonnet (Oct), and Claude Instant, Mistral: Codestral, Codestral-Mamba, Ministral 3B, Ministral 8B, Mistral 7B, Mistral Large (Feb '24), Mistral Large 2 (Jul '24), Mistral Large 2 (Nov '24), Mistral Medium, Mistral NeMo, Mistral Small (Feb '24), Mistral Small (Sep '24), Mixtral 8x22B, Mixtral 8x7B, Pixtral 12B, and Pixtral Large, Cohere: Aya Expanse 32B, Aya Expanse 8B, Command, Command Light, Command R7B, Command-R, Command-R (Mar '24), Command-R+ (Apr '24), and Command-R+, Perplexity: PPLX-70B Online, PPLX-7B-Online, Sonar 3.1 Huge, Sonar 3.1 Large, Sonar 3.1 Small , Sonar Large, and Sonar Small, xAI: Grok Beta and Grok-1, OpenChat: OpenChat 3.5, Amazon: Nova Lite, Nova Micro, and Nova Pro, Microsoft Azure: Phi-3 Medium 14B, Phi-3 Mini, and Phi-4, Upstage: Solar Mini, Solar Pro, and Solar Pro (Nov '24), Databricks: DBRX, NVIDIA: Llama 3.1 Nemotron 70B, IBM: Granite 3.0 2B, OpenVoice: Granite 3.0 8B, Reka AI: Reka Core, Reka Edge, Reka Flash (Feb '24), and Reka Flash, Other: LLaVA-v1.5-7B, AI21 Labs: Jamba 1.5 Large, Jamba 1.5 Mini, and Jamba Instruct, DeepSeek: DeepSeek V3, DeepSeek-Coder-V2, DeepSeek-V2, DeepSeek-V2.5, DeepSeek-V2.5 (Dec '24), and DeepSeek-VL2, Snowflake: Arctic, Alibaba: QwQ 32B-Preview, Qwen Max, Qwen Plus, Qwen Turbo, Qwen1.5 Chat 110B, Qwen1.5 Chat 14B, Qwen1.5 Chat 32B, Qwen1.5 Chat 72B, Qwen1.5 Chat 7B, Qwen2 72B, Qwen2 Instruct 7B, Qwen2 Instruct A14B 57B, Qwen2-VL 72B, Qwen2.5 Coder 32B, Qwen2.5 Instruct 14B, Qwen2.5 Instruct 32B, Qwen2.5 72B, and Qwen2.5 Instruct 7B, and 01.AI: Yi-Large.

Footer

Key Links

  • Mixtral 8x7b
  • Llama 2
  • GPT 4
  • GPT 3.5
  • Gemini Pro

Artificial Analysis

  • FAQ
  • Contact & Data access
  • Terms of Use
  • Privacy Policy
  • [email protected]

Subscribe to our newsletter

TwitterLinkedIn