Mobile Device Benchmark Set Methodology

Overview

The Mobile Device Benchmark Set refers to the current set of five evaluations we use to measure the intelligence of language models small enough to run on mobile phones. It applies the same independent evaluation approach we use across Artificial Analysis.

Rather than the original full-precision weights, we evaluate the quantized builds that mobile phones run in practice. Scores here can therefore differ from the same model's scores elsewhere on Artificial Analysis.

Eligibility

We define a model as theoretically small enough to run on mobile phones if it fits inside 8 GB of memory after quantization, including the KV cache at 8K context. The KV cache counts because a model needs memory for its context as well as its weights, and 8K is a conservatively large bound on the tokens needed for everyday on-device tasks featuring instructions and tool definitions.

On the results page, models that qualified but did not run properly on the selected device (at the specific quantization selected) are shown in a different color/pattern to identify that it may not be feasible to run this combination of model, quantization, inference framework and device.

How We Run the Evaluations

  • The current release uses 4-bit (Q4_K_M) or smaller GGUF quantizations, served with llama.cpp.
  • All evaluations run with a 16K token context window. A response is capped by that window and by the model's own output token limit, whichever is lower; we do not compact or trim context, so a model that runs out of tokens before giving its final answer is marked incorrect on that attempt. This context limit is lenient compared to realistic mobile phone context limits, which are often <4K in practice. A 1-minute time limit is also included in the benchmark results, to give an alternative perspective on model performance under a tighter restriction; results at a 64K context limit are coming soon.
  • Every evaluation runs 5 repeats per question with pass@1 scoring, except AA-Omniscience, which runs a single pass over its 6,000 questions.
  • Models that support both reasoning and non-reasoning modes will be evaluated in both settings and listed as separate variants. There will initially be incomplete coverage of each mode for each model.

Component Evaluations

The first version of the benchmark set consists of five evaluations, each measured independently by Artificial Analysis:

  • BFCL. Tool calling on a 640-task subset selected for small models, with three task categories weighted equally.
  • IFBench. Instruction following with precise, verifiable output constraints.
  • AA-Omniscience. Factual knowledge and hallucination resistance, split evenly into accuracy and non-hallucination.
  • GPQA Diamond. Graduate-level scientific reasoning.
  • MATH-500. Mathematical problem solving.

These evaluations were chosen to represent the mixture of instruction following, tool calling, knowledge recall, and reasoning that small models are expected to handle. They are combined in a simple average (equal weighting) to produce the final intelligence score.

We expect the evaluations to shift in the future as we further improve our small model and mobile device benchmarking practices.

BFCL

The Berkeley Function Calling Leaderboard (BFCL) v4 measures how well a model calls tools. We run a 640-task subset chosen to be meaningful for small models, drawn from three categories that each contribute a third of the BFCL score:

  • Multiple (200 tasks): choose the right function from several available and call it with the correct arguments. Scored by matching the call against the expected structure.
  • Multi-turn base (200 tasks): complete a task across several turns of tool use. Scored on the sequence of calls and the final state they produce.
  • Irrelevance (240 tasks): recognize that none of the available tools fit the request and decline instead of forcing a call.

Where we differ from the official BFCL setup:

  • All tasks come from the non-live split, written by the BFCL authors. The crowdsourced live split is a candidate for future versions of the benchmark set.
  • We test native function calling only (BFCL's FC mode), since that is how applications typically pass tools to models. We do not use BFCL's prompting mode, which passes the tools in the prompt rather than through the API's tool interface, and parses calls out of the model's reply.
  • We use each model's native tool calling, with parsing provided by llama.cpp from the model's chat template, and do not attempt to rescue tool calls from the text content: a tool call a runtime could not execute counts as a failure, as it would in a real application.
  • Requests go through our standard provider stack rather than BFCL's per-model handlers.

Dataset: the non-live split of BFCL v4, developed at UC Berkeley and maintained in the Gorilla repository.

IFBench

IFBench tests whether a model can follow precise, verifiable instructions, such as word counts, formatting rules, and sentence manipulation. It is included because reliably following instructions underpins almost everything people ask small models to do.

  • Dataset: the single-turn IFBench set of 294 prompts (allenai/IFBench_test). We do not use the multi-turn version.
  • A response only counts if it satisfies every instruction in the prompt (prompt-level accuracy).
  • Responses are checked with the official code from allenai/IFBench in loose mode, which tolerates extra text and formatting around the answer.

AA-Omniscience

AA-Omniscience is our own knowledge and hallucination benchmark: 6,000 questions across 42 topics covering business, law, health, software engineering, science, and the humanities. It rewards precise knowledge and penalizes confident wrong answers.

  • Public subset: ArtificialAnalysis/AA-Omniscience-Public.
  • Each response is graded correct, incorrect, partially correct, or not attempted by an LLM grader.
  • It contributes two equally weighted components to the average. Accuracy is the share of questions answered correctly. Non-hallucination is 1 minus the hallucination rate, where the hallucination rate is the share of questions the model answered incorrectly out of those it did not answer correctly.
  • Including non-hallucination is deliberate: small models are not expected to have extensive knowledge of obscure concepts, but need to demonstrate trustworthiness by not responding or taking an action where they do not know the correct answer.
  • Full details, including the grading approach, are in the AA-Omniscience section of our Intelligence Index methodology.

GPQA Diamond

GPQA Diamond tests graduate-level scientific reasoning with multiple-choice questions in biology, physics, and chemistry.

  • Dataset: the Diamond subset of GPQA, 198 questions with 4 answer options each, selected by the benchmark authors as the highest-quality portion of the full dataset.
  • Answers are extracted with regex and scored pass@1, using the same multiple-choice prompt as our main Intelligence Index (see the prompt template).

MATH-500

MATH-500 is a 500-problem set of high-school competition mathematics across a range of subjects and difficulty levels.

  • Dataset: HuggingFaceH4/MATH-500.
  • Models are asked to solve the problem step by step and give the final answer in a box.
  • Grading first checks the extracted answer symbolically, so equivalent forms of the same answer match. When that is inconclusive, an LLM equality checker decides whether the answer matches the reference.

Inference on Mobile Phones

Inference performance on mobile phones, including speed, latency, and memory use, is benchmarked separately through our partnership with Liquid AI, with measurements taken on the devices themselves. Liquid AI's measurement platform (and results viewer) is called Pipette.