Intelligence at pocket scale: Benchmarking small models and mobile phones

Models under a few billion parameters can now follow instructions, call tools, and answer questions on mobile phones. But available benchmark results often describe neither the quantized model artifact nor the device and runtime combination a user will run. Some of the issues that add to the difficulty of benchmarking this space:

This launch consists of intelligence benchmarking, administered by Artificial Analysis, and inference benchmarking, developed and run by Liquid AI. We test intelligence and inference on the same build, quantized to 4-bit or smaller, that a device would load, served with llama.cpp, ensuring each side is measured under as similar conditions as possible. A model qualifies for testing if it fits inside 8 GB of memory after quantization, including the KV cache required at 8K context.

Intelligence is measured by a set of five evaluations, chosen to represent the range of capabilities a phone-scale model needs: instruction following, tool calling, knowledge/recall, and reasoning.

Full details on our benchmarking process can be found on the Mobile Device Benchmark Set methodology page.

Liquid AI has developed Pipette, a mobile device inference benchmarking product, and open-sourced it on GitHub. We have independently validated Liquid AI's inference benchmarking methodology, which includes a standardized testing harness and a climate-controlled physical testing facility for repeatable results. Read more about Liquid AI's methodology on the Pipette documentation.

Nanbeige4.2-3B and LFM2.5-2.6B share the top average evaluation scores (with a 16K context limit) at launch, both scoring 63, ahead of Ornith-1.0-9B on 62 and Qwen3.5 9B (Reasoning) on 61. LFM2.5-2.6B achieves its score more efficiently: on an iPhone 17 Pro it answers a standard 1,024-token prompt in 8.0 seconds using 2.3 GB of memory, against 21.4 seconds and 4.0 GB for Nanbeige4.2-3B and more than 25 seconds and 6.9 GB for the two 9B models. The benchmark set scores 41 quantized builds, 33 of which successfully ran on the iPhone 17 Pro.

• The speed-intelligence Pareto frontier is short. Six models are unbeaten on both axes on an iPhone 17 Pro: LFM2.5-230M (27 at 0.9 seconds), MiniCPM5-1B (45 at 2.9 seconds), LFM2.5-8B-A1B (58 at 5.7 seconds), Ling 3.0 Tiny (59 at 5.7 seconds), LFM2.5-2.6B (63 at 8.0 seconds) and Nanbeige4.2-3B (63 at 21.4 seconds). LFM2.5-8B-A1B and Ling 3.0 Tiny are mixture-of-experts models that activate around a billion parameters per token, which is how they answer in under six seconds with 8B-class weights.
• The 16K context limit defines the leaderboard. Models like Qwen3.5 9B are highly intelligent but spend far more output tokens on evaluation tasks than is practical on a mobile phone: 74.5M tokens across one pass of the benchmark set, against 5.2M for Gemma 4 E4B (Reasoning), which scores within two points of it overall. Qwen3.5 9B (Reasoning) hits the 16K context limit on 29% of its generations, which reduces its scores. With the limit raised to 64K, Ling 3.0 Tiny takes first place with a score of 66, ahead of Nanbeige4.2-3B (65), with LFM2.5-2.6B and Qwen3.5 9B (Reasoning) both on 64. But a 64K window does not fit in mobile phone memory, and at 55 output tokens per second on an iPhone 17 Pro, generating 64K output tokens could mean a wait of more than twenty minutes and substantial battery use. This is why our primary results are capped at 16K, but we also publish results capped at 64K and results capped at one minute of generation time.
• A one-minute answer budget inverts the order completely. Capping each answer at what a model can generate in 60 seconds on the iPhone 17 Pro (its output speed × 60, ignoring prompt processing time) puts LFM2.5-8B-A1B first on 47, followed by LFM2-2.6B-Exp on 45 and Gemma 4 E4B (Non-reasoning) on 44. LFM2.5-2.6B falls to 38, and Nanbeige4.2-3B to 18: at 14 output tokens per second it gets about 850 tokens per answer, so most of its reasoning never finishes. Qwen3.5 9B (Reasoning) scores 14. While we aren't using the one-minute limit as the primary filter for this benchmark data, it may be more in line with the maximum wait time mobile users are willing to tolerate.
• The leading models have opposite strengths. Nanbeige4.2-3B is the most balanced: within a point of the best on BFCL (76%), third on MATH-500 (96%) and sixth on GPQA Diamond (67%). Qwen3.5 9B (Non-reasoning) is the strongest tool caller (77% on BFCL) and the strongest scientific reasoner (79% on GPQA Diamond), and Ornith-1.0-9B recalls the most facts (15% accuracy on AA-Omniscience). LFM2.5-2.6B follows instructions best of any model measured on the iPhone (59% on IFBench), clears 90% on MATH-500, and is far readier to decline than to guess: 79% non-hallucination against 33% for Nanbeige4.2-3B and 24% and 1% for the two Qwen3.5 9B variants. That component is what lifts it level with Nanbeige4.2-3B; on the other four evaluations alone, Nanbeige4.2-3B, Ornith-1.0-9B and Qwen3.5 9B (Reasoning) would all rank above it.
• Generation time spans a factor of 30, from 0.9 seconds for LFM2.5-230M to 26.7 for Falcon-H1R-7B, measured end to end on an identical 1,024-token prompt and 256-token response.
• Memory needs span an order of magnitude. Peak in-test memory at 4K context runs from 0.4 GB for the smallest models to 6.9 GB for Ornith-1.0-9B and Qwen3.5 9B, the heaviest here. On a 12 GB phone, this top end leaves limited room for the operating system and other apps.

The inference results below are measured on an iPhone 17 Pro. Results for the other devices we benchmark are on the mobile phones page, which is updated as new models and devices are added. Intelligence results are device-agnostic; every model is evaluated as the same build, quantized to 4-bit or smaller, that a device would load.

We plan to expand both the intelligence and on-device inference benchmarks. Next steps include:

• Comparing models by memory footprint rather than a single quantization, so a heavily quantized large model and a high-precision small model compete on equal terms
• Adding new models as they are released
• Incorporating results from more inference frameworks beyond llama.cpp, and more devices
• Investigating evaluations purpose-built for small models and on-device use