VIALS-AA: different paths to scientific image reasoning

Vision-enabled models are a natural fit for VIALS-AA: they can inspect a scientific figure, read its labels, and relate visual features to the question. We also wanted to test how a model without native vision would approach these tasks. We gave GLM-5.3 (max) the same image files and code execution, allowing it to investigate the figure through tools.

Without native vision, the model relies on code to extract evidence from the image. In the spot-22 trajectory below, GLM-5.3 (max) loads the image into arrays, enlarges and thresholds regions for OCR, compares pixel intensities, and prints coordinates and text maps of the image. It uses those outputs to locate labels, measure the molecular-weight scale, and choose a candidate spot. Each round of code returns another piece of evidence for the model to interpret and refine.

This selected pair shows the extra work involved, along with its limits. GLM-5.3-Flash (max) combines code with ten image views and finishes in 30 turns with the correct range. GLM-5.3 (max) takes 54 turns and 57 code calls, with no image views, and answers incorrectly.

We classified tool calls across all 19 models in our original test set, covering 9,177 rollouts. Most calls were about visual evidence. 38.3% of tool calls were viewing images, while 47.3% prepared or measured them, creating crops, enhancing views, and extracting measurements from pixels. Image measurement accounted for just 5–6% of tool calls for GPT-6 Astra (max) and GPT-6.1 Sol (max), compared with 25–27% for Claude Opus 5.5 (max) and Claude Sonnet 5.5 (max).

Not every code execution does analysis

Gemini 3.8 Flash (high) accounts for 86.7% of the cohort’s calls classified as commentary/no-op. This represents 5.6% of its tool calls, or about 3.8 of its 67.5 calls per task.

commentary/no-op tool calls cover static narration, comments, or printed text without new computation or file analysis. In Protein spot 22, a commentary/no-op tool call prints an interpretation of the spot labels. The code execution tool acts as a notepad: the model writes out its interpretation and the text returns in the next model request.

Models with higher Artificial Analysis Intelligence Index scores generally make fewer tool calls on VIALS-AA.

But similar VIALS-AA scores can come from very different inspection strategies: Opus and Sonnet make substantially more tool calls and image measurements than Astra and Sol.

Leading open weights models solve about as many tasks at least once as closed frontier models, but are much less consistent. Kimi K3.1 Pro (max), MiMo-V2.6-Pro, and MiniMax-M3.1-Flash (max) each solve 114–119 tasks at least once, against 111–118 for the seven top-scoring models, all closed. They pass all three repeats on only 48–73 tasks, against 95–103. Our headline metric, pass^3, is the share of tasks answered correctly in all three repeats. We use pass^3 because inconsistent answers in scientific work can lead to wrong conclusions and costly follow-up experiments.

Gemini 4 Carbon (high) leads or shares the lead in three of the eight domain groups: plasmid maps, phylogenetics, and flow cytometry. It shares the flow-cytometry lead with Gemini 4 Argon (high). Claude Opus 5.5 (max) leads on blotting, Claude Sonnet 5.5 (max) on cell counting, and GPT-6.1 Terra (max) on protein structure/binding. GPT-6 Astra (max) and GPT-6.1 Sol (max) share the lead on molecular structure.

Cost per task ranges from $0.02 for GPT-6 Luna (max) to $1.77 for Grok 4.7 (xhigh), a 99× spread. GPT-6.1 Sol (max) and Claude Sonnet 5.5 (max) score 59% and 60%, respectively, but Sonnet costs about 18× as much per task.

Output tokens per task range from about 2.21k for GPT-6 Astra (max) to 94.6k for MiMo-V2.6-Pro, a 43× spread. Reasoning makes up most of each model’s output. Claude Sonnet 5.5 (max) uses 63.5k, including 54.6k reasoning tokens: about 25× as many output tokens as GPT-6.1 Sol (max) for a similar score.