How Thinking Machines Lab’s Inkling performs on agentic knowledge work

See model page

Thinking Machines Lab’s Inkling scores an Elo of 836 on on our agentic knowledge work benchmark AA-Briefcase, ahead of DeepSeek V4 Flash but below leading open weights models including Nemotron 3 Ultra and GLM-5.2

Our new agentic knowledge work benchmark, AA-Briefcase, tests models on realistic tasks across thousands of input files, requiring deliverables such as spreadsheets, presentations, and UI mock-ups. Model performance is measured across three dimensions: binary rubric checks for ground-truth correctness, pairwise grading on analytical quality, and pairwise grading on presentation quality. The AA-Briefcase Elo is a single metric that combines results across all three dimensions

Key Takeaways:

➤ Scores 19.3% on the AA-Briefcase rubric, below MiMo-V2.5-Pro (21.4%) but above DeepSeek V4 Flash max (18.7%) and Gemini 3.5 Flash-Lite (14.8%). Inkling’s rubric score is worst on tasks which include non standard “Other” file types (i.e., not Excel, PowerPoint, PDF, or Word), despite its native multimodal support

➤ Performs higher on Presentation than Analytical Quality with Elo scores of 863 and 764 respectively. Presentation and Analytical Quality are both measured using separate, independent pairwise checks from model submissions. Graders compare two submissions for the same task and pick the one that is more professionally presented (Presentation) and the one with deeper, better-structured analysis (Analytical Quality)

➤ Uses 52K output tokens on average per AA-Briefcase task and 5M output tokens for the full suite, slightly more than models with similar scores on AA-Briefcase. Inkling uses the most tokens for Excel deliverables, followed by Word, PDF, PowerPoint, and Other types

➤ Has one of the highest mean turns per AA-Briefcase task (81) with also one of the widest ranges, resulting in a much lower median (49). Despite one of the higher average turns per task, Inkling comparatively uses fewer tool calls per turn on average (0.5)

Inkling scores 19.3% on the AA-Briefcase rubric, below MiMo-V2.5-Pro (21.4%) but above DeepSeek V4 Flash max (18.7%) and Gemini 3.5 Flash-Lite (14.8%)

Inkling performs higher on Presentation than Analytical Quality with Elo scores of 863 and 764 respectively

Inkling’s rubric score is worst on tasks which include non standard “Other” file types (i.e., not Excel, PowerPoint, PDF, or Word), despite its native multimodal support

Inkling uses 52K output tokens on average per AA-Briefcase task and 5M output tokens for the full suite, slightly more than models with similar scores on AA-Briefcase. Inkling uses the most tokens for Excel deliverables, followed by Word, PDF, PowerPoint, and Other types

Inkling has one of the highest mean turns per AA-Briefcase task (81) with also one of the widest ranges, resulting in a much lower median (49)

Despite one of the higher average turns per task, Inkling comparatively uses fewer tool calls per turn on average (0.5)

For more details: https://artificialanalysis.ai/evaluations/aa-briefcase