Announcing Harvey LAB-AA v1.1: adding hallucination checks to raise the bar for agentic legal work

For up-to-date results see the Harvey LAB-AA evaluation page. This article shows data as at 8 October 2026.

We collaborated with Harvey to update Harvey LAB-AA to v1.1. The update changes how the benchmark is graded and scored, so v1.1 results are not directly comparable to previously published v1.0 numbers:

• Hallucination auditing. Every deliverable a model submits is now audited against the task's source documents in a two-stage check; a task with no usable submission scores zero and is not audited. A first judge flags candidate hallucinations across three categories - contradictions of the source materials, fabricated source content, and specific assertions with no support in the sources - and a second pass by the same judge re-checks each flag against that task's source documents, dismissing flags that do not hold up and classifying the rest as material or minor.
• Hallucination-Gated All-Pass Rate headline. The headline metric is now the Hallucination-Gated All-Pass Rate: a task counts only when the deliverables satisfy every rubric criterion and contain no material hallucination.
• Criterion Pass Rate and material hallucinations. We report the share of rubric criteria passed before the hallucination gate, averaged across the three judges and pooled over all criteria, alongside the mean number of material hallucinations per checked task. The scatter compares these two metrics directly.
• Three-judge rubric panel. Rubric criteria are now graded by a panel of three LLM judges - GPT-6 Sol, Grok 4.7, and Claude Opus 5.5 - with verdicts averaged across the panel, replacing the single judge used in v1.0. We reviewed these judges and found minimal self-preference for their own model families, and averaging across all three mitigates any biases that emerge.
• Dataset update. v1.1 uses the latest private dataset from Harvey (v1.1.0), which includes improvements to the tasks and criteria.

How Harvey LAB-AA differs from Harvey's LAB

Harvey LAB-AA is our independent reimplementation of Harvey's evaluation, and there are several key differences to the original version:

• Models are run on our Stirrup agent harness, enabling features such as context compaction rather than failure when reaching context limits, with simplified Artificial Analysis-authored agent and judge prompts
• We do not include Harvey's custom tools and document-generation skill scripts (e.g. pptx, docx), instead providing a simple code execution tool to reflect raw model ability
• Deliverables must match the exact filename specified, rather than fuzzy matching when models produce incorrect filenames

In human preference studies run by Harvey, human experts flagged hallucinations as a primary determining factor in which response they prefer out of two otherwise comprehensive answers. Our hallucination check focuses on errors that could materially affect real legal work, and takes a conservative approach to flagging them. It distinguishes unsupported task-specific claims from general legal knowledge and places the latter out of scope, so models are not penalized for drawing on case law or statutes beyond the source files.

Every deliverable a model submits is audited for hallucinations against the task's source documents. A task with no usable submission scores zero and is not audited nor used in the hallucinations per task calculation. A hallucination is a claim in the work product that the sources do not support, and each falls into one of three categories:

• A contradiction of the source materials.
• Fabricated source content.
• A specific assertion with no support in the record.

Each hallucination is graded material or minor. A material hallucination would mislead a reader on a substantive point, such as a wrong contractually required date. A minor hallucination is a real error that is unlikely to meaningfully affect the legal interpretation of a deliverable. Only material hallucinations affect scoring: having one or more zeroes a task's score in the Hallucination-Gated All-Pass Rate. Criterion Pass Rate measures rubric coverage before the hallucination gate. Material and minor hallucination counts are reported per task the check ran on; minor hallucinations do not affect scores.

The GPT-6 models hallucinate least: GPT-6 Astra averages 0.03 material hallucinations per task (4 hallucinations across all 120 tasks) and GPT-6 Sol 0.07 (8 hallucinations across all 120 tasks), while Gemini 3.8 Flash averages the most in the launch set, at 13.96 per task. Completing the criteria and not hallucinating are different skills: Muse Spark 1.3 passes the most criteria (96.0%) but averages 1.68 material hallucinations per task, against 0.03 for GPT-6 Astra. Every open-weights model averages at least 2.09 material hallucinations per task. A model's hallucination rate depends far more on the model than on the practice area.

How we chose the hallucination checker

We use GPT-6 Sol (high) for both passes of the hallucination check, separately from the three-judge rubric panel. We compared six candidates for the hallucination judge: GPT-6 Sol, GPT-6 Luna, Grok 4.7, Claude Opus 5.5, Claude Sonnet 5.5 and Gemini 3.8 Flash, each at high reasoning effort, on the same 20 tasks and deliverables from eight evaluated models.

Across this subset of tasks, GPT-6 Sol and GPT-6 Luna generally identified more material hallucinations, while Claude Sonnet 5.5 and Gemini 3.8 Flash identified far fewer. Claude Opus 5.5 fell between Grok 4.7 and Claude Sonnet 5.5 in every model row. In total, Opus upheld 99 material hallucinations, compared with 219 for Grok, 57 for Sonnet and 470 for GPT-6 Sol. GPT-6 Sol identified more material hallucinations despite the check's conservative approach to flagging errors, which excludes general legal knowledge not contained in the source documents.

All six checkers found no material hallucinations in GPT-6 Astra's outputs, while GPT-6 Sol ranged from 0 to 0.20 per task. Both were among the models with the fewest material hallucinations under every checker.