🤖 AI

LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI

LegalHalluLens is a framework that enhances trustworthy legal AI by introducing typed hallucination profiles and a Risk Direction Index to reveal hidden error patterns, which then calibrate a multi-agent debate pipeline to significantly reduce fabrication rates and improve deployment safety.

Original authors: Lalit Yadav, Akshaj Gurugubelli

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lalit Yadav, Akshaj Gurugubelli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of legal assistants to read hundreds of contracts and pull out specific details, like "When does this deal end?" or "How much money is the liability cap?"

You ask them, "How often do you make mistakes?" They all say, "About 52% of the time."

That sounds bad, but it's also misleading. It's like saying a doctor has a 52% error rate without telling you what kind of errors they make. Are they misdiagnosing a broken leg (a minor issue) or missing a heart attack (a fatal one)?

This paper, LegalHalluLens, argues that in the legal world, knowing the average error rate is useless. You need to know what they get wrong and how they get it wrong.

Here is the breakdown of their findings using simple analogies:

1. The "Average" Lie (The Blindfold)

The researchers tested four different AI models (two from big tech companies and two open-source ones) on 510 real contracts.

The Old Way: If you just look at the total score, all the models look roughly the same, hovering around a 52% error rate. It's like a blindfolded archer hitting the target 52% of the time. You don't know if they are hitting the bullseye or the edge of the board.
The New Way (Typed Profiles): When the researchers looked closer, they found a massive difference based on the type of question:
- Easy Questions (Temporal): "When does the contract end?" The AI got these right most of the time (only ~30% error). It's like asking, "What color is the sky?"
- Hard Questions (Obligation & Numeric): "How much is the penalty?" or "What exactly must the supplier do?" The AI failed miserably here, with error rates jumping to 65–74%. It's like asking, "What is the exact chemical formula for this medicine?" and the AI just guessing.

The Takeaway: An AI might look "okay" overall, but it could be dangerously unreliable on the specific details that determine if a contract is legal or if a company gets sued.

2. The "Direction" of the Mistake (The Scale)

The paper introduces a new tool called the Risk Direction Index (RDI). Think of this as a scale that measures how the AI lies.

The "Inventor" (Positive RDI): This AI tends to make things up. If a contract doesn't have a liability cap, this AI might invent one and say, "Oh, it's $5 million." This creates a false sense of safety.
The "Omitter" (Negative RDI): This AI tends to hide things. If a contract says, "You must pay within 30 days," this AI might just say, "No deadline mentioned." This creates a hidden danger because the user thinks there are no rules.

The Takeaway: Two AI systems can have the exact same "52% error rate," but one is a liar who makes up rules, and the other is a forgetful assistant who misses rules. You need to know which one you are using because the legal risks are totally different.

3. The "Debate Club" Fix (The Calibrated Team)

The researchers tried to fix the "Inventor" AI (a smaller, cheaper model) using a Multi-Agent Debate.

The Generic Approach: Usually, you just tell AI agents to "argue with each other" to find the truth. This paper says that's too vague.
The Calibrated Approach: They built a specific "Debate Team" with roles tailored to the specific mistakes the AI was making:
- The Skeptic: Instead of asking random questions, this agent specifically asks, "Did you just make up that number?" or "Did you forget to include that 'unless' clause?"
- The Supporter: Defends the answer using only the exact words from the contract.
- The Gatekeeper: Uses a special rule: "We will only add a new rule if we are 100% sure it's there, but we will never delete a rule unless we are absolutely sure it's gone."

The Result: By customizing the debate to target the specific types of errors the AI made, they reduced the number of made-up facts (fabrications) by 45%.

The Magic: They took a small, cheap AI model and, by using this smart debate team, made it perform as well as (or better than) expensive, commercial "super-AI" models.

Summary

This paper is a warning and a toolkit for anyone using AI in law:

Don't trust the average: An AI can be great at dates but terrible at money and rules.
Check the direction: Know if your AI is prone to inventing rules or hiding them.
Customize the fix: You can't just tell AI to "be better." You have to build a specific process (like a debate team) that targets the exact mistakes that AI tends to make.

The authors conclude that for legal work, we need to stop asking "How accurate is this AI?" and start asking "What specific things does it get wrong, and in which direction?"

Technical Summary: LegalHalluLens

1. Problem Statement

Current legal AI systems deployed in workflows (contract review, compliance, due diligence) exhibit hallucination rates that aggregate metrics report at approximately 52%. However, the paper argues that this average is insufficient for trustworthy deployment because it conceals two critical dimensions:

Failure Concentration: Errors are not uniform; they concentrate heavily on specific claim types (e.g., numeric and obligation clauses) that carry the highest legal liability, while other types (e.g., temporal) are significantly more reliable.
Error Direction: Aggregate metrics fail to distinguish between omission (missing a present obligation) and invention (fabricating a non-existent clause). These two failure modes have distinct legal consequences (e.g., unenforceable clauses vs. false risk ceilings) but are often treated identically in standard reporting.

Consequently, compliance officers lack actionable signals to select models or design interventions, as two systems with identical aggregate hallucination rates may possess opposite risk profiles.

2. Methodology

The authors propose LegalHalluLens, an auditing framework comprising three components, evaluated on the CUAD v1.0 corpus (510 commercial contracts, 41 clause types, 249,252 clause-level instances).

A. Typed Hallucination Profiles

Instead of a single aggregate rate, the framework partitions errors into four legally motivated claim categories based on verification difficulty:

Numeric: Thresholds and values (e.g., liability caps).
Temporal: Dates and deadlines.
Obligation/Entitlement: Modal verbs, conditions, and carve-outs.
Factual: Identity claims (e.g., governing law).

The study measures HalTP (content hallucination rate among detected clauses) for each category. It defines a Typed Gap as the difference between the highest and lowest hallucination rates within a single model.

B. Risk Direction Index (RDI)

To capture the directional nature of errors, the authors introduce the Risk Direction Index (RDI). This is a signed scalar metric derived from the proportion of contradicted findings that are "missing conditions" (omissions) versus "extra conditions" (inventions).

Formula: $RDI = p_{extra} - p_{missing}$
Interpretation: Positive values indicate invention-heavy failure (overstatement); negative values indicate omission-heavy failure (understatement). This allows for the comparison of systems that have matched aggregate error rates but opposite directional biases.

C. Calibrated Multi-Agent Debate Pipeline

The framework utilizes the diagnostic outputs (Typed Profiles and RDI) to calibrate a multi-agent debate pipeline. Unlike generic debate setups, this pipeline is specifically tuned to the measured failure modes:

Typed Skeptic Challenges: The "Skeptic" agent issues specific challenges based on the claim type (e.g., verifying verbatim values for numeric claims, checking modal verbs for obligations).
Asymmetric Structural Gates: The pipeline employs an Addition Gate (requiring consensus and verification to add a clause) and a Deletion Gate (blocking removal if verification confirms presence). This asymmetry is calibrated to the specific False Acceptance Rate (FAR) vs. False Rejection Rate (FRR) profile of the target model.
Roles: The pipeline includes a Skeptic, Supporter, Re-extractor (for structural errors), Arbiter, Verifier, and Judge.

3. Key Results

Experiment 1: Typed Hallucination Profiles

Consistent Failure Ordering: Across four diverse models (two commercial APIs, two open-source models), a consistent failure ordering was observed: {Numeric, Obligation} ≫ Factual ≥ Temporal.
The Hidden Gap: While aggregate HalTP rates clustered tightly between 50.9% and 56.5%, the within-model gap between the most and least reliable claim types was approximately 38–40 percentage points.
- Numeric/Obligation claims hallucinated at 64.8–74.3%.
- Temporal claims hallucinated at only 29.0–35.1%.
Implication: Aggregate reporting hides the fact that models are 2x–2.5x more unreliable on the clauses that determine liability and enforceability.

Experiment 1: Risk Direction Index

Directional Differentiation: Two models with nearly identical aggregate HalTP (gpt-5.2 at 51.8% and qwen3-32b at 52.1%) exhibited opposite RDI values.
- gpt-5.2: Positive RDI (+0.161), indicating a tendency to invent extra conditions.
- qwen3-32b: Negative RDI (-0.202), indicating a tendency to omit existing conditions.
Significance: This distinction is critical for deployment; omission-heavy models pose risks in compliance monitoring, while invention-heavy models pose risks in liability assessment.

Experiment 2: Calibrated Mitigation

Performance: Applying the typed debate pipeline to a small open model (gemma-4-26B-A4B, 4B active parameters) allowed it to outperform commercial APIs on a matched subset.
- The debated model achieved a composite rank of 1 (under 4 of 5 weighting schemes), surpassing gpt-5.2 (rank 2).
Fabrication Reduction: The pipeline reduced fabricated detections (FAR) by 45% (from 524 to 287 instances).
Targeted Gains: Improvements were concentrated on the highest-failure categories identified in Experiment 1:
- Obligation: $\Delta$ FAR = -8.2 pp.
- Factual: $\Delta$ FAR = -5.8 pp.
- Temporal: Minimal change (+0.6 pp), consistent with its low baseline error rate.
Direction Correction: The pipeline successfully shifted the RDI of the debated model from omission-heavy (-0.078) to near-balanced (-0.014) for obligation claims.

4. Significance and Claims

The paper claims that LegalHalluLens provides a necessary shift from aggregate accuracy to direction-aware auditing for legal AI.

Actionable Diagnostics: Typed profiles and RDI serve as calibration inputs for agent design, enabling procurement decisions based on specific risk profiles (e.g., choosing an invention-heavy model for operations vs. an omission-heavy model for compliance).
Cost-Efficient Mitigation: The study demonstrates that a small, open-source model (4B active parameters) can match or exceed commercial APIs when its debate pipeline is calibrated to specific failure modes, rather than using generic tuning.
Limitations and Scope: The authors explicitly state that the numerical results apply to CUAD-style English US commercial contracts. The failure ordering is consistent across architectures but must be re-measured for other jurisdictions or document types.
Safety Warning: The framework is diagnostic, not a clearance mechanism. Even the best configuration contradicted source text in ~58% of detected contents. The authors recommend human-in-the-loop review for all flagged clauses in high-stakes categories and warn against autonomous deployment without per-deployment re-measurement.

In summary, the paper argues that trustworthy legal AI deployment requires knowing which claims fail and how they fail, rather than relying on a single aggregate hallucination rate.