💻 computer science

Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization

This paper introduces Gavel, a comprehensive evaluation framework comprising reference-based and reference-free agent components, to assess 12 frontier LLMs on long-context legal summarization, revealing that models tend to omit key information rather than hallucinate, struggle with complex details as context length increases, and that the agent-based approach significantly reduces token usage while maintaining competitive performance.

Original authors: Yao Dou, Benjamin Mamut, Wei Xu

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yao Dou, Benjamin Mamut, Wei Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge trying to review a massive legal case. This isn't just one document; it's a stack of papers so thick it's like reading five novels at once, containing over 100,000 words of court filings, motions, and decisions. Your job is to write a short, clear summary of what happened.

Now, imagine you have a team of super-smart robots (Large Language Models, or LLMs) that claim they can read this entire stack in seconds and write that summary for you. The paper you're asking about, GAVEL, is essentially a "report card" and a "new way of grading" to see if these robots are actually doing a good job.

Here is the breakdown of what the researchers found and how they tested it, using simple analogies:

1. The Problem: The "Needle in a Haystack"

The robots are being tested on cases that are incredibly long. Think of it like asking a robot to summarize a whole library by only looking at the books for a few seconds. The researchers wanted to know: Do these robots actually read the whole library, or do they just skim the first few pages and guess the rest?

2. The Solution: The GAVEL Grading System

The authors built a system called GAVEL to grade the robots. It has two main parts, like two different ways to check a student's homework:

GAVEL-REF (The "Answer Key" Method):
Imagine a human expert wrote the perfect summary of the case. GAVEL-REF compares the robot's summary against this human "gold standard."
- The Checklist: It checks if the robot mentioned specific facts, like "Who sued whom?" or "When was the case filed?" (Like checking if a student answered every question on a quiz).
- The "Leftover" Facts: Sometimes the robot adds interesting details that weren't on the checklist. GAVEL checks if those details are actually true or if the robot made them up.
- The Style Check: It also grades how the summary sounds. Does it read like a smooth story, or does it look like a messy bulleted list?
GAVEL-AGENT (The "Detective" Method):
Sometimes, you don't have a human "Answer Key" to compare against. So, GAVEL-AGENT acts like a detective. Instead of reading the whole stack of papers at once (which is expensive and slow), it gives the robot a set of tools.
- The robot can say, "I need to search for the filing date," and it uses a tool to jump straight to that page.
- It can say, "I need to read the motion," and it opens just that file.
- It builds the summary piece by piece, like a detective gathering clues, rather than trying to swallow the whole library in one bite.

3. The Big Findings: What the Robots Got Wrong

When the researchers tested 12 different top-tier robots on these massive legal cases, they found some surprising things:

They Forget More Than They Lie: The biggest problem wasn't that the robots made up fake facts (hallucinations). The biggest problem was that they forgot important details. It's like a student who writes a summary but leaves out the most important part of the story because they got tired or skipped a page.
The "Rare" Facts are Hard: The robots were great at finding easy things, like "When did the case start?" But they struggled with rare or complex things, like "Did they settle the case?" or "What was the final agreement?" These items were often missing.
Longer Cases = Worse Scores: As the stack of papers got thicker (from 32,000 words to 512,000 words), the robots got worse. Even the smartest robots started to miss more details as the task got harder.
The "Detective" Approach Saves Money: The GAVEL-AGENT method (the detective who searches for clues) was much more efficient. It used 36% to 77% fewer "tokens" (which is like using less electricity or money) than the methods that tried to read everything at once, while still getting a good grade.

4. The Human Check (Meta-Evaluation)

To make sure their grading system (GAVEL) was fair, the researchers spent 160 hours having humans grade the robots' work. They found that their automated system was very accurate and could be used to grade future robots without needing humans to do all the work every time.

5. Does it work elsewhere?

The researchers tested the "Detective" method (GAVEL-AGENT) on medical reviews (long reports about health treatments). It worked just as well there, saving even more resources (77% fewer tokens) while finding the right facts. This proves the method isn't just for lawyers; it's a general tool for reading long documents.

Summary

In short, GAVEL is a new way to test if AI can read really long, complicated documents. The study found that while current AI is good at reading, it tends to miss important details rather than make things up, especially when the documents are huge. However, a new "detective-style" approach where the AI searches for specific facts one by one is a much smarter and cheaper way to handle these massive tasks.

Technical Summary: GAVEL – Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization

Problem Statement

While modern Large Language Models (LLMs) now support context windows up to 1 million tokens, their efficacy in complex, multi-document long-context tasks remains unclear. Specifically, legal case summarization presents a unique challenge: a single litigation case often comprises dozens of documents (e.g., complaints, motions, decisions) totaling over 100,000 tokens, sometimes exceeding 1 million. Unlike news or novel summarization, where key information is often sequential or near the beginning, legal summarization requires integrating interconnected arguments across multiple documents while maintaining chronology, preserving relationships among parties and rulings, and ensuring accurate cross-references. Current evaluation methods often fail to capture the nuances of factual coverage and the reliability of automatic evaluators in this domain.

Methodology

The authors introduce GAVEL, a comprehensive evaluation framework consisting of two complementary components: GAVEL-REF (reference-based) and GAVEL-AGENT (reference-free).

1. GAVEL-REF: Reference-Based Evaluation

GAVEL-REF evaluates model summaries against expert-written human references using three distinct metrics:

Checklist Evaluation: Utilizes 26 expert-designed checklist items (e.g., filing date, parties, remedies sought). Unlike simple string matching, the framework extracts values and supporting text as lists, allowing for partial credit and handling of multi-value items. It classifies relationships (equal, containment, different) to compute precision, recall, and F1 scores.
Residual-Fact Evaluation: Recognizing that 78% of human summaries contain important facts not covered by the predefined checklist (averaging 6.8 facts per summary), this component extracts "residual" atomic facts from text spans not covered by checklist values. These are compared between model and human summaries to capture overlooked information.
Writing-Style Evaluation: An LLM judge rates the similarity between the model summary and the human reference across five dimensions (Readability/Jargon, Narrative Order, Sentence Structure, Formatting, Citation Style) on a 1–5 Likert scale.

These components are aggregated into a single score ( $S_{GAVEL-REF}$ ) weighted by content coverage ( $\alpha=0.9$ ) and the relative importance of residual facts ( $r$ ).

2. GAVEL-AGENT: Reference-Free Agent Scaffold

To address the high cost of collecting expert summaries, GAVEL-AGENT is a reference-free agent designed to extract checklist items directly from source documents.

Tool-Based Navigation: The agent is equipped with six tools (list_documents, read_document, search_document_regex, get_checklist, append_checklist, update_checklist) to navigate long document collections strategically, mimicking human expert behavior (skimming, searching, updating).
Context Management: To handle long contexts (256K+ tokens) and numerous tool calls, the agent refreshes its context after each action, keeping only the 5 most recent full responses and compressing the rest of the history.
Configurations: The authors tested single-agent, multi-agent (grouped), and 26-agent (one per item) configurations to determine the optimal extraction strategy.

3. Meta-Evaluation

To validate the reliability of GAVEL, the authors conducted a systematic meta-evaluation involving 160 hours of human annotation. This included 5,442 checklist item-level annotations, 450 comparison judgments, 281 residual fact extractions, and 375 style ratings. This dataset was used to benchmark the performance of various LLMs acting as evaluators within the GAVEL-REF framework.

Key Contributions

GAVEL Framework: A novel evaluation system combining checklist, residual-fact, and style evaluations for long-context legal summarization, and a reference-free agent scaffold for direct document-based extraction.
Large-Scale Meta-Evaluation: A dataset of 160 hours of human annotations demonstrating that GAVEL-REF can achieve accuracy comparable to state-of-the-art models (like GPT-5) using strong open-source models (e.g., GPT-oss, Qwen3), proving the feasibility of cost-effective automatic evaluation.
Comprehensive Benchmarking: An evaluation of 12 frontier LLMs (proprietary and open-source) on 100 recent (2025) legal cases across 18 diverse subject areas and five length scales (32K to 512K tokens).
Generalizability Demonstration: The successful adaptation of GAVEL-AGENT to the medical domain (Cochrane reviews), showing transferability beyond law.

Results

LLM Performance on Legal Summarization

Omission vs. Hallucination: Current models are significantly more prone to omitting key information than hallucinating incorrect content. Precision consistently outperforms recall across models.
Checklist Performance: Even the best-performing model, Gemini 2.5 Pro, achieved only 59.1 F1 on checklist evaluation (66.6 precision, 55.3 recall). Models performed well on simple, single-value items (e.g., filing date) but struggled with rare and complex multi-value items (e.g., settlements, related cases).
Length Sensitivity: Performance degrades as case length increases. Gemini 2.5 Pro's checklist F1 dropped by approximately 10% when moving from 32K-token to 512K-token cases. Open-source models degraded more sharply on longer cases due to context limits and truncation.
Residual Facts: GPT-4.1 and GPT-5 were the only models to capture more residual facts than human summaries (residual ratio > 11%), though GPT-5 produced verbose, checklist-style summaries rather than narrative ones.

GAVEL-AGENT Efficiency

Token Reduction: GAVEL-AGENT achieved competitive performance while using 36–54% fewer tokens than end-to-end and chunk-by-chunk extraction methods in the legal domain.
Medical Domain: In the medical domain, GAVEL-AGENT achieved the best performance while using at least 77% fewer tokens than baseline methods.
Agent Architecture: A configuration using 26 individual agents (one per checklist item) outperformed a single agent extracting all items jointly, suggesting that item-level specialization aids in navigating long records.

Significance and Claims

The paper claims that GAVEL provides a robust, scalable, and cost-effective solution for evaluating long-context summarization, addressing the limitations of prior work that relied heavily on manual verification or simple n-gram metrics. By demonstrating that current LLMs fail primarily through omission rather than hallucination, the work highlights a specific area for model improvement. Furthermore, the success of GAVEL-AGENT suggests that agent-based scaffolds can effectively replace expensive human references for checklist extraction, making high-quality evaluation accessible for domains where expert summaries are unavailable. The authors emphasize that their work focuses on evaluation rather than improving summarization models directly, leaving the development of models that condition on extracted checklists as future work.