Generative Gap Filling

Technical Summary: Generative Gap Filling

Problem Statement
The paper addresses a foundational empirical assumption in contract law: that when a contract is silent on a specific contingency, the remaining text provides "thin evidence" of the parties' actual intent. Consequently, legal scholars and judges often assume that gap-filling requires courts to rely on external sources—such as commercial defaults, trade usage, or the judge's own policy preferences—because the document itself has "run out." The authors argue that this assumption lacks empirical verification. If the text actually carries significant "mutual information" about missing terms, the domain for judicial policy-making is smaller than currently believed, and the worry that courts are unmoored from the parties' deal is overstated.

Methodology
To test the recoverability of missing terms from contract text, the authors employed a "masking" experimental design adapted from machine learning (specifically, masked language modeling).

• Ground Truth Creation: The researchers took real, executed contracts from court filings (PACER) and the SEC EDGAR database. They identified a negotiated term (typically a contingency provision) and masked it, creating a scenario where the "ground truth" (the actual term the parties wrote) was known to the researchers but hidden from the respondents.
• Respondents: The study utilized three distinct groups to predict the masked term based on the surrounding contract text and a specific dispute scenario:
  1. Lay Respondents: ~465 participants recruited via Prolific.
  2. Legal Experts: Law students (1L–3L) and practicing lawyers/judges.
  3. Large Language Models (LLMs): Six frontier models (including Opus 4.6, GPT-5.4, Gemini 3.1 Pro, etc.) run with temperature set to zero and randomized answer orders to prevent position bias.
• Scenarios: The initial study focused on three specific scenarios: an Artist engagement contract (cancellation due to injury), a Contingency Fee agreement (unauthorized settlement), and a Bottles supply contract (inventory based on forecasts).
• Robustness Checks:
  • Perturbation: Contracts were rewritten to flip their commercial direction (e.g., pro-buyer to pro-seller) to test if models were relying on the specific text or general defaults.
  • Blind Testing: Models were asked to predict outcomes without the contract text attached.
  • Scaling: A secondary study involved 119 recent commercial contracts from SEC filings, masking one material clause in each.

Key Results

• Human Performance: Lay respondents correctly identified the hidden terms approximately 55% of the time (double the chance rate of 25%). Legal professionals performed slightly better, with lawyers achieving ~59% accuracy. However, human performance varied significantly by scenario; for instance, lawyers struggled with the "Bottles" scenario where the deal deviated from industry norms.
• AI Performance: Large Language Models significantly outperformed all human groups, achieving an accuracy rate of 88.3% in the initial three-scenario study. In the scaled study of 119 contracts, models achieved 87% accuracy.
• Source of Accuracy:
  • Textual vs. General Knowledge: The authors estimate that roughly two-thirds of the models' success relied on general expectations of deal structure (market norms), while one-third relied on inferences drawn from specific contract language.
  • Robustness: When contracts were perturbed (flipped) or removed entirely, model accuracy dropped (to ~60% and ~68% respectively), confirming that models were not merely reciting boilerplate but were utilizing the specific text of the agreement.
  • Convergence: Different models, even with different architectures and prompts, converged on the same answers with less variance than human respondents.
• Limitations: Models struggled most with "bespoke" or anti-default terms (e.g., employment agreements or clauses reversing market standards), where accuracy fell to roughly 60%, though this still exceeded human performance.

Key Contributions

1. Empirical Validation of Textual Richness: The paper provides the first empirical evidence that contractual texts are far richer sources of intent than the literature assumes. "True gaps" (where the record runs out because no agreement existed) appear rarer than supposed; most silences can be reconstructed from the surrounding text.
2. Ground Truth for Interpretation: By using a masking design with known answers, the authors overcome the "ground truth" problem that has historically prevented the empirical testing of interpretative methods in law.
3. Generative Gap Filling as Evidence: The authors propose that AI predictions should be treated as contestable evidence in litigation, similar to expert testimony or dictionary definitions, rather than as a replacement for judicial decision-making.
4. Choice of Model Clauses: The paper introduces the concept of "Choice of Model" clauses, where parties can contractually stipulate which AI model and protocol governs the interpretation of ambiguous or omitted terms. This would allow parties to discipline the practice, fix the interpretive tool before a dispute arises, and potentially reduce litigation by clarifying silences during drafting.

Significance and Claims
The authors claim their findings challenge the "thin-evidence" assumption that drives much of modern contract theory and judicial gap-filling. They argue that:

• Judicial Role: Courts can rely on AI-generated predictions as ordinary, contestable evidence to infer what parties would have written, reducing the need for judges to inject their own policy preferences.
• Efficiency: The high accuracy of AI suggests that many disputes currently litigated as "gap-filling" cases could be resolved by reconstructing the parties' likely intent from the text, potentially reducing the volume of interpretation disputes.
• Future of Contracting: While the paper focuses on human-drafted contracts, it notes a looming challenge: as AI agents begin drafting contracts, the "signal" of human intent may disappear. The authors suggest that the "Choice of Model" framework and disclosure protocols are necessary steps to manage this transition, though they acknowledge that the legal doctrine for AI-drafted contracts remains undeveloped.

The paper concludes that while AI should not replace judges, it serves as a powerful tool to "de-bias" judicial overconfidence and extract information from deals that human readers might miss, provided the process remains adversarial and transparent.