💬 NLP

Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis

This paper argues that large-scale pretraining alone is insufficient to prepare large language models for reliable legal "ordinary meaning" analysis, as empirical evidence reveals their conclusions lack robustness against minor prompt variations and show only moderate correlation with human judgments, rendering them unsuitable for authoritative use in high-stakes judicial contexts.

Original authors: Abhishek Purushothama, Junghyun Min, Brandon Waldon, Nathan Schneider

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Abhishek Purushothama, Junghyun Min, Brandon Waldon, Nathan Schneider

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a judge in a courtroom trying to figure out what a confusing word in a law actually means to the average person. For example, does a contract term like "landscaping" cover building a trampoline? Traditionally, judges might ask themselves, "What would my neighbor think?" or look up words in a dictionary.

Recently, some legal experts and judges have started asking Large Language Models (LLMs)—the same kind of AI that powers chatbots—to answer this question. The idea is: "If an AI has read almost everything ever written on the internet, it must know exactly how ordinary people use words, right?"

This paper is a big, careful experiment to test if that idea is true. The researchers acted like "tough critics" sitting on the bench, putting these AI models through a rigorous stress test. Here is what they found, explained simply:

1. The "Broken Clock" Problem

The researchers asked 15 different AI models (ranging from small ones to the massive GPT-4) to decide if 138 different insurance scenarios were covered or not.

The Finding: Some of the AI models were like a broken clock that is stuck on "No." No matter what the story was about, they gave the exact same answer every single time.
The Metaphor: Imagine asking a weather forecaster, "Is it raining?" If they say "Yes" for a sunny day, a cloudy day, and a snowstorm, they aren't actually looking at the sky. They are just repeating a script. Several of the AI models did exactly this, failing to actually "read" the specific details of the legal case.

2. The "Magic Trick" of Prompting

The researchers then tried a magic trick: they asked the exact same question, but just changed the way they asked it.

Scenario A: "Is John covered? Yes or No?"
Scenario B: "Is John not covered? Yes or No?"
Scenario C: "Do you agree that John is covered?"
The Finding: The AI's answer would flip-flop wildly. In one version of the question, the AI might be 90% sure John is covered. In a slightly different version (like adding a "not" or flipping the order of options), the AI might become 90% sure he is not covered.
The Metaphor: It's like a suggestion box that changes its mind based on the color of the paper you write on. If you ask a model a question one way, it says "Yes." If you ask it the same question but put the "No" option first, it says "No." This means a lawyer could potentially "shop around" for the prompt that gives them the answer they want, rather than getting the truth.

3. The "Translator" Who Doesn't Speak the Language

The researchers compared the AI's answers to what real humans (ordinary English speakers) actually thought.

The Finding: The AI and the humans were only moderately in agreement. Even the best AI models (like GPT-4) only got it right about 83% of the time if you knew exactly how to interpret their probability scores. But in a real courtroom, you don't get to peek at the scores; you just get the answer.
The Metaphor: Imagine hiring a translator to tell you what a crowd of people is thinking. If the translator gets it right 83% of the time, that sounds good. But in a legal case, getting it wrong 1 in 6 times is a disaster. It's like a GPS that gets you to the right city 83% of the time but drops you in a swamp the other 17%.

The Bottom Line

The paper argues that just because an AI has read a lot of data (large-scale pretraining) doesn't mean it understands how ordinary people think.

The authors conclude that these AI models are currently too fragile and unreliable to be used as the final word in legal cases. They are not "authoritative" sources of ordinary meaning yet. If a judge relies on them without checking their work, they might be trusting a broken clock or a suggestion box that changes its mind based on the font size.

In short: The AI is a fluent talker, but it's not a reliable thinker when it comes to deciding what words mean to regular people.

Technical Summary: Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis

Problem Statement

The paper addresses the growing trend in U.S. judicial practice where judges and legal scholars utilize Large Language Models (LLMs) to ascertain the "ordinary meaning" of legal texts—a standard interpretive methodology that asks how a term would be understood by an ordinary speaker of English. Proponents, including Federal Judge Kevin Newsom, argue that the massive scale of pretraining data allows LLMs to synthesize statistical patterns of ordinary language use more effectively than traditional tools like dictionaries or small-scale surveys.

The authors challenge the assumption that large-scale pretraining and instruction tuning are sufficient to produce models capable of reliable ordinary meaning analysis. They posit that without rigorous evaluation, relying on LLMs for high-stakes legal interpretation is premature. The core problem is twofold:

Robustness: Whether LLM judgments are stable across minor variations in prompt phrasing (a known vulnerability in NLP).
Correlation: Whether LLM judgments actually approximate the interpretive consensus of lay human readers, which is the target of "ordinary meaning" inquiries.

Methodology

The authors conducted a large-scale empirical evaluation involving 15 LLMs (including base and instruction-tuned variants of Llama-3, OLMo-2, Mistral, Gemma, and GPT-2, alongside the closed-source GPT-4) across 138 legal interpretation scenarios.

Data and Task Formulation:

Scenarios: The study utilized 138 items derived from real-world insurance contract disputes (originally from Waldon et al., 2023). Each item presents a hypothetical scenario and a specific insurance term (e.g., "landscaping," "internal source") requiring a binary judgment: is the loss covered ("Yes") or not ("No")?
Human Baseline: The study compared model outputs against human judgments from 1,338 native English speakers who evaluated the same 138 scenarios.
Prompt Variants: To test robustness, the authors applied 9 systematic question variants to each item. These included:
- Standard Yes/No and No/Yes ordering.
- Negation (e.g., "is John not covered?").
- Agreement/Disagreement framing (e.g., "Do you agree that...").
- Multiple-choice options (A/B) with flipped orders.
Measurement:
- Robustness: Measured by the consistency of categorical judgments (Covered vs. NotCovered) across the 9 prompt variants for each model-item pair.
- Human Correlation: Measured using the coefficient of determination ( $R^2$ ) between the model's probability difference ( $\Delta = p(\text{Covered}) - p(\text{NotCovered})$ ) and the proportion of human "Covered" judgments. The authors treated the model's token probability as a proxy for population consensus.

Key Contributions

Systematic Evaluation of Legal Interpretation: This work provides the most extensive experimental evaluation of LLMs for legal interpretation to date, moving beyond qualitative case studies or small-scale paraphrasing tests (e.g., Choi, 2025; Waldon et al., 2025) to a controlled, large-scale dataset.
Quantification of Prompt Sensitivity: The authors demonstrate that LLMs exhibit "ubiquitous lack of consistency" across question variants. They show that trivial changes, such as negation or option ordering, can cause drastic shifts in model judgments, effectively allowing "prompt shopping" for desired outcomes.
Analysis of Human Correlation: The study rigorously tests the "omniscience" hypothesis (that training data scale equates to human-like intuition). It finds that even the best-performing models show only moderate correlation with human consensus, and this correlation is highly dependent on specific model sizes and prompt formulations.
Critique of Current Judicial Practice: The paper offers an empirical counter-argument to recent judicial opinions that rely on direct LLM queries for ordinary meaning, suggesting that current models fail to meet the necessary conditions of robustness and human alignment for such use.

Key Results

Instability and Bias:
- Stopped Clocks: Several models (6 out of 15) provided the same categorical judgment for over 92% of scenarios, regardless of the content, behaving like "stopped clocks."
- Prompt Sensitivity: In 2,061 out of 2,070 model-item combinations, the model produced conflicting judgments across the 9 prompt variants. Only 9 combinations showed full unanimity.
- Negation and Complexity: Variants involving negation (e.g., "Disagreement with Negation") frequently induced minority judgments and caused the largest distributional shifts (measured by Jensen-Shannon distance).
Correlation with Human Judgment:
- Moderate Correlation: Only larger, instruction-tuned models (specifically 70B+ parameters) showed any significant correlation with human judgments, and even then, only for specific prompt variants.
- Best Case Scenario: GPT-4 achieved the highest correlation ( $R^2 = 0.60$ ) with human judgments on the standard Yes/No variant. However, this translated to an oracle classifier accuracy of only 83% (1 in 6 error rate), which the authors argue is insufficient for legal interpretation where the stakes are high.
- Inconsistency: For many models, the correlation was negative or near zero. Some models showed strong correlation in probability distributions but produced overwhelmingly biased categorical outputs (e.g., always predicting "NotCovered" despite probability gradients suggesting otherwise).
Model Size and Tuning: While larger models generally performed better, instruction tuning introduced unpredictable biases, sometimes shifting judgments entirely (e.g., Llama-8B tuned models predominantly predicted "NotCovered," while Llama-70B tuned models predicted "Covered").

Significance and Claims

The paper concludes that large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis. The authors argue that the assumption that LLMs can reliably illuminate how ordinary people use words is not supported by empirical evidence.

Reliability Gap: The findings highlight a "reliability gap" where models are not consistent or reproducible, and an "epistemic gap" where model outputs do not accurately measure ordinary human meaning.
Judicial Caution: The authors explicitly caution against the approach taken by Judge Newsom and others who directly query LLMs to settle questions of ordinary meaning. They assert that the observed fragility (sensitivity to prompt phrasing) and weak correlation to human consensus render current LLMs unsuitable as authoritative tools for this specific legal task.
Future Work: The paper does not claim that LLMs can never be used for legal interpretation but suggests that if they are to be used, they require rigorous validation beyond current capabilities. They note that newer "reasoning" models or systems with retrieval mechanisms were not tested and might mitigate some limitations, but such claims require further empirical demonstration. The current evidence, they state, "spectacularly fails to meet the burden of proof" for using LLMs as authoritative interpreters of ordinary meaning.