💬 NLP

When Should LLMs Be Less Specific? Selective Abstraction for Reliable Long-Form Text Generation

This paper introduces Selective Abstraction, a framework that enhances the reliability of long-form LLM generation by replacing uncertain atomic claims with less specific, higher-confidence abstractions, thereby significantly improving accuracy while preserving the overall informational content of the response.

Original authors: Shani Goren, Ido Galil, Ran El-Yaniv

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shani Goren, Ido Galil, Ran El-Yaniv

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Overconfident Chef"

Imagine you hire a very talented chef (a Large Language Model, or LLM) to write a detailed biography of a famous person. The chef is great at writing, but sometimes they get a little too confident about specific details they aren't 100% sure of.

The Old Way (The "All-or-Nothing" Approach): If the chef isn't sure about one ingredient (e.g., the exact date of birth), the current safety rule says: "Throw away the whole dish." The chef refuses to serve the biography at all. This is safe, but it's wasteful. You lose all the good information about the person's life just because of one shaky detail.
The Real-World Consequence: In high-stakes situations (like law or medicine), a single wrong fact can ruin trust. But in long stories, throwing away the whole story is too extreme.

The New Solution: "Selective Abstraction"

The authors propose a new strategy called Selective Abstraction (SA). Instead of throwing the whole dish away, the chef learns to simplify the uncertain parts while keeping the rest of the meal delicious.

Think of it like a cartographer (map-maker) drawing a map of a new territory:

Too Specific (Risky): "This path leads to a red oak tree with a squirrel named Steve." (If the chef is wrong about the squirrel, the whole map is a lie.)
Too Vague (Useless): "There is a forest somewhere." (This is safe, but tells you nothing.)
Selective Abstraction (The Sweet Spot): "There is a forest with trees." (If the chef isn't sure about the specific type of tree or the squirrel, they just say "trees." The map is still useful, but now it's reliable.)

How It Works: The "Atom" Assembly Line

The paper breaks this process down into four steps, like a factory assembly line:

Generation (The Draft): The AI writes the full story first.
Atomization (Breaking it Down): The AI breaks the story into tiny, single-fact "atoms."
- Example: Instead of one sentence saying "Amelia Earhart was born in Chicago, Illinois, on July 24, 1897," it breaks this into three separate atoms:
  - Atom A: Born in the US.
  - Atom B: Born in Illinois.
  - Atom C: Born on July 24, 1897.
The Confidence Check & Abstraction (The Safety Filter): The AI asks itself, "How sure am I about each atom?"
- If it's 99% sure about "Born in the US," it keeps it.
- If it's only 60% sure about the exact date "July 24," it doesn't delete the sentence. Instead, it abstracts it. It changes "July 24, 1897" to "in the 1890s" or "in the 20th century."
- The Analogy: It's like zooming out on a camera. If the focus is blurry on the specific face, you zoom out until the whole head is in focus. The image is less detailed, but it's no longer blurry.
Reconstruction (Putting it Back Together): The AI stitches these simplified, safe atoms back into a coherent story.

The Results: More Truth, Less Detail

The researchers tested this on six different AI models using two big datasets (one about biographies, one about general facts). They compared their method to other ways of handling uncertainty, like just deleting uncertain sentences (Redaction).

The Metric: They measured the trade-off between Risk (how many lies are in the text) and Coverage (how much useful information is left).
The Finding: Selective Abstraction was the clear winner. It reduced the "Risk" (lies) significantly without throwing away as much "Coverage" (useful info) as the other methods.
- The Analogy: Imagine you have a bucket of water with some mud in it.
  - Redaction is like pouring the whole bucket out because of the mud. You have no water left.
  - Selective Abstraction is like filtering the water. You lose a little bit of volume, but you get a clean, drinkable glass of water.
- The paper found that this method improved the balance between safety and usefulness by up to 27.73% compared to just deleting sentences.

Why This Matters

The paper argues that in the real world, we don't always need the most specific answer; we need the most reliable answer.

The Bard Incident: The paper mentions a real-world example where Google's AI (Bard) confidently claimed a space telescope took the "very first pictures" of an exoplanet. This was wrong by nearly 20 years.
The Fix: If the AI had used Selective Abstraction, it might have said, "The telescope has taken pictures of exoplanets," or "It has contributed to major discoveries." It would have been less specific, but it would have been true.

Summary

This paper introduces a way for AI to say, "I'm not 100% sure about this specific detail, so I'm going to say something slightly more general that I am sure about," instead of either lying confidently or staying completely silent. It turns the AI from a rigid "all-or-nothing" robot into a flexible, honest storyteller.

Technical Summary: Selective Abstraction for Reliable Long-Form Text Generation

Problem Statement

Large Language Models (LLMs) are prone to factual errors, which erode user trust and limit adoption in high-stakes domains like healthcare and law. While uncertainty estimation mechanisms exist to mitigate this, current approaches rely on a binary "all-or-nothing" decision: either the model generates a response or it abstains entirely. This binary approach is excessively restrictive for long-form generation. Long responses often contain numerous factual claims with varying levels of uncertainty; abstaining from an entire generation due to a single uncertain detail discards valuable information. Conversely, generating the full text without modification risks including hallucinations. The paper identifies a critical need for a fine-grained approach that allows models to trade specificity for reliability without discarding entire responses.

Methodology: Selective Abstraction (SA)

The authors introduce Selective Abstraction (SA), a framework that enables LLMs to reduce the detail of uncertain content rather than removing it entirely. The core idea is to replace low-confidence claims with higher-confidence, less specific abstractions.

Framework Formalization

SA is formalized through the lenses of selective risk and coverage:

Risk: Defined as the rate of factual errors (hallucinations) in the generated text.
Coverage: Defined as the fraction of retained information relative to the original generation.
Trade-off: By varying a confidence threshold $\theta$ , SA induces a trade-off curve where higher thresholds reduce risk (by increasing abstraction) but also reduce coverage (by removing specificity).

Atom-wise Selective Abstraction

The paper proposes a specific instantiation called Atom-wise Selective Abstraction, which operates at the claim level through a four-stage pipeline:

Generation: The LLM generates an initial response $T$ to a prompt $p$ .
Atomization: The response is decomposed into atomic claims (atoms). These are short, self-contained statements, each expressing a single fact (e.g., "Amelia Earhart was born in Atchison, Kansas").
Abstraction: This is the core processing stage, consisting of three steps:
- Confidence Elicitation: The model assigns a confidence score to each atom.
- Abstraction Generation: For atoms with confidence below a threshold $\theta$ , the model generates a sequence of increasingly general abstractions (e.g., "Atchison, Kansas" $\to$ "Kansas" $\to$ "Midwestern US" $\to$ "US"). The sequence ends with full abstention ( $\top$ ).
- Selection: The model assigns confidence scores to the abstraction sequence. It selects the most specific abstraction in the sequence that satisfies the confidence threshold $\theta$ . If no abstraction meets the threshold, the atom is abstained from.
Reconstruction: The selected abstractions are reconstructed into a coherent text $T'$ .

Evaluation Pipeline

To evaluate SA, the authors developed an end-to-end pipeline for open-ended factual generation:

Risk Measurement: Uses an agentic fact-checking system that verifies atomic claims against Wikipedia. A claim is labeled "supported" if evidence is found, otherwise "unsupported." Risk is calculated as $1 - \text{accuracy}$ .
Coverage Measurement: Uses an information-theoretic measure. The information content of a claim is quantified by the reduction in entropy it provides regarding the set of compatible entities (e.g., "born in London" is more informative than "born in Europe" because it narrows the set of people more significantly). Coverage is the ratio of information retained in $T'$ to that in $T$ .
Metrics: Performance is summarized using Risk-Coverage (RC) curves and the Area Under the Risk-Coverage Curve (AURC). Lower AURC indicates a better trade-off.

Key Contributions

Selective Abstraction Framework: A general framework for trading specificity for reliability in long-form text generation, formalized via selective risk and coverage.
Atom-wise Instantiation: A claim-level procedure that selectively replaces uncertain claims with higher-confidence abstractions rather than deleting them.
Evaluation Pipeline: A novel end-to-end pipeline for open-ended generation that instantiates risk via factual correctness and coverage via an information-theoretic measure of retained content.
Empirical Validation: Demonstrations across six open-source LLMs and two benchmarks (FactScore and LongFact-Objects) showing consistent improvements over baselines.
Threshold Selection Algorithm: A risk-guided algorithm based on split conformal prediction to select confidence thresholds that meet user-defined target risks with high probability.

Results

The authors evaluated the method on six open-source models (including GPT-OSS, Qwen3, and Llama 3.3) using the FactScore and LongFact-Objects benchmarks.

Performance vs. Baselines: Atom-wise SA consistently outperformed existing baselines, including Selective Redaction (removing uncertain atoms entirely), Inline Abstraction (generating at a coarser granularity from the start), and Self-Revision (post-hoc rewriting).
AURC Improvements: Atom-wise SA improved the AURC by up to 27.73% over claim removal (Redaction) on the FactScore benchmark. Average improvements were 17.43% on FactScore and 13.77% on LongFact.
Risk-Coverage Trade-off: The method achieved a smoother trade-off curve, dominating baselines across the full coverage range. At matched coverage levels, Atom-wise SA achieved significantly lower risk (average gap of ~10 percentage points) compared to Inline and Self-Revision baselines.
Threshold Selection: The proposed algorithm successfully selected thresholds that kept empirical risk close to user-defined target risks across different models and datasets.

Significance and Claims

The paper argues that reliability in long-form generation is not a binary choice between answering and abstaining, but a spectrum of specificity versus correctness. The significance of this work lies in demonstrating that reducing specificity can boost accuracy and reliability while preserving most of the original meaning.

By operating at the level of individual claims, the proposed method avoids the abrupt information loss associated with deleting entire sentences or claims. Instead, it provides targeted, claim-specific control over uncertainty. The authors position Selective Abstraction as a step toward generation paradigms that explicitly reason about the appropriate amount of detail in a response, suggesting that in high-stakes settings, less specific responses are preferable to confident errors.

The paper remains modest regarding future directions, noting that the quality of abstraction depends heavily on the confidence scoring function (verbalized confidence was found to be superior to log-prob-based scores) and that modeling dependencies between claims (rather than treating them as independent) could further improve results.