💬 NLP

Pause or Fabricate? Training Language Models for Grounded Reasoning

This paper introduces Grounded Reasoning via Interactive Reinforcement Learning (GRIL), a multi-turn framework that trains language models to recognize information gaps and pause for clarification rather than fabricating premises, thereby significantly improving reasoning accuracy and efficiency on tasks with incomplete information.

Original authors: Yiwen Qiu, Linjuan Wu, Yizhou Liu, Yuchen Yan, Jin Ma, Xu Tan, Yao Hu, Daoxin Zhang, Wenqi Zhang, Weiming Lu, Jun Xiao, Yongliang Shen

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiwen Qiu, Linjuan Wu, Yizhou Liu, Yuchen Yan, Jin Ma, Xu Tan, Yao Hu, Daoxin Zhang, Wenqi Zhang, Weiming Lu, Jun Xiao, Yongliang Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Confident Liar"

Imagine you ask a very smart, confident friend for directions to a party. You tell them, "It's at the big red house on Main Street," but you forget to tell them which city Main Street is in.

A normal person might say, "Wait, which city is this in?"
But this paper argues that current Large Language Models (LLMs) act like a confident liar. They don't stop to ask. Instead, they immediately guess, "Oh, it must be in Springfield!" and then give you a detailed, step-by-step walking tour to a house that doesn't exist. They sound so sure of themselves that you might believe them, even though their entire answer is built on a made-up fact.

The authors call this "Ungrounded Reasoning." The model isn't bad at math or logic; it's just bad at knowing when it doesn't have enough information to start. It's like a chef who starts cooking a complex dish without checking if they have eggs, and then just invents a recipe using "imaginary eggs."

The Solution: GRIL (The "Stop and Ask" Training)

To fix this, the researchers created a new training method called GRIL (Grounded Reasoning via Interactive Reinforcement Learning).

Think of GRIL as a strict cooking instructor who changes the rules of the game. Instead of just rewarding the model for giving an answer, the instructor rewards the model for knowing when to stop.

The training happens in two distinct phases, like a game with two levels:

Level 1: The "Detective" Phase (Clarify and Pause)

The Scenario: The model is given a problem with missing info (like the "Main Street" example).
The Old Way: The model tries to solve it immediately, guesses the missing city, and gets it wrong.
The GRIL Way: The model is trained to act like a detective. If it sees a gap, it must pause and say, "I can't solve this yet; I need to know the city."
The Reward: The instructor gives a bonus point for stopping early. If the model guesses for 10 minutes before asking, it gets a tiny reward. If it asks immediately, it gets a huge reward. This teaches the model that "stopping to ask" is a victory, not a failure.

Level 2: The "Chef" Phase (Grounded Reasoning)

The Scenario: The user (or the environment) finally provides the missing info ("It's in Springfield!").
The Action: Now that the "ingredients" are complete, the model switches to "Chef Mode." It uses the new info to solve the problem correctly.
The Reward: It gets points for solving the math problem correctly.

Why This is a Big Deal

The paper shows that this simple change in training makes a massive difference:

Less Hallucination: The models stopped making up facts. They learned that if they don't know something, they should admit it.
Better Accuracy: Because they stopped guessing, they actually got the right answers much more often (up to a 30% increase in success rate).
Efficiency: They became faster. Instead of writing a long, fake story to cover up missing info, they just asked the question and got the answer. This saved a lot of "computing time" (tokens).
General Smarts: This skill wasn't just for math. The models learned to apply this "pause and check" logic to other types of questions, like reading comprehension or common sense, even ones they hadn't seen before.

The Takeaway

Before this paper, AI models were like over-eager interns who would try to finish a project even if half the data was missing, often making things up to fill the gaps.

After GRIL training, they become wise consultants. They know that a good answer starts with a good question. They have learned the most important rule of reasoning: It is better to pause and ask for clarity than to confidently guess and get it wrong.

In short: Don't just answer. Know when to stop.

1. Problem Definition: Ungrounded Reasoning

The paper identifies a critical failure mode in Large Language Models (LLMs) termed ungrounded reasoning.

The Phenomenon: When presented with incomplete inputs (missing premises), current LLMs often fail to recognize the information gap. Instead of pausing to request clarification, they "fabricate" missing information (hallucinate assumptions) and proceed with elaborate, confident, but logically invalid reasoning chains.
The Root Cause: The authors argue this is not a lack of reasoning capability, but a lack of inferential boundary awareness. Models are trained to maximize answer production (often via Reinforcement Learning from Human Feedback or standard RL) without mechanisms to detect when a valid answer is impossible due to missing data.
Metrics: The paper introduces two metrics to quantify this:
- GapRatio: The proportion of tokens generated after the model first expresses uncertainty (indicating continued fabrication).
- Premise Detection Rate: The frequency with which models correctly identify inputs as incomplete.
- Observation: Baseline models often show high GapRatios (>47%) and low detection rates (<41%), indicating they override early uncertainty signals to continue generating.

2. Methodology: Grounded Reasoning via Interactive Reinforcement Learning (GRIL)

The authors propose GRIL, a multi-turn Reinforcement Learning (RL) framework designed to teach models to distinguish between "sufficient" and "insufficient" information states.

Core Framework

GRIL decomposes the reasoning process into two distinct stages within a Markov Decision Process (MDP):

Stage 1: Clarify and Pause
- The model assesses the input.
- Action a_clarify: If the model detects missing premises, it stops and requests clarification.
- Action a_solve: If the model attempts to solve without sufficient info, it receives negative feedback.
- Reward ( $R_{detect}$ ): A time-decay reward is given for correctly identifying missing information. $R_{detect} = r_{base} \cdot \gamma^{n_{prior}}$ , where $n_{prior}$ is the number of turns before detection. This penalizes delayed detection and encourages early stopping.
Stage 2: Grounded Reasoning
- Triggered only after the environment provides the missing premise (upon successful clarification).
- The model integrates the new information and solves the task.
- Reward ( $R_{solve}$ ): A standard correctness reward is given only if the final answer is correct.

Training Environment & Rewards

Interactive Loop: The environment acts as an oracle. If the model correctly identifies an incomplete problem and asks for clarification, the environment provides the missing premise. If the model tries to solve an incomplete problem, it receives negative feedback.
Complete Problems: For problems with all information present, the model is rewarded for solving and penalized for unnecessary clarification requests (to prevent over-conservatism).
Optimization: The framework uses Proximal Policy Optimization (PPO) with a KL-divergence penalty to keep the policy close to the reference model.

3. Key Contributions

Identification of a Failure Mode: The paper formally defines "ungrounded reasoning" as a distinct failure mode where models prioritize answer generation over truthfulness, providing quantitative metrics (GapRatio, Detection Rate) to measure it.
Reframing Reasoning: It reframes reasoning under uncertainty as a sequential decision-making problem, establishing that "inferential boundary awareness" (knowing when not to reason) is a skill distinct from logical reasoning capability.
GRIL Framework: Proposes a multi-turn RL framework with stage-specific rewards that explicitly trains models to pause, detect gaps, and resume reasoning only after premises are established.
Empirical Validation: Demonstrates that GRIL significantly outperforms Supervised Fine-Tuning (SFT) and prompting baselines, proving that RL is necessary to learn the "stop" behavior effectively.

4. Experimental Results

The authors evaluated GRIL on GSM8K-Insufficient and MetaMATH-Insufficient (datasets where critical premises were systematically removed) across various model scales (Qwen2.5-1.5B to Qwen3-1.7B).

Premise Detection: GRIL drastically improved the ability to detect missing information.
- On Qwen2.5-1.5B, detection rates jumped from 4.6% (Base) to 90.8% (GRIL).
Task Success: By stopping fabrication and waiting for clarification, success rates increased massively.
- On Qwen2.5-1.5B, Success Rate (SR) improved from 1.8% to 61.6%.
- On Qwen2.5-3B, SR reached 73.5%, outperforming SFT (62.7%).
Efficiency: GRIL reduced average response length by 20–40% (e.g., 810 to 479 tokens for 1.5B models) because models stopped generating hallucinated reasoning chains early.
Generalization:
- Complete Problems: GRIL did not degrade performance on standard, complete benchmarks (GSM8K, MATH500); in fact, it improved performance by up to 7%, suggesting that learning to check premises improves overall reasoning rigor.
- Out-of-Distribution: The method generalized to non-math domains (HotpotQA, CommonsenseQA) and remained robust against noisy user feedback (e.g., irrelevant answers or "I don't know").

5. Significance

Safety and Reliability: GRIL addresses a fundamental safety issue in LLMs: the tendency to hallucinate confidently. By teaching models to admit uncertainty and request missing data, it produces more reliable and grounded outputs.
Paradigm Shift: It challenges the standard "single-turn, information-complete" evaluation paradigm, advocating for interactive, multi-turn reasoning where the model has agency to control the flow of information.
Efficiency: By reducing the generation of "ungrounded" tokens, GRIL offers a more computationally efficient reasoning process, saving inference costs while improving accuracy.
Future Direction: The paper highlights that future AI agents must possess "inferential control"—the ability to decide when to reason and when to stop—rather than just being better at reasoning.

In summary, GRIL demonstrates that with the right interactive RL training, LLMs can learn to recognize the limits of their knowledge, stop fabricating facts, and achieve superior performance in both incomplete and complete reasoning scenarios.