Don't Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning

1. Problem Definition

Large Language Models (LLMs) are prone to factuality hallucinations, particularly when responding to queries containing False Premises (FPQs). A false premise is a statement within a user's query that contradicts established facts (e.g., "Is penicillin safe for treating shellfish allergies?" implies a false connection between penicillin and shellfish allergies).

• The Challenge: LLMs often trust the embedded assumptions in the prompt and generate plausible-sounding but factually incorrect responses.
• Limitations of Existing Solutions:
  • Post-hoc mitigation: Methods like contrastive decoding or uncertainty analysis often occur after generation or require access to model logits (white-box), limiting their use with black-box models.
  • Training-based approaches: Fine-tuning or pre-training to detect false premises is computationally expensive and requires extensive data.
  • Semantic flow: False premise questions often maintain normal semantic flow, making them difficult to detect via traditional out-of-distribution detection.

2. Methodology

The authors propose a retrieval-augmented logical reasoning framework that prevents hallucinations by detecting and flagging false premises before the LLM generates a response. The process consists of three main stages:

A. Logical Form Extraction

Instead of processing the raw natural language query, the system first converts the user query ($q$) into a structured Logical Form ($L(q)$).

• Process: An LLM (e.g., GPT-4o-mini) parses the query to extract predicates and entities, transforming the text into a symbolic representation (e.g., Nominated("Lord of the Rings", "Best Director")).
• Purpose: This abstraction isolates the core factual claims, making it easier to verify specific relationships against a knowledge base without being distracted by linguistic nuances.

B. Structured Retrieval and Verification

The system uses a Retrieval-Augmented Generation (RAG) approach to verify the logical form against a Knowledge Graph (KG).

• Retrievers: The paper evaluates three types of retrievers to find relevant evidence (triples/subgraphs) in the KG:
  1. Embedding-based: Uses pre-trained encoders (e.g., all-roberta-large-v1) to compute cosine similarity between the logical form and graph triples.
  2. Non-parametric (G-retriever): Uses the Prize-Collecting Steiner Tree (PCST) algorithm to extract relevant subgraphs without learnable parameters.
  3. LLM-based: Uses models like GraphRAG/ToG to score the relevance of retrieved subgraphs.
• Verification: The system compares the retrieved evidence ($G^*$) with the logical assertions in the query. If the query contradicts the retrieved evidence, a False Premise Indicator ($F(q)$) is triggered ($F(q)=1$).

C. Hallucination Mitigation (Prompt Engineering)

Once a false premise is detected:

• Flagging: The original query is updated by appending a specific note: "Note: This question contains a false premise."
• Generation: The LLM generates the final response using this modified prompt. This explicit instruction forces the model to reject the false assumption rather than hallucinating a confirmation.
• Advantage: This method requires no access to model logits, no fine-tuning, and operates entirely at the inference level (black-box compatible).

3. Key Contributions

1. Logical Form Representation: The introduction of logical forms as an intermediate representation for query analysis. The paper demonstrates that this structured approach significantly improves the accuracy of premise detection across various retriever types compared to using raw text.
2. Explicit Premise Detection: A novel framework that explicitly identifies false premises and signals them to the LLM before generation, shifting the paradigm from "mitigating hallucinations after they happen" to "preventing them proactively."
3. Model-Agnostic Efficiency: The approach is lightweight, requiring zero training cost and no access to internal model weights, making it easily integrable into existing black-box LLM pipelines.

4. Experimental Results

The method was evaluated on three datasets: KG-FPQ (Knowledge Graph-based False Premise Questions), CREAK (Commonsense reasoning), and FEVER (Fact extraction).

• Premise Detection Performance:
  • On the KG-FPQ dataset, using logical forms in both retrieval and detection stages with the G-retriever achieved the highest performance: 94.44% True Positive Rate (TPR) and 97.12% F1 Score.
  • This significantly outperformed baselines like "Direct Claim" (LLM answering Yes/No directly) and standard RAG without logical forms.
  • Multi-hop vs. Single-hop: The method showed the most significant gains on multi-hop questions (complex reasoning), where detection accuracy jumped from ~68% (direct claim) to 95.2% (G-retriever with logical forms).
• Hallucination Mitigation:
  • The proposed method achieved the highest accuracy across multiple LLMs (GPT-4o-mini, GPT-3.5, Llama-3.1, Qwen2.5).
  • For example, on Qwen2.5, the method achieved 95.2% accuracy, surpassing baselines like Majority Vote, Direct RAG, and Perplexity AI.
  • Unlike "Prompt" methods that vaguely warn of potential errors, this method's explicit flagging of the specific false premise led to better performance on valid questions (avoiding unnecessary caution).
• Efficiency:
  • The method adds minimal computational overhead (approx. 0.6s extra running time per query) compared to Contrastive Decoding (approx. 10.6s), which requires repeated decoding steps.

5. Significance and Impact

• Safety and Reliability: The framework offers a principled way to prevent LLMs from engaging with misinformation, particularly in high-stakes domains like healthcare and finance where false premises can lead to dangerous advice.
• Scalability: By avoiding fine-tuning and logit access, this approach is immediately deployable for black-box models, addressing a critical gap in current hallucination mitigation research.
• Reasoning Enhancement: The results suggest that converting natural language into logical forms enhances the model's ability to perform deep, multi-hop reasoning and fact-checking, rather than relying on surface-level pattern matching.

In conclusion, the paper presents a robust, efficient, and model-agnostic solution to the "false premise" problem, demonstrating that pre-generation verification via logical reasoning is a superior strategy to post-hoc correction for ensuring factual consistency in LLM outputs.