Geometric Deviation as an Unsupervised Pre-Generation Reliability Signal: Probing LLM Representations for Answerability

Technical Summary: Geometric Deviation as an Unsupervised Pre-Generation Reliability Signal

Problem Statement
Hallucinations – the generation of confident but incorrect responses – remain a critical reliability challenge for deployed Large Language Models (LLMs). While prior work has investigated uncertainty calibration, internal probing classifiers, and representation steering, many existing methods rely on labeled error data, require access to model outputs (post-generation), or are sensitive to specific model architectures. The central question this work addresses is whether unsupervised representation geometry alone can serve as a practical pre-generation reliability signal. Specifically: Can the model recognize that a request is unanswerable before generating tokens, without labeled error examples or output sampling?

Methodology
The authors propose a minimal, unsupervised approach that measures the cosine distance of a prompt's hidden state from the centroid of a reference set of known answerable prompts ("Representations of the Answerable Class").

• Experimental Design: The study utilizes three instruction-tuned models of similar size (7–8 billion parameters): Llama 3.1-8B, Qwen 2.5-7B, and Mistral-7B.
• Prompt Construction (Paired Pairs): To isolate answerability from superficial variations (domain, length, syntax), the authors construct paired pairs for three prompt forms:
  • MATH: Well-defined arithmetic/algebraic questions (Answerable, A), paired with structurally identical variants containing undefined quantities or operations (Unanswerable, U).
  • FACT: Verifiable factual questions, paired with variants referring to unobservable future events or non-existent entities.
  • CODE: Python expressions with deterministic return values, paired with variants that are undefined, raise exceptions, or require infinite computations.
• Signal Extraction: Hidden states are extracted, mean-pooled over input tokens, and mean-centered to remove anisotropy. The reliability score is the cosine distance of a prompt to the centroid of the answerable (A) set within its specific form.
• Baselines: The geometric signal is compared against:
  1. Refusal Baseline: A keyword classifier on the generated output (detection of explicit refusals).
  2. Self-Consistency (SC): A post-generation baseline using 5 samples to measure output deviation.

Key Contributions and Findings

1. Geometry Encodes Task Form: Representation geometry is primarily organized by the syntactic form of the task (Math, Facts, Code) rather than just semantic content. Unanswerable prompts within a specific form tend to remain within the geometric cluster of that form, although their position relative to the centroid varies.
2. Form-Conditional Reliability Signal:
   • MATH (Strong Signal): Unanswerable math prompts consistently deviate from the centroid of answerable prompts. This yields a strong separation across all three models (ROC-AUC 0.78–0.84), significantly outperforming both the refusal baseline and self-consistency. The signal captures cases where models hallucinate without explicitly refusing.
   • FACT (No Signal): No reliable geometric separation occurs for factual unanswerability (p > 0.34). Factual unanswerability (e.g., future events) is syntactically indistinguishable from answerable facts, leaving the geometric cluster intact.
   • CODE (Mixed/Variable Signal): Large effect sizes are observed (Cohen's d > 1.0), suggesting the phenomenon generalizes to code, although statistical significance is mixed due to higher variance within the group for Python expressions compared to math.
3. Layer-wise Signal Profile: The answerability signal is an early-layer phenomenon. The gap between answerable and unanswerable representations peaks in layers 2–5 and generally decreases toward the final layer. This attenuation is caused by answerable representations migrating to a "generation-ready" manifold space in deeper layers, not by the fading of unanswerable signals.
4. Cross-Model Consensus vs. Behavioral Divergence: There is strong geometric agreement between models regarding certain types of unanswerable prompts (e.g., "the largest prime number"), indicating that the signal reflects input-side structural properties. However, behavioral responses diverge: Qwen tends to refuse these prompts, while Llama often hallucinates. This suggests that alignment training influences how models react to geometric anomalies, but not the existence of the anomaly signal itself.

Implications and Claims
The work claims that geometric deviation serves as a lightweight, unsupervised, pre-generation reliability signal, but with clear limitations:

• Effectiveness: It is highly effective in structured domains where unanswerability introduces structural inconsistencies (e.g., mathematically undefined operations, mistyped code). In these cases, it outperforms output-based baselines (refusal keywords, self-consistency) and operates with zero inference latency.
• Limitations: The signal is form-conditional and not universal. It fails to detect unanswerability in open factual queries where syntax remains valid despite the content being unrecognizable.
• Mechanism: The signal is established early in the network (layers 2–5), suggesting the model encodes structural violations before deciding on a generation strategy.

Conclusion
The authors conclude that while geometric deviation is not a universal hallucination detector, it offers a principled, low-cost mechanism for pre-generation reliability checks in domains with formal answerability constraints. The results underscore that the relationship between internal geometry and model behavior is modulated by alignment training, and that activations in early layers may provide stronger reliability signals than the final output layer.