Density Ridge Selective Prediction for LLM and VLM Hallucination Detection under Calibration Label Scarcity

Technical Summary: Density Ridge Selective Prediction for LLM and VLM Hallucination Detection

Problem Statement

The paper addresses the challenge of hallucination detection in Large Language Models (LLMs) and Vision-Language Models (VLMs) under conditions of calibration-label scarcity. While hallucination detection is increasingly framed as a selective prediction task (where a model abstains from answering if confidence is low), existing approaches face significant limitations:

• Unsupervised sampling detectors (e.g., Semantic Entropy, EigenScore) avoid the need for labels but often plateau in detection quality.
• Supervised hidden-state probes (e.g., SAPLMA) achieve strong in-distribution performance but degrade sharply when the number of available calibration labels is low.

The authors propose exploiting a complementary signal: the geometry of generation-time hidden state trajectories. They posit that repeated sampling at a fixed query traces a multimodal distribution in embedding space, where the modes encode distinct response strategies. Recent work suggests the curvature evolution of these trajectories is diagnostic of reasoning quality.

Methodology

The proposed method, Density Ridge Selective Prediction, recovers the "response manifold" of an LLM as a density ridge of a Kernel Density Estimate (KDE) built on kinematic features.

1. Kinematic Feature Map:
   For each sampled completion, the model extracts a hidden-state trajectory ($H_{i,j}$) from the final-layer, last-token states. A feature map $\phi$ projects these high-dimensional trajectories into a six-dimensional space ($\mathbb{R}^6$) using the following kinematic descriptors:

   • Mean curvature ($K_n$) and peak curvature ($K_{max}$).
   • Displacement linearity ($R^2$).
   • Mean and peak per-step displacement ($\|\Delta h\|$, $\|v\|_{max}$).
   • Normalized argmax of curvature ($\tau^\star$).
     These features are z-scored using training statistics.

2. Ridge Construction:

   • The method fits a Gaussian KDE ($\hat{p}$) only on the subset of correct trajectories ($\tilde{X}^+$).
   • It computes the gradient ($g$) and Hessian ($\nabla^2 \log \hat{p}$) of the log-density.
   • A 1-D density ridge ($R_1$) is defined as the set of points where the projected gradient onto the normal subspace is zero, and the corresponding eigenvalue is negative.
   • Subspace Constrained Mean Shift (SCMS) is used to iteratively project points to fixed points (vertices $V = \{v_k\}$) on this ridge.
   • Intrinsic dimension verification (via TwoNN) confirms the ridge is 1-dimensional ($r=1$).
   • Three coordinate variants are explored to represent the ridge: Global LTSA, Hastie–Stuetzle geodesic arclength, and a Stitched Local-Chart Atlas.

3. Scoring and OOD Interpretation:

   • For a test query, the model generates $N$ trajectories, maps them to the 6-D feature space, and calculates the off-ridge distance ($z_{off}$) as the minimum Euclidean distance to the nearest ridge vertex.
   • The final confidence score is the negated mean off-ridge distance: $s(q) = -\frac{1}{N} \sum z_{off}$.
   • Since the ridge is fitted exclusively on correct data, incorrect (hallucinated) generations are treated as Out-of-Distribution (OOD) regardless of their own density, resulting in larger distances and lower scores.

Key Contributions

1. Response-Manifold Detector: A novel detector that recovers the LLM response manifold as a density ridge over supervised, low-dimensional trajectory-curvature features.
2. Label-Scarce Evaluation: A rigorous evaluation across nine models (text and vision) and seven benchmarks (including HaluEval-QA, TriviaQA, GSM8K, POPE, ScienceQA, A-OKVQA) using a deliberately scarce calibration protocol ($n_{cal}=200$ queries).
3. Ablation Studies: Analysis isolating the contribution of ridge geometry across three parameterization variants (LTSA, Arclength, Atlas) and kernel configurations, demonstrating that the ridge structure, rather than generic embedding distance, drives performance.

Experimental Results

The method was evaluated against Semantic Entropy, log-probability, and various topological/geometric baselines (e.g., LID-MLE, Flow Matching, kNN).

• Performance Gains: The ridge-based score consistently outperformed all baselines in AUROC, showing gains of 5 to 20 percentage points across various model-dataset combinations.
  • Example: On Idefics3/A-OKVQA, AUROC improved from 0.795 (log-probability) to 0.972 (Ridge LTSA Chart).
  • Example: On SmolVLM/ScienceQA, AUROC rose from 0.790 to 0.990 (Ridge Arclength).
• Error Metrics: The method concurrently reduced AURC (Area Under the Risk-Curve) and AUGRC, indicating better risk management when abstaining.
• Stability: While non-ridge baselines (like Topology LID-MLE) degraded to near-chance levels (AUROC ~0.47–0.51) in certain multimodal settings, the ridge method maintained high performance.
• Quantization Robustness: The performance hierarchy (Ridge > Log-Probability > Non-Ridge) persisted under aggressive nf4 quantization, not just in bf16 precision.
• Complementarity: The geometric signal proved complementary to token-level confidence (log-probability), as the ridge method improved AUROC even on datasets where log-probability was already strong (e.g., HaluEval-QA).

Significance and Claims

The paper claims that the primary differentiator between faithful and hallucinated generations is not the raw location of the hidden states, but the shape of the trajectory-feature space. Specifically:

• Geometry over Location: Generic embedding distances (kNN, PCA) are insufficient; the recovered 1-D manifold density ridge is the critical signal utilized for detection.
• Robustness to Scarcity: The method demonstrates "tempered degradation" under label scarcity, recovering strong performance where supervised probes typically fail.
• Limitations: The authors note that the head-to-head comparison is restricted to cells common to all detectors. They also acknowledge that Semantic Entropy can degenerate to chance performance under data scarcity and that the supervised projection front-end requires both-class labels at fit time. Future work is suggested to include genuine per-axis kernel ablations and lifting distribution-free conformal guarantees to the ridge score.