Tethered Reasoning: Decoupling Entropy from Hallucination in Quantized LLMs via Manifold Steering

1. Problem Statement

The paper addresses a fundamental dilemma in deploying quantized Large Language Models (LLMs) (specifically 4-bit precision) across varying sampling temperatures ($T$):

• Low Temperature ($T \to 0$): Yields repetitive, mode-collapsed outputs due to rigid determinism.
• High Temperature ($T > 2.0$): Conventionally causes "hallucination," semantic incoherence, and factual errors. Standard inference frameworks often cap $T$ at 2.0 to prevent this.
• The Quantization Penalty: Quantization introduces noise ($\epsilon_q$) that exacerbates the temperature-entropy trade-off. Quantized models tend to collapse (lose coherence) at lower temperatures than full-precision models because the quantization noise effectively raises the "temperature" of the system.

Core Hypothesis: The authors challenge the assumption that high-temperature "hallucination" is a semantic collapse (loss of knowledge). Instead, they posit it is trajectory divergence: hidden-state activations escaping the structurally coherent regions (manifold) of the representation space.

2. Methodology: The Helix Framework

The authors propose Helix, a geometric inference-time intervention framework that decouples output entropy from hallucination by tethering hidden-state trajectories to a pre-computed "truthfulness manifold."

A. Unified Truth Score (UTS)

Helix introduces a per-token uncertainty metric that combines two signals:

1. Semantic Entropy ($S_E$): Shannon entropy of the token distribution.
2. Manifold Distance ($S_D$): The Mahalanobis distance of the hidden activation vector from a pre-computed Truthfulness Manifold ($\mathcal{M}_{truth}$).

The UTS is calculated as a weighted sum:
$$\text{UTS}_t = \beta(T) \cdot S_E^{(t)} + (1 - \beta(T)) \cdot S_D^{(t)}$$

• Adaptive Handover: A sigmoid function $\beta(T)$ dynamically shifts weight based on temperature. At low $T$, entropy drives the score (precision); at high $T$ (where entropy saturates), manifold distance drives the score (stability).

B. Geometric Grounding (Truthfulness Manifold)

• Construction: The manifold is constructed unsupervised from hidden activations of factual prompts (from TruthfulQA, WikiText, GSM8K) generated at conservative temperatures ($T=0.1$).
• Hybrid Architecture Strategy: The framework targets Granite 4.0 H Small, a hybrid Mamba-2 (State-Space Model) and Transformer architecture. The authors found that steering only the sparse Transformer attention layers (approx. 10% of layers, specifically layers 4, 12, and 20) is sufficient to correct drift in the Mamba-2 state-space formulation.

C. Graduated Steering

When the UTS falls below a threshold, a steering vector (negative gradient of the potential field) is applied to the logits:

• Mechanism: A penalty is applied to the top logit proportional to the uncertainty.
• Efficiency: Intervention is triggered for only 0.2%–2.5% of tokens.
• Overhead: Computation occurs within the single forward pass (extracting existing logits/activations), adding negligible latency.

3. Key Contributions

1. Trajectory Divergence Hypothesis: Demonstrates that high-temperature failure is geometric (escaping the manifold) rather than semantic, allowing for targeted correction.
2. Unified Truth Score (UTS): A real-time, single-pass uncertainty metric combining entropy and geometric distance, eliminating the need for ensembles.
3. Quantization Recovery: Shows that geometric steering can allow 4-bit quantized models to exceed full-precision baselines.
4. Temperature-Invariant Reasoning: Achieves stable performance across a wide temperature spectrum ($T=0.5$ to $T=3.0$).
5. High-Entropy Creative Reservoir: Identifies a previously masked phenomenon where high temperatures, when tethered, unlock vast, non-overlapping creative spaces without sacrificing logical coherence.
6. Multi-Temperature Synthesis: A new inference paradigm querying across temperatures to generate 200%+ more unique concepts than single-temperature inference.

4. Experimental Results

The framework was evaluated on Granite 4.0 H Small (32B total, 9B active, 4-bit quantized) and cross-validated on Qwen3-30B-A3B (MoE).

• Quantization Recovery: At $T=1.0$, the steered 4-bit model (91.80%) outperformed the full-precision baseline (87.27%) by 4.53 percentage points.
• Steering Efficiency: Even at $T=3.0$, over 97% of tokens were generated without intervention.
• Creative Reservoir: In a songwriting task, high-temperature ($T>2.0$) generation with steering resulted in only 5–20% idea duplication (vs. 70–80% at low $T$), generating 30 unique concepts across 11 temperature points (200%+ increase over single-temp).

5. Significance and Implications

• Breaking the $T=2.0$ Barrier: The paper argues that the $T=2.0$ cap in standard frameworks is an artificial infrastructure limitation, not a model constraint. Helix enables reliable operation up to $T=3.0$.
• Democratizing AI: By enabling 4-bit quantized models to outperform full-precision baselines, Helix allows high-fidelity reasoning on resource-constrained hardware (e.g., consumer GPUs, edge devices).
• New Paradigm for Hybrid Architectures: The finding that steering sparse Transformer layers corrects drift in Mamba-2 state-space models offers a new control strategy for hybrid SSM-Transformer architectures.
• Creative Exploration: The discovery of the "High-Entropy Creative Reservoir" suggests that temperature can be treated as a creative dimension. By tethering the model to structural coherence (syntax/logic) rather than specific facts, the model can explore diverse semantic spaces without hallucinating.
• Observability: The framework provides per-token telemetry (UTS, Mahalanobis distance), offering unprecedented visibility into model uncertainty during inference without ensembles.

Conclusion: The paper demonstrates that hallucination in quantized LLMs is a geometric problem solvable via manifold steering. This approach not only stabilizes high-temperature inference but also unlocks latent creative potential, transforming temperature from a trade-off parameter into a tool for optimization and discovery.