🤖 AI

Neural Diversity Regularizes Hallucinations in Language Models

This paper introduces "neural diversity" as a third scaling axis for language models, proposing the ND-LoRA method to reduce hallucinations by up to 25.6% through decorrelated parallel representations and Barlow Twins regularization, while establishing formal bounds that link lower neural correlation to higher reliability.

Original authors: Kushal Chakrabarti, Nirmal Balachundhar

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kushal Chakrabarti, Nirmal Balachundhar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a tricky riddle, but instead of asking just one person, you ask a whole team. If everyone on the team thinks exactly alike, they might all make the same silly mistake and give you the wrong answer together. This is the core problem with modern "language models"—the super-smart computer programs that write stories, answer questions, and chat with us. Even though these programs are getting bigger and smarter, they still sometimes make things up, a glitch called "hallucination." They might confidently state a fake fact or invent a story that never happened.

To fix this, scientists usually try to make the models bigger or feed them more data, like trying to solve a puzzle by buying a bigger box. But this paper suggests a different approach: instead of making the team bigger, make the team members different. The authors borrow an idea from finance called "diversification." Just as an investor doesn't put all their money in one stock because if that one fails, they lose everything, they spread their money across many different stocks. If one stock crashes, the others might hold steady, keeping the portfolio safe. The paper asks: Can we do the same thing with computer brains? If we force the different parts of the model to think in slightly different, uncorrelated ways, can we stop them from all making the same mistake at the same time?

This is exactly what the researchers set out to test. They propose a new method called Neural Diversity, which acts like a "diversity coach" for the computer's brain. Instead of letting all the internal pathways of the model collapse into thinking the same way, they force these pathways to stay distinct and independent. They built a tool called ND-LoRA to do this. Think of it as giving a single computer model four different "hats" to wear at once. Each hat represents a slightly different way of processing the same sentence. The model then combines the answers from all four hats.

The results are quite promising. By using this "four-hat" approach and a special math trick to keep the hats from becoming too similar, the researchers found they could reduce the number of made-up facts by up to 25.6% on specific tests, with an average improvement of 14.6%. This happened without needing to make the computer model bigger or train it on more data. In fact, it only cost a tiny bit more computing power—about 0.008% more training time and just 1.1 times the speed to run.

However, the paper also discovered a twist: more diversity isn't always better. It's like a choir; if everyone sings a different note, it sounds like noise. If everyone sings the exact same note, it's boring and prone to mistakes. There is a "sweet spot" where the voices are different enough to catch errors but similar enough to agree on the truth. The researchers found that this sweet spot changes depending on what the computer is doing. For tasks involving creative storytelling or checking facts, having more "different" voices (specifically, 4 to 8 parallel streams) worked best. But for simple memory tasks, like recalling a specific name, having just one voice was actually the most reliable.

The authors are careful to say that while their math proves this should work and their experiments on a small model show it does work, this is a specific solution for a specific type of error. They argue that hallucinations are often caused by the model's internal parts getting stuck in a loop of similar, wrong ideas, rather than just a lack of knowledge. By breaking that loop with diversity, they can make the model more reliable without needing to build a giant, expensive supercomputer. It's a clever, low-cost way to make AI a little less prone to lying, proving that sometimes, the best way to get a better answer is to ask a few different questions at once.

Technical Summary: Neural Diversity Regularizes Hallucinations in Language Models

Problem Statement

Despite scaling parameters, compute, and data, language models (LMs) continue to suffer from persistent hallucinations. This reliability crisis is particularly acute for small language models (SLMs), which are increasingly favored for edge and agentic applications but possess compressed representations that make them vulnerable to generating factually incorrect or inconsistent content.

Current mitigation strategies largely focus on optimizing average performance (first-moment improvements) rather than controlling tail risk. Techniques such as Reinforcement Learning from Human Feedback (RLHF), Retrieval-Augmented Generation (RAG), and contrastive decoding aim to improve mean harmlessness or factual grounding. However, these methods lack formal guarantees regarding the probability of catastrophic failures (hallucinations). Furthermore, existing parallel scaling methods often suffer from "representational collapse," where parallel streams converge to similar representations, failing to realize reliability gains. The paper posits that hallucinations arising from correlated representational errors, rather than missing knowledge, constitute a second-moment reliability problem amenable to portfolio-theoretic diversification.

Methodology

Theoretical Framework: Portfolio Theory for Hallucinations

The authors reframe hallucination probability as a second-moment reliability problem governed by representational covariance. Drawing on Markowitz's portfolio theory (1952), they propose that decorrelated parallel representations ("neural diversity") can reduce the tail bound of hallucination probability.

Signal-Noise Model: The model assumes $P$ parallel computational streams processing the same input. Each stream produces a hidden representation $Z_i = z^* + \epsilon_i$ , where $\epsilon_i$ is noise.
Neural Diversity Index ( $D$ ): A cosine-based index measures cross-stream diversity. $D=0$ implies perfect orthogonality, while $D=1$ implies complete collapse.
Theoretical Bounds:
- Theorem 1: Proves that hallucination probability $P(H_\delta)$ is bounded by a function of neural diversity $D$ and the number of streams $P$ . Specifically, with perfect decorrelation, $P(H_\delta)$ scales as $O(1/P)$ .
- Theorem 2: Demonstrates non-monotonic scaling behavior. While increasing $P$ initially reduces hallucinations, excessive parallelism can lead to increased correlation (due to optimizer constraints), causing reliability to degrade. This results in a U-shaped curve where an optimal $P^*$ exists for a given task.

ND-LoRA: Neural Diversity Low-Rank Adaptation

To operationalize this theory, the authors introduce ND-LoRA, a parameter-efficient architecture combining parallel LoRA adapters with explicit decorrelation regularization.

Architecture: The model employs $P$ $P$ parallel streams. Each stream utilizes:
- Stream-specific LoRA Adapters: Applied at each layer to perturb the representation.
- Learnable Prefix Tokens: 48 tokens prepended to the input sequence.
- Multi-Stream Aggregator: A learned mechanism to combine stream outputs with dynamic weights, preventing attention collapse.
Regularization: The training objective incorporates Barlow Twins regularization ( $L_{BT}$ ) applied to pairs of streams at a designated design layer. This forces the cross-correlation matrix of whitened features between streams to approach the identity matrix, explicitly minimizing representational redundancy.
Cost Efficiency: The backbone remains frozen; only the parallel adapters and aggregator are trained. This incurs a negligible pretraining cost (+0.008% of continued pretraining) and a modest inference latency increase (1.1×).

Key Contributions

Theoretical Linkage: The paper provides the first formal framework for hallucination probability tail bounds in ensembled LMs. It proves that neural diversity reduces these bounds and establishes the existence of an optimal parallelism level ( $P^*$ ) that minimizes hallucinations, explaining 94.3% of reliability variation across configurations in their setting.
Constructive Demonstration (ND-LoRA): The authors demonstrate empirical feasibility by reducing hallucinations by up to 25.6% (14.6% on average) on a small language model (Qwen2.5-0.5B) without degrading general capabilities. This is achieved at near-zero additional compute cost.
Mechanistic Analysis: Through ablations and causal interventions, the paper establishes that neural diversity is the mediating factor.
- Causality: Artificially corrupting diversity leads to statistically significant accuracy drops ( $p < 0.001$ ).
- Synergy: LoRA adapters and Barlow Twins regularization act synergistically; neither alone suffices for maximal gains.
- Task Dependence: Different tasks require different optimal diversity levels. Hallucination-focused tasks benefit from higher diversity, while knowledge retrieval tasks often peak at $P=1$ .

Experimental Results

The evaluation was conducted on Qwen2.5-0.5B trained on 20M tokens from The Pile, evaluated across 12 tasks.

Hallucination Reduction: ND-LoRA achieved significant improvements on hallucination benchmarks compared to parameter-matched baselines:
- HaluEval (Summarization): +25.6% relative improvement ( $P^*=4$ ).
- HaluEval (QA): +23.4% relative improvement ( $P^*=4$ ).
- TruthfulQA (MC2): +9.5% relative improvement ( $P^*=2$ ).
- MemoTrap v2: +8.8% relative improvement ( $P^*=8$ ).
Preservation of Capabilities: General capabilities (measured by NQ, Winogrande, Wikitext BPB) were preserved or slightly improved, indicating no "knowledge tax."
Comparison to Baselines: ND-LoRA outperformed inference-time interventions (Context-Aware Decoding, Activation Decoding) and training-time disagreement regularization, achieving a +14.6% average hallucination reduction compared to +4.1% for the next-best method.
Theoretical Fit: The empirical relationship between neural diversity and reliability matched the theoretical prediction with an $R^2$ of 0.943.

Significance and Claims

The paper claims to offer a paradigm shift in addressing LM reliability by reframing hallucinations as a second-moment problem (variance and correlation) rather than a first-moment problem (mean accuracy).

Orthogonal to Scaling: Neural diversity enables reliability gains without massive scaling of parameters or data. It controls tail probability through variance and correlation structure rather than capacity.
Cost-Effectiveness: The approach achieves substantial reliability improvements with near-zero pretraining overhead and only a 1.1× inference latency penalty, making it suitable for latency-tolerant, safety-critical deployments.
Task-Specific Optimality: The work highlights that reliability is not a monolithic property; different tasks require different degrees of neural diversity. This suggests a move away from "one-size-fits-all" scaling laws toward principled architectural design based on task characteristics.

The authors conclude that while their demonstration is on a small model, the theoretical framework and low overhead suggest that neural diversity is a scalable, practical mechanism for improving the reliability of language models in high-stakes domains.