💬 NLP

How Independent are Large Language Models? A Statistical Framework for Auditing Behavioral Entanglement and Reweighting Verifier Ensembles

This paper introduces a statistical framework to audit and quantify behavioral entanglement among large language models using information-theoretic metrics, demonstrating that shared failure modes undermine independent verification systems and proposing a reweighting strategy that significantly improves ensemble accuracy by mitigating correlated biases.

Original authors: Chenchen Kuai, Jiwan Jiang, Zihao Zhu, Hao Wang, Keshu Wu, Zihao Li, Yunlong Zhang, Chenxi Liu, Zhengzhong Tu, Zhiwen Fan, Yang Zhou

Published 2026-04-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenchen Kuai, Jiwan Jiang, Zihao Zhu, Hao Wang, Keshu Wu, Zihao Li, Yunlong Zhang, Chenxi Liu, Zhengzhong Tu, Zhiwen Fan, Yang Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Are These AI Models Actually Independent?

Imagine you are a teacher grading a stack of essays. To make sure your grading is fair, you ask three other teachers to review the same essays. If all three say, "This essay is perfect," you feel confident that it really is perfect. You assume that because three different people looked at it, their agreement is strong proof of quality.

But what if those three teachers all went to the same school, read the same textbooks, and were taught by the same strict principal?

If they all agree, it might not be because they independently verified the essay. It might be because they all learned the same mistakes to make, or they all have the same blind spots. They aren't three independent voices; they are three echoes of the same voice.

This is the problem this paper tackles with Large Language Models (LLMs). We assume that when we use different AI models (like GPT-4, Llama, or Claude) to check each other's work, they are independent. But the authors suspect they are secretly "entangled"—like twins who finish each other's sentences because they grew up in the same house, not because they are thinking the same thoughts independently.

The Problem: The "Echo Chamber" Effect

AI models are trained on massive amounts of data. Often, they share:

The same training books: They read the same internet data.
The same teachers: One model might be "distilled" (taught) by another, like a student copying a teacher's notes.
The same alignment: They are all tuned to behave politely and safely in similar ways.

Because of this, when two different AI models make a mistake, they often make the exact same mistake in the exact same way.

In the world of AI safety, we use a "Judge" model to check if an "Answer" model is right. If the Judge and the Answer model are secretly "entangled" (like cousins), the Judge might say, "Yes, this answer is great!" not because it's true, but because it sounds familiar to the Judge's own training. This leads to over-confidence in wrong answers.

The Solution: A New "Lie Detector" for AI

The authors built a statistical framework to catch this hidden connection. Instead of just asking, "Do they agree?" (which is easy to fake), they ask, "Do they fail together in weird ways?"

They use two main tools, which we can think of as a Multi-Resolution Audit:

1. The "Easy Task" Trap (Difficulty-Weighted Entanglement)

Imagine a test with very easy questions, like "What color is the sky?"

If two students get this wrong, it's a huge red flag. It means they aren't just guessing; they share a fundamental misunderstanding.
The authors' first tool looks at easy tasks. If two AI models both fail an easy question, it's strong evidence they are "entangled." If they fail a hard question, maybe it was just too hard. But failing an easy one together? That's suspicious.

2. The "Wrong Turn" Detector (Cumulative Information Gain)

Imagine two drivers taking a wrong turn.

Scenario A: Driver A turns left into a ditch. Driver B turns right into a tree. They both crashed, but for different reasons. This is good! It means they are independent.
Scenario B: Driver A turns left into a ditch. Driver B also turns left into the same ditch.
The authors' second tool looks at direction. If two models get a multiple-choice question wrong, did they pick the same wrong answer? If they both picked "Option C" when the answer was "Option A," it suggests they are following the same faulty logic path. This is a deep sign of entanglement.

What They Found

The researchers tested 18 different AI models. Here is what they discovered:

The "Family Resemblance": Models from the same family (like different versions of Llama) are heavily entangled. They make the same mistakes.
The "Hidden Cousins": Even models from different companies (like GPT and Claude) showed signs of entanglement. They likely share training data or similar "alignment" techniques, making them think alike.
The Danger: When a Judge model is entangled with the model it is judging, it becomes a "yes-man." It agrees with the answer model even when the answer is wrong. The more entangled they are, the more the Judge lies (or rather, confidently endorses errors).

The Fix: Reweighting the Jury

So, how do we fix this? The authors propose a new way to combine AI opinions, called De-entangled Verifier Reweighting.

Imagine you are a jury foreman. You have 10 jurors.

Old Way: You ask everyone to vote. If 6 say "Guilty," you convict.
The Problem: You realize that Jurors 1, 2, and 3 are all brothers who live together. Their vote isn't worth 3x; it's worth 1x because they are just echoing each other.
The New Way: You look at who is related to whom. You give less weight to the brothers and more weight to the jurors who seem independent.

The authors did this with AI. They calculated how "entangled" each AI was with the others.

If an AI is highly entangled with the others, they lower its weight (trust it less).
If an AI is independent, they raise its weight.

The Result: By doing this, they improved the accuracy of the AI verification system by 4.5% compared to the standard "majority vote." They stopped the "echo chamber" from drowning out the truth.

The Takeaway

We can no longer assume that because we have many different AI models, we have many independent opinions. They are often secretly linked, sharing the same blind spots.

To trust AI, we need to stop looking at how much they agree and start looking at how they fail together. If they fail together on easy tasks or pick the same wrong answers, they aren't independent. By detecting these hidden links and adjusting our trust accordingly, we can build safer, more reliable AI systems.

1. Problem Statement

The rapid proliferation of Large Language Models (LLMs) has led to a reliance on multi-model systems (e.g., LLM-as-a-judge pipelines, ensemble verifiers) that implicitly assume models provide independent signals. However, shared pretraining data, knowledge distillation, and common alignment pipelines create latent behavioral entanglement.

The Core Issue: Apparent agreement between models often reflects correlated errors (shared blind spots) rather than independent verification.
Limitations of Current Methods: Existing diagnostics (e.g., output overlap, embedding similarity, Pearson correlation) are insufficient because they cannot distinguish between independent reasoning converging on the same answer and models failing in the same way due to shared lineage. They also fail to account for task difficulty, where easy tasks naturally induce agreement even among independent models.

2. Methodology: A Multi-Resolution Statistical Framework

The authors propose a hierarchical statistical framework to audit behavioral entanglement in black-box LLMs by analyzing the joint failure manifold rather than just successful outputs. The framework consists of three levels:

Level 1: Difficulty-Weighted Behavioral Entanglement Index (BEI $_w$ )

Goal: Detect synchronized failures, particularly on easy tasks, where coincident errors are statistically unlikely under independence.
Mechanism:
1. Task Difficulty Quantification: Uses empirical failure rates ( $d_t$ ) to define task easiness ( $a_t = 1 - d_t$ ).
2. Conditional Independence Null: Models the probability of failure given difficulty ( $p_m(d)$ ) using logistic regression to remove first-order difficulty effects.
3. Residual Interaction: Calculates the deviation of actual failures from expected failures ( $R_{m,t}$ ).
4. Metric: Computes the weighted sum of pairwise residual products, emphasizing tasks with high easiness ( $a_t$ ).
5. Significance: Uses a sign-flip randomization test to determine if observed co-failures exceed the null expectation.

Level 2: Cumulative Information Gain (CIG)

Goal: Capture the directionality of errors (e.g., in Multiple-Choice Questions, do models choose the same wrong distractor?).
Mechanism:
1. Directional Collision: Identifies tasks where two models fail and select the same incorrect option.
2. Null Baseline: Calculates the expected probability of collision based on the empirical attractiveness of distractors ( $p_{k,t}$ ).
3. Surprisal Weighting: Weights tasks by $-\log(c^{null}_t)$ , giving higher importance to unlikely coincident errors.
4. Metric: Aggregates the deviation of observed directional agreement from the null expectation. Positive CIG indicates systematic alignment in incorrect reasoning paths.

Level 3: De-entangled Verifier Reweighting

Goal: Mitigate bias in ensemble verification by adjusting model weights based on inferred independence.
Mechanism:
- Defines a pairwise entanglement score $E(i, j)$ combining BEI and CIG.
- Calculates Internal Dependence ( $\Delta^{in}$ ): Redundancy of a verifier relative to the ensemble.
- Calculates Target Dependence ( $\Delta^{tar}$ ): Alignment between a verifier and the target model being evaluated.
- Reweighting Formula: Assigns weights to verifiers based on their competence score ( $q_m$ ) penalized by their entanglement scores ( $\eta_1 \Delta^{in} + \eta_2 \Delta^{tar}$ ). This downweights verifiers that are highly correlated with the target or other verifiers.

3. Key Contributions

Statistical Framework: Introduced a rigorous, multi-resolution hierarchy (BEI $_w$ and CIG) to audit behavioral entanglement, moving beyond surface-level output similarity to structural error analysis.
Empirical Discovery: Conducted extensive experiments on 18 LLMs across 6 families (including GPT, Llama, Qwen, Claude, Gemini, DeepSeek).
- Found widespread intra-family and cross-family entanglement.
- Identified cross-generation entanglement (e.g., between frontier models like GPT-5 and Claude 4.6), suggesting shared signals extend beyond direct lineage.
Bias Quantification: Demonstrated a statistically significant association between entanglement metrics and judge precision degradation.
- CIG showed strong correlation with over-endorsement bias (Spearman $\rho = 0.64$ for GPT-4o-mini, $\rho = 0.71$ for Llama3-based judges, $p < 0.01$ ).
Practical Mitigation: Proposed and validated a De-entangled Verifier Ensemble method that improves verification accuracy by penalizing dependent models.

4. Experimental Results

Entanglement Detection: The framework successfully identified hidden dependencies that standard Pearson correlation missed. BEI highlighted shared blind spots on easy tasks, while CIG revealed specific alignment in erroneous reasoning paths.
Impact on Evaluation:
- Standard Pearson correlation failed to predict evaluation bias.
- Entanglement-aware metrics (BEI/CIG) strongly predicted over-endorsement bias (Type I errors), where judges incorrectly validate answers from entangled models.
Ensemble Performance:
- In LLM-as-a-judge verification tasks, the proposed Entanglement-based Reweighting achieved 0.896 Accuracy and 0.906 Precision.
- This represents a 4.5% accuracy gain over standard Majority Voting (0.847) and outperformed accuracy-based reweighting (0.881).
- The method adaptively reduced the weight of verifiers with high target dependence (e.g., down-weighting Gemini 3.1 Pro when evaluated by a GPT-5 judge if they were entangled).

5. Significance and Conclusion

This paper fundamentally challenges the assumption of independence in multi-LLM systems. It provides:

A Diagnostic Tool: A statistical method to "audit" the LLM ecosystem for hidden dependencies caused by distillation and shared data.
A Warning: Apparent consensus among LLMs is not evidence of correctness; it may be a consensus of correlated errors.
A Solution: A practical algorithm to reweight verifier ensembles, significantly improving the reliability of automated evaluation and safety mechanisms.

The work suggests that future multi-model systems must explicitly account for dependency structures to avoid the "echo chamber" effect where entangled models reinforce each other's errors.