Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models

Technical Summary: Value Entanglement in Large Language Models

Problem Statement
The alignment of Large Language Models (LLMs) with human values requires an empirical understanding of how these models represent different kinds of "good." Human valuation is characterized by the ability to distinguish between distinct categories of value, such as moral goodness (e.g., a virtuous deed), grammatical goodness (e.g., a well-formed sentence), and economic goodness (e.g., an expensive object). The authors investigate whether LLMs similarly distinguish between these orthogonal value dimensions or if they suffer from "value entanglement"—a conflation where the representation of one value type is overly influenced by another. This confusion poses a risk to value alignment, as an agent unable to distinguish between a "good sentence" and a "good deed" may fail to act reliably according to specific moral objectives.

Methodology
The study employs a multi-faceted approach combining behavioral evaluation, internal activation geometry analysis, and causal interventions across a diverse set of closed and open-weight models (including Qwen, Gemma, Mistral, GPT, and Claude families).

1. Stimulus Design: The researchers constructed two orthogonal stimulus sets:

   • MoralGrammar68: 68 sentences varying independently in moral valence (from virtuous to harmful) and grammaticality (from 0 to 4+ errors).
   • MoralEconomic68: 68 sentences varying independently in moral valence and economic value (by embedding objects of varying retail costs, from low to high).
     Human participants validated these stimuli, confirming that moral and grammatical ratings, as well as moral and economic ratings, were uncorrelated (orthogonal) in human judgment.

2. Behavioral Evaluation: Models were prompted to rate the sentences on Likert scales (-10 to +10) for morality, grammaticality, and economic value. The study measured the correlation between model ratings on these distinct dimensions.

3. Activation Geometry: For open-weight models, the authors identified "attribute vectors" in the residual stream using a means-difference approach (contrasting activations for positive vs. negative examples of each value type). They projected the target sentences onto these vectors to measure the internal representation of each value dimension.

4. Directional Ablation: To establish causality, the authors performed directional ablation during inference. By "zeroing out" or flipping the variance along specific attribute vectors (e.g., the morality vector), they tested whether removing moral information altered the model's behavior regarding grammatical or economic judgments.

Key Results

• Behavioral Entanglement: Many models, particularly open-source variants (e.g., Qwen2.5, Gemma-2/3, Mistral-Small), exhibited significant correlations between moral and grammatical ratings, and between moral and economic ratings. For instance, in Qwen2.5 7B, a sentence with a morally wrong content but perfect grammar received a lower grammaticality rating than a neutral sentence with errors. This "inflated correlation" was largely absent in human raters and some closed-weight models (e.g., GPT-5.4, Claude Sonnet).
• Asymmetry: The entanglement was asymmetric. While moral judgments remained faithful to human norms (high correlation with human moral ratings, low correlation with grammar/economics), grammatical and economic judgments were heavily contaminated by moral content. Models often rated grammatically perfect sentences as "ungrammatical" if they described immoral acts.
• Internal Representations: Activation projections confirmed that the internal vectors for grammaticality and economic value were highly correlated with the morality vector in the residual stream. In many models, the representation of grammatical or economic value was significantly predicted by the moral level of the input, mirroring the behavioral findings.
• Causal Intervention: Directional ablation of the morality vector resulted in a recovery of performance. When moral information was removed, models' ability to judge grammaticality and economic value improved, showing higher correlation with ground truth human ratings. This suggests that moral information actively interferes with the processing of other value dimensions.
• Training Origins: Entanglement was observed in both pre-trained (base) and instruction-tuned models, suggesting that the phenomenon arises during pre-training, likely due to the polysemous nature of valenced language (e.g., the word "good" appearing in diverse contexts), rather than being solely a product of Reinforcement Learning from Human Feedback (RLHF).

Significance and Claims
The paper claims to provide empirical evidence that many LLMs suffer from value entanglement, confusing distinct types of "good." The authors argue that this representational conflation is problematic for value alignment because it distorts valuation relative to human norms. If a model cannot distinguish between a grammatically correct sentence and a morally virtuous action, its decision-making in tasks requiring multi-dimensional value assessment may be fundamentally flawed.

The study contributes to AI safety by:

1. Offering a method to empirically measure and detect value entanglement.
2. Demonstrating that targeted interventions (ablation) can repair these conflations, suggesting a potential pathway for mitigating such alignment issues.
3. Highlighting that the root of the problem may lie in the pre-training phase, implying that future alignment strategies must address the foundational representations of value, not just post-training fine-tuning.

The authors maintain a modest scope, noting that while entanglement is pervasive in many tested models, it is not universal (some closed models showed less entanglement), and the findings are specific to the value dimensions tested (moral, grammatical, economic). They do not claim to have solved the alignment problem but rather to have identified a specific, measurable failure mode in how models represent value.