Technical Summary: Mechanistic Analysis of Alignment Algorithms in Language Models

Problem Statement

Post-training alignment algorithms, designed to steer Large Language Models (LLMs) toward helpfulness, harmlessness, and honesty, are predominantly evaluated as "black boxes." Success is measured via aggregate behavioral metrics (e.g., benchmark scores, human preference ratings) rather than an understanding of how these algorithms reshape the model's internal computations. This diagnostic gap prevents the rigorous prediction of unintended side effects, such as the degradation of specific capabilities or the emergence of deceptive behaviors. While the field has diversified from the initial Proximal Policy Optimization (PPO) paradigm to include direct preference objectives like Direct Preference Optimization (DPO), Simple Preference Optimization (SimPO), Odds Ratio Preference Optimization (ORPO), Kahneman-Tversky Optimization (KTO), and Group Relative Policy Optimization (GRPO), there is a lack of comparative mechanistic insight into whether these diverse objectives converge on similar internal representations or induce distinct feature-level changes.

Methodology

The authors present a systematic mechanistic analysis across six preference-optimization methods (PPO, DPO, SimPO, ORPO, GRPO, KTO) applied to three open-weight model families: Llama-3.2-3B, SmolLM3-3B, and Qwen3-4B. The models were aligned using Low-Rank Adaptation (LoRA) on the UltraFeedback dataset. The analysis employs a three-pronged interpretability framework:

1. Linear Probing: To localize preference representations, the authors train logistic regression classifiers on the residual stream activations of chosen versus rejected completions. This measures the linear separability of preference signals across different layers, identifying the "best" layers where preference information is most decodable.
2. Sparse Autoencoders (SAEs): To decompose the superposed, polysemantic activations into interpretable, monosemantic features, Batch Top-K SAEs were trained on the base models. These SAEs were then used to measure the activation of specific "anchor" features in the aligned models, allowing for the quantification of feature suppression, amplification, or variance changes.
3. Crosscoders: To compare the geometry of latent spaces between base and aligned models, crosscoders were trained on paired hidden states. This allows for the classification of features as shared, base-specific, aligned-specific, attenuated, or redirected. Geometric statistics (norm ratios and angles) were used to determine if alignment induced constructive sharing or non-constructive rotation/attenuation.

Key Contributions

The paper establishes that alignment is a heterogeneous intervention rather than a uniform behavioral fix. The primary contributions are:

• Comprehensive Alignment Diagnostics: The study provides the first white-box comparison of six leading alignment methods across three distinct architectures, moving beyond output-level benchmarking to internal representation analysis.
• Distinct Alignment Signatures: The authors identify qualitatively different geometric transformations induced by different objectives:
  • KTO and GRPO: These methods enhance the linear separability of preference representations. KTO achieves this through constructive feature sharing and the recruitment of sparse, high-salience features. GRPO amplifies gradients based on intra-group variance, leading to strong discriminative power.
  • DPO and ORPO: These methods degrade linear separability relative to the base model. DPO is associated with non-constructive geometric rotation of shared features, shifting representations into subspaces orthogonal to the original preference-encoding directions. ORPO attenuates preference-relevant activations, significantly dampening specific feature signals (e.g., a 400% reduction in median activation for certain features in Qwen3).
  • PPO and SimPO: These methods largely preserve the baseline geometry, maintaining the linear decodability of preferences without the constructive enhancement seen in KTO/GRPO or the degradation seen in DPO/ORPO.
• Architecture-Dependent Variability: The internal effects of a specific alignment objective are not uniform across model families. For instance, while ORPO causes catastrophic performance collapse and feature suppression in Llama-3.2 and Qwen3, it maintains activation levels comparable to the base model in SmolLM3. This demonstrates that claims about alignment mechanisms must be scoped to the specific base architecture and layer being analyzed.

Results

• Linear Probing: The "camel-hump" pattern of preference discriminability (concentrated in early-mid or mid-late layers) persists across methods. However, KTO and GRPO show the highest discriminative power, while DPO and ORPO show reduced separability compared to the base model.
• SAE Analysis: ORPO induces marked suppression of anchor features in Llama and Qwen models, whereas KTO introduces higher variance in feature activations, suggesting greater context dependence. SmolLM3 exhibits different heterogeneity, where GRPO shows reduced activation only at the probe-best layer but elevated activation at the training layer.
• Crosscoder Geometry: KTO's feature sharing is "constructive," borrowing existing discriminability and improving it. In contrast, DPO and ORPO exhibit "non-constructive" distortion; DPO rotates features away from the preference subspace, and ORPO attenuates shared features. PPO and SimPO show feature distributions that reify the preservation of baseline geometry.
• Behavioral Correlation: The MT-Bench scores reveal that surface-level performance does not always correlate with internal health. For example, ORPO caused a drastic drop in Llama-3.2 scores (6.33 to 2.12), which was diagnosed as internal representation drift and feature attenuation rather than simple output noise.

Significance

The paper argues that behavioral alignment does not imply uniform internal restructuring. By characterizing alignment as a set of distinct representational transformations, the authors motivate the need for standardized feature-level auditing for safety and interpretability. The findings suggest that mechanism-aware optimization objectives are necessary to ensure that alignment protocols not only improve output behavior but also maintain or enhance the geometric fidelity of internal representations. This work moves the field toward a "white-box" understanding of alignment, providing a diagnostic framework to detect whether degradation stems from representation drift, over-optimization, or the loss of instruction-following circuits.

Limitations

The authors acknowledge that their experiments are constrained to 3B–4B parameter models, and representational changes may scale non-linearly in larger production models. Furthermore, the analysis is diagnostic rather than causal; while it identifies decodable information and feature geometry changes, it does not perform component-level causal ablations (e.g., activation patching) to establish necessary or sufficient circuits for the observed behaviors.