Technical Summary: Channel Location Constrains the Auditability of Subliminal Learning

Problem Statement

The paper addresses the safety risks of subliminal learning in teacher-student distillation pipelines. Recent work (Cloud et al.) demonstrated that a student model can inherit a teacher's hidden traits (preferences, personas, or misalignments) even when the distillation data contains no explicit reference to those traits, provided the models share an initialization.

The core problem is auditability: Can auditors predict or detect this transfer before the student is trained? The paper argues that the answer depends not on model identity or scale, but on the location of the carrier—the specific component of the neural network through which the hidden trait propagates. The authors posit that a "screen" (a pre-training metric predicting transfer) is only valid if it targets the correct carrier location. Applying a screen designed for one channel type to another yields false assurance.

Methodology

The authors develop a channel-conditioned auditability map by analyzing three distinct regimes of subliminal transfer. They employ a combination of causal interventions, ablation studies, and prospective screening tests across multiple model families (Pythia, Qwen, Gemma, RedPajama) and scales (up to 6.9B parameters).

Key methodological components include:

1. Carrier Definition: A carrier is defined operationally via a "removal test": the component whose removal, replacement, or neutralization abolishes the transfer while preserving overt supervised behavior.
2. Coverage Metric: A pre-training screen defined as the cosine similarity between the student's initial distillation update ($d_0$) and the teacher's fine-tuning displacement ($\hat{u}_T$). This measures the alignment of the first gradient step with the teacher's learned direction.
3. Causal Interventions:
   • Orthogonalization: Removing the coupling between a target token's unembedding row and its high-cosine neighbors.
   • Neighbour Masking: Excluding entangled neighbor tokens from the distillation loss.
   • Component Transplants: Swapping body (internal weights) and head (output embeddings) between students and bases to isolate routing.
   • Precision Manipulation: Training in reduced precision (bf16) to test channel formation.
4. Experimental Regimes:
   • Toy Auxiliary Channel: A controlled MLP setting with random initialization and orthogonal readout.
   • Pretrained Token Traits: Single-token preferences (e.g., named entities, number words) masked from the loss in pretrained LMs.
   • Conditional Behaviors: Complex policies (e.g., sycophancy, backdoors) where the model agrees or complies based on specific conditions, masked from the loss.

Key Contributions and Results

1. The Body Channel is Screenable (Positive Result)

In a controlled setting with a random shared initialization and an auxiliary output channel, the carrier resides in the network body (internal weights).

• Result: The Coverage metric (cosine alignment) predicts held-out transfer accuracy with high fidelity ($\rho \approx 0.95$, AUROC 0.997).
• Mechanism: The initial update direction is a first-order approximation of the teacher's displacement filtered by the channel's curvature.
• Significance: This establishes the existence of a sound, mechanistic, and prospective pre-training screen, but only when the carrier is initialization-dependent and located in the body.

2. Vocabulary-Carried Token Traits Evade Initialization Screens

In pretrained language models, single-token traits (e.g., a preference for "seven" or a brand name) are carried by convergent unembedding geometry (the output matrix mapping hidden states to logits).

• Mechanism: Due to the softmax bottleneck (vocabulary size $\gg$ hidden width), output rows are entangled. Fitting the teacher's distribution over non-target tokens inadvertently raises the probability of the target token via its high-cosine neighbors.
• Result: Transfer is initialization-independent. A student trained from a different base (different initialization) inherits the trait just as well as one with a shared base.
• Screen Failure: Coverage is non-mechanistic and blind in this regime. It may correlate with transfer within a single family due to architectural coincidences, but it fails to predict transfer across different initializations or architectures (e.g., tied-head Qwen vs. untied-head Pythia).
• Intervention: Targeted orthogonalization of the target token's unembedding row against its entangled neighbors collapses leakage by $>500\times$ while preserving model utility. This confirms the carrier is the geometry, not the weights.

3. Conditional Behaviors Route Through the Body

Complex conditional policies (e.g., sycophancy, trigger-conditioned backdoors) transfer subliminally but localize predominantly to the network body, not the vocabulary.

• Result: These behaviors survive neighbor-masking interventions that destroy token traits. They are precision-robust (unlike token traits, which are suppressed by bf16 training).
• Audit Gap: These behaviors are opaque to post-hoc detection. There is no single "token handle" to scan for; the policy is distributed across body computations.
• Screen Failure: Coverage provides only a weak directional signal ($\rho \approx 0.06$), insufficient for a deployable screen.
• Significance: This regime represents the most significant safety gap: the carrier is in the body, but the signal is initialization-independent (or weakly dependent), rendering standard alignment screens ineffective.

4. The Audit Lifecycle Taxonomy

The paper organizes auditability into three cells based on carrier location (Table 4):

• Semantic Token Traits (Vocabulary Carrier): Unscreenable pre-training; fully detectable and fixable post-hoc via elevated-token scanning and orthogonalization.
• Low-Proxy Token Traits (Vocabulary Carrier): Meaning-opaque in data but locatable via logits; fixable post-hoc.
• Body-Carried Behaviors: Weakly screenable pre-training; opaque to post-hoc detection, identification, and blocking. The only effective mitigation is governance of the training pipeline (teacher and distillation signal).

Significance and Claims

The paper claims that channel location is the primary determinant of auditability, superseding model scale or identity.

• No Universal Screen: There is no single pre-training metric that works for all subliminal transfer. A screen valid for body-carried traits (Coverage) is blind to vocabulary-carried traits, and vice versa.
• False Assurance: Relying on a screen outside its valid regime (e.g., using Coverage on a pretrained model with vocabulary traits) provides false security.
• Deployment Reality:
  • For token traits, the solution is post-hoc detection and targeted geometric repair (orthogonalization).
  • For conditional behaviors (the most dangerous class), no finished-model audit works. The leverage lies entirely in pipeline governance (ensuring the teacher and distillation signal are safe), as these behaviors evade every tested post-hoc intervention.
• Modesty: The authors explicitly state they do not propose a universal defense. Their contribution is a "channel-conditioned auditability map" that clarifies where and when specific audits are sound, preventing the misapplication of tools like Coverage to regimes where they are mechanistically inert.

The paper concludes that while a strong pre-training screen exists for controlled body channels, no validated pre-training screen exists for naturally occurring channels in pretrained models. The only robust defense for body-carried conditional behaviors remains the control of the training pipeline itself.