🤖 machine learning

Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experiment

This paper demonstrates that sustained, weakly positive gradient alignment mediates subliminal learning in multi-step MNIST auxiliary logit distillation, revealing that mitigation strategies like liminal training may fail to suppress unintended trait acquisition when first-order gradient drives dominate.

Original authors: Chayanon Kitkana, Shivam Arora

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chayanon Kitkana, Shivam Arora

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a young apprentice (the Student) to mimic the cooking style of a master chef (the Teacher). Usually, you want the apprentice to learn only the specific recipes you give them. But in this paper, the researchers discovered a sneaky problem: even when you tell the apprentice to ignore the main recipes and only practice on "dummy" ingredients (like empty bowls), the apprentice still accidentally learns the master chef's secret, unwanted habits.

In the world of AI, this is called Subliminal Learning. The apprentice picks up a "trait" (like a specific way of holding a knife) that the master has, even though you never explicitly taught that trait.

Here is what the paper found, broken down into simple stories and analogies:

1. The Invisible Push (Gradient Alignment)

Think of the learning process as a hiker trying to walk up a hill.

The Goal: The hiker wants to reach the top of "Distillation Hill" (learning the dummy recipes).
The Secret: There is a gentle, invisible wind blowing them toward "Trait Valley" (learning the unwanted habit).

The researchers found that even though the wind is very weak, it blows in the same direction as the hiker's steps almost the entire time. It's like walking on a treadmill that is slightly tilted toward a trap. Because the wind pushes in the same direction as the steps, the hiker slowly drifts into the trap, step by step, over the whole journey.

2. The "Stop the Drift" Experiment

To prove that this invisible wind was actually causing the drift, the researchers tried a new trick. They put up a wall that blocked only the wind pushing toward the trap, while letting the hiker keep walking toward the goal.

The Result: The hiker reached the top of the goal hill perfectly, but they never fell into the trap.
The Lesson: This proved that the "wind" (the alignment of the learning steps) was the only reason the apprentice learned the bad habit. If you block that specific push, the bad habit disappears, even if the rest of the training looks exactly the same.

3. The "Soft Brake" That Didn't Work

There was a popular method called Liminal Training (think of it as a "Soft Brake" or a "Safety Net"). The idea was to gently nudge the apprentice back toward their original self whenever they started to drift too far, hoping to stop the bad habit.

What happened: The Soft Brake did slow down the drift at the very beginning. It made the invisible wind feel weaker for a short while.
The Catch: The brake didn't actually stop the wind; it just made it softer. Once the brake was released (as the training finished), the apprentice drifted into the trap anyway.
The Lesson: A gentle nudge or a temporary slowdown isn't enough. If the wind is pushing you in the wrong direction, you need to completely block that specific direction, not just slow down.

The Big Takeaway

The paper concludes that when an AI learns, it's like a boat being pushed by a current.

If the current (the learning steps) is even slightly pointing toward a dangerous reef (the unwanted trait), the boat will eventually hit it.
Trying to just "slow down" or "steer gently" away from the reef doesn't work if the current keeps pushing you there.
To truly stop the AI from learning the bad habit, you have to physically remove the part of the push that is pointing toward the reef.

In short: You can't just hope a gentle correction will fix the problem. If the learning process itself is secretly pushing the AI toward a bad behavior, you have to cut that specific push out entirely to stop it.

1. Problem Statement

The paper addresses the phenomenon of subliminal learning in Knowledge Distillation (KD).

Context: In KD, a student model is trained to mimic a teacher model. Recent work (Cloud et al., 2025) demonstrated that if a student is initialized close to a teacher, it can unintentionally acquire "traits" (e.g., misaligned behaviors or specific classification capabilities) even when the distillation process explicitly targets only "no-class" logits (auxiliary outputs) and the training data contains no examples of the trait.
The Gap: Existing theory explains this via gradient alignment between the distillation objective and the trait objective, but this theory relies on a single-step gradient descent assumption. It remains unclear whether this alignment persists and causally drives trait acquisition in a realistic multi-step training setting where parameters evolve over time.
Mitigation Challenge: A recent mitigation strategy, Liminal Training (Yanagisawa et al., 2025), claims to suppress trait acquisition in Large Language Models (LLMs) by using an annealed KL regularizer. It is unknown if this method works in settings where first-order gradient effects dominate, or if it merely delays the issue.

2. Methodology

The authors conducted empirical experiments using the MNIST MLP Auxiliary Logit Distillation setup (reproducing Cloud et al., 2025).

Experimental Setup:
- Model: A Multi-Layer Perceptron (MLP) with 10 class logits and 3 "no-class" (auxiliary) logits.
- Initialization: The teacher and student share an identical initialization.
- Teacher: Trained on MNIST using only the 10 class logits (Cross-Entropy).
- Student: Trained to minimize KL divergence on the 3 no-class logits only.
- Goal: Observe if the student acquires the ability to classify digits (the "trait") despite never seeing class labels during training.
Investigations:
1. Gradient Alignment Monitoring: The authors tracked the inner product (alignment) between the trait gradient ( $\nabla_\theta L_{trait}$ ) and the distillation gradient ( $\nabla_\theta L_{distill}$ ) across 100 training seeds.
2. Causal Intervention (Gradient Projection): To test causality, they modified the distillation update rule. They projected the distillation gradient onto the normal plane of the trait gradient whenever the two were positively aligned, effectively removing the trait-aligned component of the update:
  $\tilde{g}_{distill} = \begin{cases} g_{distill} - \frac{\langle g_{distill}, g_{trait} \rangle}{\|g_{trait}\|^2} g_{trait} & \text{if } \langle g_{distill}, g_{trait} \rangle > 0 \\ g_{distill} & \text{otherwise} \end{cases}$
3. Mitigation Evaluation (Liminal Training): They applied Liminal Training, which adds a KL regularizer to keep the student's auxiliary logits close to the base model's distribution, with the regularization strength annealing to zero over time.

3. Key Contributions

Empirical Validation of Sustained Alignment: Demonstrated that positive gradient alignment between the trait and distillation objectives persists throughout multi-step training, albeit weakly.
Causal Proof: Proved that this alignment is the primary driver of subliminal learning by showing that removing the trait-aligned component of the gradient completely stops trait acquisition without hindering the distillation process.
Limitation of Current Mitigations: Showed that Liminal Training fails to suppress trait acquisition in this regime, suggesting that merely attenuating early alignment is insufficient when first-order effects dominate.

4. Results

A. Gradient Alignment Persists

Observation: Across 100 seeds, the alignment was positive in 78.1% of training steps.
Magnitude: The average cosine similarity was low ( $0.0075 \pm 0.0017$ ), indicating gradients are near-orthogonal but with a slight, consistent positive bias.
Correlation: Trait acquisition (measured by digit classification accuracy) was fastest during the early training phase where alignment was highest.
Performance: The student achieved an average final test accuracy of 55.28% (significantly above the 10% chance level), confirming subliminal learning occurred.

B. Causal Role of Alignment (Gradient Projection)

Intervention: When the trait-aligned component was removed from the distillation gradient:
- Trait Acquisition: Collapsed to near-chance levels (10.14% accuracy vs. 55.28% in control).
- Distillation Quality: Remained unaffected. The KL divergence (distillation loss) curves were visually identical to the control group.
- Conclusion: This confirms that the slight positive alignment is the causal mechanism for trait acquisition and that first-order effects dominate in this setup.

C. Failure of Liminal Training

Mechanism: Liminal training successfully reduced gradient alignment during the first epoch.
Outcome: Despite reduced early alignment, it failed to suppress trait acquisition.
- Final accuracy with Liminal Training: 48.97% (compared to 55.28% control).
- The trait acquisition was merely delayed. As the KL regularization strength decayed to zero, the alignment and trait acquisition resumed, leading to a final state similar to the unmitigated control.

5. Significance and Implications

Mechanism Clarification: The paper establishes that subliminal learning in multi-step settings is driven by a sustained, weak positive alignment of gradients, rather than a one-time initialization effect.
Critique of Mitigation Strategies: The results challenge the efficacy of "soft" mitigation strategies like Liminal Training in regimes dominated by first-order gradient effects. Simply slowing down or attenuating alignment early in training is insufficient; the trait-aligned component must be explicitly removed to prevent transmission.
Safety Implications: If a student model inherits misalignment from a teacher via this mechanism, standard safety filters on the distillation dataset may be ineffective. The paper suggests that robust mitigation requires active intervention in the gradient space (e.g., projection) rather than just regularization.
Future Research: The authors note that while gradient projection is effective, it requires knowledge of the "ground truth" trait gradient, which is often unavailable in real-world scenarios. Developing methods to identify and remove these components without explicit ground truth remains a critical open challenge.