Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering

1. Problem Statement

Large Language Models (LLMs) can be controlled at inference time through two distinct methodologies:

1. In-Context Learning (ICL): Modifying behavior via input prompts (e.g., few-shot examples, instructions).
2. Activation Steering: Modifying behavior by directly intervening on the model's hidden layer activations (e.g., adding a steering vector).

While these methods are often treated as separate phenomena with different theoretical underpinnings (ICL as Bayesian inference, steering as linear representation manipulation), they share the goal of controlling model output. The paper addresses the lack of a unifying theoretical framework that explains how these disparate interventions interact, how they scale, and whether they operate on a common internal mechanism. Specifically, the authors seek to explain the "sudden learning" curves observed in many-shot ICL and how steering vectors shift these dynamics.

2. Methodology

Theoretical Framework: Bayesian Belief Dynamics

The authors propose a unified Bayesian Belief Dynamics model. They posit that both ICL and activation steering operate by updating the LLM's belief in latent concepts ($c$).

• Latent Concepts: The model maintains a space of concepts (e.g., "Machiavellian persona" vs. "Neutral persona").
• ICL as Evidence Accumulation: In-context examples act as evidence ($x$) that updates the likelihood $p(x|c)$. As more examples are added, the posterior belief $p(c|x)$ increases.
• Steering as Prior Alteration: Activation steering is modeled as an intervention that shifts the prior probability $p(c)$ of a concept, effectively changing the baseline belief before any context is seen.

The Mathematical Model:
The core of the model describes the log-posterior odds of a concept $c$ versus its complement $c'$:
$$\log o(c|x) = \log \frac{p(c)}{p(c')} + \log \frac{p(x|c)}{p(x|c')} + a \cdot m$$
Where:

• $\log \frac{p(c)}{p(c')}$: The log prior odds (baseline belief).
• $\log \frac{p(x|c)}{p(x|c')}$: The log-likelihood ratio (evidence from ICL). The authors model this as a sub-linear power law ($\gamma N^{1-\alpha}$) to account for the diminishing returns of adding more context shots.
• $a \cdot m$: The effect of activation steering, where $m$ is the steering magnitude and $a$ is a scaling constant derived from the Linear Representation Hypothesis (LRH).

The final probability of the target behavior is given by the sigmoid of the log-odds:
$$p(y|x) = \sigma(\log o(c|x))$$

Experimental Setup

• Models: Llama-3.1-8B (primary), Qwen-2.5-7B, Gemma-2-9B, and Llama-3.1-70B.
• Datasets:
  • Persona Matching: "Dark Triad" traits (Psychopathy, Machiavellianism, Narcissism) and Moral Nihilism concepts.
  • High-Prior Concepts: Traits like "Openness" to test if the model holds for concepts the model already favors.
  • Flipped-Label Sentiment: A non-binary task where the model must learn a reversed mapping between sentiment labels and text.
• Interventions:
  • ICL: Varying the number of in-context shots ($N$) from 0 to 128.
  • Steering: Using Contrastive Activation Addition (CAA) to generate steering vectors. The magnitude ($m$) was varied from -10 to +10.
• Analysis: The authors fitted the Bayesian model parameters ($\alpha, \gamma, a, b$) to the empirical data using L-BFGS optimization and validated predictions via 10-fold cross-validation.

3. Key Contributions

1. Unified Theory of Control: The paper establishes that ICL and activation steering are dual mechanisms acting on the same latent belief state. ICL updates the likelihood (evidence), while steering updates the prior.
2. Sigmoidal Learning Curves: The model mathematically derives why many-shot ICL follows a sigmoidal (S-shaped) curve. This is a result of the sub-linear accumulation of evidence ($N^{1-\alpha}$) combined with the sigmoidal mapping of log-odds to probability.
3. Additivity in Log-Belief Space: The authors demonstrate that the effects of ICL and steering are additive in log-space. This allows for the prediction of "phase boundaries" where the model's behavior suddenly flips from one concept to another.
4. Predictive Phase Boundaries: The model can predict the exact crossover point ($N^*$) where a model switches behavior based on the interplay between the number of shots and the steering magnitude.

4. Key Results

• High Predictive Accuracy: The Bayesian model achieved a correlation of $r = 0.98$ (averaged across 5 domains) between predicted and actual LLM behavior on held-out data.
• Sigmoidal Dynamics: The experiments confirmed that as the number of in-context shots increases, the probability of adopting a persona follows a sigmoid curve. Activation steering shifts this curve horizontally (left for positive steering, right for negative), effectively requiring fewer or more shots to reach the same behavioral threshold.
• Phase Transitions: By varying both $N$ (shots) and $m$ (steering magnitude), the authors mapped a "phase diagram." They found distinct regions where the model is dominated by the target concept or the baseline concept, separated by a sharp transition boundary. The model accurately predicted these boundaries.
• Generalizability:
  • The results held across different model architectures (Llama, Gemma, Qwen) and scales (8B to 70B).
  • The model successfully explained high-prior concepts (where the transition happens faster) and flipped-label sentiment tasks (non-binary outputs).
• Limits of Linearity: The study found that the Linear Representation Hypothesis holds well within a specific range of steering magnitudes. However, at very large magnitudes, the linear effect breaks down, and model behavior converges toward chance (random guessing), suggesting the model's internal representation space has limits.

5. Significance and Implications

• Mechanistic Interpretability: The work bridges the gap between high-level behavioral control (prompting) and low-level mechanistic interventions (activation steering), suggesting they are two sides of the same Bayesian coin.
• AI Safety and Jailbreaking: The ability to predict sudden behavioral shifts (phase transitions) is critical for AI safety. It implies that "many-shot jailbreaking" (where a model suddenly adopts a harmful persona after many examples) is a predictable phase transition. Practitioners can use the model to calculate the exact threshold of context or steering magnitude required to break safety alignments.
• Control Methodology: The findings suggest that combining ICL and steering is highly efficient. A small amount of steering can drastically reduce the number of in-context examples needed to achieve a desired behavior, offering a more robust and computationally efficient method for controlling LLMs.
• Theoretical Foundation: It provides a rigorous mathematical foundation for understanding how neural networks represent and update "beliefs," linking cognitive science concepts (Bayesian inference) with modern deep learning mechanisms (linear representations).

In summary, this paper provides a powerful, predictive framework that demystifies how LLMs learn from context and respond to internal interventions, revealing that both are governed by the dynamics of updating latent concept beliefs.