Can In-Context Learning Support Intrinsic Curiosity?

Technical Summary: Can In-Context Learning Support Intrinsic Curiosity?

Problem Statement
Effective machine learning relies not only on modeling data but on the automated selection of data, a process often termed "intrinsic curiosity." The theoretical gold standard for intrinsic curiosity is Bayesian Information Gain (BIG), which measures the expected information gained about an environment's latent parameters ($\theta$) from an observation. However, computing BIG is generally intractable because it requires explicit Bayesian inference over a hypothesis space.

Classic alternatives, such as Learning Progress (LP), reward agents based on the improvement in a world model's predictive ability after acquiring new data. While theoretically sound, traditional LP implementations require expensive inner loops of gradient descent updates to the world model for every trajectory step. This computational bottleneck renders LP impractical for large-scale sequence models. The central question of this work is whether In-Context Learning (ICL)—where sequence models act as amortized, update-free predictors—can serve as a viable world model to compute intrinsic curiosity rewards efficiently, specifically by approximating BIG without explicit parameter updates.

Methodology
The authors formalize the setting as a Bayes-Adaptive Markov Decision Process (BAMDP) where a policy $\pi_\phi$ interacts with environments sampled from a prior $p(\theta)$. The goal is to maximize the mutual information $I(s_t; \theta | h_t)$ (BIG) using only the prediction errors of a frozen, pre-trained sequence model $\rho$ (an in-context learner).

The authors propose and analyze three types of intrinsic rewards derived from $\rho$:

1. Surprisal ($r_{sur}$): The negative log-likelihood of the observation, $-\log \rho(s_t | h_t)$.
2. Description Length Reduction ($r_{dl}$): The improvement in log-likelihood if the current transition were included in the model's context, theoretically requiring a counterfactual update of the posterior.
3. Summed Future Improvement ($r_{sum}$): A novel reward measuring the telescoping sum of prediction error improvements for all future steps in a trajectory if the current state were observed versus masked. This is defined as:
   $$r_{sum}^t := \sum_{t'=t+1}^T [\log \rho(s_{t'} | h_{t'}) - \log \rho(s_{t'} | h_{t' \setminus t})]$$
   where $h_{t' \setminus t}$ represents the history with $s_t$ masked.

Theoretical analysis is conducted to determine if these rewards can unbiasedly estimate BIG in general BAMDPs versus the restricted Bayesian Experimental Design (BED) setting, where transitions are conditionally independent given the action and latent parameters (e.g., active learning).

Key Contributions

1. Theoretical Limits in General BAMDPs:
   The authors prove that for general BAMDPs, no finite-horizon reward implementable via an in-context learner's interface can be an unbiased estimator of BIG.

   • Surprisal ($r_{sur}$) is shown to be biased by aleatoric entropy (the "noisy TV" problem), rewarding unpredictable noise rather than learnable structure.
   • Learning Progress ($r_{dl}$) generally cannot be implemented via ICL because it requires modifying the model's posterior distribution, which lies outside the standard predictive interface of a frozen sequence model.
   • Summed Improvement ($r_{sum}$) is implementable but suffers from a persistent "abductive" bias (a coupling between the current state and the next observation not explained by $\theta$) and a residual term that only vanishes as the trajectory length $T \to \infty$.

2. Positive Results in Bayesian Experimental Design (BED):
   In BED settings (where $p(s_t | s_{t-1}, a_{t-1}, \theta) = p(s_t | a_{t-1}, \theta)$), the structural obstructions vanish:

   • The abductive bias in $r_{sum}$ disappears, making it an asymptotically unbiased estimator of BIG as $T \to \infty$.
   • $r_{dl}$ becomes implementable via a counterfactual action commitment (reusing the action) and converges to BIG as $t \to \infty$, bounded by a Jensen gap.
   • The authors establish that $r_{sum}$ and $r_{dl}$ provide lower and upper bounds on BIG, respectively, in BED settings.

3. Empirical Validation:
   Experiments were conducted across three environments: Gaussian Process (GP) function estimation, Mastermind (code-breaking), and Alchemy (rule discovery).

   • Noise Robustness: In noisy variants of these environments, policies trained with $r_{sur}$ failed significantly, often performing worse than random baselines due to the noisy TV problem.
   • Performance: Policies trained with $r_{sum}$ and $r_{dl}$ successfully learned to ignore aleatoric noise and focus on epistemic uncertainty, achieving performance comparable to or exceeding policies trained with task-specific rewards ($r_{task}$).
   • Model Efficiency: The experiments demonstrated that pre-trained sequence models (Prior-Fitted Networks) can effectively approximate the complex Bayesian oracles required for these rewards, validating the use of ICL for intrinsic curiosity.

Significance and Claims
The paper claims to provide the first rigorous theoretical characterization of the relationship between prediction-based intrinsic rewards and Bayesian Information Gain when using in-context learners. It establishes that while ICL cannot perfectly replicate BIG in general sequential decision-making processes due to inherent biases and interface limitations, it successfully supports intrinsic curiosity in BED settings.

The significance lies in demonstrating a computationally efficient pathway to curiosity-driven exploration that bypasses the need for expensive gradient-based world model updates. By proving that $r_{sum}$ and $r_{dl}$ can asymptotically recover BIG in specific, relevant settings (like active learning), the work suggests that large sequence models can serve as effective "world models" for data collection policies, enabling agents to explore optimally without explicit task definitions or parameter updates. The authors remain modest, noting that extending this to general BAMDPs with temporal structure remains an open challenge requiring further architectural or methodological innovations.