Technical Summary: In-Context Reinforcement Learning under Non-Stationarity: A Survey

1. Problem Definition

This survey addresses Non-Stationary In-Context Reinforcement Learning (ICRL), defined as the challenge of adapting a sequential decision-making agent to a changing environment without updating the model's parameters at test time.

In standard ICRL, a pretrained or fine-tuned model infers latent task rules from interaction history (context) to improve future actions. However, existing surveys largely focus on stationary settings where accumulated context serves as evidence for a fixed task. In non-stationary settings, the decision process changes over the agent's lifetime. Consequently, accumulated context is not merely "more evidence"; it can become stale, misleading, or invalid due to shifts in:

• Reward specifications (e.g., preference changes, reward relabeling).
• Transition kernels (dynamics drift).
• Observation channels (sensor mapping changes).
• Action interfaces (action semantics or remapping).
• Task distributions or behavior policies.

The core problem is context validity: a fixed-parameter policy must not only infer the current decision rule but also determine which parts of its accumulated evidence (trajectories, rewards, demonstrations, retrieved memories) still support that rule. The agent must adapt through the context channel (selection, compression, retrieval, forgetting) rather than through weight updates.

2. Methodology and Framework

The paper does not propose a single new algorithm but establishes a taxonomy and evaluation framework to organize and analyze the field.

2.1 Taxonomy of Non-Stationarity

The survey structures the problem along three independent axes to distinguish between qualitatively different adaptation challenges:

1. Source of Change (What changes?):
   • Reward/Preference, Dynamics, Observation, Action Semantics, Instruction/Tool Interface, Task Distribution, Behavior Policy, and Horizon/Resource constraints.
2. Temporal Pattern (How does it change?):
   • Abrupt Shift: Requires rapid change-point detection and suppression of pre-shift evidence.
   • Gradual Drift: Requires tracking moving targets and smoothing noisy evidence.
   • Recurring Modes: Requires memory indexing to retrieve old context when a regime returns.
   • Adversarial/Worst-Case: Requires robustness against strategically timed or poisoned context.
3. Observability (How visible is it?):
   • Explicitly Signaled: Change is announced via tokens or instructions.
   • Partially Observable: Change is inferred from prediction errors or delayed outcomes.
   • Latent/Confounded: The same observations are compatible with multiple shift explanations.
   • Delayed/Noisy: Evidence of the shift arrives late or is corrupted.

2.2 Context Lifecycle Management

The survey reframes ICRL methods as context-management mechanisms. It decomposes the adaptation process into a lifecycle of operations:

• Write: Which interactions are stored?
• Index: How are items keyed for retrieval?
• Retrieve: Which items enter the context window?
• Compress: How are long histories summarized within a budget?
• Trust/Weight: How strongly does each item influence the policy?
• Forget/Isolate: When is context removed or partitioned?

Methods are categorized by how they handle these operations:

• Algorithm Distillation: Learns improvement patterns from histories; risks over-trusting stationary histories.
• Supervised ICRL/Decision Transformers: Scales via offline data; risks next-action loss rewarding stale imitation.
• Retrieval-Augmented ICRL: Selects external memories; risks retrieving invalid old modes based on surface similarity.
• Value-Aware/Model-Based ICRL: Infers values or dynamics from context; risks invalid context corrupting value estimates or planning.
• Efficient Backbones (S4, Mamba): Extend memory length; risks compressed states keeping invalid evidence.

2.3 Evaluation Protocols

The paper argues that average return is insufficient for non-stationary ICRL. It proposes a lifecycle evaluation approach focusing on:

• Post-Shift Recovery Time: Steps to return to near-oracle performance.
• Dynamic Regret: Gap to a changing optimal policy.
• Stale-Context Sensitivity: Performance drop when invalid old evidence is injected.
• Re-adaptation Latency: Speed of recovery when a previous mode recurs.
• Context Efficiency: Return per token or retrieval cost.

It advocates for paired stress tests (e.g., comparing valid context vs. valid + stale context) and oracle baselines (e.g., oracle change-point truncation) to isolate whether adaptation comes from the policy capacity or the context management strategy.

3. Key Contributions

1. Formal Definition: Clearly delineates Non-Stationary ICRL from stationary few-shot adaptation, domain randomization, online continual learning (which updates weights), and standard offline RL. The defining constraint is fixed parameters with context-only adaptation.
2. Unified Taxonomy: Introduces the Source–Pattern–Observability vocabulary. This allows precise characterization of benchmarks (e.g., distinguishing "reward/abrupt/explicit" from "dynamics/gradual/latent") and prevents overbroad claims of "handling non-stationarity."
3. Context Management Perspective: Shifts the focus from model architecture to context validity. It identifies that the adaptation mechanism is the pipeline of writing, retrieving, compressing, and weighting evidence, not just the forward pass of the transformer.
4. Evaluation Standards: Proposes a rigorous reporting checklist (Table 16) requiring lifecycle curves, stale-context ablations, and specific shift specifications. It highlights that success on broad procedural benchmarks does not imply success in non-stationary ICRL unless within-lifetime shifts and recovery through context are demonstrated.
5. Research Agenda: Identifies specific gaps, including the need for validity-aware retrieval, adaptive forgetting, finite-context theory under stale evidence, and context poisoning threat models.

4. Results and Findings

As a survey, the paper synthesizes existing literature rather than reporting new experimental results. Its primary findings are:

• Current Limitations: Many existing ICRL systems (e.g., Decision Transformers, Retrieval-Augmented agents) are evaluated on stationary or domain-randomized tasks. They often fail when context validity changes because they lack mechanisms to detect invalidation or distinguish between recency and relevance.
• Failure Modes: The survey identifies specific failure modes such as Stale Imitation (copying high-return pre-shift trajectories), Context Inertia (long context dominating new evidence), Retrieval Contamination (retrieving invalid old modes), and Over-Forgetting (losing useful recurring modes).
• Training Data Gap: Standard training datasets often lack invalidation events. To learn non-stationary adaptation, models must be trained on data where old context becomes wrong, and supervision must reward the model for changing behavior accordingly.
• Evaluation Gap: Current benchmarks often conflate generalization with adaptation. A model that generalizes to new tasks (held-out) may still fail to adapt when the task changes within a lifetime.

5. Significance and Claims

The paper claims that Non-Stationary ICRL represents a critical frontier for deploying fixed-parameter agents in real-world environments where conditions change.

• Shift in Focus: The field must move from asking "Does context help?" to "When should context be trusted?" and "How does a frozen agent know the difference between valid and invalid evidence?"
• Practical Necessity: In real-world deployments (e.g., robotics, interactive agents), online weight updates are often infeasible or unsafe. The ability to adapt through context is therefore a prerequisite for robustness.
• Scientific Rigor: The survey argues that without the proposed taxonomy and lifecycle evaluation, claims of "non-stationary adaptation" are often unverified. It calls for a mature literature that reports specific shift types, temporal patterns, and context-management mechanisms rather than generic performance metrics.
• Theoretical Gap: It highlights a lack of theoretical understanding regarding how regret and sample complexity scale with context length, retrieval error, and the presence of stale evidence in a bounded context window.

In summary, the paper serves as a foundational framework to unify disparate lines of work (meta-RL, retrieval-augmented agents, decision transformers) under a common problem definition, providing the necessary tools to evaluate, compare, and advance agents capable of adapting to changing worlds without retraining.