Technical Summary: Deployed Reinforcement Learning should be Continual

Problem Statement

The paper identifies a fundamental misalignment between the theoretical nature of real-world Reinforcement Learning (RL) deployment and the prevailing engineering paradigm, termed "train-then-fix." In this standard paradigm, agents are trained offline (via simulation, self-play, or demonstrations) and then frozen upon deployment. Learning ceases until performance degrades significantly enough to trigger a human-led retraining cycle.

The authors argue that this paradigm is theoretically insufficient for "measurable deployment" settings. Measurable deployment is defined as a scenario where:

1. The agent operates in a "Big World" regime where environmental complexity exceeds the agent's representational capacity and available computational resources.
2. An evaluative reward signal remains available after deployment (e.g., user engagement, ride completion, suggestion acceptance).

Under these conditions, the "train-then-fix" approach leaves performance gains on the table. The paper posits that any agent receiving evaluative feedback in a non-stationary environment where an optimal policy is unrepresentable or unreachable is inherently facing a Continual Reinforcement Learning (CRL) problem. The core issue is that the traditional Markov Decision Process (MDP) formalism, which assumes stationarity and the existence of a fixed optimal policy ($\pi^*$), fails to capture the "never-ending" nature of real-world adaptation.

Methodology and Theoretical Framework

1. Theoretical Foundation: History Process Formalism

To rigorously define the problem, the paper moves away from the standard MDP tuple $\langle S, A, P, R \rangle$. Instead, it adopts the History Process formalism (Bowling et al., 2023; Abel et al., 2023).

• Environment ($e$): Defined as a function mapping finite-length histories ($H$) and actions ($A$) to a distribution over observations ($O$).
• Agent State ($S$): Represents the agent's compression of its history, distinct from the MDP state.
• Learning Rule ($\sigma$): A mapping from histories to a distribution over policies ($\Pi$).
• CRL Definition: An RL problem is an instance of CRL if the "best agent" (one maximizing a performance measure over reachable policies) never stops learning. In the history process, because the agent cannot reset the environment or revisit exact past histories, and because the environment is non-stationary, the search for an optimal policy never terminates.

2. Characterization of Non-Stationarity

The paper identifies four specific sources of non-stationarity that necessitate continual learning in measurable deployment:

1. Action-Induced Non-Stationarity: The agent's own actions alter the distribution of future histories (e.g., a recommendation system changing user preferences).
2. Changes in Environment Dynamics: External factors such as seasonal variations, hardware aging, or market shifts that occur outside the agent's control.
3. Evolving Goals: The reward function itself changes over time due to shifting stakeholder priorities, regulatory updates, or safety constraints.
4. Emergent Novelty: The "Big World Hypothesis" dictates that agents will inevitably encounter action-observation sequences (including "black swan" events) that lie outside the distribution of their finite training phase.

3. Analysis of Learning Rules

The authors contrast two learning rules (strategies for updating policies):

• Learning Rule A (Train-then-Fix): The agent searches for a high-performing policy, settles on a fixed $\pi_{final}$, and stops. Performance degrades over time until a threshold triggers a new search cycle, resulting in a "sawtooth" performance pattern.
• Learning Rule B (Continual Learner): The agent recognizes that no reachable policy is globally optimal. It never terminates its search, continuously adapting its policy using the available evaluative signal to track changing targets.

Key Contributions

1. Paradigm Shift Proposal: The paper argues that measurable deployment is not merely a problem solved by continual learning algorithms, but is inherently a CRL problem. The distinction lies in the problem setting (environmental characteristics) rather than just the algorithmic solution.
2. Formal Justification: By utilizing the history process formalism, the authors demonstrate that the assumption of a fixed optimal policy is invalid in real-world deployment due to the four identified sources of non-stationarity.
3. Empirical Case Studies: The paper analyzes three successful real-world deployments that implicitly or explicitly utilize continual learning:
   • Cursor Tab: A code completion system using online policy gradients. It faces emergent novelty (new code patterns) and environment dynamics (evolving libraries). Its tight deployment-training loop (1.5–2 hours) allows it to adapt to these shifts, resulting in a 21% reduction in suggestions and a 28% higher acceptance rate.
   • Lyft: A ridesharing matching algorithm using online RL. It primarily faces action-induced non-stationarity (driver relocations) and environment dynamics (demand fluctuations). The system generates over $30M in additional annual revenue by adapting online rather than relying on fixed models.
   • Sim-to-Real Gap: The paper argues that the gap between simulation and reality is a manifestation of all four non-stationarity sources. Even with domain randomization, policies require real-time data for reliable performance, making post-deployment learning essential.
4. Actionable Recommendations:
   • For Practitioners: Build feedback loops where production data is training data; validate continually via online monitoring (e.g., switchback experiments); and deliberately introduce non-stationarity during development to stress-test adaptability.
   • For Researchers: Re-evaluate conventional optimizers (e.g., Adam, RMSProp) in continual settings; derive algorithm properties for history processes (moving beyond Markovian guarantees); adopt evaluation metrics like deviation regret that do not assume weakly communicating environments; and explicitly account for finite resource constraints in theoretical models.

Results and Evidence

The paper does not present new experimental results generated by the authors but synthesizes evidence from existing literature and industry reports:

• Cursor Tab: Demonstrated that continuous on-policy updates lead to superior performance metrics compared to static model releases.
• Lyft: Showed that online RL matching systems outperform fixed policies in terms of driver utilization and revenue, validating the ability to adapt to induced non-stationarity.
• Theoretical Evidence: Cites the "Rusting Pendulum" environment (Appendix A) where a train-then-fix policy fails as friction accumulates, while a continual learner maintains performance.
• Literature Trends: Highlights a surge in academic interest in CRL (Figure 1) and the maturation of theoretical foundations (e.g., history process formalism) that support this shift.

Significance and Claims

The paper claims that the "train-then-fix" paradigm is a delay mechanism for the inevitable continual learning problem, not a solution to it. Its significance lies in:

• Reframing Deployment: It asserts that deploying an agent with evaluative feedback in a complex world is a CRL problem by definition. Ignoring this signal and relying on human intervention for updates is a suboptimal strategy that sacrifices performance.
• Safety and Adaptability: Contrary to the view that fixed policies are safer, the paper argues that fixed policies degrade unsafely in non-stationary environments. Continual learning, when designed with safety constraints, is necessary to maintain alignment with evolving goals and regulations.
• Community Direction: It calls for a shift in research focus from algorithms that converge to fixed artifacts toward agents capable of sustained, open-ended learning, and urges practitioners to leverage the feedback signals already present in their systems.

The paper concludes that the path to better performance in real-world RL is not better offline training, but the acceptance of the continual nature of deployment and the design of agents that never stop searching for better policies.