COOPO: Cyclic Offline-Online Policy Optimization Algorithm

Technical Summary of COOPO: Cyclic Offline-Online Policy Optimization

Problem Statement
Reinforcement learning (RL) faces a fundamental trade-off between data efficiency and performance. Pure online RL methods (e.g., PPO, SAC) require prohibitive numbers of environment interactions to learn optimal policies, making them unsuitable for safety-critical cyber-physical systems (CPS) where exploration is costly or dangerous. Conversely, offline RL learns exclusively from static datasets, avoiding online interaction but suffering from distributional shift and policy suboptimality due to the inability to explore unseen states.

Recent hybrid "offline-to-online" approaches attempt to bridge this gap by pre-training on offline data and then fine-tuning online. However, the paper identifies two critical failure modes in existing hybrid methods:

1. Catastrophic Forgetting: During online fine-tuning, the policy often overwrites the knowledge learned from the offline dataset, leading to significant performance degradation.
2. Distributional Shift: The transition from the static offline distribution to the dynamic online distribution causes the policy to drift into regions of the state space unsupported by the data, resulting in inaccurate value estimates and unstable learning.

Existing solutions, such as warm-start phases or blending data in replay buffers, often fail to preserve long-term stability or do not sufficiently reduce the sample complexity of the online phase.

Methodology: COOPO Framework
The authors propose COOPO (Cyclic Offline-Online Policy Optimization), a unified framework that resolves these issues by repeatedly cycling between constrained offline training and online fine-tuning. Unlike traditional methods that perform a single offline pre-training followed by online adaptation, COOPO interleaves these phases in a loop.

The core mechanism operates as follows:

1. Cyclic Structure: The algorithm alternates between an Offline Phase and an Online Phase over $K$ cycles.
2. Offline Phase (Stability & Anchoring):
   • The policy is updated using a KL-regularized Advantage Weighted Actor-Critic (AWAC) algorithm.
   • This phase employs an explicit Kullback-Leibler (KL) divergence penalty to constrain the new policy ($\pi_{new}$) to remain close to the previous policy ($\pi_{old}$) and the behavior policy ($\pi_\beta$) in the dataset.
   • A value function is explicitly trained during this phase to support the subsequent online transition and to provide robust advantage estimation, mitigating out-of-distribution (OOD) errors.
   • Crucially, the offline phase acts as a "corrective stabilizing term," re-anchoring the policy to the high-quality static dataset to prevent drift and forgetting.
3. Online Phase (Adaptation & Exploration):
   • The policy is fine-tuned using a trust-region on-policy algorithm (specifically PPO in the experiments).
   • To maintain critic stability, the Q-function trained during the offline phase is frozen during the online PPO updates. This prevents value drift caused by distribution mismatches between offline and online states.
   • The online phase is kept short (few episodes) to minimize environment interactions while allowing for necessary exploration.
4. Synergy: The cyclic behavior creates a feedback loop where offline phases stabilize the policy and correct errors, while online phases refine the policy using real-world signals. This allows for the amortization of offline data costs across multiple cycles.

Key Contributions

1. Generalized Cyclic Framework: The authors introduce a novel framework that alternates KL-regularized offline training with adaptive online fine-tuning. This design explicitly prevents catastrophic forgetting and distributional shift by periodically returning the policy to the offline dataset.
2. Theoretical Guarantees:
   • Performance Bound: The paper derives a performance difference bound (Theorem 1) showing that the return increase per cycle is guaranteed under standard data coverage assumptions.
   • Sample Complexity: The authors prove that COOPO achieves a total sample complexity of $\tilde{O}(\frac{1}{\epsilon^2} \log \frac{1}{\epsilon})$. Notably, the offline sample complexity is independent of the dataset size $|D|$ due to data reuse, and the online sample complexity scales with $H^3$ (where $H$ is the horizon), which is an improvement over pure online PPO's $H^4$ scaling.
   • Convergence: Theorem 2 establishes linear convergence to the neighborhood of the optimal policy.
3. Empirical Superiority: Extensive experiments on D4RL benchmarks (HalfCheetah, Hopper, Walker2D) demonstrate that COOPO outperforms state-of-the-art offline RL (e.g., IQL, CQL) and offline-to-online baselines (e.g., Cal-QL, Uni-O4) in terms of final return and sample efficiency.

Results

• Performance: COOPO achieved top performance in 7 out of 9 D4RL tasks, surpassing pure offline methods and one-step hybrid approaches. For instance, on the halfcheetah-medium-expert task, COOPO achieved a return of 9242.2, outperforming the next best baseline (Uni-O4 at 8859.3).
• Sample Efficiency: COOPO significantly reduced the number of online environment interactions required to reach high performance compared to pure online PPO. Theoretical analysis and empirical results (Figure 7) confirm that COOPO reduces online interactions by a constant factor.
• Stability: The cyclic architecture demonstrated robust asymptotic performance, avoiding the early plateauing seen in methods like AWAC and IQL. Ablation studies showed that balancing offline epochs ($E$) and online episodes ($T$) is critical; specifically, emphasizing offline training in early stages and online fine-tuning in later stages yields optimal results.
• Hyperparameter Sensitivity: The KL coefficient ($\lambda$) was found to control the trade-off between convergence speed and update conservatism. A moderate $\lambda$ (e.g., 3) provided the best balance of convergence speed and low variance.

Significance and Claims
The paper claims that COOPO sets a new paradigm for sample-efficient and stable adaptive learning, particularly for safety-critical cyber-physical systems. By formalizing a "cyclic synergy," the method transforms the limitations of offline RL (coverage constraints) and online RL (sample inefficiency) into complementary strengths.

The authors emphasize that COOPO provides a principled foundation for reliable deployment in real-world applications where large-scale exploration is infeasible. The framework offers theoretical guarantees on performance improvement and bounded online interaction, addressing the critical need for algorithms that remain stable under distribution shifts and adversarial perturbations. The work suggests that this looped synergy effectively mitigates catastrophic forgetting and distributional drift, offering a robust solution for adaptive control in environments with limited data and uncertain dynamics.