Active Offline-to-Online Reinforcement Learning

Technical Summary: Active Offline-to-Online Reinforcement Learning

Problem Statement
Offline Reinforcement Learning (RL) allows for policy training from static datasets, avoiding the costs and risks of initial online exploration. However, policies pretrained offline often suffer from distributional shift, leading to unpredictable or poor performance in real environments. The Offline-to-Online (O2O-RL) paradigm addresses this by using a limited budget of online interactions to fine-tune pretrained policies.

Existing O2O-RL approaches typically focus on improving the offline pretraining stage or assume a single policy is selected for fine-tuning. A critical gap exists in active policy selection under a limited interaction budget. Standard pipelines often commit to a single policy for fine-tuning or divide the budget equally among all candidates. This is risky because:

1. Pretrained policies can perform arbitrarily poorly in real environments.
2. Fine-tuning performance is highly sensitive to algorithm choice, hyperparameters, and random seeds.
3. Fine-tuning trajectories are non-monotonic; they may stall, regress, or exhibit high variance, making it difficult to identify which policy will ultimately succeed without wasting the entire budget on a suboptimal candidate.

The core challenge is the trade-off between allocating online interactions to evaluating multiple candidates to identify high-performing ones versus fine-tuning a specific candidate to improve its performance.

Methodology
The authors propose an Active O2O-RL framework that jointly performs policy selection and fine-tuning. The approach operates in two stages:

1. Offline Stage: A diverse pool of $K$ candidate policies is trained using various offline RL algorithms (e.g., AWAC, IQL, CalQL, ReBRAC) and hyperparameter configurations (e.g., learning rates, batch sizes) on the static dataset.
2. Online Stage (Active Selection & Fine-Tuning):
   • Local Linear Forecasting: For each candidate policy lineage, the system maintains a history of value estimates obtained through online evaluation. It fits a local linear regression model to the most recent $w$ value estimates to forecast future performance trends. This model captures short-term upward or downward trends and estimates time-varying uncertainty (residual variance).
   • Upper Confidence Bound (UCB) Selection: The system calculates an Upper Confidence Bound (UCB) for the future performance of each policy. The UCB is derived from the forecasted value plus a scaled prediction interval.
   • Adaptive Exploration: Instead of using a fixed exploration parameter, the method dynamically adjusts the exploration degree based on the current best observed value ($v^*_j$). It selects the policy that requires the minimum "scale" to reach the current best value, effectively balancing the need to explore uncertain but promising policies against exploiting known high performers.
   • Iterative Loop: In each iteration, the policy with the highest UCB is selected, fine-tuned for one step, and evaluated online. The value estimate is added to the lineage, the local linear model is updated, and the UCBs are recalculated. If a different policy's UCB exceeds the current best, the system switches to that policy.

Key Contributions

• Novel Problem Formulation: To the authors' knowledge, this is the first work to address active policy selection for fine-tuning under a limited interaction budget in O2O-RL settings. It explicitly models the trade-off between evaluation and fine-tuning.
• Active Selection Algorithm: The paper proposes a method that uses local linear regression to forecast future performance and employs a UCB criterion to actively select which policy to fine-tune. This avoids premature commitment to suboptimal policies.
• Empirical Validation: The approach is evaluated across a diverse suite of simulated robotics benchmarks, including navigation (Point Maze), classic control (Inverted Pendulum, Swimmer, Reacher, Pusher), and legged locomotion (Hopper, HalfCheetah, Walker2d, Ant).

Results
Experiments were conducted with an online interaction budget of 1,000K transitions (and a smaller budget of 750K in ablation studies). The proposed ACTIVE method was compared against several baselines:

• RANDOM: Selecting a policy at random without fine-tuning.
• BEST: Selecting the best offline-estimated policy without fine-tuning.
• FTS (Fine-Tuning Single): Fine-tuning a single randomly selected policy with the entire budget.
• FTA (Fine-Tuning All): Dividing the budget equally among all candidates.

Findings:

• Consistent Outperformance: The ACTIVE approach consistently outperformed all baselines across most environments, achieving an overall average normalized return of 82.3%, compared to 64.3% for FTS and 63.1% for FTA.
• Adaptability: In environments where policies improve slowly or require long warm-up periods (e.g., legged locomotion), ACTIVE successfully identified promising lineages and allocated the majority of the budget to them, outperforming FTA which wasted resources on weak candidates. Conversely, in environments where policies could regress or stall, ACTIVE switched away from failing lineages, outperforming FTS which would get stuck in local optima.
• Limitations: The method struggled in specific environments like Swimmer and Ant, where policy lineages showed little indication of improvement for extended periods. In these cases, the system sometimes exhausted the budget attempting to identify an improving lineage that never materialized.
• Sensitivity: An ablation study on the window size ($w$) for the local linear model showed that while $w=3$ is generally effective, larger windows can improve performance in high-dimensional, noisy environments (like Ant) by producing less noisy forecasts, though they may reduce sensitivity to rapid changes.

Significance and Claims
The paper claims that its framework advances offline RL toward practical deployment in real-world systems where online interaction is costly or risky. By demonstrating that active selection and dynamic budget allocation are more effective than committing to a single policy or equally dividing resources, the work provides a robust mechanism for handling the volatility of fine-tuning pretrained policies.

The authors modestly note that while their approach outperforms strong baselines, there is room for improvement. They suggest future directions such as leveraging exploration rollouts for monitoring progress, incorporating policy similarities, and developing OPE methods specifically tailored to O2O-RL to better eliminate poor candidates early. The work extends their previous conference paper by providing a more comprehensive analysis, expanded experiments across more tasks, and detailed ablation studies.