Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning

Technical Summary: Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning

1. Problem Formulation

The article addresses a critical challenge in Offline-to-Online Reinforcement Learning (O2O-RL): How can a limited budget of online interactions be used efficiently to select and refine the best policy from a set of candidates that were pre-trained offline?

In standard O2O-RL pipelines, multiple candidate policies are trained offline using various algorithms and hyperparameters. These are typically evaluated via Off-Policy Evaluation (OPE) to select a single policy for deployment, which is then fine-tuned. The authors identify two main shortcomings in this approach:

1. Unreliability of OPE: OPE estimates are often inaccurate due to distributional shifts, making it risky to deploy a policy based solely on these scores.
2. Inefficiency of Online Evaluation (OE) and naive fine-tuning: Exhaustively evaluating all candidates online consumes the interaction budget without refining any policy. Conversely, committing to a single policy for fine-tuning is risky, as pre-trained policies may stagnate or regress during online interaction, especially in non-stationary environments. Furthermore, it is often impossible to know a priori which policy will benefit most from fine-tuning.

The core problem is defined as minimizing Regret within a fixed interaction budget $N$. The goal is to find a procedure that distributes $N$ online transitions across $K$ candidate policies (where each may undergo multiple fine-tuning steps) to maximize the value of the final selected policy, rather than simply selecting the best initial candidate or evaluating all candidates uniformly.

2. Methodology

The proposed framework, Adaptive Policy Selection and Fine-Tuning, operates in two stages:

A. Offline Stage

• A diverse pool of $K$ candidate policies is trained using various Offline-RL algorithms (e.g., AWAC, IQL, CalQL, ReBRAC) and hyperparameter settings.
• Off-Policy Evaluation (OPE) is performed to generate initial performance estimates. These estimates serve to rank the policies but are not treated as ground truth for value estimation during the online phase.
• Initial value estimates for the online phase are set to the value of the behavior policy ($\hat{v}_B$), derived from the dataset, and enriched with pseudo-observations to address identification issues in early phases.

B. Online Stage (Adaptive Loop)
The online phase uses an Upper Confidence Bound (UCB) approach to dynamically allocate the interaction budget $N$. The process proceeds iteratively as follows:

1. Selection: A priority queue (Max-Heap) selects the policy with the highest max-UCB. The UCB is calculated based on a forecast of the policy's future performance.
2. Fine-Tuning & Evaluation: The selected policy undergoes one iteration step of fine-tuning using a portion of the budget, followed by online evaluation to obtain a new value estimate.
3. Value Prediction Model: The authors model the evolution of policy values during fine-tuning using a linear Autoregressive (AR(2)) process combined with a conditionally heteroskedastic (ARCH(1)) process. This captures both short-term trends (upward/downward) and time-varying variance (volatility), which is crucial since fine-tuning curves can be irregular, stagnate, or regress.
4. Prediction & Update: Using the fitted AR-ARCH model, the system simulates $R$ future value paths to calculate the 95th percentile (UCB) for future steps. The max-UCB (the maximum UCB over the remaining budget horizon) is computed.
5. Re-queuing: The policy is updated with its new value and max-UCB and pushed back into the priority queue.
6. Switching: If the max-UCB of another candidate exceeds that of the currently selected policy, the system switches focus to the new candidate. This allows the algorithm to abandon policies that are regressing or stagnating and allocate resources to more promising candidates.

The process continues until the interaction budget is exhausted, returning the policy with the highest estimated value from the set of all evaluated policies.

3. Main Contributions

• New Framework: The article proposes the first approach combining adaptive policy selection and fine-tuning under a strict online interaction budget. It moves beyond the binary choice of "select first, then fine-tune" or "evaluate all" to a dynamic allocation strategy.
• Modeling Fine-Tuning Volatility: The authors introduce an AR(2)-ARCH(1) model to predict the non-stationary evolution of policy values during fine-tuning. This explicitly accounts for the possibility of performance regression and variance, which are often overlooked by standard regression models.
• UCB-based Allocation: By using a UCB criterion for predicted future performance, the method efficiently balances exploration (evaluating new candidates) and exploitation (fine-tuning promising candidates), ensuring the budget is not wasted on policies unlikely to improve.

4. Experimental Results

The approach was evaluated on four standard locomotion tasks for legged robots from the D4RL benchmark (Hopper, Cheetah, Walker, Ant) across four dataset quality levels (Random, Medium, Medium-Replay, Medium-Expert).

• Benchmarks: The method was compared with:
  • OPE: Selecting the best policy based solely on offline estimates.
  • OE: Uniformly splitting the budget for online evaluation of all candidates without fine-tuning.
  • FT: Selecting the OPE-best policy and fine-tuning with the entire budget.
  • Best: A hypothetical upper bound using the actually best policy.
• Performance:
  • The proposed method consistently achieved the highest average values across all environments and budget sizes (160K and 320K transitions).
  • It significantly outperformed FT, which often suffered from committing the entire budget to a single policy that might regress (e.g., in Walker-Medium-Expert).
  • It outperformed OPE and OE by effectively using the budget to both identify high-potential candidates and refine them, thereby avoiding the "opportunity loss" from lack of fine-tuning or the "regret" from fine-tuning the wrong policy.
  • Results showed that adaptive allocation enables the system to recover from poor initial OPE rankings and prevents wasting resources on policies that do not improve through online interaction.

5. Significance and Claims

The authors claim that this work offers a practical solution for deploying RL in real-world systems where online interactions are costly, risky, or time-consuming. By acknowledging that pre-trained policies can perform arbitrarily poorly and that fine-tuning results are volatile, the proposed framework provides a robust mechanism to navigate these uncertainties.

The article modestly notes that while the approach outperforms strong benchmarks, it is not a definitive solution. Limitations include the computational cost of frequent online evaluations and dependence on specific hyperparameter settings. The authors suggest future directions such as using exploration rollouts to reduce explicit evaluation costs, integrating policy similarity metrics, and developing OPE methods specifically tailored for O2O-RL rankings. Ultimately, the framework aims to bridge the gap between theoretical offline RL and practical, deployable systems.