Technical Summary: Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

1. Problem Formulation

The paper investigates the efficacy of pretraining Q-functions on offline data when fine-tuning a pretrained policy using online Reinforcement Learning (RL). The setting involves a Markov Decision Process (MDP) where an agent possesses a fixed offline dataset $D_{offline}$ and a pretrained base policy $\pi_{base}$ (typically trained via supervised learning or imitation learning on $D_{offline}$). The goal is to perform online RL fine-tuning to maximize returns.

The central question is whether the Q-function ($Q_\phi$), which guides the policy's action selection during online updates, should also be pretrained on $D_{offline}$ alongside the policy, or if it should be randomly initialized and learned entirely online. Conventional wisdom in offline-to-online RL suggests that joint pretraining of the policy and Q-function is beneficial. However, recent works have shown that starting with a pretrained policy and a randomly initialized Q-function can yield highly performant results. This paper systematically tests the intuition that pretraining the Q-function provides a necessary head start for downstream performance.

2. Methodology

Experimental Setup

The authors evaluate their hypotheses on six challenging continuous control manipulation tasks from Robomimic (lift, can, square) and OGBench (cube, scene, puzzle). These tasks involve sparse rewards and long-horizon reasoning.

• Base Algorithm: The study utilizes EXPO, a sample-efficient off-policy RL algorithm. EXPO employs a "base flow policy" ($\pi_{base}$) and a lightweight "edit policy" ($\pi_{edit}$) that predicts action corrections. The final action is selected by maximizing the Q-value over a set of base and edited actions.
• Baselines:
  1. Naive Pretraining: Pretrain both $\pi_{base}$ and $Q_\phi$ on $D_{offline}$ using supervised learning and offline TD learning, respectively, before online fine-tuning.
  2. Random Initialization: Pretrain only $\pi_{base}$ on $D_{offline}$; initialize $Q_\phi$ randomly and learn it entirely during online fine-tuning.
  3. Offline Value Maximization: Attempts to bridge the gap between offline and online targets by performing offline value maximization (e.g., learning an edit policy or distilling a one-step policy) during pretraining.

Proposed Method: Initialization via Policy Ensemble (IPE)

Motivated by the finding that naive pretraining fails, the authors propose IPE, a method to better utilize offline data for Q-function initialization without the mismatch issues of standard pretraining.

• Core Idea: A single pretrained policy often produces a narrow action distribution, causing the Q-function to collapse toward the value function of that specific policy ($Q_{\pi_{base}} \approx V_{\pi_{base}}$) rather than learning a generalizable action-value structure.
• Procedure:
  1. Train a base policy $\pi_{base}$ on the offline dataset.
  2. Train $N$ additional diverse policies ($\pi_1, \dots, \pi_N$) on the same action distribution (using the same dataset or subsets with different seeds).
  3. Roll out all $N+1$ policies in the environment to collect diverse transitions.
  4. Pool these rollouts to initialize the replay buffer for online RL fine-tuning.
     This approach provides the Q-function with coverage over a wider range of actions at each state, enabling it to learn a Q-function closer to the optimal online target ($Q^*_{RL}$) rather than being constrained by the narrow distribution of a single policy.

3. Key Findings and Analysis

The Failure of Naive Pretraining

Contrary to conventional intuition, the authors find that naively pretraining the Q-function provides little to no benefit over random initialization. In several tasks, it even hinders online learning.

• The Mismatch: The Q-function learned during offline pretraining ($Q_{\pi_{base}}$) targets the value function of the base policy. However, online RL fine-tuning converges to a different optimal Q-function ($Q^*_{RL}$).
• Preference Accuracy: Even when the base policy is near-optimal, the offline-pretrained critic does not align with the critic learned by online RL. The offline Q-function fails to correctly rank actions from the optimal policy ($\pi^*$) versus the base policy ($\pi_{base}$) in the way required for online convergence.
• Ineffectiveness of Fixes: Attempts to close this gap via offline value maximization (e.g., training an edit policy or applying action gradients during pretraining) fail to improve performance or preference accuracy, indicating that online interaction is necessary to learn the correct target.

The Mechanism of IPE

The paper demonstrates that the failure of naive pretraining stems from a lack of action diversity. A single policy's rollouts result in a Q-function with very low variance, effectively making it deterministic and unable to distinguish between alternative actions.

• Diversity: By using an ensemble of policies, IPE generates rollouts with significantly higher action diversity.
• Result: This diversity allows the Q-function to learn meaningful action-value structures ($Q^*_{RL}$) rather than collapsing to the value of a single policy's behavior.

4. Results

• Performance Improvement: Across the suite of continuous control benchmarks, IPE yields an average 1.26x improvement (approximately 26%) in fine-tuning performance compared to naive Q-function pretraining.
• Comparison to Random Initialization: IPE outperforms both naive pretraining and random initialization, demonstrating that the quality of the Q-function initialization is critical, but the source of that initialization must be diverse.
• Scaling: Performance scales with the number of policies ($N$) used in the ensemble. In the experiments, $N=5$ provided the best results, suggesting that broader coverage of behavior policies leads to better downstream fine-tuning.
• Preference Accuracy: IPE-trained Q-functions show higher preference accuracy for optimal actions compared to baselines, confirming that the method successfully bridges the gap toward $Q^*_{RL}$.

5. Significance and Claims

The paper challenges the prevailing assumption that "more pretraining is always better" in the context of offline-to-online RL. Its primary contributions are:

1. Empirical Evidence: It systematically demonstrates that directly pretraining a Q-function on offline data often fails to accelerate or improve online RL fine-tuning when starting from a pretrained policy.
2. Theoretical Insight: It identifies a fundamental mismatch between the offline Q-function ($Q_{\pi_{base}}$) and the online target ($Q^*_{RL}$), showing that offline value maximization alone cannot resolve this discrepancy.
3. Practical Solution: It introduces IPE, a simple and effective method that leverages policy ensembles to bootstrap Q-function learning. By increasing action diversity in the pretraining phase, IPE enables online RL to learn the correct value function more efficiently.

The authors conclude that while pretraining the policy is essential, the strategy for initializing the Q-function must account for the diversity of actions required to learn the optimal value landscape, rather than simply fitting the offline data distribution of a single policy. The work is limited to off-policy RL settings, and the authors note that reducing the data-collection overhead of IPE remains a direction for future work.