Combining Trained Models in Reinforcement Learning

Technical Summary: Combining Trained Models in Reinforcement Learning

Problem Statement
Deep Reinforcement Learning (DRL) has achieved significant milestones in domains such as Atari and Go, yet it faces two persistent practical weaknesses: high sample costs and poor transferability when task dynamics, observation processes, or environments shift. A common strategy to mitigate these issues is the reuse of information from previously trained models (pretrained knowledge) rather than initializing training from scratch for every target task. However, the literature on mechanisms for this reuse—specifically transfer learning, policy distillation, ensemble methods, and federated training—is fragmented. Existing comparisons are often difficult to interpret due to inconsistencies in tasks, baselines, and compute budgets, making it unclear under what conditions pretrained knowledge reuse genuinely outperforms training a single agent from scratch.

Methodology
This paper presents a PRISMA-guided systematic review of empirical studies on pretrained knowledge reuse in DRL. The review process involved:

• Search Strategy: Queries were conducted across IEEE Xplore, the ACM Digital Library, and through citation tracing using specific intervention terms (e.g., "policy distillation," "federated reinforcement learning") combined with RL-context terms (e.g., "deep reinforcement learning," "actor-critic").
• Screening: From an initial pool of 589 records, 570 unique records were screened. After full-text assessment and eligibility auditing, 15 empirical studies were selected for the main synthesis.
• Eligibility Criteria: Included studies had to be empirical DRL works published between 2015 and 2025, where pretrained model reuse was a central methodological component, and which compared the proposed method against a meaningful baseline (trained without reused knowledge) on performance, sample efficiency, or robustness.
• Synthesis Approach: Due to heterogeneity in environments, metrics, and model classes across the selected studies, the authors employed a qualitative narrative synthesis rather than a meta-analysis. The studies were analyzed across three factors: source-target similarity, diversity among reused models, and the fairness of comparisons against from-scratch baselines.

Key Contributions

1. Scope Definition: The paper establishes a narrower, internally consistent scope for reviewing pretrained knowledge reuse in DRL, distinguishing between actual knowledge reuse during learning and adjacent work such as inference-only filtering or hardware-throughput acceleration.
2. Empirical Synthesis: It synthesizes evidence across four mechanism families:
   • Distillation: A teacher policy supervises a student model.
   • Transfer/Policy Reuse: Reusing policies, skills, or priors on related tasks.
   • Ensembles: Multiple models jointly influencing control or value estimation.
   • Federated Training: Decentralized agents aggregating updates without centralizing raw data.
3. Independence Spectrum: The authors propose a provisional "independence spectrum" to describe how reused models differ in training experience, ranging from seed diversity (random initialization) to data, task, and reward diversity. This is presented as a hypothesis for future benchmarking rather than a validated metric.

Results and Patterns
The analysis of the 15 included studies reveals three recurring patterns:

• Structural Overlap is Critical: Positive results are concentrated in settings where the source and target tasks share substantial structure (e.g., shared dynamics, task families, or policy semantics). When structural mismatch is high, reuse can be detrimental unless the method includes explicit gating, alignment, or adaptive weighting mechanisms to manage the transfer.
• Sparse Evidence for Aggregation: While evidence for ensembles and federated aggregation is promising, it is sparse and largely limited to narrow settings. For instance, ensemble results are currently supported by only one study in the corpus, and federated studies focus on relatively homogeneous client settings. Consequently, broad claims about the superiority of these mechanisms over transfer or distillation are not supported by the current evidence.
• Weak Compute Reporting: A significant limitation in the existing literature is the lack of compute-matched comparisons. Most studies do not normalize training budgets, wall-clock time, or pretraining costs against the target-task baseline. This makes it difficult to validate claims regarding efficiency gains over stronger single-agent baselines trained from scratch.

Significance and Claims
The paper modestly claims that its primary contribution is a "cleaner map" of the current literature's capabilities and limitations. It argues that the field does not need more inflated claims about universal performance improvements. Instead, the evidence suggests that:

• Pretrained knowledge reuse is not a single mechanism but a family of interventions whose success is heavily dependent on source-target compatibility.
• Claims about the general superiority of ensembles or federated learning are premature due to the thinness of the empirical base.
• Future work requires compute-controlled baselines, clearer reporting of uncertainty, and direct benchmarks that vary source-target overlap while holding target tasks fixed.

The authors conclude that while reuse is effective under specific structural conditions, the current literature lacks the rigorous, compute-normalized evidence required to make definitive statements about efficiency or universal applicability.