Technical Summary: Adaptive Multi-Horizon Reinforcement Learning

Problem Statement

Effective decision-making in complex environments requires balancing short-term and long-term outcomes. In standard Reinforcement Learning (RL), this trade-off is governed by a fixed discount factor ($\gamma$), which imposes a single, exponentially discounted planning horizon. This static assumption often fails in continual learning scenarios where tasks and environments evolve, requiring agents to shift between short-horizon (immediate outcome) and long-horizon (delayed reward) decision-making strategies.

Biological systems, in contrast, appear to integrate predictions across multiple temporal horizons and flexibly adjust their reliance on them. Recent neuroscience studies suggest that distinct dopamine neurons and striatal circuits encode value over different discounting timescales. The central challenge addressed by this paper is: How can an agent dynamically adapt its planning horizon to different task demands, similar to the flexible decision-making observed in biological systems?

Methodology

The authors propose a Multi-Horizon Reinforcement Learning approach that adaptively combines temporal horizons, eliminating the need for manual discount-factor tuning. The core methodology involves a Mixture-of-Q-Values framework:

1. Multiple Critics (Experts): The agent maintains multiple action-value functions, $Q_i(s, a)$, each learned independently using tabular Expected SARSA($\lambda$) with a distinct, fixed discount factor $\gamma_i$. These experts cover a range of temporal horizons.
2. State-Dependent Gating Network: A learnable gating network takes the state representation $x = \phi(s)$ and computes logits $z = Wx + b$. These are normalized via a softmax to produce mixing weights $w_i(s)$ for each expert.
3. Mixed Value Function: The final action-value function is a weighted sum:
   $$Q_{mix}(s, a) = \sum_{i=1}^{K} w_i(s) Q_i(s, a)$$
4. Learning Dynamics:
   • The individual experts are updated using their specific $\gamma_i$.
   • The mixed policy $\pi(a|s)$ is derived from $Q_{mix}$ using an $\epsilon$-greedy strategy.
   • The gating network parameters ($W, b$) are optimized to minimize the squared Temporal Difference (TD) error of the mixed value function (where the update uses $\gamma=1$ for the mixture). This allows the network to learn which temporal horizon is most effective for a given state.

Key Contributions

• Adaptive Horizon Selection: The paper introduces a mechanism that dynamically selects and weights multiple planning horizons based on the current state, rather than relying on a single fixed $\gamma$.
• Continual Learning Adaptability: The method is specifically designed for continual learning settings where tasks switch sequentially, requiring the agent to re-evaluate the optimal temporal scale without manual intervention.
• Empirical Validation of Biological Principles: The work provides empirical evidence supporting the hypothesis that adaptive, multi-timescale learning improves parameter efficiency and adaptability, mirroring principles observed in biological decision-making.

Experimental Results

The authors evaluated the approach in MiniGrid environments, focusing on three sequential tasks in a continual learning setting:

1. Foraging: Collecting 40 rewards distributed in a grid.
2. Goal Reaching: Reaching a high-value goal while avoiding lava penalties.
3. Four Rooms: A composite task involving a time-limited "jackpot" and local rewards.

Key Findings:

• Task-Dependent Optimal Horizons: Experiments demonstrated that the optimal discount factor varies significantly by task and reward structure.
  • Foraging: Optimal effective horizons were found to be 16–32 steps.
  • Goal Reaching: Optimal horizons ranged from 32–512 steps.
  • Four Rooms: Optimal performance required an effective horizon of 128 steps.
• Reward Sparsity and Reachability: The study showed that as reward dispersion increases (higher spatial variance) or reward frequency decreases, the optimal discount factor shifts toward larger values (longer horizons) to mitigate sparsity.
• Continual Learning Performance: In the sequential task-switching experiment, the adaptive mixture approach consistently maintained near-optimal performance (episode returns between 36 and 40) across all three tasks.
• Gating Behavior: The gating network successfully learned to assign higher weights to the specific discount factors that were optimal for the current task. For instance, it prioritized shorter horizons for foraging and longer horizons for the time-constrained jackpot task.
• Robustness: While the agent consistently learned to collect the time-constrained jackpot (a difficult sub-goal), collecting remaining local rewards showed some variability depending on the random seed and remaining episode length.

Significance and Claims

The paper claims that its adaptive multi-horizon approach offers a novel framework for combining multiple critics with distinct capabilities. The primary significance lies in:

• Eliminating Manual Tuning: The method removes the need for manual discount-factor tuning, which is particularly beneficial in environments where reward structures change over time.
• Biological Plausibility: The results suggest that adaptive temporal discounting is a viable principle for improving both artificial and biological learning systems, aligning with neuroscientific findings on multi-timescale value encoding.
• Efficiency in Continual Learning: The approach demonstrates improved adaptability when switching between tasks with divergent temporal requirements, suggesting that learning a mixture of horizons is more robust than learning a single fixed horizon.

The authors remain modest regarding future work, noting that further evaluation across a larger number of seeds is needed to characterize variability, and that extending the framework to neural network function approximators and jointly adapting eligibility traces are promising directions for future investigation.