Technical Summary: Forager – A Lightweight Testbed for Continual Learning with Partial Observability

Problem Statement

Continual Reinforcement Learning (CRL) aims to develop agents that can learn, act, and explore indefinitely in a "big world" that is partially observable and non-stationary. Current CRL research has largely focused on loss of plasticity—the inability of agents to continue learning after task switches—often within fully observable Markov Decision Processes (MDPs) where non-stationarity is artificially introduced.

A critical gap exists in studying CRL under partial observability. While partial observability is a fundamental characteristic of real-world environments (the "Big World Hypothesis"), most existing benchmarks either:

1. Are fully observable (e.g., ALE, Mujoco), where agents can theoretically simulate the environment perfectly.
2. Are prohibitively expensive to run for long-term CRL experiments (e.g., Minecraft, Craftax), often requiring massive parallelism or privileged state information to achieve success.
3. Suffer from technical limitations, such as growing memory footprints that hinder rapid iteration (e.g., Jelly Bean World).

Consequently, the interplay between state construction (using memory/recurrence to handle partial observability) and continual learning (tracking changing dynamics) remains underexplored.

Methodology: The Forager Environment

The authors introduce Forager, a lightweight, configurable grid-world environment designed to serve as a scalable testbed for CRL.

• Environment Structure: Forager simulates an infinite toroidal world where an agent navigates to collect mushrooms with varying rewards and respawn behaviors. The world is procedurally generated but does not rely on complex physics or rich graphics.
• Partial Observability: The agent operates with a limited, agent-centric Field of View (FOV). The observation is either an RGB image or a binary tensor of the visible area. The agent cannot see the global state, necessitating memory to track object locations and respawn times outside the FOV.
• Non-Stationarity: The environment supports various forms of non-stationarity, including:
  • Task Switching: Periodic changes in reward functions for specific object types (e.g., a mushroom type that was previously positive becomes negative).
  • Dynamic Rewards: Rewards that decay over time or change based on hidden schedules.
  • Unending Streams: A variant where new mushroom species replace extinct ones, creating an infinite sequence of tasks.
• Performance Characteristics: Forager is optimized for speed and memory efficiency. It runs at up to $10^5$ frames per second (FPS) with a constant memory footprint, regardless of the agent's exploration history. This contrasts with benchmarks like Jelly Bean World, where memory usage grows with exploration.
• Implementation: The environment is available in a CPU implementation and a JAX-based implementation (Foragax) that enables hundreds of parallel trials on a single GPU.

Key Contributions

1. A Resource-Efficient Testbed: Forager provides a configurable environment for rapid prototyping of CRL algorithms. It allows researchers to control the degree of partial observability (via FOV size) and problem difficulty (via object placement and reward dynamics) without the computational overhead of complex game engines.
2. Demonstration of State Construction Necessity: The paper provides empirical evidence that standard deep RL agents (DQN, PPO) fail to perform well in Forager due to partial observability. It highlights that state construction (via memory traces or recurrent architectures) is more critical for continual performance in this setting than standard plasticity mitigation techniques.
3. Comprehensive Benchmarking Suite: The authors evaluate a wide range of algorithms, including:
   • Baselines: DQN and PPO.
   • Plasticity Mitigations: Regularization towards initialization, cReLU activations, Shrink & Perturb, L2 regularization, and Permanent-Transient Q-learning (PT-DQN).
   • State Construction: Exponential memory traces, Deep Recurrent Q-Networks (DRQN), and Recurrent Temporal Unit PPO (RTU-PPO).
4. Unending Task Variant: A specific configuration that generates an unending stream of new tasks, forcing agents to learn forever without converging to a static policy, thereby highlighting the limitations of current CRL agents.

Experimental Results

1. Impact of Partial Observability (Field of View)

• Observation: As the FOV decreases, the performance of feed-forward DQN agents drops significantly.
• Finding: With large FOVs, DQN approaches the performance of an Oracle Search. With small FOVs (where the agent cannot see across "biome" gaps), DQN fails to learn optimal foraging strategies, behaving like a random or nearest-neighbor search. This confirms that the environment effectively requires memory to track hidden states.

2. Loss of Plasticity and Mitigations

• Observation: In a task-switching variant, PPO exhibited severe loss of plasticity (performance degraded over time), while DQN showed only minor degradation.
• Mitigation Testing: Several proposed mitigations (cReLU, L2 Init, Shrink & Perturb, PT-DQN) were tested.
  • Results: While some mitigations (notably L2 Init and cReLU) improved adaptation speed or prevented total collapse, none allowed the agents to approach the performance of the Oracle Search baseline.
  • Conclusion: Mitigating loss of plasticity alone is insufficient for CRL in partially observable environments; the agents still lack the ability to construct a useful internal state.

3. The Role of State Construction

• Simple Memory: Adding an exponentially weighted memory trace of recent rewards significantly improved performance for both DQN and PPO.
• Recurrent Architectures:
  • DRQN: A modified DRQN with GRUs did not outperform standard DQN, suggesting the specific implementation or algorithm was not extracting useful state in this context.
  • RTU-PPO: A recurrent PPO variant using a Recurrent Temporal Unit (RTU) layer achieved the best performance among all learning agents, nearly matching the Oracle Search baseline.
  • Continual Learning Evidence: When the RTU-PPO agent was "frozen" (stopped learning) halfway through the experiment, its performance dropped significantly. This proves the agent was not converging to a static policy but was actively continually relearning and adapting to task switches while leveraging its recurrent state to handle partial observability.

4. Unending Task Stream

• In the most challenging variant (four biomes, dynamic species replacement, hidden cues), even the best recurrent agents (RTU-PPO) plateaued and failed to reach Oracle performance.
• Interpretation: The combination of rapid reward non-stationarity, large world size, and limited FOV creates a challenge that current algorithms cannot fully overcome, particularly in a single-stream (synchronous) learning setting.

Significance and Claims

The paper claims that Forager fills a critical gap in CRL research by providing a scalable, memory-efficient environment that isolates the challenges of partial observability and state construction.

• Modest Claims: The authors explicitly state that Forager is not a realistic simulation and lacks complex physics. Its value lies in its ability to balance rapid prototyping with sufficient complexity to challenge current algorithms.
• Core Insight: The primary finding is that while mitigating loss of plasticity is important, state construction (via recurrence or memory) is the dominant factor in enabling agents to learn continually in partially observable worlds.
• Future Direction: The unending task variant serves as a "challenge problem" to drive future research into algorithms that can handle rapid adaptation, temporal abstraction, and effective exploration in big, partially observable worlds.

The paper concludes that current CRL agents have clear limitations in these settings and that Forager offers a scientific testbed to investigate these limitations without the computational prohibitions of larger-scale environments.