Technical Summary: Remember to be Curious

Problem Statement

Exploration is a fundamental prerequisite for learning useful behaviors in sparse-reward, long-horizon tasks within 3D environments. While curiosity-driven reinforcement learning (RL) offers a mechanism to generate intrinsic rewards based on prediction errors, scaling this to complex, photorealistic environments has proven difficult. Existing approaches often suffer from "amnesiac" exploration, where agents become trapped in local loops. This occurs because:

1. Lack of Episodic Context: Without a history of past observations, agents repeatedly revisit the same locations, receiving "fresh" novelty rewards for states they have already forgotten.
2. Lack of Spatial Persistence: Standard learned world models (e.g., inverse/forward dynamics modules) often act as statistical priors over experience rather than persistent records. Consequently, they fail to maintain scene consistency, generating spurious prediction errors in revisited areas and providing false novelty signals.

The core challenge is to enable curiosity-driven agents to navigate photorealistic 3D scenes using only visual inputs (RGB) at deployment, without relying on explicit geometric maps or depth sensors, while avoiding repetitive behavior.

Methodology

The paper proposes an end-to-end framework, Remember to be Curious, which couples a persistent world model with an episodic agent.

1. Persistent 3D Forward Model (World Model)

To provide a reliable curiosity signal, the authors utilize an online 3D Gaussian Splatting (3DGS) model as a proxy for the forward world model.

• Mechanism: The model ingests a stream of privileged training-time observations (RGB, depth, camera poses) to build a persistent 3D representation ($G_t$) of the scene.
• Curiosity Reward: At each step, the model renders a prediction of the next observation ($\hat{o}_{t+1}$) based on the current 3DGS state and the agent's action. The curiosity reward ($r^{cur}_t$) is derived from the visual prediction error between the rendered view and the actual observation ($o_{t+1}$).
• Persistence: Unlike standard neural forward models, 3DGS maintains a continuous, updated record of the environment. This ensures that prediction errors only arise from genuinely novel regions, not from the model "forgetting" previously visited states.
• Reward Formulation: The error is low-pass filtered and down-sampled to focus on meaningful structural novelty rather than high-frequency noise. A binary reward is issued: $r_{new}$ for unexplained views and a small penalty $r_{old}$ for views already explained by the model.

2. Episodic Agent Architecture

The agent is a transformer-based policy that operates purely on a sequence of RGB observations and actions, maintaining an internal episodic memory without explicit geometric mapping.

• Input Processing: Each timestep's RGB frame is paired with the preceding action (encoded as a Plücker-ray image). These are processed by an image encoder and enriched with DINOv2 features.
• Memory Mechanism: The agent employs a hybrid attention architecture:
  • Sliding Window Causal Attention: Processes local temporal context efficiently.
  • Global Linear-Attention Memory: Inspired by TTT and LoGeR architectures, this module maintains a running hidden state ($h_t$) that allows information to propagate across the entire episode, enabling long-horizon planning (e.g., backtracking to find new branches).
• Training: The policy is trained via PPO using the curiosity reward as the sole signal. To prevent collapse into repetitive behavior when rewards become sparse, the training rollout includes a mixture of the learned policy and a uniform random policy (scheduled from 20% to 0% over training), alongside entropy regularization.

3. Deployment

At test time, the agent acts solely on the stream of RGB frames. The 3DGS forward model is not used during deployment; the agent relies entirely on its learned internal representation of episodic context to navigate.

Key Contributions

1. Dual Requirement for Curiosity: The paper demonstrates that effective curiosity in photorealistic 3D environments requires both a persistent world model (to avoid false novelty from forgotten states) and an episodic agent (to plan toward novel regions using historical context).
2. End-to-End RGB-Only Policy: The proposed agent requires no explicit geometric maps, depth sensors, or localization modules at deployment. It learns to navigate complex 3D scenes purely from visual inputs.
3. Persistent 3DGS as a Proxy: The work utilizes online 3D Gaussian Splatting to provide the necessary spatial persistence for curiosity signals, addressing the limitations of standard learned forward models that suffer from spatial forgetting.
4. Emergent Long-Horizon Behaviors: The agent learns to traverse already-seen regions (backtracking) to discover unseen "branches" of the environment, a behavior typically difficult to learn in sparse-reward settings without hierarchical goal selection or imitation learning.

Experimental Results

The agent was trained purely on curiosity using the Habitat Matterport 3D (HM3D) dataset.

• Exploration Performance: On HM3D and Gibson datasets, the agent outperforms state-of-the-art RL-based active mapping baselines (ANS, OccAnt) in terms of 3D scene completeness and average distance to unobserved points. It achieves higher coverage faster while requiring only RGB input, whereas baselines often rely on ground-truth depth or explicit maps.
• Ablation Studies:
  • World Model: Removing persistence (using short-term memory or standard ICM) causes the agent to collapse into local loops or get stuck.
  • Agent Memory: Reducing the agent's context window or using RNNs degrades performance. Asymmetric memory (only the critic having context) performs worse than full episodic memory, confirming the need for the policy to retain history.
• Downstream Adaptation: The pre-trained agent was fine-tuned for a few episodes on sparse-reward downstream tasks:
  • Apple Picking: Outperformed agents trained from scratch, particularly in scenarios with very sparse rewards (fewer apples).
  • Image-Goal Navigation: Successfully redirected its general exploration capabilities to reach specific target viewpoints, outperforming scratch-trained baselines.
• Zero-Shot Generalization: The agent generalized to out-of-distribution, AI-generated 3D worlds (Hobbit World, Spaceship) without further training, navigating corridors and avoiding collisions despite differences in rendering pipelines and aesthetics.

Significance and Claims

The paper claims that coupling a persistent model of the world with an episodic agent provides a practical pathway for scaling curiosity-driven exploration to complex, photorealistic 3D environments.

• Modest Claims on World Models: The authors acknowledge that their use of 3DGS is a proxy for a "dynamics-free" world model, confining the agent to static scenes. They posit that as action-conditioned video generative models mature, they may eventually replace 3DGS, provided they can achieve similar spatial persistence and closed-loop online updating.
• Insight for Future Work: The study serves as a guide for future world models, emphasizing that spatial persistence and the ability to continuously update internal representations are strictly required for reliable intrinsic motivation.
• Generalization: The work demonstrates that curiosity-driven exploration, when grounded in persistent memory, can yield agents that are not only effective explorers but also highly adaptable to diverse downstream tasks without the need for explicit mapping or imitation learning bootstrapping.