Modernising Reinforcement Learning-Based Navigation for Embodied Semantic Scene Graph Generation

1. Problem Statement

The paper addresses the challenge of Embodied Semantic Scene Graph Generation (ESSG). In this task, an autonomous agent must explore an unknown environment to construct a Semantic Scene Graph (SSG)—a structured representation of objects, their attributes, and spatial relations—within a fixed action budget (a limited number of steps).

The core problem identified in existing literature (specifically the baseline by Li et al.) involves two major limitations:

1. Unstable Learning: The use of the REINFORCE algorithm leads to unstable training dynamics and poor convergence.
2. Inefficient Exploration: The action space is severely restricted (coarse discretization), limiting the agent's ability to navigate efficiently and discover objects in complex environments.
3. Lack of Adaptability: Navigation is not treated as an explicitly optimizable component that can adapt to utility signals or safety constraints.

The goal is to modernize the navigation component to maximize SSG completeness (Node Recall) while ensuring execution safety (collision-free motion) under a fixed step budget.

2. Methodology

The authors propose a modular navigation framework that isolates the navigation policy from the downstream SSG construction. The methodology focuses on three main areas:

A. Policy Optimization Algorithm

• Replacement of REINFORCE: The study replaces the unstable REINFORCE algorithm with Proximal Policy Optimization (PPO). This change aims to improve training stability, sample efficiency, and generalization without altering the reward shaping function.

B. Action Space Design

The paper compares different formulations of the discrete action space to determine the optimal balance between resolution and learnability:

1. Single-Head (SH) Atomic Actions: The policy selects a single action ID from a large set.
   • Compact: 16 actions (8 rotations × 2 step lengths).
   • High-Resolution: 504 actions (24 rotations × 21 step lengths, allowing longer translations up to 2.0m).
2. Multi-Head (MH) Factorized Actions: The policy outputs three independent heads:
   • Rotation angle (24 options).
   • Step length (21 options).
   • Stop decision (binary).
   • Benefit: This factorization allows the agent to learn motion components (rotation vs. translation) separately, potentially improving controllability in large action spaces.

C. State Representation & Reward Shaping

• Multimodal Input: The agent observes RGB images (and optionally depth maps), the previous action, and two graph representations:
  • Local SSG (LSSG): Objects visible in the current view.
  • Global SSG (GSSG): The accumulated graph over the episode.
• Reward Shaping: A dense reward function is used, combining:
  • Potential-based shaping: Based on node/edge recall and viewpoint diversity.
  • Event-based terms: Rewards for discovering new objects, penalties for collisions, and rewards for timely stopping.
• Auxiliary Supervision: An optional collision prediction head is added when depth inputs are available to provide dense supervision for safety.

D. Curriculum Learning (CL)

The study evaluates whether gradually expanding the action space (from 16 to 504 actions) helps stabilize learning in high-resolution settings.

3. Key Contributions

1. Algorithmic Modernization: Demonstrates that switching from REINFORCE to PPO significantly improves SSG completeness (by ~21% relative to baseline) and training stability under identical reward shaping.
2. Action Space Characterization: Analyzes the trade-offs between compact and high-resolution action spaces. It finds that while larger action spaces offer higher potential completeness, they are harder to learn without factorization.
3. Factorized Control Superiority: Shows that a factorized Multi-Head (MH) policy outperforms an equivalent Single-Head (SH) high-resolution policy. MH achieves higher completeness with better safety and faster convergence by decomposing the decision process.
4. Safety vs. Completeness Trade-offs:
   • Curriculum Learning: In MH settings, CL improves execution safety (Move Success Rate) early in training but slows down convergence, offering a speed-safety trade-off.
   • Depth Input: Depth significantly improves safety in compact action spaces but offers marginal benefits in high-resolution factorized settings.

4. Experimental Results

The experiments were conducted in the AI2-THOR simulator using FloorPlans 1–27 for training and 28–30 for evaluation.

• Completeness (Node Recall):
  • REINFORCE baselines remained low and flat.
  • PPO (Compact SH16) showed rapid improvement.
  • PPO (Factorized MH504) achieved the highest Node Recall, outperforming the atomic high-resolution (SH504) variant.
• Safety (Move Success Rate):
  • REINFORCE had very low safety.
  • PPO + Depth significantly improved safety in compact settings.
  • MH + Curriculum Learning achieved the highest safety rates, preventing collisions more effectively than atomic high-resolution policies.
• Efficiency:
  • MH policies produced more spatially distributed trajectories with frequent direction changes, leading to broader coverage.
  • SH504 policies tended to move in straight lines (corridor-like), often failing to utilize the full action budget effectively.
• Key Finding: Replacing the optimizer (PPO) alone yielded a 21% relative improvement in completeness. Combining PPO with a factorized, high-resolution action space yielded the strongest overall trade-off between completeness and efficiency.

5. Significance and Implications

• Organic Computing (OC): The work validates the OC principle of treating navigation as a self-optimizing component. By maximizing a utility signal (SSG completeness) under resource constraints, the agent autonomously adapts its behavior.
• Practical Deployment: The findings suggest that for embodied AI systems, factorized action spaces combined with PPO are superior to monolithic high-resolution action sets. This architecture is more robust, safer, and easier to train.
• Future Directions: The paper lays the groundwork for integrating these navigation modules into full end-to-end ESSG pipelines with realistic perception (handling noise and detection errors) and embedding them into higher-level self-healing and self-configuration loops.

In summary, the paper successfully modernizes the ESSG navigation problem by moving away from unstable, coarse-grained RL approaches toward stable, factorized, and high-resolution control strategies, significantly enhancing the quality of semantic world models generated by embodied agents.