C-STEP: Continuous Space-Time Empowerment for Physics-informed Safe Reinforcement Learning of Mobile Agents

1. Problem Statement

Safe navigation in complex, dynamic environments remains a significant challenge for Reinforcement Learning (RL) in robotics. While existing methods utilize Control Barrier Functions (CBFs), risk-avoidance strategies, or robustness prioritization, they often struggle with:

• Continuous Space-Time Dynamics: Traditional information-theoretic measures (like empowerment) are often designed for discrete-time, stochastic systems and fail in deterministic continuous domains where conditional entropy becomes infinite (due to perfect predictability).
• Reward Design Limitations: Current safe RL approaches often rely on additive penalty terms for collisions or hard constraints. These can be brittle, difficult to tune, and may not inherently encourage the agent to learn "safe" behaviors that maximize future maneuverability.
• Lack of Physics-Informed Intrinsic Motivation: Many methods do not fully leverage the agent's internal states (e.g., velocity) and forward dynamics to assess safety proactively.

2. Methodology: C-STEP Framework

The authors propose C-STEP (Continuous Space-Time Empowerment), a novel framework that redefines empowerment for deterministic, continuous-time dynamical systems to serve as an intrinsic safety reward.

A. Rethinking Empowerment for Deterministic Systems

• Theoretical Shift: In standard definitions, empowerment is the channel capacity from actions to future states. In deterministic continuous systems, this capacity is infinite because the next state is perfectly predictable.
• New Definition: The authors redefine empowerment for deterministic agents as the differential entropy of the reachable state distribution.
  • Mathematically, for a state $x$, empowerment $E(x)$ is the logarithm of the volume of the set of reachable states within a time horizon $T$ that remain collision-free ($X_{free}$).
  • $E(x) = \log(\lambda(R_{T,free}(x)))$, where $\lambda$ is the Lebesgue measure (volume).
  • This transforms empowerment from a measure of information transfer to a measure of maneuverability and safety margin.

B. The C-STEP Reward Function

The core contribution is a multiplicative reward function that integrates this safety measure with standard task rewards:
$$r(x) = r_d(x) \cdot \log(c \cdot \lambda(R_{T,free}(x)))$$

• $r_d(x)$: The standard task reward (e.g., distance to goal).
• $c$: A positive safety coefficient that scales the influence of the empowerment term.
• Mechanism: By multiplying the task reward by the log-volume of the reachable safe set, the agent is intrinsically motivated to stay in states where it has the most future options (large reachable volume), effectively avoiding narrow passages or high-speed approaches that lead to collisions.

C. Sampling-Based Approximation (Algorithm 1)

Calculating the exact volume of reachable sets is often intractable. The authors propose a practical, sampling-based approximation:

1. Trajectory Sampling: Sample $N$ admissible control trajectories over a time horizon $T$.
2. Simulation: Propagate these trajectories using the system's forward dynamics.
3. Classification: Classify final states as either "safe" (collision-free) or "terminal" (collision).
4. Volume Estimation: Approximate the volume of the safe reachable set ($\hat{R}_T$) and the terminal set ($\hat{T}_T$) using convex hulls of the sampled endpoints.
5. Calculation: Compute the approximation $\hat{E}_T(x) = \log(\lambda(\hat{R}_T) - \lambda(\hat{T}_T))$.

Key Features:

• Physics-Informed: Uses the agent's internal states (velocity, mass, etc.) and forward dynamics.
• Environment Agnostic: Does not require a global map or prior obstacle knowledge; it only requires a simulator to detect collisions.
• Interpretable: High empowerment corresponds to high safety margins.

3. Key Contributions

1. Novel Definition of Empowerment: A reformulation of empowerment tailored for deterministic, continuous-time dynamical systems, resolving the issue of infinite entropy.
2. C-STEP Framework: A physics-informed intrinsic reward design that augments existing navigation rewards rather than replacing them, promoting safety through the maximization of future maneuverability.
3. Sampling-Based Algorithm: A practical method to approximate CST-Empowerment without requiring differentiable transition functions or explicit safe set characterizations.
4. Integration Capability: The method is designed to work on top of standard RL algorithms (like PPO) and can be combined with other safe RL paradigms (e.g., CBFs).

4. Experimental Results

The authors evaluated C-STEP using Proximal Policy Optimization (PPO) in 2D and 3D environments.

• Point Maze (2D):

  • Scenario: A ball must navigate a maze with a narrow (fast/risky) and wide (slow/safe) path.
  • Result: Unempowered agents learned to take the risky narrow path, resulting in frequent collisions. Empowered agents learned to prioritize the wider, safer path, demonstrating a behavioral shift toward safety without explicit collision penalties during training.

• Drone Navigation (3D PyBullet):

  • Scenario: A drone navigates dynamic environments with randomized obstacle widths.
  • Success Rates:
    • Unempowered baseline: 82% success rate.
    • Empowered ($c=0.5$): 99% success rate.
    • Empowered ($c=1$): 92% success rate.
  • Safety Metrics: Empowered agents spent significantly less time in proximity to obstacles. For example, the agent with $c=1$ spent 72.6% less time within 0.1m of an obstacle compared to the baseline.
  • Efficiency: The safety improvements came with only a marginal increase in travel time (e.g., 18.4% slower for the $c=1$ case), indicating a highly efficient trade-off.

5. Significance and Conclusion

• Paradigm Shift: C-STEP moves away from "punishing" unsafe behavior (additive penalties) to "rewarding" safe maneuverability (multiplicative intrinsic motivation). This encourages emergent safe behaviors, such as maintaining distance from obstacles to preserve future options.
• Model Flexibility: Unlike many model-based safe RL methods, C-STEP does not require differentiable dynamics or explicit safe set definitions, making it applicable to a broader range of robotic systems where only a forward simulator is available.
• Interpretability: The safety coefficient $c$ provides a tunable knob for safety margins, and the empowerment value itself offers an interpretable metric of how "safe" a state is based on reachable volume.
• Future Work: The authors suggest combining C-STEP with formal methods like Control Barrier Functions (CBFs) to achieve both intrinsic motivation and hard safety guarantees, and extending the theory to fully stochastic systems.

In summary, C-STEP offers a robust, interpretable, and physics-informed approach to safe reinforcement learning that effectively balances task completion with collision avoidance in mobile robotic systems.