Situationally-Aware Dynamics Learning

1. Problem Statement

Autonomous robots operating in complex, unstructured environments (e.g., off-road terrain, changing weather) face significant challenges due to latent, unobserved factors. These factors—such as terrain friction, payload shifts, wind gusts, or sensor biases—obscure the robot's understanding of its internal state and the external world.

• The Core Issue: When a robot relies only on partial observable state information, the same control action can yield drastically different outcomes depending on the hidden context (e.g., driving at the same speed on concrete vs. mud).
• Limitations of Current Methods:
  • Traditional Model-Based Reinforcement Learning (MBRL) often assumes fully observable states or requires privileged information (ground truth) available only in simulation.
  • Existing methods for learning auxiliary representations (like latent variables) often rely on multi-stage training, supervised learning, or large datasets, making them inefficient for real-time, online adaptation.
  • Physics-based models often fail to account for unmodeled, context-dependent dynamics.

The paper aims to solve this by enabling robots to learn online, unsupervised representations of hidden states to adapt to uncertain and dynamic conditions in real-time.

2. Methodology

The authors propose a framework based on Generalized Hidden Parameter Markov Decision Processes (GHP-MDPs). The core idea is to model the "situation" of the robot as the joint distribution of state transitions, which captures the influence of latent factors.

A. Theoretical Framework: GHP-MDP

The problem is formalized as a GHP-MDP where the transition function $T(s, \theta, a, s')$ depends on:

• $s$: Observable state.
• $a$: Action.
• $s'$: Next state.
• $\theta$: A latent parameter representing the current "situation" (the underlying data generating process).

Unlike standard Hidden Parameter MDPs (HiP-MDPs) which assume the agent knows what variables are missing, this framework infers the latent factors purely from interaction data without prior knowledge of the specific missing variables.

B. Online Situation Discovery (Multivariate BOCD)

To identify the current situation $\theta$ in real-time, the authors extend Bayesian Online Changepoint Detection (BOCD) to a multivariate setting:

1. Modeling Transitions: The robot models the joint distribution of $(s_t, a_t, s_{t+1})$ tuples as a multivariate Gaussian distribution parameterized by mean $\mu$ and precision matrix $\Lambda$.
2. Changepoint Detection: The algorithm monitors the data stream to detect when the underlying distribution changes (indicating a new situation, e.g., moving from grass to mud).
3. Probabilistic Updates: Using a Normal-Wishart prior, the system recursively updates the parameters of the current distribution. It calculates:
   • Growth Probability: The likelihood that the current run (segment of data) continues.
   • Changepoint Probability: The likelihood that a new underlying process has started.
4. Symbolic Representation: Once a situation is identified, its distribution parameters are mapped to a symbolic representation $\theta$ using the Moment Generating Function (MGF). This symbol is then fed into the dynamics model.

C. Adaptive Dynamics Learning & Control

• Dynamics Model: An ensemble of probabilistic neural networks (Gaussian MLPs) predicts the next state. The input is augmented with the current situation symbol $\theta$.
• Training: The model is trained online using a weighted Negative Log-Likelihood (NLL) loss to handle imbalanced data across different situations.
• Planning: The system uses Model Predictive Path Integral (MPPI) control. The planner generates action sequences based on the current situation-aware dynamics model, allowing it to anticipate how the current environment will affect future states.

3. Key Contributions

1. Novel Framework: Introduction of an online, unsupervised method for learning hidden state representations using a multivariate extension of BOCD. This allows robots to detect and adapt to latent factors without privileged information.
2. Symbolic Situation Representation: A method to map complex, learned transition distributions into symbolic representations (via MGF) that can be explicitly used to augment the transition model, bridging the gap between probabilistic learning and symbolic reasoning.
3. Data Efficiency & Robustness: The approach significantly improves data efficiency and policy performance compared to state-of-the-art MBRL and Model-Free RL baselines, particularly in scenarios with unmodeled dynamics.
4. Emergent Safe Behaviors: The system naturally learns adaptive behaviors (e.g., slowing down on slippery terrain, backing up when stuck) without explicit programming, driven by the situation-aware planner.

4. Experimental Results

The framework was validated on two distinct platforms: an Unmanned Ground Vehicle (UGV) and a Quadrotor.

A. Unmanned Ground Vehicle (UGV)

• Setup: Navigation in unstructured terrains (simulation and real-world) involving diverse surfaces (mud, grass, concrete, mulch) and steep slopes.
• Baselines: Compared against Context-aware Dynamics Models (CaDM), PE+LSTM, TD-MPC2, PETS, Model-Free RL (PPO, SAC, TD3), and a physics-based Dubins model.
• Key Findings:
  • Performance: The proposed method achieved the highest success rate (100% in real-world trials) and reached waypoints faster than all baselines.
  • Data Efficiency: It learned effective policies in fewer episodes than other MBRL methods.
  • Safety: In a challenging steep descent with hidden bumps, the Dubins baseline caused the robot to flip, while the proposed method successfully navigated the terrain by adapting its velocity and trajectory.
  • Emergent Behaviors: The robot autonomously developed strategies like "backing up" to overcome obstacles and "velocity modulation" (slowing down on slippery slopes).

B. Quadrotor

• Setup: Flying through gates in a simulator with randomly switching wind fields (thermal updrafts, wind gradients).
• Baselines: Compared against a standard Probabilistic Ensemble (PETS) without situation awareness.
• Key Findings:
  • Data Efficiency: With only 100k transitions, the situation-aware model (SA-PETS) outperformed the baseline by 34%. With 5M transitions, the gap widened to 43.6%.
  • Robustness: In complex wind environments, the baseline model's performance degraded significantly as more data (and thus more latent factors) was introduced, whereas the SA-PETS model remained robust, demonstrating its ability to disentangle hidden factors.

5. Significance and Impact

• Bridging the Sim-to-Real Gap: The method allows robots to adapt to real-world dynamics without needing extensive retraining or privileged simulation data.
• Safety-Critical Autonomy: By explicitly modeling uncertainty and latent factors, the system generates safer control policies, reducing the risk of catastrophic failures in unstructured environments.
• Interpretability: Unlike "black-box" latent spaces, the situation representations are derived from the statistical properties of the transition dynamics, offering a degree of interpretability regarding why the robot is adapting (e.g., detecting a change in surface friction).
• Generalizability: The framework is platform-agnostic, demonstrated successfully on ground (UGV), aerial (quadrotor), and aquatic (simulated boat in appendix) platforms.

In conclusion, this paper presents a significant step forward in making autonomous robots more resilient to the "unknown unknowns" of real-world environments by enabling them to learn and reason about their operational context online and unsupervised.