Mission-Aligned Learning-Informed Control of Autonomous Systems: Formulation and Foundations

1. Problem Statement

The paper addresses the critical challenge of deploying autonomous agents in safety-critical, human-centric environments (specifically robotic care for the elderly). Current state-of-the-art approaches, particularly Deep Reinforcement Learning (RL), suffer from three fundamental limitations in this context:

• Lack of Safety Guarantees: RL is a statistical model that lacks structural incorporation of physical laws, making it difficult to provide formal guarantees against catastrophic failures (e.g., collisions).
• Black-Box Nature: RL policies are often opaque, lacking interpretability required for user trust and regulatory certification.
• Poor Generalization: Standard RL struggles with out-of-distribution scenarios and requires extensive trial-and-error exploration, which is unacceptable in single-life settings (e.g., patient care).

The authors propose a solution that moves away from pure end-to-end RL toward a hybrid architecture that integrates Reinforcement Learning (RL), Model Predictive Control (MPC), and Classical Planning. The goal is to create a system that learns from data to adapt to user preferences while maintaining the safety, interpretability, and physical constraints of model-based control.

2. Methodology: A Bilevel Hierarchical Framework

The core contribution is a bilevel optimization framework that decouples decision-making into two interacting layers:

A. Upper Level: Mission-Aligned Planning (Symbolic Layer)

• Function: Handles high-level, discrete decision-making (e.g., "feed the patient," "deliver medication").
• Mechanism: Uses Classical Planning (based on relational logic and PDDL) to generate sequences of tokenized actions.
• Learning: Employs RL to learn a preference vector ($w$). This vector scalarizes a multi-objective cost function (e.g., balancing time, safety, energy, and user comfort) based on human feedback.
• Interpretability: Actions are logical and semantically meaningful to humans, ensuring the "mission" is clear.

B. Lower Level: Safe Physical Control (Continuous Layer)

• Function: Executes the physical movements required to achieve the high-level actions.
• Mechanism: Uses Model Predictive Control (MPC). The MPC solves a constrained nonlinear optimization problem at a fast time scale to generate continuous control inputs (velocity, torque).
• Safety: MPC explicitly enforces physical constraints (collision avoidance, dynamics) using known physics models, guaranteeing safe trajectories.
• Adaptation: The MPC parameters (cost matrices $Q, R$, target states) are not fixed; they are dynamically adjusted by the Upper Level based on learned preferences.

C. The Integration Mechanism: "Differentiation through Fuzzification"

The novel link between the discrete symbolic layer and the continuous control layer is a Fuzzy Logic Interface:

1. Fuzzification: Continuous physical states (e.g., robot position $z$) are mapped to fuzzy membership values ($\mu \in [0,1]$) representing logical atoms (e.g., "Robot is at Location A").
2. Differentiability: These membership functions are designed to be differentiable.
3. Gradient Propagation: The system uses Implicit Function Theorem (IFT) sensitivities to compute gradients of the MPC solution with respect to its parameters.
4. End-to-End Learning: Gradients flow from the human feedback (Upper Level) $\to$ through the fuzzy interface $\to$ into the MPC parameters (Lower Level). This allows the system to learn how to tune the physical controller to satisfy high-level mission goals without breaking safety constraints.

3. Key Contributions

1. Bilevel Optimization Formulation: A formal mathematical schema integrating RL-assisted MPC (lower level) and RL-assisted Classical Planning (upper level), solving the dependency between discrete planning and continuous control.
2. Differentiable Fuzzy Interface: A methodology to bridge symbolic logic and continuous control using fuzzy membership functions, enabling gradient-based learning across the symbolic-continuous boundary.
3. Safe Exploration via RLMPC: The framework embeds MPC within the RL loop, ensuring that even during the learning/exploration phase, the system operates within physically safe constraints (unlike pure RL which may explore unsafe states).
4. Task-Agnostic Adaptation: The architecture allows for rapid adaptation to new tasks (e.g., switching from medication delivery to meal prep) by redefining the planning problem while reusing the learned physical dynamics and user preference models.

4. Experimental Results

The authors validated the framework in a simulated hospital environment with a differential-drive robot performing two distinct tasks: Medication Delivery and Meal Preparation.

• Setup: The system interacted with five distinct "patient profiles," each with unique ground-truth preferences (e.g., "Safety-First," "Speed-Oriented," "Presentation-Focused").
• Convergence:
  • The system successfully learned the correct preference weights for 22 out of 25 runs.
  • Profiles with sharp preference peaks (e.g., Safety-First) converged rapidly (within 10–15 episodes).
  • The system correctly identified the dominant preference dimension in 100% of runs, even when full numerical convergence was not achieved.
• Behavioral Adaptation:
  • Safety-First profiles resulted in routes avoiding high-risk zones (e.g., the stove area), even if longer.
  • Speed-Oriented profiles selected the shortest paths and fastest meal types (sandwiches).
  • Presentation-Focused profiles selected elaborate meal types (full meals) to maximize "approach quality."
• Execution Quality: The MPC maintained a 100% task completion rate with no safety violations, demonstrating that the learning process did not compromise physical safety.

5. Significance and Impact

This paper provides a foundational blueprint for Trustworthy Autonomous Systems in sensitive domains:

• Bridging the Gap: It effectively bridges the gap between the flexibility of data-driven RL and the rigor of model-based control (MPC) and logic-based planning.
• Regulatory Viability: By ensuring safety through MPC constraints and interpretability through symbolic planning, the framework addresses the "black box" barrier that currently prevents widespread adoption of AI in healthcare and critical infrastructure.
• Scalability: The approach allows for the reuse of physical dynamics models across different tasks, reducing the computational and data burden compared to training separate end-to-end RL agents for every new scenario.
• Future Direction: It establishes a research program for "Learning-Informed Control," suggesting that future autonomous systems should not rely solely on statistical learning but must be grounded in first-principles physics and logic, with learning serving to tune parameters and align with human values.