Technical Summary: Training Observable Control Policies to Expose Agent State Through Actions

Problem Statement

Effective collaboration among autonomous agents often relies on mutual state awareness, typically achieved through direct communication. However, physical or operational constraints (e.g., underwater environments, bandwidth limitations, or hardware incompatibility) frequently render communication unreliable or impossible. While Partially Observable Markov Decision Processes (POMDPs) offer a theoretical framework for handling such uncertainty, they suffer from scalability issues and require retraining all agents when task parameters change, limiting their practicality in federated systems.

An alternative approach involves "decoupling" agents by explicitly formulating an estimator to infer the state of other agents based on observable actions. While agents' control actions are often visible without explicit communication (e.g., an aircraft banking), standard control policies optimized solely for task performance often lack "observability." In such cases, different states may produce identical action sequences (e.g., a constant bank angle in a circular orbit), making state reconstruction impossible or prone to divergence. The core problem addressed is how to train a control policy that remains effective for its primary task while simultaneously generating actions that allow an external observer to accurately reconstruct the agent's state.

Methodology

The authors propose a framework where a control policy is trained using Reinforcement Learning (RL) with a reward function that explicitly incorporates the performance of a state estimator.

Agent and Environment Formulation

• Task: A fixed-wing Unmanned Aerial System (UAS) must track a ground target. The agent controls the aircraft's bank angle ($\phi$) to maintain proximity to the target.
• State: The agent's decision state consists of the relative position ($x, y$) and velocity ($v_x, v_y$) with respect to the target.
• Policy: A neural network maps the state to a pseudo-control distribution (Gaussian). The mean is output by the network, while the variance is fixed to ensure stable exploration. The sampled pseudo-control is mapped to a bank angle command via a saturation function.
• Estimator: An Unscented Kalman Filter (UKF) is employed to estimate the agent's state. The filter uses the agent's system dynamics and treats the agent's control policy as the measurement function. The "observation" is the sampled pseudo-control action ($\tilde{u}$) generated by the policy.

Reward Embedding

Two policies are trained and compared:

1. Task-Only Policy: Trained with a reward ($r_{task-only}$) that decays exponentially with the distance to the target.
2. Embedded Estimator Policy: Trained with an augmented reward ($r_{embedded}$) defined as:
   $$r = r_{task-only} + R(\hat{x}, x)$$
   where $R(\hat{x}, x)$ is a reward term promoting actions that minimize the error between the true state ($x$) and the estimated state ($\hat{x}$). This term is formulated as an exponential decay of the error norm weighted by a covariance matrix.

By embedding the estimator within the training loop, the RL agent is incentivized to select actions that not only achieve the tracking objective but also reduce the uncertainty of the state estimate, effectively optimizing for observability.

Observability Analysis

The authors analyze observability using two metrics:

1. Linearized Observability Matrix: Computed at each time step via Taylor series approximation to assess local observability.
2. Stripped Observability Matrix (SOM): Constructed by stacking linearized matrices over a time trajectory to assess dynamic observability. The singular values of these matrices indicate the degree to which the state space is observable.

Key Contributions

1. Estimator Implementation: The authors implement a state estimator that relies exclusively on the control policy's output (actions) as observations, without requiring explicit state telemetry.
2. RL for Observability: They demonstrate a method to improve the observability of control policies by embedding the estimator's performance directly into the RL reward function.
3. Performance Validation: Through Singular Value Decomposition (SVD) of observability matrices and Monte Carlo simulations, they show that the resulting policies significantly improve estimator performance with minimal degradation to the nominal tracking task.

Results

The study evaluated 14,500 simulation episodes comparing the "Task-Only" and "Embedded" policies.

• Estimator Performance: The embedded policy significantly outperformed the task-only policy.
  • Position Error: At the median performance level, the embedded policy reduced position error by a factor of approximately four compared to the task-only policy. In the 90th percentile, the embedded policy kept errors below 10m in 90% of cases, whereas the task-only policy achieved this in only 60%.
  • Divergence: The task-only policy exhibited a high rate of filter divergence (approx. 30% of episodes), where the estimate drifted significantly from the true state. The embedded policy showed negligible divergence.
  • Convergence: When excluding divergent cases, the embedded policy still demonstrated a 44.2% reduction in position error and a 32.1% reduction in velocity error compared to the task-only policy at the end of the simulation.
• Task Performance: The inclusion of the estimator reward resulted in a negligible "penalty" to the primary tracking task. The tracking reward for the embedded policy was only 1.2% lower than the task-only policy at the 80th percentile, indicating that observable policies can be achieved without sacrificing nominal task efficacy.
• Observability Metrics:
  • Local Observability: Neither policy was fully observable at any single point in the state space (smallest singular values were near zero).
  • Dynamic Observability: The SOM analysis revealed that the embedded policy generated trajectories with significantly larger singular values over time. Specifically, after approximately 50 seconds of data, the smallest singular value for the embedded case increased from 0 to 1, indicating the sequence became observable, whereas the task-only case remained less observable.

Significance and Claims

The paper claims that integrating an estimator into the training process allows for the generation of control policies that are "observable" by design. The primary significance lies in enabling coordination between autonomous agents in communication-limited scenarios. By making control actions informative about the agent's internal state, agents can coordinate based on observation alone.

The authors argue that the policy space is sufficiently complex that one can select a policy that satisfies both the task objective and the observability objective without significant trade-offs. This approach offers a "decoupled" alternative to fully integrated POMDPs, allowing agents to be trained and updated independently, provided their control policies are shared.

Furthermore, the work suggests that observable policies could enhance human-autonomous teaming. If autonomous systems act in ways that make their state and intent more transparent (observable) through their actions, human operators may better understand and anticipate system behavior, improving safety and comfort in mixed operations. The paper concludes that while it does not fully solve human-machine coordination, it establishes a foundational method for making state reconstruction from actions feasible and efficient.