Technical Summary: Cross-Platform Control for Autonomous Surface Vehicles via Adaptive Reinforcement Learning

1. Problem Statement

Autonomous Surface Vehicles (ASVs) exhibit significant diversity in hydrodynamic and actuation characteristics. Current control strategies face a deployment bottleneck:

• Model-based controllers (e.g., MPC) require accurate system identification and platform-specific tuning, which is labor-intensive for heterogeneous fleets.
• Standard Reinforcement Learning (RL) approaches typically train on a single, fixed platform or a simulator that must accurately capture the target's specific dynamics. Deploying these policies to a new vessel usually necessitates re-identification and retraining.

The core challenge is to develop a single, generalist controller capable of zero-shot deployment across different ASV platforms of the same class without prior knowledge of the deployment platform's dynamics or per-platform fine-tuning. This is framed as a partial observability problem, where the true system dynamics ($\theta$) are unknown at deployment time, making the state partially observable despite the Markovian nature of the underlying physics.

2. Methodology

The authors propose an adaptive reinforcement learning framework utilizing a Teacher–Student architecture to infer latent dynamics from interaction history.

A. System Dynamics and Formulation

• Model: The system uses a simplified 3-DoF (surge, sway, yaw) horizontal-plane model based on Fossen's formulation. The dynamics are parameterized by seven physical parameters ($\theta$), including mass, added-mass coefficients, linear damping, and inertia.
• MDP Formulation: Trajectory tracking is formulated as a Partially Observable Markov Decision Process (POMDP). The state $s_t$ is observable, but the dynamics parameter $\theta$ is not. The policy conditions on interaction history to restore approximate Markovianity.
• Action Space: The policy outputs normalized body-frame force and torque commands ($\bar{\tau} \in [-1, 1]^3$). These are scaled by platform-specific actuator limits at deployment, decoupling the policy from absolute force magnitudes.

B. Teacher–Student Architecture

The approach employs a two-phase training pipeline inspired by privileged learning:

1. Phase 1: Teacher Training (Privileged Learning)

   • A "Teacher" policy is trained in simulation with access to a privileged vector $e \in \mathbb{R}^9$. This vector encodes the normalized dynamics parameters (peak accelerations, damping rates, and Coriolis coupling coefficients) derived from the 3-DoF equations of motion.
   • The teacher learns a latent representation $z = \mu(e)$ of the system dynamics.
   • Training utilizes Proximal Policy Optimization (PPO) with domain randomization over a wide range of dynamics parameters.

2. Phase 2: Student Training (Adaptation)

   • The privileged vector $e$ is removed. The encoder $\mu$ is replaced by an Adapter module $\phi$.
   • The Adapter, implemented as a Gated Recurrent Unit (GRU) followed by an MLP, estimates the latent representation $\hat{z}$ solely from the interaction history $h_t$ (recent state differences, velocities, and normalized actions).
   • The Student policy (the frozen Teacher's actor) is conditioned on the Adapter's estimate $\hat{z}$.
   • The Adapter is trained via supervised learning to minimize the reconstruction error $\|\hat{z} - z\|^2$, where $z$ is the ground truth latent from Phase 1.

C. Training Setup

• Simulation: Training relies on a lightweight analytical dynamics model rather than a high-fidelity hydrodynamic simulator.
• Curriculum: A progressive curriculum increases difficulty by expanding the range of randomized dynamics parameters and trajectory complexity (from straight lines to tight turns).
• Disturbances: Environmental disturbances (currents, wind, waves) are modeled as Ornstein–Uhlenbeck processes. Actuator delays are randomized.
• Reward: A composite reward function includes terms for position, heading, and velocity tracking, alongside auxiliary bonuses for forward motion and penalties for lateral drift, excessive yaw, and action magnitude/smoothness.

3. Key Contributions

1. Zero-Shot Cross-Platform Policy: The first demonstration of a single RL policy deployed zero-shot across multiple distinct ASV platforms (same class, different dynamics) without retraining or system identification.
2. Teacher–Student Adaptation: Adaptation of the legged locomotion teacher–student framework to marine robotics, enabling the inference of latent dynamics from interaction history alone.
3. Simplified Training Pipeline: The method achieves high performance using only a simple 3-DoF analytical model, avoiding the computational cost and complexity of high-fidelity hydrodynamic simulators.
4. Interpretability: The learned latent space effectively captures the relevant dynamics, with principal component analysis showing that the 9-dimensional latent occupies a low-dimensional subspace.

4. Experimental Results

A. Real-World Experiments

Experiments were conducted on two distinct physical platforms (Platform A and Platform B) with different masses, inertias, and actuation limits.

• Performance: The adaptive Student policy achieved a Position Mean Absolute Error (MAE) of 4.0 cm on both platforms.
• Comparison:
  • Outperformed the non-adaptive general PPO baseline by up to 58% (Platform B: 4.0 cm vs. 9.5 cm).
  • Outperformed the platform-specific PPO baseline (trained only on one platform) in position error.
  • Approached the tracking accuracy of a platform-specific tuned Model Predictive Controller (MPC) (Student: 4.0 cm vs. MPC: 4.1–4.6 cm).
• Heading: The Student achieved comparable or better heading accuracy than the non-adaptive baselines.

B. Simulation Experiments

Simulations across five platforms (including Roboat variants) under ideal conditions confirmed the real-world findings:

• The Student reduced position error by 44% and heading error by 55% compared to the non-adaptive general PPO.
• The Student performed nearly identically to the "Teacher" (oracle) policy, validating that the Adapter successfully recovers the necessary latent information.
• Ablation studies showed that replacing the Adapter's output with a static mean latent ("Student avg. z") more than doubled the tracking error, confirming the necessity of online adaptation.

5. Significance and Claims

The paper claims that this work provides a practical recipe for generalist ASV control. By conditioning on interaction history, a single policy can generalize across a fleet of heterogeneous vessels without the engineering overhead of per-platform tuning or identification.

• Efficiency: The entire training pipeline (Teacher and Student phases) completes in approximately 30 minutes on a single consumer GPU (NVIDIA RTX 4090).
• Scalability: The approach removes the "per-platform bottleneck," allowing for the deployment of heterogeneous fleets using a unified control strategy.
• Modesty on Limitations: The authors acknowledge that the current 3-DoF linear-damping model is valid for low-speed displacement regimes. The adapter handles velocity-dependent effects (like quadratic drag) by re-estimating effective damping within the randomized range, but the approach does not yet cover high-speed planing or strong heave/roll/pitch coupling. Future work is suggested for online continual adaptation and tighter integration with model-based control objectives.