⚡ electrical engineering

Differentiable Reinforcement Learning for Path Tracking by an Agile Fish-Like Robot

This paper addresses the challenges of controlling agile fish-like robots by introducing a computationally efficient simulation platform and a differentiable reinforcement learning approach that optimizes PID gains through backpropagation and curriculum training, successfully transferring the learned policy from simulation to a physical robot for accurate path tracking.

Original authors: Prashanth Chivkula, Kartik Loya, Venkata Ravindhra Reddy Varikuti, Phanindra Tallapragada

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Prashanth Chivkula, Kartik Loya, Venkata Ravindhra Reddy Varikuti, Phanindra Tallapragada

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots don't just roll on wheels or buzz on propellers, but swim with the fluid grace of a real fish. For decades, engineers have been trying to build these "bio-inspired" machines because fish are incredibly fast, efficient, and stealthy. They move by wiggling their bodies and interacting with the water in complex ways, creating invisible swirls called vortices that actually help push them forward. However, teaching a robot to do this is a nightmare. Water is messy, and the physics of how a robot's body pushes against the fluid is so complicated that writing a perfect math formula for it is nearly impossible. If you try to program a robot with rigid rules, it often ends up flailing like a fish out of water.

This is where a clever trick called "Reinforcement Learning" comes in. Think of it like training a dog, but instead of treats, the robot gets a score for doing well. The robot tries things, makes mistakes, and learns from the feedback to get better. But there's a catch: to learn effectively, the robot needs to practice millions of times. Doing this in a real swimming pool would take forever and might break the robot. So, scientists usually build a "simulator"—a video game version of the water. The problem is that most simulators are either too slow to train quickly or too simple to be accurate. This paper tackles that exact bottleneck: how to build a simulator that is both fast enough to learn from and accurate enough to work in the real world, all while teaching a robot to swim in a straight line and follow a winding path at the same time.

The Agile Swimmer and the Magic Simulator

Meet the "agile fish-like robot." It's not a robot with a propeller on the back; it's more like a mechanical eel with a secret weapon inside. Inside its sealed, hydrofoil-shaped head (about the size of a small tablet, 250 mm long), there is a spinning rotor. By spinning this internal weight back and forth, the robot creates a push against the water, propelling itself forward without any external propellers. It has a flexible tail and a little rudder to steer. The goal? To make this robot swim along a specific path, like a letter "S" or a circle, while hitting a specific speed, like a car on a highway.

The challenge is that the robot's movement is a delicate dance. If it turns too hard, it slows down. If it speeds up too much, it might lose its balance. Traditionally, engineers use "PID controllers" to manage this. You can think of a PID controller as a very strict coach who gives three types of instructions: "How far off are you?" (Proportional), "How long have you been off?" (Integral), and "How fast are you drifting off?" (Derivative). The coach adjusts the robot's steering and speed based on these numbers. The tricky part is that the "gains" (the strength of the coach's voice) need to change depending on how curvy the path is. A straight line needs a gentle coach; a sharp turn needs a loud, urgent one.

The "Differentiable" Breakthrough

The authors of this paper realized that manually tuning these coaches for every possible curve and speed is impossible. Instead, they decided to let the robot learn the perfect "coaching style" itself using a method called Differentiable Reinforcement Learning.

Here is the magic analogy: Imagine you are trying to learn to ride a bike down a hill. Usually, you just try, fall, and get back up. But what if you could rewind time, see exactly how your foot placement caused the fall, and instantly adjust your muscles to fix it? That is what "differentiable" means here. The authors built a special computer simulation of the robot and the water that is "differentiable." This means the computer can mathematically trace how a tiny change in the robot's steering or speed affects its final position, all the way back to the very first instruction.

Instead of just guessing, the computer runs a "backpropagation" (a fancy term for working backward from the mistake) through the entire simulation. It calculates exactly how to tweak the PID gains to make the robot swim better. They didn't just throw the robot into a random pool; they used a "curriculum." Just like a human student learns to walk before running, the robot started by swimming in straight lines at slow speeds. Once it got good, the simulation made the paths curvier and the speeds faster. This step-by-step training allowed the robot to master complex maneuvers without getting overwhelmed.

From Code to Water: The Real-World Test

The most exciting part of this paper is that they didn't stop at the computer screen. They took the "brain" they trained in the simulation and put it directly onto the physical robot.

In the lab, they set up a pool with a camera system to watch the robot swim. They asked the robot to trace the letters "I," "R," "O," and "S" at different speeds, ranging from 0.10 meters per second (a slow drift) to 0.25 meters per second (a brisk swim).

The results were impressive. The robot, guided by the AI-learned gains, successfully followed the paths.

The "I" path: A straight line. The robot swam straight and true.
The "R" and "S" paths: These had sharp curves. The robot adjusted its steering and speed automatically. When the path turned sharply, the AI learned to slightly relax the speed control to focus on turning, preventing the robot from spinning out.
The Speed: The robot managed to stay close to the target speeds, though it was slightly more accurate at higher speeds. At very slow speeds, it was a bit wobbly, which the authors explain is because steering corrections naturally slow the robot down, creating a trade-off between staying on the line and maintaining speed.

The data showed that the robot's "cross-track error" (how far it drifted off the line) was very small, often less than 10 centimeters, even on the tricky "S" shape. The robot's internal gains (the "coach's" settings) changed smoothly as the path changed, proving that the robot wasn't just memorizing a single trick but was actually learning a flexible strategy.

What This Means

This paper suggests that we don't need to perfectly understand every drop of water physics to build great swimming robots. Instead, by building a smart, fast simulator that can "learn from its mistakes" mathematically, we can train robots to handle the messy reality of water. The authors demonstrated that a policy learned in a computer simulation could be transferred directly to a physical robot, allowing it to swim agilely and track paths with high precision.

While the current robot only swims on a flat plane (2D), the success of this method suggests that the same approach could eventually be used to teach robots to dive, roll, and swim in 3D space. It's a step toward a future where underwater robots can navigate complex environments, inspect pipelines, or explore the ocean with the same natural ease as the fish that inspired them.

Technical Summary: Differentiable Reinforcement Learning for Path Tracking by an Agile Fish-Like Robot

1. Problem Statement

The control and motion planning of bio-inspired fish-like robots have historically been hindered by two primary challenges: the difficulty of accurately modeling complex fluid-structure interactions and the nonlinear, underactuated dynamics of such systems. While Reinforcement Learning (RL) has advanced control for ground and aerial robots, its application to fish-like robots has been limited by the lack of simulation environments that balance computational speed with accuracy. Furthermore, existing data-driven approaches for fish robots often operate in low-speed regimes (less than a fraction of a body length per second) where hydrodynamic complexity is reduced, and they typically focus on tracking a reference heading rather than simultaneously controlling both speed and path.

This paper addresses the challenge of controlling the speed and heading angle of an agile, planar fish-like robot to track curved paths at varying reference speeds. The specific difficulty lies in the coupled nature of propulsion (generated by an internal rotor) and steering (via a rudder), where the dynamics are governed by limit-cycle behaviors that make manual PID tuning nontrivial.

2. Methodology

Robot Design and Modeling

The authors developed a modular, control-oriented experimental testbed: a fish-like robot with a rigid hydrofoil head and a flexible tail. Propulsion is achieved via an internal balanced rotor driven by a brushless motor, creating periodic inertial excitation, while steering is managed by a servo-actuated tail rudder.

To enable efficient training, the authors constructed a reduced-order mathematical model based on Chaplygin sleigh dynamics. This model:

Captures essential propulsion mechanisms and steady-state limit-cycle behavior without requiring computationally expensive fluid-structure simulations (e.g., CFD).
Models tail flexibility using torsional springs between links.
Incorporates hydrodynamic added mass and viscous drag via Rayleigh dissipation functions.
Enforces a non-holonomic constraint (zero lateral velocity at the tail end) to approximate the Kutta-Joukowski vortex shedding condition.

The model parameters were identified using experimental data from 12 trials, and the model was validated against physical robot data, showing strong agreement in limit-cycle behavior and steady-state velocity scaling.

Control Framework

The control objective is simultaneous forward velocity tracking and geometric path tracking. The authors employ a structured dual-loop PID controller:

Heading Loop: Regulates the rudder angle ( $\alpha$ ) based on heading error.
Speed Loop: Regulates the throttle variable ( $s$ , related to rotor amplitude and frequency) based on velocity error.

Instead of manually tuning PID gains, the gains are treated as decision variables to be optimized. The control policy is a neural network that outputs time-varying PID gains ( $K_p, K_i, K_d$ ) for both loops based on the robot's state (normalized heading, velocity, and tracking errors).

Differentiable Reinforcement Learning

The core methodological contribution is the use of Differentiable Reinforcement Learning via Backpropagation Through Time (BPTT).

Differentiable Simulator: The Chaplygin sleigh model is implemented as a differentiable simulator, allowing gradients to propagate directly through the system dynamics, the PID update laws, and the throttle mapping.
Training: The policy is trained in simulation using a curriculum learning approach. The curriculum progressively increases task difficulty by varying commanded speeds ($0.1$ to $0.3$ m/s) and path curvature (from sinusoidal to circular trajectories with decreasing radii).
Optimization: The objective is a finite-horizon optimal control problem minimizing a cost function that includes cross-track error, heading error, velocity error, and penalties for large or rapidly changing gains.

3. Key Contributions

Differentiable Simulation Platform: Development of a computationally efficient, differentiable simulator for a carangiform fish-like robot that approximates fluid-structure interactions via reduced-order mechanics, enabling gradient-based learning.
Gain-Learning PID Framework: A novel control architecture where a neural network learns to modulate the gains of a structured PID controller in real-time, adapting to path curvature and speed requirements.
Sim-to-Real Transfer: Successful deployment of the simulation-trained policy on a physical robot without further fine-tuning, demonstrating accurate path and speed tracking in real hydrodynamic conditions.
Curriculum Learning Strategy: A training regimen that systematically introduces speed and curvature variations to ensure the policy learns stable regulation before tackling complex maneuvers.

4. Results

Simulation Results

The policy converged rapidly, achieving a mean cross-track error of approximately 0.01 m across sinusoidal and circular trajectories.
The learned gains exhibited curvature-dependent modulation: as the robot entered higher-curvature segments, proportional and integral gains increased, while derivative gains slightly decreased to prevent excessive damping of necessary turning motions.
The controller maintained negligible heading error and successfully tracked reference speeds, though a small persistent velocity error (~0.05 m/s) was observed during high-speed circular turns, reflecting a trade-off between strict velocity tracking and path following due to hydrodynamic constraints.

Experimental Results

The policy was deployed on the physical robot to track "I", "R", "O", and "S" shaped trajectories at speeds ranging from 0.10 to 0.25 m/s.
Tracking Performance: The robot successfully followed all geometric shapes. Root-Mean-Square (RMS) cross-track errors ranged from 0.011 m to 0.122 m, with generally lower errors at higher speeds.
Gain Adaptation: Online monitoring confirmed that the policy dynamically adjusted yaw (steering) gains based on trajectory curvature, while velocity gains remained relatively stable.
Stability: The system demonstrated stable closed-loop behavior with bounded overshoots and rapid convergence to commanded speeds.

5. Significance and Claims

The paper claims that the proposed framework successfully bridges the gap between complex fluid-structure dynamics and efficient control learning. By utilizing a differentiable physics simulator, the authors enable the direct optimization of control policies via gradient descent, bypassing the need for hand-crafted models or slow, sample-inefficient RL methods.

The significance of the work lies in:

Demonstrating Sim-to-Real Transfer: Showing that a policy learned entirely in a reduced-order simulation can control a physical, agile swimming robot in real water.
Structured Control: Maintaining the interpretability of PID control while leveraging the adaptability of deep learning to tune gains online.
Agile Maneuvering: Achieving path tracking at speeds up to 2 body lengths per second (though experiments focused on 0.1–0.25 m/s), a regime where hydrodynamic effects are more complex than in previous low-speed studies.

The authors modestly note that the current results are limited to planar swimming and that future work will address three-dimensional motion control. They do not claim to have solved all fluid-structure interaction modeling issues but rather provided a practical, efficient approximation suitable for control-oriented learning.