💻 computer science

Swim2Real: VLM-Guided System Identification for Sim-to-Real Transfer

Swim2Real introduces a novel pipeline that leverages vision-language model feedback and backtracking line search to automatically calibrate a 16-parameter robotic fish simulator directly from swimming videos, effectively closing the sim-to-real gap and enabling zero-shot reinforcement learning transfer to physical aquatic robots without manual system identification.

Original authors: Kevin Qiu, Kyle Walker, Mike Y. Michelis, Marek Cygan, Josie Hughes

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kevin Qiu, Kyle Walker, Mike Y. Michelis, Marek Cygan, Josie Hughes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot fish to swim in a real pool. You start by building a perfect digital twin of the fish in a computer game. But here's the problem: the computer fish swims differently than the real one. Maybe its tail wiggles too much, or it doesn't push enough water to move forward.

In the past, fixing this "digital vs. real" gap was like trying to tune a radio by guessing. Engineers had to manually break the problem into tiny pieces, use complex math for each piece, and hope they got it right. It was slow, tedious, and often failed.

Swim2Real is a new, smarter way to do this. Think of it as hiring a super-smart, visual detective (an AI called a Vision-Language Model) to watch the robot fish and fix the computer simulation automatically.

Here is how it works, broken down into simple steps:

1. The Detective (The VLM)

Instead of just looking at numbers, the system shows the AI two videos side-by-side: one of the real fish swimming and one of the computer fish swimming.

The Old Way: A computer would just say, "The error is 50 millimeters." That's like a teacher saying, "You got a C," without telling you why.
The Swim2Real Way: The AI detective watches the videos and says, "Hey, look! The computer fish's tail is bending way too sharply at the top, and it's not pushing hard enough at the bottom. It looks like the 'hinges' in its spine are too loose, and the water resistance is too high."

The AI doesn't just give a number; it gives a reasoned diagnosis based on physics, just like a human expert would.

2. The "Baby Steps" (Backtracking Line Search)

Here is the tricky part. The AI detective is great at figuring out what is wrong and which way to fix it, but it's sometimes bad at guessing how much to fix it.

The Problem: The AI might say, "We need to tighten the hinges!" and suggest tightening them by 100%. But if you tighten them that much, the fish might become stiff as a board and stop swimming.
The Solution: Swim2Real uses a "baby steps" strategy. It tries the AI's big suggestion first. If the fish swims worse, it doesn't give up. Instead, it tries half that amount. If that's still too much, it tries a quarter. It keeps shrinking the step until it finds the "Goldilocks" zone where the simulation finally matches the real fish.

This simple trick turned a system that only worked 14% of the time into one that works 42% of the time.

3. The Result: A Perfect Match

After about 40 tries (which takes less than 20 minutes), the system has tuned all 16 different settings of the computer fish (like how stiff the tail is, how the motor works, and how water pushes against it).

The computer fish now swims almost exactly like the real one.
It is 43% more accurate than the previous best methods.
It never fails completely (unlike other methods that sometimes crash and produce a robot fish that flops uselessly).

4. The Grand Finale: Real-World Swimming

Once the computer fish is perfectly calibrated, the system trains a "brain" (a Reinforcement Learning policy) on the computer fish to learn how to swim efficiently. Because the computer fish is so accurate, the brain learns the right lessons.

Then, they take that brain and put it on the real robot fish.

The Test: They send the real fish into a pool to swim to a target.
The Outcome: The real fish swims 12% farther than fish trained with older methods. It successfully navigates the water, proving that the "digital twin" was accurate enough to teach the real robot how to move.

Why This Matters

Think of this like teaching a child to ride a bike.

Old Method: You manually adjust the seat height, handlebar width, and wheel balance one by one, guessing each time.
Swim2Real Method: You put the child on a bike, watch them wobble, and an AI coach instantly says, "Your seat is too low, and you're pedaling too hard. Let's lower the seat a little bit and try again."

This paper shows that we can now use AI to automatically tune complex robots just by watching videos of them move. It removes the need for human experts to spend weeks tweaking settings, paving the way for robots that can be deployed in oceans, rivers, and disaster zones much faster and more reliably.

1. Problem Statement

The paper addresses the challenge of Sim-to-Real transfer for soft aquatic robots, specifically a tendon-driven robotic fish.

The Core Difficulty: Calibrating simulators for soft robots is difficult due to nonlinear fluid-structure coupling, chaotic parameter landscapes, and the complexity of fluid dynamics. Traditional methods often require manual decomposition of the problem into multiple stages (e.g., tuning stiffness via FFT, then geometry via grid search, then fluid via Bayesian Optimization), which is labor-intensive and prone to human bias.
The Limitation of Current Methods: Existing black-box optimization techniques (like Bayesian Optimization or CMA-ES) rely solely on scalar error metrics (e.g., trajectory distance). They ignore the rich visual and physical structure of the discrepancy, often leading to poor convergence or catastrophic failures in high-dimensional spaces (16 parameters in this case).
The Goal: To create an end-to-end, fully automatic pipeline that calibrates a 16-parameter simulator using only raw video of the real robot, without manual intervention, domain-specific search strategies, or hand-crafted cost functions.

2. Methodology: The Swim2Real Pipeline

The proposed system, Swim2Real, integrates a Vision-Language Model (VLM) with a backtracking line search to perform system identification.

A. Problem Formulation

The objective is to find a parameter vector $\theta$ (16 dimensions) that minimizes the L2 error between simulated and real marker trajectories across multiple actuation frequencies. The simulator uses MuJoCo with a stateless ellipsoid fluid model.

B. VLM-Guided Calibration

Instead of treating the simulator as a black box, the method uses a VLM (Gemini 2.5 Pro) as a "physics reasoner."

Input: The VLM receives side-by-side video overlays of simulated (blue) vs. real (red) markers, numerical error data, and the current parameter history.
Reasoning: The VLM analyzes the visual discrepancies (e.g., "tail bends too sharply," "insufficient thrust") and maps these physical observations to specific simulator parameters (e.g., hinge stiffness, fluid drag).
Output: The VLM proposes a new parameter vector $\theta'$ . Unlike scalar optimizers, this provides an interpretable, physically grounded direction for the update.

C. Backtracking Line Search

A critical component of the pipeline is the Backtracking Line Search, which addresses the VLM's tendency to identify the correct direction of improvement but overestimate the magnitude (step size).

Mechanism: Given the current best parameters $\theta_{best}$ and the VLM proposal $\theta'$ , the algorithm tests step sizes $\beta^k \cdot (\theta' - \theta_{best})$ where $\beta \in (0,1)$ .
Validation: It evaluates the objective function at decreasing step sizes. If the full step fails, it tries half, then quarter steps, until an improvement is found or the budget is exhausted.
Benefit: This recovers proposals where the direction is correct but the step is too large, significantly increasing the acceptance rate of VLM suggestions.

D. Parameter Space

The system simultaneously calibrates 16 parameters covering:

Fluid Coefficients (5): Drag and lift parameters for the ellipsoid fluid model.
Motor Geometry (1): Arm length of the crank-slider mechanism.
Mechanical Properties (10): Stiffness and damping for each of the 5 hinge joints in the tail.

3. Key Contributions

End-to-End Automated Pipeline: Swim2Real is the first system to calibrate a high-dimensional (16D) soft robot simulator directly from video using VLM feedback, eliminating the need for manual, multi-stage search strategies.
Backtracking Line Search Integration: The method introduces a line search that triples the VLM proposal acceptance rate (from 14% to 42%) by correcting magnitude errors while preserving the VLM's physical reasoning.
Comprehensive Validation: The pipeline is validated not just on calibration error, but on downstream Reinforcement Learning (RL) performance and real-world deployment, proving that the calibrated simulator is sufficient for zero-shot transfer.

4. Experimental Results

The authors compared Swim2Real against Bayesian Optimization (BayesOpt), CMA-ES, Random Search, and several ablations (e.g., without line search).

Calibration Accuracy:
- Swim2Real achieved the lowest mean marker error (51.3 mm) with the tightest variance (all 5 seeds within a 3 mm range).
- It outperformed BayesOpt (52.4 mm) and significantly outperformed CMA-ES, which suffered catastrophic failures on 2 of 5 seeds (errors > 120 mm).
- Velocity Match: Crucially, Swim2Real achieved the best forward swimming velocity match (MAE = 7.4 mm/s), which is 43% lower error than the next-best method (BayesOpt). This metric was not part of the training objective, proving the simulator captured true physical dynamics.
Sample Efficiency:
- The line search made each VLM call 2.5x more productive. Swim2Real reached optimal performance in ~16 VLM calls, whereas the version without line search required ~39 calls to reach similar accuracy.
Downstream RL & Real-World Deployment:
- Policies trained in the Swim2Real-calibrated simulator swam 12% farther than those trained with BayesOpt-calibrated simulators and 90% farther than CMA-ES.
- Real-World Test: Motor commands generated in simulation were deployed on the physical fish at 50 Hz. The robot successfully executed directed swimming and target-reaching tasks, confirming the sim-to-real gap was effectively closed.

5. Significance and Conclusion

Automated System Identification: Swim2Real demonstrates that VLMs can serve as effective "physics reasoners" for system identification, replacing the need for human experts to design complex, multi-stage calibration pipelines.
Interpretability: Unlike black-box optimizers, the VLM provides diagnostic feedback (e.g., identifying "locking up" behavior at high frequencies), allowing for physically intuitive corrections.
Generalizability: While tested on a robotic fish, the approach suggests a path toward automated simulator tuning for other underwater or soft robotic platforms where simulation-reality discrepancies are visually observable.
Limitations: The method currently relies on a proprietary VLM (Gemini) and has a computational overhead (VLM inference time) that triples wall-clock time compared to simulation-only baselines, though this is negligible compared to the time required for real-world data collection.

In summary, Swim2Real successfully bridges the sim-to-real gap for aquatic robots by combining the physical reasoning capabilities of VLMs with robust optimization techniques, enabling high-fidelity simulation and successful real-world deployment without manual system identification.