💻 computer science

DefVINS: Visual-Inertial Odometry for Deformable Scenes

This paper introduces DefVINS, the first visual-inertial odometry pipeline designed for deformable scenes that combines a rigid IMU-anchored state with a non-rigid embedded deformation graph, supported by a new real-world benchmark (VIMandala) and an observability analysis demonstrating its superior performance over existing rigid and non-rigid baselines.

Original authors: Samuel Cerezo, Javier Civera

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Samuel Cerezo, Javier Civera

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to walk through a crowded room while blindfolded, but you have a friend shouting directions to you every second. This is how most robots and phones figure out where they are: they use a camera (your eyes) and an accelerometer/gyroscope (your inner ear) to guess their position. This technology is called Visual-Inertial Odometry (VIO).

Usually, this works great because we assume the world is rigid—meaning walls, tables, and floors don't wiggle, stretch, or twist. Your brain (and the robot's software) assumes that if a chair looks bigger, you are just getting closer to it, not that the chair is inflating like a balloon.

The Problem: The "Wobbly World"
The paper introduces a new system called DefVINS to solve a specific problem: What happens when the world isn't rigid?

Imagine walking through a room filled with:

People waving their arms.
Clothes hanging on a line flapping in the wind.
A giant, wobbly jelly sculpture.

If a standard robot tries to navigate this, it gets confused. It thinks the waving arm is a wall moving toward it, or it thinks the flapping shirt is the floor rising up. It gets "drunk" on the movement, leading to a severe loss of direction (drift). It tries to force a rigid map onto a squishy world, and the math breaks.

The Solution: DefVINS
The authors created DefVINS, a new navigation system designed specifically for these "squishy" environments. Here is how it works, using simple analogies:

1. The "Two-Brain" Approach

Instead of trying to solve the whole puzzle with one brain, DefVINS splits the job into two parts:

The Rigid Brain (The Anchor): This part trusts the IMU (the inner ear sensor). It knows that gravity is always down and that the robot itself doesn't suddenly stretch or shrink. It acts like a heavy anchor, keeping the robot from floating away into nonsense.
The Wobbly Brain (The Deformation Graph): This part accepts that the environment is moving. Imagine the scene is a spiderweb. If you pull one strand, the whole web shifts. DefVINS builds a "spiderweb" of points in the scene. It allows these points to stretch and slide relative to each other, but it keeps them connected so the web doesn't fall apart.

2. The "Elastic and Viscous" Rules

To make sure the "wobbly brain" doesn't go crazy, the system uses two imaginary rules (constraints):

The Elastic Rule (Rubber Bands): Imagine connecting all the points in the scene with rubber bands. If you stretch them too far, the system says, "Whoa, that's too much!" It prevents the scene from stretching unrealistically.
The Viscous Rule (Honey): Imagine the points are moving through thick honey. If one point moves, its neighbors should move in a similar way. This stops the scene from looking like a chaotic explosion of pixels and keeps the motion smooth and natural.

3. The New "Gym" (VIMandala Dataset)

To test this, the authors couldn't just use normal videos because they needed to know exactly where the camera was. So, they created a new dataset called VIMandala.

The Analogy: Imagine filming a dancer spinning a giant, colorful, stretchy cloth (a mandala).
They recorded the video, the sensor data, and the exact true path of the camera. This is like having a "gold standard" answer key to see if their new robot navigation system is actually smart or just guessing.

Why This Matters

The paper proves that by combining the stability of the IMU (the anchor) with the flexibility of the deformation model (the spiderweb), the robot can navigate through chaotic, moving environments without getting lost.

Old Way: Tries to force a rigid map on a moving crowd. Result: The robot gets dizzy and crashes.
DefVINS: Realizes the crowd is moving, holds its own balance, and maps the movement of the crowd without losing its own way.

In a Nutshell:
DefVINS is like a dancer who can walk through a mosh pit. Instead of getting knocked over by the crowd (the deformation), they use their own balance (the IMU) to stay upright while gracefully stepping around the moving people (the non-rigid model). It's the first system to successfully do this for robots and AR devices.

1. Problem Statement

Classical Visual-Inertial Odometry (VIO) and SLAM systems rely fundamentally on the assumption of scene rigidity. In real-world scenarios involving deformable objects (e.g., human bodies, clothing, cables), this assumption is violated. When a rigid model is applied to deformable scenes, the system suffers from two primary failure modes:

Over-fitting: The rigid model attempts to fit local non-rigid motion, leading to incorrect pose estimates.
Severe Drift: When deformation-induced parallax dominates the inter-frame motion, the system loses track of the true camera trajectory, causing significant drift.

Furthermore, existing non-rigid visual odometry methods (purely visual) often lack metric scale and suffer from ill-conditioning because camera motion and scene deformation are strongly coupled. There is a critical lack of benchmarks and methods that jointly address inertial sensing and non-rigid deformation modeling.

2. Methodology: DefVINS

The authors propose DefVINS, the first VIO pipeline explicitly designed for deformable environments. The core innovation is the decomposition of the state into a rigid, IMU-anchored component and a non-rigid scene warp.

A. State Definition

The system operates on a sliding window of $N$ keyframes. The state vector $\xi$ includes:

Rigid State: Camera orientation ( $R$ ), velocity ( $v$ ), and position ( $t$ ) for each keyframe, plus IMU biases ( $b_g, b_a$ ) and gravity direction ( $\hat{g}$ ).
Non-Rigid State: A set of deformation nodes $\{x_i\}$ representing the 3D positions of tracked features that are allowed to move non-rigidly.

B. Optimization Framework

The system formulates a non-linear least-squares optimization problem minimizing a cost function $L$ composed of:

IMU Preintegration Residuals: Constraints on rotation, velocity, and position between keyframes based on IMU data.
Reprojection Error: Standard visual error between observed features and projected 3D points.
Non-Rigid Residuals ( $L_{nr}$ ): A novel term modeling the deformation graph with three specific constraints:
- Elastic Constraint: Penalizes excessive stretching or compression between connected nodes in the deformation graph, preserving local shape.
- Viscous Constraint: Encourages nearby nodes to move similarly over time, ensuring smooth and coherent temporal deformation.
- Photometric Constraint: Uses a semi-direct approach (Lucas-Kanade) to enforce brightness constancy, ensuring nodes move consistently with image texture flow.

The total cost function balances these terms using a scalar weight $\lambda_{nr}$ :
$L = \sum (L_{imu} + L_{rep} + \lambda_{nr} L_{nr}) + L_{prior}$

C. Observability Analysis

The paper provides a rigorous observability analysis of the visual-inertial deformable problem.

Key Finding: In purely visual non-rigid systems, camera motion and deformation are ambiguous (ill-posed).
Role of IMU: Inertial measurements anchor the rigid body dynamics, decoupling the global camera trajectory from the local scene deformation. This significantly improves the conditioning of the system, making the rigid state observable even when deformation is high.
Strategy: The authors derive a conditioning-based activation strategy to avoid ill-posed updates when motion excitation is poor.

3. Key Contributions

DefVINS Algorithm: The first VIO system that integrates an embedded deformation graph within a visual-inertial optimization framework, explicitly decoupling rigid IMU-anchored motion from non-rigid scene warping.
VIMandala Benchmark: The creation of the first real-world benchmark for deformable VIO. It consists of 7 synchronized RGB-D-IMU sequences capturing a deforming mandala cloth with ground-truth camera poses.
Synthetic Extension: Augmentation of the existing "Drunkard's" synthetic dataset with simulated IMU measurements to allow controlled evaluation of inertial effects.
Observability Analysis: A theoretical characterization showing how inertial constraints resolve the ambiguity between ego-motion and non-rigid deformation, proving that IMU data is essential for metric consistency in deformable scenes.

4. Experimental Results

The authors evaluated DefVINS against ORB-SLAM3 (rigid VIO baseline) and NR-SLAM (non-rigid visual baseline) on both synthetic and real datasets.

Synthetic (Drunkard's Dataset):
- DefVINS consistently outperformed baselines as deformation levels increased.
- In "Extreme" deformation scenarios, DefVINS reduced Absolute Trajectory Error (ATE) by ~40-45% compared to ORB-SLAM3 and ~50% compared to the visual-only variant.
- It maintained tracking on significantly more frames than rigid baselines, which failed to track under high deformation.
Real-World (VIMandala Dataset):
- In near-rigid scenarios, ORB-SLAM3 performed slightly better (due to mature engineering), but DefVINS remained competitive.
- As deformation increased (Sequences R4–R6), rigid models failed drastically (tracking coverage dropped below 20% in R6).
- DefVINS (Full) maintained high accuracy and tracked 85–95% of frames.
- Ablation Study: The full system (Inertial + Non-rigid) was necessary.
  - Visual-only (V-NR): Suffered from accumulated drift.
  - Rigid VIO (VI-R): Stabilized rotation but failed to model deformation, leading to large errors.
  - Full DefVINS: Achieved the best global consistency and lowest trajectory error.

5. Significance

This work addresses a critical gap in mobile robotics and Augmented Reality (AR). Most current VIO systems fail in dynamic, non-rigid environments (e.g., walking through a crowd, interacting with soft objects).

Robustness: DefVINS enables reliable localization in environments previously considered too dynamic for standard VIO.
Metric Consistency: By leveraging IMU data, the system maintains metric scale and prevents the "scale drift" common in purely visual non-rigid SLAM.
Foundation for Future Work: The introduction of the VIMandala benchmark and the theoretical observability analysis provide a foundation for future research in deformable scene understanding and state estimation.

In conclusion, DefVINS demonstrates that inertial sensing and non-rigid modeling are complementary: inertial data stabilizes the global trajectory, while non-rigid modeling captures local scene dynamics, together enabling robust odometry in highly deformable environments.