💻 computer science

TwinTrack: Bridging Vision and Contact Physics for Real-Time Tracking of Unknown Objects in Contact-Rich Scenes

TwinTrack is a physics-aware perception system that achieves robust, real-time 6-DoF pose tracking of unknown dynamic objects in contact-rich scenes by integrating Real2Sim for joint geometry and physical property estimation with Sim2Real for adaptive fusion of visual and contact dynamics cues.

Original authors: Wen Yang, Zhixian Xie, Yiting Wang, Abhijit Tadepalli, Heni Ben Amor, Shan Lin, Wanxin Jin

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wen Yang, Zhixian Xie, Yiting Wang, Abhijit Tadepalli, Heni Ben Amor, Shan Lin, Wanxin Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to catch a slippery, invisible soap bar while it's bouncing off the walls of a shower. The water is spraying (motion blur), and your hands are blocking your view of the soap (occlusion). If you only rely on your eyes, you'll miss it every time. You need to use your intuition about how soap bounces and slides to guess where it will be next.

That is exactly what the paper TwinTrack does, but for robots.

Here is the story of how this system works, broken down into simple concepts:

The Big Problem: "Blind" Robots

Robots are great at seeing things when they are still and clearly visible. But when a robot tries to juggle, catch, or manipulate an object that is moving fast, hitting walls, or getting hidden behind its own fingers, standard cameras fail. The image gets blurry, or the object disappears completely.

If a robot relies only on its camera, it loses track of the object the moment it gets covered or moves too fast.

The Solution: TwinTrack

The researchers built a system called TwinTrack. Think of it as a robot with two superpowers working together:

The Eagle Eye (Vision): It sees what is there.
The Inner Sense (Physics): It "feels" how the object should move based on the laws of physics (gravity, bouncing, friction).

The magic happens because these two powers talk to each other constantly.

How It Works: The Two-Step Dance

The system has two main parts that work in a loop, like a teacher and a student.

1. The Teacher: Real2Sim (Learning the Rules)

The Job: This part works in the background (slower). It looks at the video footage and tries to build a perfect digital twin of the object.
The Problem: The camera view is messy. The object might look weirdly shaped because of shadows or missing parts.
The Fix: Real2Sim says, "Wait, if this object is a box, it shouldn't be sliding like a pancake. It must be heavier or have more friction."
The Analogy: Imagine you are trying to guess the shape of a mystery box by looking at it through a foggy window. It looks round. But then you hear it hit the floor and bounce high. You realize, "Oh, it can't be a heavy rock; it must be a hollow ball!"
The Result: Real2Sim updates the robot's internal model. It fixes the shape (geometry) and figures out the weight and bounciness (physics) so the simulation matches reality.

2. The Student: Sim2Real (The Real-Time Tracker)

The Job: This part runs super fast (20+ times a second) to tell the robot where the object is right now.
The Strategy: It uses two guesses:
1. The Visual Guess: "I see the object here."
2. The Physics Guess: "Based on how it was moving last second, it should be there."
The Magic: It blends these two guesses.
- If the camera sees the object clearly, it trusts the camera.
- If the object is hidden behind a finger or blurry, it leans heavily on the Physics Guess.
The Analogy: Imagine playing a game of "Hot and Cold" with a friend who is hiding a ball.
- When you can see the ball, you point directly at it.
- When the ball is hidden, you don't guess randomly. You remember, "My friend threw it to the left, and it hit the wall, so it must be bouncing back to the right." You use the physics of the throw to find it even when you can't see it.

Why Is This Special?

Most robots try to "see" their way through a problem. TwinTrack realizes that touching and hitting things gives you information too.

It fixes the "Blur": When an object moves so fast it becomes a blur, the camera fails. But the physics engine knows, "If it was moving that fast and hit that wall, it must be at this specific spot now."
It fixes the "Hiding": When a robot's hand covers the object, the camera sees nothing. TwinTrack uses the physics model to say, "The hand pushed it, so it's sliding under my palm."
It learns on the fly: It doesn't need to know the object beforehand. It figures out the weight and bounciness while it's watching the object fall and bounce.

The Bottom Line

TwinTrack is like giving a robot a "sixth sense." It combines what the robot sees with what the robot knows about how the world works. This allows the robot to catch, juggle, and manipulate objects in chaotic, messy environments where a normal camera would give up and say, "I lost it!"

It's the difference between trying to catch a ball in the dark by guessing, versus catching it by feeling the wind and knowing exactly how the ball was thrown.

1. Problem Statement

The paper addresses the challenge of real-time 6-DoF pose tracking for previously unseen, highly dynamic objects in contact-rich scenes (e.g., in-hand manipulation, object falling).

Core Challenges:
- Visual Degradation: Pure vision-based trackers fail under heavy occlusion, motion blur, and abrupt impacts common in contact-rich environments.
- Unknown Properties: The system must estimate not only the pose but also the object's geometry and physical properties (mass, inertia, friction) without prior CAD models.
- Non-Smooth Dynamics: Contact interactions introduce discontinuities that challenge gradient-based optimization and learning.
- Real-Time Constraints: The system must operate at high frequencies (>20 Hz) to be useful for robotic control.

2. Methodology: The TwinTrack Framework

TwinTrack is a physics-aware perception system that closes the loop between Real2Sim (backend) and Sim2Real (frontend). It takes monocular RGB-D images and robot proprioceptive data as input.

A. Real2Sim (Backend: Learning & Refinement)

This module runs asynchronously to build a simulation-ready model of the object and its environment. It operates in two phases:

Visual Geometry Reconstruction:
- Uses Gaussian Splatting (GS) on RGB-D keyframes to jointly optimize object geometry and keyframe poses.
- Converts the GS representation into a neural Signed Distance Function (SDF) ( $SDF_{vision}$ ) to serve as a collision geometry proxy.
Contact Dynamics & Geometry Compensation:
- Builds a contact dynamics model using a customized, GPU-accelerated MJX physics engine.
- Geometry Residual ( $\delta$ ): Recognizing that visual geometry is often noisy or incomplete, the system learns a constant geometric residual to refine the collision SDF: $SDF_{collision} = SDF_{vision} - \delta$ .
- Parameter Learning: Simultaneously estimates physical parameters (mass, inertia, friction) and the geometry residual.
- Optimization Strategy: Due to the non-smooth nature of contact dynamics (which hinders gradient-based methods), the authors use a sampling-based Cross-Entropy Method (CEM) variant with GPU parallelization to minimize the one-step prediction error between simulation and observed tracking data.

B. Sim2Real (Frontend: Real-Time Tracking)

This module runs online to estimate the object's pose for every frame.

Feature-Based Visual Tracking:
- Uses SAM2 for segmentation, SuperPoint for feature extraction, and LightGlue for matching features between the current frame and optimized keyframes from Real2Sim.
- Provides a visual pose estimate ( $q_{vision}$ ).
Physics-Aware Pose Estimation:
- Uses the learned contact dynamics model to predict the next state ( $q_{sim}$ ) based on the previous state.
Adaptive Fusion:
- The final pose ( $q_k$ ) is determined by minimizing a weighted sum of the visual feature loss and the dynamics prediction loss:
  $\min_{q_k} w_k L_{vision}(q_k) + (1 - w_k) L_{dynamics}(q_k)$
- Adaptive Weighting ( $w_k$ ): The weight $w_k$ is dynamically adjusted based on the quality of visual feature matches (e.g., number of inliers). If vision is degraded (occlusion/blur), the system relies more heavily on the physics simulation.

3. Key Contributions

Closed-Loop Architecture: A novel framework integrating Real2Sim (learning geometry/physics from data) and Sim2Real (using that model for robust tracking), enabling a feedback loop where tracking history improves the model, and the model improves tracking.
Geometry Residual Learning: A specific mechanism to learn a geometric correction term ( $\delta$ ) alongside physical parameters. This bridges the gap between noisy visual reconstructions and the precise geometric requirements of physics simulators.
Sampling-Based Optimization for Contact: Implementation of a GPU-accelerated, sampling-based optimizer (CEM variant) to handle the non-differentiable, non-smooth nature of contact dynamics, avoiding the pitfalls of gradient-based methods in stiff contact scenarios.
Real-Time Performance: A highly optimized implementation using a customized MJX engine and efficient neural rendering (Gaussian Splatting) achieving >20 Hz tracking speeds.

4. Experimental Results

The method was evaluated on two challenging scenarios: Object Free-Fall (impacts with walls/ground) and In-Hand Manipulation (LEAP robotic hand).

Datasets: Various unseen objects (snack boxes, bottles, 3D-printed ducks) with RGB-D video.
Metrics: Average Distance of Model Points (ADD) and ADD-S.
Key Findings:
- Accuracy: TwinTrack significantly outperformed baselines (including pure vision trackers like FoundationPose and ablation variants without dynamics or geometry residuals).
  - Example: On the "snack box" scenario, TwinTrack achieved an ADD of 1.37 cm, compared to 2.20 cm for the baseline and 1.54 cm for the "w/o dynamics" variant.
- Robustness: The system maintained tracking during heavy occlusion and motion blur where vision-only methods failed or drifted.
- Physics Alignment: The learned contact dynamics model (including the geometry residual) produced simulations that closely matched real-world trajectories (e.g., correct rebound and settling behavior), whereas models without the residual showed "floating" or unrealistic contact.
- Speed: The system processes frames at >20 Hz (approx. 48ms total latency per frame on an RTX 4090), meeting real-time requirements.

5. Significance and Future Work

Significance: TwinTrack demonstrates that integrating contact physics cues is essential for robust perception in dynamic, occluded environments. It successfully bridges the "Sim-to-Real" gap by learning a physics model directly from visual data, enabling robots to manipulate unknown objects without pre-existing models.
Limitations:
- Currently limited to single rigid objects.
- Requires a reasonably accurate initial state; poor initialization can lead to drift.
- Uses a uniform geometric residual, which cannot handle spatially varying geometry errors.
Future Directions: Extending to multi-object tracking, handling deformable objects, and learning spatially varying geometry compensations. The aligned simulation environment is also proposed as a substrate for training reinforcement learning policies.