💻 computer science

Physics-Informed Tracking (PIT)

The paper introduces Physics-Informed Tracking (PIT), a video-based framework that utilizes a split-bottleneck autoencoder and a differentiable physics module to track single particles by enforcing physical consistency on trajectories, achieving sub-pixel accuracy through both unsupervised (PILL) and supervised (PILLS) learning strategies.

Original authors: Emil Hovad, Allan Peter Engsig-Karup

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Emil Hovad, Allan Peter Engsig-Karup

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to follow a single, bouncing ball in a video. Sometimes the video is crystal clear, but other times it's grainy, foggy, or full of static noise, making the ball almost invisible.

Usually, computer programs try to solve this by just "looking" at the video and guessing where the ball is in each frame. It's like a student trying to memorize a list of answers without understanding the math behind them. If the test gets hard (noisy video), the student fails.

This paper introduces a new method called Physics-Informed Tracking (PIT). Think of it as teaching the computer not just to see the ball, but to understand how balls move in the real world.

Here is the breakdown of how it works, using some everyday analogies:

1. The "Split Brain" Architecture (The Autoencoder)

Imagine the computer has a brain with two distinct jobs, separated by a narrow hallway (the "bottleneck"):

Job A (The Tracker): This part of the brain is obsessed with finding the ball. It creates a "heat map"—a glowing spot that gets brighter where the ball is.
Job B (The Background Cleaner): This part ignores the ball and focuses on the messy background (the noise, the static, the shadows). It tries to reconstruct the whole picture.

By separating these two jobs, the computer learns to ignore the noise and focus purely on the ball, even when the video is terrible.

2. The "Physics Coach" (The Differentiable Module)

This is the magic sauce. In standard tracking, the computer just guesses the ball's position in frame 10 based on frame 9.

In PIT, there is a "Physics Coach" sitting next to the computer.

The Rule: The Coach knows the laws of physics. It knows that if a ball is falling, it must accelerate downward due to gravity. It knows that if the ball hits a wall, it must bounce back with less speed.
The Correction: If the computer's guess says the ball suddenly teleported or moved sideways without reason, the Coach says, "No, that's impossible! Gravity doesn't work that way." The computer then adjusts its guess to fit the laws of physics.

3. Two Ways to Learn (Supervised vs. Unsupervised)

The paper tests two ways to train this system:

The "Teacher" Method (PILLS): The computer is given a textbook with the correct answers (the exact position, speed, and bounce time of the ball). The Physics Coach checks the computer's work against the textbook. This is Supervised Learning.
The "Self-Check" Method (PILL): The computer doesn't have the textbook. It doesn't know where the ball actually is. However, the Physics Coach still checks its work. If the computer predicts a path that looks like a wobbly snake instead of a smooth parabola, the Coach says, "That doesn't look like gravity." The computer learns to make its predictions physically consistent, even without knowing the "right" answer. This is Unsupervised Learning.

4. The Results: Why It Matters

The researchers ran thousands of experiments (like a massive science fair with 64 different combinations of settings) to see what worked best.

The Winner: The system that used the "Teacher" method (PILLS) was the most accurate. It could track the ball with sub-pixel accuracy—meaning it could tell you where the ball was even more precisely than the individual dots (pixels) on the screen.
The Bonus: Because the system understands physics, it doesn't just tell you where the ball is. It also tells you how fast it's going and when it will bounce, all in a single glance. Standard video trackers usually can't do this; they just guess the location.

The Big Picture

Think of this like teaching a child to ride a bike.

Old Way: You just tell them, "Pedal here, turn there." If the wind blows, they fall because they don't understand balance.
PIT Way: You teach them the principles of balance and momentum. Even if the wind blows (noise), they instinctively adjust their body to stay upright because they understand the physics of the bike.

In short: This paper shows that if you teach a computer the laws of physics (gravity, bouncing, speed) and let it apply those rules while watching a video, it becomes a much better tracker, even when the video is blurry or noisy. It turns a simple "guessing game" into a smart, physics-aware prediction engine.

1. Problem Statement

Object tracking in video is a fundamental computer vision task. While deep learning methods (e.g., Faster R-CNN, YOLO, CenterNet) have achieved success, standard approaches often treat tracking as a purely data-driven regression or classification problem. They lack physical consistency, meaning predicted trajectories may violate known laws of motion (e.g., gravity, momentum conservation), especially in noisy conditions or when ground-truth labels are scarce.

The authors address the challenge of tracking a single particle (a ball) in video sequences where:

The particle follows specific physical dynamics (parabolic motion under gravity with inelastic collisions).
Data may be noisy, making standard heatmap-based localization unreliable.
There is a need for a framework that can operate in both supervised (with ground truth) and unsupervised (without ground truth) settings while enforcing physical laws.

2. Methodology: Physics-Informed Tracking (PIT)

The proposed framework, PIT, is a video-based autoencoder architecture that integrates a differentiable physics module directly into the tracking pipeline.

A. Architecture: Split-Bottleneck Autoencoder

The model processes a sliding window of three consecutive frames ( $t-1, t, t+1$ ).

Shared Encoder: Downsamples input frames (224x224) to a latent space (56x56).
Split Bottleneck: The latent representation is split into two distinct streams:
1. Tracking Stream (Landmark Heatmaps): A single-channel heatmap where the peak represents the particle's location.
2. Residual Stream (Background/Noise): Multi-channel features capturing background texture and noise, used for image reconstruction.
Multi-Scale Decoder: Uses skip connections to refine heatmaps at three scales: 56, 112, and 224 pixels.
Differentiable Physics Module: A Velocity-Verlet integrator with bounce handling (using torch.where for differentiability) takes the estimated positions and evolves the system state to predict refined positions, velocities, and bounce events.

B. Key Components and Loss Functions

The training procedure is governed by six binary factors (A–F) evaluated via a replicated $2^6$ factorial design. The core innovations are the loss functions:

Physics-Informed Landmark Loss (PILL) - Unsupervised:
- Compares the raw heatmap-derived landmarks against the output of the physics module.
- Enforces that the predicted trajectory satisfies physical laws (e.g., gravity) without requiring ground-truth labels.
- Goal: Self-consistency.
PILLS (PILL Supervised) - Supervised:
- Compares the physics-refined predictions directly against ground-truth position, velocity, and bounce indicators from the simulation.
- Enables end-to-end backpropagation through the entire physics module.
Reconstruction & Heatmap Losses:
- $L_{AE}$ & $L_{cone}$ : Standard autoencoder reconstruction losses (BCE) and cone-guided losses to ensure the background is reconstructed and the object is localized.
- $L_{hm}$ : Focal heatmap loss (similar to CenterNet) for supervised landmark localization.

C. Differentiable Physics Model

The physics module operates in pixel space using a discretized time step ( $\Delta t = 1$ frame). It assumes:

Constant gravitational acceleration ( $g$ ).
No horizontal acceleration.
Inelastic collisions with boundaries (velocity reflection scaled by a restitution coefficient $e$ ).
The module outputs position ( $\hat{p}$ ), velocity ( $\hat{v}$ ), and bounce indicators ( $\hat{b}$ ) in a single forward pass.

3. Key Contributions

Split-Bottleneck Autoencoder: A novel architecture that explicitly separates tracking-related structure (landmarks) from background noise, allowing the model to learn robust features even in noisy conditions.
Physics-Informed Landmark Loss (PILL): An unsupervised loss function that constrains landmark trajectories to satisfy known dynamics (gravity, collisions) without ground-truth labels.
Supervised Physics Variant (PILLS): An end-to-end trainable framework that learns position, velocity, and bounce timing simultaneously by backpropagating through a differentiable physics solver.
Comprehensive Factorial Evaluation: The authors conducted a rigorous $2^6$ factorial design experiment ( $n=4$ replicates, 64 configurations) to isolate the impact of architectural choices, supervision levels, and noise conditions on tracking performance.

4. Results

The model was evaluated on synthetic video data of a bouncing ball under clean ( $\sigma=0$ ) and noisy ( $\sigma=1$ ) conditions.

Accuracy: The PILLS configuration (supervised physics) consistently achieved sub-pixel tracking accuracy (errors $\le 0.42$ pixels) at intermediate and high resolutions (112 and 224 scales), outperforming standard heatmap baselines.
Robustness: Under noisy conditions, physics-informed constraints significantly improved stability. While standard baselines (CenterNet-style) suffered large errors or failed to converge in noise, the physics-constrained models maintained low error rates.
Velocity and Bounce Prediction: A unique advantage of PIT is its ability to output velocity and bounce timing directly from the forward pass. The model successfully predicted velocity sign flips corresponding to wall bounces, a capability absent in standard heatmap trackers.
Factorial Analysis:
- The supervised physics loss (Factor E) and the multi-scale decoder (Factor A) were the dominant factors reducing error.
- Unsupervised physics (PILL, Factor D) helped but was less consistent than supervised physics.
- Hard argmax (integer pixel) was less accurate than bilinear/bicubic soft-argmax combined with physics refinement.

5. Significance and Future Work

Bridging the Gap: PIT successfully bridges the gap between deep learning-based visual tracking and scientific machine learning (PINNs), demonstrating that physical laws can be embedded directly into autoencoder training for tracking.
Data Efficiency: The unsupervised PILL loss suggests that physical consistency can be learned even without labeled trajectories, which is crucial for real-world applications where ground truth is unavailable.
Beyond Tracking: The framework naturally extends to generative settings, allowing for the synthesis of trajectories that are both visually plausible and physically consistent.
Limitations: Current experiments are limited to single-particle tracking in synthetic data. Future work aims to address multi-particle tracking with crossing trajectories and validation on real-world video data.

In conclusion, Physics-Informed Tracking (PIT) demonstrates that integrating differentiable physics modules into autoencoder architectures significantly enhances tracking accuracy, robustness to noise, and the ability to estimate physical states (velocity, bounce) beyond simple position localization.