GOT-JEPA: Generic Object Tracking with Model Adaptation and Occlusion Handling using Joint-Embedding Predictive Architecture

The paper introduces GOT-JEPA, a model-predictive pretraining framework that enhances generic object tracking generalization by predicting tracking models from corrupted frames, and further proposes OccuSolver to refine occlusion handling through iterative visibility estimation and point-centric tracking.

The Gist

Here is an explanation of the paper "GOT-JEPA: Generic Object Tracking with Model Adaptation and Occlusion Handling using Joint-Embedding Predictive Architecture" using simple language and creative analogies.

The Big Picture: The "Super-Spy" Tracker

Imagine you are playing a game of "Where's Waldo?" but the video is moving fast, the lighting changes, and sometimes a giant hand covers Waldo completely. Most computer programs trying to find Waldo get confused when he disappears or when the background gets messy. They are like a detective who only knows what Waldo looks like in a sunny park; if Waldo steps into a dark alley or gets covered by a tree, the detective gives up.

This paper introduces a new system called GOT-JEPA (with a helper named OccuSolver) that acts like a Super-Spy. Instead of just memorizing what the target looks like, this spy learns how to learn. It can adapt to new targets, handle bad weather, and even guess where the target is when it's completely hidden.

---

Part 1: The Brain Training (GOT-JEPA)

The Concept: Learning to Predict the Future, Not Just the Present.

Most trackers are like students who memorize the answers to a specific test. If the test questions change slightly, they fail. The authors wanted a tracker that learns the skill of tracking, not just the answers.

They used a training method called JEPA (Joint-Embedding Predictive Architecture). Think of this as a Teacher-Student Gym Class:

1. The Teacher (t-Predictor): This is the expert. It looks at a clear, perfect video frame and says, "Okay, here is exactly how we should track this object right now." It creates a "perfect tracking plan."
2. The Student (s-Predictor): This is the trainee. It looks at the same video, but the image is corrupted. Maybe it's blurry, has a fake object pasted over it, or is dark.
3. The Challenge: The Student has to look at the messy, corrupted image and guess the exact same "perfect tracking plan" that the Teacher made from the clear image.

The Analogy: Imagine a chef (the Teacher) making a perfect cake. Then, a student (the Student) is given the same recipe but with the flour mixed with dirt and the oven temperature wrong. The student has to figure out how to bake the exact same perfect cake despite the bad ingredients. By practicing this over and over, the student learns to ignore the dirt and focus on the core recipe.

The Result: The tracker becomes incredibly robust. It doesn't panic when the video gets messy because it has learned to predict the "tracking model" even when the input is terrible.

---

Part 2: The Detective's Magnifying Glass (OccuSolver)

The Concept: Seeing Through the Fog.

Even a super-spy gets stuck if an object is completely hidden behind a wall. Traditional trackers treat the object as a single box. If half the box is covered, they often lose track.

The authors added a tool called OccuSolver. Think of this as giving the tracker X-Ray Vision or a Pointillist Painter's Eye.

1. The Problem: Standard trackers look at the whole "blob" (the bounding box). If a tree covers half the blob, the tracker gets confused.
2. The Solution: OccuSolver breaks the object down into hundreds of tiny dots (points). It asks: "Is this specific dot visible? Is that dot hidden?"
3. The Magic: It uses a "Point Tracker" (a tool that follows tiny dots on surfaces) but teaches it to care about the specific object the main tracker is following.
   • If a dot is on the target's face and gets covered by a hand, OccuSolver marks it as "Invisible."
   • If a dot is on the target's shoulder and is still visible, it marks it as "Visible."

The Analogy: Imagine trying to follow a friend in a crowded party.

• Old Tracker: Looks at the whole crowd and says, "I think my friend is in that group." If someone blocks the view, it loses them.
• OccuSolver: Focuses on specific features: "I see the red hat (visible), I see the blue shoe (hidden), I see the left hand (visible)." Even if the friend is 80% hidden, the tracker knows exactly where the visible parts are and can guess where the hidden parts must be.

---

Part 3: How They Work Together

The paper describes a beautiful loop where these two parts help each other:

1. The Brain (GOT-JEPA) creates a strong, adaptable tracking model that can handle messy videos.
2. The Eyes (OccuSolver) look at the video and tell the Brain, "Hey, the target is partially hidden. Here are the specific dots we can still see."
3. The Feedback Loop: The Brain uses this "dot information" to create an even better "tracking plan" for the next frame. This new plan helps OccuSolver find even better dots.

It's like a dance partnership: The Brain provides the rhythm and strategy, while the Eyes provide the precise steps. Together, they never miss a beat, even when the music stops or the lights go out.

Why Does This Matter?

• Real-World Use: This isn't just for video games. This technology helps self-driving cars track pedestrians when they step behind a bus, helps drones follow people through forests, and helps security cameras spot intruders even when they try to hide.
• Generalization: Most AI needs to be retrained for every new scenario. This system learns a general "skill" of tracking, so it works on things it has never seen before (like a new type of animal or a weirdly shaped object).

Summary

The authors built a Super-Spy Tracker that:

1. Trains itself by trying to solve puzzles with "dirty" data (GOT-JEPA).
2. Uses X-Ray vision to track tiny dots on an object to see exactly what is hidden and what is visible (OccuSolver).
3. Combines both to create a tracker that is incredibly hard to fool, even when objects are moving fast, changing shape, or getting completely covered up.

Technical Summary

Here is a detailed technical summary of the paper "GOT-JEPA: Generic Object Tracking with Model Adaptation and Occlusion Handling using Joint-Embedding Predictive Architecture."

1. Problem Statement

Generic Object Tracking (GOT) aims to track an arbitrary target specified only by an initial bounding box across video frames. Despite advancements, current state-of-the-art trackers face two critical limitations:

• Poor Generalization: Most trackers are optimized to identify specific targets seen during training. When faced with unseen targets or dynamic environments (out-of-distribution), their "model predictors" (which update the tracker for new frames) often fail to adapt robustly.
• Coarse Occlusion Reasoning: Existing methods handle occlusions at a scene or bounding-box level (e.g., estimating a global confidence score). They lack fine-grained reasoning to determine which specific parts of an object are visible versus occluded. This leads to unreliable tracking when objects are partially hidden, as the system cannot distinguish between target and background features effectively.

2. Methodology

The authors propose a two-stage framework comprising GOT-JEPA (for robust model prediction) and OccuSolver (for fine-grained occlusion handling).

A. GOT-JEPA: Model-Predictive Pretraining

The core innovation is extending the Joint-Embedding Predictive Architecture (JEPA) from predicting image features to predicting tracking models.

• Teacher-Student Framework:
  • Teacher (t-Predictor): A frozen, pre-trained predictor that generates "pseudo-tracking models" ($\hat{\omega}$) from a clean current frame, conditioned on identical historical information (reference frames/labels).
  • Student (s-Predictor): Trained to predict the same pseudo-models ($\hat{\omega}$) but using a corrupted version of the current frame (e.g., with occlusions, distractors, or noise), while sharing the same historical context.
• Learning Objective: The student learns to recover target-background discrimination despite degraded observations. This forces the model to learn invariant representations that generalize to unseen targets and adverse conditions.
• Loss Functions:
  • Invariance Loss ($L_{inv}$): Minimizes the distance between the student's predicted model and the teacher's pseudo-model.
  • Covariance Loss ($L_{cov}$): Applied via an "Expander" head to reduce redundancy in the predicted tracking models, encouraging diverse and discriminative patterns.

B. OccuSolver: Fine-Grained Occlusion Handling

To address the lack of pixel-level visibility estimation, the authors introduce OccuSolver, which integrates a point tracker with the GOT framework.

• Object-Aware Point Tracking: Standard point trackers (like CoTracker) are point-centric and lack object awareness. OccuSolver conditions the point tracker on GOT-derived object priors (reference labels) to ensure the tracked points correspond to the target object rather than the background.
• Iterative Refinement:
  1. Refinement: An iterative Transformer refines point coordinates and appearance features using object priors.
  2. Visibility Estimation: A learnable VisHead estimates the visibility state (visible/invisible) for each point.
  3. Ensemble Network: A lightweight transformer fuses the sparse point-visibility cues with the dense visual features of the current frame. This creates a "visibility-aware" feature map.
• Feedback Loop: The refined visibility states generate higher-quality reference labels for the tracker, which in turn improves the object priors for the next iteration, creating a tight coupling between model adaptation and occlusion perception.

3. Key Contributions

1. GOT-JEPA Framework: The first application of JEPA to tracking model prediction rather than image feature prediction. It shifts the paradigm from optimizing for specific training targets to learning a general skill of model adaptation.
2. OccuSolver: A novel module that adapts point trackers for object tracking by incorporating object priors. It enables fine-grained, pixel-level occlusion reasoning, allowing the tracker to distinguish visible vs. occluded sub-regions of a target.
3. Tight Coupling: The system creates a synergistic loop where the tracker improves the point tracker's object awareness, and the point tracker provides visibility cues that stabilize the tracker's model predictions.
4. Robustness: The method explicitly trains the model to handle corrupted inputs (occlusions, distractors) during pre-training, leading to superior generalization in unseen scenarios.

4. Experimental Results

The method was evaluated on seven benchmarks, including AVisT (adverse visibility), NfS, OTB-100, GOT-10k, LaSOT, TrackingNet, and VOT2022.

• State-of-the-Art Performance: GOT-JEPA achieved top performance across multiple datasets.
  • AVisT (Out-of-Distribution): 63.7% Success Rate (SUC), outperforming PiVOT (62.2%) and LoRAT (62.0%).
  • OTB-100: 73.2% SUC, surpassing SAMURAI (71.5%) and PiVOT (71.2%).
  • GOT-10k: 79.6% Average Overlap (AO), the highest among non-segmentation-based trackers.
  • LaSOT & TrackingNet: Achieved the best Normalized Precision (NPr) and SUC, demonstrating strong in-distribution performance.
• Attribute Analysis: The tracker showed significant improvements in handling occlusion, deformation, distractors, and low visibility compared to baselines like ToMP-L and PiVOT.
• Ablation Studies:
  • Removing the JEPA pre-training or OccuSolver resulted in significant performance drops.
  • The combination of Invariance and Covariance losses was found to be optimal for generalization.
  • The "Copy-Paste" corruption strategy during pre-training was more effective than masking for simulating real-world occlusions.

5. Significance

This work represents a paradigm shift in Generic Object Tracking:

• From Feature to Model Prediction: By treating the tracking model itself as the prediction target, the system learns to adapt dynamically, mimicking human visual reasoning where the brain updates its internal model of an object based on partial observations.
• Granular Occlusion Handling: Moving from box-level confidence to point-level visibility allows the tracker to maintain accuracy even when large portions of the target are hidden, a critical capability for long-term tracking.
• Generalization: The framework demonstrates that pre-training with JEPA on corrupted data significantly improves robustness to unseen targets and adverse environmental conditions, addressing a major bottleneck in current tracking systems.

In summary, GOT-JEPA establishes a new standard for robust, adaptive, and occlusion-aware object tracking by leveraging predictive architecture and fine-grained geometric cues.