Accurate Planar Tracking With Robust Re-Detection

This paper introduces SAM-H and WOFTSAM, novel planar trackers that integrate SAM 2's robust segmentation with homography estimation and re-detection capabilities to achieve state-of-the-art performance on the POT-210 and PlanarTrack benchmarks while providing improved ground-truth annotations.

The Gist

Imagine you are playing a game of "Follow the Leader" with a flat piece of paper (like a poster or a playing card) in a video. Your goal is to keep your eyes locked on that paper as it spins, zooms in, gets blurry, or even disappears behind a wall. This is called Planar Tracking, and it's the backbone of Augmented Reality (AR) apps and robot vision.

For a long time, computers were really good at this when the paper had cool patterns and moved slowly. But if the paper got blurry, was transparent, or had no texture (like a white sheet of paper), the computer would lose it and never find it again.

This paper introduces a new team of two digital detectives, SAM-H and WOFTSAM, that solve this problem by combining two very different superpowers.

The Two Detectives

1. The "Shape Shifter" (SAM-H)

Think of SAM-H as a detective with a magical, glowing outline. It uses a tool called "SAM 2" (Segment Anything Model) which is amazing at drawing a perfect border around an object, even if the object is blurry, transparent, or has no patterns.

• How it works: It draws a box around the target. Then, it looks at the corners of that box to guess where the object is.
• The Flaw: It's great at finding the object when it's lost, but it's a bit clumsy with the exact angles. It's like someone who can point to a building from a mile away but can't tell you exactly which window you are looking at.

2. The "Pixel Perfectionist" (WOFT)

Think of WOFT as a detective with a high-powered microscope. It looks at every tiny dot of color (texture) on the object and matches them frame-by-frame.

• How it works: It tracks the patterns on the paper with incredible precision.
• The Flaw: If the paper gets blurry, covered by a hand, or moves too fast, the patterns disappear. The microscope goes blind, and the detective gives up. It has no "search and rescue" plan.

The Super Team: WOFTSAM

The authors realized that these two detectives are perfect for each other. They created WOFTSAM, a hybrid system that acts like a relay race team:

1. The Sprint (WOFT): The team starts with the "Pixel Perfectionist" (WOFT). As long as the object is clear and moving slowly, WOFT does the heavy lifting, tracking the object with sub-pixel accuracy.
2. The Rescue (SAM-H): Suddenly, the object gets covered by a hand, or the camera shakes too much, and WOFT loses the target. Instead of giving up, WOFTSAM calls in the "Shape Shifter" (SAM-H).
3. The Handoff: SAM-H scans the whole screen, finds the object again (even if it's blurry or transparent), and draws a rough outline around it. It says, "Hey, I found it! It's over here!"
4. The Finish: WOFTSAM takes that rough location from SAM-H, uses it as a starting point, and switches back to the high-precision microscope to lock on again.

Why This Matters (The "Aha!" Moments)

The paper shows this new team winning against all previous champions in two major ways:

• Handling the Impossible: In tests, they tracked objects that other trackers failed at completely. This includes:
  • Mirrors: Where the reflection tricks the eye.
  • TV Screens: Where the image on the screen is constantly changing.
  • Glass: Where the object is see-through.
  • Motion Blur: Where the object is moving so fast it looks like a smear.
• The "Map" Problem: The authors also realized that the "maps" (ground truth data) used to test these trackers were slightly wrong. It's like trying to run a race where the finish line is painted in the wrong spot. They went back and re-painted the finish lines with perfect precision. This showed that previous trackers were actually doing better than we thought, but the new team (WOFTSAM) is still the undisputed champion.

The Limitations (Where They Still Stumble)

Even superheroes have kryptonite. The paper admits that:

• Box Confusion: If you ask the system to track just the front of a box, it sometimes gets confused and starts tracking the whole box (including the sides) as it turns.
• The "Look-Alike" Problem: If there are two identical objects in the scene (like two identical playing cards), the system might get confused about which one is the "real" target.
• Total Blackout: If the object is completely hidden for a long time and then reappears in a totally different spot, the system might not realize it's the same object.

The Bottom Line

This paper is about teaching computers to be resilient. Instead of just being precise, they are now being robust. By combining a "rough but reliable" finder with a "precise but fragile" tracker, they created a system that can handle the messy, unpredictable real world.

It's the difference between a GPS that stops working when you go into a tunnel, and a GPS that knows exactly where you are, remembers the last known location, and guides you out of the tunnel the moment you re-emerge.

Technical Summary

1. Problem Statement

Planar Object Tracking involves localizing and estimating the 8-degree-of-freedom (8-DoF) homography pose of a flat object in video frames. While essential for Augmented Reality (AR), robotics, and 3D reconstruction, this task remains unsolved under general conditions.

Key Challenges:

• Visual Degradations: Motion blur, lack of texture, and strong perspective distortion.
• Target Complexity: Reflective surfaces, transparent objects, virtual planes, and targets with dynamically changing content (e.g., video screens).
• Tracking Failures: The current state-of-the-art (WOFT) struggles to recover once a target is lost due to occlusion, motion blur, or moving out of view because it lacks a robust re-detection mechanism.
• Annotation Noise: Existing benchmarks (like PlanarTrack) suffer from inaccurate ground truth (GT) annotations, particularly in initial frames, which skews the evaluation of high-precision methods.

2. Methodology

The authors propose two interconnected methods: SAM-H and WOFTSAM. The core philosophy is to combine the long-term resilience of segmentation-based trackers with the geometric precision of optical flow-based homography estimation.

A. SAM-H (Segmentation-to-Homography)

SAM-H is a module that converts the output of a general segmentation tracker (SAM 2) into a homography pose.

1. Segmentation: The system prompts SAM 2 with an initial quadrilateral mask ($X_0$) to track the object, generating a segmentation mask ($S_t$) for each frame.
2. Corner Extraction: It fits four lines to the mask contours using the Hough Transform and calculates their intersections to find candidate corners ($\tilde{X}_t$).
3. Symmetry Disambiguation: Since a quadrilateral is cyclically symmetric, the order of corners is ambiguous.
   • Standard Tracking: Uses a zero-velocity motion model to minimize distance from the previous frame's corners.
   • Re-detection: When the target is lost and re-detected, it compares DINOv2 features of the current crop against a template warped to four cyclically shifted candidate views. The shift with the highest feature similarity is selected.
4. Homography Estimation:
   • If all 4 points are visible: Direct 8-DoF homography estimation.
   • If 2-3 points are visible: Estimates a 4-DoF similarity transform (translation, rotation, scale) relative to the previous frame.
   • If 1 point is visible: Estimates a 2-DoF translation.

B. WOFTSAM (The Integrated Tracker)

WOFTSAM integrates SAM-H into the existing WOFT framework to enable robust re-detection.

1. Primary Tracking (WOFT): Uses Weighted Flow Homography (WFH) based on dense optical flow. It pre-warps the current frame using the previous frame's pose ($H_{t-1}$) and estimates a residual homography.
2. Failure Detection: If the optical flow support set is too small (indicating tracking failure), the system triggers a re-detection step.
3. Re-Detection (SAM-H): The system uses the SAM-H output ($H_{SAM}$) as a new pre-warping homography. It attempts to re-estimate the homography using WFH with this new initialization.
4. Fallback: If the second attempt fails, the system returns $H_{SAM}$ as the best-guess output.

3. Key Contributions

1. SAM-H: A novel method for estimating homography poses directly from segmentation mask contours, enabling robust long-term tracking even when texture is missing.
2. WOFTSAM: A new state-of-the-art (SOTA) planar tracker that significantly improves upon WOFT by integrating SAM-H for re-detection. It achieves SOTA on both POT-210 and PlanarTrack benchmarks.
3. Improved Ground Truth: The authors performed precise re-annotation of the initial frames for the PlanarTrack benchmark. They demonstrated that original annotations had significant errors (mean ~5.71 px), which disproportionately affected high-precision metrics (p@5) for optical flow-based methods due to scaling effects.
4. Complementarity Analysis: The paper highlights that optical flow and segmentation approaches are complementary. Optical flow excels with texture; segmentation excels with blur, transparency, and reflection.

4. Experimental Results

The methods were evaluated on POT-210 and PlanarTrack using precision metrics p@5 (error < 5px) and p@15 (error < 15px).

• PlanarTrack Performance:

  • WOFTSAM outperformed the previous SOTA (WOFT) by a large margin: +12.4 pp on p@15 and +15.2 pp on p@5.
  • SAM-H alone achieved the best results on PlanarTrack (p@15: 80.0) compared to WOFTSAM (77.2) in specific scenarios involving transparent/reflective targets where optical flow fails.
  • An "Oracle" experiment (selecting the best of WOFTSAM or SAM-H per sequence) achieved 86.9 p@15, proving the complementary nature of the two approaches.

• POT-210 Performance:

  • WOFTSAM set a new SOTA, nearly halving the failure rate (p@15) compared to WOFT.
  • Significant improvements were observed in challenging attributes: Motion Blur, Occlusion, and Out-of-View scenarios, where re-detection is critical.

• Impact of Re-annotation:

  • Using the new precise GT, the performance gap between WOFTSAM and SAM-H on PlanarTrack p@5 was reduced by more than half, confirming that previous evaluations were skewed by noisy initial annotations.

5. Significance and Limitations

Significance:

• Robustness: The integration of segmentation allows planar tracking to survive conditions (blur, reflection, transparency) that previously caused total tracking failure.
• Benchmark Integrity: The re-annotation effort provides a more accurate baseline for future research, separating algorithmic failure from annotation noise.
• Hybrid Approach: It demonstrates that combining geometric reasoning (homography) with semantic segmentation is a powerful paradigm for long-term tracking.

Limitations:

• Shape Assumption: SAM-H assumes a quadrilateral shape. It struggles if the target is not a quadrilateral or if occlusions prevent the detection of four stable lines.
• Segmentation Errors: SAM 2 can sometimes "spill" over to track the whole 3D object (e.g., a whole box) instead of the specific planar face, or fail to track arbitrary regions of a larger surface.
• Symmetry & Distractors: Symmetry disambiguation can fail on textureless or symmetric targets. Re-detection can fail if distractors are visually identical to the target.
• Partial Occlusion: While SAM-H handles partial occlusion well if the shape remains quadrilateral, it cannot recover the full 8-DoF pose if fewer than two sides are visible.

Conclusion

The paper presents WOFTSAM, a tracker that sets a new standard for planar object tracking by effectively merging the robustness of segmentation-based re-detection with the precision of optical flow. It not only improves performance metrics significantly but also addresses critical issues in benchmark evaluation through precise ground-truth re-annotation.