SCATR: Mitigating New Instance Suppression in LiDAR-based Tracking-by-Attention via Second Chance Assignment and Track Query Dropout

This paper presents SCATR, a novel LiDAR-based tracking-by-attention framework that mitigates new instance suppression and bridges the performance gap with detection-based methods through two architecture-agnostic training strategies: Second Chance Assignment and Track Query Dropout, achieving state-of-the-art results on the nuScenes benchmark.

The Gist

Imagine you are the conductor of a busy orchestra, but instead of musicians, you are tracking hundreds of cars, pedestrians, and cyclists moving through a city in real-time. Your goal is to keep a perfect scorecard of who is who, where they are, and where they are going, even as they zip past each other, get hidden behind buildings, or suddenly appear out of nowhere.

This is the job of LiDAR-based tracking, a technology used in self-driving cars. The car uses laser beams (LiDAR) to create a 3D map of the world.

For a long time, there were two ways to do this:

1. The "Detect-Then-Track" Method (TBD): First, take a snapshot and find everyone. Then, in the next snapshot, find everyone again and try to match the dots from the first picture to the second. It's like taking a photo of a crowd, then taking another photo a second later and trying to guess who is who by looking at their clothes. It works well, but it's slow and can get confused if people move fast.
2. The "Attention" Method (TBA): This is the newer, cooler approach. Instead of taking snapshots, the computer keeps a running list of "Trackers" (like little mental sticky notes) for every object it sees. As the car moves, these sticky notes update their positions. If a new car appears, a new sticky note is created.

The Problem: The "Shy Newcomer" Issue
The paper argues that the "Attention" method has a fatal flaw, especially with LiDAR data. It suffers from what the authors call "New Instance Suppression."

Here is an analogy:
Imagine a teacher (the AI) who has a class of students (the cars). The teacher has a group of "Senior Monitors" (Track Queries) who are assigned to watch specific students.

• The Flaw: If a new student walks into the classroom late (a "new instance"), the Senior Monitors get so confident in their own jobs that they accidentally ignore the new kid. The teacher's brain thinks, "Oh, I'm already watching Student A, B, and C. I don't need to look for anyone else."
• The Result: The new car appears, but the system ignores it because it's too busy tracking the old ones. This leads to "False Negatives"—the car is there, but the self-driving car doesn't see it.

The Solution: SCATR
The authors introduce SCATR, a new system designed to fix this shyness. They use two clever training tricks to teach the AI how to be more observant.

1. Track Query Dropout (The "Pop Quiz" Strategy)

The Metaphor: Imagine you are training a security guard to watch a crowd. If you always give them the exact same list of people to watch, they get lazy. They stop looking for new people because they know exactly who is on the list.

How SCATR does it:
During training, the system randomly "drops" some of the Senior Monitors from the list.

• Scenario: The system is watching Car A. Suddenly, it pretends Car A's monitor is on a coffee break (dropped).
• The Lesson: Now, the system must look at the "Newcomer List" (Proposal Queries) to find Car A. It learns that if a monitor is missing, it can't just ignore the car; it has to find a new way to track it.
• The Result: The AI becomes robust. Even if a tracker gets lost or confused, the system knows how to pick up the slack and find the car again.

2. Second Chance Assignment (The "Second Interview")

The Metaphor: Imagine a hiring manager who only hires people based on their first interview. If a candidate is good but didn't get picked in the first round, they are thrown away forever.

How SCATR does it:
In the old system, if a "Senior Monitor" (Track Query) wasn't assigned to a car, it was discarded. The system relied only on the "Newcomer List" to find new cars.

• The Fix: SCATR says, "Wait! Let's give the unassigned Senior Monitors a Second Chance."
• If a car appears and no one is watching it, the system takes the unassigned Senior Monitors and says, "Hey, you're free! Go track this new car."
• The Result: The system uses its best resources (the experienced monitors) to catch new cars, rather than relying on the weaker, less confident "Newcomer List." This drastically reduces the number of cars the system misses.

The Big Win

By using these two tricks, SCATR bridges the gap between the old, reliable "Detect-Then-Track" method and the newer, faster "Attention" method.

• Before: The new "Attention" method was missing about 30% more cars than the old method.
• After (SCATR): It catches almost as many cars as the old method, but it does it in a smoother, more continuous way (like a conductor keeping the orchestra in sync) rather than taking snapshots.

In a Nutshell:
SCATR teaches the self-driving car's brain to stop being so focused on what it already knows that it forgets to look for what's new. It does this by occasionally taking away its "safety nets" (Dropout) and giving its "experienced staff" a second chance to spot new arrivals (Second Chance). The result is a self-driving car that is much less likely to miss a pedestrian stepping out from behind a bus.

Technical Summary

1. Problem Statement

The paper addresses a critical performance gap between Tracking-by-Detection (TBD) and Tracking-by-Attention (TBA) in LiDAR-based multi-object tracking (MOT).

• The Context: Traditional TBD methods (e.g., CenterPoint + SimpleTrack) decouple detection and association, achieving high accuracy but suffering from error propagation and limited temporal exploitation. TBA methods (e.g., TrackFormer, JDT3D) unify detection and tracking in an end-to-end framework using query-based transformers.
• The Core Issue: While TBA excels in vision-based domains, LiDAR-based TBA suffers significantly from New Instance Suppression.
  • In TBA, "Track Queries" (TQs) maintain object identities across frames, while "Proposal Queries" (PQs) detect new objects.
  • The model learns to suppress PQs if a TQ is already assigned to an object. However, when an object is occluded or a new object appears, the model often fails to activate the necessary PQs because the TQs (even if dropped or missing) dominate the attention mechanism.
  • This leads to high False Negative (FN) rates and an inability to initialize tracks for newborn objects, causing LiDAR-TBA to underperform compared to LiDAR-TBD baselines.

2. Methodology: SCATR

The authors propose SCATR, a novel LiDAR-based TBA framework that retains the standard architecture but introduces two architecture-agnostic training strategies to resolve the suppression issue.

A. Architecture Overview

• Backbone: Uses a standard LiDAR backbone to encode point clouds into Bird's Eye View (BEV) features.
• Two-Stage Decoder:
  1. Detection Decoder: Uses proposal queries to detect objects in the current frame.
  2. Track Decoder: Concatenates the top-$N_{PQ}$ confident proposal queries with track queries propagated from the previous frame. It performs self-attention and cross-attention with BEV features to refine tracks and detect new objects.
• Propagation: The top-$N_{TQ}$ queries are propagated to the next frame.

B. Key Innovation 1: Track Query Dropout

Inspired by Group-DETR, this strategy improves the model's robustness to missing or switched track queries.

• Mechanism: During training, instead of propagating only the top-$N_{TQ}$ assigned queries, the model creates auxiliary query groups.
• Process: In each iteration, a random subset of assigned track queries is "dropped" (not propagated), while the total number of queries remains constant by sampling from the pool.
• Effect: This forces the decoder to learn context-dependent behaviors. If a track query for a specific object is missing in the next frame, the model learns to rely on proposal queries to re-detect or re-assign that object, rather than suppressing the proposal query due to the absence of the track query.

C. Key Innovation 2: Second Chance Assignment

This mechanism directly addresses the imbalance in supervision between detection and tracking tasks.

• Problem: In standard TBA, unassigned ground truth objects (newborns) are matched only to proposal queries. Unassigned track queries are discarded, leading to under-supervised proposal queries and over-confident track queries.
• Mechanism: Before the Hungarian matching (bipartite assignment) with unassigned ground truth objects, unassigned track queries are concatenated with the proposal queries.
• Effect: Unassigned track queries get a "second chance" to be matched to untracked ground truth objects.
  • If a track query is dropped (via Track Query Dropout) but still holds valid information about an object, it can now be assigned to that object instead of being ignored.
  • This ensures track queries receive supervision for both track continuation and track initialization, reducing the reliance on potentially weak proposal queries for new instances.

3. Key Contributions

1. Identification of New Instance Suppression: The paper systematically identifies and analyzes the root cause of the performance gap between LiDAR-TBA and LiDAR-TBD, attributing it to the conflict between detection and tracking objectives.
2. Track Query Dropout: A novel training strategy that diversifies query configurations, teaching the model to handle missing track queries by activating proposal queries effectively.
3. Second Chance Assignment: A ground truth assignment strategy that allows unassigned track queries to initialize new tracks, balancing supervision and reducing false negatives.
4. State-of-the-Art Performance: Demonstrates that targeted training strategies, rather than architectural complexity, can bridge the gap between TBA and TBD.

4. Experimental Results

Experiments were conducted on the nuScenes benchmark (tracking and detection classes).

• Performance Metrics:
  • AMOTA (Average Multi-Object Tracking Accuracy): SCATR achieves 65.0% on the test split, outperforming the previous SOTA LiDAR-TBA method (JDT3D) by 7.6%.
  • False Negatives (FN): Reduced by 26% compared to JDT3D.
  • ID Switches (IDS): SCATR achieves the lowest ID switches among TBA methods and reduces them by 69.1% compared to TBD baselines (SimpleTrack), proving superior track consistency.
  • mAP: Achieves 64.1% on the test split, significantly closing the gap with TBD methods.
• Ablation Studies:
  • Second Chance Assignment: Significantly reduces FN and ID switches without increasing False Positives.
  • Track Query Dropout: Alone, it does not improve performance due to assignment ambiguities, but combined with Second Chance Assignment, it yields the best results.
  • Vision Domain: The methods were also validated on vision-based TBA (using ResNet-50), showing similar improvements, indicating the strategies are generalizable.
• Comparison with TBD: While SCATR still slightly trails behind the best TBD combinations (e.g., FocalFormer + SimpleTrack) in raw detection mAP, it offers superior end-to-end tracking robustness with significantly fewer ID switches and FNs.

5. Significance

• Bridging the Gap: SCATR successfully narrows the long-standing performance gap between LiDAR-based TBA and TBD, proving that end-to-end tracking is viable for LiDAR without sacrificing accuracy.
• Training over Architecture: The work demonstrates that training strategies (dropout and assignment logic) are more critical than increasing model complexity for solving specific failure modes in TBA.
• Future Impact: By enabling robust handling of newborn instances and occlusions, SCATR paves the way for more reliable, unified end-to-end perception systems for autonomous driving and robotics, fully exploiting temporal information in sequential point cloud data.

Code Availability: The implementation is open-sourced at https://github.com/TRAILab/SCATR.