💻 computer science

Robust Multimodal Dynamic Object Segmentation

This paper presents a robust multimodal dynamic object segmentation framework that integrates 2D point tracks, 3D reconstruction, and semantic information via a Transformer-based network and a novel point-query SAM post-processing method to achieve precise, consistent segmentation and high-quality static scene reconstruction, outperforming existing state-of-the-art approaches.

Original authors: Zhe Xin, Hanzhi Chang, Penghui Huang, Yinian Mao, Guoquan Huang

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhe Xin, Hanzhi Chang, Penghui Huang, Yinian Mao, Guoquan Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect, frozen diorama of a busy city street, but the video you are using to build it is full of people running, cars zooming by, and birds flapping their wings. If you just try to freeze the whole scene, the moving people will turn into blurry ghosts, ruining your masterpiece. To fix this, computers need a special skill called "dynamic object segmentation." Think of this as a super-powered pair of scissors that can instantly cut out every moving thing in a video, leaving only the still, solid background behind.

For a long time, computers tried to do this by looking at two things: how pixels move across the screen (like watching a crowd of ants scurry) or by trying to rebuild the 3D shape of the world from scratch. But these methods had flaws. The "pixel movement" approach often got confused at the edges of objects, cutting off a person's arm but keeping their head. The "3D shape" approach was like trying to build a sandcastle during a storm; if the wind (or camera shake) messed up the measurements even a little bit, the whole structure collapsed. Scientists needed a way to combine the best of both worlds—tracking movement, measuring depth, and understanding what things are—to make a clean cut every time.

This is where the new paper, "Robust Multimodal Dynamic Object Segmentation," steps in with a clever solution. The authors, working with data from Meituan UAV and the University of Delaware, propose a system that acts like a detective team, bringing together three different types of clues to solve the mystery of what is moving and what is still. Instead of relying on just one type of evidence, their method uses a "multimodal" approach, combining 2D point tracking (following dots as they move), 3D reconstruction (guessing the depth of the scene), and semantic information (knowing that a "dog" is an object that usually moves).

The core of their invention is a smart network that acts like a traffic controller. It takes all these different clues and runs them through a "Transformer" (a type of AI brain good at connecting dots) and a "clustering" module. This allows the system to look at a scene and say, "Okay, in this part, the 3D depth clues are shaky, so let's trust the movement clues more," or "Here, the movement is confusing, so let's listen to the semantic clues that tell us this is a person." By letting the system decide which clue to trust based on the specific situation, it avoids the mistakes that happen when you rely on just one method.

However, even with a great detective, the initial sketch might be a bit messy. The paper introduces a second, very creative trick to clean up the edges. Previous methods tried to fix the messy sketch by handing the whole blurry shape to a tool called SAM (Segment Anything Model) and asking, "Is this one big object?" But in the real world, a single blurry blob might contain three different dogs running in different directions. The authors realized that feeding the whole blob to the tool was a bad idea. Instead, they developed a "point-query" strategy. Imagine taking a single pixel from the messy blob and asking the tool, "Is this specific dot part of a dog?" If the tool says yes and the shape looks right, they keep it. They repeat this process, dot by dot, until the messy blob is perfectly sliced into individual, clean objects.

The results of this approach are quite impressive. When tested on several datasets, including the synthetic PointOdyssey and real-world DAVIS2017 videos, their method achieved state-of-the-art performance. On the PointOdyssey dataset, their model reached an accuracy of 93.69% (with the extra cleaning step), beating previous top methods like DAS3R, which scored 92.22%. They also showed that their method is much faster, taking only 0.2833 seconds to process a frame, compared to over 27 seconds for some other advanced models.

The paper explicitly argues against relying on single-modality cues (just flow or just 3D) and against the old way of using SAM, where you treat a whole mask as a single object. They found that these older methods fail when objects are occluded (hidden) or when the camera moves in complex ways. Their experiments suggest that by combining the clues and using the point-by-point cleaning method, they can handle these tricky scenarios much better. While the method is highly effective, the authors note that it still faces challenges when dynamic objects dominate the entire image, a situation where the background clues become too scarce to work with.

Ultimately, this work doesn't just make better video masks; it enables better 3D reconstruction. By accurately removing the moving "ghosts," the system can rebuild the static world behind them with high clarity, achieving a PSNR (a measure of image quality) of 30.60 on the DAVIS dataset, which is higher than any other method tested. It's a step forward in teaching computers to see the world not just as a flat movie, but as a dynamic, 3D place where they can separate the actors from the stage.

Technical Summary: Robust Multimodal Dynamic Object Segmentation

Problem Statement

Dynamic object segmentation is a critical prerequisite for applications such as static scene reconstruction from dynamic videos, autonomous robotics, and AR/VR. The core challenge lies in accurately separating moving objects from the background in real-world scenarios characterized by complex object and camera motions, environmental interferences, and scale variations.

Existing approaches generally fall into two paradigms, both of which exhibit significant limitations:

2D Optical Flow-Based Methods: While these methods estimate pixel-level movements, they often fail to ensure consistent static/dynamic segmentation within object boundaries. Furthermore, their reliance on 2D data makes it difficult to decouple object motion from camera motion.
3D Reconstruction-Based Methods: These approaches derive dynamic masks from depth maps or 3D point clouds. However, they are highly sensitive to reconstruction errors (e.g., depth estimation inaccuracies) and can produce unreliable results when dynamic objects violate epipolar constraints.

Additionally, current state-of-the-art methods often rely on post-processing with the Segment Anything Model (SAM). A common flaw in these implementations is treating an entire dynamic mask as a single query to SAM, assuming all masked pixels belong to one object. This assumption frequently fails in real-world scenarios involving multiple moving objects, leading to incomplete or incorrect segmentation.

Methodology

The authors propose a unified framework that integrates multimodal cues—specifically 2D point tracks, 3D reconstruction, and semantic information—to generate precise and complete dynamic masks. The architecture consists of three primary stages:

1. Multimodal Feature Extraction

The system ingests a dynamic video clip and extracts three distinct types of features:

3D Reconstruction: Utilizing MonST3R, the system generates per-frame depth maps, camera intrinsic/extrinsic parameters, and attention maps. The attention maps, derived from the ViT decoder, provide valuable cues for identifying dynamic objects.
2D Tracking: Using CoTracker, the system samples 2D points from the first frame and tracks their trajectories across the video sequence.
Semantic Information: SAMv2 is employed to extract per-frame semantic feature maps.

2. Dynamic Mask Estimation via Trajectory Classification

Instead of processing frame-by-frame, the method converts input features into point-based trajectories. For each track $k$ , the system constructs a feature vector $F_{traj}$ by concatenating:

2D pixel positions and their offsets ( $\Delta x$ ).
Depth values and their offsets ( $\Delta d$ ).
Camera parameters (focal length, transformation matrices between frames).
Interpolated attention values and semantic features.

The classification pipeline involves:

Semantic Clustering: A point-based module ( $\Phi_{sem}$ ) uses differentiable pooling and CNNs to cluster semantic features, capturing global context within clusters and refining local features.
Transformer Integration: The geometric trajectory features ( $F_{traj}$ ) and the refined semantic features are concatenated and fed into a Transformer encoder. This allows the model to learn interactions both within individual trajectories and across the entire set of trajectories.
Aggregation and Classification: A decoder aggregation network further clusters and fuses these features to output motion confidence scores. A sigmoid function determines the final static/dynamic classification for each trajectory.

3. Point-Query-Based SAM Post-Processing

To address the limitation of single-object assumptions in standard SAM post-processing, the authors introduce an iterative point-query strategy:

All pixels labeled as dynamic in the coarse mask form a set $X_{dyn}$ .
The system randomly samples a single pixel $x$ from $X_{dyn}$ as a point query for SAM.
The resulting mask $M_x$ is evaluated against the coarse mask based on overlap ratio and pixel count.
If valid, $M_x$ is added to the final output, and its constituent pixels are removed from $X_{dyn}$ .
This process iterates until $X_{dyn}$ is empty, effectively handling multiple objects within a single initial mask.

Key Contributions

The paper outlines three primary contributions:

Multimodal Integration Framework: A novel network that unifies 2D point tracking, 3D reconstruction (depth and camera poses), and semantic information. This integration allows the model to adaptively determine which feature modality should dominate based on scene characteristics, mitigating the impact of feature degradation (e.g., poor optical flow or depth estimation).
Point-Query-Based SAM Refinement: A new post-processing methodology that treats individual pixels as queries rather than entire masks. This significantly improves performance in complex scenarios with multiple moving objects, overcoming the limitations of previous mask-based SAM approaches.
Comprehensive Validation: Extensive experiments demonstrating state-of-the-art (SOTA) performance in both dynamic object segmentation accuracy and the quality of static scene reconstruction.

Experimental Results

The method was evaluated on three datasets: PointOdyssey (synthetic), DAVIS2017 (real-world), and Sintel (synthetic, highly dynamic).

Segmentation Accuracy: On PointOdyssey, the proposed method achieved an accuracy of 93.70% and a recall of 81.42%, outperforming SOTA baselines like DAS3R (92.22% Acc) and Easi3R (88.17% Acc). On DAVIS2017, the method with SAM refinement reached 94.01% accuracy and 93.14% recall, demonstrating superior generalization to real-world scenes.
Static Scene Reconstruction: When integrated with 3D Gaussian Splatting (3DGS) for static scene reconstruction, the method achieved the highest PSNR (30.60) and SSIM (0.933) on the DAVIS2017 DAS3R split, surpassing MonST3R, Easi3R, and DAS3R.
Efficiency: The proposed model is significantly more efficient than baselines, with a parameter count of 2.8M and an inference time of 0.28s per clip, compared to hundreds of millions of parameters and tens of seconds for methods like MonST3R and DAS3R.

Significance and Claims

The authors claim that their framework addresses the fundamental limitations of relying on single-modal cues by leveraging the complementary strengths of geometric (3D), motion (2D tracking), and semantic data. By explicitly modeling feature trajectories and introducing a robust point-query post-processing step, the method achieves a balance between precision and completeness that existing methods lack.

The paper modestly acknowledges that while the method performs strongly across diverse benchmarks, it still faces challenges in scenarios where dynamic objects dominate the image frame. The authors identify addressing this limitation through more diverse training data and enhanced model refinements as a key focus for future work. The work is supported by the Shenzhen Science and Technology Program.