DynEoMT: Learning Object Dynamicity from Online Segmentation Queries
DynEoMT is an online framework that augments query-based video segmentation to predict region-level object dynamicity (moving vs. static) without requiring optical flow, depth, or camera pose, achieving strong performance through a novel offline supervision pipeline trained on camera-compensated optical flow.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of computer vision, machines are becoming remarkably good at recognizing what is in a picture. They can identify a car, a person, or a tree and draw a precise outline around it. But there is a subtle difference between seeing an object and understanding how it moves. When a camera moves through a scene, everything in the view appears to shift. A parked car might seem to slide across the screen simply because the camera is turning, while a walking pedestrian moves for their own reasons. For a machine to truly understand a scene, it must distinguish between objects that are moving on their own and objects that are only appearing to move because the camera is moving. This ability to tell the difference is crucial for robots that need to navigate the world or for systems that build maps of their surroundings. If a robot mistakes a stationary building for a moving obstacle, or ignores a real moving car because it thinks the building is moving, its understanding of the world breaks down.
Researchers have long taught computers to track objects over time, keeping a consistent label on a specific car or person as it moves through a video. However, these systems typically stop at identifying the object and its location; they do not explicitly state whether that object is moving independently or is just a static part of the background. A new study introduces a method called DynEoMT, which adds this missing layer of understanding. The system learns to look at the video frames and the data it has already gathered about objects, and then decides for each tracked region whether it is dynamic or static. The key achievement here is that the system does this without needing to calculate complex motion patterns, measure depth, or know the camera's physical position. It learns to infer the state of motion directly from the way it tracks the object itself.
To teach a computer this skill, the researchers first had to create a way to label the data, since existing video datasets did not include this specific information. They built an offline process that acts like a teacher. This process looks at pairs of video frames and calculates how pixels move between them. It then estimates how much of that movement is caused by the camera itself and subtracts that out. What remains is the "residual" motion, which represents the actual movement of objects in the world. If a region shows significant movement after the camera's motion is removed, the system labels it as dynamic. If it shows little to no movement, it is labeled as static. The researchers applied this logic to four major video datasets, covering everything from semantic segmentation, where every pixel is given a category, to instance segmentation, where individual objects are tracked separately. This created a massive set of training examples where every object mask came with a label indicating whether it was moving or still.
With this new training data, the researchers modified an existing video segmentation model. They added a small, lightweight decision-making component, or "head," to the system. This component looks at the internal representation of each object that the model is tracking and predicts whether it is moving or stationary. Crucially, this prediction happens in real-time as the video plays. The system only uses the current image and the information it carried over from the previous frame. It does not look back at old images, it does not calculate optical flow, and it does not require any extra sensors. The model learns to make this prediction alongside its primary job of drawing outlines around objects, ensuring that the new task does not interfere with its ability to see and track.
The results show that this approach works effectively across different types of video data. On a large dataset of video panoptic segmentation, the system correctly identified the motion state of objects with an accuracy of 84.3 percent. On other datasets focused on tracking individual instances, the accuracy ranged from 68.0 to 87.6 percent. Perhaps most importantly, adding this new capability did not hurt the system's original performance. The ability to draw accurate masks and keep track of object identities remained almost exactly the same as before. The researchers found that the internal signals the model used to track objects already contained enough information to determine if those objects were moving. By simply adding a small layer to interpret those signals, the system gained a new understanding of the world without needing a separate, complex motion-processing pipeline.
This work demonstrates that the distinction between a moving object and a static one can be learned directly from the way a model tracks objects over time. It suggests that future systems designed for robotics or autonomous navigation could gain a more robust understanding of their environment without the heavy computational cost of dedicated motion analysis tools. The method relies on a straightforward principle: if you can track an object, you can likely tell if it is moving on its own. The researchers have made their code available to the public, allowing others to reproduce these findings and build upon a system that sees not just what is there, but how it behaves.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.