ID-VTG: Image-Disambiguated Video Temporal Grounding
This paper introduces ID-VTG, a novel task and benchmark for video temporal grounding that leverages reference images alongside text queries to resolve ambiguities between visually similar events, accompanied by the proposed VGD-Agg framework which utilizes a dual-branch architecture and specialized learnable tokens to achieve state-of-the-art performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast library of video content that fills our screens every day, there is a persistent challenge for computers: finding the exact moment a specific event happens. This task, known as video temporal grounding, asks a machine to watch a long clip and point out the precise start and end times of a scene described in words. For years, researchers have trained systems to do this using only text. If a user types "the man is speaking into a microphone," the computer scans the footage to find that action. However, this approach hits a wall when the video contains multiple people or things that look nearly identical. If two different men in the same uniform both speak into microphones at different times, a text description alone cannot tell the computer which one the user means. The words are ambiguous, and the machine gets lost, often guessing the wrong segment or failing to find the right one at all.
To solve this, researchers at Peking University have introduced a new way of thinking about the problem. They realized that in the real world, when people are unsure which person or object they are talking about, they often have a picture to show. A security guard might have a photo of a suspect; a sports fan might have a snapshot of a specific athlete. By combining a text description with a reference image, the ambiguity disappears. The image acts as a strict visual filter, telling the computer exactly which individual to watch for, while the text describes what that individual is doing. This new approach, which the team calls Image-Disambiguated Video Temporal Grounding, shifts the goal from guessing based on words to pinpointing based on a visual match.
The researchers did not just propose a theory; they built the tools necessary to test it. They created two new collections of video data designed specifically to be difficult for standard text-only systems. The first collection focuses on gymnastics, where athletes wear identical uniforms and perform similar routines. In these videos, the same move, like a giant circle on the uneven bars, is performed by different people or by the same person multiple times. The text might say "the gymnast performs a circle," but without a photo, the computer cannot know which gymnast or which attempt is being described. The second collection is much broader, drawn from a wide variety of internet videos featuring animals, fictional characters, and everyday objects. Here, the challenge comes from different people performing similar actions, such as two different individuals tying their shoelaces. In both collections, the researchers paired every text query with a specific reference image of the target subject, creating a dataset where the only way to get the right answer is to use both the picture and the words.
To handle this new type of task, the team developed a method that mimics how a human might approach the problem. They built a system with two distinct ways of looking at the video. One part works quickly, scanning the whole video to generate a list of potential moments where an event might be happening. This is the fast branch, which creates a broad set of candidates. The second part works slowly and carefully, comparing every single frame of the video against the reference image provided by the user. This slow branch looks for the specific visual details that match the photo, such as a unique facial feature or a specific pattern on clothing.
The core innovation of their system lies in how it uses these two parts to make a decision. The researchers introduced a special mechanism that acts like a gatekeeper. When the system looks at a potential moment in the video, it asks a simple question: does this part of the video look like the reference image? If the answer is yes, the system highlights that moment. If the answer is no, the system actively suppresses that moment, treating it as a distraction. To make this work, the system learns to recognize "hard negatives"—these are parts of the video that look very similar to the target but are actually the wrong person or object. By training the system to distinguish between the true target and these tricky look-alikes, the researchers ensured that the computer could ignore the confusing parts of the video and focus only on the correct segment.
The results of this approach were tested against the new datasets and compared to existing methods that rely only on text. The new system significantly outperformed the older models. In the gymnastics videos, where the visual confusion was high, the system successfully identified the correct athlete and the correct time, whereas text-only models often failed. In the open-world dataset, which included a wide range of subjects and actions, the system maintained its accuracy even when tested on videos it had never seen before. This suggests that the method learned a general skill for matching images to video actions rather than just memorizing specific examples. The researchers also tested how the system handled poor-quality images, such as those that were blurry or had strange lighting. Even under these difficult conditions, the system remained robust, continuing to find the correct moments in the video.
This work demonstrates that adding a visual reference to a text query is not just a minor improvement but a fundamental shift in how machines can understand video. It moves the field away from the limitations of language, which can be vague or imprecise, toward a more direct form of communication where a picture clarifies the intent. By creating a framework that can effectively filter out visual distractions and focus on the specific subject of interest, the researchers have provided a path for more reliable video search and analysis. The findings suggest that for tasks requiring precision in complex, crowded scenes, the combination of a photo and a sentence is the most effective tool available, allowing computers to see the world with a clarity that text alone cannot provide.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.