TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understanding

Watching a long video to answer a specific question is a task that feels simple to a human but remains surprisingly difficult for artificial intelligence. When a person watches a film, they do not just see a stream of images; they build a mental map of events, remembering that a character picked up a key in the first act and that this key was used to open a door in the final scene. Current computer systems, however, often struggle with this kind of long-horizon understanding. They might look at a few snapshots of the video and guess the answer, or they might try to read a text summary of the plot and miss the visual details entirely. The core problem is that many questions about long videos depend on evidence scattered across different moments in time. If a system misses even one of those scattered moments, it cannot know the truth, yet it might still produce a confident, correct-looking answer simply by guessing. This gap between what a system claims to know and what it has actually seen is the central challenge researchers are trying to solve.

A team of researchers has introduced a new approach called TRACE, designed to force artificial intelligence to look at the right parts of a video before answering. Instead of relying on text summaries or guessing based on a few random frames, TRACE acts like a careful observer that builds its answer piece by piece. It starts by scanning the video to find small, specific clips, or "anchors," that seem relevant to the question. It then groups these clips together and asks the system to answer the question based only on what it has seen so far. The system does not stop there. It keeps adding new clips to its collection and re-asking the question. The process continues only until the answer stops changing, indicating that the system has gathered enough visual evidence to be certain. Crucially, the system checks its own work by replaying the final set of clips one last time to ensure the answer holds up. This method ensures that the final answer is not a guess based on a partial view, but a conclusion grounded in a complete set of visual facts.

To test whether this method actually works, the researchers created a new testing ground called VES-Bench. This is a collection of 600 questions drawn from 348 public videos, covering tasks like figuring out the order in which events happened or counting how many times a specific action occurred. Unlike previous tests that only checked if the final answer was right or wrong, VES-Bench audits the process itself. It tracks exactly which frames of the video the system looked at before making its decision. The test verifies whether the system actually saw every single moment required to answer the question correctly. If a system gets the right answer but missed a crucial scene, the audit marks it as a failure, because the answer was not truly supported by the evidence. This strict evaluation reveals that many existing methods often get lucky, answering correctly without having seen all the necessary parts of the video.

When the researchers applied TRACE to this rigorous test, the results showed a clear advantage. Using the same underlying technology as other advanced systems, TRACE answered 50.7 percent of the questions correctly while ensuring that it had seen at least two distinct frames from every single required scene. In comparison, other methods that looked at a similar number of frames often failed to cover all the necessary scenes, or they had to look at more than twice as many frames to achieve the same level of certainty. TRACE achieved this high level of accuracy while using fewer than 100 frames per question on average, a fraction of the cost required by systems that simply scan the video at a uniform rate. The study found that the key to success was not just looking at more data, but knowing when to stop looking. By waiting until the answer stabilized and then double-checking the raw visual evidence, TRACE avoided the trap of guessing based on incomplete information.

The researchers also discovered that simply increasing the number of frames a system looks at does not always lead to better answers. When they tested standard systems that looked at more and more frames, the accuracy improved only up to a point, after which adding more frames did not help. This suggests that the problem is not a lack of data, but a lack of a strategy for knowing which data matters. TRACE solved this by using a "trajectory" of evidence, where the system learns to recognize when it has gathered enough information to be sure. The study confirms that for long videos, the ability to locate and integrate specific visual moments is more important than simply processing a massive amount of video at once. By grounding answers in raw visual clips and stopping only when the evidence is complete, TRACE demonstrates a more reliable way for machines to understand the world as it unfolds over time.