Decomposing Queries into Tool Calls for Long-Video Keyframe Retrieval
The paper introduces ToolMerge, a novel keyframe retrieval method that leverages an LLM to decompose queries into specific tool calls and merge their results via boolean operators, demonstrating competitive performance on the newly proposed Molmo-2 Moments benchmark.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a 4-hour long movie, and someone asks you a very specific question about it, like, "What was the man in the yellow jacket doing right before he crossed the river?"
If you tried to answer this by watching the whole movie from start to finish, it would take forever. If you just picked a few random frames (like flipping through a photo album), you'd likely miss the exact moment. This is the problem the paper tackles: How do we find the exact few seconds of a long video that hold the answer, without watching the whole thing?
The authors propose a new system called ToolMerge. Here is how it works, explained through simple analogies.
1. The Problem with Old Methods
Previous systems tried to solve this in two ways, both of which had flaws:
- The "One-Size-Fits-All" Approach: They treated the whole question as one giant search term. It's like asking a librarian, "Find me the page with the man in the yellow jacket crossing the river." The librarian might find a page with a river, or a page with a man, but not necessarily the right combination.
- The "Rigid Checklist" Approach: They tried to break the question down into a fixed list of objects (e.g., "Find man," "Find jacket," "Find river"). But real life is messy. Sometimes you need to look for an action, sometimes an object, and sometimes a specific color. A rigid checklist misses the nuance.
2. The Solution: ToolMerge (The Smart Detective)
ToolMerge acts like a smart detective who knows how to use different tools to solve a case. Instead of trying to find the answer in one giant leap, the detective breaks the job down.
Step 1: The Planner (The Brain)
First, a "Planner" (an AI brain) reads the question. Instead of just searching for the whole sentence, it breaks the question into smaller, specific tasks.
- Analogy: If the question is "What did the man in the yellow jacket do?", the Planner doesn't just shout "Yellow Jacket!" into the void. It says, "Okay, I need three things: 1. Find a scene with a river. 2. Find a man wearing yellow. 3. Find someone crossing water."
Step 2: The Tools (The Specialists)
The Planner sends these small tasks to different "specialist" tools.
- Tool A (SigLIP): This is the Scene Specialist. It looks at the whole picture to find general vibes (e.g., "a river," "a canyon").
- Tool B (T-REN): This is the Object Specialist. It zooms in to find specific things (e.g., "a man," "a yellow jacket").
- Tool C (OCR): This is the Text Reader. It reads any words written on signs or screens in the video.
Step 3: The Merge (The Judge)
Now, each tool has ranked every single frame of the video based on how well it matches its specific task. The "Merge" step combines these rankings.
- The "AND" Rule: If the question requires both a river and a man, the system looks for frames where both tools gave a high score. It's like a Venn diagram; the answer must be in the middle where the circles overlap.
- The "OR" Rule: If the question asks "Did he cross the river OR the bridge?", the system looks for frames that satisfy either condition.
Finally, the system picks the top 3 to 8 frames that scored the best across all these combined rules and hands them to a final "Answerer" AI to give the actual answer.
3. The New Test: Molmo-2 Moments (M2M)
To prove their system works, the authors realized existing tests were unfair. Old tests had questions that could be answered by guessing or by looking at random parts of the video.
They built a new test called Molmo-2 Moments (M2M).
- The Analogy: Imagine a teacher who writes a test based only on a specific 10-second clip of a movie. The teacher knows exactly what happens in those 10 seconds. If a student answers correctly, they must have watched that specific 10 seconds. They couldn't have guessed it from the rest of the movie.
- This ensures that if the system gets the answer right, it actually found the right frames.
4. The Results
When they tested ToolMerge:
- It was better at finding the right frames than previous methods, especially for complex questions involving multiple objects or actions.
- It performed particularly well on caption retrieval. If you give the system a long, detailed description of a scene (like a caption), ToolMerge is much better at finding the exact video clip that matches that description compared to older methods.
- They also showed that if you "train" the Planner a little bit using a specific reward system (GRPO), it gets even better at knowing which tools to use.
Summary
ToolMerge is a system that doesn't try to guess the answer to a long video question all at once. Instead, it acts like a project manager: it breaks the question into small, specific tasks, sends those tasks to different specialized tools, and then combines the results to find the exact few seconds of video that hold the truth. They proved it works by creating a new, stricter test where the answer is locked to a specific moment in time.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.