ShotFinder: Imagination-Driven Open-Domain Video Shot Retrieval via Web Search

This paper introduces ShotFinder, a new benchmark and a three-stage retrieval pipeline that addresses the gap in open-domain video shot retrieval by formalizing editing requirements into keyframe-oriented descriptions with five controllable constraints, revealing that current multimodal models still struggle significantly with complex attributes like color and visual style compared to human performance.

The Gist

🎬 The Big Idea: Finding a Needle in a Haystack of Movies

Imagine you are a video editor working on a movie. You need a very specific clip: "A long-haired woman sitting at a table, leaning forward, looking sad, with warm sunset lighting and jazz music playing."

In the old days, you'd have to scroll through hours of footage, hoping to find that exact moment. Today, we have super-smart AI (Large Language Models) that can read and understand the world. But can they find that specific 5-second clip inside a 2-hour movie just by reading your description?

The Answer: Not really. Not yet.

This paper introduces ShotFinder, a new "test" (benchmark) and a new "method" to see how good AI really is at this specific task.

---

🧪 Part 1: The Test (The "ShotFinder" Benchmark)

The researchers realized that while AI is great at finding text or static images, it's terrible at finding specific moments in videos. So, they built a giant challenge course for AI.

The Course:
They created 1,210 specific challenges. Each challenge is a detailed description of a video clip (a "shot") taken from YouTube.

• The Description: "A man in a blue suit running up stairs."
• The Constraints: They added extra rules to make it harder, like:
  • Time: "It must happen after he falls down."
  • Color: "The whole scene must look very warm and orange."
  • Style: "It must look like a 1990s cartoon."
  • Sound: "You must hear a dog barking."
  • Quality: "It must be high-definition (1080p)."

The Goal: The AI has to go to the internet (like YouTube), find the right video, and then pinpoint the exact second that matches the description.

The Result: The AI struggled. Even the smartest models (like GPT-5 or Gemini) only got about 25% to 27% of the answers right. Humans, on the other hand, got about 88% right. It turns out, finding a specific mood or color in a video is still very hard for computers.

---

🛠️ Part 2: The Solution (The "Imagination" Method)

Since the AI was failing, the authors built a new way to help it. They call it ShotFinder (the method).

Think of this method as a three-step detective process:

Step 1: The "Imagination" Phase (The Dreamer)

If you ask an AI to search for "a sad woman at a table," it might just search for those words. But video titles are weird. A video might be titled "How to make sad coffee" or "Cinematic rain scenes."

• The Trick: The AI is told to imagine the whole movie first. It asks itself: "If this sad woman were in a movie, what would the title of that movie be? What genre is it?"
• The Analogy: Instead of looking for a specific brick, the AI imagines the whole house the brick belongs to, then looks for the house. This helps it find the right video library.

Step 2: The "Retriever" Phase (The Scavenger)

Once the AI has its "imagined" search terms, it goes to the internet (YouTube) and downloads a bunch of candidate videos that might contain the clip.

Step 3: The "Localizer" Phase (The Sniper)

Now the AI has a pile of videos. It needs to find the exact 5-second clip.

• It breaks the video down into many small snapshots (frames).
• It compares the snapshots to your description.
• The Sniper: It picks the one frame that matches your description perfectly and says, "This is it!"

---

📉 What Did They Learn? (The Takeaways)

After running the test, the researchers found some interesting things:

1. AI is getting better, but it's still a rookie: The best AI models are far behind human editors. They miss subtle details like "sadness" or "warm lighting."
2. Some things are easier than others:
   • Easy: Finding things based on Time (e.g., "after the explosion"). AI is okay at this.
   • Hard: Finding things based on Color or Art Style. AI is terrible at telling the difference between "warm sunset" and "cool morning."
3. Bigger isn't always better: Sometimes a smaller, smarter AI model did just as well as a massive, expensive one. It's not just about how big the brain is; it's about how it's wired.
4. The "Imagination" helps: The method that used "Video Imagination" (Step 1) worked much better than just searching for keywords. It proved that AI needs to understand the context of a video, not just the words.

🚀 Why Does This Matter?

This isn't just a game. This technology is the future of video editing.

• Imagine a future where you tell your computer: "Show me the clip where the hero cries in the rain," and it instantly finds it in a library of 10,000 movies.
• Right now, that's science fiction. ShotFinder is the first step to making it a reality, showing us exactly where the AI is failing so we can fix it.

In short: We taught AI to dream up the right search terms, but it still needs a lot of practice to find the perfect video clip.

Technical Summary

1. Problem Definition

The paper addresses the gap in Open-Domain Video Shot Retrieval, a task where a system must locate a specific continuous sequence of frames (a "shot") within a vast, uncurated pool of online videos based on a natural language description.

• The Challenge: Unlike text retrieval or static image search, video shot retrieval involves:
  • Temporal Structure: Understanding the sequence of events and the specific timing of a shot within a longer video.
  • Semantic Complexity: Bridging the gap between a specific "shot-level" description (e.g., "a woman leaning forward at a table") and "video-level" metadata (titles, tags, descriptions) found on platforms like YouTube.
  • Multi-Modal Constraints: Real-world editing often requires specific constraints beyond visual content, including Temporal order (before/after a specific event), Color tone, Visual style (e.g., 2D animation vs. live-action), Audio (dialogue vs. music), and Resolution.
• Current Limitations: Existing benchmarks focus on text or static multimodal retrieval. Current Multimodal Large Language Models (MLLMs) struggle to perform "Video Imagination" (inferring the full video context from a shot) and lack systematic evaluation for open-domain retrieval with fine-grained constraints.

2. Methodology

The authors propose ShotFinder, which consists of two main components: a new benchmark and a retrieval pipeline.

A. The ShotFinder Benchmark

• Dataset Construction:
  • Scale: 1,210 high-quality samples curated from YouTube.
  • Topics: Covers 20 thematic categories (e.g., Tech, Sports, Animation, Food).
  • Generation Pipeline: Uses LLMs (Gemini-3-Pro) to generate shot descriptions and constraints, followed by rigorous human verification by 10 researchers to ensure semantic accuracy and constraint consistency.
• Task Design:
  • Base Task: Retrieve a shot matching a description.
  • Controlled Constraints: Introduces five single-factor constraints to isolate difficulty: Temporal, Color, Visual Style, Audio, and Resolution.
  • Strategy: Uses a "constraint-aware topic allocation" strategy to ensure specific topics (e.g., "Fashion" for Color, "Sports" for Temporal) are naturally suited to the constraints, reducing noise while maintaining diversity.

B. The ShotFinder Retrieval Pipeline

The proposed method is a text-driven, three-stage pipeline:

1. Stage 1: Query Expansion via Video Imagination (Generator)
   • Instead of directly using the shot description as a search query (which often fails due to granularity mismatch), the model performs "Video Imagination."
   • It infers the likely full video context (theme, narrative, title style) that would contain the specific shot.
   • It generates high-level search keywords based on this imagined context to bridge the gap between shot-level details and video-level metadata.
2. Stage 2: Candidate Video Retrieval (Retriever)
   • The generated keywords are used to query a search engine (YouTube).
   • The system filters and downloads accessible candidate videos to form a local pool.
3. Stage 3: Description-Guided Temporal Localization (Localizer)
   • The system employs adaptive frame sampling based on video duration.
   • An MLLM analyzes the sampled frames (and audio, if applicable) against the original description and constraints.
   • It identifies the specific timestamp/frame that matches the query.
   • A verification step ensures the selected frame aligns with the ground truth.

3. Key Contributions

1. First Open-Domain Video Shot Benchmark: ShotFinder is the first benchmark to formalize open-domain video shot retrieval with controllable, single-factor constraints (Temporal, Color, Style, Audio, Resolution).
2. Imagination-Driven Retrieval Method: Proposes a novel pipeline that uses "Video Imagination" to expand queries, effectively solving the semantic gap between specific shots and general video titles.
3. Comprehensive Evaluation: Provides a detailed analysis of both closed-source (GPT-4/5, Gemini, Claude) and open-source (Qwen3) models, establishing a new baseline for video retrieval capabilities.

4. Experimental Results

The authors evaluated human performance against various state-of-the-art models:

• Human vs. Model Gap: There is a significant performance gap. Human accuracy averages 88.5%, while the best closed-source model (GPT-5.2) achieves only 26.9%.
• Constraint Imbalance:
  • Temporal: The most tractable category for models (models perform relatively well here).
  • Color & Visual Style: Major challenges. Models struggle significantly with fine-grained color tones and distinguishing between live-action and specific animation styles.
  • Audio: Models perform moderately but lag behind humans.
• Model Performance:
  • Closed-Source: GPT-5.2 leads overall, showing strength in Temporal and Style but weakness in Color.
  • Open-Source: The Qwen3 series (e.g., Qwen3-Omni-30B) is competitive, particularly in Temporal and Color, but still trails closed-source models in Style.
• Ablation Studies:
  • Increasing the number of search queries and candidate videos improves performance, especially for complex constraints like Style and Audio.
  • Denser frame sampling helps Temporal localization but does not fully resolve Color/Style challenges, indicating these require better visual understanding rather than just more data.

5. Significance and Future Directions

• Practical Impact: This research directly addresses real-world video editing workflows, where editors need to find specific clips based on vague memories or complex constraints without manually scrubbing through hours of footage.
• Scientific Insight: The results highlight that while MLLMs have advanced in text and static image retrieval, open-domain video retrieval remains a critical unsolved capability. The "Video Imagination" step is crucial for bridging the semantic gap.
• Limitations & Future Work:
  • Current evaluation uses keyframes as a proxy for full video shots; future work will explore full-video processing.
  • The benchmark currently focuses on single-factor constraints; multi-factor narrative constraints are a future direction.
  • The system relies on single-turn interactions; multi-turn conversational refinement is planned.

In conclusion, ShotFinder establishes a rigorous standard for evaluating how well AI can "find a needle in a haystack" within the open web of video content, revealing that current models are far from human-level proficiency in this domain, particularly regarding aesthetic and stylistic nuances.