Technical Summary: SF20K Competition 2025

Problem Definition

The Short-Films 20K (SF20K) Competition addresses a critical gap in video-language understanding: the transition from short-clip action recognition to story-level, long-form narrative comprehension. Existing benchmarks often suffer from "memorization," where large language models (LLMs) answer questions about popular films using parametric knowledge rather than genuine visual understanding.

To counter this, the competition utilizes a novel corpus of 20,143 amateur short films, ensuring models cannot rely on pre-trained internal knowledge of famous movies. The core task is Open-Ended Video Question Answering (VideoQA), requiring models to generate free-form natural language responses based on the full duration of a film (average 12 minutes). Success demands:

• Narrative-level comprehension: Tracking characters across scenes and causal chains.
• Multimodal grounding: Synthesizing visual and textual information over extended timescales.
• Robustness: Avoiding data contamination by testing on films released during the competition period (July–October 2025).

Methodology and Evaluation Setup

Competition Tracks

The competition was structured into two tracks to explore the trade-off between raw performance and efficiency:

1. Main Track: Unrestricted model size and compute.
2. Special Track: Limited to models with fewer than 8 billion parameters.

Evaluation Metric

Traditional n-gram metrics (e.g., BLEU, CIDEr) were deemed insufficient for open-ended responses. The competition adopted LLM-QA-Eval, an automated judge using GPT-4.1-nano.

• Process: The judge compares a predicted answer against the ground-truth answer given a specific question.
• Scoring: It assigns a binary correctness label (0–1) and a qualitative score (1–5).
• Final Metric: The leaderboard is ranked by the average correctness across 979 question-answer pairs in the private test set.

Baselines and Human Performance

• Blind Guessing: An LLM guessing without video context achieved only 14.7% accuracy, confirming the necessity of visual grounding.
• State-of-the-Art (SOTA): End-to-end models like Gemini-2.5-Pro reached 59.7%.
• Human Ceiling: Manual annotations by humans achieved 91.7%, establishing a significant gap for current AI systems.

Key Findings and Results

The competition attracted 22 teams and 286 submissions. The winning team, WXYZ, achieved 65.7% on the Main Track and 48.7% on the Special Track.

1. Narrative-Aware Sampling Outperforms Uniform Sampling

The most significant finding is that shot-level, narrative-aware processing is superior to standard uniform frame sampling.

• Winning Strategy (WXYZ): The team structured their pipeline around video shots as fundamental reasoning units. Their system included:
  • Two-stage keyframe selection.
  • Narrative rhythm detection (using audio energy, speaker speed, shot duration, and optical flow).
  • Shot-level story description agents.
  • Multi-stage chain-of-thought (CoT) reasoning with LLM verification.
• Efficiency vs. Scale: WXYZ achieved 65.7% using a 7B parameter model (Qwen2.5-VL-7B), matching the performance of sudook, which used a 235B parameter model (Qwen3-VL-235B). This suggests that information organization and reasoning structure are more critical than raw model capacity.

2. Multi-Stage Pipelines Compensate for Model Size

Well-designed pipelines using smaller models can match or exceed end-to-end inference with much larger models.

• Team BASELINE: Used a "perceive then reason" paradigm, separating visual perception (Qwen2.5-VL-7B for captioning) from reasoning (GPT-5 for answer generation), securing 3rd place.
• Team eloral-wxy: Achieved 2nd place in the Special Track using a zero-shot, API-based pipeline (GLM for summarization + GPT for refinement).

3. Subtitle Quality is a Dominant Factor

High-quality subtitle extraction proved to be a critical bottleneck.

• Teams that replaced the official Whisper-tiny subtitles with custom extractions (e.g., Whisper-large-v3 with deduplication) saw significant gains.
• Team R&T demonstrated that improved subtitles boosted their models' public benchmark scores from ~44% to ~55%.
• Teams WXYZ and eg utilized dual ASR systems (Whisper + Parakeet) to adapt to varying audio conditions.

4. Dense Frame Sampling Requires Better Temporal Encoding

Simply increasing the number of input frames does not guarantee better performance.

• Standard models (e.g., Qwen2.5-VL-7B) showed performance degradation when frame counts increased from 128 to 256, likely due to difficulty processing redundant visual information.
• Models with specialized temporal architectures (e.g., Keye-VL's slow-fast architecture, Qwen3-VL's MRoPE-Interleave) handled higher frame counts more effectively.

5. Fine-Tuning Yields Modest Gains

Fine-tuning on the limited SF20K training set (500 samples) provided only moderate improvements.

• Architectural innovations (shot-aware pipelines, CoT verification) contributed more to final performance than fine-tuning alone.
• This suggests that the current bottleneck lies in information selection and reasoning structure rather than the ability to learn task-specific patterns from small datasets.

Significance and Conclusion

The SF20K Competition demonstrates that the primary bottleneck in long-form video QA is not merely raw model capacity, but rather information selection, reasoning structure, and data quality.

• Pipeline Design Matters: The results indicate that a well-engineered pipeline using a 7B model can outperform or match massive 200B+ models, offering an encouraging direction for resource-constrained research.
• The Human-Machine Gap: Despite progress, the best system (65.7%) remains 26 percentage points below human performance (91.7%). The remaining challenges involve long-range causal reasoning, identifying characters across distant scenes, and integrating subtle visual cues (facial expressions, environmental changes) with dialogue.
• Future Direction: The competition establishes that future advancements in long-form video comprehension require innovations in temporal reasoning, narrative structure modeling, and multimodal integration, rather than simply scaling model parameters.

The SF20K benchmark remains a challenging and informative testbed for advancing story-level video understanding beyond the limitations of current architectures.