Technical Summary: CineCap

Problem Definition

Cinematographic captioning aims to describe how a video is filmed using professional film-language concepts, rather than merely describing what occurs in the scene. This task requires generating unified, open-form descriptions covering six specific dimensions: camera movement, shot size, shooting angle, depth of field, composition, and subject orientation.

The authors identify three primary challenges in this domain:

1. Fine-grained Inference: Models must distinguish subtle visual cues (e.g., differentiating camera motion from subject motion, or close-up from medium close-up) to infer professional concepts.
2. Temporal Compositionality: Cinematographic patterns are often dynamic and compound; a single video may contain multiple camera behaviors evolving over time, requiring temporal localization rather than a single static label.
3. Comprehensiveness vs. Accuracy: As a dense captioning task, the output must be both factually correct and cover all relevant dimensions. Existing approaches often struggle to balance these, frequently sacrificing coverage for accuracy or vice versa.

Current multimodal large language models (MLLMs) largely address this via visual question answering (VQA) or limited camera motion recognition, failing to produce unified, multi-dimensional descriptions.

Methodology: CineCap Framework

The authors propose CineCap, a two-stage framework designed to ground caption generation in explicit visual evidence while balancing descriptive completeness and factual correctness.

1. Spatio-Temporal Anchor-Based Structured Reasoning

To address the complexity of inferring professional concepts, the framework introduces Spatio-Temporal Anchors:

• Spatial Anchors: These ground abstract film concepts in observable visual evidence. For instance, camera motion is inferred from the movement of static background references, and shot size is determined by subject scale.
• Temporal Anchors: These bind dynamic attributes to specific time intervals (e.g., $[t_s, t_e]$), allowing the model to describe compound patterns where camera behavior changes over the duration of a clip.

2. Stage 1: Supervised Fine-Tuning (SFT) with Atomic CoT

Instead of unconstrained long-form reasoning, the authors construct atomic chain-of-thought (CoT) supervision.

• Atomic Structure: Reasoning is organized into compact units. For dynamic attributes, the format is [Temporal Anchor] [Spatial Anchor] → Statement. For static attributes, direct statements are used.
• Data Construction: Starting from manually annotated video-caption pairs, the system recovers the visual evidence underpinning each description to create an "evidence-to-answer" CoT. This yields 80,000 training samples.
• Goal: This stage teaches the model to organize visual evidence into structured, multi-dimensional reasoning without relying on unnecessarily long reasoning chains.

3. Stage 2: Reinforcement Learning with GRPO

The SFT model is further optimized using Group Relative Policy Optimization (GRPO) to improve the trade-off between comprehensiveness and accuracy.

• LLM-as-a-Judge: An LLM evaluates generated captions by decomposing them into atomic statements across the six dimensions, comparing them against ground truth to calculate:
  • Comprehensiveness Score ($s_{cmp}$): The proportion of ground-truth content covered.
  • Accuracy Score ($s_{acc}$): The factual correctness of the generated content.
• Gated Coverage Reward: To prevent the model from favoring short, conservative captions (which boost accuracy but hurt coverage) or long, hallucinated ones, a gated coverage reward is introduced.
  • This reward penalizes the mismatch between the number of predicted atomic statements and ground-truth statements ($|N_{gt} - N_{pred}|$).
  • Gating Mechanism: The coverage reward is only activated if the accuracy score exceeds a threshold ($\tau$). This ensures the model prioritizes factual correctness before attempting to increase descriptive coverage.

Key Contributions

1. Novel Perspective: The paper reframes cinematographic understanding by grounding caption generation in explicit spatio-temporal visual evidence, addressing the interaction of cinematic attributes and temporal composition.
2. CineCap Framework: A unified system combining spatio-temporal anchor-based structured reasoning with a gated coverage reward mechanism. This achieves a balanced trade-off between description comprehensiveness and accuracy.
3. CineCap Bench: The construction of the first comprehensive benchmark for cinematic captioning, comprising 472 manually annotated video-caption pairs covering diverse cinematographic aspects.
4. State-of-the-Art Performance: Extensive experiments demonstrating that CineCap outperforms both proprietary (e.g., Gemini-3.1-Pro) and open-source baselines (e.g., Qwen3-VL, InternVL) across all metrics.

Experimental Results

Evaluated on the CineCap Bench, the model achieves significant improvements over strong baselines:

• Overall Performance: CineCap achieves an overall F1 score of 73.57, surpassing the strongest baseline (Gemini-3.1-Pro) by 32.41% (52.45 vs. 73.57).
• Dimension-Specific Gains:
  • Camera Movement: F1 improved from ~40-52 (baselines) to 57.44 (average of Cmp/Acc).
  • Depth of Field: Achieved 81.29 (average of Cmp/Acc), a massive leap from baselines hovering around 47-48.
  • Camera Angle: Achieved 89.20, significantly outperforming other models.
• Ablation Studies:
  • Replacing direct caption SFT with Atomic CoT SFT improved F1 by ~1 point, confirming the value of structured supervision.
  • Adding GRPO with the proposed reward design further improved F1 by ~3.36 points.
  • The gated coverage reward was found to be superior to simple length penalties, which tended to increase completeness at the cost of accuracy.

Significance and Claims

The paper claims that CineCap establishes a new state of the art for cinematographic captioning. Its significance lies in:

• Moving beyond VQA: Shifting from constrained recognition to unified, open-form generation that captures the multi-dimensional nature of film language.
• Enabling Controllable Generation: Providing a foundation for controllable movie-quality video generation, where precise modeling of camera attributes is essential.
• Methodological Rigor: Demonstrating that combining explicit spatio-temporal reasoning with a carefully designed reward mechanism (specifically the gated coverage reward) is effective for balancing the competing objectives of accuracy and comprehensiveness in dense video captioning.

The authors conclude that their work promotes further research into cinematographic understanding and supports downstream tasks requiring controllable cinematic video generation.