Technical Summary: CaC – Advancing Video Reward Models via Hierarchical Spatiotemporal Concentrating

Problem Statement

Recent video generation models have achieved significant progress in global visual fidelity. However, when synthesizing complex dynamic content, they frequently produce sparse, subtle anomalies (e.g., structural distortions, physical-law violations, temporal inconsistencies, and corrupted text) that are often localized to small regions or brief time windows. Existing video reward models primarily emphasize overall visual quality and text alignment; as generation quality improves, these coarse-grained defects become less frequent, leaving sparse structural and temporal anomalies as a critical bottleneck. Current models struggle to isolate these weak anomaly signals from vast amounts of normal context due to a lack of fine-grained spatiotemporal perception, often resulting in global video-level reasoning that overlooks subtle hallucinations.

Methodology

The paper proposes CaC (Concentrate and Concentrate), a Vision-Language Model (VLM)-based reward model designed to detect and localize these sparse anomalies through a coarse-to-fine, two-turn hierarchical strategy.

1. Dataset Construction

To train and evaluate the model, the authors constructed the first large-scale dataset for sparse anomalies in generated videos (~30K videos). Key features include:

• Fine-grained Annotations: Per-frame bounding boxes, temporal anomaly windows, anomaly types, and attribution reasons.
• Anomaly Taxonomy: Covers single-frame recognizable anomalies (object/human distortion) and multi-frame anomalies (motion anomalies, physical violations, character anomalies).
• CoT Synthesis: A pipeline combining expert human labeling with VLM-based Chain-of-Thought (CoT) synthesis, utilizing a generate-verify process to ensure high-quality reasoning data.

2. Three-Stage Progressive Training

CaC employs a three-stage training paradigm to build hierarchical concentrating capabilities:

• Stage 1 (Single-frame Warmup): The model learns to recognize and localize anomalies from extracted anomalous frames, establishing a mapping from visual cues to spatial grounding and anomaly attribution.
• Stage 2 (Multi-frame SFT): The model is trained on both uniformly sampled global videos (for temporal scanning) and densely sampled clips (for spatial localization). This stage jointly optimizes global temporal scanning and fine-grained spatial grounding, preparing the model for the two-turn interaction format.
• Stage 3 (Two-Turn GRPO): The model is optimized using Group Relative Policy Optimization (GRPO) with a two-turn inference strategy:
  • Turn 1: Global temporal scan on a sparse frame sequence (4 fps) to anchor the anomalous time window.
  • Turn 2: Fine-grained spatial grounding on a densely sampled clip (8 fps) extracted from the predicted window to confirm anomalies and output bounding boxes.
  • Reward Mechanism: Beyond standard accuracy, the authors introduce Temporal IoU, Spatial IoU, and Attribution IoU rewards. These process-level rewards supervise the intermediate localization steps, guiding the model toward grounded, interpretable spatiotemporal reasoning. A saliency discount factor is applied to handle varying recognition difficulties.

3. Inference Strategy

The inference process mirrors the training strategy:

1. Global Scan: The input video is downsampled (4 fps) and analyzed to detect anomalies and predict a temporal window.
2. Local Concentration: The predicted window is cropped and resampled densely (8 fps). The model performs fine-grained spatial reasoning on this clip to confirm the anomaly and output specific bounding boxes.
3. Decision: If the second turn does not confirm the anomaly, the video is conservatively classified as normal.

Key Contributions

1. Dataset: Construction of the first large-scale dataset for sparse generated-video anomalies, featuring per-frame bounding boxes, temporal windows, and fine-grained attribution labels to support high-quality CoT synthesis and verifiable reward signals.
2. Model Architecture: Proposal of CaC, a coarse-to-fine two-turn reward model that utilizes hierarchical spatiotemporal focusing to detect subtle anomalies, converting structured judgments into actionable reward feedback.
3. Training Paradigm: Development of a three-stage progressive pipeline. Notably, the RL stage introduces Temporal, Spatial, and Attribution IoU rewards to guide the VLM toward interpretable anomaly CoT reasoning and stable judgments, moving beyond simple accuracy metrics.

Results

Extensive experiments demonstrate CaC's superiority in detecting sparse anomalies and improving video generation quality:

• Anomaly Detection: On the CaC-Bench-Main benchmark, CaC achieves 81.7% overall accuracy, a 25.7% absolute improvement over the strongest baseline.
• Sparse Anomaly Sensitivity: On the CaC-Bench-Hard split (featuring shorter duration and smaller spatial extent anomalies), CaC outperforms baselines by margins ranging from 14% to 38%, validating its effectiveness on highly sparse defects.
• Generalization: CaC improves accuracy by 19.9% on an external Skyra-derived subset, demonstrating out-of-domain generalization.
• Video Alignment: When used as a reward signal for downstream video generation (Wan-2.1-1.3B), CaC reduces the expert-annotated anomaly rate by 11.7% (from 29.0% to 17.3%) while improving subject consistency and imaging quality, without suppressing dynamic motion.

Significance and Claims

The paper claims that CaC effectively addresses the "attention dilution" problem caused by the extreme sparsity of video anomalies. By actively concentrating on anomalous frames and regions through a structured, interpretable reasoning process, CaC provides more reliable and grounded judgments than existing global reward models or detectors. The authors assert that this approach not only advances the evaluation of generated videos but also provides actionable feedback that directly improves the quality of video generation models by reducing structural and temporal hallucinations. The work highlights that as video generation matures, specialized, fine-grained reward models are necessary to tackle the remaining subtle failure modes that holistic evaluators miss.