💻 computer science

PercepCap: Video Captioner with Structured Spatio-Temporal Perception

PercepCap is a novel video captioning framework that enhances spatio-temporal understanding by explicitly generating a perception trace of object trajectories and events before producing the final caption, supported by a two-stage training strategy and a caption-anchored data construction pipeline to ensure alignment between perceptual evidence and descriptive output.

Original authors: Yifan Xu, Zihao Wang, Zhixiao Wang, Jiaming Zhang, Yichun Yang, Desen Meng, Yuanxing Zhang, Pengfei Wan, Limin Wang

Published 2026-07-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifan Xu, Zihao Wang, Zhixiao Wang, Jiaming Zhang, Yichun Yang, Desen Meng, Yuanxing Zhang, Pengfei Wan, Limin Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to watch a movie and write a review. In the world of artificial intelligence, this task is called "video captioning." For a long time, the smartest robots (known as Multimodal Large Language Models, or MLLMs) have been getting better at this, but they have a secret weakness: they are like magicians who pull a rabbit out of a hat without showing you how. They look at the video and immediately spit out a sentence like, "A dog runs across the field." But they don't show their work. They don't tell you which dog, where exactly it was running, or when it started and stopped. If the robot makes a mistake—say, it thinks the dog was running backward—the error is hidden inside the final sentence, making it hard to fix. Scientists care about this because if we want robots to truly understand the world, we need to know how they see it, not just what they say about it.

Enter PercepCap, a new framework that acts like a detective for video robots. Instead of letting the robot jump straight to the final review, PercepCap forces the robot to first write down its "detective notes." Before it writes the final caption, it must explicitly list the clues it found: "I see a dog (ID: 123) moving from the left side of the screen to the right between 2 seconds and 5 seconds." This "perceive-then-describe" chain makes the robot's thinking visible. The researchers found that by forcing the robot to show its spatio-temporal evidence (where things are and when they happen) before writing the story, the final description becomes much more accurate and detailed. They didn't just guess this would work; they trained a model using a special two-step process and tested it on several challenging benchmarks, showing that this "show your work" approach consistently beats the old "just guess the answer" methods.

The Detective's Notebook: How PercepCap Works

Think of the old way of video captioning like a student taking a test who is only allowed to write the final answer on the bubble sheet. If the student gets it wrong, the teacher has no idea if they misread the question, forgot a fact, or just guessed. The new method, PercepCap, is like giving that student a scratchpad.

The "Perceive" Step (The Scratchpad)
First, the robot watches the video and fills out a structured "detective notebook." This isn't just a random list; it's a strict, organized log of two things:

Object Trajectories: It tracks specific objects (like a person or a car) and writes down exactly where they are on the screen at different times.
Temporal Events: It notes when specific actions happen, marking the exact start and stop times.

The "Describe" Step (The Final Report)
Only after the notebook is full does the robot write the final caption. It uses the notes from the scratchpad as a guide. Because the robot has already done the hard work of identifying and timing the events, the final story is much less likely to contain hallucinations or missing details.

The Secret Sauce: How They Taught the Robot

You might wonder, "Where did the robot learn to fill out this notebook if no one gave it notebooks before?" The researchers had to invent a clever way to create training data, which they call Caption-Anchored Perception Data Construction.

Imagine you have a perfect movie review written by a human expert, but it doesn't have any timestamps or location tags. The researchers took this review and asked a super-smart AI to go back and watch the movie again, this time specifically looking for the things mentioned in the review.

Anchor: They started with the final caption (the "anchor").
Extract: They pulled out the names of the objects and events mentioned.
Ground: They made the AI watch the video again to find exactly where and when those specific things happened, drawing boxes around them and adding timestamps.

This created a perfect training set where the final caption and the "detective notes" matched perfectly.

The Two-Stage Training Camp

To teach the robot this new way of thinking, the researchers used a two-stage training strategy:

Stage 1: Supervised Fine-Tuning (The "Learn the Rules" Phase)
They showed the robot thousands of these perfect examples (caption + matching notes) and taught it to mimic the process. The robot learned: "First, I must list the clues. Then, I write the story." This is called Perceive-then-Describe Supervised Fine-tuning (PD-SFT).
Stage 2: Reinforcement Learning (The "Get a Gold Star" Phase)
Just copying isn't enough; the robot needs to get better at it. Here, they used a method called Perception-Grounded Reinforcement Learning (PG-RL). Imagine a teacher grading the robot's work in two separate categories:
- The Notes: Did you track the objects correctly? Did you get the start and end times right?
- The Story: Is the final caption accurate and complete?
The robot gets a "score" based on both. If it writes a great story but missed a key object in its notes, it gets a lower score. This forces the robot to care about the quality of its perception, not just the flow of its sentences.

The Results: Does It Actually Work?

The researchers tested PercepCap on several tough video understanding challenges, including DREAM-1K, CaReBench, ShortVidBench, and MotionBench. They compared their new model against the standard Qwen3-VL baseline (a powerful existing model that just writes captions directly).

The results were clear: PercepCap consistently outperformed the baseline.

On DREAM-1K, it achieved an F1 score of 33.9 (compared to the baseline's 29.1), meaning it was better at capturing the right details without making up fake ones.
On CaReBench, it improved the accuracy of identifying actions and objects significantly.
Even on ShortVidBench and MotionBench, which test how well a caption can answer questions about a video, PercepCap scored higher (e.g., 76.1% accuracy vs. 72.8% for the baseline).

Why This Matters (And What It Can't Do Yet)

The main takeaway is that making a robot "show its work" before it speaks makes it smarter and more reliable. By separating the act of seeing (perception) from the act of speaking (captioning), the researchers created a system that is easier to debug and harder to trick.

However, the paper is honest about its limits. Because the training data was created by AI (the "Caption-Anchored" method), if the initial AI made a mistake in the notes, that mistake could be passed down to the robot. Also, this method takes more time and computer power because the robot has to write a long list of notes before it can write the final sentence. But, for anyone who wants video AI that doesn't just guess but actually understands the timeline and location of events, PercepCap suggests a very promising new direction.

Technical Summary: PercepCap

Problem Statement

Video captioning requires fine-grained spatio-temporal understanding, encompassing the spatial localization of objects and the temporal sequencing of events. Existing Multimodal Large Language Models (MLLMs) typically employ a caption-only generation paradigm, mapping video inputs directly to final text descriptions. This approach renders the perceptual stage implicit, meaning the model identifies objects, actions, and temporal relations within a single autoregressive process without exposing the underlying evidence. Consequently, errors in object identity, action recognition, or temporal ordering are only visible in the final output, making it difficult to identify, supervise, or correct specific perceptual failures. Furthermore, standard supervision signals applied only to the final caption provide no direct feedback on whether the model's underlying perception is correct.

Methodology

To address these limitations, the authors propose PercepCap, a perception-aware framework that explicitly separates spatio-temporal perception from caption generation. The core innovation is a perceive–describe generation chain, where the model first produces a structured perception trace and then generates the final caption conditioned on this evidence.

1. Structured Spatio-Temporal Perception

The framework reformulates the output sequence as $Y = [Y_{perc}, Y_{cap}]$ , where $Y_{perc}$ is a structured trace containing:

Object Evidence: Temporally consistent identities, semantic labels, and localization sequences (2D bounding boxes with timestamps).
Event Evidence: Temporal segments (start/end times) with concise textual descriptions, linked to the tracked objects.

2. Two-Stage Training Strategy

PercepCap utilizes a two-stage Reinforcement Fine-Tuning (RFT) pipeline:

Perceive-then-Describe Supervised Fine-Tuning (PD-SFT): This stage adapts the model from direct caption generation to the perceive–describe chain. It uses next-token cross-entropy loss on serialized target sequences to teach the model to generate object trajectories and temporally grounded events before producing the final caption.
Perception-Grounded Reinforcement Learning (PG-RL): To overcome the limitations of single-token supervision (which cannot distinguish between perception errors and captioning errors), PG-RL employs Group Relative Policy Optimization (GRPO). It utilizes a composite reward function that decomposes feedback into:
- Perception-level Rewards: Spatial IoU ( $R_{iou}$ ) for object localization, Temporal IoU ( $R_{tiou}$ ) and Semantic Overlap ( $R_{sodam}$ ) for event timing and description.
- Caption-level Rewards: $R_{cap}$ evaluates the coverage of objects and events in the final text against the ground truth.
- Format Rewards: $R_{format}$ ensures the output adheres to a strict JSON structure.

3. Caption-Anchored Perception Data Construction

Since standard datasets lack intermediate perception annotations, the authors introduce a pipeline to construct caption-aligned training data:

Caption Anchor Generation: A strong video model generates a caption-only description ( $Y^*_{cap}$ ).
Content Extraction: The model extracts object labels and event descriptions from this caption.
Video-Grounded Completion: The model revisits the video to assign identities, predict timestamped bounding boxes for objects, and predict start/end times for events.
Assembly: The grounded spatial/temporal data is combined with the original caption to form the structured training pair $Y^* = [Y^*_{perc}, Y^*_{cap}]$ .

Key Contributions

Perception-Aware Framework: The paper introduces a paradigm that turns video captioning into an evidence-grounded generation process by explicitly outputting a structured spatio-temporal perception trace (object trajectories and event boundaries) prior to the final caption.
PD-SFT and PG-RL: A two-stage training strategy that first adapts the model to the perceive–describe chain and then optimizes it using decomposed rewards that separately evaluate perception grounding and caption quality.
Caption-Anchored Data Construction: A novel data pipeline that generates structured perception data aligned with final captions, enabling the training of models that require intermediate perceptual supervision.

Results

PercepCap was evaluated on multiple benchmarks, including DREAM-1K, CaReBench, ShortVidBench, MotionBench, and VidCapBench-AE.

Performance: The model consistently outperforms the Qwen3-VL baseline (an 8B open-source model) across all benchmarks. For instance, on DREAM-1K, PercepCap achieved an F1 score of 33.9 compared to the baseline's 29.1. On CaReBench, it improved Action F1 to 39.4 and Object F1 to 34.7.
Ablation Studies: Experiments confirmed that the gains are not merely due to additional fine-tuning or stronger caption supervision. Variants using only caption-level rewards or caption-only generation paradigms performed significantly worse than the full PercepCap pipeline, demonstrating the necessity of the explicit perception trace and the staged training approach.
Qualitative Analysis: Visual comparisons show that PercepCap better distinguishes key entities and follows action chains (e.g., object handoffs) more faithfully than direct captioning baselines.

Significance

The paper argues that detailed video captioning benefits from exposing the "reasoning" process of the model. By making perceptual evidence explicit, PercepCap allows for:

Targeted Supervision: Errors can be attributed to specific stages (perception vs. verbalization), enabling more precise training signals.
Auditability: The structured trace makes the model's understanding of the video inspectable, bridging the gap between raw video input and natural language output.
Improved Grounding: The framework ensures that the final caption is strictly conditioned on verified object trajectories and event boundaries, leading to higher coverage and accuracy in describing complex video content.

The authors conclude that structured perception traces serve as a practical intermediate interface, making video captioning more grounded, auditable, and effective for fine-grained understanding tasks.