💻 computer science

ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering

To address the limitations of current multimodal models in understanding temporally distributed text across video frames, this paper introduces ViTexQA, a large-scale dataset requiring cross-frame text fusion, alongside FrameThinker, a two-stage training framework combining CoT-guided supervised fine-tuning and temporally-grounded reinforcement learning that achieves state-of-the-art performance.

Original authors: Zhentao Guo, Chen Duan, Tongkun Guan, Zining Wang, Kai Zhou, Pengfei Yan

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhentao Guo, Chen Duan, Tongkun Guan, Zining Wang, Kai Zhou, Pengfei Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a mystery movie where the clues to solving the crime aren't all in one scene. Instead, the killer's name is written on a bus in the first minute, the time of the crime is shown on a clock in the middle, and the final location is revealed on a sign in the last minute. To solve the mystery, you have to remember and connect these scattered pieces of information.

Most current AI models are like viewers who only look at a single frozen frame of the movie. If you show them just the picture of the bus, they can read the name. But if you ask them a question that requires knowing the bus name and the clock time and the sign, they get stuck because they can't "connect the dots" across time.

This paper, ViTexQA, introduces a new way to teach AI to be a better movie detective. Here is the breakdown in simple terms:

1. The Problem: The "Snapshot" Trap

Current AI models are great at reading text in a single picture (like a sign on a wall). However, real-world videos are dynamic. Text often appears, moves, changes, or disappears over time.

The Issue: The researchers found that existing video tests were "cheating." They asked questions that could be answered by looking at just one single frame. It was like asking, "What color is the car?" when the answer is obvious in one snapshot.
The Reality: Real video understanding is more like reading a story where the plot unfolds over time. You need to remember what you saw 10 seconds ago to understand what is happening now.

2. The Solution: A New Dataset (ViTexQA)

The team created a massive new library of video questions called ViTexQA.

The Rule: Every single question in this library is impossible to answer by looking at just one frame.
The Analogy: Imagine a game of "Scavenger Hunt" where the clues are hidden in different parts of a long video. To win, the AI must watch the whole video, take notes on what text appears when, and then combine those notes to find the answer.
The Content: They gathered over 5,000 videos covering things like sports scores, news tickers, driving signs, and scrolling text. They even made 100 fake videos specifically designed to have text rolling across the screen to test this ability.
Human Touch: Unlike many AI projects that use other AI to write the questions, humans wrote every single question and answer here to ensure they were truly difficult and required real thinking.

3. The Method: "FrameThinker"

To teach an AI how to solve these "Scavenger Hunt" puzzles, they built a training method called FrameThinker. Think of it as a two-step training camp for the AI:

Step 1: The "Note-Taking" Class (Supervised Fine-Tuning)
First, they teach the AI to act like a careful student. Instead of just guessing the answer, the AI is forced to write out a "Chain of Thought." It has to say: "At 12 seconds, I saw 'Start' on the screen. At 30 seconds, I saw '50% Progress'. At 90 seconds, I saw 'Finish'." It learns to explicitly list the evidence it found at different times before giving an answer.
Step 2: The "Coach's Feedback" Class (Reinforcement Learning)
Next, they use a reward system. If the AI gives the right answer but skips the time-steps, or if it gets the time wrong, it gets a "bad grade." If it correctly links the text from the beginning of the video to the end, it gets a "gold star." This trains the AI to value timing and connection just as much as the answer itself.

4. The Results

When they tested this new method against the smartest AI models available:

The Gap: Even the best existing models struggled, often getting the answers wrong because they tried to solve the puzzle using only a single "snapshot" of the video.
The Win: The FrameThinker model, trained on this new dataset, significantly outperformed everyone else. It proved that when you force an AI to look at the whole timeline and connect the dots, it gets much better at understanding video text.

Summary

In short, the paper says: "Current AI is bad at reading stories in videos because it only looks at single pictures. We built a new test (ViTexQA) that forces AI to read the whole story, and we built a training method (FrameThinker) that teaches AI to take notes on when things happen. The result is an AI that can finally understand video text the way humans do—by connecting the past, present, and future."

) and answers (`).
2. Temporal Reward ( $r_{tmp}$ ): Quantifies how well the model's reasoning chain references specific ground-truth timestamps and maintains semantic consistency with the surrounding context.
3. Content Reward ( $r_{cnt}$ ): Evaluates the semantic correctness of the final answer.

Key Contributions

ViTexQA Dataset: The first large-scale dataset tailored for multi-frame temporal awareness. It uniquely enforces that every question requires integrating textual cues across multiple timestamps, constructed via a human-centric, quality-controlled pipeline.
FrameThinker Methodology: A systematic training approach combining CoT-guided SFT and Temporally-grounded RL to explicitly model temporal dependencies and evidence aggregation in video text understanding.
Comprehensive Benchmarking: Extensive evaluation revealing significant performance gaps in current MLLMs regarding temporal perception and demonstrating the efficacy of explicit temporal training.

Experimental Results

Dataset Validation: Experiments confirm that over 95% of ViTexQA questions are genuinely multi-frame dependent. Models achieve significantly higher scores (e.g., 71.6% vs. 0.0% for single-random frames) only when provided with multi-frame inputs, validating the dataset's design.
Baseline Performance: In zero-shot settings, existing state-of-the-art MLLMs (both open-source and closed-source) struggle with the dataset, particularly on "Rolling Text" scenarios, highlighting a fundamental weakness in integrating dynamic textual information.
FrameThinker Performance:
- Fine-tuning on ViTexQA data significantly improves model performance across multiple benchmarks (e.g., a 10.2% ROUGE-L increase on ViTexQA for Qwen3-VL).
- The full FrameThinker pipeline (SFT + RL) outperforms strong baselines, achieving a 73.5% average score on ViTexQA, surpassing the best zero-shot baseline (Gemini-3 Pro at 71.6%) and other fine-tuned models.
- Ablation studies confirm that both the CoT-guided SFT and the Temporally-grounded RL stages contribute incrementally to performance gains, with the full pipeline yielding a 10.0% improvement over the base model.

Significance and Claims

The paper claims that ViTexQA and FrameThinker address a fundamental gap in video understanding research: the tendency of current benchmarks to allow models to bypass temporal perception. By enforcing genuine multi-frame dependencies, ViTexQA provides a more faithful measure of a model's ability to comprehend real-world video content where information unfolds over time.

The authors position FrameThinker as a demonstration that explicit temporal perception training—moving beyond implicit frame-level correlations to structured, evidence-based reasoning—can significantly enhance MLLM capabilities. They assert that their human-centric annotation strategy and the resulting dataset serve as valuable resources for the research community to develop MLLMs capable of truly understanding the rich, temporally distributed textual information present in real-world videos. The work is presented as a step toward bridging the gap between current model capabilities and the complex demands of dynamic video text comprehension.