💻 computer science

TLG: Temporal-Logic Grounding for Video Question Answering via Source-Annotation Reconstruction and Category-Targeted Reasoning

The paper introduces TLG, a three-tier system that significantly improves video question answering on the TimeLogic Challenge by reconstructing action timelines from source annotations and executing temporal-logic programs, demonstrating that leveraging real annotations rather than scaling model size is the key to overcoming temporal grounding limitations.

Original authors: Ali Alavi

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ali Alavi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery where the clues are hidden in a video. The question isn't just "What happened?" but rather, "Did the person put the milk in the fridge before or after they cut the bread?" or "Did they always stir the pot while the water was boiling?"

This is the challenge of TimeLogic: a test that asks computers to understand the precise order and timing of events in a video, not just the objects they see.

Here is how the paper explains the problem and their solution, using simple analogies.

The Problem: The "Blurry Camera" Effect

Current AI models (Video-Language Models) are like people watching a video who only remember the most important snapshots. If you ask them, "Did the dog bark before the cat jumped?", they often guess randomly. Why? Because they treat the video as a "bag of frames"—a pile of pictures—rather than a continuous timeline. They can see the dog and the cat, but they can't pinpoint exactly when the dog barked. They are great at recognizing what is there, but terrible at knowing when it happened.

The Solution: TLG (Temporal-Logic Grounding)

The authors built a three-step detective system called TLG. Instead of trying to teach the AI to "see" time perfectly (which they found impossible), they built a system that uses a "cheat sheet" whenever possible.

Think of TLG as a three-tiered team:

Tier 1: The "Cheat Sheet" Detective (The Symbolic Solver)

This is the star of the show. The researchers realized that the videos in the test came from existing datasets that already had "scripts" or "annotations" (like a detailed logbook written by humans).

How it works: Instead of asking the AI to watch the video and guess the timing, TLG looks up the video's ID in the logbook. It finds the exact start and stop times of every action (e.g., "Egg cracked: 0:05–0:10").
The Magic: Once it has these exact times, it doesn't need to "think" or "see" anything. It just does simple math (like checking if 0:05 is before 0:10). It solves the question perfectly, like a calculator.
The Catch: This only works if the video has a logbook entry. If the specific action isn't in the logbook, this detective has to say, "I don't know."

Tier 2: The "Generalist" Detective (The Open VLM)

When the "Cheat Sheet" isn't available, the system passes the question to a standard, powerful AI model (Qwen2.5-VL-32B).

How it works: This AI watches the video and tries to answer based on what it sees. It's good at general questions but still struggles with precise timing.
The Strategy: The system uses this AI for the "Yes/No" questions where the AI is already pretty decent.

Tier 3: The "Specialist" Detective (The Frontier Model)

The researchers noticed that the "Generalist" AI was terrible at a specific type of question: Multiple Choice (where it has to pick the right order from four options).

The Fix: For these tricky multiple-choice questions where the Cheat Sheet failed, they sent the question to an even smarter, more expensive AI (Gemini-3.1-Pro).
The Result: They only sent the hard questions to this expensive expert, keeping the cost low (about $10 total for the whole test) while boosting the score significantly.

The Big Discovery: "Real Notes" Beat "Bigger Brains"

The most surprising finding in the paper is what didn't work.

The Failed Experiment: The researchers tried to build a system where the AI would watch the video and create its own timeline (like writing its own logbook). They tried this with three different powerful models.
The Result: All of them failed. They were worse than just letting the AI answer the question directly.
The Lesson: You cannot teach an AI to perfectly understand "time" just by making it bigger or smarter. The only way to get perfect timing is to have real, human-written notes (annotations) to look up.

The Scorecard

Before TLG: A strong AI model got about 47% correct (basically guessing).
After TLG: The system got 71.37% correct.
The Secret Sauce: The biggest jump in score came from using the "fine-grained" human notes (the detailed logbooks) and then only sending the hardest questions to the expensive AI.

Summary

The paper argues that for this specific type of video puzzle, you don't need a smarter brain; you need a better map. By combining a lookup system for known videos with a smart AI for the unknown ones, they solved the "TimeLogic" challenge much better than anyone else, proving that accurate data (the map) is more important than just having a larger model (the brain).

Technical Summary: TLG (Temporal-Logic Grounding) for Video Question Answering

Problem Definition

The paper addresses the TimeLogic Challenge, a benchmark for temporal-logic reasoning over video. The task requires models to determine relationships between events (e.g., precedence, co-occurrence, duration) using 16 formal temporal operators (e.g., BEFORE, AFTER, UNTIL, ALWAYS, CO-OCCUR) across boolean (Yes/No) and 4-way multiple-choice formats.

The authors identify a critical grounding gap rather than a knowledge gap. Existing end-to-end Video-Language Models (VLMs) perform near chance levels (~50%) because they treat video as a "bag of frames," exhibiting a single-frame bias that prevents them from reliably localizing actions on a timeline. The core challenge is not understanding language or recognizing objects, but precisely mapping actions to temporal intervals.

Methodology: The TLG Pipeline

The proposed solution, TLG (Temporal-Logic Grounding), is a three-tier system designed to prioritize deterministic logic over neural perception. It operates on the observation that TimeLogic questions are deterministic functions of the public temporal annotations from the source datasets.

Tier 1: Symbolic Reconstruction and Execution

Annotation Reconstruction: For each video, TLG reconstructs an action timeline $T_v = \{(a_i, s_i, e_i)\}$ $T_{v} = {(a_{i}, s_{i}, e_{i})}$ by mapping the video ID to the source dataset's public annotations.
- CrossTask: Uses step-boundary annotations (100% coverage).
- Breakfast: Utilizes 178-class fine-grained segmentation (e.g., "reach knife") rather than coarse 48-class labels.
- STAR/AGQA: Merges Charades action segments with Action Genome state relations and derives situation-graph codes.
Fuzzy Grounding: A matching module handles discrepancies between question phrasing (e.g., gerunds) and annotation labels (e.g., imperatives) using token-Jaccard similarity and verb-aware penalties. Compound labels are split based on camel-case and digits.
Symbolic Execution: Questions are parsed into temporal-logic programs. The system executes these programs deterministically using interval arithmetic. If the named actions are not found in the annotations, Tier 1 abstains.

Tier 2: Open VLM Fallback

For questions where Tier 1 abstains (due to missing annotations), the system falls back to a strong open-source VLM (Qwen2.5-VL-32B). This model uses FPS-style frame sampling and temporal-reasoning prompts. It serves as the workhorse for the ~43% of questions not covered by annotations.

Tier 3: Category-Targeted Frontier Reasoning

The authors empirically measured the VLM's performance per category and found it weak on multiple-choice (MC) questions (19–57% accuracy) but strong on boolean questions (73–79% accuracy).

Routing Strategy: Instead of sending all fallback questions to a larger model, TLG routes only the abstained multiple-choice questions (and weak boolean/UNTIL categories) to a frontier reasoning model (Gemini-3.1-Pro).
Cost Control: This tier uses low media resolution (256 tokens/image) and a low thinking budget, costing approximately $10 for the entire test set.

Key Contributions

TLG System: A three-tier architecture that maps videos to source annotations for exact symbolic solving, using neural models only as a targeted fallback.
Category-Targeted Routing: A strategy that isolates the VLM's weaknesses (MC questions) and offloads them to a frontier model, while keeping strong categories (boolean) on the local, cheaper VLM.
Ablation on Grounding vs. Scale: Extensive experiments demonstrating that real annotations are the primary driver of accuracy, while increasing model scale or attempting model-based timeline reconstruction yields diminishing returns or failure.
Performance: Achieving a test accuracy of 71.37%, a +24.5 absolute gain over the VLM baseline, placing the system within 3 points of the leaderboard top.

Experimental Results

Baseline vs. TLG: A holistic VLM baseline (Qwen3-VL-8B) achieves 46.9%. The full TLG system reaches 71.37%.
Component Analysis:
- Symbolic Reconstruction: Adding symbolic logic from CrossTask, Action Genome, and Breakfast fine-grained annotations incrementally improves accuracy. The Breakfast fine-grained annotations alone provided a +6.2 point jump, whereas coarse labels contributed ~0.
- Model Scale: Upgrading the base VLM from 8B to 32B only improved the hybrid system by +2.6 points. Larger models (72B/78B) suffered from memory issues or degenerate outputs.
- Model-Based Reconstruction Failure: Attempts to have a VLM reconstruct the timeline for abstained questions (instead of looking up annotations) resulted in lower accuracy (67.10%) than simply letting the VLM answer holistically (67.37%). This confirms that temporal grounding is an irreducible bottleneck that requires annotations, not just larger models.
- Targeted Frontier: Routing only the weak MC categories to Gemini-3.1-Pro added a final +4.0 points.

Significance and Claims

The paper claims that temporal-logic VideoQA should be reframed from a perception problem to an annotation reconstruction problem.

Grounding is Irreducible: The authors conclude that temporal grounding cannot be learned by scaling models or reconstructing timelines via neural networks; it fundamentally relies on the availability of ground-truth temporal annotations.
Annotation Dominance: The results show that "real annotations — not larger models — drive accuracy." The largest performance gains came from accessing fine-grained source annotations (Breakfast) and precise category routing, not from model size.
Efficiency: The system achieves near-state-of-the-art performance with minimal cost (~$10 for API calls), demonstrating that a hybrid symbolic-neural approach is more effective than pure end-to-end learning for this specific reasoning task.

The remaining gap to the leaderboard top (74.47%) is attributed to an annotation-access gap, where roughly 220 questions reference actions not present in any public source dataset, forcing the system to rely on models that cannot reliably ground time.