💻 computer science

From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models

This paper proposes a novel framework for egocentric action recognition that decouples visual perception from symbolic reasoning by converting videos into Temporal Action Graphs, demonstrating that leveraging Vision-Language Models as symbolic reasoners via in-context learning on structured graph representations significantly outperforms direct pixel-based inference across diverse model families.

Original authors: Bessie Dominguez-Dager, Francisco Gomez-Donoso, Miguel Cazorla, Marc Pollefeys, Daniel Barath, Zuria Bauer

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bessie Dominguez-Dager, Francisco Gomez-Donoso, Miguel Cazorla, Marc Pollefeys, Daniel Barath, Zuria Bauer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, well-read librarian (the AI) how to recognize what a person is doing in a video taken from their own eyes (like a GoPro on a head). The video is full of hands grabbing cups, cutting bread, and pouring coffee.

The problem is that if you just show the librarian the raw video frames (the pictures), they get overwhelmed. They are great at reading books and understanding stories, but they aren't very good at watching movies. They might see a hand holding a knife and a loaf of bread, but they struggle to figure out if the person is cutting the bread or just holding it still. They get confused by the tiny, fast changes between one second and the next.

The Paper's Solution: "TAG" (Temporal Action Graphs)

The authors propose a clever workaround. Instead of asking the librarian to watch the movie directly, they act as a translator. They break the video down into a structured "story" or a "recipe" that the librarian can read easily. They call this process TAG.

Here is how it works, step-by-step, using an analogy:

1. The "Snapshot" Translator (Visual Perception)

Imagine the video is a long, fast-forwarded movie. The authors don't show the whole thing at once. Instead, they chop the movie into tiny, overlapping clips (like 4 frames at a time).

The Job: They ask the AI to look at these tiny clips and write a simple sentence describing what is happening.
The Analogy: It's like a security guard watching a security camera. Instead of trying to memorize the whole day, the guard just writes a quick note every few seconds: "Hand reaches for cup," then "Hand grabs cup," then "Hand lifts cup."

2. Turning Notes into a "Recipe" (Symbolic Reasoning)

Once the AI has written these sentences, the authors take those sentences and turn them into a strict, structured list of facts, called a Graph.

The Job: They convert the sentence "Hand grabs cup" into a formal triplet: (Hand) --[grabs]--> (Cup).
The Analogy: Think of this like turning a messy handwritten grocery list into a clean, organized spreadsheet. The messy list says, "Get some milk, maybe eggs, oh and that red apple." The spreadsheet says: Item: Milk, Action: Buy. Item: Apple, Color: Red, Action: Buy.
Why this helps: The AI is a master at reading and understanding these structured lists (spreadsheets) because it was trained on billions of text documents. It is much better at reasoning about "If A grabs B, then C happens" when it's written as text than when it's hidden inside a blurry video pixel.

3. The "Show and Tell" Lesson (In-Context Learning)

This is the paper's biggest trick. Usually, if you want to teach an AI a new task, you have to retrain it (which is expensive and slow). But because the video is now just a text list, the authors can use In-Context Learning.

The Job: Before asking the AI to guess the action for a new video, they show it a few examples of other videos that have already been turned into these text lists, along with the correct answers.
The Analogy: Imagine you are taking a test. Instead of just handing you the question, the teacher whispers, "Remember that time we saw someone pouring water? Here is the list of facts for that. Now, look at this new list of facts. What are they doing?"
The Result: The paper shows that just showing the AI one or two of these text-based examples makes it significantly smarter at guessing the action. If they tried to show the AI raw video examples for this "Show and Tell," the AI's memory would fill up instantly because videos are huge. Text lists are tiny, so the AI can remember many examples easily.

What Did They Find?

The authors tested this on two famous video datasets (EGTEA and Epic-Kitchens) using 11 different AI models of various sizes.

The "Translator" Wins: In almost every case, the AI performed better when it read the text list (the graph) than when it tried to watch the raw video.
The "One Example" Boost: Adding just one or two text examples to the prompt made the AI even better, often beating the raw video approach by a large margin.
The Limitation: The system isn't perfect. If the first step (the security guard writing the notes) makes a mistake—like saying "cutting" when the person is actually "washing"—the final answer will be wrong. The system is only as good as the description it creates.

The Bottom Line

The paper argues that current AI models are like brilliant readers who are bad at watching movies. Instead of forcing them to watch the movie, we should translate the movie into a story they can read. By converting video into a structured text "graph," we unlock the AI's natural ability to reason, allowing it to understand complex actions like "cutting," "pouring," or "grasping" without needing to be retrained on millions of hours of video.

Technical Summary: From Frames to Temporal Graphs

Problem Statement

Egocentric action recognition requires capturing fine-grained temporal transitions in hand-object interactions (e.g., grasping, pouring, cutting). While Vision-Language Models (VLMs) have emerged as powerful reasoning systems, they struggle when applied directly to egocentric video. Current VLMs are predominantly trained on static image-text pairs and lack explicit video supervision. Consequently, when processing video sequences, they often fail to heuristically aggregate independent scene interpretations into a coherent understanding of motion and causality. This leads to the loss of critical temporal dynamics that distinguish similar actions (e.g., taking vs. touching). Furthermore, applying In-Context Learning (ICL) to raw video is impractical because each visual demonstration consumes thousands of image tokens, quickly exhausting the model's context window.

Methodology: The TAG Framework

The authors propose TAG (Temporal Action Graphs), a training-free framework that decouples visual perception from symbolic reasoning. Instead of feeding raw pixels directly into the reasoning module, TAG converts egocentric video into structured, text-serializable graphs. The process involves four key stages:

Temporal Decomposition: The input video frames are partitioned into overlapping temporal windows ( $W_t$ ). Each window is anchored to its final frame but includes preceding frames as context to disambiguate ongoing interactions.
Neuro-Symbolic Mapping (Two-Stage Prompting):
- Stage I (Semantic Description): For each window, the VLM generates a free-form natural language narrative describing the dominant hand-object interaction. This acts as a semantic bottleneck, forcing the model to consolidate visual evidence before symbolic formalization.
- Stage II (Symbolic Extraction): The narrative is parsed into a structured local graph ( $G_t$ ) consisting of interaction triplets: $(source, relation, object)$. The source is drawn from a fixed vocabulary (e.g., hand_left, camera_wearer), the relation is an open-vocabulary verb, and the object includes a label and a salient attribute.
Temporal Graph Construction: Local graphs are assembled into a single Temporal Action Graph ( $G_\tau$ ). This directed attributed graph captures both semantic content and temporal evolution by preserving the ordering of triplets.
Action Prediction & ICL: The serialized graph is provided to the VLM for closed-set action prediction. Crucially, because the graph is text-serializable, the framework supports efficient In-Context Learning (ICL). A few labeled graph examples (demonstrations) can be prepended to the prompt, a strategy that is computationally infeasible with raw video due to token limits.

Key Contributions

Framework Introduction: The proposal of TAG, a method that converts egocentric video into structured, open-vocabulary interaction graphs via multi-stage prompting, decoupling perception from reasoning without any fine-tuning.
Empirical Validation: Demonstration that reasoning over structured graphs consistently outperforms direct frame-based inference across 11 open-weight VLMs (ranging from 2B to 235B parameters). This holds true even in zero-shot settings where pretraining data contamination theoretically favors pixel-based memorization.
Efficient In-Context Learning: The provision of a scalable, fine-tuning-free alternative for few-shot learning. By projecting video into the language domain, TAG unlocks ICL capabilities that are otherwise impractical with raw visual inputs.
Comprehensive Evaluation: An extensive zero-shot and few-shot evaluation on two major benchmarks (EGTEA and Epic-Kitchens-100), including systematic ablations on graph construction components, ICL example counts, and input frame budgets.

Experimental Results

The framework was evaluated on EGTEA (106 action classes) and Epic-Kitchens-100 (3,806 composed action classes) using 11 VLMs.

Performance Gains: Across small, medium, and large model regimes, graph-based reasoning consistently improved accuracy over frame-only baselines. For instance, on EGTEA with the GLM-4.6V-Flash model, adding ICL to the graph representation yielded a 9.5% gain in Mean Class Accuracy (MCA) over the frame-only baseline.
ICL Effectiveness: The text-serializable nature of the graphs allowed for effective few-shot learning. In most cases, adding just one or two labeled graph examples significantly boosted performance, often surpassing the zero-shot frame-based limits.
Model Agnosticism: The benefits were observed across diverse architectures (dense and Mixture-of-Experts) and parameter scales. Notably, even models with weak frame-based baselines (e.g., Llama-4-Scout) showed substantial improvements when using graphs + ICL.
Ablation Studies:
- Sliding Windows: The use of overlapping temporal windows was the largest single contributor to performance gains.
- Multi-Stage Prompting: The two-stage process (narrative then graph) improved grounding and reduced hallucinations compared to direct graph generation.
- Frame Budget: Increasing the number of input frames in the frame-only setting improved performance, confirming that the gains from TAG are not merely due to seeing fewer frames but stem from the structured temporal abstraction.
Failure Modes: The paper notes that the graph representation is only as reliable as the per-window perception. Errors in the initial narrative (e.g., misidentifying a cut as a wash) propagate to the final prediction, though ICL can sometimes correct these downstream.

Significance and Claims

The paper claims that current VLMs function more effectively as symbolic reasoners than as direct visual observers of dynamic scenes. By projecting video into the language domain, TAG leverages the latent reasoning strengths of these models to compensate for their current limitations in temporal visual understanding.

The authors position this approach as a pragmatic bridge for the current state of AI. While end-to-end compute scaling may eventually solve video understanding natively, structured language representations provide a necessary, scalable, and training-free alternative today. The method allows researchers to utilize general-purpose VLMs for fine-grained egocentric tasks without the substantial cost of domain-specific fine-tuning (e.g., on Ego4D). The paper concludes that while the pipeline incurs higher inference latency due to multiple VLM calls, the trade-off yields competitive accuracy and unlocks efficient few-shot learning capabilities that raw video cannot support.