Keystep Recognition using Graph Neural Networks

Technical Summary: Keystep Recognition using Graph Neural Networks (GLEVR)

1. Problem Statement

The paper addresses the challenge of fine-grained keystep recognition in long-form procedural egocentric videos. The goal is to identify specific steps (keysteps) within a continuous video stream. This task is particularly difficult due to:

• Long-term dependencies: Understanding a single step often requires context from much earlier or later in the video.
• Multi-view availability: While exocentric (third-person) views provide valuable context, they are often only available during training and not during real-world inference.
• Computational Complexity: Existing dense video models struggle to process long-form videos due to high memory and compute requirements.

2. Methodology: The GLEVR Framework

The authors propose GLEVR (Graph Learning on Egocentric Videos for keystep Recognition), which reformulates keystep recognition as a node classification task on a graph.

A. Graph Construction Strategies:

• Egocentric Vision Graph: Each video clip (segment) is represented as a node. Temporal edges are drawn between subsequent nodes to model the sequential nature of actions.
• Multi-view Vision Graph: To leverage exocentric data during training, the authors add nodes for each available exocentric view. "Ego-exo" edges are created to connect corresponding segments between the egocentric and exocentric views, allowing the model to learn view-invariant representations.
• Heterogeneous Multimodal Graph (GLEVR-Hetero): The framework incorporates text as an additional modality. The authors use a video-to-text pipeline (VideoRecap + LLaMA-3) to generate segment-level summaries. These summaries are converted into features using LongCLIP and added as text nodes in a heterogeneous graph.

B. Key Technical Features:

• Sparsity: Unlike dense Transformers that attend to every frame, GLEVR uses a sparse graph structure, making it highly efficient for long videos.
• Training vs. Inference: The model is trained using multi-view and multimodal data but is designed to perform inference using egocentric video only.
• Feature Extraction: Instead of processing raw pixels, GLEVR operates on pre-extracted high-level visual features (Omnivore Swin-L), significantly reducing the computational burden.

3. Key Contributions

1. Novel Formulation: Transforming keystep recognition from a sequence labeling/classification task into a graph-based node classification problem.
2. Multi-view Alignment: A method to fuse variable numbers of exocentric views during training to improve egocentric-only performance at test time.
3. Multimodal Integration: A heterogeneous graph approach that successfully integrates LLM-generated video narrations.
4. Efficiency: A lightweight architecture that outperforms much larger, dense models while requiring significantly less memory and compute.

4. Experimental Results

The authors evaluated GLEVR on the Ego-Exo4D dataset and achieved several significant results:

• State-of-the-Art (SOTA) Performance: GLEVR outperformed the previous best method (VI Encoder) by 16.76% on validation and 12.12% on test sets.
• Contextual Reasoning: Moving from "no context" (treating segments independently) to "full context" (using the whole video graph) improved accuracy by over 14%, proving the effectiveness of the graph in capturing long-term dependencies.
• Multimodal Gains: Adding LLM-generated summaries (GLEVR-Hetero) provided a 1.29% improvement over the vision-only model.
• Multi-view Utility: Unlike previous models that saw performance drops when adding exocentric views, GLEVR showed a 2% gain by effectively leveraging multi-view alignment.
• Model Efficiency: GLEVR (91 MB) is orders of magnitude smaller than models like EgoVLPv2 (4300 MB).

5. Significance

This work is significant because it provides a scalable and efficient solution for understanding complex, long-duration human activities. By moving away from dense, frame-based processing toward sparse, graph-based reasoning, the authors demonstrate that models can achieve higher accuracy with a much smaller computational footprint. This approach is particularly relevant for real-world applications in robotics, augmented reality (AR), and wearable technology, where long-term temporal context and resource efficiency are critical.