🤖 AI

Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models

This paper proposes decoding pedestrian crossing intentions from egocentric video by framing the task as visual question answering, demonstrating that parameter-efficient fine-tuning of Vision Language Models—particularly when augmented with contextual cues like eye gaze and motion—significantly outperforms both zero-shot VLMs and specialized transformer baselines to establish a new state of the art.

Original authors: Danya Li, Xiang Su, Yan Feng, Rico Krueger

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Danya Li, Xiang Su, Yan Feng, Rico Krueger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess what a stranger is about to do at a busy crosswalk. Usually, traffic cameras watch from the side (like a security guard), but this paper asks a different question: What if we could see the world through the pedestrian's eyes?

The researchers wanted to teach a super-smart computer (an AI) to look at a short video clip from a pedestrian's point of view and guess: "Is this person about to step into the street, or are they going to wait?"

Here is how they did it, explained simply:

1. The "Smart Assistant" vs. The "Specialist"

The team tried using two types of AI:

The "Generalist" (Vision-Language Models): Think of these as highly educated librarians who have read millions of books and seen billions of photos. They know a lot about the world but haven't specifically studied traffic rules. The researchers asked them, "Based on this video, will the person cross?" without giving them any special training.
The "Specialist" (Traditional AI): This is like a traffic cop who has only ever studied crosswalks. It's a specialized computer program built just for this one job.

The Result: The "Generalist" librarians were okay at guessing (better than flipping a coin), but they weren't great at understanding the complex logic of traffic. They often missed the subtle cues. The "Specialist" was much better.

2. The "Tutoring" Session (Fine-Tuning)

Since the librarians were smart but untrained for this specific job, the researchers gave them a crash course. They didn't rebuild the librarians from scratch; instead, they used a technique called Fine-Tuning.

Think of this like giving a brilliant student a specific textbook on "Pedestrian Safety" and letting them study just the relevant chapters.

The Outcome: After this quick training, the "Generalist" AI became a star student. It actually beat the "Specialist" traffic cop by a significant margin (about 9% more accurate). It learned to combine what it saw in the video with its general knowledge to make better guesses.

3. Adding Extra Clues (Context)

The researchers realized that just watching the video wasn't enough. Humans don't just look; we also feel our own movement and where we are looking. So, they gave the AI extra "sensory" data:

Ego Motion: How fast is the pedestrian walking?
Vehicle Motion: How fast is the car coming?
Eye Gaze: Where is the pedestrian looking? (Did they look at the car? Did they look at the sidewalk?)

The Magic of Eye Gaze:
Imagine the AI is wearing a pair of smart glasses. The researchers drew a little red dot on the video to show exactly where the pedestrian was looking.

When the AI saw where the pedestrian was looking, combined with how fast they were walking, it became much better at predicting the future.
It's like if you see someone glance nervously at a car before stepping off the curb; that glance is a huge clue. The AI learned to spot that clue.

4. What Didn't Work?

The researchers tried some "tricks" that usually help AI, but they failed here:

Asking the AI to "Think Step-by-Step": They tried making the AI write out its reasoning before answering (like a math student showing their work). Surprisingly, this made the AI worse and slower. It seems that for this specific visual task, over-thinking in words confused the visual part of the brain.
Drawing Boxes on Objects: They tried highlighting the car and the crosswalk on the video screen to help the AI focus. This didn't help much, likely because the AI was already looking at the right things, and the extra drawings were just noise.

The Bottom Line

The paper proves that if you take a powerful, general-purpose AI and give it a little bit of specific training (fine-tuning) plus a few extra clues (like where the person is looking), it can become an expert at predicting pedestrian behavior from a first-person view.

The Champion: The best version of their system was a model called Qwen3-VL-2B, which, when guided by the pedestrian's walking speed and eye movements, achieved the highest accuracy ever recorded for this specific task.

One Catch: The researchers tested this in a Virtual Reality (VR) simulation. While the results are promising, the real world is messier and more complex than a VR game, so there is still work to be done before this is used on real streets.

Technical Summary: Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models

Problem Definition

This work addresses the challenge of predicting pedestrian crossing intentions (specifically, whether a pedestrian will "cross" or "yield") using egocentric vision (first-person perspective). Unlike traditional exocentric approaches that rely on vehicle-mounted or surveillance cameras, this study focuses on decoding intent from short (2-second) monocular video clips captured from the pedestrian's viewpoint. The goal is to predict the pedestrian's action within a 1-second future horizon.

The authors reformulate this prediction task as a closed-ended Visual Question Answering (VQA) problem. The model receives an egocentric video clip ( $V$ ), optional contextual information ( $C$ ), and a specific query ( $Q$ ), and must select the correct intention from a predefined set $\{cross, yield\}$ .

Methodology

1. Model Selection and Baselines

The study evaluates two families of Vision-Language Models (VLMs):

Multimodal Large Language Models (MLLMs): Specifically, the Qwen3-VL family (2B and 8B variants) and Qwen2.5-VL-7B, as well as InternVL3 models. These are chosen for their broad world knowledge and general reasoning capabilities.
Video-Language Pretraining (VLP) Models: Specifically, GroundVQA, which is pre-trained on egocentric video-text data but often dominated by indoor activities.
Baselines: A specialized CLIP+Transformer baseline (video-only deep learning) serves as the primary performance benchmark, alongside random guessing and majority-class prediction.

2. Prompting Strategies (Zero-Shot Evaluation)

Before fine-tuning, the authors benchmark three prompting strategies to assess the models' inherent reasoning capabilities:

Standard Prompting: A direct query asking for the most likely action.
Text Chain-of-Thought (CoT): Encouraging the model to generate intermediate reasoning steps (e.g., analyzing visual elements, risk, and logic) before answering.
Visual Prompting: Augmenting video inputs with "set-of-marks" overlays (bounding boxes and IDs) generated by a detector (GroundingDINO) to highlight relevant objects like vehicles and crossing goals.

3. Contextual Information Integration

Beyond video, the study incorporates additional modalities to improve reasoning:

Ego Motion: The pedestrian's position and speed.
Vehicle Motion: The automated vehicle's position and speed.
Eye Gaze: Represented either as textual descriptions (direction in degrees) or visual overlays (red dots on fixation points).
Personal Attributes: Demographic data (e.g., age, gender, handedness) included in the prompt.

The authors investigate two integration strategies for time-varying context: Preface (appending context to the start of the prompt) and Interleaved (inserting context alongside specific video frames).

4. Fine-Tuning Strategy

To adapt the pre-trained models to the traffic domain, the authors employ Supervised Fine-Tuning (SFT) using Low-Rank Adaptation (LoRA). This approach freezes the original model parameters and trains only low-rank updates in selected layers. They experiment with tuning:

Language encoders only.
Cross-modal fusion modules only.
Both language and cross-modal components jointly.
The vision encoder remains frozen to preserve its role as a feature extractor.

Key Results

Zero-Shot Performance

Limited Reasoning: State-of-the-art VLMs in a zero-shot setting achieve only moderate gains over random guessing (e.g., Qwen2.5-VL-7B achieves ~63% accuracy) but fail to surpass the specialized CLIP+Transformer baseline (72.7% accuracy).
Prompting Limitations: Text CoT prompting did not improve performance and significantly increased inference latency. In some cases, it degraded performance by causing models to default to "yield." Visual prompting (set-of-marks) also failed to yield measurable improvements, likely due to detection inconsistencies and the specific nature of the VR environment.
Input Sensitivity: Larger models (e.g., Qwen3-VL-8B) performed worse with increased frame sampling (4 to 32 frames), suggesting sensitivity to redundant inputs, while smaller models (Qwen3-VL-2B) benefited from more frames.

Fine-Tuned Performance

Significant Gains: Parameter-efficient fine-tuning substantially improved performance. The fine-tuned Qwen3-VL-2B achieved 77.7% accuracy, and the fine-tuned VLP (GroundVQA) achieved 78.9% accuracy.
State-of-the-Art: These results represent a 9% accuracy improvement over the specialized CLIP+Transformer baseline.
Architecture Adaptation: For Qwen models, tuning only the cross-modal modules was sufficient. For VLP models, tuning both language and cross-modal components yielded the best results.

Impact of Contextual Cues

Zero-Shot: Adding context (e.g., ego motion, gaze) generally improved zero-shot performance, with ego motion being the most informative single cue. However, combining cues without fine-tuning sometimes led to hallucinations or degraded performance.
Fine-Tuned: When fine-tuned, the models effectively learned to exploit context. The combination of ego motion and eye gaze (specifically gaze-on-screen) provided the best generalization, achieving a 14.5% accuracy improvement over the transformer baseline.
Context Formatting: The Preface strategy (appending context at the start) consistently outperformed the Interleaved strategy, suggesting that global conditioning on context is more effective than frame-by-frame interleaving for this task.

Significance and Claims

The paper claims to establish a new state-of-the-art for egocentric pedestrian intent decoding. Its primary contributions are:

Feasibility of VLMs: Demonstrating that while off-the-shelf VLMs possess non-trivial zero-shot capabilities for traffic reasoning, they require domain adaptation (fine-tuning) to match or exceed specialized deep learning baselines.
Effectiveness of Fine-Tuning: Showing that parameter-efficient fine-tuning (LoRA) is a highly effective strategy for adapting general-purpose VLMs to safety-critical traffic tasks, outperforming models trained from scratch on video data alone.
Value of Multimodal Context: Establishing that incorporating eye gaze and ego motion significantly enhances predictive accuracy and generalization, particularly when the model is trained to interpret these signals.
Methodological Insights: Providing evidence that text-based CoT prompting and visual overlays may not be beneficial for this specific task, whereas structured context integration (via preface) and fine-tuning are critical.

The authors note limitations, including the use of a VR-based dataset which may not capture all real-world complexities, and the reliance on uniformly sampled frames. They suggest future work should focus on traffic-specific pretraining, adaptive temporal sampling, and evaluation on real-world data with rare events.