FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling

1. Problem Statement

Driver attention modeling is critical for autonomous driving safety and human-AI collaboration. However, existing approaches face two major limitations:

• Data Scarcity & Cost: Traditional methods rely on large-scale, densely annotated eye-tracking datasets (often hundreds of thousands of frames) to learn spatial saliency. Collecting such data is expensive, privacy-sensitive, and lacks diversity in rare, safety-critical scenarios.
• Static vs. Semantic Understanding: Most models treat attention as a static perceptual saliency problem (predicting where a driver looks) rather than a semantically structured, forward-looking process (predicting why attention shifts). This overlooks the driver's anticipation of future events based on traffic rules and scene context.
• Task Conflict in Joint Modeling: Recent Vision-Language Models (VLMs) attempt to jointly predict gaze and generate explanations. However, forcing a single shared architecture to handle both spatial regression (gaze maps) and semantic generation (captions) often leads to task conflict, where the model overfits one task and underfits the other, especially under data-constrained conditions.

2. Methodology: FSDAM Framework

The authors propose FSDAM, a framework that achieves joint spatial attention prediction and structured explanation generation using only 90 annotated examples.

A. Structured Attention Reasoning Formulation

Instead of static saliency, the authors decompose driver attention into a structured reasoning format consisting of four components:

1. Scene Context: Global environmental layout.
2. Current Gaze: The driver's present fixation.
3. Next Gaze: The anticipated shift in attention (forward-looking).
4. Why (Causal Rationale): The safety-driven reason for the shift (e.g., "to yield to pedestrians").

B. Data Curation Strategy

• Dataset: Built upon the BDD-A dataset (braking events).
• Key Frame Selection: Uses KL-divergence mining to identify frames with maximum gaze distribution changes between consecutive frames, capturing critical attention transitions.
• Annotation: Uses GPT-4o with a strict template to generate the four-part structured captions, verified by human experts for spatial accuracy and causal correctness.

C. Dual-Pathway Architecture

To mitigate task interference under few-shot learning, FSDAM employs a decoupled architecture built on a frozen LLaVA-Next (Mistral-7B + CLIP-ViT) backbone:

1. Gaze Pathway: A dedicated convolutional decoder that upsamples visual features to predict the spatial gaze heatmap ($\hat{G}$).
2. Caption Pathway: A separate cross-attention module that generates the structured text explanation.
3. Training-Only Vision-Language Alignment: A novel mechanism where a contrastive loss (InfoNCE) aligns the gaze-weighted pooled visual features with the text embeddings.
   • Crucial Design: This alignment module is active only during training. It injects semantic priors into the spatial learning process without adding computational overhead during inference.

D. Loss Functions

The total loss is a weighted combination of three objectives:

1. Gaze Loss ($\mathcal{L}_{gaze}$): Forward KL divergence between predicted and ground-truth gaze, augmented with a blur-gap regularizer to enforce spatial sharpness and prevent diffuse predictions.
2. Caption Loss ($\mathcal{L}_{cap}$): Autoregressive cross-entropy loss for text generation.
3. Alignment Loss ($\mathcal{L}_{align}$): InfoNCE contrastive loss to align visual and semantic representations (training only).

3. Key Contributions

1. First Few-Shot Joint Modeling: The first work to demonstrate effective joint spatial attention prediction and natural language explanation generation using only 90 training examples (two orders of magnitude more efficient than prior joint models like LLada).
2. Structured Anticipation: Reformulates attention modeling as a forward-looking, semantically grounded task rather than static saliency, explicitly modeling the "next gaze" and "why."
3. Dual-Pathway with Training-Only Alignment: Introduces an architecture that decouples tasks to prevent negative transfer while using a temporary alignment mechanism to guide spatial learning with semantic priors.
4. Data Efficiency & Generalization: Demonstrates that semantically grounded attention modeling enables high performance in data-constrained environments with strong zero-shot generalization.

4. Experimental Results

The model was evaluated on four benchmarks: BDD-A, DADA-2000, DR(eye)VE, and W3D.

• In-Domain Performance (BDD-A):
  • Trained on only 90 samples, FSDAM achieved a KL divergence of 1.13, outperforming the fully supervised LLada (trained on ~70k samples) by 2.6% and the fully supervised DBNet by 13%.
  • It matched the CC (Correlation Coefficient) of LLada (0.60) while using 777x less data.
• Zero-Shot Cross-Dataset Transfer:
  • FSDAM (trained on 90 BDD-A samples) outperformed fully supervised baselines trained on massive datasets (e.g., DADA-2000 with 658k frames) on unseen datasets like DR(eye)VE and W3D.
  • On DR(eye)VE, it achieved the lowest KL divergence (0.77), improving over LLada by 26%.
• Caption Generation:
  • On the W3D dataset, FSDAM achieved BLEU scores of 0.42 (Normal Driving) and 0.33 (Safety-Critical), closely approaching fully supervised baselines (LLada: 0.44) and significantly outperforming few-shot baselines (DeepGazeIIE+LLaVA: ~0.11).
• Few-Shot Dynamics:
  • Performance saturated rapidly; the model achieved 96.7% of full-data performance with only 5 support samples.

5. Significance and Impact

• Scalability: FSDAM provides a scalable path for developing explainable driver attention systems in environments where collecting large-scale eye-tracking data is impossible (e.g., rare accident scenarios or specific weather conditions).
• Interpretability: By generating structured "Why" explanations, the model bridges the gap between black-box AI decisions and human understanding, crucial for trust in autonomous vehicles.
• Methodological Insight: The paper proves that semantic supervision (via language) can effectively guide spatial learning even with minimal data, provided the architecture is designed to decouple conflicting tasks and inject priors only during training.

In conclusion, FSDAM demonstrates that semantically grounded attention modeling allows for highly data-efficient learning, achieving competitive or superior performance to fully supervised models while offering interpretable, context-aware reasoning for autonomous driving safety.