Automated Detection of Mutual Gaze and Joint Attention in Dual-Camera Settings via Dual-Stream Transformers

1. Problem Statement

The detection of Mutual Gaze (MG) and Joint Attention (JA) is fundamental to developmental psychology, particularly in understanding infant-caregiver interactions. However, traditional analysis relies on labor-intensive, manual coding of video recordings, which is slow, prone to human error, and difficult to scale.

• Challenges: Automated detection in multi-camera laboratory settings is computationally difficult due to complex cross-camera relational dynamics.
• Limitations of Existing Methods:
  • Eye-tracking: High precision but low ecological validity (requires wearing devices or screen-based stimuli).
  • Convolutional Baselines: Often fail to capture the spatial and semantic relationships between two individuals across different camera views.
  • Multimodal LLMs: While powerful, they often lack the fine-grained temporal sensitivity required for specific behavioral boundaries and suffer from high computational costs and latency.

2. Methodology

The authors propose a Dual-Stream Transformer Architecture designed to process synchronized video streams from two cameras (one focused on the infant, one on the caregiver) to perform frame-level binary classification for MG and JA.

A. Architecture Overview

• Dual-Stream Input: The model processes two parallel video streams simultaneously.
• Frozen Backbones (GazeLLE): Instead of training a feature extractor from scratch, the authors utilize GazeLLE (a gaze-aware model based on DINOv2) as a frozen backbone. This model extracts rich visual priors and gaze-aware features directly from RGB inputs.
  • Two GazeLLE modules operate in parallel with shared weights.
  • A linear projection layer downsamples the high-dimensional embeddings to 512 dimensions to reduce computational overhead.
• Token Fusion Mechanism:
  • The projected token sequences from both camera views are concatenated.
  • A learnable [CLS] token is prepended to the sequence to serve as a global representation of the dual-camera state.
  • This sequence is fed into a Transformer Encoder (3 layers, 4 attention heads) which jointly models spatial and cross-view relationships, allowing the [CLS] token to aggregate contextual information from both the parent and the child.
• Classification Head: The updated [CLS] token is passed through a Multi-Layer Perceptron (MLP) with ReLU activations and LayerNorm to output a single logit, which is converted to a probability via a sigmoid function.

B. Dataset and Preprocessing

• Data Source: A semi-naturalistic dataset of 13 caregiver-infant dyads participating in seven observational sessions.
• Setup: Three synchronized cameras were used (Infant view, Parent view, Top-down), though the model utilizes only the Infant and Parent views.
• Preprocessing Pipeline:
  1. Synchronization: Audio cross-correlation was used to align the two camera streams, correcting hardware-induced offsets of up to 1 second.
  2. Frame Extraction: Videos were sampled at 1 fps to balance behavioral dynamics with computational efficiency.
  3. Face Detection: RetinaFace was used to detect faces and generate head bounding boxes (masks), which are required inputs for the GazeLLE backbone.
  4. Annotation: A custom React-based tool was developed for manual annotation, allowing researchers to label MG and JA events with high temporal precision.
• Data Splitting: A temporal split was used (first 10% for validation, rest for training) to prevent data leakage. The test set consisted of two held-out sessions.

3. Key Contributions

1. Novel Architecture: Introduction of a dual-stream Transformer that leverages frozen gaze-aware backbones (GazeLLE) and a custom token fusion mechanism to bridge the spatial gap between dual-camera views.
2. Extensive Evaluation: Rigorous comparison against a lightweight Convolutional Neural Network (CNN) baseline and a state-of-the-art Multimodal Large Language Model (LLM, Llama-4 Vision).
3. Open Science: The authors open-sourced the model architecture, pre-trained weights, and the test dataset, providing a parameter-efficient foundation for other researchers to fine-tune models for their specific laboratory environments.
4. Custom Tooling: Development of a custom annotation tool and a visualization interface for reviewing model predictions against ground truth.

4. Results

The proposed model was evaluated on a held-out test set using Accuracy, Precision, Recall, F1-score, and ROC-AUC.

• Mutual Gaze (MG) Detection:
  • Proposed Model: F1-score 0.825, AUC 0.834.
  • CNN Baseline: F1-score 0.655, AUC 0.429 (near random guessing).
  • LLM Baseline: F1-score 0.480, AUC 0.602.
• Joint Attention (JA) Detection:
  • Proposed Model: F1-score 0.767, AUC 0.862.
  • CNN Baseline: F1-score 0.589, AUC 0.681 (high variance in recall).
  • LLM Baseline: F1-score 0.610, AUC 0.595.

Key Findings:

• The Transformer model significantly outperformed both baselines in balancing precision and recall.
• The CNN baseline failed to generalize, suggesting that purely local spatial features are insufficient for modeling cross-camera relational dynamics.
• The LLM baseline showed a bias toward over-predicting events and lacked the temporal precision to delineate strict behavioral boundaries, resulting in lower F1 scores.
• Efficiency: The proposed model achieved an inference throughput of 9.46 samples/second on an NVIDIA A100, whereas the LLM achieved only 0.14 samples/second and required significantly more GPU memory (two 40GB GPUs vs. a single smaller footprint for the proposed model).

5. Significance and Impact

• Bridging the Gap: This work effectively bridges computational modeling and applied behavioral science by providing a scalable, automated tool for analyzing social interactions in naturalistic settings.
• Scalability: By reducing the reliance on manual coding, the tool enables the analysis of large-scale psychological datasets that were previously infeasible to process.
• Accessibility: The open-sourcing of pre-trained weights allows other labs to fine-tune the model with minimal data, overcoming the "domain shift" problem common in computer vision (e.g., different camera angles or lighting).
• Future Directions: The authors identify future work including temporal smoothing to reduce noise, expanding dataset diversity, integrating identity tracking to handle unconstrained interactions, and extending the architecture to multi-party (multi-camera) scenarios.

In conclusion, the paper presents a highly efficient, accurate, and accessible solution for automating the detection of critical social behaviors, demonstrating that specialized Transformer architectures outperform both traditional CNNs and general-purpose LLMs in this specific domain.