🤖 machine learning

Inclusive AI for Group Interactions: Predicting Gaze-Direction Behaviors in People with Intellectual and Developmental Disabilities

This paper advances inclusive AI for group interactions involving people with Intellectual and Developmental Disabilities by introducing the MIDD dataset, analyzing gaze behavior differences compared to neurotypical populations, and evaluating machine learning models alongside therapist insights to guide the development of more effective, human-centered tools.

Original authors: Giulia Huang, Maristella Matera, Micol Spitale

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giulia Huang, Maristella Matera, Micol Spitale

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to play a game of "Hot Potato" with a group of friends. The robot has been trained for years watching groups of typical, neurotypical people. It knows the rules: If someone looks at you, they are talking. If they look away, they are listening.

But now, you introduce a new group of friends: people with Intellectual and Developmental Disabilities (IDD). Suddenly, the robot gets confused. It thinks everyone is silent because no one is making "perfect" eye contact. It misses the turns, it interrupts, and it fails to understand the conversation.

This paper is about fixing that robot so it can play fairly with everyone, not just the "average" person.

Here is the story of their journey, broken down into simple parts:

1. The Problem: The Robot's "Rulebook" is Outdated

The authors explain that most AI systems today are like students who only studied one specific textbook. That textbook was written by and for neurotypical people.

The Issue: In typical groups, people look at each other when they speak. In groups with IDD, people might look at the floor, look at a wall, or use gestures instead of eyes to communicate.
The Result: The AI, relying on its old textbook, thinks, "Oh, no one is looking, so no one is talking!" It fails to include people who communicate differently.

2. The Solution: A New "Field Guide" (The MIDD Dataset)

To fix this, the researchers went out and filmed real conversations with 13 adults with IDD. They called this new collection of data MIDD (Multi-party Interaction with Intellectual and Developmental Disabilities).

Think of this as the robot finally getting a new field guide that shows how these specific friends actually behave.

They filmed them in a circle, just like the old textbook did, so they could compare apples to apples.
They found that in these groups, people look away much more often (66% of the time vs. 34% in typical groups).
They found that the lighting was often dimmer and the camera angles were trickier.

3. The Experiment: Testing the Robot's Eyes

The researchers took three different types of "robots" (AI models) and tested them:

The Old School Robot (SVC & XGBoost): These are like calculators. They look at numbers (head position, eye direction) and make a guess.
The Super-Brain Robot (FSFNet): This is a deep learning model, like a student who has read millions of books and can "see" patterns in faces.

What happened?

When they tested these robots on the old textbook data (neurotypical people), the Super-Brain was amazing.
When they tested them on the new field guide (MIDD), the robots stumbled. They were so used to seeing "perfect" eye contact that they got lost when people looked away.
The Fix: They tried "fine-tuning" the robots. It's like giving the Super-Brain a crash course specifically on the new field guide. It helped, but the robots still struggled with the heavy imbalance (so many people looking away).

4. The "Team-Up" Strategy

The researchers realized that one robot wasn't enough. So, they created a Dream Team.

They combined the "Super-Brain" (which is great at seeing faces) with the "Calculator" (which is good at tracking who is speaking).
The Analogy: Imagine a detective (Super-Brain) who looks at the suspect's face, working with a witness (Calculator) who knows who was talking. Together, they are much better at solving the mystery than either one alone.
The Result: This team-up improved the accuracy significantly, making the AI much better at guessing who was talking, even when eye contact was messy.

5. The Human Element: Talking to Therapists

Here is the most important part. The researchers didn't just trust the numbers; they sat down with six therapists (experts who work with these individuals) to ask, "What does this data actually mean?"

The therapists offered a "lightbulb moment":

The "Stimming" Misunderstanding: The AI saw a person rocking back and forth or looking at the floor and thought, "They are disengaged." The therapist said, "No! That's them regulating their emotions. They are still listening."
The Role of the Teacher: In these groups, a teacher often acts as a "social bridge," guiding the conversation with a glance or a touch. The AI didn't know to look for the teacher's help.
Silence is Golden: Sometimes, silence combined with a subtle look is a deliberate way of saying, "I'm thinking." The AI often missed this.

The Big Takeaway

This paper teaches us that AI cannot be "one size fits all."

If you build a tool to help people interact, you have to build it with the specific needs of all people in mind. You can't just train it on the "average" person. You need:

New Data: Real recordings of diverse groups.
New Tools: Robots that look at more than just eyes (like head position and who is speaking).
Human Wisdom: Listening to experts to understand why people behave the way they do, not just what they are doing.

In short: By updating the robot's "rulebook" and listening to human experts, we can build AI that doesn't just see people, but truly understands them, making group conversations fair and inclusive for everyone.

1. Problem Statement

Current Artificial Intelligence (AI) systems designed to mediate group interactions (e.g., detecting turn-taking, gaze direction, and attention) are predominantly trained on neurotypical data. This creates a significant barrier for Inclusive AI, particularly for individuals with Intellectual and Developmental Disabilities (IDD).

The Gap: Individuals with IDD often exhibit atypical interaction patterns, including reduced or indirect gaze, reliance on non-verbal cues (gestures, vocalizations), and turn-taking rhythms that diverge from neurotypical norms.
The Consequence: Models trained on standard datasets fail to generalize to IDD populations, leading to systematic errors in gaze detection. This risks marginalizing individuals with IDD by misinterpreting their engagement or failing to facilitate their participation in group settings.
The Challenge: There is a lack of annotated datasets capturing multi-party interactions involving adults with IDD, making it difficult to train or adapt robust eye-contact detection models.

2. Methodology

A. Dataset Creation: MIDD

The authors introduced the Multi-party Interaction with Intellectual and Developmental Disabilities (MIDD) dataset to address the data scarcity.

Participants: 13 adults (9 female, 4 male) with heterogeneous IDD profiles (including Down syndrome, autism, and moderate/severe intellectual disabilities).
Setup: Six group conversations (4–5 participants each) recorded in a semi-circular arrangement using 4 synchronized webcams and a central microphone, mirroring the setup of the neurotypical MultiMediate benchmark.
Data Structure: 24 synchronized videos (~476 minutes total), segmented into non-overlapping 10-second chunks.
Annotations:
- Gaze Direction: Categorized as None, Left, Front, or Right relative to the participant.
- Speaker Identity: Active speaker labeling.
- Quality Control: Two independent annotators achieved 82% inter-annotator agreement.
Imbalance: The dataset is highly imbalanced, with 66.6% of samples labeled as "no eye contact."

B. Comparative Analysis

The authors performed a rigorous statistical comparison between MIDD and the neurotypical MultiMediate dataset:

Gaze Engagement: MIDD showed significantly lower gaze engagement (33% vs. 65% in MultiMediate).
Speaking Activity: Active speaking events were less frequent in MIDD (62% vs. 92%).
Inequality: Speaking time distribution was more unequal in MIDD (Gini coefficient 0.52 vs. 0.34), indicating a few participants dominated the conversation.
Image Quality: MIDD frames exhibited lower brightness and contrast, posing greater visual challenges for computer vision models.

C. Modeling and Evaluation

The study evaluated three distinct model architectures on the MIDD dataset:

SVC (Support Vector Classifier): A traditional machine learning model using handcrafted features (OpenFace 2.0 gaze vectors, head pose, and speaker cues).
XGBoost: A gradient-boosted tree ensemble trained on the same 18-dimensional feature vectors.
FSFNet: A state-of-the-art deep learning model (ResNet-50 backbone + Transformer encoder) designed for multi-party eye contact detection.

Experimental Strategies:

Zero-Shot/Transfer Learning: Models pre-trained on MultiMediate were tested on MIDD.
Fine-Tuning: Models were fine-tuned on MIDD data, including strategies for class imbalance (e.g., WeightedRandomSampler).
Ensemble Methods: The authors tested Weighted Averaging and Stacking to combine the visual strengths of FSFNet with the structured feature strengths of SVC/XGBoost.
Ablation Study: Investigated the impact of specific feature subsets (gaze, pose, speaker cues) on performance.

3. Key Contributions

MIDD Dataset: The first publicly available, carefully annotated corpus of multi-party group conversations involving adults with IDD, designed to mirror existing benchmarks while capturing neurodiverse behaviors.
Behavioral Analysis: A quantitative characterization of the distributional shifts between neurotypical and IDD interactions, highlighting severe class imbalance, reduced gaze frequency, and variable image quality.
Model Benchmarking: A comprehensive evaluation of classical (SVC, XGBoost) and deep learning (FSFNet) models, demonstrating that while fine-tuning improves performance, significant limitations remain in generalizing to IDD populations.
Expert Validation: Integration of insights from a focus group with six therapists to interpret quantitative findings, revealing that "atypical" behaviors (e.g., lack of gaze, repetitive movements) are often valid forms of communication or self-regulation, not disengagement.

4. Key Results

Performance Metrics

Baseline Failure: Models trained solely on neurotypical data (MultiMediate) performed poorly when applied directly to MIDD (Zero-shot), confirming the domain gap.
Fine-Tuning: Fine-tuning on MIDD improved performance, but models still struggled with the extreme class imbalance.
- FSFNet: Achieved the highest validation accuracy (~~82%) and macro-F1 (~~0.71) after fine-tuning.
- Classical Models (SVC/XGBoost): Suffered from bias toward the majority class ("no eye contact"). For instance, SVC achieved 69% accuracy but only a 0.44 macro-F1 score.
Ensemble Success: The best results were achieved by ensembling FSFNet (visual features) with XGBoost (structured features).
- Best Configuration: Weighted average (0.6 FSFNet / 0.4 XGBoost).
- Performance: 70% Accuracy, 0.55 Macro-F1, and a 0.60 Composite Score.
- Significance: McNemar's tests confirmed the ensemble's superiority over base models ( $p < .001$ ).

Ablation Findings

Pose vs. Gaze: Models relying heavily on head pose features performed well on validation data but collapsed on the test set (predicting almost exclusively the majority class). This was attributed to overfitting and sensitivity to spatial layout inconsistencies (e.g., 5-person groups vs. 4-person training assumptions).
Robust Features: Configurations using gaze direction and speaker cues (excluding pose) generalized better, yielding more balanced predictions across minority classes.

Expert Insights

Therapists clarified that:

Scaffolding: Educators often guide turn-taking via non-verbal cues, which AI models must account for.
Multimodality: Communication in IDD groups relies heavily on gestures, touch, and symbolic tools, not just gaze.
Reinterpretation: "Stimming" (repetitive movements) and silence should not be interpreted as disengagement by AI systems.

5. Significance and Implications

This work provides a foundational step toward Inclusive AI by demonstrating that:

Neurotypical models do not generalize: Direct transfer of gaze detection models to IDD populations fails due to behavioral and environmental differences.
Data-driven strategies are essential: Building inclusive tools requires:
- Multimodal Feature Fusion: Combining gaze, head pose, and speaker identity to capture the full spectrum of interaction.
- Domain Adaptation: Techniques to align distributions between neurotypical and neurodiverse data.
- Context-Aware Modeling: Integrating expert knowledge to interpret subtle or non-standard communicative acts.
Practical Application: The findings suggest that future AI agents for therapeutic or well-being contexts must move beyond simple gaze detection to understand complex, scaffolded, and multimodal social dynamics to ensure equitable participation for people with IDD.