🤖 machine learning

Attention-Guided Dual-Stream Learning for Group Engagement Recognition: Fusing Transformer-Encoded Motion Dynamics with Scene Context via Adaptive Gating

This paper proposes DualEngage, a novel dual-stream framework that fuses transformer-encoded individual motion dynamics with scene-level spatiotemporal context via adaptive gating to achieve high-accuracy group engagement recognition in classroom videos.

Original authors: Saniah Kayenat Chowdhury, Muhammad E. H. Chowdhury

Published 2026-04-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Saniah Kayenat Chowdhury, Muhammad E. H. Chowdhury

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy classroom as a bustling orchestra. Some students are playing their instruments with passion (highly engaged), some are just tapping their feet occasionally (medium engagement), and others are daydreaming or looking out the window (low engagement).

For a long time, teachers and computers have tried to figure out who is paying attention. Most systems only looked at one musician at a time, checking if their eyes were open or if they were nodding. But in a group setting, the magic isn't just about the individual; it's about the vibe of the whole room. Sometimes a whole class is silent and still, but everyone is deeply focused. Other times, they are moving around, but not really learning.

This paper introduces a new system called DualEngage (Dual = Two). It's like hiring two different detectives to solve the mystery of "Who is engaged?" and then having them compare notes before making a final decision.

The Two Detectives (The Two Streams)

Detective #1: The "Motion Tracker" (The Primary Stream)

What they do: This detective zooms in on every single student. They don't just look at faces; they watch how people move. They use a special tool (called Optical Flow) that acts like a "wind sensor" for pixels. It sees the subtle shifts in posture, the fidgeting, the leaning forward, or the turning of the head.
The Superpower: It uses a "Transformer" (think of it as a super-smart memory bank) to remember how a student moved over time. Did they lean in at the start of the lecture and stay there? Or did they start fidgeting after five minutes?
The Problem: Sometimes, a student is perfectly still because they are too focused. If this detective only looked at movement, they might think, "Oh, no movement means no engagement!" which would be a mistake.

Detective #2: The "Room Vibe" Expert (The Secondary Stream)

What they do: This detective steps back and looks at the entire classroom as one big picture. They don't care about individual students; they care about the "group energy."
The Superpower: They use a 3D camera brain (a 3D ResNet) to see patterns like: Is the whole class leaning forward together? Are they all looking at the teacher? Is the room synchronized?
The Problem: This detective might miss the quiet student in the back who is actually the most engaged, because they are too focused on the big picture.

The Smart Manager: The "Gated Fusion"

Here is the genius part. In the past, computers would just take the notes from both detectives and mash them together equally. But that's like asking a weather forecaster and a traffic cop to decide if you should wear a coat, without letting them talk to each other first.

DualEngage has a Smart Manager (called Softmax-Gated Fusion).

How it works: The manager looks at the current situation and asks: "Which detective is more reliable right now?"
- Scenario A: The room is chaotic, and students are moving a lot. The manager says, "Okay, Detective #1 (Motion) is doing great work. Let's listen to them more."
- Scenario B: The room is silent and still. The manager says, "Detective #1 is confused because everyone is still. But Detective #2 (Room Vibe) sees that everyone is staring at the board in unison. Let's trust the Room Vibe more!"

The manager dynamically adjusts the "volume" of each detective's voice based on what's happening in the video.

The Results: A Standing Ovation

The researchers tested this system on real classroom videos with groups of students.

The Score: It got it right 96% of the time.
Why it matters: Previous systems struggled when students were quiet or when the camera angle was tricky. This system figured out that "stillness" can mean "focus" if the whole group is synchronized, and "movement" can mean "distraction" if the group is out of sync.

The Bottom Line

Think of DualEngage as a teacher who has a superpower: they can watch every single student's body language and feel the energy of the whole room at the same time. They know when to focus on the individual and when to focus on the group.

By combining individual motion (how you move) with group context (how the room feels) and using a smart switch to decide which one matters more at any given second, this system can finally understand the complex, messy, and beautiful reality of a real classroom. It's a big step toward helping teachers know exactly when their students are truly learning.

1. Problem Statement

Student engagement is a critical predictor of academic success, yet automated recognition of group-level engagement in physical classrooms remains a significant challenge. Existing methods suffer from several limitations:

Individual-Centric Focus: Most models analyze single students or rely on facial cues (expressions, gaze), which fail to capture collective group dynamics and social interactions.
Environmental Constraints: Traditional pose-based methods are fragile in "in-the-wild" classroom settings due to occlusions, varying lighting, and crowded layouts.
Lack of Context: Current systems often neglect the global scene context (e.g., group synchrony, teacher pacing) that influences individual behavior.
Motion vs. Stillness: High engagement does not always correlate with high motion (e.g., students listening intently), while low engagement can sometimes involve subtle movements. Models need to dynamically weigh motion cues against scene context.

The authors propose DualEngage, a novel framework designed to recognize group engagement levels (High, Medium, Low) by fusing individual motion dynamics with global scene context.

2. Methodology: DualEngage Architecture

The proposed architecture is a two-stream deep learning framework that processes video clips through parallel branches before fusing them via an adaptive gating mechanism.

A. Primary Stream: Person-Level Motion Dynamics

This stream focuses on extracting and modeling the temporal evolution of individual student movements.

Detection & Tracking: Students are detected and tracked across frames using OpenMMLab libraries (Faster R-CNN for detection, Deep SORT for tracking) to generate unique IDs and bounding boxes.
Optical Flow Extraction: Instead of relying on unstable keypoints, the system computes dense optical flow using the RAFT (Recurrent All-Pairs Field Transforms) network. This captures subtle, non-rigid motions even under partial occlusion.
Motion Tubes: For each tracked student, a "motion tube" is created by cropping the optical flow fields corresponding to their bounding box over time.
Temporal Encoding (Transformer): A shared Transformer Encoder processes the sequence of motion features for all students. Unlike RNNs, the Transformer's self-attention mechanism effectively captures long-range temporal dependencies and behavioral patterns.
Attention Pooling: To handle varying engagement levels among group members, the model uses attention pooling rather than simple mean pooling. It learns to assign higher weights to students exhibiting the most behaviorally salient motion, creating a unified motion representation ( $M$ ).

B. Secondary Stream: Scene-Level Global Context

This stream captures the holistic dynamics of the entire classroom, which cannot be attributed to any single student.

Backbone: A pre-trained 3D ResNet-18 (R3D) processes the full video clip (RGB frames).
Feature Extraction: It extracts spatiotemporal features representing group synchrony, shared attention shifts, and overall classroom layout.
Output: This produces a global scene embedding ( $G$ ) that is invariant to the number of students.

C. Adaptive Gated Fusion

Instead of simple concatenation, the authors introduce a Softmax-Gated Fusion mechanism.

The motion representation ( $M$ ) and scene embedding ( $G$ ) are concatenated and passed through a lightweight MLP.
The MLP outputs two scalar weights ( $\alpha, \beta$ ) via a Softmax function, ensuring $\alpha + \beta = 1$ .
The final fused representation is a convex combination: $h = \alpha \cdot M + \beta \cdot G$ .
Significance: This allows the model to dynamically adapt. If individual motion is ambiguous (e.g., students are still), the model relies more on scene context ( $\beta$ ). If individual behaviors are distinct, it relies more on motion ( $\alpha$ ).

3. Key Contributions

Dual-Stream Architecture: The first engagement recognition system to explicitly model engagement as a joint function of individual motion dynamics and collective scene context.
Transformer-Encoded Motion: Utilization of a Transformer encoder to model long-range temporal dependencies in dense optical flow, capturing subtle engagement-linked movements better than RNNs or CNNs.
Attention Pooling: A mechanism to weight individual student contributions based on behavioral salience, avoiding the noise introduced by averaging all students equally.
Softmax-Gated Fusion: An adaptive fusion strategy that dynamically balances the reliance on local motion vs. global context based on the specific input sample.
Comprehensive Evaluation: Extensive ablation studies validating the necessity of each component (temporal encoding, attention, gating) and backbone comparisons.

4. Experimental Results

Dataset: Evaluated on the OUC-CGE (Ocean University of China Classroom Group Engagement) dataset, containing ~6,700 video clips labeled as High, Medium, or Low engagement.
Performance:
- Accuracy: $0.9621 \pm 0.0161$
- Macro-Averaged F1 Score: $0.9530 \pm 0.0204$
- Macro-Recall: $0.9561 \pm 0.0165$
Ablation Study Findings:
- Single Streams: The Primary Stream alone achieved ~53% accuracy; the Secondary Stream alone achieved ~62%. This confirms that neither motion nor context alone is sufficient.
- Temporal Modeling: Removing the Transformer encoder caused a sharp drop in performance, proving the necessity of modeling long-range temporal dependencies.
- Fusion Strategy: Replacing the gated fusion with simple concatenation or removing attention pooling resulted in significant performance degradation (dropping F1 to ~0.87).
- Backbones: While R(2+1)D and I3D backbones performed well, the 3D ResNet-18 offered the best balance of accuracy and efficiency for this dual-stream setup.

5. Significance and Future Work

Significance: DualEngage addresses a critical gap in educational technology by moving beyond individual facial analysis to a holistic, group-level understanding of engagement. It demonstrates that motion cues (even subtle ones) combined with global context are powerful predictors of learning outcomes. The adaptive gating mechanism makes the model robust to scenarios where students are physically still but mentally engaged.
Limitations: The approach is computationally expensive due to dense optical flow (RAFT) and per-student tracking. It also relies on accurate detection; heavy occlusion or tracking failures can degrade performance.
Future Directions: The authors suggest incorporating teacher behavior analysis, fine-grained gaze estimation, and posture cues to further disambiguate cases where motion and scene context provide conflicting signals.

In conclusion, this work establishes a new state-of-the-art for group engagement recognition by effectively leveraging the synergy between individual motion dynamics and collective scene context through a sophisticated, attention-guided dual-stream architecture.