💻 computer science

HST-HGN: Heterogeneous Spatial-Temporal Hypergraph Networks with Bidirectional State Space Models for Global Fatigue Assessment

The paper proposes HST-HGN, a novel framework combining a hierarchical hypergraph network for capturing high-order spatial facial synergies and a bidirectional Mamba module for efficient long-range temporal modeling, to achieve state-of-the-art, real-time global driver fatigue assessment from untrimmed videos under constrained computational budgets.

Original authors: Changdao Chen

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Changdao Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard watching a security camera feed of a driver. Your job is to spot the exact moment they start to fall asleep (yawn) versus when they are just talking to a passenger.

The problem is that the video is long, the driver might be moving their head around, and the signs of sleepiness are tiny and subtle. Old computer programs trying to do this are either too slow (like a snail trying to run a marathon) or too dumb (they miss the big picture).

This paper introduces HST-HGN, a new "super-smart" AI system designed to be the ultimate security guard. Here is how it works, broken down into simple concepts:

1. The "Skip-Reading" Strategy (Global Sparse Sampling)

Imagine you have a 1-hour movie, but you only need to know if the main character is tired. Instead of watching every single second (which takes forever), you quickly flip through the movie, stopping only at key moments to check the character's face.

What the paper does: Instead of processing every single frame of a long video, the AI picks a few key frames spread out across the whole video. This saves massive amounts of computer power while still giving it the "big picture" of what's happening over time.

2. The "Two-Stream Detective" (Heterogeneous Feature Construction)

To understand a person, you need to look at two things: their shape and their skin.

The Shape Stream (Geometry): The AI looks at the "skeleton" of the face (where the nose, eyes, and mouth are). It uses a special trick called 3D Alignment to ignore if the driver is turning their head left or right. It only cares about the shape of the face changing (like the mouth opening wide).
The Skin Stream (Texture): The AI also zooms in on the actual pixels of the eyes and mouth. Is the eye actually closed? Is the skin around the mouth wrinkled?
The Magic: It combines these two streams. It's like having one detective checking the skeleton and another checking the skin, then having them share notes instantly.

3. The "Group Chat" vs. The "Handshake" (Hypergraph Networks)

This is the most creative part.

Old Way (Pairwise Graphs): Imagine a group of friends. Old AI models only let two friends talk to each other at a time (a handshake). If you want to understand a group hug, you have to simulate thousands of handshakes. It's inefficient and misses the group dynamic.
New Way (Hypergraphs): The HST-HGN uses a Hypergraph. Think of this as a Group Chat. One message can go to the eyes, the mouth, and the cheeks all at once.
Why it matters: When you yawn, your eyes squint, your jaw drops, and your cheeks move simultaneously. A "Group Chat" model sees this whole coordinated event as one single unit, making it much better at spotting a real yawn versus just talking.

4. The "Time-Traveling Librarian" (Bidirectional State Space Models)

To tell the difference between a yawn and talking, you need to see the whole story, not just a snapshot.

The Problem: Old AI models (like RNNs) read a story like a person reading a book from left to right. By the time they get to the end, they might have forgotten the beginning.
The New Way (Bi-Mamba): The AI uses a Bidirectional State Space Model. Imagine a librarian who can read the book forwards and backwards instantly.
- Forward: "The driver is opening their mouth."
- Backward: "The driver is closing their mouth."
- Result: By seeing the whole sequence at once, the AI knows, "Ah, this is a complete yawn cycle," rather than getting confused by a random mouth movement. It does this very quickly (linear speed), unlike older models that get slow as the video gets longer.

5. The "Focus Filter" (Attention & Pooling)

Finally, the AI has to decide: "Is this driver tired?"

It uses a Max Pooling filter. Imagine a sieve that only lets the most important grains of sand through. If the driver was normal for 100 seconds, then yawned for 2 seconds, the AI ignores the 100 seconds of "boring" data and focuses entirely on those 2 critical seconds where the yawn happened.

Why is this a big deal?

Speed: It's so light and fast that it could run on a small chip inside a car (an "edge device") without needing a supercomputer.
Accuracy: It beats all previous methods at spotting fatigue, even when the driver is moving their head or the lighting is bad.
Clarity: It can explain why it made a decision (e.g., "I saw the mouth open and the eyes close together"), which helps engineers trust the system.

In a nutshell: HST-HGN is a lightweight, super-fast AI that watches a driver's video, ignores the boring parts, uses a "group chat" to understand how facial muscles move together, and reads the whole timeline backwards and forwards to catch the exact moment a driver falls asleep.

1. Problem Statement

The paper addresses the challenge of driver fatigue assessment from untrimmed, long-duration videos under constrained computational budgets (edge devices). Key difficulties include:

Subtle Expressions: Fatigue manifests as gradual, subtle facial deformations (e.g., microsleep, yawning) that are hard to distinguish from similar actions like speaking.
Long-Range Dependencies: Accurate assessment requires capturing the full physiological lifecycle of an action (onset to offset) over extended temporal horizons, which traditional sliding-window methods fail to do.
Computational Constraints: Existing high-performance models (e.g., Transformers) suffer from quadratic complexity ( $O(T^2)$ ), making them unsuitable for real-time, resource-constrained in-cabin deployment. Conversely, lightweight models often lack the capacity to model high-order synergies or global context.
Pose Variations: Head movements and occlusions in driving scenarios can overwhelm subtle fatigue signals.

2. Methodology: HST-HGN Framework

The authors propose HST-HGN, a novel architecture combining a Heterogeneous Spatial-Temporal Hypergraph with Bidirectional State Space Models (Bi-Mamba). The framework consists of three core stages:

A. Global Sparse Sampling

Instead of dense sliding windows, the method employs a global sparse sampling strategy. From an untrimmed video of $N$ frames, a fixed-length sequence of $T=128$ frames is uniformly sampled. This reduces spatial redundancy while providing the temporal module with a global receptive field covering the entire video duration.

B. Heterogeneous Feature Construction (Spatial)

The model extracts dual-stream representations to capture complementary cues:

Geometry Stream:
- Extracts 68 facial landmarks using MediaPipe.
- Applies 3D Canonical Alignment via Procrustes analysis to remove rigid head pose variations (rotation, scale, translation), isolating pure non-rigid muscular deformations.
Texture Stream:
- Extracts localized Regions of Interest (ROIs) for the left eye, right eye, and mouth.
- Processes these patches using a lightweight Micro-CNN to generate "Texture Super-Nodes" representing appearance states (e.g., eye closure).

C. Spatial Dynamic Hypergraph Modeling

To model high-order synergies beyond pairwise connections, the authors construct a Heterogeneous Hypergraph:

Nodes: A set of 71 vertices comprising 68 geometric nodes and 3 texture super-nodes.
Hyperedges:
- Geo-Geo Hyperedges: Connect geometric landmarks to their $k$ -nearest neighbors to capture local structural dynamics.
- Tex-Geo Hyperedges: Use a star-topology to broadcast texture semantics (e.g., mouth texture) to all relevant geometric nodes (jaw, lips, cheeks).
Propagation: A Hypergraph Convolutional Neural Network (HGNN) layer propagates information, fusing local texture semantics with global geometric structures.
Pooling: An Adaptive Attention Pooling mechanism aggregates node features into a single frame-level vector, filtering noise and focusing on salient fatigue indicators.

D. Temporal Evolution Modeling (Bi-Mamba)

To handle long sequences efficiently:

Bi-Mamba Module: Replaces RNNs and Transformers with a Bidirectional State Space Model (Mamba).
Mechanism: It utilizes a selective scan mechanism with linear complexity ( $O(T)$ ). It processes the sequence in both forward and backward directions to capture complete temporal contexts (past and future relative to the current frame).
Advantage: This allows the model to dynamically filter noise (e.g., normal driving) while retaining critical transient events (e.g., the onset of a yawn), effectively distinguishing between similar actions like yawning and speaking.
Aggregation: A Global Max Pooling layer extracts the most discriminative temporal feature vector for final classification.

E. Optimization

The model is trained using a joint loss function:

Focal Loss: Addresses class imbalance and penalizes hard-to-distinguish boundary samples.
Center Loss: Enforces intra-class compactness, ensuring features for the same class (e.g., "Yawning") cluster tightly regardless of individual identity or pose.

3. Key Contributions

Novel Architecture: First integration of Heterogeneous Hypergraphs (for high-order spatial synergy) with Bidirectional SSMs (for efficient long-range temporal modeling) in driver fatigue detection.
Pose-Invariant & Multimodal: Introduction of a 3D Canonical Alignment mechanism and a Texture-Geometry broadcasting strategy that decouples pose variations from fatigue signals.
Efficiency & Interpretability: Achieves linear temporal complexity ( $O(T)$ ), making it suitable for edge deployment. The framework offers inherent interpretability by highlighting critical spatial regions and temporal frames.
State-of-the-Art Performance: Demonstrates superior accuracy and efficiency compared to existing generic video models and domain-specific fatigue detectors.

4. Experimental Results

The method was evaluated on the YawDD dataset and cross-validated on UTA-RLDD, FatigueView, and DMD.

Accuracy: On YawDD, HST-HGN achieved 98.57% Accuracy and 98.28% Macro F1, outperforming the second-best method (LiteFat at 97.14%) and generic models like VideoMAE (92.86%).
Generalization: Maintained robust performance (e.g., 97.10% Acc on DMD) in cross-dataset evaluations, proving strong domain adaptability.
Efficiency:
- Parameters: Only 0.30M (vs. 33M+ for SlowFast/VideoMAE).
- Computation: 2.90 G FLOPs per clip.
- Throughput: 26.45 clips/s on an NVIDIA RTX 3060, satisfying real-time requirements (>25 FPS) while maintaining high accuracy.
Ablation Studies: Confirmed that removing 3D alignment, Hypergraphs, Texture fusion, or Bi-Mamba significantly degrades performance. The joint Focal+Center loss was crucial for resolving the "Talking vs. Yawning" ambiguity.

5. Significance

This work bridges the gap between high-accuracy fatigue detection and real-time edge deployment. By replacing computationally expensive attention mechanisms with efficient State Space Models and utilizing hypergraphs to capture complex muscle synergies, HST-HGN provides a practical solution for in-cabin monitoring systems. Its ability to distinguish subtle physiological states (like yawning vs. speaking) with low latency makes it a significant advancement for automotive safety applications.