💻 computer science

MSGL-Transformer: A Multi-Scale Global-Local Transformer for Rodent Social Behavior Recognition

The paper proposes MSGL-Transformer, a multi-scale global-local transformer architecture featuring a Behavior-Aware Modulation block that effectively recognizes rodent social behaviors from pose-based temporal sequences, achieving state-of-the-art performance on both the RatSI and CalMS21 datasets.

Original authors: Muhammad Imran Sharif, Doina Caragea

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammad Imran Sharif, Doina Caragea

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a wildlife documentary filmmaker trying to understand the secret social lives of rats and mice. You have hours of video footage showing them interacting. Your goal is to figure out exactly what they are doing at any given moment: Are they just hanging out alone? Are they chasing each other? Are they having a nose-to-nose chat? Or are they fighting?

The Problem: The "Human Eye" is Too Slow and Too Tired
Traditionally, scientists had to watch these videos frame-by-frame and manually label every action. It's like trying to read a novel by looking at one letter at a time. It takes forever, and humans get tired, making mistakes. Sometimes, a rat makes a tiny, quick movement that looks like a sneeze but is actually a signal to attack. A human might miss it, or confuse it with just moving away.

The Solution: A Super-Powered AI Detective
The authors of this paper built a new AI detective called MSGL-Transformer. Think of this AI not just as a camera, but as a super-observant detective who can see time in three different ways simultaneously.

Here is how it works, using some fun analogies:

1. The "Zoom Lens" of Time (Multi-Scale Attention)

Most AI models are like a camera with a fixed zoom. They either look at the whole scene (long-term patterns) or just the immediate action (short-term movements), but rarely both at once.

The Short-Range Lens: Imagine watching a rat's head twitch. This happens in a split second. The AI has a "short-range lens" that focuses on these tiny, rapid movements (like a nose twitch or a quick turn).
The Long-Range Lens: Imagine watching a rat follow another rat for a long time. This is a slow, steady pattern. The AI has a "long-range lens" that looks at the whole story over many seconds.
The Magic: The MSGL-Transformer is unique because it uses both lenses at the same time. It can see the tiny twitch and understand that it's part of a longer chase. This helps it tell the difference between "moving away" (a quick retreat) and "following" (a long pursuit), which often look similar if you only look at one part of the timeline.

2. The "Highlighter Pen" (Behavior-Aware Modulation)

Imagine you are reading a long, boring book, but you need to find the exciting parts. You use a highlighter pen to mark the important sentences and ignore the boring filler words.

The AI has a special tool called the BAM block (Behavior-Aware Modulation). Before it tries to understand the story, it acts like a highlighter pen. It scans the rat's movements and says, "Okay, this specific movement is really important for knowing if they are fighting, so I'll turn up the volume on that signal. This other movement is just random noise, so I'll turn it down." This ensures the AI focuses on the clues that actually matter.

3. No Pre-Made Maps (Skeleton-Free Learning)

Many old AI models for animal movement are like hikers who need a pre-drawn map of the terrain. They are told, "The nose is connected to the ear, and the ear is connected to the tail," and they can only move along those lines.

The MSGL-Transformer is different. It doesn't need a pre-drawn map. It just looks at the coordinates of the body parts (like dots on a graph) and figures out the relationships on its own. This makes it very flexible. It's like a hiker who can navigate any forest without a map, just by looking at the trees and the path.

The Results: A Master of Two Worlds

The researchers tested this AI on two different "forests":

RatSI: A dataset of rats (12 body points tracked).
CalMS21: A dataset of mice (28 body points tracked).

The Amazing Part: They used the exact same AI brain for both rats and mice. They didn't have to rebuild the model or change its logic. They just changed the size of the "input box" to fit the different number of body points.

On Rats: It beat all the previous methods (like standard video watchers) by a significant margin.
On Mice: It was even better, improving accuracy by over 10% compared to the best existing tools.

Why Does This Matter?

Think of this AI as a translator for the animal kingdom. By accurately understanding these behaviors, scientists can:

Study how stress or drugs affect social interactions without human bias.
Understand diseases that change how animals behave.
Save thousands of hours of human labor.

In a Nutshell:
The MSGL-Transformer is like a super-smart, tireless observer that can watch a rat or mouse, zoom in on a split-second twitch, zoom out to see a long chase, highlight the most important clues, and tell you exactly what's happening—all without needing a pre-drawn map or getting tired. It's a giant leap forward in letting computers understand the complex social lives of our furry lab friends.

1. Problem Statement

Recognizing rodent social behavior is critical for neuroscience and pharmacological research, yet traditional manual scoring is labor-intensive, prone to human error, and struggles with rapid or overlapping movements. While deep learning has automated this process, existing approaches face two primary limitations:

Temporal Scale Mismatch: Rodent behaviors occur at vastly different temporal scales. Some behaviors (e.g., "Solitary" or "Following") span long durations with continuous motion, while others (e.g., "Moving Away" or "Nose Contact") are brief, subtle, and occur over only a few frames. Models focusing solely on local context miss long-range dependencies, while those focusing only on global context miss subtle, short-term cues.
Class Imbalance: Datasets are heavily dominated by long-duration behaviors (e.g., solitary activity), making it difficult for models to learn rare but behaviorally significant events (e.g., aggression or pinning).
Rigid Architectures: Many existing skeleton-based transformers rely on predefined spatial graphs or fixed temporal windows, limiting their flexibility and generalizability across different species or pose configurations.

2. Methodology: MSGL-Transformer

The authors propose MSGL-Transformer, a lightweight, pose-based sequence model designed to capture multi-scale temporal dynamics without relying on predefined spatial graphs.

Core Architecture

The model processes temporal sequences of pose coordinates ( $T \times D$ ) through five key components:

Input Embedding & Global Token: Pose vectors are linearly projected to a model dimension ( $d=64$ ). A learnable global token (similar to [CLS] in ViT) is prepended to summarize the entire sequence.
Behavior-Aware Modulation (BAM) Block: Inspired by Squeeze-and-Excitation (SE) networks, this block dynamically reweights temporal embeddings. It flattens non-global tokens, passes them through a two-layer FC network with ReLU and Sigmoid activations to generate a modulation vector, and applies this to the features. This emphasizes behavior-relevant motion patterns before attention is applied.
Multi-Scale Attention Module: This is the core innovation, consisting of three parallel branches to capture dependencies at different scales:
- Short-range Branch: Processes the first $\lfloor T/2 \rfloor$ frames using causal masking to capture rapid, local motion cues.
- Medium-range Branch: Processes all $T$ frames using causal masking to capture intermediate dynamics.
- Global Branch: Processes all $T+1$ tokens (including the global token) using full bidirectional attention (no masking) to capture long-range dependencies.
- Fusion: The outputs of the short and medium branches are averaged on overlapping positions to form a local representation ( $A_{local}$ ), which is then combined with the global representation ( $A_{global}$ ) via residual addition and layer normalization.
Lightweight Transformer Encoder: A standard 2-layer transformer encoder processes the fused representation.
Classification Head: The global token is extracted, passed through dropout and a linear layer, and classified using softmax.

Data & Preprocessing

Datasets: Evaluated on RatSI (rats, 12D pose, 5 classes) and CalMS21 (mice, 28D pose, 4 classes).
Windowing: Uses a sliding window of $T=35$ frames with a stride of 1. The label for a window is the behavior of the final frame.
Preprocessing: Includes mean imputation for missing keypoints and standardization (zero mean, unit variance).

3. Key Contributions

Multi-Scale Global-Local Attention: A novel architecture that explicitly models short, medium, and long-term temporal dependencies in parallel, addressing the multi-scale nature of rodent behaviors.
Behavior-Aware Modulation (BAM): A mechanism that dynamically reweights temporal features to highlight behavior-relevant dynamics, improving sensitivity to subtle cues.
Graph-Free Flexibility: Unlike prior skeleton-based methods (e.g., ST-GCN), MSGL-Transformer operates directly on pose coordinate sequences without predefined spatial graphs, allowing it to generalize across different species and keypoint layouts with minimal architectural changes.
State-of-the-Art Performance: Demonstrated superior performance on two distinct datasets, outperforming both sequential baselines (TCN, LSTM) and other transformer/graph-based methods.

4. Experimental Results

RatSI Dataset (Rats)

Setup: 9-fold cross-validation (leave-one-video-out).
Performance: Achieved a mean accuracy of 75.4% and F1-score of 0.745.
Comparison: Outperformed TCN, LSTM, and Bi-LSTM baselines.
Ablation: Removing either the BAM block or the Multi-Scale Attention (MSA) module resulted in significant performance drops, confirming the necessity of both components.
Challenges: The "Moving Away" class (4.4% of data) remained difficult (F1 ~0.19) due to severe class imbalance and motion ambiguity.

CalMS21 Dataset (Mice)

Setup: Official Task 1 test set.
Performance: Achieved 87.1% accuracy and 0.8745 F1-score.
Comparison:
- Surpassed HSTWFormer (the previous SOTA) by +10.7% in average per-class accuracy.
- Outperformed other keypoint-based methods (ST-GCN, MS-G3D, CTR-GCN, STGAT) by significant margins (e.g., +15.5% over CTR-GCN).
Generalization: The exact same architecture and hyperparameters used for RatSI were applied to CalMS21, with only input dimension ( $D=28$ ) and output classes ( $C=4$ ) adjusted. This proves strong cross-species generalization.
Boundary Analysis: Errors were concentrated near behavior transition points (accuracy dropped from ~92% in stable segments to ~49% at transitions), highlighting a future area for improvement.

5. Significance and Conclusion

The MSGL-Transformer represents a significant advancement in automated animal behavior analysis by effectively bridging the gap between local motion cues and global interaction patterns. Its ability to generalize across different species (rats vs. mice), pose configurations (12D vs. 28D), and behavior vocabularies without architectural redesign makes it a highly robust tool for neuroscience research.

While the model excels at recognizing dominant behaviors, the authors acknowledge that class imbalance remains a critical challenge for rare behaviors (e.g., "Attack" in mice, "Moving Away" in rats). Future work will focus on class-aware loss functions and data augmentation to further improve the detection of these minority classes. The code and models are planned for public release to facilitate further research in this domain.