MTT-Bench: Predicting Social Dominance in Mice via Multimodal Large Language Models

Technical Summary: MTT-Bench: Predicting Social Dominance in Mice via Multimodal Large Language Models

1. Problem Statement

Quantifying animal behavior is a cornerstone of neuroscience and ethology, yet traditional methods often rely on manual annotation or domain-specific, supervised models that require handcrafted features (e.g., pose estimation). A critical challenge in behavioral studies is predicting social dominance hierarchies. While the "mouse tube test" is a gold-standard behavioral assay, scaling its analysis through automation remains difficult because dominance is a latent trait derived from complex, unstructured, and subtle temporal interactions rather than simple locomotor patterns.

2. Methodology

The authors propose MTT-Bench (Mouse-Tube-Test-Benchmark) to evaluate whether general-purpose Multimodal Large Language Models (MLLMs) can reason about social hierarchy from raw video without task-specific training.

• Dataset Construction: The researchers conducted 11 groups of mouse tube experiments. Each group included four training videos (showing individual mouse behavior) and one or two testing videos (showing pairwise confrontation). To compensate for the small sample size, they employed data augmentation (flipping, grayscale adjustments, and rotation), expanding the dataset eightfold to 88 experimental groups.
• Prediction Frameworks: The paper evaluates two distinct approaches:
  1. Zero-Shot Learning (ZSL) Prediction: This method leverages the reasoning capabilities of MLLMs. The model is prompted with three sequential questions: (1) Assign a personality score (0–1) to Mouse A based on video frames; (2) Assign a score to Mouse B; (3) Synthesize these scores to predict the winner of a tube test and provide a rationale.
  2. K-means Clustering Prediction: An unsupervised approach used as a baseline. The model encodes videos into high-dimensional embeddings. Using a small subset of "known" dominant and submissive videos, the method establishes "behavioral prototypes" (centroids). New mice are classified based on their Euclidean distance to these dominant/submissive centroids.
• Evaluation Baselines: The models are compared against Human Agents (30 individuals with no prior knowledge) to establish a biological/human performance baseline.

3. Key Contributions

• MTT-Bench: A novel multimodal benchmark specifically designed to link naturalistic mouse interaction videos with validated social dominance labels.
• New Paradigm for Ethology: Demonstrates a shift from domain-specific, supervised "pose-to-behavior" pipelines toward using foundation models for high-level, zero-shot social reasoning.
• Comparative Analysis: Provides a rigorous comparison between semantic-based reasoning (ZSL) and feature-based clustering (K-means) in the context of animal behavior.

4. Results and Analysis

The performance varied significantly across different model architectures (Table I):

• Top Performers (InternVL Series): The InternVL3-2B model achieved the highest accuracy at 76.14% using the Zero-Shot method, outperforming both the human baseline (51.44%) and larger models like InternVL3-8B (72.27%).
• Model Scaling Nuances: Interestingly, the largest model (InternVL3-8B) was more "conservative," often producing neutral or uncertain responses when behavioral cues were subtle, whereas the smaller 1B model made "bolder" but potentially less accurate predictions.
• Failure of GPT Series: GPT-4o and GPT-4-turbo performed poorly (approx. 21%), largely because they struggled to interpret the video content or refused to assign numerical personality scores, citing a lack of specific methodology.
• K-means vs. ZSL: The K-means method (using InternVL3-1B) achieved 54.17% accuracy. While it slightly outperformed humans, it was significantly less accurate than ZSL. However, K-means showed superior computational efficiency, allowing for much higher frame counts (256 frames) compared to the token-limited ZSL method.

5. Significance

This work demonstrates that MLLMs possess the latent capability to internalize complex ethological constructs like social hierarchy. By proving that zero-shot reasoning can outperform both human intuition and unsupervised clustering, the paper opens a new frontier in AI-driven ethology. It suggests that future biological research can move away from designing specialized models for every new behavior and instead leverage the reasoning power of foundation models to interpret complex social dynamics in the animal kingdom.