Animal behavioral analysis and neural encoding with transformer-based self-supervised pretraining

This paper introduces BEAST, a scalable self-supervised framework that pretrains vision transformers using masked autoencoding and temporal contrastive learning to enable accurate behavioral analysis, pose estimation, and action segmentation across diverse species with minimal labeled data.

The Gist

Imagine you are trying to understand a complex orchestra by only listening to the conductor's hand movements. You know the music (the brain's activity), but you need to figure out exactly what the conductor is doing (the animal's behavior) to understand how they create that music.

For a long time, scientists studying animal behavior have been stuck in a bottleneck. They have thousands of hours of video footage of mice, fish, and other animals, but to make sense of it, they had to manually label every single frame. It's like trying to learn a new language by having a teacher translate every single word for you. It's slow, expensive, and limits how much we can learn.

This paper introduces BEAST (BEhavioral Analysis via Self-supervised pretraining of Transformers), a new AI tool that acts like a "super-intern" for neuroscientists. Here is how it works, explained simply:

1. The Problem: The "Labeling Bottleneck"

Think of animal behavior videos as a massive library of books written in a language no one speaks yet.

• Old Way: To understand a book, a human had to sit down and translate every sentence (label every frame). This took forever.
• The Limitation: Because labeling was so hard, scientists could only study a tiny fraction of the data they had. They were missing the "big picture."

2. The Solution: BEAST's "Self-Study" Method

BEAST is a type of AI that learns by reading the books without a teacher. It uses a technique called "self-supervised learning."

Imagine a student who is given a stack of blank flashcards with holes punched in them.

• The Masked Autoencoder (The "Fill-in-the-Blanks" Game): The AI looks at a video frame, but someone covers up 75% of it with a black box. The AI has to guess what's under the black box based on the tiny bits it can see. This forces the AI to learn the details of the animal's fur, whiskers, and posture, just like you learn a face by remembering the eyes and nose even if the mouth is covered.
• The Temporal Contrastive Learning (The "Spot the Difference" Game): The AI is shown a frame, then asked to find the frame that comes immediately after it. It learns that a mouse running forward looks slightly different than a mouse standing still. This teaches the AI about movement and time, not just static pictures.

By playing these two games simultaneously on thousands of hours of unlabeled video, BEAST builds a massive internal dictionary of "what animal movement looks like."

3. The Payoff: Three Superpowers

Once BEAST has "studied" the videos, scientists can use it for three specific jobs, and it does them better than the old methods:

A. Predicting Brain Activity (The "Mind Reader")

• The Goal: Scientists want to know: "If the mouse moves its whisker this way, what is happening in its brain?"
• The Old Way: They had to manually track the whisker tip (like drawing a dot on a moving target). If the fur was messy, the dot was wrong, and the brain prediction failed.
• BEAST's Way: BEAST looks at the whole video and understands the context of the movement. It predicts brain activity more accurately because it sees the "whole story," not just a single dot. It's like understanding a movie by watching the whole scene, rather than just tracking one actor's hand.

B. Tracking Body Parts (The "Super Tracker")

• The Goal: Tracking exactly where a mouse's paw, nose, or tail is at every moment.
• The Old Way: Required thousands of labeled examples to teach the AI where the paws are.
• BEAST's Way: Because BEAST already "studied" the videos, it only needs a tiny handful of examples (like 100 frames) to learn the rest. It's like a student who has read a whole textbook needing only one practice problem to solve the rest of the exam. It works even on weird animals like fish or when two mice are fighting (which confuses older AI).

C. Cutting and Pasting Behaviors (The "Editor")

• The Goal: Automatically identifying when a mouse is grooming, sleeping, or fighting.
• The Old Way: Scientists had to manually draw boxes around behaviors or rely on complex pose-tracking first.
• BEAST's Way: BEAST can look at the video and say, "Ah, this 5-second clip is 'grooming,' and this one is 'investigating.'" It skips the middleman entirely. It's like a video editor that automatically cuts a movie into scenes without needing a script.

4. Why This Matters

The most exciting part of BEAST is that it turns unlabeled data (the vast, unused mountains of video labs already have) into gold.

• For Small Labs: You don't need a team of 20 people to label videos anymore. You can just feed the raw video to BEAST, and it does the heavy lifting.
• For Big Science: It allows researchers to study behavior in ways that were previously impossible, revealing hidden connections between what an animal does and what its brain is thinking.

In a nutshell: BEAST is a smart AI that teaches itself how animals move by watching hours of raw video. Once it learns, it helps scientists decode the brain's secrets much faster, cheaper, and more accurately than ever before. It turns the "noise" of raw video into a clear "signal" of understanding.

Technical Summary

1. Problem Statement

Understanding the relationship between brain activity and behavior is a central challenge in neuroscience. While cameras are widely used to capture animal behavior, analyzing these videos presents significant technical hurdles:

• Data Scarcity: Most existing methods (e.g., DeepLabCut, specialized action segmentation tools) rely on extensive, manually labeled datasets, which are time-consuming and expensive to produce.
• Task Fragmentation: Current approaches typically require separate, task-specific models for different downstream tasks (e.g., one model for pose estimation, another for neural encoding, another for action segmentation).
• Underutilized Data: Neuroscience experiments generate vast amounts of unlabeled video data that are currently ignored by standard pipelines.
• Limitations of General Models: General-purpose foundation models (like DINOv2 or VideoMAE) trained on internet data often fail to capture the specific, subtle dynamics of controlled animal behavior experiments (static backgrounds, repetitive movements, specific morphologies).

The authors propose BEAST (BEhavioral Analysis via Self-supervised pretraining of Transformers) to address these gaps by creating a scalable, self-supervised framework that leverages unlabeled video data to build robust backbone models for diverse neuro-behavioral tasks.

2. Methodology

BEAST is a self-supervised pretraining framework based on Vision Transformers (ViT). It combines two complementary learning objectives to capture both static appearance and temporal dynamics.

Core Architecture

• Backbone: Uses a standard ViT-B/16 architecture.
• Input: Raw video frames are split into patches.
• Loss Functions:
  1. Masked Autoencoding (MAE): Randomly masks 75% of the patches and trains the model to reconstruct the missing pixels. This captures rich, low-level frame-level appearance details (crucial for pose estimation).
  2. Temporal Contrastive Learning: Uses an InfoNCE loss to learn temporal dynamics.
     • Anchor Frame: A frame at time $t$.
     • Positive Pair: A frame at $t \pm 1$ (immediate temporal neighbor).
     • Negative Pairs: Frames far apart in time ($t \pm N$) or frames from different videos.
     • Novelty: Unlike prior works (e.g., VIC-MAE) that allow any two frames from the same video to be a positive pair, BEAST restricts positives to immediate neighbors. This forces the model to learn specific behavioral transitions against static backgrounds, which is more effective for long-duration animal recordings.

Training Strategy

• Frame Selection: To avoid redundancy in long videos, BEAST employs a specific frame selection strategy. It filters out low-motion frames and uses K-means clustering on downsampled frames to select diverse "anchor" frames, ensuring the model sees a wide variety of poses while maintaining local temporal context.
• Pretraining: Models are initialized with ImageNet weights (MAE-pretrained) and then further pretrained on experiment-specific unlabeled videos using the combined loss ($L_{MSE} + \lambda L_{InfoNCE}$).
• Efficiency: By processing frames individually (rather than 3D video cubes like VideoMAE), BEAST significantly reduces computational requirements (GFLOPS and memory) while maintaining high performance.

3. Key Contributions

1. Unified Framework: BEAST provides a single backbone model that can be fine-tuned for three distinct, fundamentally different tasks: Neural Encoding, Pose Estimation, and Action Segmentation.
2. Domain-Specific Pretraining: Demonstrates that pretraining on specific experimental unlabeled data significantly outperforms general-purpose foundation models (like DINOv2 or VideoMAE) for neuroscience applications.
3. Novel Contrastive Strategy: Introduces a strict temporal sampling strategy ($t \pm 1$) for contrastive learning that is optimized for the statistical properties of animal behavior videos (long duration, repetitive behaviors, static backgrounds).
4. Elimination of Intermediate Steps: For action segmentation and neural encoding, BEAST allows researchers to skip the labor-intensive pose estimation pipeline entirely, using raw video embeddings directly.

4. Results

The authors evaluated BEAST across multiple species (mice, fish), experimental setups (head-fixed, freely moving, single/multi-animal), and datasets.

A. Neural Encoding (Predicting Brain Activity from Video)

• Task: Predicting neural firing rates or principal components from video.
• Performance: BEAST significantly outperformed all baselines, including:
  • Keypoint-based features (traditional approach).
  • PCA and CEBRA (contrastive learning on raw video).
  • DINOv2 and VideoMAE (general foundation models).
• Key Finding: BEAST features captured richer behavioral information than keypoints, which often miss subtle movements obscured by fur or feathers. Domain-specific pretraining (IN+PT) provided substantial gains over ImageNet-only pretraining.

B. Pose Estimation (Keypoint Localization)

• Task: Tracking anatomical landmarks with limited labeled data (100 frames).
• Performance: BEAST achieved state-of-the-art results, outperforming ResNet-50 (AP-10K), DINOv2, and DeepLabCut.
• Key Finding: The MAE component was critical for pixel-level accuracy. While contrastive loss alone (VIT-C) performed poorly for pose, the combination in BEAST provided complementary benefits, yielding smoother and more accurate trajectories with fewer errors than ResNet baselines.

C. Action Segmentation (Behavior Classification)

• Task: Classifying discrete behaviors (e.g., grooming, attack, mounting) on a per-frame basis.
• Performance: BEAST (using patch embeddings with attention pooling) achieved an F1 score of 0.84 on the CalMS21 dataset, placing it in the top 15 of the Multi-Agent Behavior Challenge.
• Key Finding: BEAST surpassed specialized keypoint-based pipelines (like SimBA and TREBA) and general foundation models. Crucially, it achieved this without requiring a pose estimation step, drastically reducing the labeling burden.

5. Significance and Impact

• Democratization of Analysis: BEAST lowers the barrier to entry for labs with limited computational resources or labeling budgets. By leveraging unlabeled data, labs can create custom, high-performance models without needing thousands of labeled frames.
• Scientific Insight: The framework reveals that raw video embeddings contain more predictive power for neural activity than traditional pose estimates, suggesting that current behavioral metrics may be missing critical neuro-behavioral correlations.
• Efficiency: The frame-based approach is computationally more efficient than native video models (VideoMAE), making it feasible for labs to pretrain models on standard GPU clusters (e.g., 8x A40 GPUs) rather than requiring massive supercomputing resources.
• Future Directions: The authors suggest that BEAST could be extended to foundation models trained across diverse datasets (rather than per-experiment) and adapted for naturalistic settings (e.g., zoos, field studies), though frame sampling strategies would need adjustment for less controlled environments.

In summary, BEAST represents a paradigm shift in behavioral neuroscience analysis, moving from task-specific, label-heavy pipelines to a unified, self-supervised foundation model that maximizes the utility of the vast amounts of unlabeled video data generated by modern neuroscience experiments.