FERAL: A Supervised Video-Understanding System for Direct Video-to-Behavior Mapping

Technical Summary: FERAL

Problem Statement
The quantitative study of animal behavior has traditionally relied on manual annotation or automated pipelines that first extract keypoints (pose estimation) before classifying actions. While pose-based tools (e.g., DeepLabCut, SLEAP) have revolutionized kinematic analysis, they face significant limitations: they require tightly controlled imaging conditions, struggle with occlusions and complex social interactions, and discard contextual visual information essential for interpreting specific behaviors. Furthermore, existing video-based approaches that map raw pixels directly to behavior (e.g., DeepEthogram) or video foundation models (e.g., DAP, ChimpVLM) have either been limited to specific datasets, lack generalizability across species, or remain as benchmarks rather than deployable, species-agnostic tools. There is a critical need for a system that can map raw video directly to frame-level behavioral labels without relying on intermediate pose extraction, capable of generalizing across diverse species, recording modalities (lab, field, aerial), and levels of organization (single animal to collective behavior).

Methodology
FERAL (Feature Extraction for Recognition of Animal Locomotion) is a supervised video-understanding toolkit designed to bypass pose estimation entirely. It maps raw video frames directly to user-defined behavioral labels (ethograms).

• Architecture: FERAL leverages a pretrained video foundation model, specifically V-JEPA 2 (a ViT-L/16 model pretrained on over one million hours of internet video and fine-tuned on Diving48), as its backbone. Instead of retraining from scratch, FERAL fine-tunes the upper twelve of twenty-four transformer layers.
• Processing Pipeline:
  1. Preprocessing: Raw videos are resized (default 256×256) and re-encoded into seekable formats. User annotations from various tools (e.g., BORIS, EthoVision) are converted into a standardized frame-aligned JSON schema.
  2. Chunking: To handle computational limits, videos are divided into overlapping temporal chunks (64 frames with 50% overlap).
  3. Feature Extraction & Classification: The backbone generates spatiotemporal embeddings for each chunk. These are processed by an attention-based pooling module to aggregate features into temporally aligned embeddings, which are then projected by a lightweight classification head into frame-level class logits.
  4. Ensembling: Predictions from overlapping chunks are averaged to produce stable frame-level probabilities.
• Training Strategy: The system employs class-weighted cross-entropy loss (using the square root of inverse frequency) to handle class imbalance. It utilizes data augmentations (TrivialAugment and MixUp) and supports gradient checkpointing to reduce VRAM requirements.
• Configurations: Three presets are provided to balance speed and accuracy: Lite (faster, smaller backbone), Default (standard performance), and Max (higher overlap and smoothing for peak accuracy).

Key Contributions

1. Direct Video-to-Behavior Mapping: FERAL eliminates the intermediate pose-extraction step, allowing for behavioral segmentation directly from raw pixels, preserving full visual and temporal context.
2. Species and Modality Agnosticism: The toolkit is designed to generalize across a wide range of species (from nematodes and flies to mice, ants, zebras, and apes) and recording environments (controlled labs, field drone footage, and camera traps).
3. Scalability and Accessibility: Released as an open-source package, FERAL integrates with existing pipelines and can run on consumer-grade GPUs or cloud platforms (e.g., Google Colab, RunPod), making high-performance behavioral quantification accessible to researchers with limited computational resources.
4. Handling Collective Behavior: Unlike pose-based systems that struggle with dense groups, FERAL successfully identifies emergent collective states (e.g., ant raids) directly from scene appearance without requiring individual tracking.

Results
FERAL was benchmarked against state-of-the-art pose-based and video-based methods across multiple datasets:

• CalMS21 (Mouse Social Interactions): FERAL achieved 94.2% mAP, outperforming the competition Top-1 entry, Google's VideoPrism, and self-supervised pose-pretraining baselines. Notably, it achieved 92.1% mAP using only 25% of the training data, surpassing models trained on the full dataset.
• MABe22 (Rove Beetle/Ant Interactions): On the beetle-ant subset, FERAL achieved a macro F1 score of 0.92, significantly exceeding the competition's best submission (0.758) which used a different training protocol.
• DeepEthogram Datasets: FERAL matched or exceeded DeepEthogram's frame-level F1 scores on five of six datasets, including Mouse-Social, Mouse-Homecage, and Fly.
• Field Recordings:
  • Zebra Vigilance: On aerial drone footage of Grevy's zebras, FERAL achieved a macro F1 of 0.785, whereas pose-based classifiers trained on the same data collapsed to near-chance performance (~0.55) due to the difficulty of detecting vertical head posture from overhead views.
  • PanAf500 (Wild Apes): FERAL achieved 90.8% Top-1 accuracy and 81.0% class-averaged recall on wild chimpanzee and gorilla behavior, outperforming X3D, I3D, 3D ResNet-50, and primate-specific foundation models (ChimpVLM, DAP, PriVi).
• Collective Dynamics: FERAL successfully detected binary raiding events in ant colonies with 97.8% mAP and resolved finer-grained raid substructures (search, recruitment, response) with 70.0% mAP.

Significance and Claims
The paper claims that FERAL establishes a robust foundation for behavioral segmentation that is scalable, species-agnostic, and capable of operating in complex, naturalistic environments where traditional pose-based pipelines fail. By shifting the paradigm from "where the animal is" (pose) to "what the animal is doing" (behavior) directly from video, FERAL broadens the experimental paradigms available for studying animal behavior.

The authors emphasize that FERAL is a supervised tool; it does not autonomously discover novel behaviors but excels at recognizing predefined categories with high precision. It is presented as complementary to unsupervised discovery tools (like MoSeq or B-SOiD) and pose-tracking systems (like DeepLabCut), offering a unified solution for behavioral quantification. The system's ability to generalize from laboratory settings to field recordings with minimal annotation effort (demonstrated by high performance with as little as 1-5% of training data) positions it as a transformative tool for ethology, neuroscience, and ecology.