Automated Segmentation and Tracking of Group Housed Pigs Using Foundation Models

1. Problem Statement

Precision livestock farming (PLF) requires continuous, automated monitoring of individual animals to ensure health and welfare. However, current computer vision (CV) pipelines for group-housed pigs rely heavily on supervised learning models (SLMs). These traditional approaches suffer from several critical limitations:

• Data Dependency: They require extensive, labor-intensive, pixel-level labeled datasets.
• Scalability Issues: Models must be retrained and fine-tuned for every new farm, lighting condition, or camera setup.
• Generalization: They often fail to generalize to unseen environments or complex scenarios like heavy occlusion and pig stacking.
• Long-term Tracking: Existing solutions struggle to maintain consistent animal identities (ID) over extended periods (multi-hour videos) without identity switches, particularly when pigs are occluded or inactive.

The study aims to address these challenges by developing a fully foundation model (FM)-based, zero-shot pipeline that eliminates the need for task-specific training data while achieving robust detection, segmentation, and long-term tracking.

2. Methodology

The authors propose a modular workflow centered on Grounded-SAM2 (a combination of Grounding-DINO and Segment Anything Model 2), enhanced with farm-specific post-processing logic. The system operates without fine-tuning the core models, relying instead on their pre-trained visual representations.

Core Components

1. Detection (Grounding-DINO):

   • Uses a vision-language foundation model to detect pigs based on the text prompt "pig."
   • Generates bounding boxes without prior training on pig-specific data.
   • Hyperparameters (Box and Text thresholds) were tuned to optimize the F1 score.

2. Segmentation (SAM2):

   • Takes the bounding boxes from Grounding-DINO as prompts to generate high-resolution, pixel-level instance masks.
   • Utilizes a memory attention mechanism to propagate masks across video frames, enabling short-term tracking.

3. Long-Term Tracking Pipeline:
   To extend short-term segmentation into multi-hour monitoring, the authors developed a six-module pipeline:

   • Initializer: Selects a clean "anchor frame" (ideally with 10 visible pigs) to establish initial identities.
   • Tracker: Propagates masks forward and backward within 1-minute clips using SAM2's video predictor.
   • Matcher: Links identities between consecutive clips by matching masks from the end of one clip to the start of the next.
   • Mask Refinement: Applies geometric constraints (pen boundaries) and blob analysis to remove noise, duplicates, and invalid masks.
   • Re-identification (ReID): Uses a lightweight feature extractor (SAM2 vision encoder) combined with centroid proximity and the Hungarian algorithm to resolve ID switches and re-identify lost pigs.
   • Post-Quality Control (Post-QC): Automatically flags frames with errors (e.g., zero-area masks, overlaps) for review or re-processing.

Data & Evaluation

• Dataset: Collected from 12 pens (120 pigs) at Iowa State University. Includes 0.97 TB of video (RGB day, IR night) at 10 fps.
• Subsets:
  • Detection: 1,418 annotated static images.
  • Short-term: 550 one-minute video clips (2.96 million frames).
  • Long-term: A continuous 132-minute video sequence.
• Metrics: Intersection over Union (IoU), Precision, Recall, F1, Region Similarity ($J$), Contour Accuracy ($F$), Multiple Object Tracking Accuracy (MOTA), and Multiple Object Tracking Precision (MOTP).

3. Key Contributions

• First Zero-Shot Long-Term Pipeline: This is the first study to utilize foundation models alone (without supervised components) for long-term, multi-hour video monitoring of group-housed pigs.
• Modular FM Workflow: Demonstrates how to combine a general-purpose FM backbone with lightweight, rule-based post-processing (spatial filtering, bidirectional tracking, ReID) to solve specific agricultural challenges like occlusion and identity drift.
• Label Efficiency: The system requires labels only for post-hoc evaluation, not for training. This drastically reduces the manual annotation burden compared to SLMs.
• Robustness to Occlusion: By utilizing bidirectional tracking and mask refinement, the system recovers from temporary occlusions better than traditional Kalman-filter-based trackers.

4. Results

The system was evaluated across three tasks with the following outcomes:

A. Animal Detection

• Performance: Achieved high precision and recall under daytime RGB conditions.
• Challenges: Performance dropped in night-vision (IR) and heavy occlusion.
  • Day Vision TPR: 85.20%
  • Night Vision TPR: 57.53%
• Insight: Many "false positives" were actually pigs outside the pen or partially visible, which the model correctly detected but human annotators excluded based on strict labeling rules.

B. Short-Term Video Segmentation (1-minute clips)

• Accuracy: Over 80% of active tracks were fully correct after post-processing.
• Error Reduction:
  • Duplicated Labels: Eliminated via mask-level Non-Maximum Suppression (NMS).
  • ID Switches/Lost Tracks: Significantly reduced by switching from SAM2-Tiny to SAM2-Large and using bidirectional tracking from valid anchor frames.
  • Incorrect Masks: Mitigated by the long-term pipeline's ability to carry forward correct masks from adjacent clips.

C. Long-Term Video Segmentation (132-minute continuous video)

• Segmentation Quality:
  • Region Similarity ($J$): 0.83
  • Contour Accuracy ($F$): 0.92
  • Combined $J\&F$: 0.87
• Tracking Performance:
  • MOTA: 0.99 (indicating near-perfect tracking with minimal false positives/negatives).
  • MOTP: 90.7%
  • Identity Switches: 0 observed in the evaluated frames.
• Comparison: These metrics are comparable to or exceed state-of-the-art supervised methods, despite using zero-shot inference.

5. Significance and Future Directions

• Paradigm Shift: The study validates a shift from "training-intensive" supervised models to "inference-efficient" foundation models for PLF. This enables scalable deployment across diverse farms without retraining.
• Practical Impact: The system provides a viable path toward autonomous precision livestock farming, capable of monitoring animals for hours without human intervention.
• Limitations & Future Work:
  • Occlusion: While improved, fully hidden pigs (deep stacking) remain difficult to track.
  • Night Vision: IR footage lacks texture, reducing segmentation quality; future work may involve pre-processing or depth-oriented transformations.
  • Computational Cost: SAM2 is resource-heavy; future iterations should explore lightweight variants (e.g., MobileSAM) or knowledge distillation for edge deployment.
  • Domain Generalization: Validation across diverse farm environments (different pen designs, densities) is required to confirm external validity.

In conclusion, this paper demonstrates that foundation models, when augmented with smart modular logic, can replace traditional supervised pipelines, offering a scalable, label-efficient, and highly accurate solution for long-term animal monitoring.