Technical Summary: Pose Splatter

Problem Statement

Accurate and scalable quantification of animal pose and appearance is essential for neuroscience research, particularly for studying subtle behavioral changes associated with neurological diseases. Current 3D pose estimation techniques face significant limitations:

• Keypoint-based methods rely on sparse anatomical landmarks, requiring labor-intensive manual annotation and failing to capture the full body geometry, color, or texture.
• Mesh-based methods offer complete surface reconstruction but typically require per-frame optimization, specialized template models (e.g., SMAL), and often fail to generalize to unseen postures or different species.
• Existing 3D Gaussian Splatting (3DGS) approaches generally rely on per-scene optimization or assume dense multi-view overlap, making them unsuitable for sparse-view animal reconstruction where inter-view overlap is minimal.

These bottlenecks hinder large-scale, longitudinal behavioral analysis needed to map genotype, neural activity, and behavior at high resolutions.

Methodology: Pose Splatter

Pose Splatter is a feed-forward framework that leverages shape carving and 3D Gaussian Splatting (3DGS) to reconstruct the complete 3D pose and appearance of laboratory animals without manual annotations, per-frame optimization, or species-specific templates.

Core Pipeline

1. Mask Generation: The system utilizes a pre-trained Segment Anything Model (SAM2) to generate foreground masks from multi-view video frames, eliminating the need for manual labeling.
2. Pose and Rotation Estimation: To ensure rotation invariance, the system estimates the animal's 3D center via robust triangulation of mask centroids. The azimuthal rotation is derived from the principal axis of a 3D Gaussian distribution fitted to the animal's mass, tracked smoothly over time.
3. Shape Carving: Multi-view masks are back-projected into a shared 3D voxel grid to create a coarse "visual hull." Voxels outside the intersection of visual cones are removed. Colors are assigned to remaining voxels based on visibility and weighted averaging to handle occlusions.
4. Refinement and De-voxelization:
   • A stacked 3D U-Net architecture refines the coarse voxel volume, producing an occupancy channel and additional feature channels.
   • A compact Multi-Layer Perceptron (MLP) processes voxel features to predict 3D displacement vectors, covariance matrices, and appearance features.
   • Crucially, the mean position of the Gaussian is calculated by adding the predicted displacement to the voxel coordinate, effectively "de-voxelizing" the representation to capture fine-grained geometry.
5. Rendering: The resulting 3D Gaussian parameters are rendered using standard Gaussian splatting to produce high-fidelity images from novel viewpoints.

Visual Embedding Technique

To facilitate downstream behavioral analysis, the authors propose a rotation-invariant visual embedding:

• Rendering: The 3D Gaussian scene is rendered from 32 virtual cameras distributed on a sphere surrounding the animal.
• Encoding: Each 224×224 rendering is passed through a pretrained ResNet-18 to extract 512-dimensional feature vectors.
• Spherical Harmonic Expansion: Features are expanded using a truncated spherical harmonic basis. To achieve invariance to horizontal (azimuthal) rotation, the squared magnitudes of the harmonic coefficients are used, eliminating phase dependence.
• Adversarial PCA: To remove residual encoding of the animal's specific azimuthal angle (potentially caused by uneven lighting), an adversarial PCA is applied to find a 50-dimensional subspace that maximizes pose variance while minimizing the ability to predict the sine and cosine of the rotation angle.

Key Contributions

• Annotation-Free 3D Reconstruction: Pose Splatter recovers complete 3D animal geometry and appearance without manual landmark annotation, addressing the scalability issues of keypoint-based methods.
• Template-Free and Optimization-Free: Unlike mesh-based approaches, the model performs inference via a single forward pass and requires no species-specific templates or per-frame optimization.
• Sparse-View Robustness: The framework is specifically designed for sparse-view settings (4–6 cameras) with minimal inter-view overlap, a scenario where existing feed-forward 3DGS methods typically fail.
• Efficiency: The model operates with a low memory footprint (approx. 2.5 GB VRAM) and achieves inference speeds of ~30 ms per frame, significantly faster than optimization-based baselines.
• Rotation-Invariant Embedding: The proposed visual embedding provides a compact, low-dimensional descriptor of pose and appearance that is invariant to animal position and horizontal rotation, suitable for behavioral analysis.

Experimental Results

The authors evaluated Pose Splatter on datasets of mice, rats, and zebra finches.

• Rendering Quality: Pose Splatter outperformed state-of-the-art sparse-view 3DGS baselines (including 3DGS, FSGS, and GaussianObject) and feed-forward methods (PixelSplat, MVSplat) on metrics such as Intersection over Union (IoU), L1 color error, PSNR, and SSIM. Notably, while optimization-based methods like GaussianObject achieved comparable metrics, they required roughly one hour of optimization per scene, whereas Pose Splatter required only ~30 ms.
• Generalization: The model demonstrated robust cross-species generalization. Models trained on one species (e.g., mice) could reconstruct unseen views of another species (e.g., rats) with only modest performance degradation, suggesting the learned representations capture generalizable animal shape features.
• Behavioral Analysis:
  • Nearest Neighbor Preference: In a study with 102 participants, the visual embedding was preferred over 3D keypoint-based nearest neighbors (54% vs. 50%) for identifying similar poses, despite the keypoints being derived from manually annotated data.
  • Subtle Movement Detection: The visual embedding successfully captured subtle variations in pose, such as feather expansion and head shaking, which were reflected in the principal components of the embedding.
  • Downstream Prediction: The visual embedding outperformed 3D keypoints in predicting manually annotated behaviors (6 out of 8 behaviors) using logistic regression, despite requiring no manual supervision.

Significance and Claims

The paper claims that Pose Splatter enables the analysis of large-scale, longitudinal animal behavior by removing the bottlenecks of manual annotation and computational optimization. By providing accurate 3D geometries and a rotation-invariant visual descriptor, the framework facilitates deeper understanding of the genetic and neural underpinnings of behavior.

The authors acknowledge limitations, noting that the current method is designed for single-subject recordings and may struggle with prolonged occlusions in multi-animal interactions. They also note that while the visual embedding is effective, further work is needed to improve interpretability and direct cross-species comparisons. However, the framework represents a significant step toward scalable, high-resolution behavioral quantification in neuroscience.