Technical Summary: CoGS – Compositional Dynamic Human-Object Scenes Gaussian Splatting from Monocular Video

1. Problem Statement

Reconstructing dynamic scenes involving human-object interactions from a single monocular video is a challenging inverse problem. The core difficulty lies in the fact that the scene comprises three distinct components—the articulated human, the manipulated rigid object, and the static background—that obey different motion models yet share the same image pixels.

Existing dynamic radiance field and Gaussian splatting methods often treat the entire scene as a single deforming field. This approach leads to "motion entanglement," where the motion of an object leaks into the human representation or the static background, and vice versa. Furthermore, monocular supervision is inherently underconstrained, particularly for human regions that are rarely observed, occluded, or self-occluded. Without explicit separation, these methods struggle to maintain geometric consistency and photometric fidelity in complex interaction scenarios.

2. Methodology: CoGS Framework

The authors propose CoGS, a compositional Gaussian-splatting framework that decomposes the video into three coordinated branches, each with its own motion model and supervision strategy. The pipeline consists of preprocessing, branch-specific modeling, and a staged optimization curriculum.

2.1. Compositional Decomposition

CoGS separates the scene into:

1. Articulated Human Branch: Initialized from a complete canonical prior (SMPL-based). It uses surface-based skinning weights to deform canonical Gaussians according to estimated body poses.
2. Rigid Object Branch: Models the manipulated object as a separate Gaussian field with its own rigid trajectory. The object's motion is parametrized by a compact set of keyframes (translation and rotation) interpolated via cubic-Hermite splines and spherical linear interpolation (SLERP) to ensure temporal smoothness.
3. Static Scene Branch: Represents the background as a static Gaussian field, initialized from COLMAP sparse geometry. It is optionally regularized by weak planar primitives extracted from the scene to stabilize large, low-texture surfaces.

2.2. Visibility-Aware Anchoring

A critical innovation is the visibility-aware human anchoring mechanism. Since monocular videos often lack observation of certain body parts (e.g., due to occlusion by the object or self-occlusion), CoGS does not rely solely on photometric evidence for these regions.

• For each human Gaussian, a visibility score is calculated based on whether it falls within the human mask, outside the object mask, and contributes non-negligible opacity.
• Gaussians with low visibility scores are "softly tethered" to the canonical human seed via an anchor loss, preventing them from drifting or collapsing.
• Well-observed Gaussians are allowed to move freely to fit the image evidence.

2.3. Six-Stage Optimization Schedule

To prevent competition between branches for the same pixels and to stabilize the optimization, CoGS employs a six-stage curriculum:

1. Human Stabilization (Stages I–II): The human branch is trained independently using the prior seed, RGB/mask supervision, and visibility anchoring. Densification is disabled to preserve the prior structure.
2. Object Stabilization (Stages III–IV): With the human branch frozen, the object branch is optimized using silhouette, pose, and motion constraints. Object densification is delayed and bounded to prevent pose errors from being hidden by uncontrolled Gaussian growth.
3. Compositional Fusion (Stages V–VI): The stabilized human and object branches are fused with a fresh scene field. The full renderer is trained with full-image supervision, foreground consistency, and delayed scene-planar regularization. A final "true-resume" pass refines the appearance and alpha consistency.

3. Key Contributions

The paper identifies four primary contributions:

• Compositional Formulation: A novel Gaussian-splatting approach that explicitly separates the video into articulated human, rigid object, and static scene branches, assigning distinct motion models to each.
• Reliability-Aware Supervision: The introduction of visibility-aware anchoring, which preserves weakly observed human regions by tethering them to a canonical prior while allowing observed regions to adapt to photometric evidence.
• Staged Optimization Curriculum: A six-stage training schedule that stabilizes individual components before fusing them, reducing inter-branch competition and delaying aggressive density changes until components have stable support.
• Empirical Validation: Comprehensive evaluation on HOSNeRF (human-object interaction) and NeuMan (in-the-wild human-scene) benchmarks, demonstrating improvements in both full-frame and human-focused reconstruction quality.

4. Experimental Results

The authors evaluated CoGS against state-of-the-art dynamic scene and human-scene baselines (e.g., K-Planes, 4DGS, NeuMan, HUGS).

• HOSNeRF (Human-Object Interaction): CoGS achieved the best PSNR on all six tested scenes and the best LPIPS on five of six scenes. It showed significant improvements in scenarios involving object occlusion (e.g., Backpack, Suitcase) and fast articulated motion (e.g., Tennis, Dance).
• NeuMan (Human-Scene): In full-frame evaluations, CoGS outperformed all previous methods across all metrics (PSNR, SSIM, LPIPS), with PSNR improvements ranging from 8.46% to 16.46% and LPIPS reductions of 40.91% to 61.54%.
• Human-Only Regions: When evaluating only the dynamic human regions (excluding the static background), CoGS showed even larger gains, with LPIPS reductions of 66.67% to 81.25% on several sequences. This highlights the effectiveness of the visibility-aware prior in reconstructing partially observed body parts.
• Ablation Studies: Removing the human prior seed or object seed resulted in increased LPIPS (+0.026 and +0.029, respectively), confirming that initialization is crucial for handling occlusions. Similarly, removing human alpha/boundary losses or delayed scene-planar regularization degraded performance, indicating that each component of the pipeline addresses specific failure modes.

5. Significance and Claims

The paper claims that CoGS represents a significant step forward in monocular dynamic scene reconstruction by addressing the fundamental issue of motion entanglement. By explicitly separating the scene into components with distinct physical constraints, the method allows each part to be optimized under the most appropriate supervision signals.

The authors emphasize that their approach does not require multi-view data or complex hardware, relying instead on standard monocular inputs and off-the-shelf preprocessing tools (COLMAP, SMPL estimation, SAM). The significance lies in achieving stronger fidelity and perceptual quality in "in-the-wild" scenarios where objects occlude humans and vice versa, a common challenge that single-field methods struggle to resolve.

Limitations acknowledged by the authors:

• Dependence on the quality of monocular preprocessing (e.g., pose estimation, segmentation).
• Inability to fully recover identity or clothing details in regions that are never observed.
• The use of planar primitives provides only weak, rather than complete, scene geometry.
• The assumption that manipulated objects are rigid, which limits applicability to deformable objects.

Future work is suggested to focus on jointly refining preprocessing and reconstruction, supporting deformable objects, and strengthening scene geometry recovery.