Technical Summary: Scene and Human in One World: Reconstruction in a Feedforward Pass

Problem Statement

Reconstructing humans within dynamic scenes from moving monocular cameras remains a significant challenge due to scale ambiguity, misalignment between human and scene geometry, and occlusion interference. Existing pipelines often treat human mesh recovery and scene reconstruction as separate tasks, aligning them only post-hoc using fragile localization cues (e.g., heuristic body anchors or detector-based estimates). This decoupled approach frequently results in reconstructions that are individually plausible but physically inconsistent when combined, particularly regarding metric scale, global position, and physical interaction. While recent feed-forward methods have begun to unify these tasks, their coupling is often limited; geometry models act as general-purpose 3D predictors without explicit human-awareness, and human models provide only post-alignment cues rather than active constraints on scene geometry.

Methodology: SHOW

The authors propose SHOW (Scene and Human in One World), a mask-promptable, feed-forward framework that couples 3D scene reconstruction and Human Mesh Recovery (HMR) within a unified metric space. The method operates in a single pass to predict dense scene point maps and parametric human models (SMPL-X) simultaneously.

Core Architecture

The pipeline is built upon adapting the visual geometry foundation model VGGT and consists of three main stages:

1. Human-Aware Geometry Encoding:

   • Mask Prompting: Instead of relying on sparse heuristics like head detection, SHOW utilizes a human foreground mask (obtained from off-the-shelf segmentation models) as a dense 2D prompt. A lightweight mask encoder processes this mask, and its features are fused with the latent geometry features from the VGGT backbone via an adaptation module.
   • DensePose Auxiliary Supervision: To ensure the latent features explicitly encode human body information rather than just generic foreground, the model is supervised with an auxiliary DensePose prediction task during training. This forces the geometry encoder to learn features that are predictive of human body regions, providing stronger cues for scale and placement.

2. Geometry-Aware SMPL-X Decoding:

   • A Transformer-based decoder fuses two feature streams: human-centric features (extracted from PromptHMR) and the adapted scene geometry features.
   • The decoder utilizes three types of query tokens: SMPL tokens (for pose, shape, and translation), Scale tokens (for point map scaling), and Register tokens (for information exchange).
   • An asymmetric cross-attention mechanism is employed: SMPL and register tokens attend to both human and geometry features, while scale tokens attend only to geometry features. This design encourages pose/shape estimation to rely on human priors while grounding scale in the reconstructed scene.
   • A ray encoder injects camera intrinsic information into the human branch to ensure spatial consistency.

3. Joint Training for Alignment:

   • The model undergoes a final joint fine-tuning stage to enforce consistency between the predicted human mesh and the scene point map.
   • Scale and Placement Consistency: The model predicts a scale factor to map the normalized point map to the metric scale of the human body. Temporal regularization is applied to ensure scale stability across video frames.
   • SMPL-Guided Point-Map Alignment: A Chamfer distance loss is computed between the rescaled point map in human regions and the visible surface of the predicted SMPL-X mesh. This pulls the scene geometry toward the human surface, ensuring physical plausibility.

Key Contributions

• Unified Feed-Forward Framework: SHOW introduces a single-pass model that jointly estimates dense scene geometry and SMPL-X human meshes in a shared metric coordinate system, eliminating the cascading errors typical of multi-stage alignment pipelines.
• Bidirectional Coupling: The method explicitly injects human semantic and metric-scale priors into scene reconstruction while using scene geometry to constrain human mesh recovery. This enables the joint learning of human-aware geometric features and geometry-constrained human features.
• Mask-as-Prompt Mechanism: The authors propose using human segmentation masks as dense 2D prompts for visual geometry foundation models. This facilitates target-aware feature extraction, improves surface localization, and enhances robustness against clutter and multi-person interference compared to bounding box or head-detection prompts.
• Evaluation Protocol: The paper provides a comprehensive evaluation protocol for human-scene alignment, introducing new metrics (Human-Scene Chamfer and Human-Scene Variance) to benchmark existing feed-forward methods.

Experimental Results

The authors evaluated SHOW on standard datasets including 3DPW, EMDB (subsets 1 and 2), and RICH, using the synthetic BEDLAM2.0 dataset for training.

• Local Mesh Recovery: On 3DPW and EMDB-1, SHOW achieves competitive performance in local pose and shape accuracy (PA-MPJPE, MPJPE, PVE), outperforming other joint human-scene feed-forward approaches like Human3R and UniSH. The authors attribute this to the robustness of mask-based prompting, which handles occlusion better than head-detection or bounding-box methods.
• Global Motion Estimation: On EMDB-2 (moving camera), SHOW demonstrates clear advantages in global trajectory accuracy (WA-MPJPE, RTE), outperforming methods that rely on off-the-shelf scale estimators. Performance on the static-camera RICH dataset is slightly lower, which the authors attribute to the visual geometry models' reliance on multi-view observations.
• Human-Scene Alignment: SHOW achieves the best performance on human-scene consistency metrics (HS-CF and HS-V) among methods that do not explicitly predict metric point maps using additional real-world data. The ablation studies confirm that the mask prompting, ray encoder, and joint optimization are critical for reducing scale ambiguity and achieving physical alignment.
• Qualitative Results: Visual comparisons show that SHOW produces physically plausible interactions (e.g., feet resting on the ground, no penetration into furniture), whereas baseline methods often exhibit significant misalignment, such as body penetration into scene geometry.

Significance and Claims

The paper claims that SHOW represents a shift from treating human and scene reconstruction as separate, loosely coupled tasks to formulating them as an explicitly coupled problem where the two domains mutually inform each other. By integrating human semantics directly into the geometry foundation model and using scene geometry to constrain human recovery, SHOW achieves metric-scale consistency and improved alignment in a single feed-forward pass.

The authors emphasize that this approach reduces reliance on fragile heuristic localization cues and provides a simple yet effective framework for unified 4D perception. While the method shows strong results, the authors modestly acknowledge a sim-to-real generalization gap, noting that the model still struggles with extreme out-of-distribution cases, such as scenes with extreme scale variations or humans occupying very small portions of the image.