Technical Summary: CapFrame – Text-Instructed Viewpoint Grounding in 3D Gaussian Scenes

1. Problem Definition

While 3D Gaussian Splatting (3DGS) enables photorealistic, real-time novel view synthesis, the process of placing a virtual camera to capture a specific desired frame remains largely manual. Existing language-guided 3D methods primarily focus on object-centric grounding (determining what to observe) but fail to address compositional constraints (determining how the scene should appear). Real-world instructions often specify complex photographic attributes, such as subject orientation (e.g., "left view"), spatial layout (e.g., "on the right side"), and camera tilt, which current trajectory generation or object localization methods cannot enforce.

To address this, the authors formalize a new task: Text-Instructed Viewpoint Grounding (TIVG). The goal is to identify a 6-DoF camera pose ($T_{CW} \in SE(3)$) within a 3D Gaussian scene such that the rendered frame aligns with a natural language instruction describing subject presence, composition, and viewpoint.

2. Methodology: CapFrame

The authors propose CapFrame, a partially differentiable framework that converts language instructions into geometric pseudo-labels for camera pose optimization. The framework operates via a three-stage Retrieve–Translate–Refine pipeline:

Stage 1: Retrieve (Semantic-aware Initialization)

Since exhaustively searching the continuous $SE(3)$ space is computationally prohibitive, CapFrame initializes the optimization using a discrete subspace of training images.

• Coarse Filtering: The system uses FG-CLIP to perform global semantic alignment between the text instruction and training views, retaining the top-$M$ candidates.
• Question-Evaluation (QE): To overcome the limitations of holistic embeddings in cluttered scenes, a Multimodal Large Language Model (MLLM) decomposes the instruction into specific questions (e.g., "Is the subject on the right?", "Is the left view visible?"). The MLLM evaluates each candidate view against these questions to generate granular scores, selecting the top-$K$ views as high-quality initialization points.

Stage 2: Translate (Language to Geometric Pseudo Labels)

Natural language rarely provides numeric optimization targets. CapFrame translates compositional instructions into structured geometric pseudo-labels:

• Subject-Gaussian Association: The MLLM identifies key subjects. Using Grounded SAM on the retrieved images and 3DGS depth, the system back-projects 2D masks to 3D space to associate specific subsets of Gaussians with each subject.
• Orientation Pseudo Labels: For asymmetric subjects, the MLLM infers relative orientation offsets (azimuth, elevation, roll) from the text. These are combined with estimated subject orientations (via Orient Anything) to create target orientation vectors.
• Layout Pseudo Labels: The MLLM infers target 2D bounding boxes for subjects within the normalized image frame. These boxes are adjusted for camera roll to ensure consistency with the orientation constraints.

Stage 3: Refine (Gradient-Based Optimization)

Starting from the retrieved pose, CapFrame performs differentiable optimization in 3DGS to find the optimal camera pose.

• Differentiable Rendering: The system exploits the differentiability of 3DGS rasterization to backpropagate losses directly to the camera pose parameters ($\tau \in \mathfrak{se}(3)$).
• Loss Function: The optimization minimizes a multi-objective loss $\mathcal{L} = \mathcal{L}_{\text{layout}} + \mathcal{L}_{\text{orient}}$:
  • Layout Loss: Enforces subject placement by penalizing deviations of the subject's soft centroid from the target layout box, as well as penalizing mask leakage outside the box and lack of coverage inside it.
  • Orientation Loss: Enforces viewpoint constraints via three terms: a gravity term (for upright camera), a forward-facing term (aligning camera direction with the subject's target orientation), and a look-at term (keeping the subject centroid in view).

3. Key Contributions

1. New Task Formulation: The paper introduces TIVG, shifting the focus from object localization to full-frame compositional alignment in 3D scenes.
2. CapFrame Framework: A novel pipeline that leverages MLLMs to generate geometric pseudo-labels (orientation and layout) from text, bridging the gap between abstract instructions and differentiable 3D optimization.
3. Question-Evaluation Strategy: A retrieval mechanism that uses MLLMs to decompose instructions into specific questions, providing more robust initialization than standard VLM similarity scoring.
4. Differentiable Optimization: A method to refine camera poses in 3DGS using layout and orientation losses derived from pseudo-labels, avoiding the need for additional training of the 3D representation.

4. Experimental Results

The authors evaluated CapFrame on 38 real-world scenes (from Mip-NeRF 360, Deep Blending, Tanks and Temples, LERF-OVS, and DL3DV-10K) with 135 curated text instructions.

• Baselines: The method was compared against heuristic search methods (Interpolation-based Viewpoint Search, Sampling-based Viewpoint Search) and adapted trajectory generation methods (ChatCam, SplaTraj).
• Quantitative Performance: CapFrame achieved the highest scores across all metrics:
  • VLM Metrics: Highest text-image similarity scores (CLIP and SigLIP2).
  • MLLM Judges: Highest alignment scores (AS) and win rates (WR) when evaluated by GPT-5.4-mini and Gemini-2.5-Flash.
  • User Study: 33 participants, each completing questionnaires with 12 groups, preferred CapFrame's outputs over baselines in 77.7% of cases, with an average rating of 4.50/5.0.
• Ablation Studies:
  • Removing the QE strategy resulted in poor initialization, often missing the target subject.
  • Removing orientation loss caused the camera to fail in achieving the specified view (e.g., wrong angle).
  • Removing layout loss caused subjects to drift from their intended positions.
  • The Refine stage consistently improved alignment scores over the initial retrieved views across all datasets.
• Sensitivity: The system showed robustness to paraphrased text and moderate noise in pseudo-labels, though performance degraded slightly when initialized from lower-ranked (Top-5) images.

5. Significance and Claims

The paper claims that CapFrame enables intuitive, language-driven exploration of complex 3D environments by automating the selection of viewpoints that satisfy compositional constraints. Unlike previous methods that focus on "what" to see, CapFrame addresses "how" to frame the scene.

The authors position this work as a first solution to the TIVG task. They acknowledge limitations, noting that the framework's reliance on MLLMs introduces sensitivity to prompt quality and that the current approach does not involve additional training of the 3DGS model itself. Future work is suggested to explore more robust prompting strategies and prompt optimization. The code is made available to facilitate further research in this direction.