Two-View Accumulation as the Primary Training Lever for Hybrid-Capture Gaussian Splatting: A Variance-Decomposition View of When Gradient Surgery Helps

1. Problem Statement

Hybrid-Capture Novel View Synthesis (NVS) involves reconstructing 3D scenes using cameras at vastly different distances (e.g., ground-level street views vs. aerial drone views).

• The Conflict: Standard 3D Gaussian Splatting (3DGS) fails in these settings because it optimizes a single set of Gaussian parameters using a single averaged photometric loss.
  • Near views demand high-frequency details, sharp textures, and precise local geometry.
  • Far views demand spatially averaged appearance and stable global consistency.
• The Failure Mode: When these heterogeneous views are aggregated into a single gradient update, the optimization signal becomes biased toward the dominant regime (often the far views), causing the model to either over-smooth aerial renderings or fail to reconstruct street-level details.
• The Gap: Standard 3DGS trained for 30K iterations under-fits the minority regime by 1–3 dB compared to compute-matched baselines on hybrid-capture benchmarks.

2. Methodology: CrossGrad-GS

The authors propose CrossGrad-GS, a training-side framework that modifies how gradients are aggregated without changing the underlying Gaussian representation, rasterizer, or density control rules.

Core Components:

1. Altitude-Balanced View Pairing:

   • Training cameras are split into Near and Far groups based on the median Euclidean distance from the scene center.
   • At each optimizer step, the method samples one near view and one far view.
   • Gradients are computed separately for both views on the shared Gaussian parameters.

2. Symmetric Gradient Reconciliation (Gradient Surgery):

   • If the near and far gradients are aligned ($g_{near}^T g_{far} \geq 0$), they are summed normally.
   • If they conflict ($g_{near}^T g_{far} < 0$), the method applies a symmetric projection to remove the destructive component of each gradient that opposes the other:
     $$g'_{near} = g_{near} - \frac{g_{near}^T g_{far}}{\|g_{far}\|^2 + \epsilon} g_{far}$$
     $$g'_{far} = g_{far} - \frac{g_{near}^T g_{far}}{\|g_{near}\|^2 + \epsilon} g_{near}$$
   • The final update uses the sum of these reconciled gradients: $\theta \leftarrow \theta - \eta(g'_{near} + g'_{far})$.

3. Variance-Decomposition Framework:

   • The authors derive a theoretical framework decomposing gradient variance into within-regime ($\sigma^2_w$) and between-regime ($\sigma^2_b$) components.
   • Key Insight: They prove that the variance reduction from structured pairing (Near+Far) vs. random pairing depends on the ratio $\sigma^2_b / \sigma^2_w$.
   • Empirical Finding: In hybrid-capture 3DGS, $\sigma^2_b \ll \sigma^2_w$. This implies that the specific pairing rule (structured vs. random) matters less than the structural change of rendering two views per step. The primary benefit comes from halving the variance of the gradient estimator by using two views, rather than the specific geometric projection operator.

3. Key Contributions

1. Diagnostic of Gradient Conflict: The paper quantifies that 33–83% of shared parameter tensors exhibit negative inner products (conflict) between near and far views during training, establishing this as a measurable property of hybrid capture.
2. Identification of the Primary Lever: Through extensive ablation, the authors demonstrate that the structural change of rendering two views per optimizer step accounts for 80–100% of the performance gain.
   • Surprisingly, the specific pairing rule (structured Near/Far, random, or active loss-disparity) yields statistically indistinguishable PSNR results across all benchmarks.
   • This contradicts the intuition that "direction-aware" surgery is the primary driver; instead, the variance reduction from two-view accumulation is dominant.
3. Negative Results & Honest Characterization: The paper reports that many sophisticated gradient surgery techniques (GradNorm, CAGrad, MGDA, confidence gating, active pairing) fall within the seed variance of a simple random two-view control. This delineates the boundary of what actually moves the needle for hybrid-capture 3DGS.
4. Backbone Transfer: The method is shown to be complementary to representation-level improvements, successfully transferring gains to Scaffold-GS and Pixel-GS.

4. Experimental Results

The method was evaluated on five hybrid-capture benchmarks: UC-GS (NYC/SF), MatrixCity, and HorizonGS (Road/Park).

• Performance Gains:
  • CrossGrad-GS (Vanilla backbone) improved PSNR by +1.01 dB (NYC), +2.51 dB (MatrixCity), +3.35 dB (Road), and +0.73 dB (Park) over standard 30K Vanilla 3DGS.
  • Crucially, it outperformed a compute-matched Vanilla 3DGS (60K iterations) on NYC, MatrixCity, and Road, proving the gain is not merely due to seeing more training views but due to better gradient balancing.
• Ablation Insights:
  • Two-View vs. One-View: Moving from 1 view/step to 2 views/step closed the gap to the 60K baseline.
  • Pairing Rules: Structured Near/Far pairing, Random pairing, and Active pairing all performed within seed variance.
  • Grouping Sensitivity: The method relies on the Near/Far split matching the true visual regimes. On UC-GS SF (anisotropic camera layout), the default radial split failed, highlighting that grouping is a critical modeling choice, not just an implementation detail.
• Qualitative Results: CrossGrad-GS successfully recovered high-frequency textures in near views while maintaining global consistency in far views, whereas Vanilla 3DGS collapsed to a blurred average or lost one regime entirely.

5. Significance and Conclusion

• Paradigm Shift: The paper challenges the prevailing focus on complex gradient surgery operators (like CAGrad or PCGrad) for this specific problem. It argues that for hybrid-capture 3DGS, the variance reduction achieved by simply processing two views per step is the dominant factor, rendering complex direction-aware projections largely redundant (within statistical noise).
• Practical Impact: It provides a simple, low-overhead training recipe (render 2 views, sum reconciled gradients) that significantly boosts performance on difficult aerial-ground datasets without requiring new Gaussian representations or heavy architectural changes.
• Theoretical Contribution: The variance-decomposition framework offers a new lens for understanding when gradient surgery is necessary. It suggests that if between-regime variance is low relative to within-regime variance, simple accumulation is sufficient, and complex surgery adds little value.

In summary, the paper identifies that hybrid-capture failure is primarily a training-side gradient aggregation problem best solved by two-view accumulation, with the specific pairing strategy being secondary to the structural change of the training loop.