3D Multi-View Stylization with Pose-Free Correspondences Matching for Robust 3D Geometry Preservation

1. Problem Statement

The thesis addresses the challenge of artistic style transfer on multi-view captures of a 3D scene. While style transfer is well-established for single images and videos, applying it to unordered multi-view datasets (e.g., for 3D reconstruction, SLAM, or AR/VR) introduces critical failures:

• View-Dependent Artifacts: Independent per-view stylization causes "texture swimming," edge bending, and inconsistent shading across viewpoints.
• Geometric Degradation: These artifacts break the cross-view correspondences (keypoints and descriptors) required by Structure-from-Motion (SfM) and Simultaneous Localization and Mapping (SLAM) systems.
• Depth Instability: Stylization alters local contrast and shading, confusing monocular depth priors and leading to inconsistent 3D reconstructions.
• Dependency on Poses: Existing 3D-aware methods often require accurate camera poses and explicit 3D reconstructions (NeRF, Gaussian Splatting) as prerequisites, making them brittle in scenarios with sparse views or unknown camera parameters.

Goal: Develop a pose-free multi-view stylization framework that produces visually stylized outputs while explicitly preserving the geometric cues (correspondences and depth) necessary for downstream 3D tasks.

2. Methodology

The proposed approach trains a lightweight, feed-forward neural stylizer per-scene using a composite objective function. The system does not require camera poses or an explicit 3D representation during training.

A. Network Architecture

The stylizer $g_\theta$ maps an input view $I_i$ and a style reference $S$ to a stylized output $\hat{I}_i$. Two architectures are evaluated:

1. Residual CNN: A compact network of residual blocks, acting as a baseline that encourages near-identity mapping to preserve coarse geometry.
2. Conditional U-Net: An encoder-decoder architecture with skip connections to better preserve high-frequency details (edges, corners). In the multi-style setting, it uses Conditional Instance Normalization (CIN) to modulate features based on a style ID.

B. Loss Functions

The training objective combines perceptual style transfer with novel geometric constraints:

1. Perceptual Losses (Style & Content):

   • Content Loss ($L_{content}$): Uses a frozen VGG-19 encoder to match deep features between input and output, preserving semantic layout.
   • Style Loss ($L_{style}$): An AdaIN-inspired loss that matches the channel-wise mean and variance of feature statistics between the output and the reference style image.

2. Correspondence-Based Geometry Loss ($L_{SG}$):

   • Mechanism: Utilizes SuperPoint (for keypoint detection) and SuperGlue (for matching) to enforce cross-view consistency.
   • Process: A "stabilized" anchor view is stylized. SuperGlue matches keypoints between this stylized anchor and the original (unstylized) cached views of the scene.
   • Constraint: The loss penalizes the drift in descriptors between matched keypoints. If the stylizer distorts the local structure (e.g., bends an edge), the descriptor match fails, and the loss increases. This forces the network to preserve repeatable local structures without needing camera poses.

3. Depth Preservation Loss ($L_{Depth}$):

   • Mechanism: Uses a frozen MiDaS/DPT monocular depth estimator.
   • Process: Compares the normalized depth map of the stylized output against a reference depth map derived from the original image.
   • Domain Alignment: To prevent the depth model from failing due to the stylized color distribution (Out-of-Domain shift), a global color transfer step aligns the input dataset's statistics to the style image before depth inference.
   • Constraint: Penalizes deviations in predicted depth structure, ensuring global volumetric consistency.

C. Optimization Strategy

• Warmup + Ramp Scheduling: The geometric losses ($L_{SG}$ and $L_{Depth}$) are initially set to zero. They are gradually ramped up over the first 600 iterations. This prevents the network from collapsing into a "geometry-only" solution (ignoring style) or failing to converge due to unstable correspondences in early training.
• Multi-Style Extension: The framework is extended to handle multiple styles in a single model using CIN, allowing the network to learn a shared geometry-preserving representation while modulating appearance via style codes.

3. Key Contributions

1. Pose-Free Geometric Regularization: Introduction of a SuperPoint/SuperGlue-based loss that enforces descriptor consistency across views without requiring camera intrinsics/extrinsics or explicit 3D reconstruction.
2. Depth-Aware Stylization: A depth-preservation loss with a practical domain-alignment step to ensure monocular depth priors remain valid after stylization.
3. SLAM-Based Evaluation Protocol: A novel evaluation metric that runs DROID-SLAM on stylized sequences to quantitatively measure trajectory error (ATE/RTE) and 3D structural consistency (Chamfer Distance), moving beyond standard image-level metrics.
4. Conditional Multi-Style Generalization: Demonstration that a single conditional U-Net can generalize to unseen scenes and multiple styles while maintaining geometric stability.

4. Results and Evaluation

The method was evaluated on the Tanks and Temples and Mip-NeRF 360 datasets, comparing against the state-of-the-art MuVieCAST baseline.

• Quantitative Metrics:

  • SLAM Consistency: The proposed method significantly outperformed MuVieCAST in 3D consistency.
    • ATE (Absolute Trajectory Error): Reduced by up to 93% (e.g., in the Francis scene).
    • Chamfer Distance (3D Geometry): Reduced by 47–91% across various scenes, indicating that the reconstructed point clouds from stylized images were much closer to the original geometry.
  • Static Metrics: While the method sometimes showed a slight trade-off in pure color histogram distance (CHD) compared to baselines, it consistently achieved superior DINO Structure Distance (DSD), proving better preservation of edges and structural layout.

• Qualitative Analysis:

  • Visual results showed that the proposed method eliminated "texture swimming" and edge bending common in per-view baselines.
  • The Residual CNN variant offered the most conservative, geometry-stable results, while the U-Net offered richer stylistic expression with slightly higher (but still controlled) geometric risk.
  • Ablation Studies: Confirmed that both $L_{SG}$ and $L_{Depth}$ are necessary. $L_{Depth}$ alone could lead to "garbage" color regimes (degenerate solutions), while $L_{SG}$ alone improved local structure. Together, they provided the most robust 3D consistency.

• Real-World Domain Translation: The method successfully transferred real-world weather/seasonal styles (e.g., Rainy, Snow, Autumn) to scenes while preserving architectural geometry, demonstrating utility beyond artistic painting.

5. Significance

This thesis makes a significant contribution to the field of 3D computer vision and generative AI by bridging the gap between artistic style transfer and geometric usability.

• Practical Impact: It enables the creation of stylized 3D datasets that remain usable for robotics, AR/VR, and autonomous navigation, where standard style transfer would break tracking and reconstruction pipelines.
• Methodological Shift: It moves away from heavy, pose-dependent 3D optimization (like NeRF/3DGS stylization) toward a lightweight, image-space approach that implicitly learns 3D consistency through correspondence and depth constraints.
• Evaluation Standard: It establishes a rigorous protocol for evaluating stylization not just by "how it looks," but by "how well it works" for downstream 3D tasks, using SLAM trajectory error as a primary metric.

In summary, the work proves that by explicitly regularizing for correspondence stability and depth consistency, it is possible to generate artistic multi-view imagery that does not sacrifice the geometric integrity required for robust 3D reconstruction.