SplAttN: Bridging 2D and 3D with Gaussian Soft Splatting and Attention for Point Cloud Completion

Technical Summary: SplAttN

Problem Statement
While multi-modal learning has advanced point cloud completion by leveraging 2D images as semantic priors, the theoretical mechanisms underpinning this success remain unclear. The authors identify a critical failure mode in existing state-of-the-art methods: Cross-Modal Entropy Collapse. Standard approaches utilize deterministic "hard projection" to map sparse 3D point clouds onto 2D image planes. This process results in an extremely sparse support on the image plane, creating a high divergence between projected features and the true latent distribution required by visual encoders. Consequently, this sparsity severs the gradient flow between modalities, hindering the model's ability to learn a robust cross-modal connection and causing the system to degenerate into a unimodal template retriever insensitive to visual input.

Methodology: SplAttN
To address this limitation, the paper proposes SplAttN, a framework that bridges the 2D-3D gap by replacing hard projection with Differentiable Gaussian Splatting. The methodology is grounded in Multimodal Learning Theory, specifically addressing the need for a learnable "Connection" between heterogeneous modalities.

1. Theoretical Reformulation: Instead of modeling projection as a discrete mapping (Dirac delta functions), SplAttN reformulates it as a continuous probabilistic density estimation using a Gaussian kernel. This transforms the sparse point cloud into a dense, continuous feature manifold on the image plane. Theoretically, this ensures non-vanishing gradients and strictly expands the valid information support, preventing entropy collapse.
2. Gaussian Splatting Bridge (GS-Bridge): This is the core differentiable module. It employs a hybrid geometric tokenizer (combining EdgeConv for local curvature and Graph Transformers for global topology) to generate geometric tokens. These tokens act as Queries in an attention mechanism, while the visual features derived from the Gaussian Splatting process serve as Keys and Values. This "Active Query" mechanism allows the geometric stream to dynamically retrieve and align with dense visual priors, functioning as a differentiable dictionary lookup.
3. Hybrid Global-Local Encoder: The architecture satisfies both local isometry (via EdgeConv) and global homeomorphism (via Transformers) to approximate the underlying 3D manifold accurately.
4. Global-Local Decoder: A hierarchical decoder reconstructs the point cloud from a coarse skeleton to fine details. It integrates structure-aware attention to model geometric consistency and cross-attention to inject local features for fine-grained refinement, utilizing a weighted Arc-Chamfer Distance loss to handle outliers.

Key Contributions

• Identification of Cross-Modal Entropy Collapse: The authors pinpoint the sparsity induced by hard projection as the bottleneck restricting the learnable connection in multi-modal point cloud completion.
• SplAttN Framework: The proposal of a framework utilizing Differentiable Gaussian Splatting to maximize Point-wise Mutual Information. The authors theoretically prove this acts as a continuous density estimator, bridging the modality gap and enabling active alignment.
• Counter-Factual Evaluation on KITTI: The paper introduces a rigorous stress test using the real-world KITTI benchmark. By employing a Semantic Consistency Score (SCS) and severing the visual branch during inference, the authors demonstrate that baseline methods (like SVDFormer and GeoFormer) effectively degenerate into unimodal template retrievers with negligible sensitivity to visual removal. In contrast, SplAttN shows a significant performance drop when visual cues are removed, empirically validating a genuine cross-modal dependency.
• State-of-the-Art Performance: The method achieves superior results on standard benchmarks (PCN, ShapeNet-55/34) and demonstrates robust generalization on unseen categories in ShapeNet-34/21.

Results

• Quantitative Performance: On the PCN dataset, SplAttN achieves a state-of-the-art average Chamfer Distance (CD) of 6.36, outperforming previous methods like GeoFormer (6.42) and SVDFormer (6.54). On ShapeNet-55, it achieves the highest F1-Score (0.520) and lowest average CD (0.77).
• Generalization: The model demonstrates superior performance on both seen and unseen categories in ShapeNet-34/21, indicating effective learning of underlying manifold structures rather than overfitting to specific training distributions.
• KITTI Stress Test: In the counter-factual evaluation on KITTI, removing visual input caused SplAttN's consistency score to drop by 26.1%, whereas baselines showed negligible fluctuations or even anomalous gains. This confirms that SplAttN relies on active cross-modal alignment rather than memorized priors.
• Information Throughput: The paper reports a Cross-Modal Information Throughput (CMIT) of 200.5 for SplAttN on KITTI, orders of magnitude higher than baselines, validating the effectiveness of the dense density estimation.

Significance and Claims
The paper claims that resolving Cross-Modal Entropy Collapse is fundamental to bridging the gradient gap in sparse geometric completion. By reformulating projection as continuous density estimation, SplAttN establishes a bona fide cross-modal connection rather than a decoupled generation process. The authors argue that their method moves beyond heuristic fusion to provide a theoretically grounded mechanism that maximizes information flow between 2D and 3D modalities. The work suggests that future progress in multi-modal 3D vision depends on maintaining differentiable, dense connections between modalities to prevent models from reverting to unimodal template retrieval.