MAGE: View-guided Point Cloud Completion with Efficient Modality Alignment and Adaptive Geometry Enhancement

Technical Summary of MAGE: View-guided Point Cloud Completion with Efficient Modality Alignment and Adaptive Geometry Enhancement

Problem Statement
Point cloud completion aims to reconstruct complete 3D shapes from partial, incomplete observations. While deep learning has advanced single-modality completion, real-world sensors often produce data with significant sparsity and missing regions due to occlusion and limited viewpoints. Relying solely on partial point clouds often leads to uncertainty and structurally inconsistent reconstructions. To address this, view-guided point cloud completion (ViPC) utilizes a single-view RGB image to provide complementary visual and structural cues. However, existing ViPC approaches suffer from two primary limitations:

1. Insufficient Modality Alignment: Current methods often implicitly align image and point cloud features via attention or feature transfer without explicit geometric constraints. This lack of geometry-aware consistency leads to cross-modal geometric inconsistencies, such as missing structural parts (e.g., tail wings) or broken components.
2. Limited Self-Geometry Enhancement: Existing methods struggle to balance the preservation of local geometric details with global structural coherence. Local graph-based methods have limited receptive fields and accumulate noise, while conventional global attention tends to oversmooth features, losing high-frequency details critical for thin structures and intricate patterns.

Methodology
The authors propose MAGE (Modality Alignment and adaptive Geometry Enhancement), a unified geometry-aware framework designed to address cross-modal geometric inconsistency and enhance both global and local shape recovery. The architecture consists of five key components:

1. Tokenization and Initial Alignment:

   • The input image is divided into non-overlapping patches, and the partial point cloud is downsampled using Farthest Point Sampling (FPS) to extract representative anchor points.
   • Both modalities are fed into a shared Vision Transformer (ViT) to initially align their features in a shared latent space.

2. Efficient Modality Alignment:

   • To enforce explicit geometric consistency, the framework employs cross-modality reconstruction supervision. The aligned image features are decoded into a reconstructed point cloud using a decoder identical to the main point cloud decoder.
   • This reconstruction is supervised by the ground-truth point cloud using an L2 Chamfer Distance loss. This process explicitly forces the image and point cloud features to converge in the latent space, ensuring they describe the same 3D structure.

3. Anchor Refinement:

   • Initial anchors derived from the partial point cloud only exist in known regions. MAGE introduces a geometry-perceptive anchor refinement module that updates anchor locations.
   • This module concatenates global shape representations, point token features, and original anchor coordinates. A Multi-Layer Perceptron (MLP) predicts refined anchor positions, supervised by the ground truth, allowing the model to infer missing regions more accurately than methods relying solely on global shape priors.

4. Adaptive Geometry Enhancement:

   • After cross-attention with image features, an Adaptive Geometry-Aware Self-Attention (AGSA) module further refines the point cloud features.
   • AGSA combines kNN-based local attention (to capture local geometric relationships) with global self-attention (to capture long-range dependencies).
   • Crucially, it employs a spatially-variant feature fusion mechanism. A mask is generated to distinguish between valid (observed) and missing regions. This mask guides the interpolation between original features and attentive features, mitigating the oversmoothing effect of standard attention and preserving high-frequency details in missing areas.

5. Decoder:

   • A multi-branch upsampling network maps the enhanced features to the final completed point cloud, predicting sub-regions of the missing parts and combining them with downsampled inputs.

Key Contributions

• Efficient Modality Alignment: A strategy combining a shared self-attention Transformer with cross-modality reconstruction supervision to explicitly align image and point cloud features with geometric consistency.
• Adaptive Geometry Enhancement: A novel mechanism integrating kNN-based local attention, global self-attention, and spatially-variant feature fusion to simultaneously refine global structures and local geometric details.
• Geometry-Perceptive Anchor Refinement: A supervised module that updates anchor point locations based on both global shape and local input information, improving the inference of missing regions.

Experimental Results
The authors evaluated MAGE on the ShapeNet-ViPC dataset (both known and unknown categories) and real-world KITTI car scans.

• Quantitative Performance: MAGE achieved state-of-the-art results on all metrics (Chamfer Distance and F-Score) across eight known categories (e.g., Airplane, Chair, Watercraft) and four unknown categories (e.g., Bench, Monitor). On the known categories, it reduced the average Chamfer Distance by 0.117 compared to the previous best (EGIInet) and improved the average F-Score by 0.020.
• Qualitative Performance: Visual comparisons demonstrated that MAGE successfully recovers full global structures and reasonable local details (e.g., thin chair legs, airplane wings, and watercraft sails) where competitors like XMFnet, CSDN, and EGIInet failed to fill regions or produced oversmoothed/structurally inconsistent results.
• Ablation Studies: Removing any of the three core components (Anchor Refinement, Adaptive Geometry Self-Attention, or Cross-Modality Reconstruction) resulted in significant performance degradation, validating the necessity of each module.

Significance and Claims
The paper claims that MAGE addresses the long-standing issue of cross-modal geometric inconsistency in view-guided point cloud completion. By integrating efficient modality alignment and adaptive geometry enhancement, the method bridges the gap between 2D image features and 3D point cloud representations more effectively than prior art. The authors assert that their approach provides a more robust solution for reconstructing complete and plausible 3D shapes from partial observations, particularly in scenarios requiring the recovery of fine-grained geometric details and thin structures.

Limitations
The authors acknowledge that the method may struggle to restore fine-grained details when the reference images are blurry or of low resolution due to viewpoint limitations. In such cases, the model can recover overall contours but may fail to reconstruct complex internal components. The paper suggests that future work could incorporate multi-view images, generative priors (e.g., diffusion models), or large-scale pre-trained backbones to further enhance detail recovery.