💻 computer science

ProjFormer: Point Cloud Completion via Geometric-Projective Transformer and Cross-Modal Semantic Constraints

ProjFormer is a lightweight cross-modal framework for point cloud completion that addresses the ill-posed nature of the task by enforcing geometric consistency through explicit projection and adaptive feature routing to effectively integrate 2D semantic constraints with 3D structural refinement.

Original authors: Sheng Liu, Meng Wang, Ruihui Li, Huilong Pi, Zhuo Tang, Kenli Li

Published 2026-08-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Sheng Liu, Meng Wang, Ruihui Li, Huilong Pi, Zhuo Tang, Kenli Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer vision, machines are constantly learning to see the world in three dimensions. While a standard photograph captures a flat, two-dimensional slice of reality, many modern applications—from self-driving cars navigating busy streets to robots assembling delicate parts—require a full, volumetric understanding of space. To provide this, scientists use point clouds, which are essentially massive collections of individual dots floating in 3D space. Each dot represents a specific location on a surface, and together, they form a digital skeleton of an object. However, the real world is messy. Sensors often miss parts of an object due to shadows, other objects blocking the view, or the sheer distance of the scan. This leaves the computer with a fragmented, incomplete shape, like a puzzle with half the pieces missing. The challenge for researchers is to teach machines to look at these broken fragments and mentally reconstruct the whole object, filling in the gaps with a shape that makes geometric sense.

For years, the most successful attempts to solve this problem relied on two distinct approaches. Some methods tried to guess the missing parts using only the 3D dots they had, essentially asking the computer to imagine the rest based on patterns it had seen before. Others tried to borrow ideas from how humans see the world, using 2D images to guide the 3D reconstruction. The problem with the image-based methods was that they often treated the 3D dots and the 2D pictures as separate things that needed to be glued together. This "gluing" process was often imprecise; the computer might know what a chair looks like from a photo, but it struggled to know exactly which specific dot in the 3D cloud corresponded to which part of that photo. This lack of a direct, physical link between the image and the 3D point often led to reconstructed shapes that looked blurry or had strange, warped artifacts.

A team of researchers at Hunan University has introduced a new approach called ProjFormer, which changes how the computer connects the 2D image to the 3D dots. Instead of trying to learn a vague relationship between the two, their method uses the strict rules of geometry to draw a direct line between them. Imagine trying to match a specific spot on a 3D model to a spot on a photograph; the researchers' system does this by mathematically projecting the 3D dot onto the image, just as a camera lens would project a real object onto film. This ensures that every piece of information the computer pulls from the image is perfectly aligned with the specific 3D point it is trying to repair. By enforcing this strict geometric consistency, the system avoids the guesswork that plagued earlier methods, allowing it to retrieve the exact visual details needed to fill in the missing parts of the shape.

The core of this new system involves a process the researchers call "projective guided view attention." In simpler terms, the computer looks at the incomplete 3D object from several different angles, creating a set of virtual maps. For every single dot in the incomplete cloud, the system calculates exactly where that dot would appear on these virtual maps. It then reaches out to those specific locations on the maps to grab the visual features—like texture or edge information—and brings them back to the 3D dot. This happens with a level of precision that previous methods could not achieve, because the connection is not learned through trial and error but is dictated by the laws of perspective. The system also includes a smart filter that weighs how reliable each piece of information is, paying more attention to views where the object is clearly visible and less attention to views that might be obscured or too far away.

Once the system has gathered these precise visual clues, it faces another challenge: deciding how to use them. Some parts of the object are clearly visible, while others are completely missing. The researchers found that a single, rigid rule for combining information does not work well for the whole object. To solve this, they built a "geometry-aware routing" mechanism. This acts like a dynamic traffic controller for the data. For the parts of the object that are already visible, the system leans heavily on the detailed visual clues it just gathered. But for the parts that are completely missing, it shifts its focus to broader structural patterns, using its knowledge of what the whole object should look like to fill in the void. This ability to switch strategies depending on the local situation allows the system to produce results that are both detailed where they should be and structurally sound where the data is missing.

The results of this new approach are significant. When tested on standard datasets containing thousands of different objects, from airplanes to cars, the new method produced more accurate and complete shapes than many of the leading techniques currently available. On a benchmark dataset known as PCN, the system achieved an average error score of 6.34, a figure that indicates a very close match to the true shape of the objects. Beyond just accuracy, the system is remarkably fast. While other methods that try to combine 2D and 3D data can take over 26 milliseconds to process a single object, this new approach completes the task in about 15.85 milliseconds. This speed is crucial for real-time applications, such as a robot needing to understand its environment instantly to avoid a collision.

The researchers also tested their system on real-world data collected from autonomous vehicles, a scenario where the input is often noisy and incomplete. In these tests, the system generated shapes that were much closer to the realistic forms of cars found in the real world compared to previous methods. While there was a slight trade-off where the system was more aggressive in filling in missing parts, the overall quality of the reconstruction was superior. The study demonstrates that by respecting the fundamental geometry of how light and space interact, rather than relying solely on complex statistical guesses, machines can learn to see the world more clearly. This work suggests that the future of 3D vision lies not just in gathering more data, but in building smarter connections between the different ways we represent the world.

Technical Summary: ProjFormer

Problem Statement

Point cloud completion is an inherently ill-posed task due to the severe sparsity and ambiguity of partial observations. While recent multi-view methods attempt to alleviate this by incorporating 2D semantics, they often rely on learned attention mechanisms and fixed fusion strategies that lack explicit geometric consistency and adaptability. Existing approaches face three primary limitations:

Lack of Localized Correspondence: Even when using coordinate-consistent representations (e.g., Canonical Coordinate Maps), representation-level consistency does not guarantee explicit point-to-pixel correspondences, leading to weakened spatial alignment between observed 3D points and multi-view features.
Computational Inefficiency: Dense attention mechanisms for fine-grained point-to-view associations scale poorly with the number of views and spatial tokens, and their learned weights do not explicitly encode known projective geometry.
Suboptimal Fusion: Heterogeneous features (structural priors, local geometry, and view-conditioned cues) are typically fused via concatenation or learned attention, which fails to explicitly balance structural priors against observation-driven features on a point-wise basis, especially in regions with varying degrees of incompleteness.

Methodology

The authors propose ProjFormer, a cross-modal framework that enforces geometry-consistent 2D–3D interaction through explicit projection and adaptive feature routing. The architecture consists of three core components:

1. Canonical Coordinate Maps (CCMs)

Following GeoFormer, ProjFormer encodes partial 3D geometry into multi-view maps. Points are normalized into a canonical coordinate system and projected onto predefined viewpoints. This process generates CCMs that preserve metric coordinate information while enabling the use of mature 2D encoders.

2. Projective Guided View Attention (PGVA)

To address the lack of localized correspondence and computational inefficiency, ProjFormer introduces PGVA. Unlike standard cross-attention, PGVA uses deterministic projection to align 3D points with multi-view features.

Mechanism: For each observed 3D point, the module projects it onto multiple CCM views to retrieve localized features at the corresponding raster locations.
Geometry-Aware Weighting: It employs a depth-gated attention mechanism. The attention weight is modulated by the consistency between the projected point's canonical coordinate and the aggregated CCM coordinate, as well as a normalized depth prior. This ensures that only geometrically reliable observations contribute significantly to the feature aggregation.
Complexity: This approach reduces the complexity of multi-view feature aggregation from $O(V \cdot N \cdot HW)$ (dense attention) to $O(V \cdot N)$ , where $N$ is the number of observed points and $V$ is the number of views.

3. Geometry-Aware Routing Refinement

To handle the integration of heterogeneous features, the authors propose a progressive refinement module that performs point-wise adaptive routing.

Feature Decomposition: The module disentangles inputs into three pathways:
1. Structural Pathway: Captures global dependencies via self-attention.
2. Local Geometric Pathway: Preserves fine-grained details from the partial input via cross-attention.
3. Image Guidance Pathway: Utilizes the point-aligned image tokens generated by PGVA.
Adaptive Gating: A learnable gate dynamically balances the structural pathway against an observation-conditioned pathway (a combination of local and image features). This allows the model to rely more on structural priors in severely missing regions while emphasizing observation-driven features in well-observed areas.
Progressive Refinement: The fused features are decoded into residual coordinate offsets in a cascaded manner to progressively recover coarse topology and fine-grained geometry.

Key Contributions

The paper claims four primary contributions:

ProjFormer Framework: A geometry-aware framework for point cloud completion that leverages explicit 2D–3D projection to improve localized point-to-view alignment without requiring external RGB images.
PGVA Module: A novel attention mechanism using deterministic projection and depth-gated weighting to achieve point-aligned view aggregation with $O(VN)$ complexity.
Routing Refinement: A Geometry-Aware Routing Refinement module that performs point-wise adaptive routing between prior-driven structural completion and observation-conditioned geometric refinement.
Performance and Efficiency: Extensive experiments demonstrating competitive accuracy with significantly reduced inference latency compared to existing cross-modal methods.

Experimental Results

The authors evaluated ProjFormer on PCN, ShapeNet-55/34, and KITTI datasets.

PCN Dataset: ProjFormer achieved an average Chamfer Distance ( $CD_{L1}$ ) of 6.34, outperforming state-of-the-art methods like GeoFormer (6.42) and SVDFormer (6.55). It also demonstrated a ~40% reduction in inference latency (15.85 ms vs. 26.37 ms for GeoFormer), even accounting for the overhead of rendering 2D images.
ShapeNet-55/34: The method achieved the lowest $CD_{L2}$ scores across seen and unseen categories, including simple, moderate, and hard difficulty levels.
KITTI (Real-world): Using Minimum Matching Distance (MMD) as a metric for realism (since ground truth is unavailable), ProjFormer achieved the lowest MMD (0.48), indicating predictions closer to the manifold of realistic complete car geometries.
Ablation Studies: Removing CCMs, point-aligned tokens, depth gating, or the PGVA module individually resulted in performance degradation, confirming the necessity of each component.

Significance and Claims

The paper positions ProjFormer as a solution that bridges the gap between geometric consistency and computational efficiency in multi-modal point cloud completion. The authors claim that by replacing learned, dense cross-attention with explicit geometric projection and adaptive routing, the method achieves:

Improved Structural Completeness: Fewer artifacts and better topological correctness compared to global fusion methods.
Efficiency: A lightweight design that significantly lowers inference latency while maintaining competitive accuracy.
Interpretability: The use of deterministic projection provides geometric interpretability that learned attention weights often lack.

The work is presented as a step toward more efficient and geometrically grounded 3D reconstruction, particularly in scenarios where external RGB data is unavailable and self-view completion is required.