🔬 optics

CSPR-Net: Self-supervised Curved Surface Projection Rectification Network for Geometric Distortion Correction in Non-planar Projections

CSPR-Net is a self-supervised deep learning framework that utilizes dual coordinate-based neural networks and cycle-consistency constraints to automatically correct geometric distortions in non-planar projections without requiring manual calibration or ground-truth data.

Original authors: Kejin Peng, Jia Wei, Xiang Hao

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Kejin Peng, Jia Wei, Xiang Hao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to project a movie onto a bumpy, uneven rock wall or a curved marble statue. Instead of a crisp, flat screen, the movie looks stretched, squashed, and "wavy." This is a classic problem in projection mapping.

Usually, to fix this, engineers have to do one of two things: either spend hours manually measuring every bump with expensive 3D scanners (like taking a medical CT scan of the wall), or use complicated patterns of light to "map" the surface (like trying to wrap a piece of gift wrap perfectly around a jagged rock). Both are a huge headache.

This paper introduces CSPR-Net, a smart "digital brain" that fixes these distortions automatically, without needing any manual measurements or expensive tools.

Here is how it works, explained through three simple analogies:

1. The "Mirror Image" Trick (Self-Supervised Learning)

Imagine you have two people, Projector Pete and Camera Cam, standing on opposite sides of a bumpy mountain. Pete throws a ball (an image) at the mountain, and Cam tries to catch it. Because the mountain is bumpy, the ball bounces in weird directions, and Cam sees a distorted version of what Pete threw.

Instead of telling them exactly where the ball should have landed (which would require a map of the mountain), the researchers teach them a game: "The Loop Challenge."

Pete throws the ball $\rightarrow$ Cam catches the distorted version $\rightarrow$ Cam tries to "reverse" the distortion to send it back to Pete.
If the ball Pete receives at the end is exactly what he originally threw, they win!

By playing this "loop" game millions of times, the system learns the shape of the mountain perfectly just by seeing how the "ball" (the image) gets bent. This is what the paper calls "Cycle-Consistency."

2. The "Elastic Rubber Sheet" (Neural Networks)

Traditional methods try to fix the image using rigid math formulas (like trying to fix a crumpled shirt by using a wooden ruler). It works okay for big curves, but it misses the tiny wrinkles.

CSPR-Net uses Neural Networks, which act more like an intelligent, stretchy rubber sheet. Instead of using a stiff ruler, the system learns exactly how to pull and stretch every tiny pixel of the image so that when it hits the bumpy surface, it "snaps" into a perfect, flat-looking rectangle. It’s like having a digital tailor who can see every microscopic wrinkle in the fabric and adjust the pattern perfectly.

3. The "Edge Detective" (Gradient-Based Loss)

One big problem in the real world is messy lighting—shadows or bright sun can confuse the computer. To solve this, the researchers gave the system a special "detective" tool.

Instead of just looking at the colors (which can change if the light changes), the system looks at the edges (the sharp lines). Even if the light is dim, a sharp edge is still a sharp edge. By focusing on these "structural clues," the system ensures the image stays sharp and doesn't get blurry or "mushy" when it hits the surface.

The Result?

The researchers tested this on real-world curved surfaces (like W-shaped and V-shaped objects).

The Old Way: The image looked "mostly okay" but had visible wobbles and stretches at the edges.
The CSPR-Net Way: The image looks perfectly flat and crisp, as if it were being projected onto a high-quality cinema screen.

In short: CSPR-Net is like an automated, digital "eye" that learns to see the bumps in a surface and pre-distorts an image in the exact opposite way, so that when it hits the bumps, it looks perfectly straight.

Technical Summary: CSPR-Net

Title: CSPR-Net: Self-supervised Curved Surface Projection Rectification Network for Geometric Distortion Correction in Non-planar Projections

1. Problem Statement

When projecting images onto non-planar surfaces (e.g., cylindrical, W-shaped, or arbitrary manifolds), the resulting images suffer from severe geometric warping. Traditional correction methods face a significant trade-off:

Feature-based methods (e.g., structured light/checkerboards) are sensitive to ambient light and require manual intervention if detection fails.
Reconstruction-based methods (e.g., 3D scanning) require expensive, specialized hardware and complex calibration.
Supervised Deep Learning requires massive datasets with pixel-perfect "ground-truth" deformation maps, which are extremely difficult to obtain in physical environments.

The core challenge is to achieve high-precision, automated geometric rectification for complex surfaces without manual labeling or expensive 3D sensors.

2. Methodology

The authors propose CSPR-Net, a self-supervised deep learning framework that treats rectification as a bi-directional mapping problem between the projector and camera spaces.

Dual-Path Neural Architecture: Instead of standard CNNs, the model uses two independent coordinate-based Multi-Layer Perceptrons (MLPs): $Net_{p2c}$ (forward mapping: projector $\to$ camera) and $Net_{c2p}$ (backward mapping: camera $\to$ projector). These networks act as residual learners, predicting local displacement fields ( $\Delta u$ ) rather than absolute coordinates, which stabilizes training.
Differentiable Warping: A Spatial Transformer mechanism is integrated to allow end-to-end optimization, enabling loss signals from the image space to flow back to the MLP parameters via bilinear interpolation.
Self-Supervised Learning Objective: The network is trained using a composite loss function ( $L_{total}$ $L_{t o t a l}$ ) that eliminates the need for ground truth:
- Photometric & Structural Consistency: Uses a combination of $L_1$ loss and a gradient-based loss (using a Sobel operator) to ensure edges and high-frequency details align, even under non-uniform lighting.
- Cycle-Consistency Loss: Enforces that mapping from $A \to B \to A$ returns the original coordinates, ensuring the mapping is bijective (one-to-one).
- Geometric Regularization: A Total Variation (TV) smoothness loss prevents unphysical "pixel folding" or discontinuities.
- Mask Consistency Loss: Aligns the projected content boundaries with the camera's Field-of-View (FOV) to prevent mapping errors in background regions.

3. Key Contributions

Implicit Coordinate Representation: Moves away from rigid parametric models (like polynomials) to a flexible, continuous neural representation.
Fully Self-Supervised Paradigm: Enables autonomous learning directly from raw projection-capture image pairs, removing the need for manual annotation or 3D scanning.
Robustness to Environmental Noise: The introduction of gradient-consistency loss allows the model to ignore photometric discrepancies caused by ambient light and surface reflectance.

4. Results

The performance was validated through high-fidelity ray-tracing simulations and physical experiments using a DLP projector and CMOS camera.

Simulation Results: CSPR-Net significantly outperformed a 3rd-degree polynomial baseline. In the "Pre-warped Domain" (the most critical metric for projection), CSPR-Net achieved a PSNR of 22.73 dB and an SSIM of 0.9282, compared to the baseline's 14.29 dB and 0.7779.
Physical Experiments: The model demonstrated a 20.7% improvement in end-to-end fidelity (SSIM). It successfully rectified images on complex W, V, and M-shaped surfaces.
Generalization: When tested on an unseen scenic landscape image (not used during training), the network successfully compensated for the non-linear distortions of various surface geometries, proving its ability to generalize beyond simple calibration patterns.

5. Significance

CSPR-Net represents a shift from manual/hardware-heavy calibration to data-driven, autonomous rectification. Its ability to handle high-frequency geometric variations on arbitrary manifolds makes it highly valuable for Spatial Augmented Reality (SAR), projection mapping, and immersive displays. By solving the geometric problem self-supervisely, it lowers the barrier to entry for high-fidelity projection in dynamic or complex environments.