GS-ProCams: Gaussian Splatting-based Projector-Camera Systems

The paper introduces GS-ProCams, a novel Gaussian Splatting-based framework that achieves view-agnostic projector-camera systems with superior simulation quality while eliminating the need for additional hardware and drastically reducing computational costs compared to existing NeRF-based methods.

The Gist

Imagine you are trying to paint a picture on a bumpy, crumpled piece of paper using a projector. If you just shine a standard image, it will look distorted, stretched, and blurry because the surface isn't flat. This is the problem Projector-Camera Systems (ProCams) try to solve: they use a camera to "see" the surface and a computer to "fix" the image before projecting it, so it looks perfect to your eye.

However, existing methods for doing this are like old, heavy, single-use cameras. They are slow, require special dark rooms, and if you move your head (change the viewpoint), the whole system breaks and needs to be recalibrated from scratch.

Enter GS-ProCams. Think of this as the smartphone upgrade for projector technology. It's fast, works in normal light, and lets you walk around the object while the projection stays perfect.

Here is a breakdown of how it works using simple analogies:

1. The Old Way vs. The New Way

• The Old Way (NeRF/CNN): Imagine trying to understand a 3D object by taking a million photos and feeding them into a giant, slow brain (a Neural Network). It takes hours to learn the object, needs a lot of electricity (GPU memory), and if you move even an inch, the brain gets confused. It's like trying to learn a dance routine by watching a video at 1% speed.
• The New Way (GS-ProCams): This uses 2D Gaussian Splatting. Imagine the object isn't a solid block, but a cloud of millions of tiny, flat, colorful stickers (Gaussians) floating in space. Each sticker knows its own shape, color, and how shiny it is.
  • The Magic: Because these stickers are simple and explicit, the computer can instantly calculate how light hits them from any angle. It's like swapping a slow, heavy brain for a super-fast, lightweight calculator.

2. How It Handles the "Messy" Real World

In a real room, light doesn't just come from the projector; it bounces off walls, ceilings, and other objects (Global Illumination).

• The Problem: Previous systems either ignored this (making things look fake) or needed a second light source right next to the camera to measure it (which is impractical).
• The GS-ProCams Solution: It treats the "extra" light as a residual color. Think of it like a painter who first paints the main subject perfectly, and then adds a subtle, transparent layer of "ambient glow" on top to match the room's lighting. It learns this glow automatically without needing extra hardware.

3. Fixing the Blur (The PSF)

Projectors aren't perfect lasers; they have a "Point Spread Function" (PSF), which means light from one pixel bleeds slightly into its neighbors, causing blur (defocus), especially on curved surfaces.

• The Analogy: Imagine shining a flashlight through a foggy window. The light spreads out.
• The Fix: GS-ProCams learns a tiny "blur filter" (a 5x5 grid) that simulates how the projector's lens naturally smears the light. It uses this to pre-correct the image so that when it hits the bumpy surface, it looks sharp.

4. Why It's a Game Changer

The paper highlights three massive improvements, which we can compare to upgrading from a film camera to a modern smartphone:

1. Speed (900x Faster):
   • Old Way: Takes hours to learn a scene and seconds to show a result.
   • GS-ProCams: Takes minutes to learn and shows results instantly (like a live video feed).
2. Efficiency (1/10th the Memory):
   • Old Way: Requires a massive, expensive supercomputer to run.
   • GS-ProCams: Runs on a standard gaming laptop.
3. View-Agnostic (Walk Around Freely):
   • Old Way: If you move your head, the projection looks wrong. You have to stop and retrain the system.
   • GS-ProCams: You can walk around the object, and the projection stays perfectly aligned and realistic, no matter where you stand.

Real-World Applications

Because it's so fast and flexible, GS-ProCams opens the door to things that were previously too hard:

• Diminished Reality: Imagine projecting a "hole" in a wall so you can see through it, or hiding a messy cable by projecting a clean wall texture over it.
• Text-Driven Mapping: You could type "Make this wall look like a tiger" and the system instantly projects a realistic tiger onto the wall, adapting to the wall's curves and lighting.
• Interactive Art: Artists can project onto moving crowds or complex sculptures, and the image will stay locked to the object even as people walk around it.

The Bottom Line

GS-ProCams is like giving a projector a pair of smart glasses and a super-fast brain. It understands the 3D shape of the world, knows how light behaves, and corrects the image in real-time, allowing us to paint light onto the real world with the ease of using a digital tablet.

Technical Summary

1. Problem Statement

Projector-Camera Systems (ProCams) are essential for applications like projection mapping, augmented reality, and industrial inspection. These systems require establishing accurate geometric (where light hits) and photometric (how light reflects) mappings between the projector and the camera.

Existing methods face significant limitations:

• View-Specific Methods (CNN-based/LTM): Traditional approaches (e.g., Light Transport Matrix, CNN-based methods) are tightly coupled to a specific viewpoint. They fail to generalize to novel viewpoints, requiring retraining and recalibration for every new camera angle.
• View-Agnostic Methods (NeRF-based): Recent Neural Radiance Field (NeRF) approaches allow for view-agnostic simulation but suffer from high computational costs, slow inference speeds, and heavy memory usage. Crucially, they often require additional hardware (a co-located light source) and dark room conditions to separate geometry from material properties, limiting real-world applicability.
• Efficiency Gap: There is a lack of a framework that is both view-agnostic and computationally efficient enough for real-time or practical deployment without specialized hardware constraints.

2. Methodology: GS-ProCams

The authors propose GS-ProCams, the first framework to utilize 2D Gaussian Splatting (2DGS) for ProCams. The core idea is to represent the scene using 2D Gaussians augmented with physical attributes to model complex light-surface interactions.

Key Technical Components:

• Scene Representation:

  • The scene is represented by 2D Gaussians, which inherently provide geometric consistency across viewpoints.
  • Augmentation: Each Gaussian point is extended with learnable Bidirectional Reflectance Distribution Function (BRDF) attributes: Albedo ($b$) and Roughness ($r$).
  • Global Illumination: To handle environmental lighting and indirect effects without expensive global illumination solvers, the method approximates residual global illumination as a view-dependent residual color ($c_g$) using Spherical Harmonics (SH) coefficients.

• Differentiable Physically-Based Rendering (PBR):

  • The framework models the rendering equation where the outgoing light is a sum of Projector Direct Light and Global Illumination.
  • Projector Modeling: The projector's direct light ($L_p$) is modeled by applying a learnable Point Spread Function (PSF) kernel ($\kappa$) to the projected pattern, accounting for defocus blur. It also incorporates projector gamma ($\gamma_p$) and gain ($G_p$).
  • Lighting Equation: $L_o = f(\omega_o, \omega_p, x_s) \cdot L_p \cdot (\omega_p \cdot n_s) + C_g$, where $C_g$ captures the residual environment light.

• Joint Optimization:

  • The system jointly optimizes the Gaussian parameters (position, rotation, scale, opacity, SH, albedo, roughness) and projector response parameters ($\gamma_p, G_p, \kappa$) using a differentiable rendering pipeline.
  • Loss Functions:
    • Photometric Loss: L1 and DSSIM between simulated and captured images.
    • Geometric Regularization: Depth distortion and normal consistency losses to ensure surface smoothness.
    • Material Regularization: Bilateral smoothness guided by albedo to prevent noise in roughness maps.
    • Mask Entropy: Ensures opacity aligns with the region of interest (e.g., projector FOV).

• Applications:

  • ProCams Simulation: Synthesizing camera images for arbitrary projector patterns and viewpoints.
  • Projector Compensation: Solving an inverse rendering problem to find the optimal projector input pattern that, when projected, results in a desired appearance on the surface, even from novel viewpoints.

3. Key Contributions

1. First Gaussian Splatting Framework for ProCams: Introduces GS-ProCams, integrating 2D Gaussians with physically-based rendering to model geometric and photometric mappings efficiently.
2. View-Agnostic without Extra Hardware: Unlike NeRF-based methods, GS-ProCams operates under ambient lighting without requiring a co-located light source or a dark room.
3. Superior Efficiency:
   • Training Memory: Uses only 1/10 of the GPU memory required by NeRF-based methods.
   • Inference Speed: Achieves 900x faster inference speeds compared to NeRF-based approaches.
4. Unified Optimization: Eliminates the need for multi-stage training (separating geometry and material) by jointly optimizing all parameters in a single differentiable system.
5. New Benchmark: Introduces a comprehensive real-world benchmark dataset for view-agnostic ProCams, featuring various viewpoints, textured surfaces, and environmental lighting conditions.

4. Experimental Results

The authors evaluated GS-ProCams on both synthetic (Nepmap) and real-world datasets, comparing it against state-of-the-art methods (Nepmap, DeProCams, DPCS, CompenHR).

• Synthetic Dataset (Nepmap):
  • Quality: GS-ProCams achieved higher PSNR (31.97 vs. 27.24), SSIM, and lower LPIPS compared to Nepmap.
  • Efficiency: Training time dropped from ~3 hours to 3.6 minutes. Inference FPS increased from 0.3 to 297.
• Real-World Benchmark:
  • View-Agnostic Performance: GS-ProCams successfully generalized to novel viewpoints without retraining, whereas view-specific baselines (DeProCams, DPCS) required retraining for every new angle and failed to generalize.
  • Compensation Quality: In projector compensation tasks, GS-ProCams achieved the best PSNR (24.59) and SSIM (0.8652) on novel viewpoints, outperforming view-specific methods that had been retrained on those specific views.
  • Robustness: The method maintained high quality even with as few as 4 training viewpoints.
• Ablation Studies:
  • Removing the PSF model resulted in unrealistically sharp projections and artifacts on complex geometries, confirming the necessity of modeling projector defocus.
  • The method is robust to varying numbers of training patterns and viewpoints.

5. Significance and Impact

GS-ProCams represents a paradigm shift in Projector-Camera Systems by bridging the gap between high-fidelity physical modeling and real-time efficiency.

• Practicality: By removing the need for dark rooms and co-located lights, it makes advanced projection mapping and compensation feasible in standard indoor environments.
• Scalability: The massive reduction in memory and time requirements allows for deployment on consumer-grade hardware and enables real-time interactive applications.
• Future Direction: It opens the door for dynamic, view-agnostic AR and VR applications where the system can adapt to moving observers and changing lighting conditions without heavy computational overhead.

Limitations: The current framework assumes static scenes and struggles with highly specular or transparent materials (e.g., glass) and complex depth-of-field blur variations, which are identified as areas for future improvement.