💻 computer science

Breaking the Scalability Limit of Multi-Projector Calibration with Embedded Cameras

This paper introduces a novel calibration framework that embeds cameras into the target surface to capture simultaneous projections, thereby reducing the calibration time for multi-projector systems from linear to nearly constant with respect to the number of projectors while maintaining high accuracy.

Original authors: Takumi Kawano, Kohei Miura, Daisuke Iwai

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Takumi Kawano, Kohei Miura, Daisuke Iwai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "One-by-One" Bottleneck

Imagine you are trying to tune a massive choir of 25 singers (projectors) to sing in perfect harmony on a giant stage (the wall or screen). In the old way of doing this, you had to ask each singer to sing a specific note, wait for the conductor (a camera) to hear it, and then ask the next singer to do the same.

If you have 25 singers, you have to repeat this process 25 times. The more singers you add, the longer it takes. This is the "scalability limit" the paper talks about. For huge systems with dozens of projectors, the calibration process becomes painfully slow and tedious because you can't let them all "sing" at once; their voices would just blend into a confusing mess that the conductor couldn't untangle.

The Solution: Giving the Stage "Eyes"

The researchers came up with a clever trick: instead of having a camera standing in the audience trying to listen to the singers, they embedded tiny cameras directly into the stage itself (the calibration board).

Think of it like this:

Old Way: A person in the back of the room tries to guess which singer is making which sound, but when everyone talks at once, it's just noise.
New Way: Every singer has a tiny microphone built right into their own mouthpiece. Even if 25 singers talk at the exact same time, each microphone only hears the sound coming from its specific direction.

By placing cameras flat against the calibration board, the system can tell the difference between light coming from Projector A and Projector B based purely on the angle at which the light hits the camera. It's like how your ears can tell if a sound is coming from the left or the right. Even if the light beams cross over each other on the board, the embedded cameras can separate them because they arrive from different directions.

The "Slightly Crooked" Fix

There was one small catch. When you build a board with cameras inside it, it's very hard to make the camera's "eye" (the optical center) sit perfectly flat on the surface. It's usually a tiny bit higher or lower, like a table leg that is slightly too long.

If you don't fix this, the math gets messy, and the projectors won't align perfectly. The authors created a special "correction algorithm" to measure exactly how crooked the cameras are and mathematically straighten them out. It's like a software update that tells the system, "Hey, your eyes are tilted 2 millimeters to the left, so let's adjust the picture to compensate."

The Results: Speed and Clarity

The team built a prototype board with four cameras and tested it with up to 25 projectors. Here is what they found:

Speed: Instead of taking hours to calibrate 25 projectors one by one, the new method did it all in one go. They reduced the number of patterns needed by 95%. It went from a linear slog (more projectors = way more time) to a nearly constant speed (more projectors = almost no extra time).
Accuracy: With the "crookedness" fix, the alignment was just as sharp as the old, slow methods. In fact, because the cameras were right on the board, the image was sometimes even clearer than using a camera far away in the room.
Bright Lights: The system worked even in bright sunlight. Because the embedded cameras look directly at the projector light, they don't get confused by the ambient light washing out the image, unlike a camera in the room which would see a washed-out, invisible pattern.

Summary

The paper presents a new way to calibrate multiple projectors by turning the calibration board itself into a sensor. By embedding cameras into the board, the system can listen to all projectors simultaneously, separating their signals based on direction. This breaks the old speed limit, allowing massive projector arrays to be set up quickly and accurately, even in bright conditions.

1. Problem Statement

Conventional multi-projector calibration systems face a fundamental scalability bottleneck.

Sequential Requirement: To calibrate multiple projectors, structured light patterns (e.g., Gray codes) must be projected and captured sequentially. If patterns from multiple projectors are projected simultaneously, they overlap on the target surface, making individual patterns indistinguishable to an external camera due to diffuse reflection.
Linear Time Complexity: Consequently, calibration time and effort increase linearly with the number of projectors ( $O(M)$ , where $M$ is the number of projectors).
Impact: This hinders the deployment of large-scale systems requiring dozens or hundreds of projectors, such as high-brightness stacking, super-resolution displays, light-field displays, and shadow-suppression systems.

2. Methodology

The authors propose a novel calibration framework that replaces the traditional printed calibration target with a board embedded with cameras.

A. Core Concept: Inverse Setup

Instead of using an external camera to observe patterns reflected off a target, the system embeds cameras directly into the calibration target with their optical centers aligned to the board's plane.

Directional Encoding: When multiple projectors simultaneously project structured light, the light rays from different projectors enter the embedded cameras from different incident angles.
Separation: Because the cameras capture light based on direction (similar to light-field cameras), the system can distinguish which projector emitted the light observed at a specific pixel, even if the patterns overlap spatially on the board surface.

B. Calibration Process

Pattern Projection: All $M$ projectors simultaneously project structured light patterns (Gray codes for coarse ID/position, line-shift patterns for sub-pixel accuracy) and a binary temporal sequence to encode the Projector ID.
Data Capture: Embedded cameras capture the light. For each camera pixel receiving light, the system decodes:
- The Projector ID (from the temporal sequence).
- The Projector Pixel Coordinate ( $p_m$ ) (from Gray codes and line-shift patterns).
- The Camera Pixel Coordinate ( $c_n$ ).
Correspondence: This establishes a correspondence between the projector pixel and the camera's optical center on the board. By repeating this for multiple board poses, intrinsic and extrinsic parameters are estimated using Zhang's method.

C. Misalignment Compensation

A critical technical challenge is that the optical centers of embedded cameras cannot be perfectly aligned with the physical board surface due to lens housing and compound optics.

Problem: If the optical center is offset, the intersection point of the light ray on the board ( $x_n$ ) varies per projector, introducing geometric errors.
Solution: The authors introduce an offline calibration step to estimate a homography transformation ( $M_n$ ) for each embedded camera.
- They project a single line from a known projector position to an embedded camera.
- An external camera measures the intersection of this ray with the board.
- This data maps the camera pixel coordinate to the true board intersection coordinate, compensating for the optical offset during the main calibration phase.

3. Key Contributions

Scalability Breakthrough: The method reduces the number of required projection-capture cycles from linear ( $O(M)$ ) to nearly constant ( $O(1)$ ) relative to the number of projectors. All projectors are calibrated in a single batch.
Directional Separation: It solves the "pattern overlap" problem by utilizing incident direction as a separation key, enabling simultaneous decoding of overlapping structured light.
Robustness to Ambient Light: Since embedded cameras receive light directly (high contrast) rather than relying on diffuse reflection, the system remains functional even under strong ambient illumination (e.g., outdoor sunlight), where conventional methods fail.
High-Precision Compensation: The proposed algorithm effectively corrects for optical center misalignment, achieving sub-pixel accuracy comparable to or better than conventional methods.

4. Experimental Results

The authors validated the system using a prototype board with 4 embedded cameras and experiments with up to 25 projectors.

Alignment Accuracy:
- In a 2-projector experiment, the proposed method with compensation achieved alignment accuracy comparable to the conventional method (RMS reprojection error < 1.0 pixel).
- Without compensation, errors increased to ~2.4 pixels, validating the necessity of the correction algorithm.
- In a 25-projector experiment, the compensated method produced sharper images than the conventional method. The conventional method suffered from lower apparent resolution because the external camera was far from the board, whereas embedded cameras are at the source.
Efficiency:
- 25-Projector System: The conventional method required 1,100 patterns (44 per projector) and ~12 minutes. The proposed method required only 54 patterns total, achieving a 95% reduction in pattern count and time.
Ambient Light Robustness:
- Tested outdoors under ~70 klux (direct sunlight). The proposed method successfully separated and decoded patterns, while the conventional external-camera method failed completely due to washed-out patterns.
Separation Limit: The system successfully separated projectors with an angular separation as small as 0.88 degrees.

5. Significance

This work addresses a four-decade-old scalability issue in projection mapping and multi-projector systems. By shifting the paradigm from "observing reflection" to "directing capture," the authors enable:

Rapid Deployment: Large-scale arrays (e.g., hundreds of projectors for light-field displays) can now be calibrated in minutes rather than hours or days.
New Applications: It facilitates the practical use of complex systems like shadow-suppression displays and high-brightness stacking, which were previously too difficult to calibrate.
Future Potential: The directional data captured by the embedded cameras opens the door for further refinements, such as estimating lens aberrations, aperture shapes, and reducing the number of required board poses in future iterations.