⚡ electrical engineering

Optimal Multispectral Imaging using RGB Cameras

This paper presents a physics-driven framework that optimizes the allocation of target wavelengths across off-the-shelf RGB cameras and narrow multi-band filters by minimizing the spectral condition number of the measurement system, thereby enabling accurate, stable, and noise-robust multispectral imaging.

Original authors: Tomislav Matulić, Ivan Škrabo, Dubravko Babić, Damir Seršić

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tomislav Matulić, Ivan Škrabo, Dubravko Babić, Damir Seršić

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a standard digital camera, like the one in your phone. It sees the world in three colors: Red, Green, and Blue (RGB). It's great for taking photos of your cat or a sunset, but it's a bit "blind" to the hidden details of the world. It can't tell the difference between two types of red paint that look identical to the eye but have different chemical compositions, or spot a bruise on a piece of fruit before it turns brown.

To see these hidden details, scientists use Multispectral Imaging. Instead of just seeing Red, Green, and Blue, they want to see the scene in many specific "slices" of light (wavelengths), like 410nm, 500nm, 620nm, and so on.

The Problem:
Building a camera that sees all these specific slices is usually expensive and complicated. It's like trying to build a custom piano that can play every single note perfectly, but you have to order every key from a specialized factory.

The Clever Solution:
The authors of this paper, Tomislav and his team, came up with a clever, low-cost hack. They said: "Why build a new camera? Let's just use four cheap, off-the-shelf RGB cameras and put special colored glasses (filters) in front of them."

Think of it like this:

You have 4 identical cameras.
You put a special 3-color filter in front of each one.
Each filter only lets through 3 specific "slices" of light.
Camera 1 sees slices A, B, and C.
Camera 2 sees slices D, E, and F.
And so on.

Together, these four cameras can see 12 different slices of light. But here's the catch: The cameras don't just see "Slice A." Because of how the filters and camera sensors work, Camera 1 sees a mix of Slice A, Slice B, and Slice C all at once. It's like trying to figure out the ingredients of a smoothie just by tasting the blended drink. You know the taste, but you need to do some math to figure out how much strawberry, how much banana, and how much milk went in.

The Big Challenge: The "Math Puzzle"
The team needed to figure out which 3 slices to assign to which camera.

If you pick the wrong combination, the math becomes a nightmare. Imagine trying to solve a puzzle where two pieces look exactly the same. If you mix up the ingredients too similarly, the computer gets confused, and tiny errors (like a little bit of digital noise) get blown up into huge mistakes. The result is a blurry, inaccurate picture of the hidden world.

The "Golden Rule" of the Paper:
The authors created a step-by-step guide to solve this puzzle. They treated the problem like a game of Tetris or seating guests at a dinner party.

The Goal: You have 12 specific wavelengths (guests) and 4 cameras (tables) that can each hold 3 guests.
The Rule: You want to seat the guests so that no two tables are "too similar." If two tables have very similar guests, the system gets unstable.
The Secret Weapon: They used a mathematical concept called the "Condition Number."
- Analogy: Imagine a bridge. If the bridge is perfectly balanced (Condition Number = 1), it's super stable. If the bridge is wobbly and leaning to one side (High Condition Number), a tiny wind (noise) could make it collapse.
- The team's goal was to arrange the cameras so the "bridge" was as perfectly balanced as possible.

How They Did It:
They wrote a computer program that tried every single possible way to arrange the 12 wavelengths across the 4 cameras. There were over 15,000 different ways to do this! The computer calculated the "stability score" for each arrangement and picked the winner.

The Winner:
They found the "Golden Arrangement." For example, they decided:

Camera 1 should look at wavelengths 410, 620, and 720.
Camera 2 should look at 430, 520, and 700.
...and so on.

This specific arrangement made the math "sturdy." Even if the camera sensors were a little noisy, the computer could still perfectly reconstruct the hidden details of the scene.

Why This Matters:

It's Cheap: You don't need a $50,000 scientific camera. You can build this with standard cameras you can buy at a store.
It's Flexible: If you want to look at different wavelengths later, you just swap the filters. You don't need to buy new cameras.
It's Robust: By adding a little bit of "redundancy" (having some wavelengths measured by more than one camera), you can make the system even more accurate, like having a backup plan.

In a Nutshell:
This paper is about taking a messy, expensive problem (seeing hidden light colors) and solving it with a clever, low-cost setup of standard cameras. The authors figured out the perfect seating chart for the light waves so that the math works out perfectly, allowing us to see the invisible world clearly and accurately without breaking the bank.

1. Problem Statement

Multispectral imaging (MSI) is crucial for applications like remote sensing, medical imaging, and food safety, but traditional MSI hardware is often expensive and complex. While off-the-shelf RGB cameras combined with narrow multi-band optical filters offer a low-cost alternative, they present a significant challenge: spectral reconstruction is an ill-posed inverse problem.

When an RGB camera views a scene through a multi-band filter, the measured RGB values are linear mixtures of unknown spectral components. The accuracy of recovering the original spectrum depends heavily on the numerical conditioning of the system matrix. Poorly conditioned systems amplify noise, leading to unstable and inaccurate spectral estimates. Furthermore, existing methods often treat filter selection as a heuristic or isolated problem, lacking a systematic framework for allocating specific wavelengths across multiple cameras to ensure global numerical stability.

2. Methodology

The authors propose a physics-driven, deterministic framework to optimize the allocation of target wavelengths across a multi-camera system equipped with RGB sensors and narrow multi-band filters.

A. Linear Forward Model

The paper establishes a linear measurement model where the camera response $y$ is a linear mixture of spectral irradiance $x$ at discrete target wavelengths:
$y = Ax + n$

Mixing Coefficients: The matrix $A$ is constructed based on the overlap between the camera's spectral sensitivity functions ( $S_c(\lambda)$ ) and the optical filter transmittances ( $F_i(\lambda)$ ).
Global System: For a system with $N_{cam}$ cameras, each equipped with a $k$ -band filter, the individual camera models are stacked to form a global system matrix $A_{C_j}$ .

B. Wavelength Allocation as a Design Problem

The core task is formulated as a combinatorial optimization problem:

Target Set: A predefined set of $p$ target wavelengths ( $K_0$ ) is selected.
Configuration Space: The system must assign $k$ wavelengths to each of the $N_{cam}$ cameras such that the union of all assignments covers $K_0$ .
Constraints: The configuration must ensure the system is determined or overdetermined ( $N_{cam} \times C \geq p$ ) and that the resulting matrix has full column rank.

C. Optimization Criterion: Spectral Condition Number

Instead of heuristic selection, the authors use Frame Theory to define the optimality criterion. They propose minimizing the spectral condition number ( $\kappa$ ) of the global system matrix:
$\kappa(A) = \frac{\sigma_{max}(A)}{\sigma_{min}(A)}$

Theoretical Justification: Minimizing $\kappa(A)$ maximizes the worst-case output Signal-to-Noise Ratio (SNR). A condition number close to 1 indicates a "tight frame," where the transformation preserves signal energy and minimizes noise amplification during the least-squares inversion.
Search Strategy: Since the design space is finite (though large), the authors employ an exhaustive search to evaluate all feasible wavelength allocations and select the one with the minimum $\kappa$ .

D. Handling Redundancy

The framework explicitly supports redundant configurations where $N_{cam} \times C > p$ . In these cases, some wavelengths are measured by multiple cameras. The authors demonstrate that this redundancy provides additional degrees of freedom, further improving the conditioning of the inverse problem and noise robustness.

3. Key Contributions

Systematic Design Framework: A deterministic procedure to allocate wavelengths across multiple RGB cameras and filters, moving beyond heuristic or task-specific designs.
Global Optimization: Unlike previous works that optimize filter subsets in isolation, this method optimizes the entire multi-camera system matrix, ensuring the specific configuration yields the most stable inversion.
Theoretical Link to Noise Robustness: The paper rigorously links the spectral condition number to worst-case SNR using frame theory, providing a clear mathematical justification for the design criterion.
Sensor Agnosticism: The framework is applicable to any camera sensor (RGB or multispectral) provided the channel-wise spectral sensitivities are known.
Redundancy Analysis: The paper demonstrates how adding extra cameras (overdetermined systems) can be systematically optimized to enhance reconstruction quality.

4. Results

The method was validated using a representative example:

Setup: 4 RGB cameras, each with a 3-band (triband) filter, targeting 12 specific wavelengths (410 nm to 780 nm) relevant to the food industry.
Sensor: Sony IMX411 sensor characteristics were used.
Search Space: The algorithm evaluated 15,400 feasible configurations.
Optimal Configuration: The search identified a specific wavelength distribution (e.g., Camera 1: {410, 620, 720} nm, etc.) that minimized the condition number to $\kappa_{min} = 9.0840$ .
Redundancy Cases: The authors also simulated scenarios with 11 wavelengths (requiring one repetition) and 5 cameras (15 measurements for 12 wavelengths), showing that the framework successfully identifies optimal allocations even in non-minimal, redundant setups.

5. Significance

This work provides a critical bridge between low-cost hardware and high-fidelity spectral data. By treating wavelength allocation as a rigorous numerical stability problem, the authors enable the construction of customizable, high-performance multispectral systems using mass-produced components.

Practical Impact: It allows researchers and engineers to design MSI systems without expensive custom sensors, simply by selecting the optimal combination of off-the-shelf filters and cameras.
Flexibility: The ability to reconfigure the system post-manufacturing (by swapping filters) offers a level of adaptability not found in fixed-mosaic multispectral cameras.
Robustness: The focus on minimizing the condition number ensures that the resulting spectral reconstructions are reliable even in noisy real-world environments, making this approach viable for industrial and scientific applications.

In conclusion, the paper establishes a mathematically grounded, exhaustive design methodology that transforms the challenge of spectral reconstruction from a heuristic guess into a solvable optimization problem, significantly advancing the feasibility of low-cost multispectral imaging.