CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis

The paper introduces CARL, a novel camera-agnostic representation learning model that utilizes a self-attention-cross-attention spectral encoder and feature-based self-supervision to overcome spectral heterogeneity across RGB, multispectral, and hyperspectral imaging modalities, demonstrating superior robustness and generalizability in diverse domains like medicine, autonomous driving, and remote sensing.

The Gist

Imagine you are trying to teach a computer to recognize objects in photos. You have a huge library of pictures, but here's the catch: every photo was taken with a different camera. Some cameras see only three colors (Red, Green, Blue), like your phone. Others see dozens of colors, like a high-tech medical scanner. Some see infrared light; others see ultraviolet.

Currently, if you train an AI on photos from Camera A, it gets confused when you show it photos from Camera B. It's like teaching a student to speak only French, and then expecting them to understand a conversation in German. They share the same alphabet (pixels), but the "words" (wavelengths of light) are different. This forces scientists to build a separate, expensive AI model for every single type of camera they own, wasting data and limiting how smart these systems can get.

Enter CARL: The Universal Translator for Light.

The paper introduces CARL (Camera-Agnostic Representation Learning), a new AI model designed to solve this exact problem. Think of CARL not as a camera, but as a universal translator that sits between the camera and the brain of the AI.

Here is how it works, using some simple analogies:

1. The "Spectral Encoder": The Smart Translator

Imagine you have a book written in a language with 100 words (a hyperspectral camera) and another book with only 10 words (a multispectral camera). A normal AI tries to read them literally, getting confused by the different lengths.

CARL uses a special Spectral Encoder. Think of this as a translator who doesn't just read the words; they understand the meaning behind them.

• Wavelength Awareness: The translator knows that "Red" on Camera A is slightly different from "Red" on Camera B. It uses a special map (called positional encoding) to align the colors correctly, even if the cameras use different shades.
• Distillation: Instead of trying to memorize every single color channel, the encoder acts like a master summarizer. It takes all that complex light information and distills it down into a few "Golden Nuggets" of meaning (called spectral representations). No matter if the input has 3 channels or 100, the output is always the same clean, organized summary.

2. The "Self-Supervised" Gym: Learning Without a Teacher

Usually, to teach an AI, you need a human to label thousands of pictures ("This is a tumor," "This is a car"). This is slow and expensive.

CARL uses a trick called Self-Supervised Learning. Imagine a student in a gym who is trying to learn to juggle. Instead of a coach telling them exactly what to do, the student is given a ball, then the ball is hidden, and they have to guess what the ball looked like based on the other balls they are holding.

• The Masking Game: CARL takes an image, hides (masks) some of the color channels, and asks the AI: "Based on the colors you can see, what were the hidden colors?"
• The Teacher-Student Duo: The AI has a "Student" network that tries to guess, and a "Teacher" network (which is a slightly older, smarter version of the student) that knows the answer. The student learns by trying to match the teacher's predictions.
• The Result: Because CARL learns the structure of light itself, it doesn't need a human labeler for every single photo. It can learn from millions of unlabeled images from any camera.

3. Why This Matters: The "One Model to Rule Them All"

The researchers tested CARL in three very different worlds:

• Medicine: Distinguishing between healthy and diseased organs during surgery using different medical cameras.
• Autonomous Driving: Recognizing traffic lights and signs using both standard car cameras and expensive hyperspectral sensors.
• Satellite Imaging: Analyzing Earth from space using data from satellites that have completely different sensors.

The Magic Result:
In every test, CARL didn't just work; it thrived.

• When other models got confused by a new camera type, CARL kept performing perfectly.
• It could take knowledge learned from a standard RGB camera (like a phone) and apply it to a complex medical scanner, and vice versa.
• It successfully identified objects (like "poles" in a city scene) that were missing from the training data of one camera type, simply because it had learned the concept of "poles" from another camera type.

The Big Picture

Think of the current state of AI in spectral imaging as having a different dictionary for every language in the world. If you want to learn a new language, you have to buy a whole new dictionary.

CARL is the Rosetta Stone. It creates a single, universal dictionary of "light" that works for any camera, past or future. This means we can finally combine all our scattered data silos into one massive, powerful brain that can see the world clearly, no matter what kind of eye (camera) is looking at it.

In short: CARL teaches AI to understand the essence of light, rather than just memorizing the specific settings of a camera, making it robust, versatile, and ready for the future of imaging.

Technical Summary

1. Problem Statement

Spectral imaging (including RGB, multispectral, and hyperspectral) offers rich data for applications in medicine, autonomous driving, and remote sensing. However, a critical bottleneck exists: spectral heterogeneity. Different cameras capture varying numbers of channels (dimensionality) at different specific wavelengths.

• Current Limitations: Existing deep learning models are typically "camera-specific," requiring retraining for every new sensor configuration. They fail to generalize across datasets with different spectral properties.
• Data Silos: Valuable data exists in isolated silos (e.g., a specific hospital's hyperspectral camera vs. a satellite's multispectral sensor), but knowledge cannot be transferred between them because standard models (CNNs, ViTs) assume fixed channel dimensions and ignore wavelength information.
• Self-Supervised Learning (SSL) Gap: While SSL is powerful for scaling, existing spectral SSL methods are either limited to spatial encoding (ignoring spectral relationships) or require fixed channel dimensions, preventing them from leveraging massive, heterogeneous cross-camera datasets.

2. Methodology: The CARL Framework

The authors propose CARL (Camera-Agnostic Representation Learning), a unified framework designed to learn representations that are invariant to the specific camera sensor but aware of the physical wavelengths.

A. Architecture

The model consists of two main stages:

1. Spectral Encoder ($E_{spec}$):
   • Input: A spectral image $I \in \mathbb{R}^{H \times W \times C}$ with arbitrary channel count $C$.
   • Wavelength Positional Encoding: Instead of treating channels as arbitrary indices, the model uses the physical wavelength ($\lambda_i$) of each channel to generate a positional encoding $PE(\lambda_i)$ using sinusoidal Fourier features. This allows the model to understand the spectral relationship between channels regardless of the camera's specific setup.
   • Self-Attention & Cross-Attention: The encoder processes spectral tokens (patches + wavelength encoding) using a self-attention mechanism. Crucially, it employs cross-attention with a set of $K$ learnable spectral representations (spectral tokens). This mechanism distills the salient spectral information from the variable-length input into a fixed-size set of $K$ representations.
   • Aggregation: The distilled spectral tokens are aggregated (via summation) to produce a camera-agnostic feature map enriched with spectral attributes.
2. Spatial Encoder ($E_{spat}$):
   • The camera-agnostic feature map is passed to a standard spatial encoder (e.g., ViT or EVA-02) to capture geometric and spatial relationships.
   • This decoupling allows the model to leverage powerful pre-trained spatial backbones while handling spectral variability in the front-end.

B. Self-Supervised Pre-training Strategy (CARL-SSL)

To train on large-scale unlabeled data across different sensors, the authors introduce CARL-SSL, a novel feature-based self-supervised framework:

• Dual Masking: The strategy employs two parallel masking tasks:
  1. Spectral SSL: Randomly masks specific channels (wavelengths) in the input. The student network must predict the masked spectral features using the unmasked channels and the wavelength positional encoding.
  2. Spatial SSL: Adapts I-JEPA (Joint Embedding Predictive Architecture) to mask spatial regions of the feature map.
• Teacher-Student Distillation: A teacher network (updated via Exponential Moving Average) processes the full input to generate target features. The student network predicts these targets using only the masked inputs.
• Loss Function: The model is optimized using VICReg loss, which ensures invariance between predictions and targets while preventing feature collapse by maximizing variance and minimizing covariance.

3. Key Contributions

1. First Camera-Agnostic Spatio-Spectral Encoding: CARL is the first method to simultaneously achieve wavelength awareness, channel invariance, and spatio-spectral encoding. It can process images from any sensor (RGB to Hyperspectral) without architectural changes.
2. Novel Spectral SSL Framework: CARL-SSL introduces the first self-supervised strategy that learns spatio-spectral relationships in a camera-agnostic manner, enabling pre-training on massive, heterogeneous datasets.
3. Cross-Domain Validation: The approach was validated across three distinct domains:
   • Medical Imaging: Organ segmentation in hyperspectral surgery.
   • Automotive: Urban scene segmentation (RGB + Hyperspectral).
   • Remote Sensing: Satellite image classification and segmentation.

4. Experimental Results

The paper demonstrates that CARL outperforms both camera-specific baselines and existing channel-invariant methods (e.g., Spectral Adapter, DOFA, HyperFree, SpectralGPT+).

• Medical Imaging (Organ Segmentation):
  • CARL maintained high mIoU scores even when the training set was progressively replaced with synthetic multispectral images of varying channel counts and wavelengths.
  • Baseline models suffered significant performance drops as spectral heterogeneity increased, while CARL remained robust.
• Automotive (Urban Scene Segmentation):
  • Zero-Shot Transfer: In the HSICity dataset, the "pole" class was absent from the training set but present in the test set. A camera-specific model failed completely (0% IoU). CARL, leveraging RGB data from Cityscapes during pre-training, successfully transferred the "pole" concept to the hyperspectral domain, achieving the highest IoU.
  • CARL-SSL achieved a 50.1 mIoU, outperforming the next best method (49.6).
• Remote Sensing (Satellite Imaging):
  • Pre-trained on ~800k images from Sentinel-2 (multispectral) and EnMAP (hyperspectral).
  • Out-of-Distribution (OOD) Generalization: CARL achieved the best average rank (1.6) across 11 benchmark datasets, including those from unseen sensors (e.g., Gaofen-5 with 116 bands, LandSat-8). It significantly outperformed other foundation models on OOD sensors, demonstrating superior cross-sensor generalization.

5. Significance and Impact

• Unlocking Data Silos: CARL enables the joint training of diverse spectral datasets that were previously incompatible, effectively breaking down data silos in medical, industrial, and remote sensing applications.
• Foundation Model Potential: By decoupling spectral and spatial learning, CARL provides a scalable backbone for future spectral foundation models, capable of handling the extreme variability of real-world sensors.
• Efficiency vs. Performance Trade-off: While slightly more computationally expensive than purely spatial encoders, the paper argues that the cost is justified by the massive gains in generalization and robustness, especially in scenarios where retraining for every new sensor is infeasible.

In conclusion, CARL represents a paradigm shift from "sensor-specific" to "sensor-agnostic" spectral AI, leveraging physical wavelength information and self-supervised learning to create robust, transferable representations for the future of spectral imaging.