Scale Equivariance Regularization and Feature Lifting in High Dynamic Range Modulo Imaging

This paper proposes a learning-based HDR restoration framework for modulo imaging that combines scale-equivariant regularization with a feature lifting input design to effectively distinguish natural edges from wrapping artifacts and achieve state-of-the-art reconstruction performance.

The Gist

Imagine you are trying to take a photo of a scene that has both a blindingly bright sun and a pitch-black cave.

The Problem: The "Overflowing Bucket"
Standard camera sensors are like buckets with a fixed size. If the water (light) pours in faster than the bucket can hold, it overflows. In a photo, this means the bright parts turn into a flat, white blob, and you lose all the details. This is called "saturation."

The Modulo Solution: The "Rolling Odometer"
To fix this, the researchers use a special camera technique called Modulo Imaging. Instead of letting the bucket overflow and spill, imagine the bucket has a magical reset button. Every time the water hits the top, it instantly drops back to zero and starts filling up again.

• The Catch: If you look at the water level later, you see a number like "5 gallons." But was that 5 gallons, or 105 gallons (5 + 100 reset cycles), or 205 gallons? You don't know how many times it "wrapped around." This is the Unwrapping Problem. The photo looks like it has weird, artificial jagged lines (discontinuities) where the water reset, making it hard to tell where a real shadow ends and where the "reset" happened.

The New Solution: A Smart AI Detective
The paper proposes a new AI (a deep learning network) to solve this puzzle. It uses two clever tricks to figure out the real picture:

1. The "Feature Lifting" (Giving the Detective More Clues)

Usually, you'd just show the AI the messy, wrapped photo and hope it figures it out. But this paper says, "Let's give the AI a cheat sheet." They feed the AI three different versions of the same scene at once:

• The Raw Photo: The messy, wrapped image (the crime scene).
• The Edge Map: A version that highlights the jumps in the water level. This helps the AI see where the real edges of objects are versus where the "reset" happened.
• The Rough Sketch: A quick, math-based guess of what the big picture looks like (the lighting).

Analogy: Imagine trying to solve a jigsaw puzzle. Instead of just dumping the pieces on the table, you also give the AI the picture on the box (the rough sketch) and a highlighter that marks the edges of the pieces (the edge map). It makes the job much easier and faster.

2. Scale Equivariance (The "Zoom Test")

This is the paper's most unique idea. It teaches the AI a rule about how the world works: If you change the brightness of the light, the photo changes, but the shape of the objects stays the same.

• How it works: The researchers train the AI by showing it the same scene twice: once with normal light, and once with the light turned up or down (simulating different exposure times).
• The Rule: The AI learns that if the light gets brighter, the "wrapped" numbers change, but the AI must be smart enough to realize, "Oh, that's just the same scene, just brighter!" It learns to ignore the fake "reset lines" caused by the brightness and focus on the real shapes of the trees and buildings.

Analogy: Think of a rubber stamp. If you press it hard, the ink is dark; if you press it lightly, the ink is pale. A smart detective knows that a pale stamp and a dark stamp are the same stamp, just applied with different pressure. This AI learns to ignore the "pressure" (exposure) and focus on the "design" (the scene).

The Result

By combining these two tricks—giving the AI extra clues and teaching it to ignore brightness changes—the new system creates a High Dynamic Range (HDR) photo that is incredibly sharp and realistic.

• Old methods often left weird "stripes" or color glitches where the light was too bright.
• This new method successfully reconstructs the blinding sun and the dark cave simultaneously, looking like a photo taken by a human eye rather than a broken camera.

In short: They built a smarter AI that uses extra hints and learns the rules of light to fix photos that would otherwise be ruined by too much brightness.

Technical Summary

1. Problem Statement

High Dynamic Range (HDR) Imaging Limitations: Conventional sensors (CCD/CMOS) suffer from limited well capacity and quantization precision, leading to signal clipping (saturation) in bright regions and loss of detail.
Modulo Imaging Solution: Modulo imaging addresses this by cyclically wrapping pixel intensities that exceed a threshold ($2^b$) rather than clipping them. This allows the capture of signals far beyond the sensor's native dynamic range.
The Core Challenge: The resulting "modulo" images contain artificial discontinuities (wrapping edges) that are indistinguishable from natural image edges. Reconstructing the original HDR scene (the "unwrapping" problem) is an ill-posed inverse problem. Existing methods (e.g., PnP-UA, AHFD, UnModNet) often struggle in high-lighting conditions, failing to distinguish between true scene structures and wrapping artifacts, leading to reconstruction errors and color distortions.

2. Methodology

The authors propose a learning-based HDR restoration framework that integrates two primary strategies: Feature Lifting (input design) and Scale Equivariance Regularization (training constraint).

A. Feature Lifting (Input Construction)

Instead of feeding the raw modulo image ($y$) alone into the network, the authors employ a "feature lifting" strategy. They concatenate three distinct input channels to provide the network with diverse priors:

1. Raw Modulo Image ($y$): Preserves the full scene content but includes wrapping artifacts.
2. Wrapped Finite Differences ($M_b(\Delta y)$): Calculates the finite differences of the modulo image. This explicitly highlights edge information and corrects gradient discontinuities, helping the model distinguish true edges from wrapping jumps without learning these filters from scratch.
3. Closed-Form Initialization ($x_0$): A physics-based initial estimate derived from the 2D unwrapping problem (solved via 2D DCT). This captures large-scale illumination trends, allowing the deep network to focus on refining textures and correcting residual wraps rather than learning global structure from zero.

B. Scale Equivariance Regularization ($R_{eq}$)

The authors introduce a regularization term based on the principle of Equivariant Imaging.

• Concept: Changing the exposure time of a scene scales the true intensity values ($x \to \alpha x$) but does not change the underlying scene structure. Consequently, the optimal reconstruction network should produce scaled outputs for scaled inputs.
• Implementation: During training, the network is exposed to pairs of HDR images and their modulo measurements where the HDR image is scaled by a random factor $\alpha$ (sampled from a uniform distribution, e.g., $[0.9, 1.1]$).
• Loss Function: The network is penalized if the reconstruction of the scaled modulo image ($f_\theta(W_b(\alpha x))$) is not equal to the scaled reconstruction of the original image ($\alpha f_\theta(W_b(x))$).
• Goal: This forces the network to learn invariance to exposure scaling, effectively teaching it to ignore modulo discontinuities (which shift with scale) and focus on natural image edges.

C. Network Architecture

The restoration model is a lightweight variant of DRUNet (4 downsampling levels, 8 feature channels at the first scale), trained with a combined loss function:
$$\mathcal{L} = \mathcal{L}_{rec} + \gamma \mathcal{R}_{eq}$$
Where $\mathcal{L}_{rec}$ is the Mean Squared Error (MSE) between the ground truth and the prediction, and $\mathcal{R}_{eq}$ is the scale-equivariance loss.

3. Key Contributions

1. Scale-Equivariant Regularization: A novel training strategy for modulo imaging that enforces consistency under exposure variations, significantly improving the network's ability to generalize and distinguish artifacts from true structures.
2. Feature Lifting Input Design: A multi-channel input approach combining raw data, gradient information, and a physics-based initialization. This reduces the learning burden on the network and improves convergence.
3. State-of-the-Art Performance: The proposed method outperforms existing deep learning and optimization-based baselines (UnModNet, AHFD, PnP-UA, SPUD) across both perceptual and linear metrics.

4. Experimental Results

Experiments were conducted on the UnModNet dataset using both perceptual (PU21 encoding) and linear HDR domains.

• Input Analysis:

  • Using the raw image ($y$) alone provided a stable baseline.
  • Adding wrapped finite differences ($M_b(\Delta y)$) significantly boosted performance by enhancing edge detection.
  • The closed-form initialization ($x_0$) alone performed poorly and slightly degraded results when combined with others, suggesting the network learns better representations than the simple DCT-based initialization for this specific task.
  • Optimal Configuration: The combination of $y$ and $M_b(\Delta y)$ yielded the best balance of fidelity and efficiency.

• Quantitative Performance:

  • Perceptual Metrics (PU21): The proposed method achieved 25.30 dB PSNR-Y (with regularization), outperforming the previous best (UnModNet at 20.72 dB) by ~4.6 dB. It also improved PSNR in PU21 by ~2.5 dB over UnModNet.
  • Linear Metrics: The method achieved 36.47 dB PSNR-L and 0.973 SSIM-L, significantly surpassing baselines which hovered around 27–29 dB PSNR-L.
  • Regularization Impact: Adding the scale-equivariance term ($R_{eq}$) consistently improved all metrics, confirming its role in enhancing generalization.

• Qualitative Results:

  • Visual comparisons show that previous methods (AHFD, PnP-UA) fail in highly saturated regions (e.g., light sources), introducing color distortions and false discontinuities.
  • The proposed method successfully reconstructs high-fidelity HDR scenes with accurate luminance gradients and minimal artifacts, even in complex lighting conditions.

5. Significance

This work bridges the gap between physical signal processing constraints and deep learning optimization. By integrating physics-informed priors (via feature lifting) and geometric constraints (via scale equivariance), the authors demonstrate that HDR modulo imaging can be solved with high fidelity. The approach offers a robust solution for capturing extreme dynamic ranges in resource-constrained settings, potentially enabling the deployment of modulo cameras in real-world applications where traditional HDR fusion is impractical. The significant gains in perceptual quality (PU21) are particularly notable, as many existing methods fail to preserve color consistency and visual realism in the high-dynamic range regime.