Edge-based natural image reconstruction provides a unified account of many lightness illusions

This paper demonstrates that a deep learning framework based on edge-driven image reconstruction can unify diverse lightness illusions as emergent properties of surface filling-in mechanisms, challenging the need for complex 3D scene inference or distinct processing mechanisms.

The Gist

Imagine your eyes are like a high-tech security camera, but instead of sending a full, crystal-clear video feed to your brain, it only sends a sketchy, black-and-white outline of the world. It tells your brain, "Here is where a wall ends and a door begins," or "Here is a sharp line between light and dark." It doesn't send the actual colors, textures, or smooth surfaces.

So, how does your brain turn that sketchy outline into the rich, colorful, 3D world you see? It has to fill in the blanks.

This paper is about a team of scientists who built a computer brain (a deep learning model) to see if that "filling in" process is actually what causes our visual illusions.

The Big Idea: The "Sketch-to-Painting" Machine

The researchers asked a simple question: If you give a computer only the edges of an image and ask it to "paint" the rest of the picture based on what it has learned from looking at millions of real photos, will it make the same mistakes humans do?

They built a model they called "Edge-Net."

• The Input: They took real photos and ran them through a filter that erased everything except the edges (like a rough pencil sketch).
• The Task: They trained the computer to look at that sketch and try to reconstruct the original, full-color photo.
• The Twist: They never taught the computer about optical illusions. They just let it learn how to paint natural scenes.

The Magic Result: The Computer Gets "Tricked"

When they tested this Edge-Net on famous optical illusions, something amazing happened. The computer started seeing things that weren't there, exactly the way humans do.

Here are three examples of how they matched:

1. The Cornsweet Illusion (The "Magic Edge"):

   • The Trick: Imagine two gray squares side-by-side. They are the exact same color. But if you put a tiny, sharp line between them (one side slightly lighter, one slightly darker), your brain thinks the whole left square is dark and the whole right square is light.
   • The Computer: The Edge-Net looked at the sketch (which only had that tiny line) and "painted" the left side dark and the right side light. It didn't know the squares were supposed to be the same! It just followed the edge.

2. The Moon Illusion (The "Hazy Sky"):

   • The Trick: If you put a gray moon against a dark, hazy sky, it looks bright. If you put the exact same gray moon against a bright, hazy sky, it looks dark. Your brain assumes the moon is being lit differently by the atmosphere.
   • The Computer: Even though the moon pixels were identical, the Edge-Net painted the moon in the dark sky as bright white and the moon in the bright sky as dark gray. It "inferred" the lighting conditions just like a human.

3. The Checkerboard Illusion (The "Shadow"):

   • The Trick: On a checkerboard, a square in a shadow looks much lighter than an identical square in the sun. Your brain knows the shadow is dimming the light, so it "brightens" the square in your mind.
   • The Computer: The Edge-Net successfully reconstructed the shadowed square as lighter than the sunlit square, mimicking our brain's ability to see past the shadow.

The "Denoising" Test: It's About Edges, Not Just "Fixing"

To make sure this wasn't just a fluke, they tried a different kind of computer brain. They built a model trained to take a noisy, static-filled TV picture and clean it up.

• The Result: This "Denoising" model did not see the illusions. It saw the squares as the same color.
• The Lesson: This proves that it's not just about "filling in missing information." It's specifically about how our brains process edges and contrasts. Our visual system is wired to interpret edges as boundaries of objects, and that specific wiring is what creates the illusions.

The Limitations: Where the Computer Gets Stuck

The computer was great at many illusions, but it failed at a few tricky ones (like the Koffka Ring or White's Illusion).

• Why? These illusions require the brain to understand grouping (e.g., "These lines belong to one object, not another"). The Edge-Net was good at local painting (filling in a patch based on its immediate neighbors) but bad at understanding the "big picture" of how objects are grouped together.
• The Metaphor: The computer is like a painter who is amazing at blending colors within a single room but doesn't quite understand that the room is part of a larger house.

The Takeaway: Why We See What We See

For a long time, scientists thought our brain was like a detective solving a complex mystery: "Is that a shadow? Is that a light source? Is that a 3D object?" They thought we had to do heavy math to figure out the "true" nature of the world.

This paper suggests a simpler, more elegant answer: We don't solve the mystery; we just paint the picture.

Our visual system's main job is to take a compressed stream of edge data from our eyes and reconstruct a smooth, complete surface. The "illusions" aren't bugs in our system; they are side effects of a highly efficient painting algorithm. We see the world the way we do because our brain is optimized to turn a sketchy outline into a vivid reality, and sometimes, in doing so, it gets a little creative.

In short: Your brain is a master artist who only has a pencil sketch to work with. It fills in the rest of the canvas so beautifully that sometimes, it paints things that aren't actually there. And that's exactly how we see the world.

Technical Summary

1. Problem Statement

The human visual system transforms local measurements of light (retinal inputs) into a rich, coherent perception of surface lightness and color. However, identical physical luminance values can appear dramatically different depending on context (e.g., shadows, surrounding haze, or adjacent edges), a phenomenon known as lightness illusions.

• The Core Question: How does the visual system reconstruct surface properties from primarily local, edge-based contrast information?
• The Competing Hypotheses:
  1. Inverse Graphics: The visual system performs complex, high-level inference to disentangle intrinsic surface properties from ambient lighting, 3D geometry, and shadows. This requires sophisticated scene decomposition and is computationally expensive.
  2. Edge-Based Filling-In: The visual system uses a simpler, modular mechanism where local contrast edges trigger a "filling-in" process to reconstruct surfaces, without explicit 3D or lighting inference.
• The Gap: While "inverse graphics" theories explain complex illusions (like the Checkerboard or Moon illusions), they lack computational models that operate on natural images and explain these phenomena as emergent properties rather than explicit goals. Conversely, simple filling-in models have struggled to explain complex illusions.

2. Methodology

The authors utilized a deep learning framework to test the Edge-Based Reconstruction Hypothesis. They posited that if the sole computational goal is to reconstruct a full image from edge-based inputs (mimicking retinal ganglion cell responses), lightness illusions should emerge naturally as reconstruction errors.

A. Model Architecture (Edge-Net)

• Architecture: A standard U-Net (8-layer encoder, 2048-dimensional bottleneck, 8-layer decoder).
• Input: Natural images processed through a Laplacian edge filter (second-order gradient). This simulates center-surround receptive fields, preserving local contrast relationships while discarding absolute intensity information.
• Output: A full grayscale reconstruction of the original image.
• Training Objective: Minimize the Mean Absolute Error (MAE) between the reconstructed image and the ground truth grayscale image.
• Datasets: Trained on four variations of ImageNet subsets (two small sets of ~1k images, two large sets of ~8k images) to test robustness.

B. Control Condition (DeNoise-Net)

To ensure the illusions were not merely a result of any generative reconstruction task, the authors trained control models with the same architecture and datasets but a different input:

• Input: Natural images corrupted with Gaussian noise (low and high levels).
• Goal: Denoise the image to recover the original.
• Rationale: Both tasks involve "filling in" missing information, but denoising preserves absolute intensity information, whereas edge-based reconstruction relies solely on contrast.

C. Evaluation

• Stimuli: The models were tested on classic lightness illusions (Cornsweet, Moon, Checkerboard, Koffka, White, etc.) that were never seen during training.
• Metric: Illusion Magnitude was calculated as the difference in mean pixel intensity between two regions that are physically identical in the ground truth but perceived differently by humans.
• Human Comparison: A behavioral experiment was conducted where participants performed a "Pattern Match Task" (adjusting brightness/contrast of a moon region to match their perception) to allow direct quantitative comparison with the model's reconstruction errors.

3. Key Contributions

1. Unified Computational Account: The paper demonstrates that a single, modular objective—reconstructing surfaces from edge contrasts—can explain a wide spectrum of lightness illusions, ranging from simple edge propagation (Cornsweet) to complex scene-based illusions (Moon, Checkerboard).
2. Emergent Phenomena: The illusions are not programmed into the model; they emerge naturally as by-products of learning to reconstruct natural images from compressed edge data.
3. Refutation of "Any Reconstruction" Theory: By showing that denoising models (which also reconstruct images) fail to produce these specific illusions, the authors prove that the specific constraint of contrast-edge input is critical for generating human-like lightness perception.
4. Quantitative Human-Model Alignment: The study provides a novel method for directly comparing human perceptual judgments and model reconstruction errors, showing strong alignment in the Moon Illusion task.

4. Key Results

A. Reproduction of Core Illusions

The Edge-Net models successfully recapitulated three major illusions across all training datasets:

• Cornsweet Illusion: The model correctly "filled in" adjacent panels with different brightness levels based on a thin edge gradient, despite the panels being identical gray in the ground truth.
• Moon Illusion: The model reconstructed the moon in a "light haze" as darker and the moon in a "dark haze" as lighter, mirroring human perception, even though the moon pixels were identical.
• Checkerboard Illusion: Models trained on larger datasets (8k) successfully reconstructed the shadowed square (Region B) as lighter than the non-shadowed square (Region A), inferring a lighter surface material despite identical luminance.

B. The Necessity of Edge Constraints

• Denoising Models Failed: The DeNoise-Net models produced veridical (accurate) reconstructions for the Cornsweet and Checkerboard illusions and showed a reversal of the Moon illusion. This confirms that the illusions arise specifically from the loss of absolute intensity and reliance on local contrast edges, not just from the difficulty of image reconstruction.

C. Generalization and Limitations

• Successes: The model generalized to other illusions like Simultaneous Contrast, Adelson Haze, and Cornsweet Cube variants.
• Failures (Boundary Conditions): The model failed to replicate the Koffka Ring (when the ring is connected) and the White Illusion (parametric width variations).
  • Reasoning: In these cases, human perception relies on grouping and border ownership (segmenting the image into objects). The Edge-Net, lacking explicit segmentation or "anchoring" mechanisms, could not resolve competing edge cues when a region was surrounded by both light and dark influences.

D. Human-Model Quantitative Match

In the Moon Illusion experiment:

• Humans showed a specific trend where illusion strength decreased as the contrast of the moon increased (in the Pattern Match task).
• The Edge-Net models exhibited the exact same quantitative trend, aligning both qualitatively and quantitatively with human data. This suggests the model's "filling-in" mechanism is sensitive to edge strength in a way that mirrors human biology.

5. Significance and Conclusion

• Paradigm Shift: The study challenges the dominant "Inverse Graphics" view that lightness perception requires explicit, high-level inference of 3D geometry and lighting sources. Instead, it suggests that many complex illusions are reconstruction by-products of a simpler, early-stage goal: recovering a complete surface representation from a compressed, edge-based code.
• Biological Plausibility: The findings align with neurophysiological evidence that the retina and LGN transmit primarily edge/contrast information, and that the cortex must "fill in" the rest.
• Future Directions: The paper identifies that while edge-based reconstruction explains a vast amount of lightness perception, it lacks grouping/segmentation mechanisms (anchoring). Future models incorporating "border ownership" or segmentation-aware normalization could bridge the gap to explain the remaining illusions (like White's illusion) that the current Edge-Net misses.

In summary, the authors provide a powerful, unified framework suggesting that the visual system's primary drive is not to "solve" the physics of the world, but to efficiently reconstruct surfaces from edge data, with lightness illusions serving as the inevitable artifacts of this efficient coding strategy.