🤖 AI

The Importance of Phase in Neural Representations: An Internal Oppenheim-Lim Test of Image Classifiers

This study demonstrates that deep image classifiers, including CNNs and Vision Transformers, internally encode image identity primarily through phase or sign information rather than magnitude, providing a mechanistic explanation for the texture-shape gap between these architectures.

Original authors: Alper Yıldırım

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alper Yıldırım

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Idea: The "Skeleton" vs. The "Clothes"

Imagine you have two people standing in a room. One is wearing a bright red suit, and the other is wearing a blue t-shirt.

In 1981, scientists Oppenheim and Lim discovered something strange about how our eyes and brains process images. They found that if you take the shape (the skeleton, the outline, the pose) of the person in the red suit, but dress them in the texture and brightness (the clothes, the lighting) of the person in the blue t-shirt, your brain will still recognize it as the person in the red suit.

Phase = The Shape/Skeleton (Where things are, edges, contours).
Magnitude = The Clothes/Texture (How bright it is, the color intensity, the "energy").

The brain cares much more about the shape (phase) than the clothes (magnitude) when trying to identify who someone is.

The Question: Do AI Brains Work Like Human Brains?

We know this is true for human vision. But what about Artificial Intelligence (AI) image classifiers? These are computer programs trained to look at pictures and say, "That’s a cat" or "That’s a dog."

The authors of this paper asked: Does this "shape-over-clothes" rule happen inside the AI’s brain, too?

Most previous studies only looked at the input (the raw pixels). This paper looks inside the AI’s hidden layers—the internal thoughts the computer has while it’s deciding what the image is.

The Experiment: The "Chimera" Test

To test this, the researchers created "Chimera" images (like the mythical half-horse, half-eagle creature).

They took two different images (Image A and Image B).
They went inside the AI’s brain at a specific layer.
They built a fake internal representation that had Image A’s "Clothes" (Magnitude) but Image B’s "Skeleton" (Phase/Sign).
They asked the AI: "What do you see?"

If the AI said "Image B," it means the AI relies on the Skeleton (Phase).
If the AI said "Image A," it means the AI relies on the Clothes (Magnitude).

The Results: The AI Loves the Skeleton

The researchers tested four different types of AI architectures (different ways of building the brain):

PRISM2D (A complex-valued network).
GFNet (A spectral network).
ViT (Vision Transformer, very popular in modern AI).
ResNet (A classic Convolutional Neural Network).

The Big Surprise:
In all four models, the AI’s final decision followed the Skeleton (Phase/Sign). Even when they deleted all the specific "clothing" details (magnitude) and replaced them with generic average clothing, the AI could still identify the image correctly.

This means that by the time the AI makes its final decision, it has thrown away the specific texture/brightness details and is relying almost entirely on the structural shape.

The Twist: ResNet is Playing Hide-and-Seek

At first, the classic ResNet model looked like an exception. When the researchers tried to swap the "sign" (which is the AI equivalent of phase for standard networks), nothing happened. The AI didn’t care.

Why?
ResNet uses a filter called ReLU. Think of ReLU as a strict bouncer who throws out any negative numbers. Since "sign" is about positive vs. negative, the bouncer destroyed the sign information before the researchers could check it.

The researchers realized they had to check the information before the bouncer (ReLU) kicked it out. When they did that, they found that ResNet did have a strong sign code, but it was hiding it.

How ResNet Works:

Early Layers: ResNet carries the identity in the spatial structure (the shape).
Late Layers: It compresses that shape information into a single "average" number (the DC term) that the final part of the brain reads. It’s like taking a detailed sketch and summarizing it into a single keyword.

The Difference Between AI Types

The paper explains why some AIs are better at recognizing shapes (like humans) and others are better at recognizing textures (like patterns on fabric).

Transformers (ViT): They commit to the "Skeleton Code" (Phase/Sign) very early in their processing. They are "shape-first" thinkers.
CNNs (ResNet): They start with shape, but because of their design (the ReLU bouncer), they have to work harder to keep that shape information alive, eventually packing it into a summary statistic.

Conclusion

The paper concludes that all these different AI architectures, despite being built differently, converge on the same strategy: Identity lives in the angle/phase (the shape), not the length/magnitude (the brightness/texture).

This gives us a mechanical explanation for why some AI models look more "human-like" (focusing on shape) and others look more "robot-like" (focusing on texture). It’s not magic; it’s about how and when the AI decides to prioritize the skeleton over the clothes.

Technical Summary: The Importance of Phase in Neural Representations: An Internal Oppenheim–Lim Test of Image Classifiers

Problem Statement
The paper investigates whether the phase-magnitude asymmetry observed in natural images—where phase encodes structural identity (edges, contours) and magnitude encodes energy profile—is replicated within the hidden layers of trained image classifiers. While prior work (e.g., Chen et al., 2021; Yang and Soatto, 2020) has explored this asymmetry in pixel space via data augmentation or domain adaptation, it remains unclear where decision-relevant information resides inside a trained model and whether it can be causally manipulated. The authors ask if modern classifiers converge on a phase-dominant identity code in their latent representations, analogous to how frequency-domain structures have been identified in language models.

Methodology
The authors propose a causal intervention protocol, termed the "Neural Oppenheim–Lim" test, to probe the semantic role of phase and magnitude in latent representations.

Intervention Protocol: Given two images $A$ and $B$ from different classes, the authors construct chimeric hidden states at a chosen layer $\ell$ . The chimera $\tilde{h}_A$ retains the magnitude of $A$ ( $|h_A|$ ) but adopts the phase-like component of $B$ ( $\phi(h_B)$ ). The model’s prediction is then recorded to determine if it follows the magnitude donor ( $A$ ) or the phase donor ( $B$ ).
Phase-like Components: The definition of "phase" varies by architecture:
- Complex Stream (PRISM2D): Exact per-channel complex phase.
- Real Feature Maps (GFNet, ResNet): Spatial-spectral phase derived via an analyst-computed 2D Fourier transform over the spatial grid.
- Real Scalars (ViT, ResNet pre-activation): The sign of the activation ( $\text{sign}(h)$ ), serving as the common probe for real-valued streams.
Models Evaluated: Four architectures spanning complex, spectral, convolutional, and attention-based designs: PRISM2D, GFNet-Ti, ResNet-50, and ViT-B/16.
Controls: To rule out trivial explanations (e.g., magnitudes becoming uninformative with depth), the authors employ:
- Phase Necessity: Randomizing phase or setting magnitude to unity.
- Magnitude Sufficiency: Replacing image-specific magnitude with batch-mean magnitude.
- Geometry Checks: Measuring magnitude correlation and cosine similarity between chimeras and donors.
- DC-only Control (ResNet): Isolating the global mean (DC term) from spatial structure to understand global average pooling.

Key Contributions

Causal Layer-Resolved Intervention: Recasts the Oppenheim–Lim phenomenon as a causal test on latent representations, using per-unit sign as a universal probe for complex, spectral, and real-valued networks.
Unified Phase/Sign Code: Demonstrates across four diverse architectures that class identity is carried by the phase-like component of intermediate features, while the readout is largely magnitude-invariant.
Mechanistic Account of Rectification: Reveals that in ResNet, the sign code is latent and hidden by ReLU rectification; it is only recoverable via pre-activation interventions, explaining the apparent magnitude-dependence of post-ReLU features.
Readout-Aware Explanation: Provides a unified account linking the texture–shape gap between CNNs and attention models to differences in when and how identity is committed to phase, shaped by readout geometry (class token vs. global pooling) and inductive biases.

Results

Phase Dominance: In all four models, predictions follow the phase/sign donor. By the final layers, ViT reaches 91%, ResNet 88%, PRISM2D 76%, and GFNet 75% follow-phase rates.
Magnitude Dispensability: Deleting image-specific magnitude (via batch-mean substitution) leaves accuracy near baseline for spectral models and ViT, confirming that identity rides on phase.
Architecture-Specific Dynamics:
- ViT: Commits to a sign-based identity code in the first block and strengthens it monotonically.
- Spectral Models (PRISM2D, GFNet): Consolidate identity into phase by early-to-middle depth.
- ResNet: Initially appears magnitude-dependent due to ReLU. However, pre-ReLU sign swaps show strong phase coding in late blocks. Identity transitions from distributed spatial structure (early layers) to a channel-wise spatial average (DC term) consumed by global average pooling in the final blocks.
Controls: Magnitudes remain image-specific (correlation $\approx 0.6$ ), chimeras remain geometrically distinct from donors, and destroying phase collapses accuracy to chance, ruling out trivial explanations.

Significance
The paper argues that image classifiers of diverse designs converge on a phase/sign-dominant identity code by the time of readout. The differences between architectures lie in the onset depth and the basis of this code, driven by readout geometry and layer inductive biases. This reframes the behavioral gap between texture-biased CNNs and shape-aligned attention models not as a categorical split between phase and magnitude, but as a difference in how readily and where identity is encoded in phase. The findings suggest that explicit phase representation may offer useful inductive biases for vision tasks.