Technical Summary: Low-Frequency Shortcuts in Texture-Driven Visual Learning

Problem Statement

Neural networks are prone to shortcut learning, where models learn features that perform well on training data but fail to generalize to in-distribution (ID) or out-of-distribution (OOD) test sets. While existing research on shortcut learning and spectral bias has largely focused on shape-driven standard benchmarks (e.g., CIFAR-10, ImageNet) where classification relies on global object shapes, many real-world application domains are texture-driven. In these domains (e.g., histopathology, textile classification, ground terrain recognition), "shape" is ill-defined or irrelevant, and classification relies on repetitive, fine-grained spatial patterns.

The paper investigates whether shortcut learning manifests differently in texture-driven domains compared to standard benchmarks. Specifically, it asks: Do texture-driven models rely on specific frequency components that act as shortcuts, and how does this reliance affect robustness to OOD corruptions?

Methodology

The authors employ a frequency-based diagnostic framework using the Discrete Cosine Transform (DCT) to analyze learning dynamics.

1. Frequency Pruning Pipeline:

   • Images are transformed into the frequency domain.
   • Frequency components are grouped into diagonals (from top-left to bottom-right), where oscillation rates and spatial complexity increase.
   • The authors apply pruning strategies by zeroing out specific sets of diagonals: Low-Frequency Components (LFCs), Mid-Frequency Components (MFCs), and High-Frequency Components (HFCs).
   • Both training and test sets are pruned according to the strategy, and models are trained from scratch to evaluate the impact on ID and OOD performance.

2. Spectral Behavior Analysis:

   • Accuracy Contributions: The authors measure the "accuracy contribution" of individual frequency components by pruning them one by one from the test set and attributing the resulting accuracy drop to that component.
   • Definition of Low-Frequency Shortcut: A phenomenon where a model's accuracy contributions are concentrated on a small number of LFCs, despite the classification information residing in higher-frequency, fine-grained details.

3. Experimental Setup:

   • Datasets: Six datasets were used: SP-Colorectal (histopathology), TextileNet (textiles), GTOS (ground terrain), GalaxyMNIST, EuroSAT (land use), and CIFAR-10 (standard benchmark).
   • Models: ResNet-50, MobileNet-V3, ViT-Small, ViT-Tiny, and foundation models (DinoV2, CLIP).
   • OOD Corruptions: ImageNet-C corruptions were used, categorized into low-frequency (fog), high-frequency (Gaussian blur), and mixed (elastic transform).

Key Findings and Results

1. Existence of Low-Frequency Shortcuts in Texture-Driven Domains

• Skewed Spectral Behavior: In texture-driven domains (SP-Colorectal, TextileNet, GTOS), models trained on unpruned data exhibit a highly skewed spectral behavior. They make the majority of their decisions based on a few Low-Frequency Components (LFCs).
• Contrast with Shape-Driven Data: Unlike CIFAR-10, where accuracy contributions are more balanced across the spectrum (relying on smooth, shape-defining structures), texture-driven tasks rely on LFCs (e.g., background shading, camera effects) even though the discriminative information lies in high-frequency, fine-grained patterns.
• Persistence: This shortcut behavior is persistent across different model architectures (CNNs and Transformers), sizes, hyperparameters, and optimizers. It also affects pretrained vision foundation models (DinoV2) and language models (CLIP), though the severity varies.

2. Impact of Pruning LFCs on ID Performance

• Elimination of Shortcuts: Pruning LFCs from both training and test sets forces the model to rely on higher-frequency features.
• Accuracy Improvement: This intervention eliminates the shortcut and provides a more balanced spectral behavior, resulting in up to 8% improvement in ID accuracy for texture-driven tasks.
• Trade-off: For shape-driven tasks like CIFAR-10, pruning LFCs reduces ID accuracy because the model loses access to the smooth, low-frequency structures that define object shapes.

3. Impact on OOD Robustness

• Vulnerability to Low-Frequency Corruptions: Models exhibiting low-frequency shortcuts are highly vulnerable to OOD corruptions that modify low frequencies (e.g., fog). This leads to accuracy drops of up to 70% compared to ID performance.
• Improved Robustness via Pruning: Pruning LFCs significantly improves robustness to low-frequency corruptions (e.g., fog), yielding up to 40% higher OOD accuracy. By removing the reliance on the corrupted LFCs, the model becomes more robust.
• Trade-off with High-Frequency Corruptions: The improvement in robustness comes with a trade-off. As models are forced to rely more on high-frequency features (due to LFC pruning), they become more vulnerable to high-frequency corruptions (e.g., Gaussian blur). The final OOD accuracy depends on the interaction between the improved generalization from balanced spectral behavior and the increased sensitivity to high-frequency noise.

4. Model Architecture Differences

• ViTs vs. CNNs: Vision Transformers (ViTs) exhibit a stronger bias toward low-frequency features than CNNs. In domains with severe low-frequency noise (e.g., TextileNet with its white backgrounds and body shapes), ViTs suffer more severely from low-frequency shortcuts. Consequently, pruning LFCs sometimes decreases ID accuracy for ViTs on these specific tasks, whereas it consistently improves performance for CNNs.

Significance and Claims

The paper claims to identify and characterize a specific type of shortcut learning—low-frequency shortcuts—that is prevalent in texture-driven domains but distinct from the shape-based shortcuts observed in standard benchmarks.

• Novelty: The work extends frequency analysis beyond standard object-centric datasets to critical application domains where texture is the primary semantic cue.
• Mechanism: It demonstrates that simplicity bias causes models to latch onto a few simple, low-frequency features (backgrounds, lighting) rather than the complex, high-frequency textures required for the task.
• Solution: The authors propose frequency pruning (specifically of LFCs) as a simple, effective regularization technique. This method not only mitigates shortcut learning but also improves both ID accuracy and robustness to specific OOD corruptions in texture-driven domains.
• Implications: The findings suggest that current standard benchmarks may not fully capture the learning dynamics of real-world texture-driven applications and that spectral analysis is a crucial tool for diagnosing and improving model generalization in these domains.

The paper concludes that while pruning LFCs offers a path to better generalization and robustness, the optimal strategy depends on the specific interaction between the domain's spectral characteristics, the model architecture, and the nature of the OOD corruptions.