💻 computer science

From Infrared to Ultraviolet]{From Infrared to Ultraviolet: A Renormalization Group Approach to Frequency-Aware Progressive Training

This paper proposes a frequency-aware progressive training strategy inspired by the renormalization group that sequentially activates low-to-high frequency branches with decaying learning rates, demonstrating consistent improvements in low-dimensional, few-shot regression tasks while clarifying that its efficacy diminishes in high-dimensional or smooth-process scenarios where scale separation is less relevant.

Original authors: 英熙朱

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: 英熙朱

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world. Usually, we throw all the data at it at once: the big shapes, the tiny details, the smooth curves, and the jagged edges, all mixed together in one giant smoothie. The robot tries to learn everything simultaneously. But what if the robot gets confused? What if the tiny, noisy details distract it from learning the big picture first? This is the problem of "data scarcity"—when you don't have enough examples for the robot to figure out both the big picture and the small details at the same time.

To solve this, scientists often look to physics for inspiration. One famous idea from physics is called the "Renormalization Group" (RG). Think of RG like looking at a map. First, you look at a map of the whole country to see the major highways and mountain ranges (the big picture). Only after you understand the country do you zoom in to look at the specific streets in a single town (the fine details). If you tried to memorize every single street address before you even knew which country you were in, you'd get lost. This paper asks: Can we teach artificial intelligence the same way? Can we force the computer to learn the "big picture" (low-frequency patterns) first, and only then let it worry about the "fine details" (high-frequency patterns), just like a physicist zooming in from a map of a continent to a street view?

The authors of this paper, led by Yingxi Zhu, say "yes, but with a catch." They developed a new training method called "Frequency-Aware Progressive Training." Instead of letting the AI learn everything at once, they split the AI's brain into different "frequency branches." They turn on the "low-frequency" branch first, letting it learn the smooth, broad shapes of the data. Once that branch is good at the basics, they "unfreeze" the "mid-frequency" branch to add some texture, and finally the "high-frequency" branch to add the sharp, tiny details. Crucially, as they add these new branches, they slow down the learning speed of the older branches so the new, noisy details don't mess up the solid foundation the AI already built.

The results are fascinating, but they aren't a magic bullet for every problem. In tests where the AI had to guess a function or restore a blurry image with very few examples, this "coarse-to-fine" method worked wonders. For instance, when trying to predict a wavy line with only 10 data points, the new method was about ten times more accurate than the standard way of training. It was like the AI finally understood the shape of the wave before trying to guess the tiny ripples on top of it.

However, the paper also draws a very clear line in the sand about where this trick doesn't work. If the task is to tell the difference between two very similar-looking objects (like distinguishing a cat from a dog), or if the data is so complex that the "big picture" doesn't actually help much, this method can actually make things worse. In fact, when the researchers tried to teach the AI to recognize different types of images (classification) using this method, it performed worse than the standard approach. The authors found that for this "infrared-to-ultraviolet" strategy to succeed, two things must be true: the AI must be able to strictly separate the "big" signals from the "small" signals (using special math tools like wavelets), and the "big picture" must actually contain useful information for the specific task.

So, the paper suggests that while we can't just apply this physics trick to every AI problem, we have found a powerful new tool for specific situations—especially when data is scarce and the goal is to reconstruct or predict smooth, structured patterns. It's a reminder that sometimes, to learn the details, you really do have to learn the big picture first.

Technical Summary: From Infrared to Ultraviolet: A Renormalization Group Approach to Frequency-Aware Progressive Training

1. Problem Statement

Standard end-to-end training of neural networks often struggles to exploit the multi-scale structure of real-world data, particularly in regimes characterized by data scarcity (few-shot learning). In such scenarios, low-frequency, global patterns are typically learnable from very few examples, whereas high-frequency details require significantly more data to be pinned down accurately. When all frequency components are forced to evolve simultaneously, high-frequency noise can disrupt the early-stage learning of coarse, low-frequency structures. Consequently, the model may fail to establish a stable foundational representation necessary for subsequent refinement, leading to poor generalization and extrapolation capabilities.

2. Methodology

The authors propose Frequency-Aware Progressive Training, a strategy inspired by the Renormalization Group (RG) framework from statistical physics (Wilson, 1971; Wilson and Kogut, 1974). The core philosophy is to train neural networks "from infrared to ultraviolet"—learning the coarse, low-frequency structure first, and progressively refining the fine, high-frequency details.

The method consists of three primary components:

A. Frequency-Decomposed Architecture

The network is explicitly split into frequency branches ( $f_{IR}, f_{mid}, f_{UV}$ ), each processing a specific frequency band.

Decomposition Operators: The input signal is separated using operators such as Discrete Wavelet Transforms (DWT) for images, or Fourier basis functions for synthetic 1D/2D data.
Strict Separation: The authors emphasize the necessity of "strict" frequency separation (e.g., via orthogonal Fourier bases or FFT masking) to ensure functional decoupling. Approximate methods like spatial downsampling are shown to cause functional overlap between branches, degrading performance.
Aggregation: For regression, outputs are summed; for classification, features are concatenated.

B. Sequential Branch Activation

Training proceeds in $K+1$ phases (where $K$ is the number of frequency branches):

Phase 0 (Infrared): Only the low-frequency branch ( $f_{IR}$ ) is active and trained. High-frequency branches are frozen.
Phase $k$ : The next frequency branch is unfrozen. The previously trained branches remain active but their learning rates are decayed.
Fine-tuning: A final phase with a uniform, small learning rate allows all scales to coordinate.

C. Layer-wise Learning Rate Decay

To protect the coarse representations learned in early phases, the learning rates of already-trained branches are exponentially decayed as new, higher-frequency branches are introduced.

Mechanism: If $\eta_0$ is the base learning rate and $\gamma \in (0,1)$ is the decay factor, the learning rate for a branch trained in phase $j$ during phase $k$ ( $j < k$ ) is set to $\eta_0 \cdot \gamma^{k-j}$ .
Rationale: This mimics the RG concept where low-energy parameters are largely insensitive to high-energy fluctuations, effectively holding the coarse structure in a "trust region."

3. Key Contributions and Results

Empirical Findings

The paper systematically evaluates the method across synthetic function approximation, physical simulations (Burgers' and Heat equations), image restoration (denoising, dehazing, inpainting), and image classification.

Success Regimes: The method consistently outperforms end-to-end (E2E) baselines in low-dimensional, few-shot regression tasks with clear frequency separation.
- Example: In 1D function extrapolation with only 10 data points, the method reduced MSE by an order of magnitude compared to E2E (0.0118 vs. 0.0968).
- Example: In MNIST denoising with extreme scarcity, it achieved a 0.9 dB PSNR improvement over E2E.
Failure Regimes: The advantage shrinks or reverses in:
- High-dimensional sparse settings: Where coarse priors are insufficiently sampled.
- Discriminative tasks: In image classification (e.g., CIFAR-10/100 few-shot), the method underperforms E2E because low-frequency structures lack the discriminative information required for class separation.
- Physically smooth processes: In the heat equation, where high frequencies are naturally dissipated, the coarse-first constraint becomes an unnecessary hindrance.
- Approximate Separation: When using spatial downsampling instead of strict spectral separation, the method often performs worse than E2E.

Comparative Analysis

vs. Reverse Training: Training from Ultraviolet to Infrared consistently underperforms, confirming the irreversibility and necessity of the coarse-to-fine order.
vs. Curriculum Learning (CL): While CL (ordering data by difficulty) helps in some tasks, the structural prior of frequency-aware training is superior in others (e.g., denoising). A hybrid approach showed mixed results, suggesting structural and data priors are not always synergistic.
vs. Multirate Training (MRT): The method significantly outperforms MRT. Since both use layer-wise learning rate adjustments, this gap isolates the specific benefit of explicit frequency decomposition and sequential activation over mere rate scheduling.

Theoretical Insights

The paper posits that the method works via Scale Decoupling. By isolating frequency bands, the conditional entropy of high-frequency components given low-frequency ones is reduced. Furthermore, the method acts as a structural regularizer, limiting the hypothesis space in early phases to prevent overfitting to noise, analogous to the Information Bottleneck principle.

4. Significance and Claims

The authors position this work not as a universal solution, but as a precise delineation of boundary conditions for scale-separated training.

Actionable Guidance: The primary contribution is mapping exactly when a coarse-to-fine prior pays off (strict separation + task-relevant coarse structure) and when it does not (discriminative tasks, smooth dynamics, approximate separation).
Physics-ML Dialogue: The work demonstrates how a conceptual principle from the Wilsonian Renormalization Group (integrating out fast modes before refining slow modes) can be translated into a concrete, practical training algorithm without requiring explicit physical governing equations.
Practical Implications: For practitioners dealing with data-scarce, multi-scale problems, the paper advises that investing in precise frequency decomposition (e.g., wavelets, Fourier) yields higher returns than coarse pooling, and that the learning rate decay factor ( $\gamma$ ) must be tuned based on the fragility of the task's coarse structure.

The paper concludes that while the method is not a panacea, it offers a principled, physics-motivated approach to optimization that is highly effective within its specific, well-defined regime.