Technical Summary: Spectral Evolution Search (SES)

1. Problem Statement

Inference-time scaling offers a paradigm for aligning visual generative models with downstream objectives (e.g., aesthetic quality, human preference) without requiring parameter updates or retraining. However, existing methods face significant inefficiencies:

• Trajectory Optimization: Relies on intrusive interventions in denoising paths, often coupling with Stochastic Differential Equation (SDE) solvers, which hinders compatibility with efficient Ordinary Differential Equation (ODE) samplers or Flow Matching models.
• Gradient-Based Optimization: Requires differentiable reward functions, precluding the use of non-differentiable human preference metrics and susceptible to "reward hacking" via imperceptible high-frequency artifacts.
• Initial Noise Search: While treating inference as a black-box search over initial noise is broadly compatible, current approaches suffer from the curse of dimensionality. They typically employ an implicit isotropic parameterization that treats all search directions equally. This assumption leads to severe computational redundancy, as many search directions (particularly high-frequency perturbations) exert negligible influence on the final generation, resulting in diminishing returns.

2. Methodology: Spectral Evolution Search (SES)

The authors propose Spectral Evolution Search (SES), a plug-and-play framework for initial noise optimization that addresses these inefficiencies by exploiting a spectral bias in generative dynamics. The core insight is that model sensitivity to initial perturbations diminishes rapidly as frequency increases; low-frequency perturbations significantly reshape image structure, while high-frequency perturbations of equivalent energy have minimal visual impact.

SES operates through two primary modules:

A. Wavelet-Based Spectral Decoupling

To mitigate the curse of dimensionality, SES constrains the search space to a compact low-frequency manifold:

1. Decomposition: The initial noise $x_{init}$ is decomposed using a Discrete Wavelet Transform (DWT) into low-frequency coefficients ($c_{LL}$) and high-frequency details ($c_{H}$).
2. Freezing: High-frequency components are frozen as a static background.
3. Optimization Space: Optimization is restricted exclusively to the low-frequency vector $u \equiv c_{LL}$. This reduces the search dimensionality by a factor of $4^J$ (where $J$ is the decomposition level).
4. Reconstruction: Candidate low-frequency vectors are combined with the frozen high-frequency background and reconstructed via Inverse DWT (IDWT) to form valid initial noise for the sampler.

B. Cross-Entropy Optimization (CEM)

Within the reduced low-frequency subspace, SES employs a gradient-free evolutionary strategy:

• Initialization: A Gaussian distribution $p(u; \mu, \Sigma)$ is initialized to match the pre-trained diffusion prior ($\mu=0, \Sigma=I$).
• Sample-Evaluate-Update Loop:
  1. Sampling: Candidates are drawn from the current distribution.
  2. Evaluation: Candidates are reconstructed, denoised, and scored using a reward function $R$ (which can be non-differentiable).
  3. Update: The distribution parameters are updated using the statistics of the top-$K$ "elite" candidates, employing a smoothing factor $\gamma$ to prevent premature convergence.
• Final Generation: The final image is generated by sampling from the optimized distribution.

3. Theoretical Analysis

The paper derives the Spectral Scaling Prediction to theoretically substantiate the focus on low frequencies. By analyzing perturbation propagation dynamics in continuous-time generative flows (modeled as ODEs), the authors show that the cumulative gain $G(\omega)$ of an initial perturbation follows an inverse power-law scaling with respect to the frequency norm $\|\omega\|$:
$$G(\omega) \propto \|\omega\|^{-\beta/2}$$
This derivation, based on the assumption that the denoiser acts as a Wiener filter and natural images exhibit power-law spectral decay, confirms that the effective degrees of freedom are intrinsically sparse and concentrated in the low-frequency band. High-frequency perturbations exhibit negligible cumulative gain, rendering them ineffective control variables.

4. Experimental Results

Extensive experiments were conducted on mainstream models (SDXL, FLUX.1-dev, Qwen-Image) across diverse reward objectives (CLIP, PickScore, HPS, ImageReward, Aesthetic Score).

• Performance: Under a fixed computational budget (NRE = 200), SES consistently outperformed strong baselines, including Best-of-N (BoN), Zero-Order Search (ZO-N), and trajectory-based methods (SMC, SVDD, Demon). SES achieved the highest reward scores across all metrics and architectures.
• Scaling Behavior: While BoN exhibited logarithmic-like scaling that plateaued quickly, SES demonstrated a steep, continuous scaling trajectory up to NRE = 1000, indicating sustained performance growth without saturation.
• Ablation Studies:
  • Subspace Selection: Searching in the low-frequency subspace significantly outperformed full-frequency, high-frequency, and random subspaces of equivalent dimensionality, confirming that gains stem from the specific control authority of low frequencies, not just dimensionality reduction.
  • Optimization Strategy: The Cross-Entropy Method (CEM) substantially outperformed random search within the fixed subspace.
• Efficiency: A proxy-guided search mechanism (using LCM or reduced-step ODEs) reduced evaluation costs by ~85% while maintaining ranking consistency and high structural fidelity.
• Generalization: SES remained effective even when applied to models already aligned via training-time methods (DPO, SPO), demonstrating orthogonality to training paradigms. It also showed robustness across both deterministic ODE and stochastic SDE samplers.

5. Significance and Claims

The paper claims that SES significantly advances the Pareto frontier of generation quality versus computational cost. By identifying and exploiting the spectral bias inherent in generative dynamics, SES provides a theoretically grounded, model-agnostic solution that:

1. Eliminates Redundancy: Focuses computational budgets exclusively on the most influential degrees of freedom (low frequencies).
2. Ensures Compatibility: Works with non-differentiable rewards and various sampler types (ODE/SDE) without intrusive modifications to the generation process.
3. Prevents Reward Hacking: By restricting optimization to the low-frequency manifold, SES implicitly regularizes the search, preventing the exploitation of imperceptible high-frequency artifacts that often plague gradient-based methods.

The authors conclude that SES offers a versatile and efficient paradigm for inference-time scaling, enabling static models to adapt to specific objectives more effectively than existing search strategies.