Technical Summary: FiRe (Frequency Reparameterization)

1. Problem Statement

Periodic Implicit Neural Representations (INRs), such as SIREN and FINER, are designed to encode continuous signals by replacing standard ReLU activations with sinusoidal functions. However, these architectures suffer from spectral bias, a phenomenon where gradient descent naturally fits low-frequency components of a signal first, while high-frequency details (fine textures, sharp edges) are recovered slowly.

The core limitation of existing periodic INRs is that they assign a single global frequency multiplier ($\omega_0$) to every neuron. This approach inefficiently allocates the representational budget because different parts of a signal (e.g., a sharp edge vs. a smooth background) require different local frequencies. While methods like FINER attempt to make frequency input-dependent by feeding pre-activation magnitudes back into the frequency envelope, they still rely on fixed global parameters or specific activation structures. The challenge is to make the per-neuron frequency learnable and input-dependent without simply increasing the global frequency multiplier (which changes the function class) or adding excessive parameters.

2. Methodology: FiRe (Frequency Reparameterization)

FiRe introduces a mechanism to reparameterize the frequency of periodic INRs by assigning a bounded, input-dependent frequency scale to each neuron.

Core Mechanism

Instead of folding the frequency into the main weight matrix, FiRe inserts a low-rank gating path that multiplies the pre-activation before the activation function.
For a standard SIREN activation $h = \sin(\omega_0(Wx + b))$, FiRe modifies this to:
$$h = \sin(\omega_0 \cdot s(x) \odot (Wx + b))$$
Where $s(x) \in (0, 2)$ is a learned gate implemented as a low-rank bottleneck:
$$s(x) = 2 \sigma(U_{\uparrow} U_{\downarrow} x + b_{\uparrow})$$
Here, $U_{\uparrow}$ and $U_{\downarrow}$ are low-rank matrices, and $\sigma$ is the logistic sigmoid. This structure ensures the frequency remains bounded and stable.

Initialization and Controls

• Initialization: The gate is initialized such that $U_{\uparrow} = 0$ and $b_{\uparrow} = 1$, resulting in a constant gate value $s_i \approx 1.46$ at the start. This ensures FiRe starts as a standard periodic network with a scaled frequency.
• Frequency Equalization ($w_{scale}$): To ensure performance gains are not merely due to a higher global frequency, the authors employ a weight-rescaling control. The weights of the baseline network are scaled by $1/1.46$ so that the realized per-neuron frequency of the baseline matches that of FiRe exactly at initialization.
• Parameter Matching: Baselines are widened to match the parameter count of FiRe (adding 1.4%–7.7% overhead depending on rank), ensuring gains are not due to increased capacity.

3. Theoretical Mechanism: NTK Decomposition

The paper provides a theoretical explanation based on the Neural Tangent Kernel (NTK).

• Additive Decomposition: Because the gate path parameters ($\theta_g$) and the carrier/signal path parameters ($\theta_c$) are disjoint, the empirical NTK ($K$) decomposes exactly into an additive sum:
  $$K = K_c + K_g$$
• Implicit Preconditioning: The gate path contributes a positive-semidefinite block ($K_g$) to the kernel. Even though the gate is functionally constant at initialization, its gradient with respect to its own parameters is non-zero. This adds new tangent directions to the optimization landscape without altering the forward map.
• Spectral Effects: By Weyl's inequality, adding $K_g$ raises the eigenvalues of the total kernel. This increases the effective rank of the tangent space (by 23–44% in experiments) and improves the conditioning of the optimization problem at initialization.
• Convergence Implication: In the "lazy" NTK regime, a kernel with higher eigenvalues and effective rank allows the residual error to decay faster along more eigendirections. This explains why FiRe accelerates early training convergence, specifically for high-frequency components which correspond to the slower-decaying eigenmodes.

4. Key Results

The authors evaluated FiRe on 2D image fitting tasks using DIV2K and Kodak datasets across various resolutions (256², 512², 1024²) and ranks.

• Convergence Acceleration: FiRe significantly outperforms parameter-matched baselines in the early stages of training (short budgets typical of INR fitting).
  • At 256² resolution, FiRe-SIREN and FiRe-FINER achieved +0.5 to +0.7 dB PSNR gains over baselines at 2000 epochs.
  • The advantage is most pronounced at low ranks (1–4) and diminishes as the rank increases (e.g., rank 8), suggesting the benefit comes from a coherent preconditioning mode rather than raw capacity.
• Resolution Dependence: The performance gain shrinks as image resolution increases. This is attributed to the fact that higher resolutions are further from convergence at a fixed budget; the "head start" provided by FiRe becomes a smaller fraction of the total optimization required.
• Spectral Localization: The gains are concentrated in the high-frequency band. Spectral analysis shows FiRe tracks the ground truth's high-frequency power more closely than the baseline, while low/mid frequencies are fitted equally well by both.
• No Asymptotic Advantage: Crucially, as training progresses to full convergence (e.g., 8000 epochs), the parameter-matched baseline eventually overtakes FiRe. This confirms that FiRe does not change the function class or the final solution, but rather acts as a preconditioner for the optimization trajectory.
• Stability: The realized frequency of FiRe remains matched to the baseline throughout training, confirming the gains are due to reparameterization, not a hidden frequency shift.

5. Significance and Claims

The paper positions FiRe as a method to accelerate optimization rather than to expand the representational capacity of the network.

• Mechanism Isolation: The authors emphasize that previous apparent gains in INR literature may have been confounded by uncontrolled increases in global frequency or parameter count. By strictly controlling for these factors, they isolate the optimization-geometry effect of the gate.
• Implicit Preconditioning: FiRe is identified as an implicit preconditioner that improves the conditioning of the NTK at initialization. This provides a theoretical bridge between reparameterization and faster convergence in the high-frequency regime.
• Practical Utility: The method is most valuable in scenarios where INRs are trained to a fixed, short budget (common in practice) rather than to full convergence. In these settings, FiRe yields higher quality reconstructions of fine details and textures.
• Modesty: The authors explicitly state that FiRe offers no advantage at full convergence and that the NTK-based explanation relies on the "lazy regime" approximation, which weakens as training progresses. The method is currently limited to 2D image fitting, and its extension to 3D (SDFs, NeRF) remains an open question.

In summary, FiRe demonstrates that reparameterizing the frequency of periodic neurons via a low-rank gate can significantly improve the conditioning of the optimization landscape, leading to faster convergence and better high-frequency reconstruction, provided the comparison is made against strictly frequency- and parameter-matched baselines.