🤖 machine learning

Spectral Gating via Damped Oscillations for Adaptive Implicit Neural Representations

This paper proposes a novel Implicit Neural Representation method that models neuron activations as damped harmonic oscillators to enable adaptive, curriculum-based spectral gating, allowing networks to naturally learn from coarse to fine details without requiring explicit regularization or task-specific hyperparameter tuning.

Original authors: Alex Costanzino, Pierluigi Zama Ramirez, Giuseppe Lisanti, Luigi Di Stefano

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alex Costanzino, Pierluigi Zama Ramirez, Giuseppe Lisanti, Luigi Di Stefano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to draw a picture or recreate a sound. To do this, the robot uses a "neural network," which is essentially a giant, complex machine made of many small switches (neurons). The way these switches turn on and off is determined by an activation function.

For a long time, researchers faced a frustrating dilemma with these switches, which they call the "Spectral Dilemma."

The Problem: The "All-or-Nothing" Switch

Think of the activation function as a radio tuner.

Type A (The Wild Radio): Some tuners are great at picking up high-pitched, detailed sounds (like the crisp snap of a drum or the fine lines in a drawing). However, they are too sensitive. They pick up everything, including static and noise. If you try to tune them to a song, they might accidentally memorize the background hiss as part of the music.
Type B (The Muffled Radio): Other tuners are very good at ignoring noise. They only let through the deep, smooth bass notes (the big shapes and smooth curves). But they are so cautious that they miss all the high-pitched details. The result is a blurry, low-quality image or a muffled sound.

Existing solutions tried to fix this by building a "smart switch" that could change its settings, but these were often fragile, required a human to constantly tweak the knobs, or were too slow to run.

The Solution: The "Damped Oscillator"

The authors of this paper, Alex Costanzino and his team, decided to look at physics for inspiration. They modeled each neuron not as a simple switch, but as a damped harmonic oscillator.

The Analogy: A Swing in the Wind
Imagine a child on a swing (the neuron).

The Forcing Frequency: This is the wind blowing on the swing (the input data).
The Natural Frequency: This is how fast the swing wants to move on its own.
The Damping: This is the friction (air resistance or a brake) that slows the swing down.

In their system, the "height" the swing reaches (the amplitude) depends on how well the wind matches the swing's natural rhythm and how much friction is applied.

If the wind blows at the exact right rhythm, the swing goes high (capturing the signal).
If the wind is random or chaotic (noise), the swing doesn't go very high because the friction stops it.

How It Works: The "Spectral Gate"

The magic of this paper is that the "friction" (damping) and the "rhythm" are learnable. The network teaches itself how much friction to apply.

Starting Small: At the beginning of training, the network is set up like a swing with heavy brakes. It can only move a little bit. This forces the network to focus on the big, easy-to-see shapes (low frequencies) first. It ignores the tiny, messy details.
The "Gate" Opens: As the network learns, it realizes, "Hey, there's a specific pattern in the data that matches my rhythm!" It then loosens the brakes (reduces damping) just enough to let that specific pattern through.
Rejecting Noise: If the data is just random noise (like static), it doesn't match the rhythm. The brakes stay on, and the noise is filtered out. The network refuses to memorize the garbage.

This creates a coarse-to-fine learning curriculum. The network naturally learns the big picture first, then slowly adds in the fine details, only when it's sure they are real and not just noise. It doesn't need a human to tell it when to stop or start; the physics of the "swing" does it automatically.

The Results

The authors tested this "swing" network (which they call FDHO) against many other methods on various tasks:

Drawing Images: It recreated photos with sharp edges and smooth gradients, often better than the competition.
Recreating Sound: It captured complex music and speech clearly without turning the background hiss into part of the song.
Fixing Noisy Data: When given a noisy image, it successfully ignored the noise and reconstructed the clean picture underneath, whereas other methods often got confused and memorized the noise.

The Best Part:
Unlike other methods that need a human expert to tune specific settings for every new task (like adjusting the radio for every different station), this method uses the same settings for everything. It adapts itself automatically. It's like a radio that automatically finds the perfect station and volume for any song you play, without you ever touching the dial.

In short, they replaced a fragile, hard-to-tune switch with a physics-based "swing" that naturally knows how to balance between seeing the big picture and spotting the tiny details, all while ignoring the static.

Technical Summary: Spectral Gating via Damped Oscillations for Adaptive Implicit Neural Representations

1. Problem Statement

Implicit Neural Representations (INRs) encode continuous signals (images, audio, shapes) into the weights of coordinate-based neural networks. A fundamental challenge in this domain is the spectral dilemma:

Periodic activations (e.g., SIREN) possess infinite bandwidth, enabling the representation of high-frequency details. However, they act as all-pass filters with no intrinsic regularisation, making them susceptible to memorising noise and suffering from unstable convergence (overshooting).
Spatially compact activations (e.g., Gaussian) provide inherent regularisation and smoothness but suffer from a strong low-frequency bias, limiting their ability to capture fine-grained or oscillatory content.
Existing attempts to resolve this trade-off (e.g., WIRE, FINER, Fourier Reparametrisation) often introduce computational overhead, require task-specific hyperparameter tuning, or lack a formal mechanism to adapt bandwidth robustly against noise.

The authors posit that no existing method provides a mechanism to adapt the network's spectral bandwidth dynamically during training to distinguish between coherent signal content and random noise without explicit regularisation.

2. Methodology: Forced Damped Harmonic Oscillator (FDHO)

The proposed method models each neuron's activation as the steady-state response of a sinusoidally-forced damped harmonic oscillator.

Activation Definition

Given learnable parameters $\theta = (\omega, \omega_n, \xi, \varphi)$ , where $\omega$ is the forcing frequency, $\omega_n$ is the natural frequency, and $\xi$ is the damping factor, the activation function $\sigma: \mathbb{R} \to \mathbb{R}$ is defined as:
$\sigma(z; \theta) = A(\omega, \omega_n, \xi) \sin(\omega z + \varphi)$
The amplitude $A$ is derived from the transfer function of a second-order linear system:
$A(\omega, \omega_n, \xi) = \frac{\omega_n^2}{\sqrt{(\omega_n^2 - \omega^2)^2 + (2\xi\omega_n\omega)^2}}$
Crucially, the amplitude is not a free parameter but is physically coupled to the frequencies and damping. The network jointly optimises the oscillator parameters $\theta$ alongside the standard weights $(W, b)$ .

Spectral Gating Mechanism

The core theoretical contribution is the derivation of implicit spectral regularisation through the gradient dynamics of the damping factor $\xi$ .

The Gradient Decomposition: The gradient of the reconstruction loss $\mathcal{L}$ $L$ with respect to $\xi$ $ξ$ decomposes into two competing terms:
1. Self-interaction term ( $T_{self}$ ): Always negative, pushing $\xi$ to increase (narrowing the bandwidth). This acts as a regularising force.
2. Signal term ( $T_{signal}$ ): Proportional to the correlation between the target signal and the carrier frequency. It is positive only when the target contains coherent energy at the carrier frequency, pushing $\xi$ to decrease (widening the bandwidth).
The Gating Condition: The spectral gate opens (allowing high-frequency content) if and only if the signal correlation exceeds a threshold determined by the self-interaction.
- Coherent Signal: The gate opens, allowing the network to fit the frequency.
- Random Noise: The correlation is near zero; the self-term dominates, keeping the gate closed and preventing noise memorisation.
- Mismatched Frequency: The gate remains closed due to orthogonality.

Coarse-to-Fine Learning Curriculum

The method is initialised with parameters placing the system in the "stop-band" (low amplitude, high damping). As training progresses and the loss landscape demands richer frequency content, the bandwidth naturally expands. This creates a coarse-to-fine learning curriculum without explicit scheduling: the network captures low-frequency structures first and admits high-frequency details only when justified by the reconstruction objective.

Multi-Layer Composition

In deep networks, cascading these layers produces a frequency-modulated signal. The effective modulation index at layer $\ell$ is $\beta_\ell = \omega_\ell w_\ell \prod_{k=1}^{\ell-1} A_k$ . Since amplitudes $A_k$ are learnable and depend on damping, the network gains explicit control over its harmonic bandwidth at each depth, extending the Bessel-function governed harmonics observed in SIREN to a learnable regime.

3. Key Contributions

Physics-Grounded Activation: A novel activation function derived from the steady-state response of a damped harmonic oscillator, introducing a physical coupling between amplitude and frequency.
Implicit Spectral Gating: A theoretical proof (Corollary 1) demonstrating that the gradient dynamics of the damping factor naturally distinguish between coherent signal and random noise, rejecting noise without explicit loss penalties or architectural changes.
Adaptive Bandwidth: A mechanism where the network autonomously adjusts its spectral selectivity during training, eliminating the need for task-specific hyperparameter tuning.
Unified Framework: A single architecture and parameter set that performs competitively across diverse tasks (signal fitting, denoising, inpainting, super-resolution, CT reconstruction) without modification.

4. Experimental Results

The authors evaluated FDHO against eight established INRs (SIREN, Gauss, WIRE, FINER, BACON, MFN, Fourier Features, FR) across multiple tasks.

Signal & Audio Fitting: FDHO achieved the highest final PSNR across all tested 1D signals and audio clips (Bach, Counting). Unlike SIREN, which often showed a large gap between peak and final performance (indicating instability/overshooting), FDHO demonstrated stable convergence.
Image Fitting: FDHO achieved state-of-the-art or competitive results on five diverse images, often outperforming SIREN and FINER by significant margins (e.g., up to 10 dB on the "Bikers" image). It eliminated the "overshoot-then-decay" behaviour typical of SIREN.
Denoising: In image denoising tasks (without explicit denoising losses), FDHO significantly outperformed all baselines. It successfully separated signal from noise purely through its inductive bias, whereas SIREN and others degraded to the level of low-frequency-biased methods.
CT Reconstruction: FDHO led in CT scan reconstruction from noisy, sparse projections, achieving nearly 3 dB higher PSNR than the second-best method (MFN), demonstrating robustness to both noise and ill-posedness.
Inpainting & Super-Resolution: FDHO remained competitive, matching or exceeding SIREN in inpainting and achieving top results in super-resolution, though it showed slightly lower performance on highly regular, repetitive textures where variable-periodic methods (FINER) excel.
SDF Fitting: FDHO performed competitively on 3D shape reconstruction, particularly on complex geometries where it outperformed SIREN.

Computational Cost: FDHO introduces a negligible parameter overhead (0.007% increase over SIREN) and a moderate training time increase (~1.4x SIREN), primarily due to the dual-optimizer setup and learning rate scheduling, but remains within the range of existing complex INRs.

5. Significance and Claims

The paper claims that FDHO resolves the spectral dilemma by embedding spectral control directly into the activation's differentiable structure. The primary significance lies in the robustness and adaptability of the method:

Noise Rejection: The method provides a formal mechanism for noise rejection that does not rely on fixed spectral ceilings or explicit regularisation terms.
Stability: By coupling amplitude to frequency via a physical transfer function, the method avoids the degenerate solutions and parameter drift observed in "adaptive SIREN" variants where parameters are decoupled.
Generalisation: The approach requires no task-specific tuning of hyperparameters, yet achieves state-of-the-art or competitive results across a wide spectrum of inverse problems and signal types.

The authors conclude that the interaction between activation design and training dynamics is central to INR performance, and that deriving activations from physical systems with provable spectral gating properties offers a principled path forward. They acknowledge that while the method excels in noisy and ill-posed settings, it may offer less advantage in scenarios with clean, spatially incomplete observations or highly regular textures where the spectral gate is less critical. Future work includes extending the formulation to per-neuron parameters and formalising multi-layer gradient dynamics further.