On the Failure of Step-Response Tests to Certify Admissibility of Spectral Averaging Operators

1. Problem Statement

In signal processing and numerical analysis, linear smoothing operators (such as spectral truncation or low-pass filters) are frequently applied to signals with physical or semantic bounds, such as probabilities, normalized intensities, or concentrations in the interval $[0, 1]$. A critical requirement for such operators is admissibility: the property that if an input vector $x$ lies within $[0, 1]^N$, the output $Tx$ must also lie within $[0, 1]^N$.

Practitioners commonly use step-response tests as a heuristic diagnostic for admissibility. The logic is that if a filter applied to a step function (a transition from 0 to 1) produces no overshoot (values $>1$) or undershoot (values $<0$), the filter is assumed to be "safe" or "admissible."

The Core Problem: This paper demonstrates that step-response tests are fundamentally insufficient and can be misleadingly optimistic. Specifically, for periodic, translation-invariant operators defined by spectral truncation, a step response can appear perfect (zero overshoot) while the operator is provably non-admissible, capable of producing outputs far outside $[0, 1]$ for other valid binary inputs.

2. Methodology and Theoretical Framework

2.1 Setting and Definitions

• Domain: Discrete periodic signals $x \in \mathbb{R}^N$ (where $N$ is even).
• Operator: Linear time-invariant operators realized as circular convolution with a real kernel $h \in \mathbb{R}^N$.
• Normalization: The operator preserves constants (unit mass), meaning $\sum h[m] = 1$.
• Admissibility: An operator $T$ is admissible if $x \in [0, 1]^N \implies Tx \in [0, 1]^N$.

2.2 Theoretical Characterization (Theorem 3.1)

The paper establishes a necessary and sufficient condition for admissibility in this setting:

Theorem: A periodic convolution operator with unit mass is admissible on $[0, 1]^N$ if and only if its convolution kernel $h$ is entrywise nonnegative ($h[m] \ge 0$ for all $m$).

• Proof Logic:
  • Sufficiency: If $h \ge 0$ and $\sum h = 1$, the output is a convex combination of inputs, guaranteeing the output stays within $[0, 1]$.
  • Necessity: If any $h[m] < 0$, one can construct a binary witness input $x \in \{0, 1\}^N$ that isolates the negative coefficient, forcing the output below 0 (or above 1).

2.3 Constructive Witnesses

The paper provides a constructive method to generate "worst-case" inputs if admissibility fails:

• If $S_- = \sum_{h[m]<0} h[m] < 0$, there exists an input $x_{neg}$ (1 where $h$ is negative, 0 otherwise) such that the output is exactly $S_-$.
• Similarly, an input $x_{pos}$ (1 where $h$ is positive) yields an output of $1 - S_- = 1 + |S_-|$.
• This provides a certified lower bound on the magnitude of the violation: $|S_-|$.

2.4 Spectral Operators Analyzed

The authors analyze three standard spectral constructions parameterized by a cutoff $K$:

1. Fejér (Cesàro) Averaging: Uses a triangular window.
2. Sharp Spectral Truncation: A hard cutoff (Dirichlet kernel).
3. Signed Spectral Control: Keeps low frequencies, flips high frequencies to -1.

3. Key Findings: The "Near-Nyquist Blind Spot"

The most significant contribution is the identification of a blind spot in step-response testing, particularly near the Nyquist frequency ($K = N/2 - 1$).

3.1 The Phenomenon

At the cutoff $K = N/2 - 1$ (removing only the Nyquist bin):

• Fejér Averaging: Remains admissible (kernel is nonnegative).
• Sharp Truncation & Signed Control: Become provably non-admissible. Their kernels contain substantial negative mass ($S_- \approx -0.5$ for sharp, $S_- \approx -1.0$ for signed).
• The Paradox: Despite having large negative mass, the step response is perfect.
  • The canonical midpoint step function $s[n]$ (0 for $n < N/2$, 1 for $n \ge N/2$) has a Nyquist Fourier coefficient of exactly zero (when $N/2$ is even).
  • Since the operator only differs from the identity at the Nyquist frequency, and the step function has no energy at that frequency, the operator reproduces the step exactly: $T s = s$.
  • Result: The step test reports zero overshoot/undershoot, falsely suggesting the operator is admissible.

3.2 Quantitative Gap

The paper quantifies the disconnect between the step test and reality:

• Observed Step Violation: $\approx 0$ (machine precision).
• Certified Worst-Case Violation:
  • Sharp Truncation: Guaranteed to produce outputs $\le -0.4995$ and $\ge 1.4995$.
  • Signed Control: Guaranteed to produce outputs $\le -0.9990$ and $\ge 1.9990$.

This gap exists because the step function aggregates kernel coefficients over a contiguous block, causing positive and negative oscillatory lobes (Gibbs phenomenon) to cancel out. However, a binary witness input can align perfectly with the negative lobes to prevent cancellation, exposing the violation.

4. Key Contributions

1. Characterization Theorem: Proved that for periodic convolution with unit mass, admissibility is strictly equivalent to kernel nonnegativity.
2. Constructive Falsification: Developed a method to generate explicit binary inputs ($\{0, 1\}^N$) that maximize the violation of admissibility bounds, providing a certified lower bound ($|S_-|$) on the error.
3. Identification of the Blind Spot: Demonstrated that step-response testing is structurally incapable of detecting inadmissibility in specific spectral regimes (near Nyquist) where the test signal is orthogonal to the operator's error mode.
4. Reproducible Certification: Provided a deterministic, $O(N \log N)$ procedure (Inverse FFT + scan for negative entries) to certify admissibility, replacing unreliable heuristic step plots.

5. Results and Visualizations

The paper includes extensive numerical experiments (with $N=2048$) sweeping cutoffs $K$:

• Figure 1 & 3: Show a plot of "Observed Step Overshoot" vs. "Certified Worst-Case Violation."
  • For Fejér: Both metrics are 0.
  • For Sharp/Signed: The "Certified Violation" remains high (order 1) across all $K$, while the "Step Overshoot" drops to near zero as $K \to N/2 - 1$.
• Figure 2: Visualizes the kernels. Fejér is nonnegative; Sharp/Signed show oscillatory negative lobes.

6. Significance and Implications

• Safety in Critical Systems: In applications involving probabilities, concentrations, or physical intensities, relying on step-response plots can lead to catastrophic failures where the system produces physically impossible negative values or values $>1$ on untested inputs.
• Design Guidelines:
  • Avoid Sharp Truncation: If boundedness preservation is required, sharp spectral cutoffs (Dirichlet kernels) are inherently unsafe because they generate negative lobes.
  • Prefer Positive Kernels: Use constructions like Fejér (Cesàro) averaging which guarantee nonnegative kernels.
• Reporting Standards: The paper proposes a new standard for reporting filter properties. Instead of showing step plots, authors should report:
  1. The minimum kernel coefficient ($\min h$).
  2. The negative mass sum ($S_-$).
  3. The explicit witness input if $S_- < 0$.

Conclusion

The paper concludes that step-response tests are not certificates of admissibility. Admissibility is a property of the operator's real-space kernel (positivity), not its frequency-domain cutoff or its response to a single probe signal. The "near-Nyquist blind spot" serves as a definitive counterexample where a filter looks perfect on a standard test but is mathematically guaranteed to fail on binary inputs. The recommended practice is to verify kernel nonnegativity directly.