Input-Side Variance Suppression under Non-Normal Transient Amplification in Continuous-Control Reinforcement Learning

1. Problem Statement

Continuous-control Reinforcement Learning (RL) often produces controllers that, while nominally stable, exhibit large closed-loop variance, high-frequency control jitter, and sensitivity to disturbances during deployment.

• Existing View: Current explanations attribute these issues primarily to disturbance sources (e.g., action noise, exploration perturbations, policy nonsmoothness, or function approximation errors). Solutions typically focus on smoothing the source (e.g., action regularization).
• The Gap: This "source-only" perspective overlooks the role of the closed-loop dynamics as an amplifier. Even in nominally stable systems (spectral radius $\rho(A_{cl}) < 1$), non-normal state-transition matrices can cause transient amplification. This means small input perturbations can be geometrically amplified into disproportionately large state excursions before decaying.
• Core Question: Why do small input perturbations still produce large downstream state covariance in stable but non-normal closed loops, and can this be mitigated without redesigning the controller?

2. Methodology: The Source–Amplifier Decomposition

The paper proposes a Source–Amplifier decomposition framework to analyze execution-time closed-loop variance:

• Source: The origin of disturbances (action noise, exploration, policy jitter).
• Amplifier: The closed-loop geometry (specifically non-normality) that transiently amplifies these disturbances.
• Intervention: A Input-Side Variance Suppression Layer is introduced. This is a post-training module placed between the learned policy and the plant input.
  • Mechanism: It acts as a causal temporal smoother (first-order filter) that modifies the statistics of the applied input ($u_{app}$) without altering the structural closed-loop operator ($A_{cl}$) or the policy itself.
  • Goal: Reduce the variance of the injected disturbance ($W_{eff}$) entering the amplifier, thereby reducing the resulting state covariance ($\Sigma_x$) without changing the structural peak gain ($G_{peak}$).

Key Analytical Tools:

• Koopman/ALE Surrogates: Used strictly as analysis and certification tools to diagnose local propagation structures and nominal stability, not as the direct mechanism for performance improvement.
• Control-Theoretic Interventions: The paper employs two specific synthetic interventions to isolate mechanisms:
  1. CI-1 (Amplifier Isolation): Varies eigenvector geometry (non-normality) while holding eigenvalues, spectral radius, and disturbance statistics fixed.
  2. CI-2 (Source-Only Intervention): Holds the non-normal geometry fixed and varies only the applied-input statistics (white vs. filtered) to isolate the effect of the suppressor.

3. Key Contributions

1. Mechanistic Insight: Identifies non-normal transient amplification as a critical, under-emphasized contributor to execution-time variance in nominally stable RL controllers. It reframes variance reduction as a dual problem of managing both the source and the amplifier.
2. Novel Intervention: Proposes a simple, post-training input-side suppressor (a causal smoother) that reduces downstream state covariance by altering applied-input statistics, requiring no retraining or policy architecture changes.
3. Rigorous Isolation Evidence: Provides controlled experimental evidence (CI-1 and CI-2) that strictly separates amplifier-side effects from source-side effects, proving that variance reduction can occur without modifying the structural peak gain.
4. External Validation: Validates the mechanism on a planar quadrotor benchmark across multiple scenarios (hover, heavy payload, agile pitch, model mismatch), demonstrating consistency in nonlinear settings.

4. Key Results

The paper presents quantitative evidence from synthetic and physical simulations:

• CI-1 (Amplifier Effect):

  • With fixed spectral radius ($\rho=0.93$) and fixed disturbance, increasing the condition number of eigenvectors (non-normality, $\kappa(V)$) led to a strong linear correlation with state covariance ($r=0.970$) and peak transient gain ($r=0.996$).
  • Conclusion: Non-normality alone drives variance growth even in stable systems.

• CI-2 (Source Suppression Effect):

  • Under a fixed, strongly non-normal geometry, applying the suppressor reduced applied-input variance by 92.0% (from 0.0399 to 0.00317).
  • This resulted in a 21.8% reduction in state covariance trace (15.85 $\to$ 12.39) and a 25% reduction in expected peak state norm (7.42 $\to$ 5.56).
  • Crucially, the structural peak gain ($G_{peak}$) remained unchanged (5.55 $\to$ 5.55), proving the reduction came from source shaping, not amplifier redesign.

• CI-3 (Quadrotor Validation):

  • Across four scenarios and three random seeds, the suppressor consistently reduced action variance and state covariance.
  • At high disturbance levels (0.20), the aggregate results showed an 87.67% reduction in action variance and a 37.47% reduction in state covariance.
  • The results held true even under model mismatch (payload mismatch), confirming the mechanism's robustness.

5. Significance and Scope

• Theoretical Shift: The paper shifts the focus from purely "smoothing the policy" to understanding the interaction between input statistics and system geometry. It demonstrates that a stable system can still be "jittery" due to non-normality, and that suppressing the input variance is an effective countermeasure.
• Practical Utility: The proposed suppressor is post-training compatible. It can be deployed on existing RL policies without retraining, architecture modification, or controller resynthesis, making it highly practical for real-world robotics.
• Limitations & Future Work: The paper explicitly avoids head-to-head benchmarking against other smooth-control RL baselines (e.g., action regularization) to maintain focus on mechanism isolation. It does not claim to solve all RL variance issues but provides a specific, evidence-backed perspective on non-normal amplification.
• Role of Surrogates: A significant methodological contribution is the strict delineation of Koopman/ALE surrogates as diagnostic tools rather than performance drivers, ensuring the observed improvements are attributed to the physical input suppression mechanism.

In summary, this letter provides a control-theoretic explanation for why stable RL controllers jitter and offers a lightweight, mechanism-consistent solution that suppresses variance by managing the input statistics fed into non-normal amplifiers.