Technical Summary: Fractional Stochastic Neural Networks

Problem Statement

Deep residual networks are increasingly interpreted through the lens of dynamical systems, where layer indices correspond to time steps. While Neural Ordinary Differential Equations (ODEs) and Neural Stochastic Differential Equations (SDEs) driven by standard Brownian motion have successfully modeled intrinsic uncertainty and epistemic uncertainty, they rely on the assumption of independent increments (Markov property). However, many physical and financial signals—such as rough volatility, geophysical records, and network traffic—exhibit long-range dependence and persistent or anti-persistent increments that standard Brownian motion cannot capture.

The primary challenge addressed in this paper is the lack of a rigorous training framework for stochastic neural networks driven by Fractional Brownian Motion (fBm). Unlike standard Brownian motion, fBm is neither a martingale nor a semimartingale, and its increments are correlated. Consequently, the standard backward stochastic differential equation (BSDE) adjoint recursions and the samplewise stochastic maximum principles used for Brownian systems cannot be directly transferred. The non-independence of increments means the conditional expectation of future noise depends on the entire observed history, complicating the derivation of unbiased gradient estimators for training.

Methodology

1. Discrete Formulation and Noise Decomposition

The authors formulate the Fractional Stochastic Neural Network (FSNN) as a controlled stochastic difference equation on a discrete depth grid $N$ with step size $h$. The forward dynamics are given by:
$$X_{n+1} = X_n + h b(n, X_n, u_n) + \sqrt{h} \sigma(n, X_n, u_n) \xi^H_n$$
where $\xi^H_n$ are normalized fractional Brownian increments.

To handle the non-martingale nature of fBm, the paper introduces a decomposition of the fractional increment $\xi^H_n$ into a predictable part (conditional on the past filtration $\mathcal{F}_n$) and a white noise innovation $\eta_n$:
$$\xi^H_n = \zeta^H_n + \beta(n, n)\eta_n$$
Here, $\zeta^H_n = \mathbb{E}[\xi^H_n | \mathcal{F}_n]$ is $\mathcal{F}_n$-measurable, and $\eta_n$ is an independent standard normal variable. The coefficients $\beta(n, k)$ are derived from the Cholesky factorization of the fBm covariance matrix. This decomposition allows the authors to separate the memory-dependent component from the fresh noise, enabling the construction of an adapted adjoint system.

2. Discrete Stochastic Maximum Principle (SMP)

The core theoretical contribution is the derivation of a Discrete Stochastic Maximum Principle for the FSNN.

• Adjoint Equation: The authors define an adapted backward stochastic difference equation (BS$\Delta$E) for the adjoint state $p_n$. The recursion incorporates both the predictable part of the noise ($\zeta^H_n$) and the innovation term ($\eta_n$), effectively absorbing the long-memory effects into the adjoint projections.
• Gradient Estimator: By differentiating the realized single-trajectory loss (pathwise reverse sweep), the authors derive a samplewise gradient estimator. They prove that for deterministic network parameters, this estimator is unbiased with respect to the true gradient of the expected cost functional. This avoids the need to solve the full adapted adjoint over the entire state space or evaluate Malliavin derivatives, which are computationally prohibitive.

3. Convergence Analysis

The paper establishes a mean square convergence theorem for Projected Stochastic Gradient Descent (SGD) applied to deterministic network parameters.

• The analysis accounts for the variance introduced by the fractional noise.
• A key finding is that the convergence rate depends on the Hurst exponent $H$ through a variance factor $V_H(N)$, which quantifies the one-step predictability of the fractional increments.
• For the specific case where the diffusion coefficient is independent of the control (i.e., $\sigma_u \equiv 0$), the authors prove a sharper convergence rate of $O(1/K)$, removing an explicit $h^{-1}$ variance term that appears in the fully trainable diffusion case.

Key Contributions

1. Theoretical Framework: The paper provides the first discrete stochastic maximum principle and corresponding adjoint recursion for neural networks driven by fractional Brownian motion, bridging the gap between rough path theory and deep learning training algorithms.
2. Unbiased Gradient Estimation: It demonstrates that a simple pathwise backpropagation (reverse sweep) on a single realized trajectory yields an unbiased gradient estimator for deterministic parameters, despite the non-Markovian nature of the driving noise.
3. Convergence Guarantees: The authors prove mean square convergence for projected SGD, explicitly characterizing how the Hurst exponent and network depth influence the error bounds.
4. Algorithmic Implementation: A practical training algorithm is proposed that requires only one forward and one backward sweep per iteration, utilizing precomputed Cholesky coefficients to reconstruct correlated noise from independent white noise innovations.

Experimental Results

The paper validates the framework through three sets of numerical experiments:

• Convergence and Uncertainty Quantification (E1):

  • In a closed-form Linear Quadratic (LQ) control problem, the measured control error decreases monotonically with the number of SGD iterations, consistent with the theoretical $O(1/K)$ bound for the control-independent diffusion case.
  • In high-dimensional noisy regression, the FSNN achieves test errors comparable to the label noise floor.
  • Uncertainty quantification tests show that the predictive bands are well-calibrated across different Hurst exponents, with rougher drivers ($H < 0.5$) producing wider, more conservative bands.

• Long Memory Time Series Generation (E2):

  • FSNNs trained with the target Hurst exponent significantly outperform Brownian SDE-Net and deterministic RNNs in recovering the scaling exponent (Hurst error) of synthetic fractional Ornstein–Uhlenbeck processes.
  • On real-world datasets (e.g., SPX, Nile, Ethernet traffic), FSNNs demonstrate superior performance in capturing long-lag autocorrelation structures compared to Brownian baselines, though marginal distributional matching remains task-dependent.

• Robustness to Perturbations (E3):

  • On image classification tasks (MNIST, Fashion-MNIST, CIFAR-10), FSNNs do not uniformly outperform deterministic CNNs on clean data.
  • However, under structured perturbations (additive fBm random fields) and common corruptions (fog, frost, blur), stochastic backbones generally improve robustness. The optimal Hurst exponent varies by dataset and corruption type (e.g., $H=0.8$ for MNIST under fBm fields, $H=0.6$ for CIFAR-10 common corruptions).

Significance and Claims

The paper claims that fractional drivers provide a principled mechanism for improving long memory recovery and robustness in stochastic neural networks relative to standard Brownian and deterministic baselines.

• Theoretical Significance: The work extends the applicability of stochastic optimal control and maximum principles to non-semimartingale settings, offering a tractable discrete-time framework that avoids the complexities of continuous-time Malliavin calculus.
• Practical Significance: The results identify specific settings where fBm-driven networks are advantageous: specifically, in modeling time series with long-range dependence and in enhancing model robustness against structured, correlated noise.
• Modesty of Claims: The authors explicitly note that the benefits are not uniform across all tasks. For instance, on saturated datasets like MNIST, the deterministic backbone remains dominant for clean accuracy, and the stochastic advantage is primarily observed under specific perturbation regimes. The paper frames the FSNN as a specialized tool for problems where the "Brownian template" is insufficient, rather than a universal replacement for existing architectures.