Technical Summary: Robust Score-Based Generative Modelling with Tsallis–Gaussian Perturbations

Problem Statement

Score-based generative modelling, particularly Score-Matching with Langevin Dynamics (SMLD), relies heavily on Gaussian perturbations to define the training objective and sampling dynamics. While Gaussian perturbations offer analytical convenience and a direct connection to Fisher divergence, they possess a critical limitation: the conditional score function is linear and unbounded with respect to the perturbation residual ($r = y - x$). Consequently, the standard Denoising Score-Matching (DSM) objective is highly sensitive to large deviations, making it non-robust against heavy-tailed data distributions and data contamination (e.g., feature-level noise or sample-level corruption).

Existing attempts to address this by simply substituting Gaussian noise with heavy-tailed alternatives (e.g., stable or fractional dynamics) often fail to provide a statistically coherent framework. They typically do not align the perturbation distribution with a corresponding divergence, loss function, or sampling procedure, leaving the robustness properties of the training target undefined. Furthermore, financial data and synthetic tabular data often exhibit heavy tails, volatility clustering, and structural breaks, necessitating generative models that can capture extreme events without being destabilized by them.

Methodology

The authors propose a unified framework, Tsallis-Weighted SMLD (TSMLD), which integrates Tsallis–Gaussian (or $q$-Gaussian) perturbations with the Tsallis Kullback–Leibler (KL) divergence. The framework is built on two complementary robustness mechanisms:

1. Tsallis–Gaussian Perturbation Kernel (Feature-Level Robustness)

Instead of standard Gaussian noise, the model injects Tsallis–Gaussian noise. For a deformation parameter $q > 1$, this distribution is heavy-tailed.

• Bounded Score: The conditional score induced by the Tsallis–Gaussian kernel, $\nabla_y \log p_j(y|x)$, is bounded and redescending in the residual $r$. Unlike the linear Gaussian score, the Tsallis score decreases toward zero for large residuals.
• Effect: This limits the influence of extreme feature-level deviations during training, preventing outliers from dominating the gradient updates.

2. Tsallis Weighting via Tsallis KL Divergence (Sample-Level Robustness)

The training objective is derived from the first-order variation of the Tsallis KL divergence, leading to a weighted explicit score-matching (ESM) loss:
$$\mathcal{L} \propto \mathbb{E} \left[ \omega_{q,j}(Y) \| \nabla_Y \log p_\theta(Y) - \nabla_Y \log p_j(Y) \|_2^2 \right]$$
where the weight is defined as $\omega_{q,j}(y) = (p_j(y) / p_\theta(y))^{1-q}$.

• Mechanism: When $q > 1$, observations where the model assigns low probability relative to the data (low density ratio) receive reduced weights.
• Effect: This downweights poorly supported or contaminated samples at the sample level, enhancing robustness against distributional shifts and outliers.

Implementation: Dual-Head Architecture

Since the theoretical weights depend on unknown density ratios, the authors propose a practical implementation using a dual-head neural network based on Noise Conditional Score Networks (NCSN):

1. Primary Head: Estimates the ordinary score $\nabla \log p_\theta$ for sampling.
2. Auxiliary Head: Estimates a scalar or vector transformation to approximate the density-power factor $p_\theta^{1-q}$, enabling the computation of the Tsallis weight without explicit density estimation.

• Stabilization: The implementation employs log-domain mini-batch normalization, a warm-up schedule for weights, and a stop-gradient operation to prevent the network from optimizing the weights to minimize loss rather than the score error.

Sampling

Sampling is performed via an annealed Tsallis-noise Langevin dynamics. The sampler traverses a noise ladder $\sigma_1 > \dots > \sigma_J$, using the learned ordinary score for drift and Tsallis–Gaussian noise for the stochastic term. This allows for broader exploration of the state space, particularly at high noise levels, compared to standard Gaussian Langevin dynamics.

Key Contributions

1. Unified Framework: The paper formulates the first score-based sampling framework that jointly aligns a heavy-tailed perturbation family (Tsallis–Gaussian) with a corresponding divergence (Tsallis KL) and a weighted objective function.
2. Dual Robustness Mechanisms: It establishes theoretically and empirically that robustness is achieved through:
   • Feature-level robustness: A bounded, redescending conditional score that dampens extreme coordinate-wise deviations.
   • Sample-level robustness: Adaptive density-ratio weighting that attenuates the influence of poorly matched observations.
3. Practical Neural Architecture: A dual-head implementation that approximates Tsallis weights in high dimensions without explicit density estimation, compatible with existing SMLD pipelines.
4. Empirical Validation: Comprehensive experiments on synthetic, financial, and image datasets demonstrating improved robustness under contamination and heavy-tailed regimes while maintaining competitive generative performance.

Results

Synthetic and Tabular Data

• Performance: On classification (HAR, ISOLET, Spambase) and regression (Superconductivity, Communities and Crime, Wave Energy) tasks, TSMLD (with moderate $q \approx 1.2$) consistently outperforms standard SMLD and other baselines (CTGAN, TVAE, TabDDPM) in downstream utility (TSTR) and distributional fidelity (MMD).
• Robustness: Under feature-level and sample-level corruption (up to 50% noise), TSMLD exhibits significantly higher stability than standard SMLD and tabular generators. It preserves task-relevant structure better, showing smaller performance degradation in accuracy and regression metrics.

Image Generation

• Quality: On MNIST, Fashion-MNIST, and CIFAR-10, TSMLD achieves competitive or superior Fréchet Inception Distance (FID) scores compared to standard SMLD and other heavy-tailed diffusion models (LIM, DLPM).
• Robustness: Under pixel-level and image-level corruption, TSMLD demonstrates superior resilience. Generated images contain fewer artifacts and maintain semantic coherence better than baselines when trained on contaminated data.

Financial Time Series

• Stylised Facts: TSMLD better recovers financial stylised facts, including tail heaviness (kurtosis), downside risk (VaR, CVaR), and market-mode dependence (eigenvalues of correlation matrices), compared to GAN-based (TimeGAN, TailGAN) and diffusion-based baselines.
• Stress Regimes: In out-of-distribution stress diagnostics (2020–2024 crisis period), TSMLD with $q=1.5$ generates scenarios with realistic tail behavior and crisis-period dynamics, whereas Gaussian SMLD and GANs tend to underestimate tail risk.
• Ablation Findings:
  • Tsallis Perturbation: The heavy-tailed perturbation is the primary driver for recovering stress-tail behavior.
  • Tsallis Weight: The adaptive weighting is crucial for calibrating the model in non-crisis regimes, preventing the over-amplification of tails in calm periods.
  • Synergy: Neither component alone is sufficient; the full TSMLD model combines heavy-tailed exploration with adaptive sample-level calibration.

Significance and Claims

The paper claims that the proposed framework offers a statistically aligned approach to robust generative modelling. It argues that simply changing the noise distribution is insufficient; a coherent framework must align the perturbation, divergence, and sampling dynamics.

The authors position their work as a significant step toward generative models that are robust to heavy-tailed data and contamination, which are prevalent in financial applications and synthetic data generation. They emphasize that the method provides a "favourable trade-off" between distributional fidelity and downstream utility, particularly in high-corruption regimes where standard methods fail.

The paper is modest in its claims regarding the theoretical scope:

• It acknowledges that the convergence of the Tsallis-noise Langevin sampler is currently proven only for the finite-variance regime ($1 < q < 5/3$).
• It notes that the terms "feature-level robustness" and "sample-level robustness" are descriptive labels for the observed effects rather than direct analogues of classical influence-function-based robustness, as the formulation of influence functions within score-based generative modelling remains an open question.
• It suggests that future work is needed to develop data-adaptive rules for selecting $q$ and to extend the framework to infinite-variance regimes through regularization.

In summary, the paper presents TSMLD as a robust, unified alternative to Gaussian SMLD, particularly effective for domains where extreme events and data contamination are critical concerns.