Dispersion of Gaussian Sources with Memory and an Extension to Abstract Sources

Technical Summary: Dispersion of Gaussian Sources with Memory and an Extension to Abstract Sources

Problem Statement
This paper addresses the finite blocklength lossy compression of information sources where components are independent but not necessarily identically distributed (non-i.i.d.). While classical rate-distortion theory characterizes the asymptotic limits (as blocklength $n \to \infty$), practical compression operates at finite blocklengths. The authors aim to quantify the operational limits under the constraint that the probability of exceeding a distortion level $d$ is at most $\epsilon$. Specifically, they seek to generalize the "Gaussian approximation" of the minimum achievable rate $R(n, d, \epsilon)$ to sources with memory, moving beyond the existing results for i.i.d. sources and scalar Gauss-Markov sources.

A central challenge is that standard typicality arguments relying on the empirical distribution fail when source components are non-identical. Furthermore, for Gaussian sources with memory, the covariance structure introduces dependencies that prevent direct application of i.i.d. dispersion results.

Methodology
The authors employ a second-order asymptotic analysis framework, building upon the non-asymptotic bounds established by Kostina and Verdú for i.i.d. sources. The methodology involves three primary technical components:

1. Decorrelation of Gaussian Sources: For Gaussian sources with memory, the authors utilize a standard orthogonal transformation to decorrelate the source vector into independent, non-identically distributed Gaussian components. This allows the problem to be mapped onto the framework of independent but non-identically distributed (i.n.i.d.) sources while preserving the squared error distortion metric.
2. Point-Mass Product Proxy Measure: A key technical novelty is the introduction of a "point-mass product proxy measure." Unlike previous works that use the empirical distribution of the observed sequence to construct typical sets (which fails for non-i.i.d. components), this proxy measure assigns probability mass to the specific sequence realization. This construction preserves the additivity of distortion and information densities across coordinates, enabling the application of the Berry–Esseen theorem to sums of independent, non-identical random variables.
3. Eigenvalue Asymptotics for Autoregressive Sources: To analyze stationary Gaussian autoregressive (AR) sources, the authors derive refined asymptotic results for the eigenvalues of the covariance matrix. They establish an $O(1/n)$ convergence rate for the $n$-th order rate-distortion function ($R_n(d)$) and source dispersion ($V_n(d)$) toward their limiting values, leveraging the properties of Toeplitz matrices and the limiting power spectrum.

Key Contributions and Results

• General Gaussian Approximation (Theorem 1): The paper proves that for sources with independent but non-identically distributed components, the minimum achievable rate satisfies:
  $$R(n, d, \epsilon) = R_n(d) + \sqrt{\frac{V_n(d)}{n}}Q^{-1}(\epsilon) + O\left(\frac{\log n}{n}\right)$$
  where $R_n(d)$ is the $n$-th order informational rate-distortion function and $V_n(d)$ is the source dispersion. This result generalizes the existing dispersion results for i.i.d. abstract sources.

• Extension to Gaussian Sources with Memory (Corollary 2): By applying the decorrelation technique, the authors provide explicit reverse water-filling expressions for $R_n(d)$ and $V_n(d)$ for Gaussian sources with memory. These expressions depend directly on the eigenvalues of the $n$-letter covariance matrix.

• Stationary Gaussian Autoregressive Sources (Corollary 3): For stationary finite-memory Gaussian AR sources, the authors show that the operational dispersion coincides with the limiting informational dispersion. They derive a second-order approximation where the first and second-order terms are expressed as integrals over the source power spectrum $S(\omega)$:
  $$R(n, d, \epsilon) = R(d) + \sqrt{\frac{V(d)}{n}}Q^{-1}(\epsilon) + O\left(\frac{\log n}{n}\right)$$
  This result sharpens the remainder term of the only previously known dispersion result for a source with memory (the scalar Gauss-Markov source) from $o(1/\sqrt{n})$ to $O(\log n/n)$.

• Numerical Analysis: The paper includes numerical comparisons of the spectrum, rate-distortion, and dispersion for various AR sources. It demonstrates that while sources with similar rate-distortion behaviors may have distinct dispersion characteristics, the dispersion is sensitive to the entire power spectrum, including inactive frequency components that do not contribute to the rate.

Significance and Claims
The paper claims to extend the second-order asymptotic analysis of rate-distortion theory beyond the i.i.d. setting and beyond the specific case of scalar Gauss-Markov sources. The significance lies in:

1. Generalization: Providing a unified framework for independent, non-identically distributed sources, which serves as a proxy for Gaussian sources with memory via decorrelation.
2. Technical Novelty: The introduction of the point-mass product proxy measure, which enables typicality analysis for non-identical components without relying on source-specific estimators (such as the maximum likelihood estimators used for Gauss-Markov sources in prior work).
3. Refinement: Improving the precision of finite blocklength bounds for stationary Gaussian AR sources by establishing a tighter remainder term and explicitly characterizing the convergence rate of the eigenvalues.

The authors note that their results rely on regularity assumptions, including a uniform lower bound on dispersion and bounded moments, which may not hold for non-stationary sources or sources with specific eigenvalue behaviors (e.g., exponentially small eigenvalues in non-stationary AR processes). The paper concludes by identifying the extension to non-stationary infinite-order sources and the zero-dispersion regime as directions for future work.