📊 statistics

Group invariance of $f$ -divergences and the Fisher--Rao distance

This paper demonstrates that $f$ -divergences and the Fisher--Rao distance are invariant under group actions in transformation models, thereby reducing their dependence on parameter pairs to maximal invariants such as double cosets, particularly within multidimensional location-scale families.

Original authors: Frank Nielsen, Kazuki Okamura

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Frank Nielsen, Kazuki Okamura

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to compare two different soups. One is a small bowl of spicy tomato soup, and the other is a giant pot of mild tomato soup.

If you just look at the bowls, they look totally different. But if you know the "recipe" (the statistical model), you realize that the only real differences are how much you stretched the pot (scale) and where you moved it on the counter (location). If you ignore the size of the pot and the counter's position, the soups are essentially the same "flavor profile."

This paper is about finding a mathematical way to ignore those irrelevant differences (like pot size and counter position) so we can measure the true difference between two distributions (like two soups) fairly.

Here is a breakdown of the paper's main ideas using simple analogies:

1. The Problem: Too Many Ways to Describe the Same Thing

In statistics, we often describe data using parameters (like a mean and a variance). But nature has "symmetries."

Translation (Location): If you move a whole dataset 5 steps to the right, the relationship between two points in that data shouldn't change.
Stretching (Scale): If you zoom in or out on a dataset, the shape of the relationship shouldn't change.

The authors ask: If we have a mathematical tool to measure the "distance" or "difference" between two distributions, does that tool respect these symmetries? In other words, if we stretch or move both distributions by the exact same amount, should the distance between them stay the same?

2. The Big Discovery: The "Group" Magic

The paper proves that a huge family of these distance-measuring tools, called f-divergences (which includes famous ones like Kullback-Leibler divergence), automatically respect these symmetries.

The Analogy: Imagine you have a rubber sheet with two dots drawn on it.

If you stretch the whole sheet (scale) or slide it across the table (location), the distance between the two dots on the sheet doesn't change.
The paper shows that for these specific statistical tools, the "distance" is like the measurement on the rubber sheet, not the measurement on the table. It is invariant.

3. The Solution: The "Maximal Invariant" (The Ultimate ID Card)

Just knowing that the distance doesn't change when we stretch or move things is good, but it's not enough. We want to know: What exactly determines the distance?

The authors introduce a concept called a Maximal Invariant. Think of this as an "Ultimate ID Card" for a pair of distributions.

If you have two pairs of distributions, and they have the same "Ultimate ID Card," then the distance between them is identical.
If they have different ID cards, the distance might be different.

This ID card strips away all the "noise" (the stretching and moving) and leaves only the relative information.

Relative Location: How far apart are the centers?
Relative Scale: How much bigger is one than the other?

4. The "Double Coset": A Geometric Shortcut

When the authors look at complex, multi-dimensional data (like 3D shapes instead of just 1D lines), they find that this "Ultimate ID Card" has a specific geometric shape called a Double Coset.

The Analogy: Imagine you are trying to describe the position of two people in a room relative to each other, but the room is rotating and the people are walking around.

Instead of tracking their absolute coordinates (which change constantly), you track their relative pose.
The "Double Coset" is the mathematical name for that relative pose. It tells you everything you need to know about how the two distributions are positioned relative to one another, ignoring the fact that the whole universe might be spinning.

5. Applying it to Real Models (Location-Scale Families)

The paper applies this to Location-Scale families (distributions defined by a center and a spread).

They show that for these models, the "Ultimate ID Card" is made of two things:
1. Singular Values: These are like the "stretch factors" in different directions.
2. Block Norms: These measure the "relative shift" of the centers, but only in the directions where the stretching happened.

This means you don't need to calculate a complex formula for every possible rotation or shift. You just need to calculate these specific numbers (singular values and block norms), and that is all the information you need to know the distance.

6. The Fisher-Rao Distance (The "Curved" Distance)

Finally, the paper looks at the Fisher-Rao distance, which is a special kind of distance used in information geometry. It's like measuring the distance between two points on a curved surface (a geodesic) rather than a flat map.

The authors prove that this curved distance also follows the same rules. Even though the math is harder, the result is the same: the distance between two distributions depends only on their relative position and scale (the "Ultimate ID Card"), not on where they are in the universe.

Summary

The paper is a guide to simplifying complex comparisons.

Symmetry exists: Moving or stretching data doesn't change the intrinsic difference between two distributions.
Tools respect symmetry: Common statistical distance tools (f-divergences) naturally ignore these moves.
The "ID Card": We can replace complex parameters with a simpler set of numbers (singular values and relative shifts) that capture the only thing that matters: the relative relationship between the two distributions.
Universal Application: This works for both standard distance measures and the more complex, curved Fisher-Rao distance.

In short: To compare two things fairly, ignore where they are and how big they are; look only at how they relate to each other. The paper gives the mathematical recipe for doing exactly that.

Technical Summary: Group Invariance of f-Divergences and the Fisher–Rao Distance

Problem Statement
Many statistical models possess natural symmetries described by group actions, where transformations of the sample space and parameter space should not alter the intrinsic comparison between two distributions. While it is intuitive that statistical divergences and distances should respect these symmetries, a rigorous characterization of how such invariance constrains the functional form of these measures is required. Specifically, the paper addresses how $f$ -divergences and the Fisher–Rao distance behave under transformation models where a group acts on both the sample and parameter spaces with a specific multiplier. The core problem is to reduce the dependence of these measures from the full parameter space to a minimal set of invariant variables (maximal invariants) that capture the relative position of two distributions.

Methodology
The authors employ the framework of transformation models and invariant statistics, drawing on classical results by Eaton and Wijsman. The methodology proceeds in three stages:

General Invariance Proof: The paper establishes a general transformation model where a topological group $G$ acts on a measure space $(X, \mathcal{B}, \lambda)$ and a parameter space $\Theta$ . The densities transform with a multiplier $\chi(g)$ such that $\chi(g)p(gx|g\theta) = p(x|\theta)$ . Under these conditions, the authors prove that any $f$ -divergence is invariant under the diagonal action of the group on pairs of parameters.
Maximal Invariant Reduction: The authors demonstrate that because the divergence is invariant, it must be a function of a maximal invariant of the pair of parameters. For transitive parameter spaces (homogeneous spaces $\Theta \cong G/H$ ), they characterize this maximal invariant using the double coset space $H \backslash G / H$ . This provides a geometric interpretation where the divergence depends solely on the relative position of two points in the homogeneous space.
Explicit Construction for Location-Scale Families: The theory is applied to multidimensional location-scale families. The authors construct explicit maximal invariants by modifying the parameter space to ensure identifiability (quotienting out orthogonal rotations). They utilize Singular Value Decomposition (SVD) to parameterize the double coset, reducing the divergence to a function of singular values and block norms of the transformed relative location.
Extension to Fisher–Rao Distance: The same invariance principles are applied to the Fisher–Rao distance (the geodesic distance of the Fisher metric). The authors show that the affine invariance of the Fisher metric allows for a similar reduction, mapping the distance between arbitrary distributions to the distance from a canonical base distribution.

Key Contributions and Results

General Invariance Theorem (Theorem 2.1): The paper proves that in a transformation model satisfying the multiplier condition, every $f$ -divergence $D_f(\theta_1 : \theta_2)$ is invariant under the group action: $D_f(g\theta_1 : g\theta_2) = D_f(\theta_1 : \theta_2)$ .
Maximal Invariant Characterization (Theorem 2.2 & 2.3): It is shown that any invariant divergence is a function of a maximal invariant $m(\theta_1, \theta_2)$ . For transitive actions where $\Theta = G/H$ , the maximal invariant is the double coset $H g_1^{-1} g_2 H$ , which completely describes the relative position of the parameters.
Canonical Form for Location-Scale Models (Section 3.1): For multidimensional location-scale models with radial base densities, the authors derive a specific maximal invariant. By identifying the parameter space with $R^d \times (GL(d, R)/O(d))$ $R^{d} \times (G L (d, R) / O (d))$ , the divergence depends only on:
- The singular values $\tau_1, \dots, \tau_k$ of the relative scale matrix $S = V_2^{-1}V_1$ and their multiplicities.
- The block norms $r_j$ of the transformed relative location vector $\nu = V_2^{-1}(\mu_1 - \mu_2)$ , where the blocks correspond to the multiplicities of the singular values.
Dual Group Actions and Bregman Divergences (Section 3.2): The paper analyzes centered scale families using dually flat geometry. It shows that while $f$ -divergence invariance arises from the transformation model, the invariance of dually flat Bregman/Fenchel-Young divergences arises from the cancellation of dual group actions. This unifies the spectral nature of these divergences (dependence on eigenvalues of $\Sigma_2^{-1}\Sigma_1$ ).
Fisher–Rao Reduction (Proposition 4.2): The Fisher–Rao distance for the location-scale family is shown to be invariant under the affine group. Consequently, the distance between two distributions is a function of the same maximal invariants (singular values and block norms) derived for $f$ -divergences. The problem is reduced to computing the distance from a canonical base point $(0, [Id])$.

Significance and Claims
The paper claims to provide a "general and geometric way to express invariant divergences" by utilizing the double coset structure of transitive parameter spaces. The primary significance lies in the reduction principle: instead of treating divergences as functions of all parameters, one can first remove group symmetry to express them through a smaller set of invariant variables.

The authors explicitly state that this approach:

Simplifies computation by reducing the dimensionality of the problem.
Identifies relevant quantities by determining exactly which invariant variables (e.g., singular values, block norms) can appear in any invariant formula.
Unifies different divergences: It shows that the same reduction applies to both $f$ -divergences and the Fisher–Rao distance.

The paper is modest regarding closed-form solutions. It notes that while the reduction principle identifies the correct variables, it "does not give a general closed-form formula" for the Fisher–Rao distance in dimensions $d > 1$ , as the geodesic boundary-value problem remains analytically difficult. However, it successfully establishes the structural dependencies required for any such formula. The work also clarifies the relationship between the affine-invariant Fisher–Rao geometry and the symmetric space geometry proposed by Lovrić et al., distinguishing the former as the specific geometry of the Gaussian location-scale family and the latter as a symmetric-space analogue.