📊 statistics

Supervised Quadratic Feature Analysis: Information Geometry Approach for Dimensionality Reduction

This paper introduces Supervised Quadratic Feature Analysis (SQFA), a dimensionality reduction method that leverages information geometry to learn linear features maximizing Fisher-Rao distances between class-conditional distributions, demonstrating competitive or superior classification performance compared to state-of-the-art methods.

Original authors: Daniel Herrera-Esposito, Johannes Burge

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniel Herrera-Esposito, Johannes Burge

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to tell the difference between a cat and a dog. You have thousands of photos, but each photo has millions of tiny details (pixels). If you try to look at every single pixel, the computer gets overwhelmed and confused. This is where dimensionality reduction comes in. It's like taking a huge, messy room full of clutter and finding just the few essential items that prove whether it's a "cat room" or a "dog room."

The paper introduces a new tool called SQFA (Supervised Quadratic Feature Analysis) to help with this sorting task. Here is how it works, explained simply:

1. The Problem: It's Not Just About the "Average"

Most old-school methods for sorting data (like LDA) act like they are looking for the average difference.

Analogy: Imagine two groups of people. Group A is tall, and Group B is short. An old method would just look for the height difference.
The Catch: What if Group A and Group B are the same average height, but Group A is very consistent (everyone is exactly 5'10") while Group B is all over the place (some are 4'10", some are 6'5")? The "average" looks the same, but the spread or variability is totally different.

The paper argues that to sort things well, you often need to look at this variability (the "spread" or "shape" of the data), not just the average.

2. The Solution: A New Way to Measure "Distance"

To find the best way to sort the data, SQFA uses a concept from a field called Information Geometry.

The Metaphor: Imagine every possible group of data (like "all cat photos") is a point floating in a giant, curved universe.
The Goal: We want to push the "cat point" as far away as possible from the "dog point" in this universe.
The Ruler: Most methods use a standard ruler (like a straight line). SQFA uses a special, curved ruler called the Fisher-Rao distance. This ruler is smart; it knows that the "shape" of the data matters just as much as where the data is located. It measures the distance by walking along the curved surface of the universe, taking into account how "spread out" the data is.

3. How SQFA Works (The "SQFA" Magic)

SQFA learns a set of filters (think of them as special lenses) to look at the data.

It looks at the data through these lenses.
It calculates the "distance" between the classes using that special curved ruler (Fisher-Rao).
It adjusts the lenses until the distance between the classes is as huge as possible.

By maximizing this specific distance, SQFA finds the best way to squish the high-dimensional data down into a few simple numbers that still keep the classes perfectly separated.

4. The Surprising Winner: The "Hellinger" Variant

The authors tested a few different versions of their special ruler.

They found that while the Fisher-Rao distance is great, a specific variation of it called the Hellinger distance (which they call SQFA-H) was the absolute champion.
The Result: In tests with real-world data (like recognizing handwritten numbers or analyzing brain signals), SQFA-H consistently beat other popular methods. It was better at helping computers classify things correctly than the standard "average-finding" methods.

5. Real-World Tests

The authors didn't just do math on paper; they tested SQFA on real things:

Street View House Numbers: Recognizing numbers in photos. SQFA was great at this, even when the numbers had weird backgrounds.
MNIST (Handwritten Digits): Another classic test. SQFA-H was the top performer.
Speed Estimation: They tested it on videos of moving textures (like a road moving past a car). SQFA learned filters that looked very similar to how real animal eyes and brains detect motion.
Brain Data: They used it on recordings from monkey brains. It successfully found the hidden patterns in the noisy brain signals that told the difference between different visual stimuli.

Summary

Think of SQFA as a new, smarter way to organize a messy closet.

Old methods just look at the average height of the clothes.
SQFA looks at the shape and spread of the clothes, using a special curved map to ensure that "shirts" and "pants" are pushed as far apart as possible in the closet.
The result? A cleaner closet where it's much easier to find what you need, and in this case, a computer that is much better at telling different things apart.

The paper concludes that using this "curved map" (Information Geometry) is a powerful, underused tool that can make machine learning smarter and more efficient, especially when the differences between groups are about their variability rather than just their average.

Technical Summary: Supervised Quadratic Feature Analysis (SQFA)

Problem Statement
Supervised dimensionality reduction aims to map high-dimensional labeled data into a lower-dimensional feature space that maximizes class separation. While nonlinear methods exist, linear methods (learning filters $F$ such that $z = F^T x$ ) remain essential for their simplicity, interpretability, and data efficiency. Traditional linear approaches, such as Linear Discriminant Analysis (LDA), typically maximize specific statistical dissimilarity measures between class-conditional distributions, such as the squared Mahalanobis distance, Bhattacharyya distance, or Kullback-Leibler (KL) divergence. However, these measures often rely on specific assumptions (e.g., homoscedasticity for LDA) or may not fully capture the geometric structure of the probability distributions.

Methodology
The authors introduce Supervised Quadratic Feature Analysis (SQFA), a supervised dimensionality reduction method that learns linear features by maximizing the Fisher-Rao distance between class-conditional distributions under Gaussian assumptions.

Information Geometry Framework: SQFA treats probability distributions as points on a statistical manifold. It utilizes the Fisher information metric to define the geodesic distance between distributions, known as the Fisher-Rao distance. This distance is conceptualized as the accumulated local discriminability along the geodesic connecting two distributions.
Optimization Objective: The goal is to find filters $F$ that maximize the sum of pairwise Fisher-Rao distances between class-conditional distributions in the projected space.
Handling Computational Constraints: Since closed-form expressions for the Fisher-Rao distance between arbitrary multivariate Gaussians are unavailable, SQFA employs two strategies:
1. Zero-Mean Case (smSQFA): For zero-mean Gaussians, the manifold is equivalent to the manifold of Symmetric Positive Definite (SPD) matrices. The Fisher-Rao distance is proportional to the affine-invariant distance, which has a closed-form solution based on generalized eigenvalues.
2. General Case (SQFA): For arbitrary Gaussians, the authors utilize the Calvo-Oller bound, a closed-form lower bound derived by embedding the distribution parameters (mean and covariance) into an SPD matrix of dimension $m+1$ . This bound serves as a practical surrogate for the Fisher-Rao distance.
Variants: The optimization framework is flexible, allowing for the maximization of other dissimilarity measures, including the Hellinger distance (SQFA-H), Bhattacharyya distance (SQFA-B), Jeffreys divergence (SQFA-J), and Wasserstein distance (SQFA-W).
Implementation: Filters are optimized using L-BFGS. The method includes regularization to prevent rank-deficient covariances and handles the non-convexity of the objective by using random initialization (with results showing robustness across seeds).

Key Contributions

Novel Objective: SQFA is the first method to explicitly use the Fisher-Rao distance as an objective for supervised dimensionality reduction.
Practical Surrogate: The paper demonstrates that the Calvo-Oller bound is a highly effective and practical proxy for the Fisher-Rao distance in real-world datasets, enabling the optimization of this geometric objective without numerical integration.
Superior Performance of Hellinger Distance: A significant finding is that the variant maximizing the Hellinger distance (SQFA-H) consistently achieves the highest classification accuracy across diverse datasets, outperforming variants maximizing the Fisher-Rao distance, Bhattacharyya distance, and other state-of-the-art methods.
Interpretability: The learned features are interpretable and, in the context of naturalistic speed estimation, resemble motion-sensitive receptive fields found in biological visual systems.

Results
The authors evaluated SQFA and its variants on four datasets: SVHN (Street View House Numbers), MNIST, a naturalistic speed-estimation dataset, and neural spike recordings from macaque visual cortex.

Digit Classification (SVHN & MNIST): SQFA variants generally outperformed PCA, LDA, SPCA, LFDA, WDA, and LMNN.
- On SVHN (where class means are less informative than covariances), SQFA and SQFA-H achieved the highest QDA and kNN accuracy.
- On MNIST (where class means are informative), SQFA-H consistently achieved the highest accuracy across all filter counts, followed closely by SQFA-B. SQFA (Fisher-Rao) was competitive but generally ranked lower than SQFA-H.
Speed Estimation: In a task involving naturalistic video speed estimation, smSQFA (which maximizes the exact Fisher-Rao distance for zero-mean Gaussians) and SQFA-H performed nearly as well as AMA-Gauss, a method specifically designed to optimize Bayesian decoding performance. The filters learned by SQFA variants were visually similar to motion-sensitive neurons in the visual cortex.
Neural Data: On neural recordings, SQFA-H again achieved the highest classification accuracy, demonstrating the method's ability to exploit stimulus-dependent covariances even in noisy, non-Gaussian data.
Geometric Analysis: The authors verified that maximizing the Calvo-Oller bound indeed leads to the largest Fisher-Rao distances in the feature space compared to other objectives. Furthermore, the pairwise Calvo-Oller bounds were highly correlated with the true Fisher-Rao distances.

Significance and Claims
The paper claims that information geometry provides a powerful and underutilized framework for supervised dimensionality reduction. Specifically:

Geometric vs. Information-Theoretic: Maximizing the Fisher-Rao distance (a geometric measure) yields features competitive with those maximizing information-theoretic measures (like Bhattacharyya) that are directly linked to Bayes error.
The Hellinger Advantage: The authors suggest that the superior performance of SQFA-H may stem from how the Hellinger distance weights class pairs. Unlike the Bhattacharyya distance, which grows rapidly for well-separated classes (potentially over-emphasizing easy pairs), the bounded Hellinger distance focuses learning on separating classes that are close together, which is often more critical for multiclass discrimination.
Robustness: The method is computationally efficient, scales well to high dimensions, and produces robust features across different classifiers (QDA and kNN).
Future Directions: The authors note that while SQFA assumes Gaussianity, the Calvo-Oller bound extends to other elliptical distributions (e.g., Student-t, Cauchy), offering a path for future work on non-Gaussian data. They also identify extending the framework to nonlinear feature learning as a future direction.

The paper concludes that SQFA, particularly the SQFA-H variant, represents a state-of-the-art approach for learning linear features that maximize classification accuracy, validating the utility of information geometry in machine learning.