🔭 astrophysics

Information Geometry meets Functional ANOVA: An Exact Fisher-Information Decomposition, with an Application to Radio Luminosity Functions

This paper establishes an exact decomposition of the Fisher information metric for nonlinear regression models into Hoeffding-Sobol' functional ANOVA channels, introducing information-theoretic analogues of Sobol' indices and demonstrating their utility in analyzing parameter constraints and combinations with low sensitivity within a radio luminosity function model for active galactic nuclei and star-forming galaxies.

Original authors: Marko Imbrišak, Krešimir Tisanić

Published 2026-08-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Marko Imbrišak, Krešimir Tisanić

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to tune a giant, complex radio to find a specific station. The radio has hundreds of knobs, and some of them are connected by invisible springs. If you turn one knob, it might wiggle another, or maybe it cancels out the effect of a third. In the world of astronomy and statistics, scientists often build mathematical models to describe the universe, like how many galaxies exist at different brightness levels. These models have "knobs" (parameters) that need to be adjusted to fit the data.

The big question is: where does the information come from that tells us how to turn these knobs? Is it the main dial, or is it a tiny, hidden screw? Usually, scientists look at the whole model as a single, messy block of numbers. But what if we could take that block apart, like a Lego set, to see exactly which piece of the puzzle is doing the heavy lifting? This is the heart of a new approach called "Information Geometry," which treats these models like shapes in a strange, multi-dimensional space. It uses a tool called "Functional ANOVA" (a fancy way of breaking a problem into its main parts and its interactions) to see how different parts of the data contribute to the final answer. The goal is to stop guessing and start knowing exactly which part of our model is actually telling us something useful, and which parts are just confusing noise.

The Great Information Breakup

In this paper, two researchers, Marko Imbrišak and Krešimir Tisanić, have discovered a mathematical magic trick. They found a way to take the "Fisher information"—a measure of how much a model knows about its own settings—and slice it up perfectly along the lines of how the data is organized.

Think of a model like a recipe for a cake. You have ingredients like flour (main effect), sugar (main effect), and maybe a secret spice that only works if you mix it with the flour (interaction). Usually, when you taste the cake, you just know it's "good" or "bad." But these authors built a scale that can weigh exactly how much the flour contributed, how much the sugar contributed, and how much the mixing of the two contributed to the final taste. Even better, they found that sometimes ingredients work against each other. If you add too much flour, you might need to add a specific amount of sugar to fix it. In their math, this shows up as a "negative" contribution, meaning the two parts are canceling each other out.

The paper proves that this breakdown isn't just an approximation; it is an exact identity. They showed that if you add up all the pieces—the main effects, the interactions, and the weird "cancellation" parts—you get the total information back, down to the last decimal point. It's like taking a complex song apart and proving that the sum of the bass, the drums, the vocals, and the silence between them equals the original song perfectly.

The "Sloppy" Secrets of the Universe

One of the most exciting things they found is how to spot "sloppy" parameters. In science, a "sloppy" parameter is one where you can wiggle it around without changing the result much, usually because another parameter is doing the exact opposite to compensate. It's like a seesaw: if one kid goes up, the other goes down, and the balance stays the same.

The authors showed that these sloppy combinations leave a specific fingerprint: a negative "cross-term" in their math. When they looked at their data, they saw these negative fingerprints clearly. This tells scientists, "Hey, don't worry too much about pinning down this specific number exactly, because it's being held in place by another number doing the opposite dance." This helps researchers know where to focus their energy and where to let go.

Testing the Theory: Plants and Radio Waves

To prove their idea works, the team tested it on two very different datasets.

First, they looked at a simple experiment with plants. They measured how much carbon dioxide grass drank under different conditions (chilled vs. not chilled, and different plant clones). It was a small, clean dataset. The math worked perfectly, showing that the main factors (the treatment and the plant type) did most of the work, with a tiny bit of interaction between them. It was a nice, clean confirmation that their "Lego-breaking" method worked on simple things.

Then, they tackled a much bigger, messier problem: the Radio Luminosity Functions of galaxies. This is a map of how many galaxies exist at different brightness levels in the radio part of the spectrum. They combined data from two different surveys: one looking at nearby galaxies (the 6dFGS–NVSS survey) and one looking at distant galaxies (the VLA-COSMOS 3 GHz survey). They had to fit a model that described two types of galaxies: Star-Forming Galaxies (SFGs) and Active Galactic Nuclei (AGN).

Here, the math revealed some surprising truths.

The Main Drivers: Most of the information about the model came from the general shape of the curve (the "mean") and the difference between the two types of galaxies (the "functional form").
The Hidden Tension: When they looked at the Star-Forming Galaxies, they found a strong "sloppy" behavior. The math showed that the information from the nearby survey and the distant survey were fighting each other. If you tried to adjust the brightness of the faint galaxies based on the nearby data, the distant data would push back. This "compensating" effect meant that the model was struggling to agree on the exact brightness of the faintest galaxies.
The Bright End: For the bright galaxies, the model was much more stable. The information was spread out evenly, and there wasn't much of that "canceling out" behavior.

Why This Matters

The authors didn't just invent a new way to crunch numbers; they gave scientists a new pair of glasses. Before this, if a model was hard to fit, you might just shrug and say, "The data is noisy." Now, you can look at the "channel decomposition" and say, "Ah, the problem is that the nearby and distant surveys are compensating for each other in the faint-end region."

They also showed that how you write your math matters. If you put the variables in the wrong place (like adding them inside a logarithm instead of multiplying them), you can create artificial "leverage" where a single data point controls the whole result. By using their method, they proved that multiplying the variables was the safer, more stable choice for their galaxy model.

In short, this paper takes the mysterious, opaque "black box" of complex statistical models and opens the door. It lets us see exactly which gears are turning, which ones are grinding against each other, and which ones are just along for the ride. It turns a vague feeling of uncertainty into a precise map of where the information lives, helping astronomers understand not just what their models say, but why they say it.

Technical Summary: Information Geometry meets Functional ANOVA

Problem Statement
Global sensitivity analysis typically employs the Hoeffding–Sobol' (functional ANOVA) decomposition to partition the output variance of a model into main effects and interactions. However, this approach addresses the variance of a fixed function, not the information content regarding the model's parameters during a fitting process. In nonlinear regression with heterogeneous or dependent covariates, it remains unclear how the Fisher information metric—the measure of parameter identifiability and uncertainty—decomposes across these same ANOVA channels. Existing derivative-based sensitivity measures (e.g., DGSM) and active-subspace methods aggregate gradients to rank inputs or find dominant directions but do not provide an exact, additive decomposition of the Fisher information matrix itself.

Methodology
The authors establish an exact identity linking the geometry of the fitted model to the Hoeffding–Sobol' decomposition.

Hilbert Space Formulation: The score contributions (derivatives of the model output with respect to parameters) are treated as vectors in a Hilbert space. The Hoeffding decomposition partitions the model output $f$ into orthogonal components (mean, main effects, interactions) based on the covariate distribution.
Exact Decomposition Theorem: The paper proves that the Fisher information metric $g_{ab} = \sum_k \partial_a f_k \partial_b f_k$ (the Gauss–Newton matrix) decomposes exactly into a sum of Gram matrices of these score components. Specifically, $g_{ab} = \sum_{u,v} T^{uv}_{ab}$ , where $T^{uv}_{ab}$ represents the cross-channel terms between ANOVA components $u$ and $v$ .
Centered vs. Uncentered: The decomposition distinguishes between the uncentered metric (including the mean channel) and the centered metric (excluding the mean), showing that the mean channel accounts for the exact difference between the two.
Cross-Term Interpretation: The cross-channel terms $T^{uv}$ ( $u \neq v$ ) are shown to be signed. Positive terms indicate complementary information (super-additive detectability), while negative terms indicate compensating or "degenerate" parameter combinations where shifts in one channel are cancelled by shifts in another, leading to flat directions in the cost landscape.
Estimators: Two practical estimators are proposed:
- A nonparametric "pick-and-freeze" estimator (cost $O(N^2)$ ) that re-pairs covariates.
- A parametric orthogonal-basis projection (cost $O(N)$ ) that projects derivatives onto an orthonormal basis of the covariate space.
  The paper notes that under true covariate independence, population-level cross-terms between specific channels (e.g., main effects and interactions) vanish, providing a theoretical noise floor for finite-sample estimates.

Key Contributions

Exact Identity: The derivation of an exact decomposition of the Fisher information matrix into main-effect, interaction, and cross-channel terms, defined as a "Sobol'-type accounting for parameter information."
Information-Theoretic Sobol' Indices: The definition of $S^u = \text{tr}(T^{uu})/\text{tr}(g)$ as an analogue to classical variance-based Sobol' indices, with the caveat that the sum of indices is not constrained to unity due to the presence of signed cross-terms.
Characterization of Degeneracy: A rigorous geometric interpretation of negative cross-terms as the signature of "degenerate" parameter combinations (Transtrum et al.), where the Fisher information is small despite large individual channel responses due to cancellation.
Independence Properties: A proof that under covariate independence, specific population cross-terms vanish, allowing the framework to distinguish genuine model interactions from sampling noise.

Results
The methodology is applied to two datasets:

Plant CO2 Uptake (Categorical): A small dataset with binary covariates (chilling treatment, replicate). The decomposition shows information dominated by the mean level and split equally between the two main effects, with negligible interaction and cross-terms. Both estimators agree, validating the method for categorical data.
Radio Luminosity Functions (Astrophysical): A combined model of Star-Forming Galaxies (SFG) and Active Galactic Nuclei (AGN) fitted to 6dFGS–NVSS and VLA-COSMOS 3 GHz data.
- Global Decomposition: Information is overwhelmingly carried by the mean level and the functional-form covariate (distinguishing SFG from AGN). The catalogue covariate and interaction terms carry a small but non-zero share.
- Spatial Resolution: By pulling the decomposition back onto the luminosity axis, the authors show that for SFGs, the bright-end variance is dominated by cross-calibration issues between surveys (the $X_2$ and interaction channels), not the shape parameters.
- Degeneracy Visualization: For the SFG branch, the cross-term is negative across most of the luminosity range, visualizing the compensating nature of parameters near the turnover ( $L^\star$ ). For the AGN branch (broken power law), the cross-term remains near zero, indicating less degeneracy.
- Coupling Sensitivity: The study highlights that multiplicative coupling of covariates in log-space models is essential for well-conditioned interaction channels, whereas additive coupling can lead to pathological leverage.

Significance and Claims
The paper claims to provide a novel, exact decomposition of the Fisher information metric that extends the Hoeffding–Sobol' framework from output variance to parameter information. It does not claim novelty for the underlying ANOVA machinery itself but rather for the specific construction of indexing the pullback of the ambient Fisher metric by ANOVA channels.

The authors emphasize that this decomposition transforms the Fisher information matrix from an opaque object into an interpretable, channel-resolved accounting of where parameter information originates. It offers a principled way to identify "degenerate" directions in parameter space and to localize which parts of a dataset (e.g., specific luminosity bins or surveys) constrain specific model components. The paper remains modest, noting that the decomposition is model-agnostic but dependent on the choice of channel structure and estimator, and that it describes the fitted model's information landscape rather than the data-generating process itself.

The Great Information Breakup

The "Sloppy" Secrets of the Universe

Testing the Theory: Plants and Radio Waves

Why This Matters

More like this