🤖 machine learning

Bridging Functional and Representational Similarity via Usable Information

This paper presents a unified framework that quantifies representation similarity through the lens of usable information, establishing formal links between stitching performance and conditional mutual information while demonstrating that representational similarity is relative to predictive capacity and sufficient but not necessary for functional similarity.

Original authors: Antonio Almudévar, Alfonso Ortega

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Antonio Almudévar, Alfonso Ortega

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two different chefs (neural networks) who have both learned to cook a specific dish, like a perfect lasagna. You want to know: Are these two chefs actually using the same "recipe" in their minds, or are they just arriving at the same delicious result by totally different methods?

This paper, titled "Bridging Functional and Representational Similarity via Usable Information," is a guidebook for answering that question. It proposes a new way to measure how similar two AI models really are by looking at the "usable information" they hold, rather than just comparing their raw outputs.

Here is the breakdown of their findings using simple analogies:

1. The Two Ways to Compare Chefs

The authors argue that we usually look at similarity in two different ways, but we often confuse them:

Functional Similarity (The "Taste Test"): This asks, "If I give Chef A's notes to Chef B, can Chef B still make the lasagna?" If Chef B can take Chef A's intermediate notes and finish the job perfectly, they are functionally similar. It doesn't matter if their notes look different on the page; what matters is if the information is usable to get the result.
Representational Similarity (The "Notebook Check"): This asks, "Do Chef A and Chef B write their notes in the exact same handwriting and format?" This looks at the internal structure of the data. If their notes are geometrically identical, they are representationally similar.

2. The Big Discovery: The "One-Way Street" Problem

The paper reveals a crucial flaw in how we usually test these chefs. We often try to translate Chef A's notes to Chef B's style (a process called "stitching") and see if it works.

The Finding: This translation is often a one-way street.
The Analogy: Imagine Chef A writes notes in a detailed, high-definition diary. Chef B writes in a rough sketchbook. You can easily translate the diary into a sketch (you lose detail, but the main idea remains). But you cannot translate the rough sketch back into a detailed diary because the information was lost in the sketch.
The Lesson: To truly say two models are similar, you must be able to translate both ways. If you can only go from A to B, but not B to A, they aren't truly functionally similar. The paper proves that "stitching" is inherently asymmetric.

3. The "Observer" Matters (Who is Reading the Notes?)

The paper introduces a fascinating concept: Similarity depends on who is reading the notes.

The Analogy: Imagine a complex map.
- If you are a rigid observer (someone who only understands straight lines and right angles), two maps might look completely different and useless to you.
- If you are a flexible observer (someone who understands curves, angles, and 3D shapes), those same two maps might look identical.
The Lesson: Two AI models might look very different to a simple mathematical tool (like a straight line), but if you use a more powerful tool (a complex neural network) to read them, they might turn out to be the exact same. "Similarity" isn't an absolute fact; it depends on the complexity of the tool you use to measure it.

4. The Hierarchy: The "Whole Picture" vs. The "Specific Task"

The authors establish a clear hierarchy between the two types of similarity:

Representational Similarity is the "Master Key": If two models have the exact same internal notes (Representational Similarity), they are guaranteed to be able to solve any task you throw at them, from the hardest to the easiest.
Functional Similarity is the "Specific Key": If two models can solve a specific task (like identifying a cat), they are functionally similar for that task. However, they might have thrown away other information (like the color of the cat's fur) that isn't needed for that specific task.
The Takeaway: Having the "Master Key" (Representational Similarity) guarantees you have the "Specific Key" (Functional Similarity). But having the "Specific Key" does not guarantee you have the Master Key. You might be able to open the front door (solve the task) without having the blueprint for the whole house (the full representation).

5. Old Tools Are Actually New Tools in Disguise

Finally, the paper looks at the standard tools scientists use today (like CKA, RSA, and stitching).

The Finding: These aren't just random math tricks. They are actually measuring "usable information" under specific constraints.
The Analogy: Think of these tools as different types of rulers. A standard ruler measures length. A flexible tape measure measures curves. The paper shows that when we use these different "rulers," we are essentially asking the models, "Can you solve this problem if you are only allowed to use straight lines?" or "Can you solve it if you can use curves?"
The Result: The paper proves that these standard tools are actually very good at estimating how much "usable information" is shared between models, provided you understand what kind of "ruler" (or predictive family) you are using.

Summary

In short, this paper tells us:

Don't just check one direction: To see if two AI models are truly similar, you must be able to translate their knowledge back and forth.
Context is king: Similarity depends on how complex the "reader" is. A simple reader might see two models as different, while a complex reader sees them as identical.
The Whole vs. The Part: If two models are identical in their internal structure, they can do anything. But if they can do one specific thing, they might still be very different internally.
Old tools are useful: The standard ways we measure AI similarity are actually valid, but they are measuring specific types of "usable information" based on the complexity of the tools we use.

Technical Summary: Bridging Functional and Representational Similarity via Usable Information

Problem Statement

The field of deep learning relies heavily on comparing internal neural representations to understand how different systems process information, transfer knowledge, and organize structures. However, the literature currently lacks a unified framework to define and measure "similarity" between representations. Existing approaches are typically bifurcated into two loosely defined categories:

Functional Similarity: Assessing whether representations support the same downstream behaviors (e.g., via model stitching or linear probes).
Representational Similarity: Assessing whether representations encode input information in the same structural way (e.g., via CKA, RSA, or Procrustes analysis).

These concepts have been studied in isolation, leading to a disconnect between functional methods (which focus on task performance) and geometric metrics (which focus on feature space alignment). Furthermore, standard metrics often lack a rigorous theoretical grounding regarding what information is actually being compared or how the capacity of the observer (the predictive model) influences the measurement.

Methodology

The authors propose a unified framework grounded in Usable Information Theory (Xu et al., 2020). This theory restricts information flow to a specific "predictive family" $\mathcal{V}$ , acknowledging that not all statistical dependence in a representation is exploitable by a downstream model of limited capacity.

Theoretical Framework

The paper redefines similarity through the lens of conditional mutual information and Markov blankets:

Functional Similarity: Two representations $Z_1$ and $Z_2$ are functionally similar with respect to a task $Y$ if they form mutual Markov blankets. This implies $I(Y; Z_2 | Z_1) = I(Y; Z_1; Z_2) = 0$ . The authors formalize Model Stitching as a mechanism to estimate this: if $Z_1$ can be stitched into $Z_2$ without loss of performance, $Z_1$ contains all task-relevant information of $Z_2$ . Crucially, they prove this relationship is inherently asymmetric; robust functional similarity requires bidirectional stitching.
Representational Similarity: Defined as functional similarity where the task $Y$ is the input $X$ itself ( $I(X; Z_2 | Z_1) = 0$ ). This implies the representations contain identical information about the input.
Hierarchy of Granularity: The authors establish a monotonic relationship: similarity on a complex task guarantees similarity on any coarser derivative task. Consequently, representational similarity (reconstructing the input) is the strongest condition, guaranteeing functional similarity for all derived tasks.

Empirical Methodology

To validate these theoretical claims, the authors conducted a large-scale empirical study involving:

Scale: 58 trained models across 14 distinct architectures (including Linear, CNNs like ResNet, DenseNet, ShuffleNet, and MobileNet) and 6 datasets (MNIST, CIFAR-10/100, SVHN, Tiny-ImageNet, ImageNet).
Comparisons: 12,459 distinct layer-to-layer comparisons.
Stitching Families: The study utilized three specific families of stitchers (predictive families $\mathcal{V}$ $V$ ) to test the impact of expressiveness:
1. Affine: Unconstrained linear maps.
2. Orthogonal + Isotropic Scaling: Procrustes-like constraints.
3. Orthogonal: Strict rotation constraints.
Metrics: The authors compared standard metrics (CKA, RSA, SVCCA) against their proposed usable information-based scores (normalized reconstruction error).

Key Contributions

Unified Framework via Usable Information: The paper theoretically links model stitching to usable conditional mutual information. It demonstrates that functional similarity is not an absolute property but is defined relative to the capacity of the predictive family $\mathcal{V}$ .
Reinterpretation of Standard Metrics: The authors empirically and theoretically show that standard geometric metrics (CKA, RSA, Procrustes) are not arbitrary measures but specific estimators of usable information under specific constraints (e.g., orthogonal transformations).
The Hierarchy of Similarity:
- Asymmetry: They prove that stitching is inherently asymmetric; a single directional stitcher is insufficient to claim functional similarity.
- Sufficiency vs. Necessity: They demonstrate that representational similarity is a sufficient but not necessary condition for functional similarity. Representations can be functionally similar (solving the same task) while being representationally distinct (encoding different nuisance variables).
- Task Granularity: They establish that representational similarity (input reconstruction) is the limit of maximum granularity, guaranteeing functional similarity for all coarser tasks.

Experimental Results

Stitching Asymmetry: Analysis of 12,459 pairs revealed a heavy-tailed distribution of accuracy differences between forward and reverse stitchers, confirming that functional similarity is frequently asymmetric. One representation often contains the information of another without the reverse being true.
Monotonicity of Representational Similarity: The study validated that representational similarity scores increase as the predictive family becomes more expressive. Representations that appear distinct under rigid Procrustes constraints (Orthogonal) often appear nearly identical under more expressive Affine transformations.
Metric Alignment: Standard metrics showed strong correlation with the proposed usable information scores. Specifically, CKA ( $R^2=0.937$ ) and RSA ( $R^2=0.912$ ) aligned closely with the representational similarity derived from Orthogonal + Scaling stitchers, supporting the hypothesis that these metrics act as proxies for usable information under specific transformation groups.
Hierarchy Validation: Using CIFAR-100 (fine classes vs. coarse superclasses), the authors confirmed that functional similarity on a coarse task consistently upper-bounds similarity on the fine task.
Sufficiency of Representational Similarity: As representational similarity approached 1.0, the probability of functional similarity converged to 1.0. However, high functional similarity was frequently observed even with low representational similarity, confirming that representational similarity is not a necessary condition.

Significance and Claims

The paper claims to resolve the disconnect between functional and representational similarity by grounding both in the theory of usable information. The authors argue that similarity is not an intrinsic property of the representations alone but is relative to the computational constraints of the predictive family used to compare them.

By unifying these concepts, the paper provides a rigorous theoretical justification for why standard metrics work and under what conditions they fail. It shifts the paradigm from seeking a "universal" similarity metric to explicitly defining the predictive family $\mathcal{V}$ that dictates what "similar" means for a specific application. The work concludes that while representational similarity guarantees functional similarity across all tasks, the reverse is not true, highlighting that models can achieve identical task performance through fundamentally different internal representations.