🧠 neuroscience

Framed RSA: Representational comparisons that honor both geometry and population-mean response preferences

This paper introduces "framed RSA," a novel analysis technique that enhances model-selection accuracy and mechanistic interpretability by augmenting traditional representational similarity analysis with reference patterns to simultaneously honor both the geometry of neural activity and population-mean response preferences.

Original authors: Taylor, J. E., Schutt, H., Kriegeskorte, N.

Published 2026-09-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Taylor, J. E., Schutt, H., Kriegeskorte, N.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

To understand how the brain thinks, scientists often look at the patterns of activity that light up when we see a face, a house, or a word. Imagine a vast, multi-dimensional space where every possible thought or perception has a specific location. When you see a cat, your brain creates a unique pattern of activity, a point in this space. When you see a dog, it creates another point. The distance between these points tells us how similar the brain considers the two animals to be. This is the core idea behind a popular method called representational similarity analysis. For years, researchers have used this approach to compare how different parts of the brain, or even layers of artificial intelligence, organize information. They have found that the shape formed by these points—the geometry of the pattern—is a powerful signature of what a brain region does.

However, this traditional method has a blind spot. It focuses entirely on the distances between points, effectively ignoring where the entire group of points sits within the larger space. It does not care if the brain is generally more active when seeing faces than when seeing houses, or if a specific region is simply more energetic overall. This missing piece of information, the average level of activity across all the neurons in a region, is biologically significant. It reflects the metabolic cost of processing different stimuli and offers clues about the specific computational function of that brain area. By discarding this data, standard methods might miss crucial differences between systems that look geometrically similar but operate with different energy profiles.

A team of researchers at Columbia University and the University of Luxembourg has introduced a new technique called framed representational similarity analysis to fix this oversight. Their goal was to create a method that respects both the shape of the neural patterns and their overall position and orientation in the response space. To do this, they added two invisible reference points to the mathematical landscape: one representing a state of zero activity and another representing a state of uniform, maximum activity. These two points act like a frame or a ruler, anchoring the neural patterns in place. By measuring the distance of every stimulus pattern to these fixed references, the new method can recover the average activity level for each stimulus. This allows the analysis to distinguish between brain regions that might have the same internal geometry but different overall preferences, such as one region that is highly active for faces and another that is highly active for houses, even if the relative distances between the face and house patterns are identical.

The researchers tested this new approach using three distinct types of data: brain scans from humans viewing thousands of natural images, electrical recordings from the brains of macaque monkeys, and the internal layers of deep neural networks trained to recognize objects. In each case, they treated the task as a matching game. They asked whether the new method could correctly identify that the visual cortex of one person was more similar to the visual cortex of another person than it was to a different brain region, such as the area that processes words. They compared the performance of their framed method against the standard technique and against a simple analysis that only looked at average activity levels.

The results showed that the framed approach consistently improved the ability to distinguish between different brain regions and network layers. In the human brain scan data, the new method was significantly better at identifying the correct brain region than the standard method, especially when the number of images shown was small. It also outperformed looking at average activity alone when the data was limited. In the recordings from macaque monkeys, the framed method matched or exceeded the performance of the standard technique across all conditions. Perhaps most revealing was the test on artificial neural networks. When the researchers added noise to simulate real-world measurement errors, the framed method proved more robust than the standard approach. It successfully identified matching layers across different versions of the same network, suggesting that incorporating the average activity profile provides a more stable and informative signature of how a system computes.

The study also explored how the choice of the reference point affects the results. They found that setting the reference point to a moderate size worked best, while making it too large actually reduced the method's accuracy. This suggests that the balance between the geometry of the patterns and their position relative to the reference frame is delicate. The researchers demonstrated that their new technique is not just a theoretical improvement but a practical tool that can be integrated into existing analysis pipelines. By uniting the study of regional activation levels with the study of pattern geometry, framed representational similarity analysis offers a more complete picture of how the brain and artificial systems represent the world. It honors the fact that where a pattern sits in the neural space is just as important as the shape it forms with other patterns.

Technical Summary: Framed RSA

Problem Statement
Representational Similarity Analysis (RSA) is a standard method for comparing neural representations across brain regions, species, and artificial neural networks (DNNs). RSA characterizes the geometry of response patterns by analyzing pairwise distances (or dissimilarities) between stimuli, typically summarized in a Representational Dissimilarity Matrix (RDM). However, standard RSA discards critical information regarding the population-mean response preferences (e.g., a region's overall stronger activation to faces than to houses) and the global location and orientation of the response patterns in the multivariate response space.

While invariance to certain transformations (like scaling or rotation) is desirable to abstract away from idiosyncratic encoding details, the authors argue that discarding mean activation profiles is problematic. The population-mean response is physiologically significant as it determines metabolic cost and reflects the tuning of individual neurons (e.g., mixed selectivity). Standard RSA is invariant to global translations and rotations that alter these mean preferences, potentially causing distinct computational functions to appear identical if their geometries are merely rotated or shifted versions of one another.

Methodology: Framed RSA
The authors introduce Framed RSA, a variant of RSA that incorporates information about the global location and orientation of response patterns relative to the all-1 vector in the multivariate response space. This is achieved by augmenting the set of stimulus patterns with two specific "framing patterns":

The Origin (0): An all-zero vector.
The Constant (c): An all- $c$ vector, where $c$ is a fixed positive real number.

These two patterns define a "frame" (specifically a hypercube boundary) in the response space. The method constructs an extended RDM ( $D_{0c}$ ) that includes:

Distances between all stimulus patterns.
Distances from each stimulus pattern to the origin ($0$).
Distances from each stimulus pattern to the constant vector ( $c$ ).

Mathematical Implications:

Recovery of Mean Activation: By expanding the squared Euclidean distance formula between a stimulus pattern $s$ , the origin $0$, and the constant $c$ , the authors demonstrate that the regional population-mean activation ( $c^T s / \|c\|_1$ ) can be recovered via linear operations on the entries of the extended RDM.
Reduced Invariance: Framed RSA removes translational invariance and reduces rotational invariance. It is no longer invariant to rotations that change the population-mean response for any stimulus, though it remains invariant to rotations about the $c$ vector.
Choice of $c$ : To maintain desirable invariances (such as invariance to uniform scaling of measurement units), the norm of $c$ is chosen to be proportional to the mean norm of the stimulus vectors (e.g., $k=1$ ).
Noise Handling: The authors adapt the use of noise-normalized dissimilarity estimators (Mahalanobis and crossnobis) and whitened RDM comparators. They derive a specific formula for the variance-covariance matrix ( $V$ ) of dissimilarity estimates that accounts for the noiseless nature of the $0$ and $c$ patterns, preventing singularity issues that would arise if standard assumptions (true distances = 0) were applied.

Key Contributions and Results
The authors validate Framed RSA using "region identification" tasks, where the ground truth is known (e.g., identifying which brain region or DNN layer generated a specific pattern). The performance is compared against standard RSA and the profile of mean activations alone.

Natural Scenes Dataset (Human fMRI):
- Framed RSA significantly outperformed standard RSA across all stimulus sample sizes when using whitened correlation or whitened cosine similarity as comparators.
- Framed RSA outperformed the mean-activation profile for small numbers of stimuli ( $\le 20$ ).
- Interestingly, for large numbers of stimuli ( $\ge 100$ ), the mean-activation profile alone performed best, suggesting that mean activation is a robust signature for fMRI data when sufficient data is available to average out noise.
Things Ventral Stream Spiking Dataset (Macaque Intracranial Recordings):
- Framed RSA matched or exceeded the performance of standard RSA across all conditions.
- It significantly outperformed standard RSA for small numbers of stimuli ( $\le 10$ ).
- Framed RSA consistently outperformed the mean-activation profile across all stimulus counts. The authors suggest that in low-noise neural recording data, the fine-grained information in the population patterns combined with mean preferences provides a stronger signature than mean preferences alone.
Deep Neural Network (DNN) Layer Identification:
- Using AlexNet instances trained with different random seeds, Framed RSA significantly outperformed both standard RSA and the mean-activation profile at low noise levels.
- At high noise levels, Framed RSA outperformed standard RSA but was outperformed by the mean-activation profile, reinforcing the finding that mean activation is particularly diagnostic under high-noise conditions.

Significance
The paper claims that Framed RSA unites two traditionally separate traditions in brain-activity analysis: regional activation analysis (univariate mean responses) and multivariate pattern geometry (RSA). By creating a "functional fingerprint" that honors both the geometry and the population-mean response preferences, Framed RSA enables more mechanistically meaningful model comparisons.

The authors argue that requiring models to correctly predict regional-mean response preferences, alongside geometry, improves the power to distinguish competing models of brain computation. This is particularly relevant because the metabolic cost of representing stimuli is a biologically important constraint. Framed RSA is implemented in the open-source RSA toolbox, allowing for easy integration into existing analysis pipelines. The method offers a simple, mathematically grounded approach to incorporating global response properties into representational comparisons without discarding the abstraction benefits of standard RSA.

More like this