🤖 machine learning

Rashomon Alignment

This paper introduces Rashomon Alignment (RA), a new geometric measure that assesses functional similarity between machine learning models across the entire input space rather than just on observed data, offering a complementary perspective to distributional methods for applications like model selection and interpretability.

Original authors: Moisés Santos, Peter van der Putten, Bernhard Pfahringer, Carlos Soares

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Moisés Santos, Peter van der Putten, Bernhard Pfahringer, Carlos Soares

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize cats. You show it thousands of pictures, and it learns to spot whiskers and pointy ears. But here's the tricky part: two robots might both get 90% of the pictures right, yet they could be "thinking" in completely different ways. One might be looking at the background, while the other is focusing on the ears. In the world of machine learning, this is a huge deal. If you only look at the final score (accuracy), you might think two robots are identical twins, when in reality, they are strangers who just happen to agree on the test you gave them. This paper dives into a specific corner of computer science called "model similarity." It asks a simple but profound question: How do we know if two AI models are actually seeing the world the same way, or if they are just lucky coincidences? The authors want to move beyond just checking the test scores and start mapping the actual "decision boundaries"—the invisible lines the models draw in their minds to separate one category from another.

The authors of this paper, Moisés Santos and his team, propose a new way to measure this similarity called Rashomon Alignment. Think of it like this: imagine two cartographers drawing maps of a mysterious island. One cartographer only draws the parts of the island where tourists usually go (the popular beaches and hotels). The other draws the entire island, including the dense, unexplored jungles and hidden caves. If you only look at the tourist areas, their maps might look identical. But if you zoom out to see the whole island, you might find that one cartographer drew a mountain where the other drew a lake.

The paper introduces two versions of this map-checking tool. The first is Distributional Rashomon Alignment (dRA). This is like the first cartographer; it checks how much the models agree on the data they have actually seen (the "tourist spots"). The second, and the paper's main star, is Geometric Rashomon Alignment (gRA). This is the second cartographer. It ignores where the data actually is and instead asks the models to guess on a perfectly uniform grid covering the entire possible space. It forces the models to show their hand on regions they've never seen before, revealing if their internal logic is truly similar or just a fluke of the specific data they were trained on.

To test this, the team ran a massive experiment using 92 different datasets from the UCI Machine Learning Repository. They pitted two types of decision trees against each other: "unpruned" trees (which are allowed to grow wild and complex, often memorizing every little detail) and "pruned" trees (which are trimmed down to be simpler and more general). They found that looking at accuracy alone was often misleading. In many cases, the two trees had very similar scores, but the gRA revealed they were making decisions in totally different ways across the full space.

The results were eye-opening. The study showed that gRA and dRA provide different, complementary views. Sometimes, models agreed perfectly on the data they saw (high dRA) but disagreed wildly everywhere else (low gRA). This is a dangerous situation: it means the models are fragile and might fail miserably if the data changes even slightly. The authors found that in about 92 datasets, the geometric alignment (gRA) varied widely, spreading across the full range from 0 to 1, whereas the distributional alignment (dRA) tended to cluster at higher values, suggesting models often agree more on the "tourist spots" than on the whole island.

Crucially, the paper suggests that relying only on accuracy or only on data-based agreement can be deceiving. For instance, they identified cases where two models had the exact same accuracy but were structurally very different (low gRA), and cases where they were structurally almost identical (high gRA) but had different accuracy scores because the test data happened to fall in a tricky spot. The authors conclude that Geometric Rashomon Alignment is a powerful new tool. It doesn't replace accuracy, but it adds a vital layer of understanding, helping us see if two models are truly aligned in their logic or just faking it on the specific data we happen to have. This could help engineers build better AI teams (ensembles) and understand why a model makes a mistake, not just that it made one.

Technical Summary: Rashomon Alignment

Problem Statement

Current methods for assessing functional similarity between machine learning models rely on distributional measures. These metrics compare model outputs on available real-world data (test sets) to quantify agreement. However, the authors argue that such measures suffer from two critical limitations:

Ecological Validity: They only reflect similarity in regions of the input space covered by the available data. If the data sample is unrepresentative of the true underlying distribution, the similarity estimate is poor.
Concept Drift Sensitivity: Even if accurate for current data, these measures become outdated if the data distribution changes (covariate shift), as they do not account for the structural alignment of decision boundaries across the entire instance space.

Existing functional similarity measures fail to distinguish between models that agree on specific data points but define structurally different decision boundaries elsewhere. Consequently, relying solely on predictive accuracy or distributional agreement can be deceptive, masking significant structural divergences between models.

Methodology: Rashomon Alignment (RA)

The paper proposes Rashomon Alignment (RA), a new framework for assessing functional similarity that incorporates a geometrical perspective. RA is defined as the probability that two models, $M_A$ and $M_B$ , produce the same prediction for a randomly selected instance $x$ from a reference distribution $Q$ . The authors distinguish between two variants based on the choice of $Q$ :

1. Distributional Rashomon Alignment (dRA)

Reference Distribution ( $Q$ ): The data-generating distribution $P(X)$ .
Estimation: Computed using a sample of observed data (e.g., a hold-out test set).
Characteristics: Reflects agreement in regions where real data exists. It is sensitive to the representativeness of the sample and changes in the data distribution. It is an unbiased estimator of the population-level agreement under $P(X)$ .

2. Geometric Rashomon Alignment (gRA)

Reference Distribution ( $Q$ ): The uniform distribution $U(F)$ over the instance space $F$ (defined by the domains of the variables).
Estimation: Computed using a synthetic sample $X_{syn}$ drawn uniformly from the instance space (typically within the bounding box of the training features).
Characteristics: Estimates similarity across the entire instance space, independent of any specific data distribution. It measures the fraction of the instance space where decision functions coincide.
Invariance: Because $X_{syn}$ is generated independently of training or test data, gRA is invariant to changes in $P(X)$ as long as the underlying decision functions remain fixed.

Theoretical Relationship

The paper establishes that dRA and gRA are related via a change of measure. The gap between them ( $|dRA^* - gRA^*|$ ) quantifies the extent to which observed data fails to represent the full instance space. A high dRA with low gRA indicates that models agree where data lies but disagree elsewhere, a form of agreement that is fragile under distribution shifts.

Experimental Setup

To validate the methodology, the authors conducted an extensive empirical study:

Datasets: 92 diverse datasets from the UCI Machine Learning Repository, covering various domains, sizes (32 to 4,600+ instances), dimensionalities (4 to 649 features), and feature types.
Models: A comparison between pruned and unpruned decision trees. This scenario was chosen because the models share the same hypothesis language (directly comparable boundaries) but exhibit different tendencies regarding overfitting, ensuring a mix of aligned and misaligned pairs.
Protocol: 5-fold cross-validation. For each fold, models were trained, and three metrics were recorded: accuracy difference, dRA (on the test partition), and gRA (on 1,000 synthetic uniform instances).

Key Results

The experiments yielded four primary insights:

Complementarity of gRA and Accuracy:
- There is a moderate positive correlation ( $r = 0.514$ ) between gRA and accuracy difference.
- However, the measures are not redundant. The study identified critical cases where models had low gRA but low accuracy difference (structurally different boundaries that happen to balance out on the test set) and high gRA but high accuracy difference (similar boundaries where disagreement is concentrated in specific test regions).
- This demonstrates that focusing solely on predictive performance can be deceiving; gRA reveals structural similarities or differences that accuracy alone cannot.
Complementarity of gRA and dRA:
- dRA values were generally higher and more concentrated than gRA values, indicating that models tend to agree more on observed data than across the full instance space.
- A strong correlation ( $r = 0.745$ ) exists between gRA and dRA, but significant deviations were observed.
- Critical Finding: Several datasets exhibited high dRA but low gRA. In these cases, models agreed on the test partition but differed substantially over most of the instance space. Relying solely on dRA in such scenarios would lead to an overestimation of structural model similarity.
- Conversely, no dataset showed high gRA with low dRA, consistent with the theoretical relationship that uniform agreement implies agreement on any subsample.
Quadrant Analysis:
- The analysis partitioned datasets into four quadrants based on the medians of gRA and absolute accuracy difference.
- The most populated quadrant was high accuracy difference / low gRA, confirming that pruning often alters boundaries significantly in complex problems.
- The high accuracy difference / high gRA quadrant highlighted cases where global alignment existed despite test accuracy discrepancies, emphasizing that disagreement was concentrated in the test set's specific region.

Significance and Claims

The paper claims that Rashomon Alignment provides a comprehensive view of decision boundary alignment that is independent of specific data distributions. The significance of the work lies in:

Introducing a Geometrical Perspective: Moving beyond distributional similarity to assess functional similarity across the entire instance space.
Revealing Hidden Divergences: Demonstrating that distributional measures (and accuracy) can mask structural differences between models, particularly under covariate shift or when data is unrepresentative.
Practical Utility: The authors propose that RA can be used for model selection, ensemble construction (by selecting diverse models), and enhanced interpretability of machine learning algorithms.

The authors maintain a modest stance, noting that RA complements rather than replaces accuracy. They suggest future work involves applying RA to other algorithms and refining methods for sampling from the instance space.