🤖 machine learning

A Geometric View of Counterfactual Behavior: Interaction of Boundary Proximity and Local Support

This paper demonstrates that counterfactual behavior—specifically the feasibility and distance of meaningful input changes—is a distinct dimension from predictive performance, driven by the interaction between decision-boundary proximity and local data support, which can vary significantly even among models with identical accuracy.

Original authors: Ioanna Gemou, Matteo Gamba, Randall Balestriero, Ritambhara Singh

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ioanna Gemou, Matteo Gamba, Randall Balestriero, Ritambhara Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot that looks at pictures and guesses what they are. You want to know: "What tiny change would make this robot change its mind?" For example, if it thinks a picture is a "cat," what small tweak would make it say "dog"? This is called a counterfactual explanation.

This paper is like a detective story investigating why some robots are easy to trick into changing their minds, while others are stubborn, even if they are equally good at guessing the right answer in the first place.

Here is the breakdown using simple analogies:

1. The Setup: The Map and the Fence

Think of the robot's brain as a giant map (called "representation space").

The Encoder: This is the part of the robot that turns a picture into a dot on the map.
The Classifier: This is the part that draws fences (decision boundaries) on the map to separate cats from dogs.

The paper asks: If two robots have the same map (they see the world the same way) and are equally accurate at guessing, why does one robot change its mind easily with a tiny nudge, while the other requires a massive shove?

2. The Two Secrets: The Fence and the Crowd

The authors discovered that the answer depends on two things working together:

Secret #1: How close are you to the fence? (Boundary Proximity)
If your dot is right next to the fence between "Cat" and "Dog," it only takes a tiny step to cross over. If you are deep in the middle of the "Cat" zone, you have to walk a long way to cross.
Secret #2: Is there a crowd waiting on the other side? (Local Data Support)
This is the twist. Just crossing the fence isn't enough. Imagine you cross the fence into the "Dog" zone, but you land in a barren, empty desert where no real dogs live. That's a bad explanation because it's not a realistic "dog."
A good explanation happens when you cross the fence and land right in the middle of a crowd of real dogs.

The Paper's Big Claim: You need both to be close to the fence and close to a crowd of the other type to get a good, realistic answer.

3. The Surprise: Same Score, Different Behavior

The researchers tested many different robots (using images of shapes, handwritten numbers, medical X-rays, and movie reviews).

The Finding: They found pairs of robots that got the exact same test score (e.g., 90% accuracy).
The Twist: Even though they were equally smart, one robot was very easy to flip its opinion, while the other was very hard to flip.
Why? It wasn't because the robots saw the pictures differently. It was because the fences they drew were in different spots relative to the crowds. One robot drew its fence right next to a crowd; the other drew its fence far away in an empty spot.

4. The "Fence Mover" Experiment

To prove this, they took one robot, froze its map (so it couldn't change how it saw things), and just redrew the fences using different methods.

Result: By just moving the fences, they could make the robot suddenly become "easy to trick" or "hard to trick," even though the robot's overall accuracy stayed exactly the same.
Lesson: The way the robot draws its lines matters just as much as how well it sees.

5. Why This Matters (The "Why Should I Care?")

The paper suggests that when we try to explain AI decisions, we can't just look at how accurate the AI is. We have to look at the geometry (the shape of the map and the fences).

They even showed that if you tell the robot's search algorithm to "aim for the crowd" (not just cross the fence), it finds better, more realistic answers. It's like telling a lost hiker: "Don't just cross the river; cross it and walk straight to the village."

Summary in One Sentence

Two robots can be equally smart, but one might be easy to convince and the other hard, simply because one draws its decision lines right next to a crowd of examples, while the other draws them in an empty desert. To get good explanations, you need to look at both the lines and the crowds.

Technical Summary: A Geometric View of Counterfactual Behavior

Problem Statement
Counterfactual explanations aim to identify small, semantically meaningful input changes that alter a model's prediction. While widely used for auditing machine learning systems, current methods often treat counterfactual generation as a local optimization problem constrained by task-specific feasibility. However, a prediction-changing perturbation may exist within a locality budget yet fail to correspond to a realistic alternative if it lands in a region of weak data support. Furthermore, models with nearly identical predictive performance can exhibit vastly different counterfactual behaviors. This work investigates why these discrepancies occur, hypothesizing that counterfactual behavior is not solely a function of predictive accuracy but is governed by the geometric interaction between decision-boundary proximity and local data support within the representation space.

Methodology
The authors propose a geometric framework and a standardized local search probe to evaluate counterfactual behavior across diverse domains (vision, language, medical imaging, and multimodal).

Geometric Framework:
- Decision-Boundary Proximity ($dbd(z)$): Defined as the distance to the nearest decision boundary, approximated via a first-order linearization of the logit gap between the top two predicted classes.
- Local Target Class Support ( $r_k(z)$ ): Defined as the Euclidean distance from an embedding $z$ to its $k$ -th nearest neighbor within the target class's reference set in the embedding space. This measures whether the target region is densely populated with data.
- Hypotheses: The work tests three hypotheses: (H1) Accuracy-matched models differ in counterfactual behavior; (H2) Varying the classifier head under fixed embeddings alters counterfactual behavior; and (H3) The interaction of boundary proximity and local support predicts counterfactual outcomes.
Experimental Setup:
- Encoders: The study evaluates several pretrained encoders (e.g., ResNet-50, ViT, BERT, CLIP, SigLIP2) across datasets like Shapes, MNIST, Chest X-ray, IMDb, and MM-IMDb.
- Classifier Heads: Linear classifiers are trained on frozen embeddings. To isolate the effect of boundary placement, the authors train multiple heads on the same embeddings using different initializations and regularization strengths (weight decay).
- Search Protocol: A standardized projected gradient descent is used in the embedding space to find a prediction change within a fixed trust region ( $\tau$ ) and iteration budget.
- Metrics: Counterfactual Success (CF-Suc), Counterfactual Distance (CF-Dist), and Optimization Effort (OptEff).

Key Contributions

Geometric Framework and Probe: Introduction of a standardized method to evaluate counterfactual behavior based on the interaction of boundary proximity and local data support, moving beyond simple predictive metrics.
Isolation of Boundary Effects: Demonstration that under fixed representations, varying only the classifier head (via initialization or regularization) can substantially alter counterfactual success and distance while leaving predictive accuracy largely unchanged.
Joint Predictive Power: Evidence that decision-boundary proximity alone is insufficient; the combination of boundary proximity and local data support is required to explain variation in counterfactual feasibility and effort.

Results

Divergence in Accuracy-Matched Models: Models with minimal accuracy gaps (e.g., <0.02) exhibited large discrepancies in counterfactual behavior. For instance, on MNIST, two models with a 0.01 accuracy gap showed a 65-percentage-point difference in counterfactual success rates.
Classifier Sensitivity: When the encoder was frozen and only the linear classifier head was varied, counterfactual success rates varied significantly (e.g., CF-Suc gaps of 0.12 to 0.32) despite stable accuracy. Increasing regularization shifted the mean boundary distance, which correlated monotonically with lower counterfactual success.
Predictive Modeling of Behavior: Regression models using only accuracy or cross-entropy loss explained little variance in counterfactual outcomes. However, models incorporating $dbd(z)$ and $r_k(z)$ achieved significantly higher $R^2$ scores. Specifically, adding local data support to boundary proximity consistently improved predictive power across all datasets.
Supported vs. Unsupported Flips: The study distinguishes between "supported flips" (reaching a target region with strong local data support) and "unsupported flips" (crossing a boundary into sparse regions). Effective counterfactual behavior was found to depend heavily on reaching supported regions, not merely crossing a boundary.
Improved Search via Geometry: In a case study on MNIST, augmenting the counterfactual search objective with geometry-aware priors (penalizing distance to target-class neighbors) successfully improved search outcomes where standard baselines failed, demonstrating the practical utility of these geometric signals.

Significance and Claims
The paper claims that counterfactual behavior constitutes a distinct dimension of model evaluation, separate from predictive performance and margin-based robustness. The authors argue that:

Counterfactual feasibility is not invariant across functionally similar models; it is critically dependent on how the classifier places boundaries relative to local data structure.
Standard predictive metrics (accuracy, loss) are insufficient for auditing counterfactual explanations or recourse systems.
Understanding the interaction between boundary proximity and local support is essential for model selection, robustness analysis, and designing more reliable counterfactual methods.
The findings suggest that future counterfactual methods should explicitly account for local data support to ensure generated explanations lie in regions supported by the data distribution.

The work concludes that while these geometric factors explain variation in current linear-head setups, extending this perspective to end-to-end nonlinear models and developing geometry-aware counterfactual objectives remains a direction for future research.