🤖 AI

Global Geometry Is Not Enough for Vision Representations

This paper demonstrates that global embedding geometry is insufficient for predicting compositional binding in vision representations, revealing instead that functional sensitivity, measured via input-output Jacobians, is a critical and reliable indicator of a model's ability to compose visual elements.

Original authors: Jiwan Chung, Seon Joo Kim

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiwan Chung, Seon Joo Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: It's Not Just About the Ingredients, It's About the Recipe

Imagine you are trying to teach a robot to recognize a sandwich.

The Old Way (Global Geometry): You tell the robot, "Make sure your list of ingredients is perfectly balanced. You should have an equal amount of bread, cheese, and ham, and no single ingredient should dominate the list." The robot creates a perfect, balanced list of what is there.
The Problem: The robot still doesn't know if the cheese is on top of the ham or underneath it. It knows the ingredients exist, but it doesn't understand how they are arranged.

This paper argues that most current AI vision models are like that robot. They are great at listing what is in a picture (global geometry), but they are terrible at understanding how those things are put together (compositional binding).

The Experiment: The "Shape Puzzle"

To test this, the researchers created a simple, fake puzzle.

The Setup: They showed the AI a "query" image with a red circle on the left and a green square on the right.
The Test: They asked the AI to find a matching image from a group of options.
The Trick: The matching images used completely different colors (e.g., blue circle, yellow square). The AI couldn't cheat by matching colors; it had to understand the structure: "Circle on the left, square on the right."

They tested 26 different AI models (like CLIP, DINO, and others) on this puzzle.

The Discovery: The "Map" vs. The "Engine"

The researchers looked at two ways to measure how good these AI models were:

1. The Map Check (Global Geometry)
This measures how evenly the AI spreads out its knowledge. Imagine a map where all the cities are spread out perfectly so no two are too close.

The Result: The researchers checked this "map" for all 26 models. They found zero connection between how "perfectly spread out" the map was and how well the model solved the shape puzzle. A model could have a perfect map and still fail the puzzle.

2. The Engine Check (Functional Sensitivity)
This measures how the AI reacts when you slightly tweak the input. Imagine pushing a car. Does the car move in many different directions depending on where you push, or does it only move forward no matter what?

The Result: The researchers measured this "sensitivity" (using something called the Jacobian Effective Rank). They found a strong connection: Models that were sensitive to many different directions (like a car that can drive forward, backward, and sideways) were the ones that actually solved the shape puzzle.

The "Why": How the Training Rules Shape the AI

Why do some models have this "engine" and others don't? It comes down to the rules (loss functions) the AI was trained with.

The "Variance Decorrelation" Rule (The Winners): Some models (like Barlow Twins) were trained with a rule that says, "Make sure every part of your brain reacts differently to every part of the image." This forces the AI to be sensitive to many directions, giving it the "engine" it needs to understand structure.
The "Contrastive" or "Masked" Rule (The Losers): Most popular models (like CLIP or DINO) were trained to just match similar pictures or fill in missing parts. These rules only care about the big picture (global geometry). They don't care about the fine details of how the image reacts to small changes. So, the AI's "engine" collapses; it becomes stiff and only reacts in a few specific ways, making it bad at understanding structure.

The Takeaway

Don't just look at the map: Just because an AI's internal representation looks "perfect" and balanced (good global geometry) doesn't mean it understands the world.
Check the engine: To know if an AI can understand how things are put together (like a red car next to a blue house), you need to check if it is sensitive to changes in its input (functional sensitivity).
The Fix: If we want AI that truly understands composition, we need to change the training rules to force the AI to be sensitive to local details, not just global balance.

In short: You can have a perfectly organized library (Global Geometry), but if the librarian doesn't know how to find a specific book based on its location on the shelf (Functional Sensitivity), the library isn't very useful. This paper tells us to stop just organizing the library and start training the librarian to navigate the shelves.

Technical Summary: Global Geometry Is Not Enough for Vision Representations

1. Problem Statement

A prevailing assumption in visual representation learning is that globally well-distributed embeddings—characterized by properties such as isotropy, uniformity, and high effective dimensionality—support robust and generalizable representations. This geometric perspective has shaped training objectives (e.g., contrastive alignment, variance decorrelation) and evaluation protocols, treating global geometry as a proxy for representational competence.

However, visual scenes are defined not only by the presence of elements but by their compositional binding (i.e., how elements are arranged and related). While global geometry effectively encodes which features are present, it may be insensitive to how they are composed. This paper investigates whether standard geometric metrics can predict a model's ability to perform compositional binding, or if this capability relies on a different, unmeasured axis of representational quality.

2. Methodology

Experimental Setup

The authors evaluated 26 pretrained vision encoders spanning seven distinct training objective families:

Variance Decorrelation: Barlow Twins, VICReg
Masked Modeling: MAE, BEiT, BEiTv2
Clustering/Distillation: DINO, DINOv2, SwAV
Contrastive: MoCo v3
Vision-Language: CLIP, SigLIP, EVA-CLIP
Supervised: ConvNeXt, ViT

To isolate compositional capability from confounding factors like texture recognition or object detection, the authors designed a synthetic Attribute Binding benchmark.

Task: Models must match a query image to a target image based on shape-position bindings (e.g., "red circle left of blue square").
Constraint: Queries and targets use disjoint color sets to prevent reliance on color co-occurrence, forcing the model to rely on abstract spatial relations.
Control: A "Same/Different" probe was used to measure basic structural discrimination, distinguishing it from true relational binding.

Metrics Evaluated

Geometric Metrics (Static):
- Global Participation Ratio (G.PR): Measures global effective dimensionality.
- Global Isotropy Score (G.Iso): Measures variance distribution beyond the top principal component.
- Local Isotropy (L.Iso): Measures the isotropy of local neighborhoods in the embedding space.
Functional Metrics (Dynamic):
- Jacobian Effective Rank (JER): Measures the effective rank of the input-output Jacobian ( $J_f(x) = \partial f(x) / \partial x$ ). This quantifies the dimensionality of local variations in the representation space induced by input perturbations, characterizing functional sensitivity.
Readout Analysis:
- The authors tested whether alternative geometry-based readouts (kNN, Local PCA) could recover binding information from frozen embeddings when cosine similarity failed.

3. Key Results

Geometric Metrics Fail to Predict Binding

Zero Correlation: Standard geometric metrics (G.PR, G.Iso, L.Iso) exhibited near-zero correlation with compositional binding accuracy across all 26 models (e.g., $r \approx 0.11$ for G.Iso, $p > 0.5$ ).
Orthogonality: High-quality geometric properties (e.g., high isotropy) do not translate to functional binding performance. Models with excellent global geometry (like DINOv2 or CLIP) performed at near-chance levels on the binding task.
Robustness: This null result held across various hyperparameter choices (neighborhood sizes $k$ ) and metric formulations (Effective Rank, Participation Ratio).

Functional Sensitivity Predicts Binding

Strong Correlation: The Jacobian Effective Rank (JER) showed a strong positive correlation with binding accuracy ( $r = 0.69, p < 0.001$ ).
Predictive Power: A bivariate regression combining JER and "Same/Different" discrimination accuracy explained 78% of the variance in binding performance ( $R^2 = 0.78$ ).
Objective Dependence:
- Variance-Decorrelation objectives (Barlow Twins, VICReg) maintained high JER (~~28.8) and achieved the highest binding accuracy (~~44%).
- Contrastive and Vision-Language objectives (CLIP, DINOv2) exhibited "rank collapse" in their Jacobian spectra, resulting in low JER (~~18–20) and near-chance binding accuracy (~~13%).
- Supervised models showed the lowest JER (~14.1).

Mechanism of Rank Collapse

Spectral Analysis: Variance-decorrelation models maintain a flat, slowly decaying singular value spectrum, indicating uniform sensitivity across input directions. In contrast, models like CLIP and DINOv2 show rapid spectral decay, concentrating sensitivity in a few dominant modes.
Depth Analysis: Rank collapse occurs at different stages depending on the objective. CLIP and DINOv2 exhibit mid-network collapse (blocks 5–8), while MAE maintains high rank in the backbone but collapses sharply at the projection layer.
Readout Limitations: Alternative geometric readouts (kNN, Local PCA) failed to improve binding accuracy for models with low JER, suggesting binding information is not recoverable through post-hoc geometric manipulation.

Generalization

The correlation between JER and binding performance generalized to natural images (SugarCrepe benchmark) within the CLIP family, with JER predicting performance ( $r = 0.69$ ) even after controlling for zero-shot ImageNet accuracy.

4. Theoretical Explanation

The paper posits that the disparity between geometric regularity and functional sensitivity arises from objective design:

Variance-Decorrelation: These objectives explicitly constrain second-order feature statistics (covariance) via augmentation. This forces the loss to directly shape the Jacobian ( $J(x)\Sigma_\delta J(x)^\top$ ), preserving sensitivity across multiple input directions and maintaining high JER.
Contrastive/Clustering/Masked/Semantic: These objectives constrain the Jacobian only through rank-deficient projections (e.g., gradients of similarity or reconstruction errors). They leave directions orthogonal to the specific task (e.g., semantic alignment or instance discrimination) unconstrained, leading to a collapse in functional rank and a loss of compositional sensitivity.

5. Significance and Claims

The authors claim that global embedding geometry captures only a partial view of representational competence.

Limitation of Current Metrics: Widely used geometric diagnostics reliably track performance on tasks driven by single-factor variation (e.g., feature presence) but systematically fail on tasks requiring genuine compositional structure.
New Diagnostic Axis: The paper establishes functional sensitivity (measured via Jacobian Effective Rank) as a critical, complementary axis for modeling composite structure.
Intrinsic Property: Linear accessibility of compositional structure is shown to be an intrinsic property induced by the pretraining objective, rather than a feature that can be recovered through alternative geometric readouts.

The work does not propose a new training algorithm but offers a diagnostic framework to understand why certain objectives (like variance decorrelation) succeed in compositional tasks while others (like contrastive learning) fail, despite both producing "well-conditioned" global geometries.