Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers
This paper reveals that contrastively pretrained vision-language models contain a substantial, invariant subspace of shared multi-modal noise that can be safely pruned without harming downstream performance, thereby offering new mechanistic insights into their latent representational structure.
Original authors:Jakub Grzywaczewski, Dawid Płudowski, Przemysław Biecek
Original authors: Jakub Grzywaczewski, Dawid Płudowski, Przemysław Biecek
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a super-smart robot assistant (like CLIP) that has been trained to understand both pictures and words. You ask it, "Show me a picture of a dog," and it finds one instantly. It seems perfect. But this paper argues that this robot is actually carrying around a heavy, useless backpack full of junk that it doesn't need to do its job.
Here is the breakdown of what the researchers found, using simple analogies:
1. The "Static" in the Signal
Think of the robot's brain as a giant radio receiver with 768 different channels (dimensions) listening to the world.
The Good Channels: Most of these channels are tuned to the "signal"—the actual meaning of the picture or the word (like "dog," "church," or "happy").
The Bad Channels: The researchers discovered that about 164 of these channels are just broadcasting "static." This isn't random noise; it's a specific, shared kind of static that appears in every picture and every word the robot sees, regardless of what the content is.
2. The "Ghost" Pattern
The most surprising part is that this "static" is identical across different groups.
The Analogy: Imagine you take a photo of a Siberian Husky and a photo of a Church building. They have nothing in common. Yet, if you look at the "junk" part of the robot's memory for both images, the junk looks almost exactly the same (91% overlap).
The Finding: The robot has a "ghost" pattern that gets stamped onto everything it sees. It's like if a printer had a smudge on the lens; every single document it prints, whether it's a recipe or a love letter, would have that same smudge in the corner. The researchers found that this smudge is so consistent that they can map it out perfectly.
3. The "Junk Drawer" Experiment
The researchers decided to test what happens if they simply throw away this "junk drawer" (the 164 dimensions of noise).
The Result: They took the robot, removed those 164 channels, and asked it to do its usual tasks (like identifying animals in photos or matching words to images).
The Surprise: The robot didn't get worse. In fact, it got slightly better at matching words to images. It was like taking a backpack full of rocks off a runner; they didn't slow down, they actually ran faster because they weren't weighed down.
4. Why Does This Happen?
The paper suggests that this "noise" isn't a bug in the code, but a side effect of how the robot was built.
The Analogy: Imagine a factory that builds cars. The engineers designed the assembly line so efficiently that the cars are great, but the vibration of the machinery accidentally imprints a tiny, useless scratch on every single car door. The scratch doesn't stop the car from driving, but it's there on every single one.
The researchers found that this "scratch" (the noise) is a fundamental part of the architecture, not just a mistake with one specific dataset.
Summary
The paper claims that modern AI models like CLIP are carrying around a massive amount of "shared noise" (about 164 dimensions in some versions) that has nothing to do with the actual meaning of the images or text.
The Good News: You can cut this noise out without hurting the AI's performance.
The Big Picture: A huge chunk of the AI's "brain space" is actually just storing this repetitive, non-meaningful pattern, rather than useful knowledge. By cleaning it out, the AI becomes slightly more efficient and accurate.
Technical Summary: "Your CLIP has 164 dimensions of noise"
Problem Statement Contrastively pre-trained Vision-Language Models (VLMs), such as CLIP and SigLIP, are widely deployed as feature extractors for tasks ranging from zero-shot classification to cross-modal retrieval. However, these shared latent spaces exhibit structural anomalies, acting as repositories for non-semantic, multi-modal noise. While prior research has identified phenomena like the "cone effect," the "modality gap," and "dimensional collapse" in single-modal models, there is a need to understand how these models encode irrelevant data within their high-dimensional latent spaces. The authors posit that a significant portion of the latent geometry in modern VLMs is governed by shared, architecture-level noise rather than task-relevant semantics.
Methodology The authors employ spectral decomposition of covariance matrices to analyze the geometry of VLM latent spaces.
Covariance Estimation: They compute empirical covariance matrices (Σtext and Σimg) for text and image embeddings separately, derived from large-scale datasets (ImageNet-1K and LAION-2B). An average cross-modal covariance matrix (Σavg) is defined to analyze the joint latent space.
Spectral Decomposition: The matrices are decomposed to identify eigenvalues and eigenvectors. The authors observe a consistent, sharp drop in eigenvalues across various models (CLIP and SigLIP variants) beyond a specific threshold (approximately 10−3.6).
Noise Thresholding: Using the "elbow method" on Σavg, they identify a cutoff threshold to separate the "signal" (top eigenvalues) from the "noise" (lower bulk of eigenvalues). Dimensions falling below this threshold are identified as the "shared noise subspace."
Subspace Alignment: To verify the universality of these noise dimensions, the authors compute the Mean Squared Cosine of Subspace Angles (mSCSA) between the globally extracted noise vectors and the low-variance eigenvectors of specific data subsets (e.g., individual ImageNet classes).
Pruning and Evaluation: They construct a noise-removing projection matrix (P=I−VVT) to project embeddings onto the orthogonal complement of the noise subspace. They evaluate the impact of this pruning on downstream tasks: ImageNet zero-shot classification and LAION-2B image-text alignment.
Key Results
Existence of Shared Noise: The analysis reveals a distinct "noise subspace" comprising the lower bulk of the covariance eigenvalues. For CLIP ViT-L/14, this corresponds to 164 dimensions (21% of the 768-dimensional space).
Subgroup Invariance: The identified noise dimensions exhibit strong invariance across distinct data subsets. The mSCSA scores between the global noise subspace and class-specific low-variance subspaces are high (often >90% for larger models), indicating that these noise directions are ubiquitous and not specific to any single class or data subset.
Impact of Pruning:
Classification: Removing the noise dimensions results in a negligible drop in ImageNet zero-shot Top-5 accuracy (e.g., CLIP ViT-L/14 drops from 91.2% to 91.1%). Randomly removing the same number of dimensions yields significantly worse performance, confirming that the noise dimensions carry little semantic information.
Alignment: Contrary to the hypothesis that pruning might harm alignment, the authors observe a systematic increase in average cosine similarity for image-text pairs on LAION-2B after noise removal. This improvement scales with the size of the latent dimension.
Qualitative Inspection: Visual inspection of samples with high activation in the noise subspace reveals that they often correspond to unavailable image placeholders or out-of-distribution artifacts (e.g., Marvel movie topics when estimated on ImageNet alone), further supporting the non-semantic nature of these dimensions.
Significance and Claims The paper claims to provide new mechanistic insights into the representational structure of modern VLMs. By formally decomposing the latent space into multi-modal semantic signals and a shared noise subspace, the authors demonstrate that:
A substantial fraction of the latent geometry in contrastively pretrained models is governed by shared, architecture-level noise rather than task-relevant semantics.
Pruning these shared noise dimensions is "mainly harmless" and can actively improve downstream alignment performance without degrading classification capabilities.
The phenomenon of dimensional collapse persists in modern VLMs but manifests as a repository for irrelevant data rather than a total collapse of the space.
The authors maintain a modest stance, acknowledging that the optimal noise threshold may depend on specific model architectures and data distributions. They note that while the eigenspectrum dip is sharp, it is not as instantaneous as the phase transition observed in previous single-modal studies, suggesting a complex underlying mechanism for latent capacity allocation. Future work is proposed to expand this evaluation to a broader spectrum of backbones and to establish a formal theoretical framework linking this noise mechanism to the emergence of the modality gap.