💻 computer science

GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations

GraSP-VL introduces a learnable, shared near-orthogonal prefix transform that reorganizes frozen vision-language embeddings into a "Semantic Matryoshka" interface, enabling controllable access to semantic granularity from coarse to fine by simply varying the embedding length while preserving the original model's full-dimensional geometry and zero-shot performance.

Original authors: Zesheng Li, Chengchang Pan, Honggang Qi

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zesheng Li, Chengchang Pan, Honggang Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart library book (a Vision-Language Model) that knows everything about pictures and words. When you ask it a question, it gives you a single, massive "summary card" (an embedding vector) that contains all the information at once: the object, the color, the action, the mood, and the whole story.

The problem is that this summary card is always the same size. If you just want to know "Is there a dog?", you have to read the whole 500-page book. If you want to know "Is the dog brown?", you still have to read the whole book. It's like trying to find a specific ingredient in a soup by tasting the entire pot every time, even if you just want to know if there's salt in it.

GraSP-VL is a new method that turns this "all-or-nothing" summary card into a Russian Nesting Doll (Matryoshka) of information.

Here is how it works, using simple analogies:

1. The "Nesting Doll" Interface

The authors realized that the information inside that big summary card is actually layered, but it's all jumbled up. They created a special tool (a "Shared Orthogonal Transform") that rearranges the card without changing the total amount of information.

Think of it like organizing a messy desk:

The Short Prefix (The Top Layer): If you only look at the first few inches of the card, you only see the main object (e.g., "Dog"). It's a coarse, big-picture view.
The Medium Prefix (The Middle Layer): If you look a bit deeper, you start seeing attributes (e.g., "Brown Dog").
The Long Prefix (The Deeper Layer): If you go even deeper, you see actions and relationships (e.g., "Dog digging a hole").
The Full Card (The Bottom Layer): If you read the whole thing, you get the full caption (e.g., "A brown dog digging a hole in front of a plant").

The magic is that you can choose how deep you want to dig. If you just need to find a dog, you stop at the top layer. If you need to find a brown dog, you go a little deeper. You don't have to process the whole thing unless you need the whole story.

2. The "Magic Rearranger" (The Transform)

How did they do this? They didn't rewrite the book or change the author's original words. Instead, they built a magic rearranger.

Imagine the original summary card is a deck of cards where the "Dog" info is mixed with "Brown" info and "Digging" info in a random order. GraSP-VL is a shuffle that moves all the "Dog" cards to the top, all the "Brown" cards to the middle, and all the "Digging" cards to the bottom.

Crucially, this shuffle is perfect. It doesn't lose any information, and it doesn't change the relationship between the cards when you look at the entire deck. It just changes the order so that the "top" of the deck tells you something specific, and the "bottom" tells you something else.

3. The "Contract"

The paper calls this a "Semantic Matryoshka." It's a contract between the user and the computer:

User: "I promise to stop reading at length X if I only want object-level info."
Computer: "I promise that if you stop at length X, you will only see object-level info, and nothing about colors or actions."

Without this method, stopping early usually just gives you a weak, confused version of the whole picture. With GraSP-VL, stopping early gives you a clean, specific answer.

4. The Results (The Proof)

The authors tested this on thousands of images and captions (like dogs, cats, and people doing things).

Before: If they just cut the card short, the computer got confused. It couldn't tell the difference between a "brown dog" and a "black dog" if they stopped reading too early.
After: With GraSP-VL, when they stopped at the "short" length, the computer was great at spotting the object (the dog) but ignored the color. When they went a bit longer, it suddenly became great at spotting the color. When they went even longer, it understood the action.

They also proved that this didn't break the original model. If you read the whole card after the shuffle, it still knows exactly what it knew before. The "whole picture" accuracy stayed the same, but now you have the option to look at just the "top layer" for faster, simpler answers.

Summary

GraSP-VL takes a frozen, "one-size-fits-all" AI brain and gives it a dial. You can turn the dial to "Coarse" for quick, simple answers (just the object), or "Fine" for detailed answers (color, action, full story). It does this by reorganizing the information inside the brain so that the "top" of the data always holds the big picture, and the "bottom" holds the details, all without changing the brain's original knowledge.

Technical Summary: GraSP-VL

Problem Statement
Frozen vision-language models (VLMs), such as CLIP, generate embeddings that contain multi-granularity semantic signals ranging from object identity to attributes, relations, and full-caption meanings. However, these models expose these signals through a fixed-length vector interface. This creates an "interface mismatch": a single embedding vector is used for queries requiring different levels of semantic resolution (e.g., identifying a "dog" vs. a "brown dog running"). Current methods to shorten embeddings, such as truncation, projection, or compression, do not guarantee that the resulting shorter vectors correspond to specific semantic granularities. For instance, the first few dimensions of a frozen CLIP embedding are not inherently guaranteed to represent object-level semantics, nor do standard dimensionality reduction techniques like PCA understand the intended hierarchy of entities, attributes, and relations.

Methodology: GraSP-VL
The authors propose GraSP-VL (Granularity-Selective Prefixes for Vision-Language representations), a method designed to turn embedding length into a controllable semantic access interface without retraining the underlying VLM.

Shared Near-Orthogonal Transform: GraSP-VL learns a single, shared transform $F_\theta$ applied to both image and text embeddings from a frozen VLM. The transform is instantiated as an orthogonal matrix $R$ parameterized via a Cayley transform ( $R = (I + A_\theta)^{-1}(I - A_\theta)$ ), where $A_\theta$ is a skew-symmetric matrix derived from trainable parameters. This ensures the transform is near-isometric, preserving the cosine similarity of the full-dimensional space exactly ( $\langle Re_a, Re_b \rangle = \langle e_a, e_b \rangle$ ).
Semantic Matryoshka Interface: The method implements a "Semantic Matryoshka" principle. Instead of treating prefix length merely as a compression budget, it assigns specific semantic roles to different prefix lengths via an explicit interface contract:
- Short prefixes (e.g., $D/16$ ) are assigned coarse roles (object identity).
- Longer prefixes (e.g., $D/8, D/4, D/2$ ) progressively expose finer distinctions (attributes, relations/actions, full captions).
Granularity-Selective Objective: The training objective combines five coupled terms to enforce this structure:
1. Multi-Granularity Alignment ( $L_{align}$ ): Aligns each prefix length to its assigned language view (object, attribute, relation, full caption).
2. Cumulative Retention ( $L_{ret}$ ): Ensures longer prefixes retain the coarse semantics of shorter ones, preventing information replacement.
3. Selective Hard-Negative Ranking ( $L_{rank}$ ): Uses typed hard negatives (object, attribute, relation, action, order, full) to enforce that specific distinctions are only resolvable at or after their assigned prefix boundary.
4. Early-Prefix Invariance ( $L_{inv}$ ): Discourages premature separation of positive/negative pairs before the assigned prefix boundary, ensuring short prefixes remain coarse and do not overfit to fine-grained details.
5. Full-Space Preservation ( $L_{pres}$ & $L_{ortho}$ ): Monitors that the full-dimensional geometry remains unchanged.

Key Contributions

GraSP-VL Framework: A shared prefix transform for frozen VLMs that reorganizes access to existing signals, exposing coarse-to-fine semantics via prefix length while preserving the original full-space geometry.
Semantic Matryoshka Principle: A formalization where embedding length is assigned semantic granularity by an explicit contract rather than relying on discovered natural boundaries.
Granularity-Selective Objective: A novel loss function combining alignment, retention, hard-negative discrimination, early-prefix invariance, and geometry preservation.
Diagnostic Protocol: A rigorous evaluation framework that separates semantic selectivity from raw truncation, PCA, generic adaptation, and coordinate permutations, using a "falsification chain" to validate the necessity of the learned interface.

Experimental Results
The method was evaluated on a 20,147-example pool from COCO and Flickr30K, with additional transfer tests on SugarCrepe-clean and CIFAR-100.

Staircase Performance: GraSP-VL achieved a staircase score of 53.01 and a hard-negative selectivity of 89.76%. This significantly outperformed baselines including direct truncation, PCA, random rotations, MRL-style, Matryoshka-style, and SMEC-style compression methods.
Geometry Preservation: The method maintained full-space drift below $10^{-6}$ , effectively preserving the original VLM's decision surface for full-dimensional queries.
Selectivity vs. Drift: Unlike generic MLP adapters which achieved high selectivity but rewrote the full embedding space (high drift), GraSP-VL achieved high selectivity with near-zero drift.
Transferability:
- On SugarCrepe-clean (an external benchmark not used in training), GraSP-VL achieved 86.03% object accuracy and 11.96 mean external emergence.
- It preserved full-dimensional zero-shot accuracy on CIFAR-100.
- The approach generalized across different backbones (OpenAI CLIP, OpenCLIP, EVA-CLIP, SigLIP) with varying dimensions.
Baselines: The study demonstrated that raw coordinates, PCA, and learned permutations fail to produce the intended semantic ladder. Only the dense, near-isometric mixing provided by GraSP-VL successfully organized the signals.

Significance and Claims
The paper positions GraSP-VL as an interface engineering solution rather than a discovery of universal semantic breakpoints. The authors claim:

Frozen VLM embeddings contain sufficient multi-granularity signals to be reorganized into a controllable prefix interface.
The "Semantic Matryoshka" interface allows users to select a prefix length corresponding to the required semantic resolution (entity, attribute, relation, or full caption) without retraining the base model or destroying the original embedding space.
This approach is particularly valuable for retrieval systems where compatibility with existing full-vector indexes and cached embeddings is critical, as the transform can be applied offline to gallery embeddings.
The authors explicitly state that the chosen dimensional breakpoints (e.g., $D/16$ for objects) are user-specified interface contracts, not necessarily natural constants inherent to the VLM. The method's success lies in realizing this contract while preserving the full geometry, rather than proving that the VLM naturally stores these signals in that specific order.