🤖 AI

Vector Linking via Cross-Model Local Isometric Consistency

This paper introduces Vector Linking, a method that leverages the local geometric consistency of independently trained contrastive encoders to iteratively recover cross-model object correspondences from a tiny seed set by matching distance-based hash representations and aggregating evidence via a Beta-Bernoulli posterior.

Original authors: Ziying Chen, Yang Cao, He Sun, Beining Yang, Tianjian Yang

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziying Chen, Yang Cao, He Sun, Beining Yang, Tianjian Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two massive libraries of books. One library was cataloged by Librarian A, and the other by Librarian B. Both librarians are brilliant, but they have very different ways of organizing things.

Librarian A might group books by the color of their covers.
Librarian B might group them by the smell of the paper.

In the world of AI, these "libraries" are vector databases (collections of data points), and the "librarians" are AI models that turn text or images into lists of numbers (vectors). The problem is that even if both librarians have the exact same book in their collection, they might place it in completely different spots on the shelf. You can't just walk from one library to the other and find the matching book because the maps don't line up.

This paper introduces a solution called Vector Linking. Here is how it works, using simple analogies:

The Core Problem: The "Map Mismatch"

Usually, if you want to merge these two libraries, you'd need to ask the librarians to reorganize their shelves together. But in this scenario, the librarians are "black boxes"—you can't talk to them, you can't see their notes, and you can't ask them to change their rules. You only have the books (the vectors) sitting on the shelves.

Furthermore, the libraries only partially overlap. They share some books, but each also has thousands of unique books the other doesn't have. Traditional methods try to force a single, global map to fit both libraries, but because the libraries are organized so differently, this usually fails.

The Big Discovery: "Local Neighborhoods" Stay the Same

The authors noticed something magical about how these AI librarians work. While their global organization is totally different (Librarian A thinks Book X is far away from Book Y, while Librarian B thinks they are neighbors), their local neighborhoods are surprisingly consistent.

The Analogy: Imagine two different maps of a city.

Global View: On Map A, the city looks like a circle. On Map B, it looks like a square. The distance from the North Pole to the South Pole is totally different on both maps.
Local View: However, if you zoom in on a single street corner, the distance between the bakery and the coffee shop is almost the same on both maps, just maybe scaled up or down by a tiny bit.

The paper proves that for AI models trained to compare things (called "contrastive encoders"), this "local consistency" is a mathematical rule. If two items are very similar, their distance in both AI systems will be proportional, even if the systems are completely different.

The Solution: "Geometric Hashing" with a Seed

To connect the two libraries without asking the librarians for help, the authors propose a method called Geometric Embedding Hashing (GEH).

Think of it like a game of "Hot and Cold" with a tiny group of friends.

The Seed (The Tiny Group): You start with a very small list of "Anchor Books" that you know are the same in both libraries (e.g., you know "The Great Gatsby" is in both). Let's say you only have 15 of these.
The Signature (The Distance Map): For every other book in Library A, you measure its distance to those 15 Anchor Books. This creates a unique "signature" or "fingerprint" based on how far it is from your friends. You do the same for Library B.
The Magic: Because of the "local neighborhood" rule, if a book in Library A is the same as a book in Library B, their signatures (distances to the anchors) will look almost identical, even though the libraries are organized differently.
The Voting System (Bootstrapping):
- Sometimes, a random book might accidentally look like a match just by chance (a "false alarm").
- To fix this, the system doesn't just use the 15 anchors once. It creates hundreds of different "views" by picking different random groups of anchors.
- If a book pair is a real match, it will keep showing up as a match in almost every view. If it's a false alarm, it will only show up once or twice.
- The system counts the votes. If a pair gets enough votes, it's declared a match.
Growing the Team: Once the system finds a new confident match, it adds that pair to the list of "Anchors." Now it has 16 anchors. It repeats the process, using the new anchor to find more matches, growing the list of known connections like a snowball rolling down a hill.

Why This Matters

The paper shows that with just a tiny seed (as few as 15 known pairs), this method can accurately link millions of items between two completely different AI systems.

It works with "Black Boxes": You don't need to know how the AI models work inside; you just need the numbers they produced.
It handles "Partial Overlap": It doesn't get confused by the millions of books that only exist in one library.
It's Robust: It filters out the "noise" (false matches) by using the voting system.

Real-World Uses Mentioned in the Paper

The authors specifically demonstrate two uses for this "bridge" they built:

Merging Vector Databases: You can now combine two separate search engines (one built by Company A, one by Company B) into one unified system so you can search them all at once.
Cross-Model Clustering: You can group similar items together even if they were processed by two different AI models, creating a single, cohesive list of topics or categories.

In short, the paper provides a universal translator for AI "languages," allowing different systems to recognize each other's data without needing to speak the same dialect or reveal their internal secrets.

Technical Summary: Vector Linking via Cross-Model Local Isometric Consistency

Problem Definition: Vector Linking
The paper addresses the problem of Vector Linking: recovering cross-model object correspondences between two embedding clouds ( $E_1$ and $E_2$ ) generated by different black-box contrastive encoders ( $f_1$ and $f_2$ ) over partially overlapping datasets ( $O_1$ and $O_2$ ). The setting is strictly post-hoc and black-box; the method has access only to the static embedding vectors and a small "seed set" of paired anchors ( $S$ ) known to correspond to the same underlying objects. Crucially, the overlap between the datasets is unknown and partial, meaning there is no global bijection between the two embedding spaces, and non-overlapping regions cannot be treated as simple outliers. This interoperability gap hinders unified retrieval, cross-index deduplication, and vector database integration.

Theoretical Foundation: Local Isometric Consistency
The core theoretical insight driving the work is the discovery of local geometric consistency across independently trained contrastive encoders.

Empirical Observation: The authors observe that while long-range distances between embeddings decorrelate quickly across different models, short-range distances remain strongly correlated.
Theoretical Justification: By analyzing a localized alignment-uniformity surrogate for contrastive learning, the paper proves that independently trained encoders induce locally isometric metrics up to a scale factor. Specifically, for two locally optimal encoders $f_1$ and $f_2$ at a point $x$ , and for any nearby point $y$ , the distance relationship holds:
$\|f_1(x) - f_1(y)\| = \kappa \cdot \|f_2(x) - f_2(y)\| + O(d_M(x, y)^2)$
where $\kappa$ is a scale factor and $d_M$ is the intrinsic geodesic distance. This implies that local neighborhoods are stable across models, whereas global arrangements are distorted.

Methodology: Geometric Embedding Hashing (GEH)
Building on the local consistency property, the authors propose Geometric Embedding Hashing (GEH), an iterative, reference-based algorithm that recovers vector links from a tiny seed set without accessing model internals or raw data.

Distance-to-Anchor Signatures: Instead of aligning global spaces, GEH represents each vector by its distances to a set of paired anchors. To handle the unknown scale factor $\kappa$ , these signatures are normalized (scale-free), typically using cosine similarity on the distance vectors.
Multi-View Voting: A single set of anchors is insufficient for all points due to the locality requirement. GEH samples many small, diverse geometric "views" (subsets of anchors) from the current pool of paired anchors.
- View Generation: Views are constructed using Furthest Point Sampling (FPS) to ensure anchor diversity and stability.
- Kernelized Hashing: To emphasize short-range consistency and suppress long-range noise, raw distances are transformed via a monotone kernel (e.g., $e^{-dist/\sigma}$ ).
- Candidate Generation: Within each view, candidate links are proposed via Mutual Nearest Neighbor (MNN) search in the hash space, using CSLS to mitigate hubness.
Posterior-Guided Bootstrapping:
- Evidence Aggregation: Matches across different views are treated as noisy votes. The paper models the success probability of a link using a Beta–Bernoulli posterior. The posterior mean serves as a confidence score.
- Iterative Growth: High-confidence links (those with high posterior scores) are promoted to become new anchors for the next iteration. This bootstrapping process expands the initial tiny seed set into a large correspondence set.
- Adaptive Thresholding: An Otsu threshold is applied to the posterior scores to separate true links from spurious collisions caused by model-specific distortion.

Key Contributions

Problem Formulation: Formalized the "Vector Linking" problem under partial, unknown overlap and strict black-box constraints.
Theoretical Insight: Established both empirically and theoretically that contrastive encoders exhibit local distance consistency, forming the basis for encoder-invariant hashing.
Algorithm: Developed GEH, a multi-view geometric hashing algorithm with posterior-guided bootstrapping that accurately recovers links using only 15–30 seed pairs.
Applications: Demonstrated the utility of recovered links for vector database integration and cross-model clustering.

Experimental Results
The method was evaluated across multiple BEIR benchmarks (NFCorpus, SciFact, ArguAna, SciDocs, FiQA, FEVER) and five pairs of embedding models (including Mistral, OpenAI, GTE, Qwen, and KaLM).

Performance: GEH consistently outperformed eight baselines (including Linear, CCA, MLP, RCSLS, Procrustes, and Unbalanced Gromov-Wasserstein). For instance, with only 15 seed pairs, GEH achieved over 90% recall on the FiQA dataset for Mistral vs. OpenAI, whereas the best baseline achieved only ~20% recall.
Robustness: The method remained robust across varying overlap ratios (15%–40%), seed budgets, and even when using out-of-domain (OOD) seeds drawn from different datasets.
Scalability: On the FEVER dataset (5.4 million documents), GEH achieved 93.8% precision and 68.9% recall in roughly 3328 seconds, demonstrating scalability comparable to or faster than linear baselines despite its iterative nature.
Downstream Impact: In vector database integration tasks, GEH-enabled unified retrieval approached the theoretical upper limit of using a single model. In cross-model clustering, it recovered consistent cluster assignments for 75–98% of overlapping objects.

Significance
The paper argues that vector linking is a practical primitive for embedding interoperability. By leveraging the inherent local geometric consistency of contrastive encoders, it bypasses the need for global alignment assumptions (which fail under partial overlap) and retraining. This enables the integration of disparate vector databases and the joint analysis of embeddings from different models, addressing a critical bottleneck in the rapidly evolving landscape of embedding-based information systems.