💻 computer science

Retrieve, Match, Escalate: Accurate and Scalable Product Linking with VLM-Distilled Cross-Encoders and Agentic VLMs

This paper presents a scalable, cost-efficient product linking system that employs a cascade architecture where a distilled cross-encoder resolves high-confidence matches and an agentic vision-language model handles ambiguous cases, thereby maximizing coverage while minimizing computational costs and human annotation requirements.

Original authors: Jian Wang, Steven Xu, Sanjyot Thete, Maryam Barouti, Tom Tang, Elaine Wu, Charu Sareen, Kyle MacDonald

Published 2026-08-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jian Wang, Steven Xu, Sanjyot Thete, Maryam Barouti, Tom Tang, Elaine Wu, Charu Sareen, Kyle MacDonald

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, chaotic marketplace of the modern internet, millions of independent sellers list the same items under different names, with different photos, and sometimes with missing or confusing details. A single product, like a specific brand of coffee maker, might appear in a catalog thousands of times, each entry slightly different from the last. For the computers that power search engines and recommendations, this creates a confusing fog. If the system cannot recognize that two different listings are actually the same item, it cannot combine their reviews, track their sales, or show them to the right customers. The goal of "product linking" is to clear this fog, acting as a digital librarian that sorts through the noise to find the single, correct identity for every item on the shelf. This is not just a matter of organization; it is the foundation of how online stores understand what they are selling.

A team of researchers at DoorDash has built a new system to solve this problem at an enormous scale, handling billions of records without breaking the bank. They realized that treating every single item the same way is a waste of resources. Some items are easy to identify because their names and codes match perfectly, while others are like twins separated at birth, requiring deep investigation to tell them apart. Instead of using a powerful, expensive computer brain to check every single item, they created a three-step process that spends more effort only when it is truly needed. First, the system quickly narrows down the possibilities to a small list of likely matches. Then, a fast, lightweight computer program checks these pairs. If the match is obvious, the system accepts it immediately. If the match is clearly wrong, it rejects it. But if the program is unsure, it escalates the difficult case to a more advanced artificial intelligence agent.

This advanced agent is the heart of the new discovery. Unlike the fast program that only reads text, this agent can look at product photos and, crucially, search the open internet for extra clues. When the system encounters a confusing pair of listings where the names are similar but the details are vague, the agent acts like a detective. It might look at a picture of a pot to see if it is made of cast iron or stainless steel, or it might search the web using a barcode number to find the manufacturer's official description. This ability to gather evidence from outside the original records allows the system to solve cases that would stump a simple text-matching tool. The researchers found that by using this "escalation" strategy, they could link nearly 77 percent of all products automatically, a significant jump from the 68 percent they could achieve with the cheaper, faster tools alone.

The team also discovered a surprising truth about how to train these systems. Instead of hiring thousands of humans to label millions of examples, they used two different advanced artificial intelligence models to grade the same items. They only kept the answers where both models agreed, creating a highly reliable set of training data. This method allowed them to teach the fast, lightweight program to make decisions with the same confidence as the expensive, slow one, but at a fraction of the cost. They also found that simply feeding the raw numbers of a barcode into the system worked better than giving it a pre-made "match" or "no match" flag, because the computer could learn to spot partial matches and errors on its own. However, they had to be careful, as relying too heavily on barcodes could lead to mistakes when two different products accidentally shared the same code. They solved this by teaching the system to double-check the product name whenever the barcode matched, ensuring it did not get fooled by rare errors.

By combining a fast filter with a smart, web-searching agent, the researchers created a system that is both accurate and affordable. They replaced a costly, closed-source artificial intelligence with a self-hosted version that costs about one-seventh as much to run, while still maintaining high accuracy. This approach means that online marketplaces can clean up their catalogs more thoroughly, ensuring that customers see a single, clear entry for every product rather than a scattered mess of duplicates. The result is a smarter, more efficient way to organize the world's goods, proving that sometimes the best way to solve a massive problem is not to use the biggest hammer for every nail, but to know exactly when to call in the expert.

Technical Summary: Retrieve, Match, Escalate

Problem Definition

The paper addresses product linking (entity resolution) at the scale of a large on-demand marketplace (DoorDash). The task involves mapping billions of noisy, multi-lingual, and multi-category merchant product records (SKUs) to a canonical catalog of tens of millions of products.

The core challenge is a cost-capability trade-off.

Scale: Resolving every candidate pair with a single, high-capacity model is computationally prohibitive.
Difficulty Variance: Many pairs are straightforward (e.g., exact barcode matches), while others are ambiguous due to missing data, conflicting attributes, or the need to distinguish subtle variants (e.g., size, flavor).
Data Quality: Merchant records often contain missing images, un-normalized barcodes, and free-text attributes, making simple text matching insufficient for the "hard tail" of ambiguous cases.
Labeling Bottleneck: Obtaining reliable human labels is difficult because product identity is conjunctive (consistent across brand, size, etc.), and annotating requires joint resolution of incomplete evidence.

Methodology

The authors propose a confidence-routed, three-stage cascade that allocates computation proportional to the difficulty of the decision.

1. Stage 1: Retrieval

Before matching, the system narrows the global catalog to a small candidate set ( $K=20$ ) per merchant record.

Mechanism: A vector index fuses three channels: text (item names/structured fields), images (product photos), and barcodes (GTIN/UPC).
Technique: Approximate Nearest Neighbor (ANN) search is used for text and images, combined with exact/partial barcode matching. Results are merged using Reciprocal Rank Fusion (RRF).
Goal: Ensure high recall; if the correct canonical product is not retrieved, it cannot be matched downstream.

2. Stage 2: Lightweight Cross-Encoder Matcher

This stage scores candidate pairs and routes them based on confidence.

Architecture: A text-only ModernBERT-base cross-encoder (150M parameters) with a sigmoid head. It concatenates serialized fields (category, name, GTIN/UPC lists, size, unit) from both records into a single 256-token sequence.
Input Strategy: Raw barcode digits are fed directly into the sequence rather than pre-computed equality flags, allowing the model to learn partial and noisy agreement patterns.
Training & Distillation: The model is trained on 5.3 million pairs generated via dual-VLM consensus. Two independent VLMs score each pair; only pairs where they agree are used as training labels. This replaces 40k human labels with 130x more consistent synthetic labels.
Routing Policy: The model outputs a calibrated score ( $s \in [0, 1]$ $s \in [0, 1]$ ).
- HIGH ( $s \geq \theta_{high}$ ): Auto-accept (precision $\geq 98\%$ ).
- LOW ( $s \leq \theta_{low}$ ): Auto-reject.
- MEDIUM: Escalate to Stage 3.

3. Stage 3: Agentic Multimodal VLM

This stage adjudicates the ambiguous "medium" band that Stage 2 cannot resolve.

Architecture: A self-hosted, open-weight Mixture-of-Experts (MoE) VLM (Qwen 3.6 35B-A3B).
Capabilities: The agent is agentic. It receives product records, images, and has access to a web search tool via an MCP interface. It follows a ReAct pattern, deciding when to search for external evidence (e.g., verifying a barcode or brand against a retailer site) to resolve conflicts absent from the records.
Optimization:
- Prompt Engineering: Specific addenda guide the model to treat empty search results as inconclusive (not proof of fabrication) and restrict identity proof to same-retailer evidence.
- Loop Control: The agent is capped at four tool rounds to prevent over-searching and latency bloat.
Cost Reduction: The system migrated from a closed frontier VLM (GPT-5.4) to the open-weight model, removing per-token vendor costs and allowing control over the search backend.

Key Contributions

Tool-Using Multimodal Inference: Formulating ambiguous product linking as a task requiring active evidence gathering (web search) and multimodal reasoning, implemented via a pretrained VLM with specific prompting.
Scalable Confidence-Routed Cascade: Demonstrating how to pair a lightweight, distilled cross-encoder with an agentic VLM. The cross-encoder resolves the high-signal majority efficiently, while the agent handles the difficult tail, optimizing the cost-accuracy trade-off.
Production Validation & Distillation Insights:
- Validated the pipeline at a production operating point (98% precision).
- Showed that an open-weight, self-hosted agent achieves precision parity with a closed frontier model at ~1/7th the cost (with a 4-point recall trade-off) without fine-tuning.
- Identified a specific failure mode in the cross-encoder (barcode over-indexing on corrupted identifiers) and resolved it via a second distillation pass using a regularized teacher, rather than manual patching.

Results

Retrieval: The retrieval stage achieves 93.06% recall on a deduplication slice (independent ground truth) and 99.12% on linked pairs.
Matcher (Stage 2): The distilled cross-encoder achieves 77.05% recall at 98% precision (R@P98) on a held-out set. It auto-accepts 43.7% of links in production at this precision bar.
Agent (Stage 3):
- vs. Humans: On hard medium-confidence tasks, the agent outperformed trained human operators by +13.7% accuracy, +18.5% recall, and +4.7% precision ( $p < 0.0001$ ).
- vs. Closed Model: The open-weight agent matches the closed frontier model's precision (98.0%) but has slightly lower recall (88% vs. 92%) at ~7x lower cost per pair.
End-to-End Impact: The cascade increases end-to-end link coverage from 68.1% (cheap stage only) to 77.1%, a 9.0 percentage point gain, by resolving the most ambiguous records.

Significance and Claims

The paper claims that labels are the bottleneck, not architecture. The results demonstrate that scaling the volume of high-consistency synthetic labels (via dual-VLM consensus) yields greater performance gains than increasing model capacity or hyperparameter tuning.

The system establishes a production operating point where:

Cost Efficiency: Computation is spent only where necessary. The per-pair cost spans five orders of magnitude, but the cascade ensures the expensive agent is only invoked for the "hard tail."
Robustness: The pipeline handles noisy, heterogeneous data (missing images, un-normalized barcodes) by combining raw token learning (for barcodes) with agentic search (for missing context).
Portability: The design (retrieve-then-match, confidence routing, distillation) is domain-independent. The specific success relies on the ability to swap models (e.g., closed to open weights) without retraining the entire pipeline, provided the prompt and tool contracts remain stable.

The authors emphasize that the system is not a "silver bullet" for 100% coverage (bounded by retrieval and catalog completeness) but represents a significant step toward scalable, accurate, and cost-effective entity resolution in large marketplaces.