💻 computer science

Cross-Granularity Ranking Disagreement for Uncertainty-Aware Image-Text Retrieval

This paper introduces Cross-granularity Ranking Uncertainty (CRU), a lightweight framework that directly estimates retrieval uncertainty from ranking-space disagreements between global and local matching distributions to improve error detection, calibration, and selective prediction in image-text retrieval without requiring multiple forward passes.

Original authors: Wei Chen, Yuhang Chen

Published 2026-08-07

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Wei Chen, Yuhang Chen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a massive library where every book has a picture on the cover, and every picture has a story written next to it. Your job is to find the perfect match: a picture that goes with a specific sentence, or a sentence that describes a specific photo. This is the world of image-text retrieval, a superpower used by search engines, shopping apps, and digital art galleries to connect what we see with what we read. For a long time, computers have gotten really good at this, using "global" views to get the general vibe of a scene. But here's the tricky part: sometimes, even when a computer gives you a list of matches, it doesn't know if it's actually right. It might be super confident but totally wrong, or it might be confused because two pictures look almost identical. In the real world, knowing when to trust the computer is just as important as getting the answer itself. If a search engine can say, "I'm pretty sure about this one, but I'm totally guessing on that last one," it becomes a much more helpful tool.

This is exactly the problem Wei Chen and Yuhang Chen tackle in their new research. They noticed that while computers are great at ranking lists of matches, they often fail to tell us how "shaky" that ranking is. To fix this, they built a new system called CRU (Cross-granularity Ranking Uncertainty). Think of it like hiring two different detectives to solve a mystery. One detective, the "Global" agent, looks at the big picture: "Is this a beach scene? Yes." The other detective, the "Local" agent, zooms in on the tiny details: "Does the person have red shoes? Yes. Is there a blue umbrella? No." Usually, these two agree. But when they disagree—when the Global agent says "Beach!" but the Local agent sees "Red shoes on a mountain"—that's a huge red flag. CRU uses this disagreement to calculate a "worry score." If the score is high, the system knows the answer might be wrong. The researchers found that by listening to both detectives and checking if they are arguing, their system not only finds better matches but also knows exactly when to say, "Hey, I'm not sure about this one," making the whole search process smarter and more reliable.

The Two Detectives and the Worry Score

In the world of image-text retrieval, computers usually try to match a photo and a sentence by turning them into mathematical numbers (called embeddings) and seeing how close they are. Traditional methods often rely on a single "global" view, which is like looking at a painting from across the room. You can tell it's a landscape, but you might miss that the dog in the corner is actually a cat. Other methods try to look at "local" details, like matching specific words to specific parts of the image, but this can be slow and messy.

The authors of this paper argue that the best way to find errors isn't to build a bigger, more complex model, but to listen to the difference between these two ways of looking. They created a framework where the computer acts like a panel of two judges.

The Global Judge: Looks at the whole image and sentence to get the main idea.
The Local Judge: Looks at tiny patches of the image and specific words to check the nitty-gritty details.

In a perfect world, both judges would agree on the top matches. But in the real world, they often disagree. Maybe the Global Judge thinks an image of a "sunset" matches a sentence about "sunset," but the Local Judge notices the sentence says "sunset over the ocean" and the image is actually a sunset over a desert.

How CRU Works: The "Disagreement" Detector

The magic of CRU lies in how it measures this disagreement. Instead of just asking, "How confident are you?" (which can be misleading if the computer is confidently wrong), CRU asks two specific questions:

Ambiguity: Is the computer confused? If both judges are spreading their votes across many different images, the system knows the answer is fuzzy.
Disagreement: Are the judges fighting? If the Global Judge is 100% sure Image A is the best, but the Local Judge is 100% sure Image B is the best, that's a conflict.

CRU combines these two feelings into a single "Uncertainty Score." If the score is low, the system is confident and the ranking is likely correct. If the score is high, the system knows something is fishy.

What makes this special is that it doesn't need to run the computer's brain multiple times (like some other methods that try to guess by running the same test over and over). It does it all in one single pass. It's like having two detectives solve the case in one go and then immediately comparing their notes to see if they are on the same page.

The Results: Smarter, Not Just Faster

The researchers tested their new system on three famous datasets: Flickr30K-1K, MS-COCO-1K, and MS-COCO-5K. These are basically giant libraries of photos and captions used to train AI.

Here is what they found:

Better Accuracy: CRU didn't just get better at spotting errors; it actually found better matches. On the Flickr30K-1K test, it achieved a score (called rSum) of 520.4. On MS-COCO-1K, it hit 535.1, and on the larger MS-COCO-5K, it scored 446.9. These numbers were higher than the previous best methods, meaning the system found the right photo more often.
Better Error Detection: When it came to spotting mistakes, CRU was a champion. Compared to the strongest previous method (called PCME++), CRU improved its ability to detect errors (measured by AUROC) by 4.3 to 4.6 points.
Calibration: This is a fancy word for "being honest." If a system says it's 90% sure, it should be right 90% of the time. CRU was much better at this than other methods. It reduced the "Expected Calibration Error" (ECE) to as low as 0.023, meaning its confidence levels were very close to reality.
Selective Retrieval: This is the ability to say, "I'm not sure about this one, so I'll skip it." When the researchers told the system to skip the most uncertain answers, the error rate dropped significantly. CRU reduced the "Area Under the Risk-Coverage Curve" (AURC) by 16.4% to 21.6% compared to the best baseline. This means that by simply ignoring the guesses it wasn't sure about, the system became much more reliable.

Why This Matters

The paper suggests that by treating global and local matching as "complementary opinions" rather than just steps to get a score, we can build systems that are not only accurate but also self-aware. The authors explicitly rule out the idea that you need complex, probabilistic models that run thousands of times to get this kind of insight. Instead, they show that a simple, deterministic (one-time) pass that checks for disagreement between two different "views" is enough.

They also note that while the system is great, it isn't perfect. It relies on the data it was trained on, and if the world changes drastically (like a sudden shift in how people take photos or write captions), the system might need to be recalibrated. But for now, CRU offers a practical, efficient way to make image-search engines that don't just give you an answer, but also tell you how much you can trust it. It turns a blind search into a conversation where the computer knows when to say, "I think I know, but let me double-check."

Technical Summary: Cross-Granularity Ranking Disagreement for Uncertainty-Aware Image–Text Retrieval

1. Problem Statement

Image-text retrieval systems are typically optimized for ranking accuracy (e.g., Recall@K) but lack mechanisms to indicate the trustworthiness of their returned lists. In real-world scenarios, queries may be incomplete, candidates may be nearly indistinguishable, or crucial attributes may be missed. A high similarity score alone does not reveal whether a result is reliable.

Existing uncertainty-aware approaches often represent modalities as probability distributions within the embedding space. However, this uncertainty is learned indirectly and may not reflect the error risk of the final ranking. Other methods, such as Monte Carlo dropout or deep ensembles, require stochastic inference or multiple models, which are computationally expensive for large-scale galleries. Post-hoc confidence measures (e.g., top-score margins) are inexpensive but often fail to detect decision conflicts where different retrieval mechanisms are individually confident but disagree on the candidate order.

2. Methodology: Cross-granularity Ranking Uncertainty (CRU)

The authors propose CRU, a deterministic framework that estimates retrieval uncertainty directly from the candidate-ranking space rather than the embedding space. The method utilizes two complementary retrieval "opinions" derived from a shared pair of encoders: a Global Semantic Branch and a Local Detail Branch.

2.1 Dual-Granularity Feature Representation

The framework employs a Vision Transformer (ViT) and a BERT text encoder.

Global Branch: Extracts global semantic tokens (e.g., [CLS]) to capture scene-level intent. It uses cosine similarity for efficient, indexable retrieval.
Local Branch: Aggregates patch-word correspondences using a temperature-controlled LogMeanExp operator. This branch verifies fine-grained details (attributes, quantities, relations) that global vectors might suppress.

2.2 Candidate-Space Ranking Distributions

To compare the two branches, CRU constructs a shared candidate support set $C_q$ by taking the union of the top- $K$ candidates from both branches. Raw scores are standardized and converted into categorical ranking distributions $p_g$ and $p_l$ using softmax.

2.3 Decomposition of Uncertainty

CRU decomposes reliability into two distinct failure modes:

Within-Branch Ranking Ambiguity ( $U_{amb}$ ): Measured by the normalized Shannon entropy of each branch's distribution. High entropy indicates that a branch cannot distinguish between candidates (diffuse evidence).
Cross-Granularity Ranking Disagreement ( $U_{dis}$ ): Measured by the Jensen–Shannon (JS) divergence between the global and local distributions. High divergence indicates that both branches are confident but prefer different candidates (conflicting opinions).

The final uncertainty score $U(q)$ is a convex combination of these two components:
$U(q) = \alpha U_{amb}(q) + (1 - \alpha)U_{dis}(q)$

2.4 Confidence-Aware Dynamic Fusion and Calibration

Dynamic Fusion: The model computes query-level fusion weights based on branch confidence ( $1 - \text{entropy}$ ) to produce a fused similarity score $S_f$ . This ensures the final ranking is coherent and interpretable.
Correctness-Aware Calibration: A lightweight logistic calibrator maps the uncertainty score $U(q)$ to a predicted top-1 error probability $\hat{r}_q$ . This is trained using a binary cross-entropy loss against the ground-truth top-1 error indicator, ensuring the uncertainty score is calibrated to empirical error rates.
Consistency Loss: A consistency term encourages agreement between branches only for queries where the fused ranking is already confident, preventing the model from forcing agreement on unresolved queries.

2.5 Inference Efficiency

CRU requires only a single deterministic forward pass. It performs global retrieval to fetch a candidate pool, applies local re-ranking only to this pool, and computes uncertainty. It avoids the computational overhead of sampling or ensembles.

3. Key Contributions

The paper claims the following contributions:

Candidate-Space Formulation: A novel formulation of retrieval uncertainty that exposes conflicts between global semantics and local details directly in the decision space.
Decomposable Score: A method combining ranking ambiguity and cross-granularity JS disagreement, coupled with query-level confidence-aware fusion.
Calibration and Selective Prediction: A correctness-aware calibration mechanism and confidence-weighted consistency that enable selective retrieval (abstention) without extra inference passes.
Comprehensive Evaluation: Extensive experiments covering retrieval accuracy, error detection, calibration, and component ablation across standard benchmarks.

4. Experimental Results

The method was evaluated on Flickr30K-1K, MS-COCO-1K, and MS-COCO-5K.

Retrieval Accuracy: CRU achieved rSum values of 520.4 (Flickr30K-1K), 535.1 (MS-COCO-1K), and 446.9 (MS-COCO-5K). These results represent improvements of 4.2, 2.6, and 3.9 points respectively over the strong GRM baseline.
Error Detection: Relative to the strongest uncertainty baseline (PCME++), CRU improved the Area Under the Receiver Operating Characteristic (AUROC) by 4.3–4.6 points and the Area Under the Precision-Recall curve (AUPR) by 4.3–4.9 points.
Calibration: The Expected Calibration Error (ECE) was reduced significantly (e.g., from 0.047 to 0.024 on Flickr30K-1K), representing a relative reduction of ~49–53%.
Selective Retrieval: The Area Under the Risk-Coverage curve (AURC) was reduced by 16.4–21.6% compared to PCME++, indicating that retaining only low-uncertainty queries yields a substantially lower error rate.
Efficiency: The framework maintains efficiency by confining local matching to a small candidate pool ( $R=100$ ) and avoiding stochastic inference.

5. Significance and Claims

The paper posits that ranking-space uncertainty is a more effective indicator of retrieval risk than embedding-space uncertainty. By treating global and local matchers as complementary opinions, CRU can detect "confident but wrong" scenarios where a single ranking mechanism would fail to signal risk.

The authors claim that CRU provides a practical route to building retrieval systems that not only rank accurately but also indicate when their outputs should not be trusted. This is achieved in a single deterministic pass, making it suitable for large-scale deployment. The results demonstrate that explicitly modeling the disagreement between different granularities of matching improves not only error detection and calibration but also the final retrieval accuracy itself.

The authors remain modest regarding limitations, noting that calibration is defined relative to available annotations (which may not be exhaustive) and that severe distribution shifts may still degrade performance, suggesting a need for future work on conformal guarantees.