📊 statistics

VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation

This contribution presents a conformal prediction framework to quantify the reliability of vision-language model judges by converting point estimates into calibrated intervals, demonstrates that evaluation uncertainty is highly task-dependent, and reveals a critical failure mode of "decoupling of ranking and scoring," wherein models can correctly order responses but fail to provide reliable absolute scores.

Original authors: Divake Kumar, Sina Tayebati, Devashri Naik, Ranganath Krishnan, Amit Ranjan Trivedi

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Divake Kumar, Sina Tayebati, Devashri Naik, Ranganath Krishnan, Amit Ranjan Trivedi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Confident but Wrong" Judge

Imagine a new robot judge (a Vision-Language Model) that looks at images and assigns them a score from 1 to 5, just like a teacher grading a test. The problem is that this robot never says, "I'm not sure about this one." It simply outputs a number.

When the robot gives an image a "4," is that a solid 4 because it's very confident? Or is it a shaky 4 because it's guessing? Currently, users have no way to tell the difference. This paper tries to fix that by giving the robot a "confidence meter."

The Solution: The "Safety Net" (Conformal Prediction)

The authors used a mathematical tool called Conformal Prediction. Imagine this as a safety net or a fuzzy halo around the robot's score.

Instead of just saying, "This image is a 4," the robot now says, "This image is a 4, but I am 90% confident that the true score lies somewhere between 2 and 5."

A narrow halo (e.g., 3.8 to 4.2): The robot is very confident. You can trust the score.
A wide halo (e.g., 1 to 5): The robot is confused. The score is unreliable, and you should not trust the exact number.

The paper tests this on 14 different types of visual tasks (such as reading charts, describing art, or solving math problems) using three different robot judges.

Key Finding 1: The "Task Difficulty" Rule

The size of the "halo" depends entirely on what the robot is looking at, not just on how intelligent the robot is.

Easy Tasks (The "Aesthetics" Test): When the robot looks at a pretty image and decides if it is "artistic," the halo is tiny. The robot is very sure.
Hard Tasks (The "Chart" Test): When the robot must read a complex chart, extract data, and do math, the halo explodes and covers almost the entire 1-to-5 scale.

The Analogy: Imagine a human judge. If you ask them, "Is this sunset beautiful?", they might say, "9 out of 10, I'm sure." But if you ask them, "Calculate the exact profit margin from this blurry stock chart," they might say, "I'd guess 4 out of 10, but it could be anything between 1 and 5." The paper shows that AI judges behave exactly the same way: they get confused by difficult visual puzzles, and their "halo" becomes huge to show this uncertainty.

Key Finding 2: The "Ranking vs. Scoring" Trap

This is the most surprising discovery. The paper found that a robot can be great at ordering things but terrible at giving exact numbers.

The Scenario: Imagine a race. The robot can correctly tell you that Runner A beat Runner B and Runner B beat Runner C. (This is Ranking).
The Trap: However, if asked how much faster Runner A was, the robot might guess wildly. It might say "1 second faster," even though it was actually "10 seconds faster." (This is Scoring).

The paper calls this Decoupling of Ranking and Scoring. Standard tests only check if the robot got the order right (Ranking). However, this paper shows that even if the robot gets the order perfect, the actual numbers it provides can be useless because the "halo" is too wide. You can trust the robot when it says "A is better than B," but you cannot trust it when it says "A is a 4.5 and B is a 3.5."

Key Finding 3: It's About the Data, Not the Brain

The authors tested whether larger, smarter robots (with more "brainpower") had smaller halos. They found that size didn't matter much.

Instead, the size of the halo depended on how clean the answers were.

The "Messy Classroom" (MLLM-as-a-Judge): The robot evaluated answers where only one human teacher gave a score. Sometimes this teacher was inconsistent. The robot's halo was huge (broad uncertainty).
The "Clean Classroom" (Polaris): The robot evaluated answers where many teachers agreed on the score. The robot's halo became tiny (narrow uncertainty).

The Result: With clean data, the robot's uncertainty dropped by a factor of 4.5. This proves that the robot's confusion stems from messy data and task difficulty, not from the robot being "dumb."

The Conclusion

The paper concludes with a simple rule for everyone using AI judges:

Look at the width of the halo. If the "safety net" is wide, the robot is uncertain. Do not trust the exact number it provided.
Use the robot for ordering, not scoring. If the halo is wide, use the robot to say "Image A is better than Image B," but do not use it to assign a specific score like "4 out of 5."
Clean data is king. If you want reliable scores, you need clear tasks and consistent human answers to train on.

In short: AI judges can tell you which answer is better, but they often cannot tell you exactly how good it is, especially when the task is difficult or the data is messy.

1. Problem Statement

Vision-Language Models (VLMs) are increasingly deployed as automated judges ("VLM-as-a-Judge") to evaluate multimodal AI systems across diverse tasks (e.g., diagram reasoning, image captioning, visual QA). However, a critical limitation exists: VLM judges provide point estimates (e.g., 4/5) without any indication of reliability.

The Reliability Gap: A score of 4/5 could represent high confidence or significant uncertainty. Users cannot distinguish between the two.
Multimodal Complexity: Errors in VLMs arise from both visual perception and linguistic reasoning, causing uncertainty to accumulate.
Failure of Metrics: Standard metrics (correlation, accuracy) do not capture instance-level reliability. For example, a judge can achieve high rank correlation (correct ordering of responses) while simultaneously assigning completely inaccurate absolute values.
Current State: State-of-the-art VLM judges achieve only an exact match of approximately 32–34% with human evaluations on a 5-point scale, with significant deviations.

2. Methodology

The authors propose using Conformal Prediction (CP), a distribution-free framework, to convert point estimates into calibrated prediction intervals.

Core Mechanism:
- Input: Instead of retraining the model, the method extracts score-token log-probabilities from the VLM's output. For a Likert scale of 1–5, this yields a 5-dimensional feature vector $x = [\log P(1), \dots, \log P(5)]$ .
- Framework: Using a calibration dataset with human-annotated values, the method computes a Nonconformity Score to determine a prediction interval $C(x) = [\hat{y} - \hat{q}, \hat{y} + \hat{q}]$ .
- Guarantee: The interval guarantees containing the true human value with a probability of at least $1-\alpha$ (e.g., 90%), without making assumptions about error distributions.
Key Adaptations:
- Boundary Adjustment: Since the Ground Truth is discrete (integers) but CP generates continuous intervals, the authors apply a boundary adjustment to map intervals to valid rating labels ( $\lceil l \rceil, \lfloor u \rfloor$ ), significantly improving coverage.
- Task-Conditioned CP (Mondrian CP): To address task heterogeneity, data is partitioned into groups (Easy, Medium, Hard) based on initial interval widths. Separate quantiles are computed for each group, resulting in narrower intervals for easy tasks and wider, safer intervals for difficult tasks.
- Ranking-Scoring Gap (RSG): A new metric defined as $RSG = |\rho| - (1 - \frac{w}{K-1})$ , where $\rho$ is the correlation and $w$ is the interval width. This quantifies the decoupling between ranking ability and scoring reliability.

3. Experimental Setup

Judges: Three VLMs with different architectures and scales:
1. LLaVA-Critic-7B: Open-source, specialized for evaluation.
2. Phi-4-reasoning-vision-15B: Large reasoning model.
3. Gemini 2.5 Flash: Closed API access, known for strong ranking but high overconfidence.
Datasets:
1. MLLM-as-a-Judge: 5,717 instances across 14 visual task categories (e.g., ChartQA, MathVista, AesBench) with Ground Truth from a single annotator.
2. Polaris: 8,726 image-caption pairs with Ground Truth averaged over multiple annotators (cleaner, well-defined task).
Protocol: 50/50 split into calibration and test data, 10 random seeds, target coverage of 90% ( $\alpha=0.10$ ).

4. Key Results and Insights

A. Task-Dependent Uncertainty

Uncertainty is not uniform; it is strongly driven by task type.

Narrow Intervals (High Reliability): Aesthetics and natural image tasks (e.g., AesBench) yield intervals covering only about 40% of the value range (width $\approx$ 2.08).
Wide Intervals (Low Reliability): Visually intensive tasks requiring complex reasoning (e.g., InfographicsVQA, MathVista) yield intervals covering about 70% of the range (width $\approx$ 3.50).
Consistency: The order of task difficulty is consistent across all three judges (Spearman $\rho \approx 0.82–0.93$ ), indicating that task difficulty, not model identity, drives uncertainty.

B. Decoupling of Ranking and Scoring

The paper identifies a critical failure mode where standard metrics are misleading:

Phenomenon: Judges can achieve high rank correlation (correct ordering of responses) while producing wide, uninformative intervals (and thus failing to assign reliable absolute values).
Example: On ChartQA, a judge achieved the highest Pearson correlation ( $\rho=0.507$ ) but produced one of the widest intervals (3.08).
Implication: High correlation does not mean the absolute value is trustworthy. For tasks with high RSG, pairwise comparison of absolute scores is preferable.

C. Influence of Data Quality

Clean vs. Noisy Data: When the same judge (LLaVA-Critic) and method were applied to the Polaris dataset (clean, multi-annotator captions), the interval width dropped from 3.05 (MLLM-Judge) to 0.68.
Result: This represents a 4.5-fold reduction in interval width, demonstrating that uncertainty is primarily driven by annotator noise and task ambiguity, not by the conformal method itself.

D. Model Performance Paradoxes

Confidence vs. Interval Width: Gemini 2.5 Flash was highly overconfident (83% of predictions had a Softmax maximum > 0.99) yet generated the narrowest intervals. This proves that interval quality depends on the discriminative signal in the log-probabilities, not the calibration of the Softmax distribution.
Specialization > Scaling: On the specialized Polaris task, the 7B LLaVA-Critic (trained for evaluation) vastly outperformed the 15B Phi-4 (general reasoning), achieving 7.1x higher correlation and 5.3x narrower intervals.

E. Method Comparison

R2CCP (Regression-as-Classification) was the best-performing method, achieving near-perfect 90% coverage with the narrowest informative intervals.
CQR (Conformalized Quantile Regression) failed and collapsed to intervals spanning the entire range ( $\approx$ 4.0), as score-token log-probabilities are weak predictors for quantiles in this discrete setting.
Feature Fusion: Combining log-probability features from multiple judges degraded performance, suggesting that feature quality (a strong judge) outweighs quantity.

5. Significance and Contributions

Quantitative Reliability Map: The paper provides the first systematic mapping of evaluation reliability across 14 visual tasks, showing that "VLM-as-a-Judge" is reliable for some tasks (aesthetics) but highly uncertain for others (complex reasoning).
New Failure Mode: It reveals the decoupling of ranking and scoring, a phenomenon where standard metrics (correlation) mask the judges' inability to provide reliable absolute values.
Practical Framework: It demonstrates that Conformal Prediction can be applied to existing VLM pipelines without retraining, using only log-probabilities to generate actionable uncertainty estimates.
Operational Guideline: The authors propose a rule of thumb: Narrow intervals indicate the evaluation is suitable for absolute values; wide intervals signal ambiguity and suggest that relative ranking or pairwise comparison is the appropriate evaluation strategy.
Insight on Data Quality: It highlights that improving annotation quality (consensus of multiple annotators) yields greater reliability gains than merely scaling model parameters.

Conclusion

The paper concludes that VLM judges are not universally reliable. Their uncertainty is structurally dependent on the task and data quality. By employing Conformal Prediction, practitioners can move from blind trust in point estimates to a nuanced understanding of when a judge's evaluation is trustworthy and when it should be treated as a noisy ranking signal.