💻 computer science

AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty

This paper introduces AnchorScore, a low-cost, CLIP-based diagnostic metric that effectively predicts the per-class annotation difficulty for Multimodal Large Language Models (MLLMs), enabling cost-efficient routing strategies and prioritization of human review without requiring expensive MLLM evaluations.

Original authors: Yan Ma, Lizhuo Zhang

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yan Ma, Lizhuo Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern landscape of artificial intelligence, a powerful new class of tools known as multimodal large language models has emerged. These systems are designed to see and understand the world much like humans do, combining the ability to process images with the ability to read and write text. Because they can interpret complex scenes, researchers and companies are increasingly using them to automatically label vast collections of photographs, a task that once required armies of human workers. However, these digital annotators are not perfect. While they might excel at identifying a person standing at a blackboard, they can struggle profoundly with more subtle actions, such as someone answering a question or raising a hand. This inconsistency creates a significant problem: if a system is used to label millions of images, how can a user know which specific categories will be handled with high confidence and which will be filled with errors? Running the full system on every single image to check its accuracy is often too slow and expensive to be practical, taking hours or even days to process a modest dataset.

A team of researchers set out to solve this dilemma by finding a way to predict these failures before they happen, using a much simpler and faster tool. They focused on a specific type of artificial intelligence model that acts as a bridge between images and words, trained on billions of pictures and their descriptions. This model, while less complex than the massive annotators, is incredibly quick to run. The researchers hypothesized that if this simpler model struggles to recognize a specific action in a photo, the more powerful, complex system would likely struggle with it too. They tested this idea on a dataset of classroom scenes, where the goal was to identify various behaviors like teaching, writing, or standing. By running the fast, simple model on thousands of images, they generated a difficulty score for each type of behavior. They then compared these scores against the actual performance of the powerful annotators. The results showed a striking connection: the classes that the simple model found difficult were almost exactly the same classes where the powerful system made mistakes. This relationship was strong enough to be statistically significant, suggesting that the simple model's performance serves as a reliable early warning system for the complex one.

The researchers did not stop at simply observing this link; they rigorously tested whether other methods could provide the same insight. They tried using the complex system's own internal confidence levels, asking it to rate how sure it was about its answers, but this approach failed to predict errors accurately. They also tested other types of visual models that do not connect images to language in the same way, and these too failed to show a meaningful correlation. The only signal that worked was the specific type of image-text matching provided by the fast, simple model. This finding is crucial because it rules out the idea that the complex system's own uncertainty or generic visual recognition is enough to spot trouble. Instead, it points to a shared difficulty: certain visual concepts are inherently hard for both types of technology to grasp, regardless of their size or complexity. The researchers further confirmed this by testing the idea on a completely different set of images showing people performing everyday actions, finding the same strong pattern. This suggests the discovery is not just a fluke of the classroom data but a general principle about how these machines learn.

Beyond simply identifying which categories are hard, the team demonstrated how this insight can be used to build smarter, more efficient workflows. They developed a strategy where the fast, simple model acts as a gatekeeper. If the simple model is confident about an image, the system accepts its label immediately, saving time and money. If the simple model is unsure, the system automatically routes that image to the powerful, expensive model for a second look. This hybrid approach allowed them to achieve much higher accuracy than using the simple model alone, while still saving a significant amount of computational cost. In one test, they improved accuracy by more than twenty percentage points while reducing the need for the expensive system by nearly half. They also used the data to improve the instructions given to the powerful system. When the simple model confused two similar actions, the researchers added a specific note to the instructions to clarify the difference, which helped the powerful system correct its mistakes. Finally, they showed that this method could help humans prioritize their work. By flagging the categories that the machines are most likely to get wrong, human reviewers can focus their attention on the most critical images, ensuring that errors are caught where they matter most.

The study concludes that this fast, simple diagnostic tool offers a practical way to manage the limitations of advanced artificial intelligence. It does not replace the powerful systems, nor does it provide a perfect prediction of their exact accuracy. Instead, it provides a low-cost map of the terrain, showing where the ground is shaky. By using a few minutes of computation on a small set of images, users can identify which tasks require extra care and which can be handled automatically. This approach turns the expensive process of evaluating these massive systems into a manageable budget, allowing practitioners to allocate their resources where they will have the greatest impact. The work highlights that even in the age of giant models, the key to efficiency often lies in understanding the shared weaknesses of the entire family of tools, rather than trying to force the most powerful one to do everything alone.

Technical Summary: AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty

Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly utilized for automated data annotation, yet their performance is highly inconsistent across different classes. For instance, on classroom behavior datasets, MLLM accuracy can range from 12% to 98% across classes. Identifying which specific classes an MLLM will struggle with a priori is critical for resource allocation but remains expensive. Evaluating a single 27B-parameter MLLM on a validation set of ~5,400 images requires approximately 14 hours of compute time, whereas a lightweight alternative is needed to rank classes by expected difficulty before deploying the full annotation pipeline. Existing methods, such as running pilot samples through the MLLM or extracting model-specific uncertainty, are computationally prohibitive or require access to model logits.

Methodology

The paper introduces AnchorScore, a low-cost, training-free diagnostic metric defined as the per-class zero-shot accuracy of a frozen CLIP model on a target dataset. The core hypothesis is that if a frozen CLIP model (a different model family) struggles to classify a specific class zero-shot, a larger generative MLLM will likely also struggle with that same class.

Key Methodological Components:

Definition: AnchorScore for a class $c_i$ is calculated as the fraction of validation images where the frozen CLIP model's prediction matches the ground truth label.
Implementation: The study uses a ViT-L/14 CLIP backbone pretrained on LAION-2B. To ensure robustness, text features are averaged over three prompt templates (e.g., "a photo of a person {cls} in classroom").
Cost Asymmetry: The method leverages the massive computational gap between CLIP and MLLMs. A full-frame CLIP inference on 5,416 images takes ~3 minutes (single GPU), compared to ~14 hours for a 27B MLLM (4 GPUs).
Validation Strategy: The study evaluates the correlation between AnchorScore and the mean accuracy of six diverse MLLMs (ranging from 7B to 35B parameters, including Qwen and Gemma families) across multiple datasets.

Key Contributions

The paper makes three primary contributions:

Diagnostic Discovery: It demonstrates that a cheap, training-free probe from a different model family (frozen CLIP zero-shot accuracy) effectively predicts where MLLM annotation is unreliable. This signal is obtainable before any MLLM inference, transforming MLLM evaluation from a per-model cost into an allocatable budget.
Shared-Difficulty Characterization: The study establishes that the correlation reflects a shared class-difficulty factor rather than a CLIP-specific artifact. Alternative predictors, including MLLM self-verbalized uncertainty, non-CLIP vision models (DINOv2, ResNet-50, SigLIP), and small-MLLM pilots, failed to show significant class-level correlations. A cross-model consensus control further suggests the signal captures a shared difficulty factor across the MLLM ensemble.
Practical Workflows: The paper outlines three resource-aware applications enabled by this diagnostic:
- Hybrid Routing: A deployable strategy where images predicted to belong to high-AnchorScore classes are handled by CLIP, while low-AnchorScore classes are routed to the expensive MLLM.
- Prompt Disambiguation: Using CLIP's confusion patterns to generate targeted prompts that clarify visual distinctions for hard classes.
- Review-Priority Prediction: Ranking classes for human verification to maximize the efficiency of limited review budgets.

Results

Correlation Analysis:

SCB5 (Classroom Behavior): On a dataset with 13 classes and 6 MLLMs, AnchorScore showed a significant Spearman correlation with per-class MLLM accuracy ( $\rho = 0.769, p = 0.002$ ).
Stanford40 Replication: An independent replication on the Stanford40 Actions dataset (40 classes) yielded a nearly identical effect ( $\rho = 0.817, p < 0.001$ ).
Cross-Domain Performance: The signal was strongest on activity-recognition data. It attenuated on medical (MedMNIST) and satellite (EuroSAT) imagery, where the correlation was weaker or non-significant within domains, suggesting the diagnostic is most effective in domains where CLIP possesses moderate visual-linguistic competence.

Baseline Comparisons:

Alternative Predictors: None of the alternative difficulty predictors (SigLIP, DINOv2, ResNet-50, MLLM self-uncertainty, or a 7B MLLM pilot) showed a significant class-level correlation ( $p > 0.05$ ) on the SCB5 dataset.
Consensus Control: Conditioning the analysis on the consensus of the other five MLLMs reduced the partial correlation of AnchorScore to near zero ( $\hat{\rho} = 0.067$ ), indicating that AnchorScore primarily captures a shared difficulty factor rather than a unique CLIP-specific error source.

Application Performance:

Hybrid Routing: On the TeacherBehavior subset, a deployable routing strategy (using CLIP for high-score classes and MLLM for low-score classes) achieved a 21.7 percentage point (pp) realized accuracy gain over CLIP-only baselines under the canonical protocol while saving 44% of MLLM inference costs. The expected value estimate was +23.3 pp, but realized runs delivered +21.7 pp under the canonical protocol and +17.5 pp under the full-frame deployment condition.
Prompt Disambiguation: For low-AnchorScore classes, injecting direct visual-distinction descriptions into prompts improved MLLM accuracy by an average of +25.0 pp compared to standard prompts, whereas negation-style hints were less effective and sometimes detrimental.
Review Prioritization: AnchorScore successfully identified the most difficult classes for human review, achieving an Area Under the Curve (AUC) of 0.938 on TeacherBehavior and 0.842 on Stanford40.

Significance and Claims

The paper positions AnchorScore not as a calibrated estimator of exact MLLM accuracy, but as a low-cost ranking signal that directs expensive MLLM evaluation to the classes where it is most informative.

Efficiency: The method reduces the computational cost of identifying difficult classes by a factor of approximately 270x compared to running the full MLLM pipeline.
Generalizability: The signal is robust across different MLLM families (Qwen, Gemma, LLaVA) and scales (7B–35B), provided the task involves human action recognition or similar visual-linguistic alignment.
Limitations: The authors explicitly note that the diagnostic is domain-dependent. It performs poorly in "floor regimes" (e.g., medical imaging where both CLIP and MLLMs fail) and "ceiling regimes" (e.g., Stanford40 where most classes are easy for CLIP). Furthermore, the correlation is a relative ranking tool; it does not estimate absolute accuracy, and large divergences can occur for composite action classes where MLLMs leverage language understanding to overcome CLIP's visual misalignment.

In summary, AnchorScore provides a practical, training-free mechanism for practitioners to optimize MLLM annotation pipelines by identifying and prioritizing classes that are inherently difficult for the model family, thereby enabling smarter resource allocation without requiring prior MLLM inference.