💬 NLP

Can Argus Judge Them All? Comparing VLMs Across Domains

This paper introduces ARGUS-EVAL, a novel framework that evaluates Vision-Language Models by balancing benchmark capability with cross-dataset reliability, revealing significant discrepancies between traditional performance rankings and real-world stability across models like Qwen-2.5VL-3B-Instruct and CLIP.

Original authors: Harsh Joshi, Gautam Siddharth Kashyap, Rafiq Ali, Ebad Shabbir, Niharika Jain, Sarthak Jain, Jiechao Gao, Usman Naseem

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Harsh Joshi, Gautam Siddharth Kashyap, Rafiq Ali, Ebad Shabbir, Niharika Jain, Sarthak Jain, Jiechao Gao, Usman Naseem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're a talent scout for a massive, high-tech circus. You need to hire the best "Vision-Language Models" (VLMs)—super-smart robots that can look at a picture and tell you what's happening, find similar pictures, or solve puzzles about them.

Usually, when you pick a robot, you just look at its report card. You see a score like "95% on the Math Test" and think, "Perfect! This is the one!" But this paper, titled "Can Argus Judge Them All?", argues that this report card is a bit of a trick. It suggests that a robot might ace the test but fall apart the moment you take it out of the classroom and into the messy, unpredictable real world.

The authors call this the "Capability-Reliability Gap." It's like hiring a gymnast who does a perfect backflip on a soft, padded mat in the gym (the benchmark), but wobbles and falls when they try it on a bouncy castle or a windy stage (the real world).

The New Scoring System: ARGUS-EVAL

To fix this, the researchers built a new judging system called ARGUS-EVAL. Instead of just looking at one final score, they check the robots on four different things:

The Test Score (Capability): How well does it do on the standard exams?
The Consistency Score: If you give it the same type of test but with slightly different questions or pictures, does it still perform well?
The "Toughness" Score: If you mess up the picture (make it blurry) or the text (add typos), does it still figure things out, or does it crash?
The Efficiency Score: How fast is it, and how much battery (or computer memory) does it eat up?

The Race: Who Won?

The team put five famous robot models through this new gauntlet: CLIP, BLIP, LXMERT, Gemma-3-4B, and Qwen-2.5VL-3B-Instruct. They tested them on three main jobs: finding matching images and text (Retrieval), describing images (Captioning), and solving logic puzzles (Reasoning).

Here is what they found, and it's a bit of a plot twist:

The "Star Student" (Qwen):
If you only looked at the traditional report card, Qwen-2.5VL-3B-Instruct was the clear winner. It had the highest scores everywhere.

In the image-finding game, it got 82.7% of the top matches right (R@1).
When describing pictures, it wrote sentences with a BLEU-4 score of 47.2 and a CIDEr score of 141.6.
In the logic puzzles, it got 97.4% right on the CLEVR test.

The paper suggests that if you need the absolute smartest robot for a job where accuracy is everything, Qwen is the one to pick.

The "Reliable Veteran" (Gemma):
But here's the catch. When the judges added the "Consistency" and "Toughness" scores to the mix, the rankings flipped. Gemma-3-4B actually became the most reliable overall.

While Qwen was slightly better at the raw tests, Gemma was more stable. It didn't panic as much when the tests got weird or the pictures got blurry.
In fact, when the authors added all the new reliability scores, Gemma's total score jumped to 0.891, while Qwen's dropped to 0.868.
The paper suggests that Gemma is the better choice if you need a robot that won't break when things get messy.

The "Speedster" (CLIP):
Then there's CLIP. It wasn't the smartest at solving puzzles or writing fancy descriptions. But it was the fastest and lightest.

It could process an image in just 31 ms (milliseconds).
It only needed 0.9 GB of memory.
The paper notes that for devices with small batteries or weak computers (like a Raspberry Pi), CLIP is the clear winner because it doesn't hog resources.

The Big Lesson

The main finding of this paper is that you can't just pick the robot with the highest test score.

The authors measured this carefully across different datasets and found that models with similar test scores can behave very differently in the real world. They explicitly argue against the idea that "higher capability always means higher reliability." They showed that in many cases, the robot that looks best on paper (Qwen) might actually be less stable than the one that looks slightly less impressive (Gemma).

They also measured how these robots run on different hardware. On a powerful server (NVIDIA A100), CLIP was 31 ms fast, while Gemma was 138 ms and Qwen was 142 ms. But on smaller devices, the gap in speed and memory usage became even more dramatic, with CLIP staying light and fast while the others got heavier.

What This Doesn't Tell Us

The paper is careful to say what it didn't do. They didn't test every possible real-world scenario, and they didn't test robots that had been specially trained (fine-tuned) for specific jobs. They only tested the robots "as they are" out of the box (zero-shot).

They also didn't test every kind of "messy" picture, just a specific set of blurry or noisy ones. So, while they suggest that Gemma is more reliable and Qwen is smarter, they aren't claiming this is the final word on every robot in existence.

The Takeaway

If you are building a system where you need the absolute best answers and you have powerful computers, Qwen might be your best bet. But if you need a robot that stays calm when things go wrong, or if you are running on a smaller device where speed matters, Gemma or CLIP might actually be the smarter choice.

The paper concludes that we need to stop looking at just one number on a report card. We need to look at the whole picture: how smart the robot is, how steady it is, and how much energy it uses. Only then can we really judge if a robot is ready for the real world.

Technical Summary: Can Argus Judge Them All? Comparing VLMs Across Domains

Problem Statement
Vision-Language Models (VLMs) are increasingly deployed in industry applications such as retrieval systems, content generation platforms, and decision-support workflows. Currently, model selection is predominantly guided by benchmark rankings derived from downstream tasks like retrieval, captioning, and reasoning. However, the authors identify a critical limitation in this approach: models with similar aggregate benchmark performance often exhibit substantially different behaviors across datasets and evaluation conditions. This discrepancy creates a "Capability-Reliability Gap," where a model maximizing benchmark capability ( $P(M)$ ) does not necessarily maximize reliability ( $R(M)$ ) in practical deployment scenarios. Traditional evaluations fail to capture performance stability across heterogeneous benchmarks or robustness under perturbations, leading to potential mismatches between theoretical rankings and observed model stability.

Methodology: ARGUS-EVAL
To address this gap, the authors introduce ARGUS-EVAL, a capability-reliability-oriented evaluation framework designed to characterize model behavior through a four-dimensional assessment vector $A(M) = \{P(M), CDC(M), RR(M), E(M)\}$ .

Benchmark Capability ( $P(M)$ ): Represents the aggregate performance across retrieval, captioning, and reasoning tasks. It is computed as the mean of normalized scores from five datasets: COCO, Flickr30k, and Visual Genome (for retrieval and captioning), and CLEVR and VCR (for reasoning).
Cross-Dataset Consistency ($CDC(M)$): Measures performance stability across heterogeneous benchmarks. It quantifies the variation in a model's score distribution across different datasets; higher values indicate smaller performance variation.
Robustness Retention ($RR(M)$): Measures performance preservation under controlled image and text perturbations. It is calculated as the ratio of benchmark capability after perturbation to the clean benchmark capability.
Efficiency ( $E(M)$ ): Captures deployment characteristics, including inference latency, memory footprint, FLOPs, and throughput. These metrics are normalized and aggregated to reflect the model's suitability for various hardware environments.

The framework evaluates five representative VLMs: CLIP, BLIP, LXMERT, Gemma-3-4B, and Qwen-2.5VL-3B-Instruct. All models are tested using official pretrained checkpoints under a common zero-shot protocol without fine-tuning.

Key Results
The empirical evaluation reveals significant divergences between capability-oriented and reliability-oriented rankings:

Capability Dominance: Qwen-2.5VL-3B-Instruct achieves the strongest overall benchmark capability ( $P(M)$ ). It records the highest scores in retrieval (R@1 = 82.7% on COCO), captioning (BLEU-4 = 47.2, CIDEr = 141.6 on COCO), and reasoning (97.4% accuracy on CLEVR).
Reliability and Robustness: When reliability metrics are incorporated, the ranking shifts. Gemma-3-4B demonstrates superior robustness retention ($RR = 0.901$) under perturbations compared to Qwen ($RR = 0.890$). Consequently, Gemma achieves the highest overall ARGUS-EVAL score (0.891) when consistency and robustness are factored in, surpassing Qwen (0.868).
Efficiency: CLIP exhibits the strongest efficiency profile, recording the lowest latency (31 ms on NVIDIA A100) and smallest memory footprint (0.9 GB), making it highly suitable for resource-constrained environments despite lower generative capabilities.
The Capability-Reliability Gap: The study confirms that the model maximizing capability (Qwen) does not coincide with the model maximizing reliability (Gemma). This validates the existence of the gap, where high benchmark scores do not guarantee consistent performance across diverse conditions.

Significance and Claims
The paper claims that ARGUS-EVAL is the first unified framework to jointly assess benchmark capability, cross-dataset consistency, robustness retention, and efficiency. The authors argue that relying solely on traditional benchmark rankings provides limited insight into model stability in real-world deployment. By demonstrating that instruction-tuned models (like Gemma) can offer a more balanced trade-off between capability and reliability compared to models that strictly maximize benchmark scores, the work advocates for a shift in evaluation methodology. The authors position ARGUS-EVAL as a tool to support responsible model assessment, encouraging practitioners to move beyond single-metric optimization toward a holistic view that includes stability and efficiency before deployment.

Limitations
The authors acknowledge that the framework currently evaluates a limited set of representative datasets and does not cover the full diversity of real-world deployment scenarios. The robustness analysis is restricted to predefined perturbations, and efficiency measurements are conducted on a specific set of hardware platforms (NVIDIA A100, Jetson Nano, Raspberry Pi 5), which may not generalize to all environments. Furthermore, the evaluation is strictly zero-shot and does not assess fine-tuned or task-specific adaptations.