🤖 AI

VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

The paper introduces VendorBench-100, a unified cross-paradigm benchmark that evaluates 36 deepfake detection models across commercial APIs, vision-language models, and open-source detectors using a curated 100-image corpus, revealing that while commercial APIs lead in median performance, a critical divergence exists between ranking ability (ROC-AUC) and reliable decision-making (MCC).

Original authors: Sharayu N. Deshmukh, Md Rashidunnabi, Nelton Tiago Gemo, Kurundkar G. D., Mahamune M. R., Nilesh K. Deshmukh

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sharayu N. Deshmukh, Md Rashidunnabi, Nelton Tiago Gemo, Kurundkar G. D., Mahamune M. R., Nilesh K. Deshmukh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a security team. Your job is to spot fake photos (deepfakes) that are being used to trick people. You have three very different types of security guards available to hire:

The High-Tech Pros (Commercial APIs): You pay a company like "Neural Defend" or "Reality Defender" to check the photos. They have expensive, specialized training just for this job.
The Smart Generalists (Vision LLMs): You ask a super-smart AI assistant (like a chatbot that can see pictures) to take a guess. They weren't trained specifically to spot fakes, but they are very smart and can reason about what they see.
The DIY Enthusiasts (Open-Source Detectors): You download free tools built by researchers and hobbyists. Some are brilliant, some are buggy, and they all run on your own computers.

The problem is that nobody has ever put these three groups in the same room to take the same test. The "Pros" have their own reports, the "Generalists" have general test scores, and the "DIY" crowd has their own leaderboards. It's like comparing a race car's lap time on a track to a truck's speed on a highway and calling them "fast."

The Paper's Big Idea: The "VendorBench-100" Test
The authors of this paper decided to build a single, unfair, and very difficult test to see who actually wins. They didn't just make a test with 1,000 easy photos. They hand-picked 100 specific, tricky photos designed to break these systems.

Think of this test like a "survival course" for detectors:

Some photos are just a face swapped onto a body, but the background is real.
Some are frames from AI videos that look like normal movies.
Some are photos where someone used a phone app to edit just one small part of a real picture.
Some are tiny, blurry, and heavily compressed (like a photo sent through a text message).

They ran 36 different models (5 Pros, 7 Generalists, and 24 DIY tools) through this exact same 100-photo gauntlet.

The Trap: Why "Accuracy" is a Liar
Here is the most important part of the paper, explained with a simple analogy.

Imagine the test has 79 fake photos and 21 real photos.
If you hired a guard who was too lazy to look at the photos and just shouted "FAKE!" for every single one, what would happen?

They would get the 79 fakes right.
They would get the 21 real ones wrong.
Their "Accuracy" score would be 79%. That sounds great!

But in reality, that guard is useless. They are just guessing based on the odds. The paper argues that looking at "Accuracy" is like judging a chef only by how many plates they served, ignoring whether the food was edible.

Instead, the authors used a smarter score called MCC (Matthews Correlation Coefficient). This score only goes up if you get both the fakes right and the real ones right. It punishes the lazy "always say fake" guard.

The Surprising Results
When they ranked everyone by this smarter score, here is what they found:

The Pros Won (Mostly): The paid commercial services generally did the best job. They were the most reliable guards.
The Generalists were Middle-of-the-Road: The smart chatbots did okay, but they weren't as good as the pros.
The DIY Crowd was a Mixed Bag: Most of the free tools were the worst. However, a few specific free tools were actually better than the smart chatbots at spotting patterns.

The Big Twist: The "Ranking" vs. The "Decision"
This is the paper's biggest discovery. They found that a model can be great at ranking (sorting fakes higher than real photos) but terrible at deciding (actually saying "Fake" or "Real" when you ask it).

The "TruthScan" Example: One commercial tool was amazing at sorting. If you gave it a list of 100 photos, it could perfectly separate the fakes from the real ones in order. Its "Ranking Score" was the highest of all.
The Problem: But when you asked it to make a final decision, it was so scared of missing a fake that it decided EVERYTHING was fake. It failed the test because it flagged all the real photos as fakes.

It's like a metal detector at an airport that beeps for everything—coins, keys, belt buckles, and guns. It has perfect "ranking" (it finds all the metal), but it's useless as a security guard because it never stops anyone from walking through.

The Takeaway
The paper concludes that you cannot just look at a single number (like a leaderboard score) to decide which detector to use.

If you only look at Ranking, you might pick a tool that is great at sorting but terrible at making decisions.
If you only look at Accuracy, you might pick a lazy tool that just guesses "Fake" all the time.

To pick the right guard, you need to check both how well they can tell the difference between real and fake, and how well they make the final call without making too many mistakes. The authors released all their data and tools so other people can run these tests again and verify the results.

Technical Summary: VendorBench-100

Problem Statement

The rapid democratization of synthetic media generation has shifted the burden of detecting deepfakes from specialized forensic laboratories to general platforms and individuals. Currently, three distinct paradigms compete to fill this detection gap, yet they are rarely evaluated under a common protocol, making direct comparison impossible:

Commercial APIs: Proprietary services offering "black-box" detection with unverified accuracy claims.
Zero-Shot Vision-Language Models (LLMs): General-purpose multimodal models prompted ad hoc for forgery detection without specific training.
Open-Source Detectors: A fragmented ecosystem of academic checkpoints and community classifiers requiring local infrastructure.

Existing benchmarks typically evaluate these paradigms in isolation, often using large, homogeneous datasets that fail to reflect the "messy, adversarial content" circulating in the real world. Furthermore, standard evaluation metrics like raw accuracy are misleading on imbalanced datasets, and there is a lack of unified frameworks to assess whether a model's ranking ability (discriminative power) translates to reliable operating-point decisions (default threshold accuracy).

Methodology

The authors introduce VendorBench-100, a unified cross-paradigm benchmark designed to evaluate 36 representative models across the three paradigms (5 Commercial APIs, 7 Zero-Shot Vision LLMs, and 24 Open-Source detectors) using a single, fixed, adversarial corpus.

The Corpus

Composition: A curated set of 100 images (79 fake, 21 real).
Design Philosophy: Optimized for difficulty rather than scale. Every image was hand-selected to probe distinct, under-served failure modes rather than to provide statistical power for training.
Edge-Case Taxonomy: The 79 fake images span eight distinct families:
1. Live face-swap smearing (entangled with webcam motion blur).
2. Near-duplicate online face swaps (global statistics remain authentic).
3. Letterboxed text-to-video stills (cinematic grading mimics post-processing).
4. AI avatar compositing seams (glowing outlines missed by frequency detectors).
5. Fully synthetic text-to-image output (high-resolution, photorealistic).
6. On-device AI photo edits of real scenes (localized synthesis in authentic backgrounds).
7. Opaque-provenance manipulations (unknown tools).
8. Compressed research-dataset frames (low-resolution, waxy skin blending).
Anti-Leakage: Images are served with neutral numeric filenames to prevent models (especially LLMs) from inferring labels from metadata.

Evaluation Protocol

Unified Schema: All models are normalized to a shared output record containing a hard label (FAKE/REAL), a confidence score $P(fake) \in [0, 1]$ , and a success flag.
Metrics:
- Primary Metric: Matthews Correlation Coefficient (MCC). Chosen because the 79:21 class imbalance makes raw accuracy trivially gameable (e.g., a "always fake" predictor scores ~79% accuracy but 0.00 MCC). MCC accounts for class balance and requires performance on both classes to be high.
- Tie-Breaker: ROC-AUC. A threshold-independent measure of ranking ability.
- Secondary Metrics: Accuracy, F1, Precision, Recall, Specificity, and Coverage.
Execution: Commercial APIs were queried live; Vision LLMs were prompted zero-shot with a shared forensic schema; Open-source detectors were run locally on GPU.

Key Contributions

Cross-Paradigm Benchmark: The first unified evaluation of 36 models across commercial, LLM, and open-source tracks using a single adversarial corpus and normalization schema.
Adversarial Corpus Construction: A deliberately difficult 100-image dataset organized around eight specific failure-mode families, accompanied by a provenance registry and anti-leakage protocols.
Metric Divergence Analysis: The identification of a consistent, critical divergence between ranking power (ROC-AUC) and operating-point quality (MCC), demonstrating that strong score discrimination does not guarantee reliable default-threshold decisions.
Reproducibility: Release of the full evaluation framework, per-image evidence, and aggregated results.

Results

The evaluation reveals a complex landscape where paradigm membership predicts typical performance but fails to predict individual model capability:

Paradigm Ordering: Commercial APIs generally achieve the strongest median performance, followed by Vision LLMs, with the open-source track trailing in median performance.
Exceptions: Individual open-source models can outperform the best Vision LLMs in pure ranking power. Specifically, the open-source model DRCT achieved an ROC-AUC of 0.866, outperforming all Vision LLMs (the best being Claude Opus 4.8 at 0.791).
The "Accuracy Trap": Several models achieved deceptively high accuracy or F1 scores by defaulting to the majority class ("fake"). For example, zai_glm52 (Vision LLM) scored 71% accuracy and 0.83 F1 but had 0% specificity and a negative MCC (-0.152), indicating it flagged all real images as fake.
The Divergence Finding: The paper's central empirical finding is the disconnect between ROC-AUC and MCC:
- TruthScan (Commercial): Achieved the highest ROC-AUC (0.915) but the worst MCC among commercial APIs (-0.130) because its default threshold flagged nearly every image as fake.
- Neural Defend (Commercial): Achieved the highest MCC (0.876) but a near-chance ROC-AUC (0.516), indicating excellent decision calibration despite weak underlying score separation.
- Implication: A model can be an excellent ranker (high AUC) but a poor decision-maker (low MCC) at its shipped threshold, and vice versa.

Significance and Claims

The paper claims that VendorBench-100 fundamentally shifts the evaluation focus from isolated leaderboard rankings to a more nuanced understanding of detector behavior.

Metric Disagreement is Central: The divergence between ranking ability and operating-point quality is presented as a more significant finding than any single model's rank. It demonstrates that reporting only one metric (e.g., AUC or Accuracy) can systematically mischaracterize a model's real-world utility.
Practical Guidance: For practitioners, the benchmark suggests that procurement decisions must weigh both MCC and ROC-AUC. Relying on a single metric can lead to deploying a model that is either unreliable (high AUC, low MCC) or conservative to the point of uselessness (low AUC, high specificity but low recall).
Paradigm Nuance: While commercial APIs lead on average, the study refutes the notion that open-source detectors are universally inferior, showing that carefully designed academic methods (like DRCT) remain competitive with general-purpose foundation models.
Limitations: The authors explicitly state that the results are diagnostic signals from a single, small (100-image), adversarial corpus. They do not claim these results represent population-level accuracy estimates or that statistical significance testing was performed across all 630 possible pairwise comparisons. The corpus is intended as a stress test to surface failure modes rather than a population benchmark.

In conclusion, the paper argues that the field must move toward unified, reproducible, and multi-metric evaluations that prioritize fair comparisons across paradigms over isolated leaderboard scores.