DOCFORGE-BENCH: A Comprehensive 0-shot Benchmark for Document Forgery Detection and Analysis

DOCFORGE-BENCH introduces the first unified zero-shot benchmark for document forgery detection, revealing that current methods suffer from severe calibration failures due to the extreme rarity of tampered pixels in documents, which renders standard fixed thresholds ineffective and highlights threshold adaptation as the critical missing step for practical deployment.

The Gist

Imagine you are a security guard at a bank. Your job is to spot fake documents—like altered receipts, doctored IDs, or tampered contracts. For years, you've been trained using photos of fake paintings and edited landscapes (natural images). You learned to spot the "brushstrokes" of a forgery in those photos.

Now, someone hands you a stack of real bank receipts and ID cards. You look at them, and your training kicks in. You can feel something is wrong. You can tell which pixels are "suspicious" and which are "safe." But when you try to point your finger and say, "This specific letter is fake," you fail miserably. You end up accusing the whole page of being fake, or you miss the tiny typo entirely.

This is exactly what the paper DOCFORGE-BENCH discovered.

Here is the story of the paper, broken down into simple concepts:

1. The Big Experiment: "No Cheating Allowed"

The researchers wanted to see if the best AI "detectives" in the world could spot fake documents without any special training on documents.

• The Old Way: Most previous tests let the AI study the specific type of document it was about to judge (like studying the answer key before the test).
• The New Way (DOCFORGE-BENCH): They took 14 different AI models, gave them their "frozen" brains (pre-trained weights), and threw them into a room with 8 different types of documents (receipts, IDs, text-heavy forms). They were not allowed to study the documents first. This is called a "zero-shot" test—it's like asking a chef who only cooks Italian food to judge a Thai dish without tasting it first.

2. The Shocking Discovery: The "Confident but Clueless" Problem

The results were surprising. The AI models were actually smart about what was fake, but stupid about how to decide.

• The Analogy: Imagine a metal detector at an airport.
  • The AI's Brain (AUC): The detector is working perfectly. It beeps loudly for a gun, a medium beep for a belt buckle, and a soft beep for a coin. It knows the difference between a threat and a harmless object. In the paper, this is called a high Pixel-AUC (the model knows what is fake).
  • The AI's Decision (Pixel-F1): The guard standing next to the detector has a rule: "If the beep is louder than a 'medium' volume, arrest the passenger."
  • The Disaster: Because the fake parts of a document are so tiny (like a single changed number on a receipt), the "beep" for a fake document is very quiet. But the guard's rule is set to only catch "loud" beeps. So, the guard ignores the quiet beeps (the fakes) and lets them through.
  • The Result: The detector works (high AUC), but the guard catches nothing (near-zero F1 score).

3. Why Did This Happen? The "Needle in a Haystack" Problem

The paper explains that documents are different from photos.

• In a Photo: If someone edits a photo, they might change a whole sky or a person's face. That's a big chunk of the image (10–30% of the pixels). The AI's "alarm threshold" is set to catch big changes.
• In a Document: A forgery is usually just changing one number or one name. That's less than 1% of the image. It's a needle in a haystack.
• The Mismatch: The AI models were trained on "big changes." When they see a "tiny change," their internal alarm doesn't ring loud enough to cross the "arrest" line. The paper calls this a Calibration Failure. The AI isn't blind; it just has the volume knob turned down too low for the job.

4. The Solution: Turning the Volume Knob

The researchers found a simple fix. They didn't need to retrain the AI or teach it new tricks. They just needed to adjust the sensitivity.

• The Experiment: They took a tiny sample of 10 real documents and asked, "At what volume does the alarm start ringing for these?"
• The Result: By simply lowering the threshold (making the AI more sensitive to quiet beeps), the detection rate jumped from near-zero to 40–50% of its potential.
• The Lesson: The problem wasn't that the AI was "dumb." The problem was that the rule for "what counts as a fake" was wrong for documents.

5. The Hard Truth

Despite having the smartest AI models in the world, none of them work perfectly right out of the box.

• If you buy a "Document Forgery Detector" today and try to use it on a real receipt, it will likely fail.
• The models trained specifically on documents were actually worse at handling new types of documents than the general models, because they had memorized the wrong patterns (overfitting).

6. The Future: The "AI-Generated" Wild West

The paper ends with a warning. All the fake documents used in this test were made with old-school editing tools (like Photoshop or Paint).

• The New Threat: Today, people are using Generative AI (like DALL-E or Stable Diffusion) to create fake documents. These forgeries look completely different.
• The Gap: The current AI detectors are like guards trained to spot "hand-drawn forgeries." They have no idea how to spot "AI-generated forgeries." The paper suggests that if we tested these models on AI-generated fakes today, they would likely fail completely.

Summary

DOCFORGE-BENCH is a reality check. It tells us that while our AI detectors are smart enough to see the forgery, they are currently too rigid to act on it when the forgery is tiny. We don't need smarter AI; we just need to teach them how to listen for the quiet whispers of a fake document. Until we fix this "calibration" issue, document forgery remains a major unsolved problem.

Technical Summary

Here is a detailed technical summary of the paper DOCFORGE-BENCH: A Comprehensive Benchmark for Document Forgery Detection and Analysis.

1. Problem Statement

The paper addresses a critical gap in the field of image forensics: the failure of existing document forgery detection methods when deployed in real-world, zero-shot scenarios.

• The Core Issue: While image manipulation detectors perform well on natural images (photos of people, landscapes), they fail catastrophically on document images (receipts, IDs, contracts).
• The Specific Failure Mode: Current methods exhibit a calibration failure rather than a discrimination failure. They can correctly rank tampered pixels above authentic ones (high AUC), but they fail to identify a usable decision threshold (near-zero F1 score) when using the standard $\tau=0.5$.
• Root Cause: Document forgeries involve extreme class imbalance. Tampered regions typically occupy only 0.27% to 4.17% of an image, whereas natural image benchmarks assume 10–30% tampered areas. This mismatch renders the standard 0.5 threshold miscalibrated, causing precision to collapse.
• Limitation of Existing Benchmarks: Previous evaluations (e.g., ForensicHub) rely on fine-tuning protocols, which mask the true "out-of-the-box" generalization capability of models and obscure the calibration issue.

2. Methodology

The authors introduce DOCFORGE-BENCH, a unified benchmark designed to evaluate methods under strict zero-shot conditions.

A. Evaluation Protocol

• Zero-Shot / Frozen Weights: All 14 evaluated methods are tested using their published pretrained weights with no domain adaptation or fine-tuning. This simulates a realistic deployment scenario where practitioners lack labeled document training data.
• Metrics: The benchmark utilizes two complementary metrics to diagnose failure modes:
  1. Pixel-F1 (@ $\tau=0.5$): Measures deployment performance without calibration.
  2. Pixel-AUC: Measures the model's ability to rank tampered pixels above authentic ones, independent of threshold calibration.
  3. Oracle-F1: The theoretical maximum F1 achievable by selecting the optimal threshold per image, used to quantify the "calibration gap."

B. Dataset Coverage

The benchmark spans 8 diverse datasets covering three primary threat models:

1. Text-Region Tampering: DocTamper, T-SROIE, RealTextManipulation, Tampered-IC13, FSTS-1.5k. (Covers synthetic and real-world text replacement/insertion).
2. Commercial Receipt Forgery: ReceiptForgery. (Price/quantity manipulation using standard editors).
3. Identity Document Forgery: FantasyID. (Face swaps and text replacement on ID cards).

• Scale: Ranges from 35 forged images (ReceiptForgery) to ~170,000 images (DocTamper).
• Annotation: Includes both pixel-level masks and bounding boxes (rasterized for evaluation).

C. Methods Evaluated

A total of 14 methods were evaluated, categorized into:

• General Image Forensics (7): TruFor, ManTraNet, MVSS-Net, CAT-Net, PSCC-Net, IML-ViT, SAFIRE. (Trained on natural images).
• Document-Specific Methods (7): DocTamper (model), DTD, FFDN, CAFTB-Net, TIFDM, ASCFormer, ADCD-Net. (Trained on document datasets like DocTamper or RTM).

3. Key Contributions

1. First Unified Zero-Shot Benchmark: DOCFORGE-BENCH is the first benchmark to evaluate document forgery detection without fine-tuning, isolating true generalization capabilities.
2. Diagnosis of the Calibration Gap: The paper empirically establishes that the primary bottleneck in document forensics is score-distribution shift, not feature discrimination. Models retain high AUC ($\ge 0.76$) but fail at fixed thresholds due to extreme class imbalance.
3. Quantitative Explanation: The authors mathematically demonstrate that the standard $\tau=0.5$ threshold is invalid for documents where tampered pixels constitute $<1\%$ of the image.
4. Calibration Recovery Proof: Through controlled experiments, the authors show that adapting a single global threshold on a tiny sample of domain data ($N=10$ images) can recover 39–55% of the performance gap between fixed-threshold F1 and Oracle-F1.
5. Identification of Open Gaps: The paper highlights that current benchmarks predate the era of Generative AI (Diffusion/LLM) editing, leaving a critical blind spot for modern attack surfaces.

4. Key Results

• No Reliable Out-of-the-Box Method: No evaluated method achieves a reliable Pixel-F1 ($\ge 0.3$) across six of the eight datasets. Document forgery detection remains an unsolved problem in zero-shot settings.
• The AUC–F1 Gap:
  • General Methods: TruFor and CAT-Net showed strong AUC ($\ge 0.8$) but low F1 on most document datasets.
  • Document-Specific Methods: Models like the DocTamper model achieved high in-domain F1 (0.91) but collapsed on out-of-domain datasets (e.g., dropping to 0.045 on T-SROIE), indicating severe overfitting to specific rendering artifacts.
  • Parity: The best document-specific method (CAFTB-Net) and the best general method (TruFor) achieved statistically indistinguishable mean F1 scores ($\approx 0.34$) across the benchmark.
• Calibration is the Bottleneck:
  • Oracle-F1 was found to be 2–10 times higher than fixed-threshold Pixel-F1.
  • This confirms that the models can distinguish forgeries, but their confidence scores are systematically shifted below 0.5.
• Threshold Adaptation: A simple threshold adjustment on just 10 domain images recovered up to 55% of the lost performance, proving that retraining is not necessary for significant improvement; only calibration is needed.

5. Significance and Future Directions

• Paradigm Shift: The paper challenges the community to move away from fine-tuning-centric evaluations and focus on calibration-aware metrics and protocols. It suggests that the "failure" of current detectors is often a deployment configuration issue, not a fundamental representation failure.
• Practical Deployment: The findings imply that practitioners can deploy existing forensic models on documents if they perform a minimal calibration step (threshold tuning) on a small set of local data.
• Future Work: The authors identify Generative AI forgeries (Diffusion models, LLM-based text editing) as the next critical frontier. Current detectors are likely to fail even more severely on these new attack vectors, and the benchmark toolkit is designed to support future evaluations of these emerging threats.

In summary, DOCFORGE-BENCH reveals that document forgery detection is currently an unsolved problem in zero-shot settings due to severe calibration mismatches caused by extreme class imbalance, rather than a lack of discriminative power in the models themselves.