AIForge-Doc: A Benchmark for Detecting AI-Forged Tampering in Financial and Form Documents

This paper introduces AIForge-Doc, the first benchmark featuring pixel-level annotated AI-inpainted financial and form documents, which reveals that current state-of-the-art forgery detectors and vision-language models fail to distinguish AI-generated tampering from authentic content, highlighting a critical vulnerability in document forensics.

The Gist

🕵️‍♂️ The Big Problem: The "Magic Eraser" for Receipts

Imagine you have a receipt from a coffee shop. It says you bought a latte for $5.00.

In the past, if a criminal wanted to change that price to $50.00, they would have to use Photoshop. They'd have to cut out the "5," paste a new "50" on top, and try to match the font. But when they did this, they left "fingerprints": weird pixel blurs, mismatched shadows, or jagged edges. Computer programs (detectives) were very good at spotting these messy fingerprints.

But now, the game has changed.

New AI tools (like Gemini and Ideogram) act like a "Magic Eraser." You tell the AI, "Change the price to $50," and it doesn't just paste a new number on top. It re-imagines the pixels from scratch. It draws the new number so perfectly that it matches the font, the paper texture, the lighting, and the background noise of the photo.

To the human eye, and even to the best computer programs we have today, the fake receipt looks 100% real.

🛠️ What Did the Researchers Do? (The "AIForge-Doc" Project)

The researchers at Duke, NYU, and others realized: "We have no way to test if our security cameras can spot these new, perfect fakes."

So, they built a new training ground called AIForge-Doc.

Think of it like a driving school for forgery detectors.

1. The Students: They took 4,000 real receipts and forms (from datasets like CORD, WildReceipt, etc.).
2. The Teachers: They used two powerful AI tools to secretly change the numbers on these receipts (changing prices, dates, or phone numbers).
3. The Answer Key: For every fake receipt, they kept a "secret map" (a pixel-perfect mask) showing exactly where the AI touched the image.

They created 4,061 of these "perfectly forged" documents. This is the first time anyone has made a dataset specifically for this kind of AI trickery.

🧪 The Test: Can the Detectives Find the Fake?

The researchers took three different "detectives" (AI programs) and asked them to look at these new forged receipts. They didn't teach the detectives anything new; they just threw the new fakes at them to see what happened.

Here is how the detectives did:

1. The "Old School" Detective (TruFor)

• Who it is: A famous program trained to spot Photoshop edits.
• The Result: It got confused. It used to be a champion (96% accuracy) on old-style fakes, but on these new AI fakes, its score dropped to 75%.
• The Metaphor: Imagine a security guard who is great at spotting people wearing fake mustaches. But now, the criminals are wearing perfect, realistic masks. The guard sees a face, but can't tell if it's real or a mask.

2. The "Document Specialist" (DocTamper)

• Who it is: A program specifically trained to find tampered receipts.
• The Result: It failed miserably. It scored 56%, which is barely better than flipping a coin.
• The Metaphor: This detective was trained to find "glue residue" (clues left by cutting and pasting). But the AI didn't use glue; it used magic. The detective looked for glue, found none, and assumed everything was safe.

3. The "Super-Intelligent Robot" (GPT-4o)

• Who it is: A very smart AI that knows a lot about the world.
• The Result: It scored 51%. This is random guessing.
• The Metaphor: You ask a genius, "Does this receipt look fake?" The genius looks at the numbers and says, "Well, $50 is a plausible price for a latte." It can't tell the difference because the forgery is so perfect that the logic of the document still holds up.

🚨 The Big Takeaway

The paper concludes with a scary but important truth: We are currently blind to AI-forged documents.

• The Gap: Our current security tools are like night-vision goggles designed for the dark forest of the past. They don't work in the bright, confusing fog of AI-generated fakes.
• The Danger: Criminals can now change a bank transfer amount, a tax form, or a medical bill in under a second for a few cents, and leave no trace.
• The Solution: The researchers released their dataset (AIForge-Doc) to the public. They are essentially saying, "Here is a pile of perfect fakes. We need you to build new detectors that can learn to spot the invisible fingerprints of AI."

💡 In a Nutshell

We used to worry about bad photocopies. Now, we have to worry about perfect forgeries that look realer than reality. The researchers built a "fake receipt gym" to train our defenses, and they found that our current defenses are completely unprepared for this new threat. It's time to build a new kind of security system.

Technical Summary

1. Problem Statement

The paper addresses a critical gap in document forensics: the inability of current detection systems to identify forgeries created by diffusion-model-based inpainting APIs (e.g., Gemini 2.5 Flash Image, Ideogram v2 Edit).

• The Threat: Unlike traditional forgeries created with tools like Photoshop (which leave compression seams, cloning patterns, or statistical noise anomalies), AI inpainting synthesizes pixels that are statistically consistent with the original document's font, texture, and background. This makes the edits visually imperceptible and statistically indistinguishable from authentic content.
• The Gap: Existing benchmarks (e.g., DocTamper, OSTF) rely on traditional editing or focus on scene-text rather than structured financial documents. Consequently, state-of-the-art detectors are "blind" to AI-generated document fraud, a rapidly growing threat where digital forgeries now outnumber physical counterfeits.

2. Methodology: Dataset Construction

The authors constructed AIForge-Doc, the first benchmark dedicated to AI-inpainted financial and form documents with pixel-level annotations.

• Source Data: The dataset is built from four public document datasets spanning nine languages:
  • CORD: Indonesian receipts (focus on prices/totals).
  • WildReceipt: English receipts (focus on phone numbers, addresses, prices).
  • SROIE: English receipts (focus on dates, totals, policy text).
  • XFUND: Multilingual forms (focus on numeric answer fields).
• Forgery Generation Pipeline:
  1. Field Selection: A prioritization logic selects high-risk numeric fields (Financial Amount > Date > ID > Quantity).
  2. Value Mutation: Plausible alternative values are generated (e.g., scaling prices by 1.15x–3.0x, perturbing dates, flipping ID digits) while matching character count to minimize visual disruption.
  3. Context-Window Inpainting: To prevent global hallucinations, the system crops a region around the target field (expanding the bounding box by 50%), creates a binary mask, and sends this to the AI API with a strict prompt to preserve font and background. Only the specific field region is pasted back into the original image.
  4. Tool Selection: Seven AI inpainting systems were evaluated. Only Gemini 2.5 Flash Image and Ideogram v2 Edit produced character-accurate, legible text. The other five (including FLUX and Stable Diffusion variants) failed due to garbled text or inability to understand multimodal context.
• Dataset Statistics:
  • Total Images: 4,061 forged images (80/20 train/test split).
  • Annotation: Pixel-precise binary masks (DocTamper-compatible format).
  • Tamper Scale: The tampered region is highly localized, comprising a median of 0.92% of the total image pixels (often <1%), making detection analogous to finding a "needle in a haystack."

3. Key Contributions

1. Novel Dataset: A large-scale, pixel-level annotated dataset of AI-forged financial documents, specifically targeting the high-risk scenario of numeric field manipulation.
2. Reproducible Pipeline: An open-source, automated pipeline for generating high-fidelity forgeries using context-window inpainting, including a prompt ablation study that proved only multimodal models with language understanding can perform this task effectively.
3. Baseline Evaluation: A rigorous zero-shot evaluation of existing detectors, revealing a catastrophic failure rate in the face of AI inpainting.

4. Experimental Results

The authors benchmarked three representative detectors on the AIForge-Doc test set without fine-tuning (zero-shot):

Detector	Type	Performance on AIForge-Doc (AUC)	Performance on Original In-Distribution	Key Finding
TruFor	General Forensic	0.751	0.96 (NIST16)	Significant degradation. While pixel-level AUC is high (0.916), image-level confidence and localization (IoU=0.358) fail.
DocTamper	Document-Specific	0.563	0.98 (Own Test Set)	Near-random performance. Pixel-level IoU drops from 0.71 to 0.020, indicating total failure to localize AI edits.
GPT-4o	VLM Judge	0.509	N/A	Essentially random chance. Semantic reasoning fails because forged numbers look valid in isolation.

Key Findings:

• General Detectors Fail: TruFor, which relies on camera noise fingerprints (NoisePrint++), struggles because AI inpainting generates noise consistent with the surrounding authentic region.
• Domain Shift is Insufficient: The failure is not just due to domain shift (receipts vs. photos) but the fundamental difference between Photoshop artifacts and diffusion synthesis.
• VLMs are Blind: GPT-4o cannot detect the forgery because the AI-generated text is semantically plausible and visually consistent with the document layout.
• Tool Variance: TruFor performed slightly better on Gemini forgeries (AUC 0.778) than Ideogram forgeries (AUC 0.521), suggesting different artifact signatures, though Ideogram's small test set requires caution in interpretation.

5. Significance and Conclusion

• Qualitative Shift: AIForge-Doc demonstrates that document forensics has entered a new, unsolved challenge. The "statistical consistency" of diffusion models renders traditional forensic features (compression artifacts, cloning traces) obsolete.
• Urgency: With AI forgery tools costing ~$0.01 per edit and taking <1 second, the barrier to entry for financial fraud has collapsed. Current defenses are ineffective.
• Future Directions: The paper calls for:
  • New detectors specifically trained on AI-inpainting artifacts.
  • Research into multi-field tampering detection (fraudsters often alter correlated values).
  • Expansion to other document types (invoices, contracts) and non-Latin scripts.

In summary, AIForge-Doc serves as a wake-up call for the computer vision and forensic communities, proving that current state-of-the-art methods are fundamentally ill-equipped to handle the next generation of document fraud.