💻 computer science

Counterfeit Answers: Adversarial Forgery against OCR-Free Document Visual Question Answering

This paper introduces a novel adversarial attack scenario called "Counterfeit Answers" that forges document content in visually imperceptible ways to manipulate OCR-free Document Visual Question Answering models into generating specific or incorrect answers, demonstrating critical vulnerabilities in state-of-the-art systems like Pix2Struct and Donut.

Original authors: Marco Pintore, Maura Pintor, Dimosthenis Karatzas, Battista Biggio

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Marco Pintore, Maura Pintor, Dimosthenis Karatzas, Battista Biggio

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, automated robot assistant whose job is to read invoices, receipts, and contracts. You ask it questions like, "How much do I owe?" or "What is the invoice number?" and it looks at the image of the document and gives you the answer. This technology is called Document Visual Question Answering (DocVQA).

This paper is a security report that says: "This robot is easily tricked by invisible magic tricks."

Here is a simple breakdown of what the researchers found, using everyday analogies:

1. The Setup: The Robot and the Document

Think of the DocVQA model as a super-fast librarian who can read a document and answer questions about it instantly.

Old way: The librarian would first use a scanner (OCR) to turn the picture of the paper into typed text, then read that text.
New way (The paper's focus): The librarian looks directly at the picture of the paper, understanding the layout, the fonts, and the numbers all at once, without needing to "type it out" first. This is faster and smarter.

2. The Problem: The "Invisible Ink" Trick

The researchers discovered that you can fool this smart librarian by making tiny, almost invisible changes to the document image.

The Analogy: Imagine you have a receipt. You take a magic marker and change a single pixel (a tiny dot of color) in the corner of the paper. To your human eye, the receipt looks exactly the same. But to the robot, that tiny change is like a loud siren that screams, "Ignore the total price! The total is now $0.00!"
The Result: The robot confidently gives you the wrong answer, even though the document looks unchanged to a human.

3. The Two Ways They Did It

The researchers tested two methods of "tricking" the robot:

The "Full-Document" Trick: They scattered tiny, invisible changes all over the entire page. This is like dusting the whole receipt with invisible powder that confuses the robot's brain.
The "Patch" Trick: They put a small, invisible "sticker" (a patch) in just one corner of the document (like the bottom right). Surprisingly, this tiny sticker was enough to make the robot forget the total amount or change the invoice ID.

4. The Three Types of Tricks

The researchers showed they could trick the robot in three specific ways:

The "Specific Lie" (Targeted Single): They made the robot say a specific wrong answer for just one question. Example: Making the robot say the total is $0.00 when it's actually $128.
The "Mass Confusion" (Targeted Multi): They tried to make the robot lie about multiple questions at once. Example: Changing the total, the date, and the invoice number all at the same time. They found this is harder to do; the robot gets confused, but it's tough to make it lie about everything perfectly at once.
The "Refusal to Answer" (Denial of Answer): They didn't care what the wrong answer was; they just wanted the robot to fail completely. Example: Making the robot say "I don't know" or "Retry" for every question. This was very easy to achieve.

5. The "White-Box" Secret

The researchers admit they had a huge advantage in their experiment. They knew exactly how the robot's brain was built (its code, its settings, and how it processes images).

The Analogy: It's like a magician practicing a trick on a friend who is wearing a headset that tells them exactly which card the magician is about to pick.
Why it matters: Even with this unfair advantage, the fact that they could break the system so easily proves the system is fragile. If a hacker can do this with full knowledge, they might eventually figure out how to do it without knowing the secrets (though that is harder).

6. The Real-World Danger

The paper warns that if we let AI agents (robots) handle money or legal documents automatically, these "invisible ink" tricks could be dangerous.

The Scenario: A bad actor could send an invoice with a tiny, invisible patch. An automated banking system reads it, gets tricked, and authorizes a payment of $0.00 instead of the real amount, or approves a fake transaction.

Summary

The paper doesn't say this technology is useless. Instead, it's a "Warning Label." It says: "We built a very smart robot to read documents, but we just found out that a tiny, invisible speck of dust can make it hallucinate completely wrong answers."

The researchers released their "magic tricks" (code) so that other scientists can study them and build stronger shields to protect these systems in the future.

Technical Summary: Counterfeit Answers: Adversarial Forgery against OCR-Free Document Visual Question Answering

Problem Statement

Document Visual Question Answering (DocVQA) systems, particularly those utilizing OCR-free end-to-end architectures, have emerged as powerful tools for reasoning over document images. These models process raw images and natural language questions to generate answers, bypassing the computational costs and error propagation associated with traditional Optical Character Recognition (OCR) pipelines. However, the vulnerability of Deep Learning (DL) models to adversarial perturbations—subtle input manipulations that trigger undesired behaviors—has not been thoroughly investigated in the context of DocVQA.

The paper identifies a critical security gap: the potential for "document forgery" via adversarial attacks. Unlike standard image classification attacks, DocVQA attacks aim to manipulate specific semantic outputs (answers) derived from complex document layouts. The authors posit that even visually imperceptible perturbations could allow adversaries to induce specific incorrect answers (e.g., altering financial totals on an invoice) or cause systematic model failure. In automated pipelines, such as those enabled by protocols like the Agents Payments Protocol (AP2), these vulnerabilities could lead directly to monetary loss or unauthorized transactions.

Methodology

Threat Model

The authors formalize a white-box threat model where the adversary possesses complete knowledge of the target DocVQA system, including its architecture, preprocessing pipelines, parameters, and gradients. The attacker has access to the original document image $x$ and the ground-truth answers $y$ , but the question $q$ is fixed (e.g., a system prompt). The attacker's goal is to generate a perturbed document $x' = x + \delta$ that causes the model to output a manipulated answer $y^*$ .

The attack operates under the following constraints:

Input Space Manipulation: Perturbations are applied directly to the raw pixel values of the document image before any model-specific preprocessing.
Imperceptibility: The perturbation $\delta$ is constrained by an $\ell_\infty$ norm ( $\|\delta\|_\infty \leq \epsilon$ ) to ensure it remains visually subtle.
Scenarios: The framework supports three attack scenarios:
1. Targeted Single-QA: Forcing a specific incorrect answer for one question.
2. Targeted Multi-QA: Forcing specific incorrect answers for multiple questions simultaneously.
3. Denial of Answer (DoA): Inducing generally incorrect answers for all questions on a document (availability violation).

Attack Formulation and Optimization

The authors employ Projected Gradient Descent (PGD) to optimize the perturbation. The objective function minimizes the loss between the model's output and a target answer $y^*$ (for targeted attacks) or maximizes the loss against the ground truth (for untargeted DoA attacks).

A key technical contribution is the handling of model-specific preprocessing. Since DocVQA models like Pix2Struct and Donut employ unique, often non-differentiable preprocessing steps (e.g., patch extraction, specific normalization, question rendering), the authors reverse-engineered these pipelines to create fully differentiable computational graphs. This allows gradients to flow back from the model output to the raw input image, enabling end-to-end optimization.

Two distinct loss functions were utilized:

Standard Fine-tuning Loss: The negative log-likelihood used in the original model training.
Custom Logit Loss: A novel loss function designed specifically to improve attack efficacy on the Donut model. This loss penalizes only those target tokens that are not currently the maximum logit, guiding the optimization to "bring out" target tokens without disturbing already correct ones.

Experimental Setup

The study evaluates attacks on two state-of-the-art OCR-free models:

Pix2Struct: A vision-language transformer that renders the question as a visual header on the image.
Donut: A transformer-based model that accepts the question as a text prompt.

Experiments were conducted on the PFL-DocVQA dataset, utilizing a subset of 1,000 unique invoice documents, each with 5 associated question-answer pairs. The authors tested both full-document attacks (perturbing the entire image) and patch-based attacks (perturbing a localized region, e.g., a 15% square in the bottom-right corner).

Key Results

Attack Success Rate (ASR)

Targeted Single-QA: The attacks were highly effective. For full-document attacks, ASR reached nearly 100% for Pix2Struct and just below 80% for Donut. Patch attacks were also successful for Pix2Struct (~100% ASR) but significantly less effective for Donut (<20% ASR).
Targeted Multi-QA: As the number of optimized questions ( $B$ ) increased, the ASR decreased, reflecting the increased difficulty of satisfying multiple constraints simultaneously. Pix2Struct maintained high ASR for small $B$ , while Donut's performance dropped rapidly, approaching zero at $B=2$ .
Denial of Answer (DoA): The untargeted attacks proved highly effective at degrading model performance. Even with a single optimization step ( $K=1$ ), the models were disrupted. The ASR for DoA remained high for $B=1$ but decreased as more questions were targeted, though the drop was less severe for Donut compared to the targeted scenario.

Collateral Damage (CDMG) and ANLS

Collateral Damage: For targeted single-QA attacks, CDMG remained low, indicating that non-targeted answers on the same document largely remained correct. However, in multi-QA and DoA scenarios, CDMG increased as expected, particularly for DoA where the goal is global disruption.
ANLS Scores: The Average Normalized Levenshtein Similarity (ANLS) metrics confirmed the attack dynamics. In targeted attacks, ANLS-B (similarity to target answers) increased, while ANLS-C (similarity to ground truth for non-targeted answers) remained high. In DoA attacks, ANLS-B (similarity to ground truth) decreased significantly, confirming the degradation of correct outputs.

Significance and Claims

The paper claims to present the first threat model specifically analyzing the adversarial vulnerability of OCR-free DocVQA systems. Its primary significance lies in demonstrating that:

End-to-End Vulnerability: Current DocVQA systems are susceptible to adversarial forgery where small, localized, or full-document perturbations can induce specific, semantically incorrect answers.
Real-World Impact: These attacks are not merely theoretical; they can be crafted to manipulate financial data (e.g., invoice totals) or trigger denial-of-service conditions in automated document processing pipelines.
Model Sensitivity: Different architectures exhibit varying levels of robustness. Pix2Struct was found to be more vulnerable to patch attacks than Donut, while Donut struggled more with multi-target optimization.
Call for Defense: The findings highlight a critical need for robust defenses, including adversarial training adapted to document modalities, detection of anomalous inputs, and hardening of preprocessing pipelines.

The authors emphasize that while their study assumes a white-box setting (an upper bound on attack capability), the results establish a clear upper bound on the effectiveness of such attacks, urging the community to develop principled mitigations before these vulnerabilities are exploited in the wild.