Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models

1. Problem Statement

The deployment of Large Vision-Language Models (LVLMs) for real-world document question answering (QA) faces a critical safety challenge: dynamic, user-defined policy preservation. Unlike static social norms, these policies are specific constraints (e.g., "Do not disclose revenue for the Middle East region") that dictate what information can be revealed based on context.

The paper identifies a "Reasoning-Induced Safety Gap":

• Cross-Modal Leakage: Models often fail to adhere to policies when answering queries that require synthesizing information across heterogeneous modalities (e.g., combining a percentage from a pie chart with a total revenue figure from text to calculate a protected value).
• Implicit vs. Explicit Failure: While models may comply with explicit requests to withhold data, they frequently leak sensitive information when the data must be inferred through complex reasoning or aggregated across multiple document sections.
• The OCR Paradox: Providing models with OCR-extracted text (instead of raw images) improves perceptual clarity but inadvertently facilitates leakage by making numerical reasoning and value extraction easier, thus bypassing safety constraints.

2. Methodology: Doc-PP Benchmark

To systematically evaluate and address these issues, the authors introduce Doc-PP, a novel benchmark constructed from real-world PDF documents (financial reports, industry reports, etc.).

A. Dataset Construction Pipeline

The construction follows a three-stage process (illustrated in Figure 2 of the paper):

1. Policy Construction:
   • Target Generation: An LLM identifies confidential targets ($t$) requiring non-disclosure (e.g., specific revenue figures, strategic roadmaps).
   • Evidence Extraction: For each target, the system extracts ground-truth evidence ($v$) including page numbers, evidence type (text, table, chart), and verbatim quotes.
   • Filtering: A checklist-based verification ensures targets are grounded, uniquely identifiable, and do not inherently leak the value. This reduced the candidate set from 838 to 635 high-quality targets.
2. Query Construction:
   • Explicit Queries: Direct questions asking for the restricted target (e.g., "What are the details on $t$?").
   • Implicit Queries: Summary-style questions designed such that a faithful answer necessitates the disclosure of the protected value $v$ (e.g., "Summarize the regional performance, including specific figures").
3. Evaluation Framework:
   • Leakage: Binary check if the model output contains the confidential value $v$.
   • Faithfulness: Measured via a Content-Coverage Checklist. This ensures the model provides useful information (excluding the protected value) without hallucinating or omitting necessary non-sensitive facts.
   • Inputs: The benchmark tests two input modalities: $D_{img}$ (raw document images) and $D_{ocr}$ (OCR-parsed text + images).

B. Proposed Mitigation: DVA Framework

To address the identified vulnerabilities, the authors propose DVA (Decompose–Verify–Aggregation), a structural inference framework:

1. Decompose: The model breaks down a prospective answer into atomic information elements (individual facts/sub-claims).
2. Verify: Each atomic element is individually checked against the user-defined policy. Elements violating the policy are marked invalid.
3. Aggregate: Only the valid elements are synthesized into the final response.

• Key Innovation: This decouples the reasoning process from policy verification, preventing the "black box" synthesis where sensitive values are hidden within complex sentences.

3. Key Results

The paper evaluates six state-of-the-art LVLMs (including GPT-5.2, Gemini-3, Qwen3-VL) on the Doc-PP benchmark.

• Systemic Reasoning Gap: Models exhibit significantly higher leakage rates on Implicit Queries compared to Explicit Queries. When reasoning is required to synthesize data (e.g., calculating a ratio from a chart), safety barriers collapse.
• The OCR Paradox Confirmed: Models using OCR inputs showed higher leakage rates than those using raw images. For example, GPT-5.2's leakage on implicit queries rose from 25.7% (Image) to 25.8% (OCR), but other models showed more drastic increases, indicating that text accessibility aids in bypassing visual-based safety filters.
• Evidence Type Impact: Leakage is highest in Cross-Modality scenarios (e.g., Chart + Text). The complexity of integrating heterogeneous sources appears to circumvent existing safety mechanisms.
• Document Type Correlation: Documents with higher text density (Business/Financial reports) exhibited higher leakage rates (up to 84.6%) compared to visual-heavy documents, suggesting text acts as a "guide" for extracting protected visual values.
• Mitigation Efficacy:
  • Standard defenses like Chain-of-Thought (CoT) and Post-Hoc Revision failed to significantly reduce leakage (often remaining >50%).
  • DVA significantly outperformed baselines, reducing leakage rates by nearly half (e.g., Gemini-3-Flash leakage dropped from 64.6% to 30.5%). It successfully prevented leaks that CoT missed by verifying facts at the atomic level.

4. Key Contributions

1. Doc-PP Benchmark: The first benchmark specifically designed to evaluate user-defined policy preservation in multimodal document QA, moving beyond text-only or implicit social norm studies.
2. Discovery of Safety Gaps: Identification of the Reasoning-Induced Safety Gap and the OCR Paradox, highlighting that improved multimodal understanding and text extraction can paradoxically degrade safety.
3. DVA Framework: A novel, lightweight structural inference method that effectively decouples reasoning from policy verification, offering a robust baseline for secure document understanding.
4. Comprehensive Evaluation: A rigorous evaluation framework using checklist-based faithfulness metrics and leakage detection across diverse document types and input modalities.

5. Significance

This work is critical for the safe deployment of LVLMs in enterprise and legal sectors where conditional information disclosure is mandatory. It demonstrates that current safety alignment techniques are insufficient for complex, multimodal reasoning tasks. By proving that standard prompting fails and proposing DVA, the paper provides a practical pathway for building document QA systems that can dynamically adhere to strict, user-defined confidentiality policies without sacrificing utility. The findings suggest that future safety research must focus on granular, atomic verification rather than holistic response generation.