Technical Summary: DocPO: Advancing Document Policy Optimization via Tailored Step-Aware Rewards

1. Problem Statement

Document parsing, the task of decoding heterogeneous 2D layouts (text, tables, formulas) into 1D sequences, faces significant challenges when applying Reinforcement Learning (RL). While Supervised Fine-Tuning (SFT) is the current mainstream, it suffers from exposure bias. Although RL offers a path to sequence-level optimization, its application to document parsing has been limited compared to standard Large Language Models (LLMs).

The core issue identified is the weak discriminability of reference-based rewards in the high-accuracy regime. Existing methods often rely on edit-distance metrics (e.g., string edit distance). As models approach high accuracy, near-correct outputs receive scores that are nearly identical to ground truth. This results in:

• Flat reward landscapes: Small quality differences between high-scoring candidates are not amplified, providing limited learning signals for hard cases.
• Misalignment: Generic string-level edit distances fail to capture structural correctness in tables (rewarding high lexical overlap despite structural corruption) or semantic equivalence in formulas (penalizing valid synonymous LaTeX variations).
• Dependency on complex rewards: Alternative approaches, such as learned render-and-compare reward models, require additional data, compute, and auxiliary training, reducing deployment simplicity.

2. Methodology

The authors propose DocPO, a document policy optimization framework that retains simple, element-specific reference-based rewards but enhances their optimization power through a novel mechanism called Step-Aware Annealing (SAA).

2.1 Base Reward Instantiation

DocPO utilizes distinct, normalized reference-based rewards tailored to specific document elements, mapped to a $[0, 1]$ scale:

• Text: Normalized Edit Distance (NED).
• Tables: Weighted Tree Edit Distance Similarity (TEDS) using the APTED algorithm. This differentiates costs for structural operations (e.g., row/column spans) versus content-only mismatches, ensuring structural errors incur heavier penalties than typos.
• Formulas: A hybrid Rubric+edit reward. This combines:
  • A deterministic syntax gate ($v_{syn}$) that assigns zero reward to invalid LaTeX (syntax validity).
  • A binary rubric ($r_{sem}$) for semantic equivalence (handling valid alternative notations like \over vs. \frac).
  • A dense NED component ($r_{struct}$) for structural supervision.

2.2 Step-Aware Annealing (SAA)

SAA is a plug-and-play reward sharpening mechanism designed to address the flatness of rewards in the high-accuracy regime. It operates independently of the specific reward definition once normalized.

• Non-Linear Reward Shaping: SAA applies a power-law transformation to the base reward $M$:
  $$f_\gamma(M) = M^\gamma$$
  where $\gamma$ is a curvature factor. When $\gamma > 1$, the relative gap between high-scoring samples is amplified. This effectively lowers the "temperature" of the selection process, making the optimization signal sharper for top-ranked candidates while preserving the original ranking.
• Adaptive Scheduling: The curvature factor $\gamma$ is not fixed but evolves during training:
  $$\gamma = \gamma_{init} + \Delta\gamma \cdot \left[1 - \exp\left(-\frac{s}{\tau_{adaptive}}\right)\right]$$
  • Early Training: $\gamma$ starts near 1 (linear), ensuring stability and alignment with the original base reward.
  • Late Training: $\gamma$ increases (up to a max of $\gamma_{init} + \Delta\gamma$), progressively sharpening the reward to distinguish subtle differences among high-quality outputs.
• Dynamic Dispersion Controller (DDC): To adapt the sharpening rate to task difficulty, $\tau_{adaptive}$ is modulated by a task-wise dispersion score ($d_{DDC}$) calculated over a rolling window of recent rewards. High dispersion (heterogeneous outcomes) triggers faster sharpening, while low dispersion maintains a steadier pace.

3. Key Contributions

1. Step-Aware Annealing (SAA): A novel reward sharpening mechanism that dynamically increases reward curvature over training steps. It improves optimization in the high-accuracy regime without requiring additional human supervision or auxiliary reward model training.
2. DocPO Framework: A unified document RL framework that integrates element-specific reference-based rewards (NED for text, TEDS for tables, Rubric+edit for formulas) with SAA.
3. Fine-Grained Benchmark: The construction of DocElemHard, a challenging dataset covering text blocks, tables, and formulas for element-level evaluation, alongside evaluations on OmniDocBench.

4. Experimental Results

Experiments were conducted using the Qwen2.5-VL-3B backbone without any modifications to the visual encoder, pre-training, or post-processing modules.

• Performance Gains:
  • OmniDocBench: DocPO achieved an overall score of 95.38, surpassing specialized VLMs like PaddleOCR-VL (94.87) and DeepSeekOCR (90.63).
  • DocElemHard: DocPO achieved the best overall performance (93.76), outperforming the strongest specialized baseline (PaddleOCR-VL at 91.47).
  • Element-Specific:
    • Text: Reduced NED error to 0.0125 (vs. 0.0358 for SFT baseline).
    • Formulas: Improved CDM score to 94.70 (vs. 92.61 for SFT).
    • Tables: Achieved 93.10 TEDS (vs. 89.30 for SFT).
• Ablation Studies:
  • SAA vs. Fixed Curvature: Dynamic SAA (93.10) outperformed static power transforms with fixed $\gamma$ (ranging 91.9–92.3), demonstrating the necessity of the adaptive schedule.
  • DDC Contribution: Removing the Dynamic Dispersion Controller dropped performance from 93.10 to 92.73, confirming the value of task-specific dispersion adaptation.
  • Training Efficiency: SAA accelerated convergence, reaching the 91.0 TEDS threshold 1.8× faster than the non-annealed baseline, while also achieving a higher final performance ceiling (+1.5 points).
• Comparison with Specialized Models: DocPO outperformed models that rely on architectural customization (e.g., dedicated high-resolution encoders, custom vision token compression, or additional multimodal pretraining) using only a standard 3B backbone and reward-level optimization.

5. Significance and Claims

The paper claims that for document parsing, optimizing the reward signal is a more effective lever than scaling or redesigning the model architecture.

• Simplicity and Deployability: Unlike recent trends that require complex visual front-end modifications or auxiliary reward models, DocPO achieves state-of-the-art results using a standard VLM backbone with no extra pre-processing or post-processing.
• Generalizability: The SAA mechanism is agnostic to the specific reward definition, offering a unified optimization strategy across diverse document elements (text, tables, formulas).
• High-Accuracy Optimization: The work demonstrates that the primary bottleneck in high-accuracy document parsing is not the model's capacity, but the discriminative power of the reward signal in the near-optimal regime. SAA effectively addresses this by progressively sharpening the learning signal.

Limitations: The authors note that RL training is computationally more expensive than SFT due to multiple rollouts and rubric scoring. Additionally, proxy rewards may miss subtle semantic nuances, and the current scope is limited to text, formulas, and tables, excluding charts and geometric diagrams.