Technical Summary: Multimodal Approaches for Visually-Rich Document Type Classification

Problem Statement

Document type classification for Visually Rich Documents (VRDs) remains a significant challenge because relevant information is distributed across textual, visual, and layout modalities. While early deep learning approaches processed these modalities separately (e.g., using CNNs for images and RNNs for OCR text), modern architectures have shifted toward unified multimodal modeling. However, the field suffers from a lack of systematic comparison due to heterogeneous model architectures (ranging from specialized Transformers to general-purpose Large Language Models) and inconsistent evaluation setups. Existing comparative studies often rely on aggregate metrics without fine-grained empirical analysis, making it difficult to assess whether performance gains reflect genuine methodological advances or simply variations in experimental design. Furthermore, the distinction between OCR-dependent and OCR-free approaches has not been rigorously evaluated within a unified framework.

Methodology

To address these gaps, the authors conducted a controlled empirical comparison of four representative models within a harmonized experimental pipeline on the RVL-CDIP benchmark (a dataset of 400,000 document images across 16 categories). The study categorizes models along two dimensions: backbone type (Transformer vs. LLM) and feature integration strategy (OCR-dependent vs. OCR-free).

The four evaluated architectures are:

1. LayoutLMv3 (OCR-Dependent Transformer): A specialized multimodal encoder integrating OCR-derived text, 2D layout coordinates, and image patches. The study utilized a publicly available fine-tuned checkpoint.
2. Donut (OCR-Free Transformer): An end-to-end image-to-text framework using a visual encoder (Swin Transformer) and an autoregressive decoder (BART), operating directly on document images without external OCR.
3. Qwen3-VL-32B-Instruct (OCR-Free LLM): A general-purpose vision-language model using a SigLIP-2 vision encoder and a dense language model decoder, evaluated via prompting without fine-tuning.
4. Qwen3-32B (OCR-Dependent LLM): A text-only LLM backbone where document classification relies entirely on text extracted via Tesseract OCR, processed via prompting.

Experimental Setup:

• Dataset: RVL-CDIP test split (40,000 images).
• Metrics: Accuracy, per-class precision/recall/F1, confusion matrices, and end-to-end inference time (including OCR processing for dependent models).
• Hardware: 4x NVIDIA H200 GPUs.
• Constraints: All models were evaluated under identical inference configurations to ensure comparability. LLMs were used in zero-shot/few-shot modes via prompting, while Transformers relied on fine-tuned weights.

Key Results

1. Performance Comparison

• Specialized Transformers vs. LLMs: Specialized multimodal Transformers (Donut and LayoutLMv3) consistently outperformed general-purpose LLMs.
  • Donut (OCR-Free Transformer): Achieved the highest accuracy (0.95), matching the performance of the original LayoutLMv3 paper despite being OCR-free.
  • LayoutLMv3 (OCR-Dependent Transformer): The publicly available fine-tuned checkpoint achieved 0.90 accuracy, notably lower than the 0.9544 reported in the original paper, highlighting the sensitivity of Transformer performance to fine-tuning quality.
  • Qwen3-VL (OCR-Free LLM): Achieved 0.75 accuracy.
  • Qwen3-32B (OCR-Dependent LLM): Achieved the lowest accuracy (0.55), demonstrating that relying solely on OCR text without visual context severely limits performance on layout-intensive documents.

2. Modality Analysis

• Visual Dominance: Image information was identified as the primary driver for robust classification. OCR-free models (Donut, Qwen3-VL) outperformed their OCR-dependent counterparts (Qwen3-32B), confirming that direct visual processing preserves critical layout and structural cues lost during OCR extraction.
• Role of Text: OCR-derived text provided useful but secondary support. It was most effective for documents with linear, text-dominant structures (e.g., emails, resumes with keywords) but insufficient for layout-heavy categories (e.g., forms, handwritten notes).
• Layout Sensitivity: Models struggled with categories exhibiting heterogeneous or weakly standardized layouts (e.g., "presentation," "form"). The "form" category, in particular, showed low performance across all models due to its visual overlap with other semi-structured documents.

3. Efficiency and Usability

• Inference Time: OCR-free models were faster. Qwen3-VL was the fastest (239 ms/image), followed by Donut (307 ms). OCR-dependent models were significantly slower due to the OCR stage (LayoutLMv3: 554 ms; Qwen3-32B: 600 ms).
• Deployment: Transformer models require domain-specific fine-tuning, which is computationally demanding and requires expertise. LLMs offer easier deployment via prompting but suffer from output instability (e.g., generating labels outside the predefined set) and higher hardware requirements.

Key Contributions

1. Structured Review: The paper summarizes the design space of multimodal VRDU architectures, explicitly contrasting OCR-dependent and OCR-free strategies within Transformer and LLM paradigms.
2. Controlled Empirical Comparison: It provides a fair, unified evaluation of four representative models on the RVL-CDIP benchmark, addressing the lack of consistent evaluation setups in existing literature.
3. Empirical Characterization: The study offers a data-driven analysis of the trade-offs between model families, demonstrating that visual features are essential for layout-intensive documents and that OCR-free approaches can match or exceed OCR-dependent specialized models in accuracy while reducing latency.

Significance and Claims

The paper claims to provide a systematic basis for comparing multimodal architectures and offers practical guidance for selecting feature combinations and model designs. Its primary findings are:

• Multimodal processing is essential for documents with pronounced layout structures; text-only approaches fail to capture necessary spatial semantics.
• Specialized Transformers remain superior to general-purpose LLMs for document type classification, provided they are well-tuned.
• OCR-free approaches (like Donut) are highly effective, offering a balance of high accuracy and inference efficiency by avoiding the overhead and error propagation of external OCR.
• Dataset limitations (such as label noise and intra-class variability in RVL-CDIP) significantly impact model performance, suggesting that future progress depends on better-curated benchmarks with clearly separated document classes.

The authors conclude that while LLMs offer flexibility, they currently lack the robustness and predictability required for strict, closed-set document classification tasks compared to specialized, fine-tuned multimodal Transformers.