InstructTable: Improving Table Structure Recognition Through Instructions

1. Problem Statement

Table Structure Recognition (TSR) is critical for downstream tasks like data extraction and knowledge graph construction. However, existing methods face two primary challenges:

• Semantic-Visual Imbalance:
  • Visual-Centric Models: Rely solely on visual features, often failing to understand complex semantic relationships (e.g., merged cells, empty cells) leading to structural misjudgments.
  • Vision-Language Models (VLMs): Leverage semantic context but often underemphasize precise visual structural details (e.g., cell alignment), resulting in poor performance on complex layouts.
• Data Scarcity and Quality: High-quality tabular data is difficult to annotate. Existing synthetic data generation methods rely on rigid templates, which lack diversity and introduce systematic errors (e.g., "implicit row" problems where merged cells create visual contradictions with ground truth annotations).

2. Methodology

The authors propose InstructTable, an instruction-guided multi-stage training framework, and Table Mix Expand (TME), a novel data synthesis method.

A. Table Mix Expand (TME)

To address data scarcity and bias, TME generates large-scale, authentic tabular data without using fixed templates.

• Atomic Cell Matrix: Instead of HTML, TME parses real tables into an atomic cell matrix (representing independent, left-merged, up-merged, and bidirectionally merged cells). This representation is invariant to cropping and reduces token counts by >80% compared to HTML.
• Mixing and Expansion: Real table data is partitioned into blocks, randomly mixed, and spliced to create new structural configurations.
• Content Generation & Validation: Large Language Models (LLMs) like GPT-4o populate the generated structures with coherent content. A second LLM (Gemini 2.5 Pro) validates the semantic coherence and structural integrity, eliminating the "implicit row" errors found in template-based methods.
• Output: This process creates the BCDSTab benchmark (900 complex tables) and provides 50k+ synthetic samples for training augmentation.

B. InstructTable Framework

InstructTable is an end-to-end TSR model designed to balance visual and semantic information through a multi-stage training strategy.

• Architecture:
  • Image Encoder: Uses TableResNetExtra (CNN-based) to extract visual features.
  • Text Encoder: Uses BERT to encode human-designed instructions into feature vectors.
  • Decoder: A Transformer-based decoder that jointly processes image and instruction features via cross-attention. It outputs three components simultaneously: atomic cell matrices, cell bounding boxes, and empty cell classifications.
• Multi-Stage Training Strategy:
  1. Initialization: The model is trained with a fixed instruction ("Recognize all cells") to acquire basic TSR competence.
  2. Instruction Pre-training: The model is exposed to diverse, fine-grained instructions (e.g., "Recognize empty cells," "Cells in row X," "Merged cells"). This directs the model's attention to specific structural patterns, enhancing its ability to handle complex layouts.
  3. TSR Fine-tuning: Domain-adaptive fine-tuning on downstream benchmarks to align with standard visual-centric paradigms while retaining semantic understanding.
• Instruction Design: Four groups of instructions are used: (1) Global structure recognition, (2) Specific location recognition, (3) Empty cell recognition, and (4) Merged cell recognition.

3. Key Contributions

1. InstructTable Framework: A novel instruction-guided TSR approach that bridges the gap between visual-centric and vision-language models. It dynamically modulates attention based on instructions to improve fine-grained structural understanding.
2. Table Mix Expand (TME): A template-free data synthesis method that generates diverse, high-quality synthetic tables by mixing atomic cell matrices. It effectively eliminates structural errors inherent in template-based generation.
3. BCDSTab Benchmark: A new benchmark comprising 900 complex, dense table images synthesized via TME, designed to rigorously evaluate TSR models in challenging, real-world scenarios.

4. Experimental Results

The authors evaluated InstructTable on FinTabNet, PubTabNet, MUSTARD, and the proposed BCDSTab.

• State-of-the-Art Performance:
  • InstructTable achieved the best results on FinTabNet and PubTabNet, outperforming both traditional visual-centric models and general VLMs.
  • On MUSTARD (multilingual, diverse scenarios), InstructTable achieved a TEDS score of 95.30%, surpassing the previous SOTA visual-centric method (SPRINT) by 10.11% and the strongest VLM (Gemini 2.5 Pro) by 1.65%.
  • On the complex BCDSTab benchmark, after incorporating TME-generated data, InstructTable set a new SOTA with a 75.45% TEDS score, significantly outperforming competitors like MinerU 2.5.
• Ablation Studies:
  • TME Impact: Adding TME-generated data improved TEDS by ~1.2% and S-TEDS by ~0.7% across models.
  • Instruction Impact: Instruction pre-training significantly boosted structural parsing accuracy. Removing the "global structure" instruction group caused the largest performance drop (2.41% S-TEDS decrease), highlighting its importance.
  • Synergy: Combining instruction pre-training and TME data yielded the highest gains (+1.6% TEDS, +1.8% S-TEDS).

5. Significance

• Paradigm Shift: The paper demonstrates that integrating specific, fine-grained natural language instructions into TSR can effectively resolve the visual-semantic imbalance that has plagued previous methods.
• Data Solution: TME offers a scalable, low-cost solution to the data bottleneck in TSR, moving away from rigid templates toward dynamic, authentic data generation.
• Practical Application: The ability to handle complex merged cells, empty cells, and diverse layouts makes InstructTable highly suitable for real-world document processing in finance, academia, and enterprise environments.
• Open Resources: The authors released the code, the TME synthesis pipeline, and the BCDSTab benchmark, fostering further research in complex table recognition.