💻 computer science

TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition

The paper introduces TRivia, a self-supervised fine-tuning method leveraging Group Relative Policy Optimization and attention-guided question answering to enable vision-language models to learn table recognition directly from unlabeled data, resulting in the open-source TRivia-3B model that outperforms existing proprietary systems on key benchmarks.

Original authors: Junyuan Zhang, Bin Wang, Qintong Zhang, Fan Wu, Zichen Wen, Jialin Lu, Junjie Shan, Ziqi Zhao, Shuya Yang, Ziling Wang, Ziyang Miao, Huaping Zhong, Yuhang Zang, Xiaoyi Dong, Ka-Ho Chow, Conghui He

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junyuan Zhang, Bin Wang, Qintong Zhang, Fan Wu, Zichen Wen, Jialin Lu, Junjie Shan, Ziqi Zhao, Shuya Yang, Ziling Wang, Ziyang Miao, Huaping Zhong, Yuhang Zang, Xiaoyi Dong, Ka-Ho Chow, Conghui He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Table Translator" Problem

Imagine you have a massive library of documents, but the most important information is trapped inside tables. Some tables are neat grids; others are messy, with merged cells, missing borders, and handwritten notes.

Your goal is to turn these messy pictures of tables into clean, digital code (like HTML or Markdown) that computers can read and use. This is called Table Recognition (TR).

The Problem:
Until now, teaching computers to do this was like teaching a child to read by showing them flashcards with the answers written on the back. You needed humans to look at thousands of tables, draw lines, and type out the answers. This is expensive, slow, and hard to do for private documents (like bank records or medical files) where you can't just upload them to a cloud server.

Big tech companies (like Google or OpenAI) have solved this by using massive amounts of human labor and super-computers. But for everyone else, the best open-source models are still struggling.

The Solution: TRivia
The authors created a new method called TRivia. Instead of needing humans to write the answers, TRivia teaches the computer to learn from unlabeled tables (tables without answers) by playing a game of "Question and Answer."

The Analogy: The "Study Group" vs. The "Strict Teacher"

1. The Old Way: The Strict Teacher (Supervised Learning)

Imagine a student (the AI) trying to learn math.

The Old Way: A teacher gives the student a worksheet with 1,000 problems and the answer key attached. The student memorizes the answers.
The Flaw: If the teacher runs out of answer keys (which happens with real-world, messy tables), the student stops learning. Also, if the student needs to study a secret textbook they can't show the teacher, they can't use this method.

2. The New Way: The Study Group (TRivia)

TRivia is like a smart study group where the student learns from a pile of textbooks without answer keys.

Step 1: The "Guessing Game" (Response-Consistency Sampling)
The student looks at a messy table and tries to solve it. They do this multiple times, getting slightly different answers each time.

The Trick: If the student gets the exact same answer every time, the table was probably too easy. But if the student gets different answers, it means the table is tricky and interesting. TRivia picks these "tricky" tables to study because they offer the most learning value.

Step 2: The "Quiz Master" (Attention-Guided QA)
Now, how does the student know if their answer is right without an answer key?

TRivia acts as a Quiz Master. It looks at the table and asks specific questions like, "What is the total in the bottom right corner?" or "Who is listed in the second row?"
The Magic: The system uses the AI's own "attention" (like a spotlight) to make sure the questions cover different parts of the table, not just the same spot over and over.

Step 3: The "Self-Correction" (The Reward Loop)
The student tries to answer the Quiz Master's questions based on their own interpretation of the table.

If the student's table structure is wrong, they will get the quiz questions wrong.
If the student's table structure is right, they will get the quiz questions right.
The Reward: The system gives the student a "score" based on how well they answered the quiz. If they do well, they get a reward. If they do poorly, they get a penalty.
Over time, the student learns to build the perfect table structure just to get a high score on the quiz. They never saw the "correct" answer key; they just learned to answer the questions correctly.

Why is this a Big Deal?

It's "Self-Supervised": It doesn't need humans to label data. It can learn from any table image found on the internet or in private documents.
It's Better than the "Experts": The paper shows that their new model, TRivia-3B, is actually smarter at reading tables than the expensive, proprietary models from big tech (like Gemini 2.5 Pro) and other specialized open-source models.
It's Private: Because it doesn't need to send data to a cloud server to get labeled, you can run it on your own computer to process sensitive documents (like legal contracts or medical records) without worrying about privacy leaks.

The "Secret Sauce" Ingredients

The "Spotlight" (Attention): The system ensures the questions asked about the table aren't all about the same corner. It forces the AI to look at the whole picture.
The "Filter" (Illegal-Sample Filtering): Sometimes the AI gets confused and produces gibberish. The system is smart enough to throw away those bad attempts so they don't confuse the learning process.
The "Group Chat" (GRPO): Instead of just saying "Right" or "Wrong," the system compares the AI's different attempts against each other. It asks, "Which of these 5 guesses was the most consistent?" and rewards that one.

The Bottom Line

TRivia is like teaching a computer to read tables by giving it a pile of messy documents and asking it, "Can you tell me what's in this cell?" instead of saying, "Here is the correct answer."

By playing this game of self-generated quizzes, the AI learns to understand tables better than ever before, without needing expensive human teachers or risking privacy. It's a giant leap forward for making documents readable by machines.

1. Problem Statement

Table Recognition (TR) involves converting table images into semi-structured formats like HTML or Markdown. While essential for document parsing, TR faces significant challenges:

Data Scarcity: High-performance TR models traditionally rely on large-scale, manually labeled datasets, which are expensive and time-consuming to produce.
Limitations of Existing Approaches:
- Synthetic Data: Lacks the visual diversity and complexity of real-world documents.
- Distillation from Proprietary Models: While effective, it is costly, caps performance at the level of the "teacher" model, and raises privacy/compliance issues.
- Open-Source Models: Often lag behind proprietary systems (e.g., Gemini 2.5 Pro) due to limited training data.
Core Question: Can unlabeled table images from the wild be leveraged to improve TR performance without human annotation?

2. Methodology: The TRivia Framework

The authors propose TRivia, a self-supervised fine-tuning framework that enables Vision-Language Models (VLMs) to learn TR directly from unlabeled data. The framework operates in two main stages: an Adaptive Data Engine (Preparation) and Self-supervised Fine-tuning (Training).

A. Core Mechanism: Table QA as Proxy Rewards

Instead of predicting full HTML structures directly (which is hard to verify without labels), TRivia uses Table Question Answering (QA) as a proxy task.

Concept: If a model correctly recognizes a table's structure and content, it should be able to answer questions about that table.
Reward Signal: The model generates a table structure. An external Large Language Model (LLM) attempts to answer specific questions based on this structure. The F1-score between the LLM's answer and the ground truth (or a verified answer) serves as the reward.
Advantage: Verifying a specific answer is easier and more robust than verifying complex structural tags (like colspan/rowspan) in raw HTML.

B. Stage 1: Adaptive Data Engine

To ensure the unlabeled data is useful and the supervision signals are high-quality, TRivia employs three key modules:

Response-Consistency Sampling (Sample Selection):
- Not all unlabeled tables are equally valuable. TRivia samples multiple recognition outputs ( $K$ responses) for each image using the current model.
- It calculates the Tree Edit Distance-based Similarity (TEDS) between pairs of responses.
- Strategy: Images with low consistency (high diversity in outputs) indicate high model uncertainty and are selected for training. This aligns with the Group Relative Policy Optimization (GRPO) principle of optimizing relative differences.
Attention-Guided Diverse QA Generation:
- To avoid generating redundant or overlapping questions, TRivia uses the attention mechanism of the VLM.
- It identifies the "visual source" (visual tokens) for each generated QA pair.
- Filtering: It greedily selects QA pairs that have minimal overlap in their visual sources (measured by Intersection-over-Union, IoU), ensuring the questions cover diverse regions of the table.
Validity Cross-Check:
- An external VLM verifies that a question is answerable only when the image is provided (visual dependency) and that the answer is correct. This filters out hallucinated or ungrounded QA pairs.
Illegal-Sample Filtering:
- During training, if the model generates an invalid table structure (e.g., broken HTML), the LLM cannot answer the questions, resulting in a zero reward. To prevent reward noise and training instability, these invalid samples are filtered out before the advantage calculation in GRPO.

C. Stage 2: Self-supervised Fine-tuning with GRPO

The framework uses Group Relative Policy Optimization (GRPO) to fine-tune the VLM.

Process: For a selected image, the model generates a group of $R$ recognition outputs.
Reward Calculation: Each output is paired with the generated QA set. An LLM answers the questions based on each output. The reward is the average F1-score across all QA pairs for that output.
Optimization: The model is updated to maximize the relative advantage of high-reward responses over low-reward responses within the group, without needing explicit ground-truth labels.

3. Key Contributions

Self-Supervised Framework (TRivia): A novel method enabling VLMs to learn TR directly from unlabeled wild data, bypassing the need for costly human annotations.
Response-Consistency Sampling: A strategy to automatically identify the most informative (uncertain) unlabeled samples for training, optimizing the data curation process.
Attention-Guided QA Mechanism: A method to generate diverse, verifiable, and non-redundant supervisory signals by leveraging visual attention maps, ensuring robust reward estimation.
TRivia-3B Model: An open-source, compact (3B parameter) model that achieves state-of-the-art performance, surpassing both specialized expert models and massive proprietary systems.

4. Experimental Results

The authors evaluated TRivia-3B (fine-tuned from Qwen2.5-VL-3B) against 12 baselines, including expert TR models, general-purpose VLMs, and document-parsing VLMs.

Benchmarks: OmniDocBench, CC-OCR, and OCRBench v2.
Performance:
- TRivia-3B achieved an overall TEDS score of 89.88, surpassing Gemini 2.5 Pro (88.93) and MinerU2.5 (86.82).
- It significantly outperformed specialized open-source models like UniTable (70.86) and PaddleOCR-VL (83.36).
- It showed particular robustness on complex, real-world tables (rotated, borderless, merged cells), where it improved performance by over 20 TEDS points on rotated tables compared to the base model.
Ablation Studies:
- Removing attention-guided QA generation caused a significant performance drop.
- Illegal-sample filtering was crucial for stabilizing training.
- Using direct distillation from a weaker teacher (Qwen2.5-VL-72B) failed, whereas the TRivia QA-reward approach succeeded, proving the value of the proxy reward mechanism.
Data Annotation Capability: TRivia-3B was used to generate pseudo-labels for a new dataset. A model distilled from these labels achieved performance nearly identical to TRivia-3B, demonstrating its potential as a scalable, automated annotation engine.

5. Significance

Breaking the Data Bottleneck: TRivia demonstrates that high-performance TR is achievable without massive labeled datasets, leveraging the vast availability of unlabeled documents.
Privacy and Accessibility: By providing an open-source model that rivals proprietary giants (like Gemini 2.5 Pro), TRivia enables organizations to process sensitive documents offline without relying on commercial APIs.
Scalability: The framework offers a path to continuously improve document parsing models by ingesting new, unlabeled data from the wild, making it a scalable solution for the future of document AI.
Generalizability: The authors suggest the QA-based reward mechanism could be extended to other OCR and document understanding tasks (e.g., Key Information Extraction), establishing a new paradigm for self-supervised multimodal learning.

Code and Model Availability: The model and code are open-sourced at https://github.com/HKU-TASR/TRivia.