🤖 machine learning

Generalizing Numerical Reasoning in Table Data through Operation Sketches and Self-Supervised Learning

The paper introduces TaNOS, a continual pre-training framework that enhances the robustness and transferability of numerical reasoning in expert-domain tables by combining header anonymization, operation sketches, and program-first self-supervised learning to overcome the limitations of supervised fine-tuning and domain shifts.

Original authors: Hanjun Cho, Gahyun Yoo, Hanseong Kim, Jay-Yoon Lee

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hanjun Cho, Gahyun Yoo, Hanseong Kim, Jay-Yoon Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Cheat Sheet" Student

Imagine you are training a student to solve math word problems based on financial reports (tables full of numbers).

If you teach this student using standard methods (Supervised Fine-Tuning), they might become very good at the specific practice tests you give them. However, they aren't actually learning how to do math. Instead, they are memorizing shortcuts.

The Shortcut: They notice that whenever the column header says "Revenue," the answer usually involves adding numbers. When it says "Profit," they subtract.
The Failure: If you give them a test where the headers are changed to "Sales" or "Earnings," or if the questions are phrased slightly differently, the student panics and fails. They relied on the words (the headers), not the logic (the math structure).

This is exactly what happens with current AI models on expert tables (like finance or biology). They are "brittle"—they break easily when the context changes.

The Solution: TaNOS (The "Architect" Approach)

The researchers created a new training framework called TaNOS. Instead of letting the AI memorize words, they forced it to learn the blueprint of the problem. They did this using three clever tricks:

1. The "Operation Sketch" (The Blueprint)

Imagine you are asking a chef to make a cake.

Old Way: You say, "Make a chocolate cake." The chef memorizes "Chocolate = Cocoa Powder." If you ask for a "Dark Chocolate Cake," the chef gets confused.
TaNOS Way: You give the chef a sketch: "Mix flour, sugar, and cocoa. Bake at 350°."
In the Paper: Before the AI tries to answer a question, the researchers give it a tiny, abstract hint (a sketch) that says what kind of math to do (e.g., "Subtract the first number from the last"). This forces the AI to focus on the logic of the calculation rather than guessing based on the words in the table headers.

2. Header Anonymization (The "Blindfold")

Imagine you are teaching someone to play chess, but you cover the names of the pieces (King, Queen, Pawn) and just call them "Piece A," "Piece B," and "Piece C."

Why do this? If the student learns that "Piece A" always moves two squares, they might fail if you switch to a game where "Piece A" moves differently. But if they learn the rules of movement regardless of the name, they can play any chess game.
In the Paper: The AI is trained on tables where the column headers (like "Revenue" or "Net Income") are replaced with random nonsense words (like "Burger" or "Zebra"). This forces the AI to ignore the specific vocabulary and focus entirely on the structure of the data.

3. Self-Supervised Learning (The "Infinite Practice Lab")

Usually, to teach an AI, humans have to write thousands of questions and answers. This is slow and expensive.

The TaNOS Trick: The researchers built a robot that writes its own practice problems.
1. The robot picks a random math operation (e.g., "Subtract 5 from 10").
2. It calculates the answer (5).
3. Then, it asks an AI to write a natural language question that matches that math.
Why it works: Because the robot started with the math, the answer is 100% guaranteed to be correct. The AI gets to practice on millions of these perfect examples without a human needing to write a single one.

The Results: Why It Matters

The researchers tested this on an 8-billion-parameter AI model (a smart but relatively small model compared to the massive ones used by big tech).

Less Data, Better Results: TaNOS achieved 80% accuracy using only 10% of the training data. Standard methods needed 100% of the data to get only 74% accuracy.
The "Shape-Shifter" Test: When they changed the domain (e.g., took a finance model and tested it on biology tables with different headers), standard models crashed (dropping performance by over 10%). TaNOS barely blinked, dropping less than 2%.
Beating the Giants: TaNOS outperformed massive, proprietary models (like GPT-5 and Gemini) that cost a fortune to run, even though TaNOS used a much smaller, open-source model.

The Takeaway

TaNOS is like teaching a student to understand the grammar of math instead of memorizing the vocabulary.

By hiding the specific words (Anonymization), giving them a logic blueprint (Sketches), and letting them practice on infinite self-generated problems (Self-Supervised Learning), the AI learns to be a flexible problem-solver. It can handle new industries, new table formats, and new questions without needing to be retrained from scratch.

In short: It stops the AI from being a parrot that repeats what it heard, and turns it into a mathematician that understands how numbers work.

1. Problem Statement

Numerical reasoning over expert-domain tables (e.g., finance, biology, engineering) remains a significant challenge for Large Language Models (LLMs). While models often achieve high accuracy within a specific training domain, they suffer from limited robustness to domain shifts.

The authors identify three primary failure modes in current supervised fine-tuning (SFT) approaches:

Reasoning Inefficiency: Models allocate excessive capacity to inferring trivial arithmetic operations based on surface-level patterns rather than learning complex contextual reasoning.
Data Scarcity for Logical Supervision: Generating high-quality, logically consistent training data for numerical reasoning is difficult; standard LLM-based generation often produces spurious or inconsistent programs.
Header Dependency: Models memorize brittle associations between specific table headers (e.g., "Revenue") and operations (e.g., "Subtract"). When headers change (domain shift), performance degrades significantly because the model relies on lexical cues rather than structural understanding.

2. Methodology: The TaNOS Framework

The authors propose TaNOS (Table Numerical Reasoning with Operation Sketches), a continual pre-training framework designed to decouple domain-specific lexical semantics from the underlying computational structure. It consists of three core components:

A. Operation Sketches

Concept: Minimal symbolic cues that encode the core computational structure of a query without revealing the final answer.
Form: Represented as a compact symbolic sequence of abstract concepts and operators (e.g., (Revenue - Revenue) / Revenue for "percentage change").
Role: These sketches act as a "scaffold," relieving the model from inferring basic arithmetic relations and allowing it to focus on higher-level contextual alignment and relational reasoning.
Availability: At inference, sketches are assumed to be provided by the user (as the query implies the operation) or extracted via rule-based templates.

B. Header Anonymization

Mechanism: An instance-level scheme where table headers are mapped to unique, arbitrary tokens (e.g., "Revenue" $\to$ "burger") via a bijective mapping specific to each training instance.
Goal: To break the correlation between specific header tokens and operations, forcing the model to learn structural reasoning patterns rather than memorizing header-operation pairs. This preserves the table's structural integrity while removing lexical bias.

C. Self-Supervised Learning (SSL) with "Program-First" Generation

Strategy: Unlike traditional pipelines that generate a question and then an answer, TaNOS adopts a reverse-generation workflow:
1. Program Construction: Randomly sample operation sequences and arguments from unlabeled tables to construct executable programs ( $G$ ).
2. Execution: Execute $G$ to obtain a guaranteed-correct answer ( $A$ ).
3. Template & Sketch Generation: Derive a template-based question ( $Q$ ) and an operation sketch ( $S$ ) from the program and table structure.
4. Paraphrasing: Use an LLM only to paraphrase the template question into natural language, ensuring semantic diversity while maintaining the correctness of the underlying logic.
Outcome: This creates a large-scale dataset (NumReason-500) with correctness-guaranteed program-question pairs, enabling scalable pre-training without manual annotation.

3. Key Contributions

Failure Mode Analysis: Identified and analyzed the specific reasons why SFT models fail under domain shifts (inefficiency, data scarcity, and header dependency).
TaNOS Framework: Introduced a unified framework integrating Operation Sketches, Header Anonymization, and Correctness-Guaranteed SSL.
Data Efficiency: Demonstrated that TaNOS can achieve high performance with significantly less labeled data compared to standard SFT.
Robust Generalization: Showed that the framework effectively bridges the gap between in-domain and cross-domain performance, reducing the "cross-domain gap" to negligible levels.

4. Experimental Results

The framework was evaluated on Financial TableQA benchmarks (FinQA, MultiHiertt, NumReason-500) and cross-domain shifts (Mechanical, Biology, Legal, Scientific).

In-Domain Performance:
- Applied to an 8B instruction-tuned model (LLaMA-3.1-8B), TaNOS achieved 85.38% execution accuracy with full supervision, outperforming the SFT baseline (73.97%) and proprietary models like GPT-5 (77.82%) and Gemini-2.5-Pro.
- With only 10% of labeled data, TaNOS achieved 80.13%, surpassing the full-data SFT baseline and proprietary models.
Cross-Domain Generalization (Domain Shift):
- SFT Baseline: Showed a performance drop of >10 percentage points (pp) when moving from training to unseen domains (e.g., Finance $\to$ Biology).
- TaNOS: Maintained stable performance with a cross-domain gap of <2pp.
- Ablation: Removing any component (sketches, anonymization, or SSL) degraded cross-domain performance, confirming their complementary roles.
Data Efficiency:
- In a "one-shot" setting (one example per operation pattern), TaNOS achieved 66.79% accuracy, vastly outperforming SFT (18.97%) and approaching the performance of SFT trained on 100% of the data.
Complex Structures:
- On MultiHiertt (hierarchical multi-table reasoning), TaNOS improved accuracy by 21pp over the SFT baseline, demonstrating scalability to complex table layouts.

5. Significance and Implications

Structural vs. Lexical Reasoning: TaNOS proves that explicitly separating structural reasoning signals (via sketches and anonymization) from lexical surface forms significantly improves model robustness.
Privacy and Cost: The framework enables high-performance numerical reasoning using compact, open-source models (3B–8B) with minimal labeled data. This is crucial for privacy-sensitive domains (like finance and healthcare) where data cannot be sent to proprietary APIs.
Future Direction: The work suggests a path toward integrating symbolic structures with neural models, moving beyond pure end-to-end learning toward "structurally grounded" reasoning.

Conclusion: TaNOS offers a practical solution to the brittleness of current LLMs in expert-domain numerical reasoning. By enforcing structural learning through operation sketches and header anonymization, it achieves superior generalization and data efficiency, making it a viable approach for real-world deployment in diverse, data-scarce, and privacy-constrained environments.