💬 NLP

Gold-Guided Programmatic Distillation for Financial Reasoning over Hybrid Tables and Text

This paper proposes a gold-guided programmatic distillation framework that transfers reliable numerical reasoning from large language models to compact student models using execution-verified Python programs instead of error-prone natural language rationales, achieving state-of-the-art performance on the TAT-QA financial reasoning benchmark.

Original authors: Yun Dong, Erica Zhao, Elana Chen

Published 2026-07-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yun Dong, Erica Zhao, Elana Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, tricky puzzle where half the clues are written in a story and the other half are hidden inside a messy spreadsheet. This is the daily life of a computer trying to answer financial questions. For a long time, scientists have been teaching computers to "think" by asking them to write out their steps in plain English, like a student showing their work on a math test. This is called "Chain-of-Thought." But here's the catch: computers are terrible at math when they try to do it in their heads while writing sentences. They often get distracted, mix up numbers, or just make up answers that sound good but are wrong. It's like asking a brilliant storyteller to also be a human calculator; they usually fail at the math part.

To fix this, researchers started teaching computers to write code instead of sentences. Code is like a strict recipe that a computer can follow perfectly without getting confused. But there's a new problem: how do you teach a small, fast computer to write these perfect recipes if the only teacher you have is a giant, slow computer that sometimes still makes mistakes? If the teacher gives a bad recipe, the student learns the wrong way. This paper tackles that exact problem. It asks: Can we teach a small computer to be a math wizard by only letting it learn from the teacher's perfect recipes, and then helping it fix the ones the teacher got wrong?

The Golden Recipe Book

The researchers, a team from Georgia Tech and Stanford, came up with a clever two-step plan called "Gold-Guided Programmatic Distillation." Think of it as a master chef (the "Teacher," a huge 72-billion-parameter model) trying to teach a young apprentice (the "Student," a smaller 7-billion-parameter model) how to cook a complex financial dish.

Step 1: The Strict Filter
Usually, when a teacher shows a student how to do something, they just say, "Here's how I did it." But in this world, the teacher is allowed to make mistakes. So, the researchers added a magical "Gold Guide." Before the teacher writes a recipe (a Python program), they are secretly shown the correct answer key. The teacher then tries to write a recipe that leads exactly to that answer.

Here's the magic trick: The team didn't just take the teacher's word for it. They ran every single recipe the teacher wrote through a strict "taste test" (a computer program that executes the code). If the recipe had a typo, crashed, or didn't result in the exact right number, it was thrown in the trash. Only the recipes that worked perfectly and matched the gold answer were saved in a special "Golden Recipe Book." The small student was then trained only on these perfect recipes. This ensured the student never learned a bad habit.

Step 2: The Comeback Tour
But what about the hard questions where even the giant teacher couldn't write a working recipe? The teacher would just give up, leaving those questions unsolved. The researchers didn't want to leave those behind. They introduced a "Recovery Stage."

They took the questions the teacher failed and asked the student to try again. The student would generate five different possible recipes for the same question. Just like before, they ran all five through the taste test. If any of the student's recipes worked perfectly, they saved it. This meant the student could "teach itself" how to solve the problems the teacher couldn't, effectively learning from its own successes. They then trained the student again on this new batch of self-discovered, perfect recipes.

The Results: Small but Mighty

The team tested this on a famous financial puzzle set called TAT-QA, which mixes tables and text. They compared their small, trained student against the giant teacher, older computer models, and other smart AI systems.

The results were surprising. The small student, after this special training, scored 87.00 on a measure called "Exact Match" (which means getting the answer exactly right) and 87.18 on a measure called "F1" (which looks at how close the answer is).

To put that in perspective:

The giant 72-billion-parameter teacher, even with the "Gold Guide" hints, only scored 78.46.
The previous best AI system (TAT-LLM) scored 82.67.
The untrained small student started at a low 46.33.

The small student didn't just catch up; it actually beat the giant teacher and the best previous systems. The researchers found that the student got especially good at the hardest parts: doing math with numbers found in both tables and stories.

Why This Matters

The paper suggests that this method is a powerful way to make smaller, faster computers reliable at math without needing the massive computing power of the giant models. By filtering out every single mistake and letting the student learn from its own recovered successes, they created a system that is both smart and precise.

However, the authors are careful to note that the problem isn't 100% solved. Even with the best student, there are still some errors—sometimes the answer is right but the unit (like "million" vs "billion") is wrong, or the answer is just plain wrong. But compared to before, the number of mistakes dropped dramatically, from hundreds of errors down to just over a hundred.

The researchers conclude that this "Gold-Guided" approach is a promising path forward. They suggest that in the future, we might build systems that can decide on their own whether to use a simple lookup or a complex code recipe, making financial AI even more helpful and trustworthy. For now, though, they've proven that a small, well-trained robot can outsmart a giant, unrefined one when it comes to crunching the numbers.

Technical Summary: Gold-Guided Programmatic Distillation for Financial Reasoning

Problem Statement

Financial question answering (QA) over hybrid tabular and textual data presents a dual challenge: it requires integrating information from multiple sources and performing precise numerical computation. While Large Language Models (LLMs) can generate intermediate reasoning steps via Chain-of-Thought (CoT) prompting, their natural-language rationales are prone to arithmetic hallucinations. This unreliability makes free-form text an ineffective supervision source for distilling reasoning capabilities into smaller models. Although Program-of-Thought (PoT) approaches mitigate arithmetic errors by delegating computation to external interpreters, transferring these reliable reasoning traces from large teacher models to compact student models remains difficult, particularly when intermediate traces are imperfect or when the teacher fails to solve complex examples.

Methodology

The authors propose Gold-Guided Programmatic Distillation, a framework designed to transfer reliable numerical reasoning from a large teacher model to a compact student using execution-verified Python programs. The framework operates in two primary phases:

1. Teacher-Driven Distillation (Phase 1)

Gold-Guided Synthesis: A large teacher model (Qwen2.5-72B-Instruct) synthesizes executable Python programs to solve financial QA tasks. Crucially, the teacher is guided by gold derivations (ground-truth reasoning steps) injected into the prompt to ensure accurate multi-step calculations.
Execution Verification: Generated programs are executed in a sandboxed Python interpreter. Only programs that compile without error and produce the exact gold numerical answer (including the correct scale) are retained.
Student Training: A compact student model (Qwen2.5-7B-Instruct) is trained via Supervised Fine-Tuning (SFT) using Low-Rank Adaptation (LoRA) on this filtered, high-quality dataset. The student learns to generate programs conditioned solely on the context and query, without access to the gold derivation.

2. Iterative Recovery and Self-Improvement (Phase 2)

Revisiting Failures: Examples that the teacher failed to solve are placed in a failure set ( $D_{wrong}$ ).
Student Sampling: The Phase-1 student model samples multiple candidate programs for these difficult examples at a higher decoding temperature.
Recovery Filtering: Candidates are filtered through the same execution-based verification process. Programs that execute correctly and yield the gold answer are added to a recovery set ( $D_{rec}$ ).
Refinement: The student undergoes a second round of LoRA fine-tuning on the augmented dataset ( $D_{Total} = D_T \cup D_{rec}$ ), allowing it to learn from execution-verified self-generated traces that extend beyond the teacher's original coverage.

Inference

During inference, the trained student generates an executable program for a given query. A Python interpreter executes this code to produce the final numerical answer, ensuring that all arithmetic is performed deterministically rather than by the model's internal logic.

Key Contributions

Gold-Guided Programmatic Distillation: A novel framework that leverages ground-truth derivations to guide teacher-side program synthesis, ensuring that only high-quality, execution-verified programs are used for distillation.
Iterative Recovery Mechanism: An extension of rejection sampling fine-tuning (RFT) and STaR that enables the student model to recover and incorporate solutions for examples the teacher failed to solve, thereby expanding the supervision signal.
Execution-Verified Supervision: The approach replaces unreliable natural-language rationales with executable code as the supervision signal, directly addressing the arithmetic error problem in financial reasoning.

Experimental Results

The framework was evaluated on the TAT-QA dataset, a benchmark for hybrid tabular and textual financial reasoning.

Performance: The best-performing student model (Qwen2.5-7B with Attention+MLP LoRA adaptation) achieved 87.00 Exact Match (EM) and 87.18 F1 on the test set.
Comparison: This result substantially outperforms:
- The 72B teacher model with derivation hints (78.46 EM).
- The strongest prior LLM-based baseline, TAT-LLM (82.67 EM).
- Task-specific baselines like TAGOP (47.05 EM).
- The untuned 7B student (46.33 EM).
Impact of Components:
- Teacher Quality: Providing gold derivations to the teacher increased its own performance from 69.55 to 78.46 EM, directly improving the quality of the distilled dataset.
- LoRA Configuration: Adapting both attention and MLP projection layers consistently outperformed attention-only adaptation.
- Phase 2 Recovery: The iterative recovery stage improved the test EM of the Attention+MLP model from 84.72 (Round 1) to 87.00 (Round 2), demonstrating the value of recovering teacher-failed examples.
Error Analysis: The framework significantly reduced errors in answer prediction, scale prediction, and joint failures compared to the baseline, with the most significant gains observed in tasks requiring reasoning over mixed tabular and textual evidence.

Significance and Claims

The paper claims that execution-verified programmatic distillation provides an effective and extensible framework for training smaller models to perform reliable numerical reasoning. By shifting the burden of arithmetic to deterministic code execution and using gold derivations to filter supervision data, the authors demonstrate that compact models can surpass much larger teacher models in complex financial QA tasks.

The authors modestly note that while the approach yields strong improvements, it does not fully resolve all error types (answer prediction, scale, and joint failures), and the evaluation is limited to a filtered subset of TAT-QA suitable for program generation. Future work is suggested to explore task-aware routing mechanisms to distinguish between queries requiring programmatic reasoning versus simple extraction, and to evaluate the framework on broader financial benchmarks like FinQA and ConvFinQA.