Technical Summary: Teaching LLMs to Reason over Tabular Data with Tri-Level Rationale Distillation (TLRD)

1. Problem Statement

Tabular data is the primary medium for real-world information storage, driving critical industrial applications in credit scoring, healthcare, and supply-chain optimization. While traditional tree-based ensembles (e.g., XGBoost, CatBoost) dominate these tasks due to their predictive performance and efficiency, they lack the ability to provide readable, case-specific explanations essential for stakeholder decision-making. Conversely, Large Language Models (LLMs) can naturally generate explanations but struggle with tabular data due to their inability to capture dataset-specific patterns, feature distributions, and interactions during pre-training.

A critical technical hurdle arises when attempting to fine-tune LLMs directly on labeled tabular datasets. Supervised solely by categorical labels, models often suffer from catastrophic forgetting of generative behaviors, leading to explanation collapse. In this state, the model optimizes strictly for discriminative mapping, producing terse, label-only outputs and failing to generate free-form, evidence-grounded rationales. Existing few-shot LLM approaches often underperform compared to strong baselines and do not adequately address the optimization of explanation quality in settings where large labeled datasets are available.

2. Methodology: Tri-Level Rationale Distillation (TLRD)

TLRD is a framework designed to convert label-only tabular datasets into structured, rationale-rich supervision for LLMs. It employs a teacher-student distillation pipeline to enable a compact student model to predict labels and generate grounded explanations using only raw features at inference time.

2.1 Tri-Level Augmented Input

To address the LLM's lack of dataset-specific context, the framework constructs a Tri-Level Augmented Input for the teacher model during corpus construction. This input extends the serialized raw row with two additional evidence sources:

1. Dataset-Level Statistics: Global distributional context, including mean, standard deviation, quartiles, and class-conditional frequencies for numerical and categorical features.
2. Retrieved Neighbors: A label-balanced set of similar historical instances retrieved based on categorical feature matches and numerical similarity (cosine similarity on standardized vectors).

2.2 Tri-Level Rationale Schema

Guided by a fixed schema, the teacher model analyzes the augmented input to generate structured rationales conditioned on the ground-truth label. The schema enforces three complementary levels of reasoning:

• Instance-Level Reasoning (Self Feature-based): Analyzes specific feature values using general domain logic and common sense to determine directional impact on the target.
• Dataset-Level Reasoning (Global Statistics-based): Contextualizes feature values against global statistics (e.g., comparing a value to class means or percentiles) to anchor the explanation in macro-level evidence.
• Comparison-Level Reasoning (Similar-cases): Uses retrieved neighbors to extract feature-combination patterns and critical deviations, summarized in <Patterns> and <Deviations> blocks, to validate predictions against local contrastive evidence.

2.3 Distillation and Deployment

The teacher generates a structured rationale report justifying the ground-truth label. A compact student model is then fine-tuned (using LoRA) on this distilled corpus. Crucially, during deployment, the student receives only the raw serialized features (no external statistics or retrieval). The model is trained to internalize the dataset-level and case-based knowledge, enabling zero-overhead inference while retaining the ability to generate the tri-level rationale.

3. Key Contributions

• Tri-Level Evidence to Rationales: The paper introduces a structured schema that guides a teacher model to synthesize rationales grounded in instance-level features, dataset-level statistics, and comparison-level neighbors. This formalizes how to transform raw tabular data into evidence-rich supervision.
• Context-to-Parameter Distillation: TLRD proposes a distillation method where dataset statistics and retrieved neighbors are used only during corpus construction. This indirectly injects dataset-specific knowledge into the student's parameters, allowing for efficient, standalone inference without external retrieval or statistical lookups.
• Comprehensive Empirical Evaluation: The framework is evaluated across six diverse datasets (classification and regression). Results demonstrate that TLRD significantly narrows the performance gap between LLMs and state-of-the-art tree ensembles while producing grounded, readable explanations.

4. Experimental Results

Experiments were conducted on datasets including ADULT, HOME CREDIT, OKCUPID, DIABETES130US, CALIFORNIA, and DIAMONDS, using various student backbones (Llama 3.1-8B, Qwen3-8B, Gemma 3-12B) and teacher models (GPT-OSS-20B/120B, Qwen3-Next-80B).

• Performance vs. Baselines: TLRD significantly outperforms zero-shot and few-shot LLM baselines. On classification tasks, TLRD variants achieve competitive performance against strong non-LLM baselines like XGBoost, CatBoost, and TabPFN. For instance, on the OKCUPID dataset, TLRD (Gemma 3-12B) achieved a Macro-F1 of 56.1, surpassing TabPFN v2.5 (55.9) and approaching CatBoost (58.9).
• Rationale Grounding: Qualitative analysis (Figure 3) shows that standard distillation often produces fluent but generic explanations based on commonsense assumptions. In contrast, TLRD generates explanations that cite specific dataset statistics (e.g., "77.8% of defaults vs 70.4% of non-defaults") and identify deviations from similar cases, offering more grounded decision support.
• Ablation Studies: Removing either dataset statistics or comparison-level retrieval degrades performance, confirming the complementary nature of the tri-level evidence. Furthermore, TLRD outperforms standard distillation and "Evidence-Augmented Distillation" (where the teacher sees evidence but lacks the structured schema), highlighting the importance of the schema in generating high-quality supervision.
• Scalability: The framework is effective even with smaller student models (e.g., 4B parameters) supervised by mid-sized teachers, suggesting a practical path for resource-constrained deployment.

5. Significance and Limitations

Significance:
TLRD offers a practical trade-off between the high predictive performance of tree-based ensembles and the interpretability of LLMs. By distilling dataset-specific reasoning patterns into a student model, it enables transparent, case-specific decision support without the latency of external retrieval or the explanation collapse associated with label-only fine-tuning. The authors posit this as a valuable reference for high-stakes decision-making where both accuracy and explainability are required.

Limitations:
The authors acknowledge several limitations:

• Spurious Correlations: The student model may inherit spurious correlations or logical leaps from the teacher, even when the teacher is conditioned on ground-truth labels.
• Scalability: The approach relies on feature selection for high-dimensional datasets and may struggle with datasets having a large number of classes due to input length constraints.
• Fine-Tuning Scope: Due to computational constraints, the study utilizes LoRA (parameter-efficient fine-tuning) rather than full supervised fine-tuning, leaving open the question of whether full-parameter adaptation would yield further gains.
• Inherent LLM Risks: The method inherits general LLM limitations, including potential hallucinations and sensitivity to prompt settings. The authors emphasize that TLRD should be deployed strictly as a decision-support tool, not an autonomous decision-maker, and recommend rigorous fairness audits and human evaluation for real-world deployment.