Technical Summary: TAROT

Problem Definition

Few-shot tabular learning addresses scenarios where acquiring sufficient labeled data is costly or difficult, such as in fraud detection or rare disease diagnosis. Existing approaches fall into two categories, both of which face significant limitations:

1. Traditional Methods: Approaches like SCARF, TabPFN, and STUNT rely on additional training on large-scale unlabeled or synthetic data. While effective, this incurs substantial computational overhead.
2. LLM-based Methods: Methods like In-context learning, TABLET, and FeatLLM leverage Large Language Models (LLMs) for reasoning. However, they are often constrained by context length limits, raise privacy concerns when feeding raw data to external models, and frequently overlook the semantic relationships between features.

The core problem identified is that both paradigms largely ignore the structural and semantic priors inherent in tabular data. In few-shot scenarios, learning feature interactions directly from sparse supervision leads to instability and susceptibility to spurious correlations. Furthermore, constructing semantic graphs in few-shot settings is challenging because: (1) feature relationships are not explicitly provided, and (2) existing graph structure learning methods require large amounts of data to infer these relationships accurately. Additionally, relying solely on LLMs to generate graph structures introduces "hallucination" noise, such as spurious edges or missing task-relevant connections.

Methodology: TAROT

The authors propose TAROT (Task-Adaptive Refinement of LLM-prior Graphs), a GNN-based framework designed to encode structural and semantic priors by constructing and refining a task-adaptive semantic graph. The framework consists of four main components:

1. Unified Semantic Tabular Node Encoder (USTNE)

To handle heterogeneous tabular data (numerical and categorical), TAROT employs a training-free encoder.

• Decomposition: Data is split into a text set (feature names and categorical values) and a numerical set.
• Encoding: A pre-trained text encoder (e.g., BERT) processes the text set. Numerical features are normalized.
• Node Representation: For categorical features, the node embedding is the concatenation of the feature name and value embeddings. For numerical features, the feature name embedding is element-wise multiplied by the normalized value. This results in a unified node representation matrix $H$.

2. Semantic Graph Structure Construction

To address the lack of explicit graph structures in few-shot settings, TAROT utilizes an LLM as an external knowledge source without accessing raw data samples (zero-shot).

• Prompting: The LLM is prompted with the task description and feature names.
• Code Generation: Instead of asking for a list of edges, the LLM is instructed to generate an executable code snippet that directly outputs the adjacency matrix $A$. This standardizes the output and avoids parsing errors.
• Output: This produces an initial "prior graph" ($A_{prior}$) based on the LLM's world knowledge.

3. Task-Adaptive Semantic Graph Refinement

To mitigate structural noise (hallucinations) introduced by the LLM, TAROT refines the prior graph using a small set of labeled samples.

• Edge Scoring: A linear model scores edge representations (derived from node embeddings) to determine their relevance to the task.
• Prior Learning Loss: A loss term ($L_{prior}$) is introduced to regularize the model toward the initial LLM-generated structure, preventing overfitting to the scarce labeled data.
• Pruning and Enhancement:
  • Pruning: Edges with semantic scores below a threshold $\tau$ are removed (Task-Adaptive Pruning).
  • Enhancement: Top-$k$ edges with the highest scores that were missing in the prior graph are added (Task-Adaptive Enhancement).
• Result: A refined adjacency matrix $A_{refine}$ that aligns the graph structure with the downstream objective.

4. Message Passing Mechanism

A Graph Neural Network (GNN) performs message passing over the refined graph $G = (A_{refine}, H)$. The model aggregates neighborhood information to capture task-related semantic dependencies, which are then passed through a linear layer for final prediction. The total loss function combines the task loss, the prior learning loss, and a sparsity regularizer.

Key Contributions

1. Novel Insight on Structural Priors: The paper proposes leveraging LLMs to induce a semantic graph based solely on task descriptions and feature names. This addresses the challenge of inferring feature relationships without large datasets, providing a strong structural prior for few-shot learning.
2. Task-Adaptive Refinement Mechanism: A novel mechanism is introduced to denoise the LLM-generated graph. By pruning task-unrelated edges and adding missing task-related ones, the method enables effective GNN-based feature interaction modeling specifically for few-shot scenarios.
3. State-of-the-Art Performance: Extensive experiments on 11 real-world datasets (covering both classification and regression) demonstrate that TAROT consistently outperforms state-of-the-art traditional and LLM-based baselines.

Experimental Results

• Classification: On eight classification datasets (e.g., Adult, Heart, Diabetes), TAROT achieved superior AUC scores compared to baselines like FeatLLM, TabPFN, and SCARF across 4-shot, 8-shot, and 16-shot settings.
• Regression: On three regression datasets (Abalone, Boston, Cholesterol), TAROT achieved state-of-the-art RMSE performance, demonstrating the applicability of the semantic graph approach to regression tasks.
• Efficiency: TAROT is computationally efficient. It requires no additional pre-training on unlabeled data (unlike traditional methods) and avoids per-instance LLM queries during inference (unlike other LLM-based methods), as the graph structure is generated once via a single API call.
• Ablation Studies:
  • Removing the Semantic Graph Structure Construction (SGSC) caused performance to drop to the level of random graph structures, confirming the value of the LLM prior.
  • Removing the prior learning loss ($L_{prior}$) led to significant performance drops, validating the necessity of constraining the refinement process with the LLM's structural knowledge.
  • Qualitative analysis on the Adult dataset showed that TAROT successfully removed spurious edges (e.g., between "Sex" and "Race" for income prediction) while retaining meaningful connections, whereas baseline graph learning methods produced noisy, near-random structures in few-shot settings.

Significance

The paper claims that TAROT establishes a new state-of-the-art in few-shot tabular learning by explicitly modeling feature interactions through a task-adaptive semantic graph. Its significance lies in bridging the gap between the reasoning capabilities of LLMs and the structural efficiency of GNNs. By using LLMs to provide structural priors and a refinement mechanism to correct LLM hallucinations, TAROT offers a robust, privacy-preserving, and computationally efficient solution for real-world applications where labeled data is scarce. The work highlights that incorporating semantic relationships between features is essential for stabilizing learning and improving predictive performance in low-data regimes.