Language Model Representations for Efficient Few-Shot Tabular Classification

1. Problem Statement

The Web contains vast amounts of structured data in the form of tables (e.g., product catalogs, knowledge bases, scientific datasets). While Large Language Models (LLMs) have become integral to web infrastructure for semantic tasks, leveraging them for few-shot tabular classification remains challenging.

• The Gap: Traditional tabular models (e.g., XGBoost, TabPFN) are specialized but lack the semantic understanding of LLMs. Conversely, using LLMs directly via in-context learning (serializing entire tables into prompts) is computationally prohibitive due to quadratic attention complexity and high token costs.
• The Question: Can existing, off-the-shelf LLM embeddings of individual table rows be reused for structured data classification without specialized training or extensive retraining?
• The Challenge: Naive application of LLM embeddings for tabular tasks underperforms compared to specialized models. This is attributed to two main factors:
  1. Anisotropy: LLM embeddings are not uniformly distributed in vector space; they cluster in narrow cones, degrading distance-based metrics like cosine similarity.
  2. Task Variability: Different classification tasks require different levels of "confidence" or attention to context examples, which a static classifier cannot adapt to.

2. Methodology: TaRL (Table Representation with Language Model)

The authors propose TaRL, a lightweight, gradient-free framework that adapts off-the-shelf LLM embeddings for few-shot classification. The method consists of four key components:

A. Serialization and Embedding

• Row-wise Serialization: Instead of serializing the entire table context (which causes token explosion), each table row is serialized individually into a text string (e.g., Feature A: 1.2, Feature B: 3.4).
• Embedding: Each serialized row is passed through a pre-trained LLM to generate a vector embedding ($\mathbf{x} \in \mathbb{R}^{d_e}$). This avoids the quadratic scaling of transformer attention with respect to the total context size.

B. Kernel-Based Classification

The task is framed as a kernel regression problem. Given a support set of $k$ labeled examples and a query:

1. Compute embeddings for the query and all support examples.
2. Calculate similarity using negative inverse cosine similarity (bounded between $[-\pi, 0]$).
3. Apply a softmax function to convert similarities into weights ($w_i$).
4. Compute a weighted sum of the label embeddings to predict the query's class.

C. Geometric Correction: Common Component Removal (CCR)

To address the anisotropy of LLM embeddings:

• The method calculates the mean embedding vector (the "common component") across the support set and the query.
• This mean vector is subtracted from all embeddings.
• Result: This forces the embeddings to be more isotropic (uniformly distributed), improving the discriminative power of the cosine similarity metric.

D. Temperature Calibration ($\gamma$)

To address task variability, the authors introduce a tunable temperature parameter $\gamma$ in the softmax function:
$$w_i = \frac{\exp(\gamma \cdot \text{sim}(\mathbf{x}_q, \mathbf{x}_i))}{\sum \exp(\gamma \cdot \text{sim}(\mathbf{x}_q, \mathbf{x}_j))}$$

• High $\gamma$: Creates a peaked distribution (nearest-neighbor behavior).
• Low $\gamma$: Creates a uniform distribution (averaging over more examples).
• mTaRL (Meta-Learning): Instead of manually tuning $\gamma$, the authors train a simple meta-learner (regression model) on auxiliary datasets. This learner predicts the optimal $\gamma$ for a new task based solely on statistics of the support set (e.g., mean similarity, variance), requiring no gradient-based training on the target task.

3. Key Contributions

1. TaRL Framework: A novel, lightweight paradigm that reuses off-the-shelf LLM embeddings for few-shot tabular classification without any gradient-based training or specialized architecture.
2. Semantic Adaptations: Identification and implementation of two critical adaptations:
   • CCR: Mitigates embedding anisotropy.
   • Temperature Calibration: Dynamically adapts classifier behavior to specific task structures.
3. Meta-Learned Calibration (mTaRL): A practical method to approximate an "oracle" (perfect $\gamma$ selection) using a meta-learner trained on handcrafted features, enabling rapid adaptation (<1 hour) compared to days of training for foundation models.
4. Label Embeddings: Demonstrated that using LLM-generated embeddings for class labels (rather than one-hot vectors) significantly boosts performance by leveraging semantic label information.

4. Experimental Results

The method was evaluated on the CARTE-CLF (semantic-rich) and TabArena benchmarks across various shot settings ($k \in \{2, \dots, 512\}$).

• Performance vs. SOTA:
  • In low-data regimes ($k \le 32$) on semantic-rich datasets (CARTE), mTaRL achieves performance comparable to state-of-the-art specialized models like TabPFN and ConTextTab.
  • On non-semantic datasets (TabArena), specialized models still hold an advantage, confirming that TaRL's strength lies in leveraging semantic content (column names, text values).
• Oracle Bound: The "Oracle" classifier (which knows the perfect $\gamma$ for every task) achieves near-perfect scores, proving that the raw LLM embeddings contain sufficient signal if properly adapted.
• Runtime Efficiency:
  • TaRL is up to 1,000x faster than autoregressive LLM inference (e.g., TabuLa-8B) for few-shot tasks.
  • With a caching scheme (pre-computing embeddings), TaRL is significantly faster than XGBoost and other baselines for small $k$.
• Model Agnosticism: The approach works effectively with various LLM backbones (Llama 3.1, TabuLa-8B, Granite, Qwen), with Llama 3.1-8B showing particularly strong results.

5. Significance and Impact

• Infrastructure Reuse: The paper demonstrates that organizations with existing LLM deployments (for search or chat) can immediately apply these models to structured data classification tasks without deploying new specialized models or incurring massive retraining costs.
• Cost-Effectiveness: By avoiding autoregressive generation and heavy fine-tuning, TaRL offers a highly efficient pathway for web-scale table understanding.
• Semantic Understanding: It validates that the semantic understanding learned by LLMs during pre-training is transferable to tabular data, provided the geometric and task-specific adaptations are applied.
• Practicality: The method is particularly valuable for scenarios with limited data (few-shot) and rich semantic content, such as categorizing user forms, triaging support tickets, or classifying product metadata.

In conclusion, the paper argues that the future of efficient tabular classification may not lie in building larger, specialized foundation models, but in properly adapting the semantic representations already learned by general-purpose LLMs.