Technical Summary: LLMs on Tabular Data with Limited Semantics

Problem Formulation

This study addresses an industrial planning challenge within automotive development: predicting the retrofit requirements for newly registered prototype vehicles. Planners must answer three specific questions for each vehicle:

1. Occurrence: Will the vehicle require a retrofit visit? (Binary classification)
2. Type: If a visit occurs, which specific retrofit package is needed? (15-way multiclass classification)
3. Duration: How long will the retrofit process take? (Regression)

The core technical challenge lies in the nature of the data. The dataset links a prototype-registration system (284,271 vehicles) with a retrofit-management system (48,716 cleaned visits). Crucially, the categorical metadata (e.g., derivative, integration step, engine type) is hashed, meaning the data lacks inherent lexical semantics. When rows are serialized into text for Large Language Model (LLM) processing, the input exposes structural patterns but very little semantic content. This setting serves as a controlled environment to distinguish what LLMs can learn from distributional patterns alone versus what requires semantic understanding.

Methodology

The authors evaluate three distinct LLM-based strategies against strong classical tabular baselines and AutoML references across the three prediction stages.

Classical Baselines

The study employs standard feature engineering (target encoding, frequency encoding, interaction terms) and trains a suite of classical models, including Logistic Regression, Random Forest, Extra Trees, Gradient Boosting, XGBoost, LightGBM, and CatBoost. Stacking and weighted soft-voting ensembles are also utilized. For the multiclass task, SMOTE is used to handle class imbalance.

LLM-Based Strategies

Three specific approaches are tested on row-serialized inputs:

1. LLM-EMBED: Structured rows are serialized into key-value text strings and embedded using Amazon Titan Embed v2 (1,024 dimensions). A logistic regression classifier is then trained on these vectors. Due to API costs, this is subsampled to 5,000 training rows.
2. LLM-PROMPTED: Direct classification using Claude Sonnet 4. The model receives serialized rows with few-shot examples and predicts the label directly without fine-tuning. Due to latency and cost, this is evaluated on small held-out subsets (200–500 rows).
3. LLM-STACKED: A hybrid approach where a meta-learner combines the probability outputs of the best classical models with the outputs derived from the LLM strategies. This tests whether LLMs function better as feature providers or complementary components rather than end-to-end predictors.

Time-Series Benchmark

In addition to per-visit predictions, the authors construct an aggregated monthly time-series benchmark (76 points) of mean retrofit duration. This is used to compare classical statistical forecasting and lag-based machine learning against time-series foundation models (Chronos, TIME-LLM) in a low-data, zero-shot setting.

Key Results

Stage 1: Binary Occurrence Prediction

• Classical Dominance: Hyperparameter-tuned CatBoost achieved the strongest standalone performance (ROC-AUC = 0.997, F1 = 0.884).
• LLM Embeddings: Embedding-based features remained useful, achieving an AUC of 0.982, though they lagged behind the best tree ensembles.
• LLM Prompting Failure: Direct prompting collapsed to random performance (AUC = 0.500) once the semantic signal was stripped by hashing.
• Hybrid Success: The LLM-STACKED approach achieved the highest F1 score (0.900) on its evaluation subset, suggesting that LLMs provide complementary signal when combined with strong classical models.

Stage 2: 15-Way Retrofit-Type Classification

• Hybrid Superiority: The LLM-STACKED model achieved the best weighted F1 (0.626) among manually configured pipelines, outperforming Random Forest (0.621) and XGBoost (0.614).
• AutoML: AutoGluon remained the overall best performer (0.654 weighted F1).
• Prompting Failure: Consistent with Stage 1, direct prompting performed poorly on the hashed data.

Stage 3: Duration Regression & Time-Series

• Regression: Classical models remained the standard for per-visit duration prediction.
• Time-Series: Lag-based machine learning outperformed time-series foundation models on the aggregated monthly benchmark. However, Chronos-small remained competitive in zero-shot forecasting despite the small dataset size.

Significance and Claims

The paper concludes that for privacy-constrained industrial tables where categorical values are hashed and semantic content is minimal:

1. LLMs are not replacements: Strong classical tabular baselines (specifically tree ensembles) remain the most effective standalone models for structured data prediction.
2. Direct prompting is ineffective: Without rich semantic content, direct LLM prompting fails to learn distributional patterns, collapsing to random chance.
3. Hybridization is valuable: LLMs serve as effective complementary components. Embedding features retain useful signal, and hybrid stacking strategies can yield the best manually built models by combining the strengths of classical ML and LLM-derived representations.
4. Deployment Implications: The results suggest that in enterprise settings with limited data semantics, LLMs should be deployed as feature enhancers or components within a stacking architecture rather than as end-to-end predictors.

The study provides a clear failure mode for direct prompting on hashed data while validating the utility of embedding-based and hybrid approaches in industrial planning pipelines.