Technical Summary: Learning When to Trust LLM Priors (Statsformer)

Problem Statement

Large Language Models (LLMs) encode rich semantic knowledge that can serve as valuable domain priors for supervised learning, particularly in data-scarce, high-dimensional domains like genomics and finance. However, LLM outputs are statistically unreliable: they can be noisy, misspecified, or hallucinated. Existing approaches to integrating LLM knowledge fall into two problematic categories:

1. Unvalidated Trust: Methods that directly inject LLM signals (e.g., feature scores) into models without validation, leaving predictions vulnerable to hallucinated or adversarial guidance.
2. Restricted Scope: Methods that validate priors but are confined to a single model class (e.g., linear models), failing to generalize across heterogeneous learners.

The central challenge is to determine when to trust an LLM-derived prior, rather than assuming the foundation model is reliable. The goal is to integrate semantic priors in a way that improves prediction when the prior is informative but automatically attenuates or ignores it when the prior is weak or adversarial, without sacrificing performance relative to a prior-free baseline.

Methodology: The Statsformer Framework

The authors introduce Statsformer, a validated framework for incorporating LLM-derived semantic priors into supervised statistical learning. The framework operates on the principle of validated prior integration, separating the injection of semantic bias from the validation of its utility.

1. Prior Injection via Monotone Transformations

Statsformer maps LLM-derived feature relevance scores ($V$) into a family of learner-specific prior-injection mechanisms. This is achieved through a monotone, temperature-controlled transformation $\tau_\alpha$ parameterized by $\alpha \geq 0$.

• Null Condition: When $\alpha = 0$, the transformation yields a uniform vector, recovering the prior-free learner.
• Injection Mechanisms: The framework supports three broad, model-agnostic injection strategies via "adapters":
  • Penalty-based: Rescaling feature-specific regularization strengths (e.g., in Lasso or GLMs).
  • Feature-reweighting: Rescaling input features prior to training (e.g., in tree-based methods or SVMs).
  • Instance-weighting: Assigning sample weights to emphasize examples where prior-identified features are active.

2. Out-of-Fold (OOF) Stacking and Validation

Instead of assuming a fixed prior strength, Statsformer treats the prior strength ($\alpha$) and the choice of base learner as hyperparameters to be learned.

• Heterogeneous Library: The framework constructs a dictionary of candidates comprising diverse base learners (e.g., Lasso, XGBoost, Random Forests, Kernel SVMs) across various prior strengths and injection mechanisms.
• Cross-Validation: The system performs $K$-fold cross-validation to generate out-of-fold (OOF) predictions for every candidate configuration.
• Meta-Learning: A meta-learner (e.g., non-negative least squares or regularized logistic regression) aggregates these OOF predictions by solving a convex optimization problem to find weights $\hat{\pi}$.
• Refitting: The final predictor is a convex combination of the base learners refitted on the full dataset, weighted by the learned $\hat{\pi}$.

Key Contributions

1. Validated Semantic-Prior Integration Across Model Classes

Statsformer is a model-agnostic aggregation framework. It integrates LLM-derived feature priors into supervised learning across diverse base learners and injection mechanisms, unified through a monotone prior-injection family. It requires only a single LLM query for feature scoring, with no fine-tuning, iterative prompting, or inference-time LLM calls, making it efficient and scalable.

2. Provable Safety Under Unreliable Priors

The paper provides a formal oracle-style guarantee. Using classical convex aggregation theory, the authors prove that the Statsformer predictor performs no worse than the best convex combination of its in-library candidates (including prior-free learners) up to statistical error.

• Implication: If the LLM signal is weak, misspecified, or adversarial, the framework automatically downweights the prior-informed learners, ensuring the final performance degrades gracefully to the level of the best prior-free ensemble. This establishes a "statistical floor."

3. Stress-Tested Robustness and Quantified Upside

The framework is evaluated on both its safety floor and its potential upside:

• Safety: Experiments with deliberately inverted (adversarial) priors show that Statsformer reverts to prior-free performance, confirming it does not catastrophically fail when the prior is misleading.
• Upside: Oracle-prior simulations (using ground-truth relevance) demonstrate substantial performance gains in high-dimensional, data-scarce regimes ($p \gg n$), quantifying the "semantic ceiling" available when high-quality priors are available.

Experimental Results

The authors evaluated Statsformer on diverse tabular datasets (e.g., Breast Cancer, Bank Marketing, ETP, Lung Cancer) with semantically meaningful feature names.

• Performance vs. Baselines: Statsformer consistently outperformed strong baselines, including classical methods (XGBoost, Lasso, Random Forests) and advanced AutoML systems (AutoGluon, AutoML-Agent).
• Comparison to No-Prior Stacking: Against a "Super Learner" baseline (stacking without priors), Statsformer showed positive relative improvements in most cases, with significant gains in underdetermined settings (e.g., ETP and Lung Cancer datasets).
• Model Robustness: The framework remained robust across different LLM choices for generating priors (from open-source models like Qwen to proprietary models like Claude and o3). While larger models yielded stronger gains, the framework maintained safety even with weaker models.
• Adversarial Robustness: When tested with corrupted priors, the win rate of Statsformer against the no-prior baseline clustered near 0.5, confirming that the framework does not degrade below the performance of a standard ensemble when the prior is harmful.

Significance and Claims

The paper positions Statsformer as a reliability-oriented approach to LLM-informed statistical learning. Its primary significance lies in shifting the paradigm from "trusting LLM knowledge directly" to "validating semantic priors against data."

• Guardrailed Learning: By decoupling prior injection from prior validation, Statsformer allows foundation-model knowledge to influence statistical estimation only when empirically justified.
• Formal Guarantees: It is the first model-agnostic framework for LLM-derived feature priors that offers a formal downstream performance floor, ensuring that the integration of LLM knowledge cannot harm the model's performance relative to the best available prior-free candidate.
• Scalability: Unlike agent-based AutoML systems that rely on complex, iterative LLM interactions, Statsformer is computationally efficient, requiring only a one-time query for feature scores and standard statistical training.

The authors conclude that while current LLMs provide valuable semantic signals, their reliability varies. Statsformer provides the necessary statistical machinery to harness these signals safely, unlocking their value in high-dimensional regimes while protecting against hallucination and misspecification.