Embedding Foundation Model Predictions in Discrete-Choice Models with Structural Guarantees

Technical Summary: Embedding Foundation Model Predictions in Discrete-Choice Models with Structural Guarantees

1. Problem Statement

Discrete-choice models are critical for policy decisions involving significant economic stakes, such as valuing time in transportation or estimating willingness-to-pay (WTP) in consumer behavior. Traditional Multinomial Logit (MNL) models satisfy the second requirement of these tasks—forecasting responses to interventions—by construction. They enforce economic logic through sign-constrained coefficients (ensuring monotonic demand), provide interpretable trade-off ratios (e.g., value of time), and assign zero probability to unavailable alternatives. However, MNL models often lack predictive accuracy compared to modern machine learning.

Conversely, modern tabular foundation models (e.g., TabPFN, Mitra) achieve superior accuracy but violate fundamental economic structures. Their predictions often exhibit:

• Non-monotonicity: Raising a price can increase predicted demand.
• Implausible WTP: Estimates can be negative or fall outside realistic ranges.
• Availability Leaks: Unavailable alternatives receive non-zero probability.

Existing remedies fail to balance these needs: architecturally constrained monotonic networks lose closed-form trade-off ratios; knowledge distillation into MNL students cannot match the teacher's accuracy; and penalized fine-tuning is incompatible with in-context learning foundation models that do not allow gradient access to weights.

2. Methodology

The authors propose a two-stage behavioral adapter that embeds a foundation model's predicted choice probabilities as a precomputed feature within a structurally constrained MNL utility function.

Architecture:
For an observation $i$ and alternative $k$, the utility $V_k(x_i)$ is defined as:
$$V_k(x_i) = \underbrace{\beta^\top \phi_k(x_i)}_{\text{Structural Utility } (V^{\text{struct}}_k)} + \underbrace{g_k(q(x_i))}_{\text{Foundation Model Correction}}$$

• $V^{\text{struct}}_k$: Standard linear utility with sign-constrained coefficients (e.g., $\beta_{\text{cost}} = -\exp(\theta)$).
• $q(x_i)$: A precomputed probability vector from a single forward pass of the foundation model.
• $g(\cdot)$: A small Multi-Layer Perceptron (MLP) with two hidden layers (width 32) that maps the foundation model's predictions to a correction term.

Two-Stage Training Procedure:

1. Stage 1: The correction $g$ is held at zero. The structural coefficients $\beta$ are fitted via maximum likelihood under standard MNL sign constraints, yielding $\beta^*$.
2. Stage 2: The structural coefficients are frozen at $\beta = \beta^*$. Only the neural correction $g$ is fitted via maximum likelihood.

Intervention Protocol (Fixed-q):
To preserve structural guarantees during evaluation, the authors use a "fixed-$q$" protocol. When perturbing an attribute (e.g., increasing cost) to test monotonicity, the foundation model's input $x_i$ is perturbed, but the precomputed probability vector $q(x_i)$ is held fixed at its original value. This ensures the correction term $g(q)$ does not react to the perturbation, isolating the structural component's response.

3. Theoretical Contributions

The paper provides formal proofs regarding the preservation of economic structure:

• Proposition 1 (MRS Preservation): Under the two-stage training and fixed-$q$ protocol, the Marginal Rate of Substitution (MRS) between any two attributes entering only through the structural utility is exactly $\beta^*_j / \beta^*_{j'}$. Consequently, the value of time (VOT) and WTP are analytically guaranteed to match the standalone MNL estimate, regardless of the foundation model or the complexity of $g$.
• Proposition 2 (Failure of Joint Training): If $\beta$ and $g$ are trained jointly, structural identifiability breaks. If the cost attribute is recoverable from the foundation model's predictions ($q$), a one-parameter family of solutions exists where the structural cost coefficient and the correction term cancel each other out. This leads to the collapse of the structural coefficient (e.g., $\beta_{\text{cost}}$) and the loss of interpretable trade-off ratios.

4. Experimental Results

The method was evaluated on three discrete-choice datasets (Swissmetro, LPMC, IoT-Wearables) using two foundation models (Mitra, TabPFN).

• Accuracy: The adapter recovers most of the foundation model's accuracy advantage. It outperforms the standalone MNL by an average of 6.4 percentage points (pp) (up to 12.8 pp) and trails the raw foundation model by at most 2 pp.
• Behavioral Validity:
  • Monotonicity: The adapter achieves 100% monotonicity on cost across all datasets, whereas raw foundation models fail significantly (e.g., TabPFN on LPMC: 28.9% monotonic).
  • Trade-off Ratios: The adapter preserves the MNL's VOT and WTP exactly. In contrast, raw foundation models produce implausible values (e.g., negative VOT) and feature-augmented MNL baselines suffer from "drift," producing VOT estimates orders of magnitude off (e.g., 1509 CHF/hr vs. 84.4 CHF/hr).
  • Availability: The adapter maintains zero probability for unavailable alternatives, whereas raw models leak probability (up to $2 \times 10^{-3}$).
• Counterfactual Evaluation: Under a +10% cost perturbation, the adapter and MNL consistently predict a decrease in aggregate market share. Raw foundation models violated this monotonicity in 6 of 16 scenarios, predicting increased demand for higher prices.
• Robustness: Even when the foundation model's context is restricted to 10% of the original size, the adapter retains at least 6 pp of accuracy gain over the MNL.

5. Significance and Claims

The paper claims to offer a "model-agnostic" recipe that resolves the tension between predictive accuracy and economic consistency in discrete-choice modeling.

• Structural Guarantees as a Design Choice: The authors argue that economic guarantees (monotonicity, interpretable WTP) should not be empirical accidents but mathematical properties of the model architecture. The two-stage adapter achieves this by construction.
• Efficiency: The method requires no gradient access to the foundation model's weights, making it compatible with in-context learning models like TabPFN.
• Audit Pipeline: The work introduces a generalizable audit pipeline for discrete-choice datasets that evaluates monotonicity, trade-off ratios, and availability compliance without requiring knowledge of the model's internals.

The authors acknowledge limitations, such as elevated calibration error (ECE) on the Swissmetro dataset and the dependency of Proposition 2 on the specific dataset's ability to recover cost from foundation model predictions. However, they conclude that the adapter successfully embeds foundation model predictions into a structurally constrained utility, paying a minimal accuracy cost (at most 2 pp) to ensure full structural validity.