Technical Summary: Measuring Behavior Portability in Large Language Models

Problem Statement

Large language models (LLMs) are increasingly deployed as autonomous decision-makers, mapping inputs to distributions over actions. A critical challenge in their evaluation is behavioral portability: the extent to which a behavioral mapping learned in one decision environment transfers to another environment that preserves the same underlying incentive structure (payoff-relevant features) but differs in surface presentation (payoff-irrelevant framing).

Current evaluation practices rely on finite prompt suites, implicitly assuming that once a task is fixed, the model's induced action distribution remains stable under payoff-irrelevant reframings. However, this assumption is untested and potentially fragile. If an LLM's behavior depends on payoff-irrelevant features (e.g., narrative tone, specific labels, or framing), finite evaluations cannot reliably certify performance in open-ended, real-world deployment scenarios where the "contract" between the principal and the agent is incomplete. The paper posits that if a model is truly aligned with an evaluator's objectives, its behavior should depend solely on payoff-relevant variables ($x$) and be invariant to payoff-irrelevant variables ($z$).

Methodology

Conceptual Framework

The authors formalize a decision environment as $e = (x, z)$, where $x$ represents payoff-relevant features and $z$ represents payoff-irrelevant features.

• Alignment: A policy $\pi$ is aligned if it selects the utility-maximizing action distribution for a given $x$, regardless of $z$.
• Portability: A policy is $T$-portable if a behavioral summary $T$ (e.g., the action distribution) depends only on $x$.
• Theoretical Implication: Under a uniqueness condition for the evaluator's optimum, alignment implies portability. Consequently, a behavioral model fitted on source environments using only $x$ should predict behavior in a held-out target environment (with the same $x$ distribution) as accurately as a model fitted directly on the target data.

Empirical Protocol

The authors introduce a train-on-environments, test-on-another protocol to measure portability:

1. Task Selection: Seven canonical economic decision problems are used: Dictator, Ultimatum, Trust, Public Goods, Beauty Contest, Lottery Choice, and Normal Form games. These tasks offer transparent, low-dimensional payoff structures.
2. Environment Generation: For each task, 40 distinct textual environments ($z$) are generated. These vary in narrative framing, metaphors, and style while keeping the payoff parameters ($x$) and action spaces identical.
3. Data Collection: LLMs are queried across these environments with varying payoff parameters. Actions are extracted and parsed.
4. Model Fitting:
   • Source-trained: An interpretable behavioral model (Ordinary Least Squares regression) is fitted on pooled data from a subset of source environments, mapping $x$ to predicted actions.
   • Target-trained: The same model architecture is fitted directly on data from a held-out target environment.
5. Evaluation: Both models are evaluated on the held-out target test set.
6. Metric: The primary metric is the Total Variation (TV) distance between the joint distribution of (predicted action, realized action) induced by the source-trained model versus the target-trained model in the target environment.
   • This metric is loss-agnostic: A TV distance of $\epsilon$ implies that for any bounded evaluation criterion defined on the prediction-action pair, the expected difference in performance between the two models is at most $\epsilon$.

Models and Conditions

The study evaluates several models: GPT-4.1-nano, Gemma-3-12B, Llama-3.1-8B, Llama-3.1-70B, and DeepSeek-R1. Experiments are conducted under two prompting regimes:

• Non-CoT: Direct answer generation.
• Chain-of-Thought (CoT): Step-by-step reasoning followed by an answer.

Key Results

1. Substantial Portability Losses

The study finds that behavioral portability is imperfect across all tested models.

• Magnitude: The mean TV distance across models and games is 0.349, indicating significant divergence. This implies that a behavioral mapping learned in one wording often fails to predict behavior in a payoff-equivalent alternative.
• Task Variance: Portability varies by task type.
  • High Portability: Lottery Choice (TV $\approx$ 0.052), where decisions are explicitly quantitative.
  • Low Portability: Social and strategic tasks like Dictator (TV $\approx$ 0.525), Trust (TV $\approx$ 0.525), and Ultimatum (TV $\approx$ 0.480). These tasks, involving fairness, cooperation, or reciprocity, show the largest sensitivity to framing.

2. Impact of Chain-of-Thought (CoT)

CoT prompting improves portability on average but does not do so monotonically or uniformly.

• Average Improvement: CoT reduces the mean TV distance by 0.047 (12.6%) compared to Non-CoT.
• Heterogeneity: The effect of CoT depends on the interaction between the model, the specific game, and the framing.
  • In Normal Form games, CoT drastically reduces portability loss (e.g., -79.7% for GPT-4.1-nano), likely by anchoring reasoning to the explicit payoff matrix.
  • In Ultimatum and Dictator games, CoT sometimes increases portability loss (e.g., +21.3% for GPT-4.1-nano in Ultimatum), suggesting that reasoning traces can amplify sensitivity to fairness narratives or specific wording rather than abstracting away from them.
• Diagnostic: Adding environment indicators to predictive models increases $R^2$ more under Non-CoT than CoT, confirming that CoT reduces but does not eliminate residual dependence on payoff-irrelevant framing.

3. Reasoning Models

DeepSeek-R1, a model specialized for reasoning, demonstrates superior portability in the tasks where it was tested (Lottery Choice, Beauty Contest, Normal Form).

• It consistently lies near the best observed portability frontier.
• Mechanism: DeepSeek-R1 exhibits high "numerical grounding," mentioning payoff-relevant parameters in nearly 100% of responses. Its reasoning motifs are organized around formal task structures (e.g., expected value, Nash equilibrium) rather than frame-specific narratives.
• Nash Support: In Normal Form games, DeepSeek-R1 selects actions supported by a Nash equilibrium in 99.5% of cases, significantly outperforming other models.

Significance and Claims

The paper claims to provide a formal framework and empirical evidence that behavioral characterizations of LLMs obtained in one decision environment cannot be assumed to transfer reliably to structurally equivalent alternatives.

• Evaluation Implications: The results challenge the standard practice of extrapolating performance from finite prompt suites to unseen textual variants. If portability is low, a model's behavior in a test suite may not reflect its behavior in deployment, even if the underlying incentives are identical.
• Alignment: The findings suggest that current LLMs are not fully "aligned" in the sense of being invariant to payoff-irrelevant features. Their decision-making processes are sensitive to the "contract" of the prompt's surface form.
• Robustness: While CoT and reasoning-specialized models (like DeepSeek-R1) can improve portability by recovering the underlying payoff-relevant structure, they do not guarantee it. The sensitivity to framing remains a fundamental property of the models tested.

The study concludes that portability is an empirical object to be measured, not a property that can be certified by a finite suite, and that achieving true behavioral portability in LLMs remains a significant challenge.