Benchmarking Large Language Model Rationality Using Measurement Axioms

Technical Summary: Benchmarking Large Language Model Rationality Using Measurement Axioms

Problem Statement
As Large Language Models (LLMs) are increasingly deployed as decision-makers in high-stakes domains such as credit risk, healthcare, and policy, current evaluation paradigms remain insufficient. Existing benchmarks prioritize accuracy and factual grounding, often ignoring the structural properties required for coherent and interpretable decision-making. In realistic environments lacking a clear "ground truth," benchmark accuracy fails to capture whether a model's decision process is internally consistent. Specifically, there is a lack of formal frameworks to determine if LLM-generated preferences satisfy the structural conditions—such as transitivity—necessary for rational agency and utility representation. Without transitivity, preferences cannot be coherently ordered, undermining the interpretability of model outputs and complicating alignment with human values.

Methodology
The authors introduce a measurement-theoretic framework grounded in utility theory to evaluate AI decision-making. The study defines a decision system as rational if it admits a utility function, a condition necessitated by the axiom of transitivity of preference. To address the non-deterministic nature of LLM outputs, the authors employ direct first-token log-probability extraction rather than relying on generated text or post-hoc parsing. This allows for the recovery of deterministic choice probabilities for any given input.

The study evaluates 20 distinct LLMs (including Meta's Llama series, Alibaba's Qwen, MistralAI, and Google's Gemma) across three experimental manipulations:

1. Generation Temperature: Tested across 10 settings (0.2 to 2.0) to determine if increasing stochasticity destabilizes the logical structure of preferences.
2. Contextual Memory: Compared "Memory-Less" conditions (independent trials) against "Memory-Full" conditions (where the model receives a record of its 10 most recent choices) to assess if context history stabilizes or degrades preference consistency.
3. Question Format: Varied the presentation of probability (fraction vs. percentage) and monetary value (numeric, "$" prefix, "dollars" suffix) across six formats to test framing effects on structural coherence.

The stimuli consisted of five gamble sets adapted from human transitivity experiments (Tversky, 1969; Davis-Stober et al.), comprising 20 choice trials per set. The authors evaluated adherence to four probabilistic models of transitivity:

• Weak Stochastic Transitivity (WST): The least restrictive condition ($P_{AB} \ge 0.5 \land P_{BC} \ge 0.5 \implies P_{AC} \ge 0.5$).
• Medium-Certain Transitivity (MCT): A stricter condition ($P \ge 0.75$).
• High-Certain Transitivity (HCT): A high-certainty condition ($P \ge 0.9$), designed for agents producing deterministic probability computations.
• Mixture Model of Transitive Preference (MMTP): Allows for stochastic choice via sampling over transitive deterministic preferences.

Key Results

• Transitivity Adherence: Adherence to the least restrictive model (WST) was stable across temperatures but varied significantly by model type, with instruction-tuned models generally outperforming base models. However, adherence to stricter models (MCT and HCT) was near zero for almost all models, with only a few 70B instruction-tuned variants showing non-zero counts that declined as temperature increased.
• Temperature Effects: While MMTP adherence increased slightly with higher temperatures, the authors argue this is likely a mathematical artifact of probabilities compressing toward 0.5 (random guessing) rather than an improvement in coherence.
• Question Format: Adherence was modestly higher when probabilities were presented as percentages rather than fractions. The representation of monetary value had negligible impact.
• Contextual Memory: Contrary to the hypothesis that context might stabilize preferences, providing a memory of prior choices decreased adherence to WST and MMTP. Furthermore, the specific ordering of prior choices introduced significant variability, with some models showing adherence counts spanning the full possible range (0 to 30) depending on the sequence.
• Intransitivity Patterns: When violations occurred, they were not random. Many models exhibited "human-aligned intransitivity," replicating the lexicographic semi-order patterns documented in human decision-making (Tversky, 1969). However, the rate of violation in LLMs was substantially higher than in human respondents.

Key Contributions

1. Formal Framework: The paper provides a principled, measurement-theoretic framework for evaluating AI rationality that moves beyond accuracy to assess structural coherence.
2. Methodological Rigor: By utilizing direct log-probability extraction, the study bypasses the noise of stochastic text generation to measure intrinsic choice probabilities, offering a reproducible standard for evaluating preference consistency.
3. Systematic Benchmarking: The study systematically varies operational parameters (temperature, memory, format) to demonstrate how these factors impact the logical structure of LLM decisions, revealing that current models lack stable utility representations.
4. Comparative Analysis: The work offers a direct structural comparison between LLMs and human decision-making, highlighting that while LLMs can mimic human intransitive patterns, they do so with significantly higher frequency and less stability.

Significance and Claims
The authors claim that measurement axioms offer a complementary foundation for evaluating AI systems, distinct from and necessary alongside benchmark accuracy. The study demonstrates that current LLMs fail to satisfy the structural conditions required for coherent decision-making, particularly under stricter rationality criteria (MCT, HCT) and when exposed to contextual memory.

The paper modestly asserts that its framework evaluates the structure of outputs, not the internal cognitive processes of the models. It concludes that while LLMs may appear to make structured decisions, their inability to maintain consistent preferences across varying conditions indicates an absence of a stable utility representation. The authors suggest that this framework can be extended to other domains (e.g., medical triage, moral trade-offs) and other choice axioms (e.g., monotonicity), serving as a foundational step toward a measurement-oriented evaluation of artificial agent rationality.