Technical Summary: Irrationality on Top of Value Misalignment in LLM Reasoning

Problem Statement

Despite significant progress in aligning Large Language Models (LLMs) with target value functions through methods like Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and Direct Preference Optimization (DPO), a critical gap remains. Even when an LLM is well-aligned during training, it may fail to maximize the aligned value during inference. This paper argues that value misalignment (where the learned value function itself is incorrect) is not the sole source of failure; rather, irrational reasoning persists. This irrationality is defined as the model's inability to select the best available response from its sampled candidates, even when those high-utility candidates exist within the model's output distribution.

Methodology

Formalization of Rational Value Risk

The authors mathematically formalize this phenomenon as Rational Value Risk (RVR).

• Rational Reasoning: Defined as the strategy that maximizes expected utility in the steepest direction under a given value function.
• Rational Value Risk ($R$): The utility discrepancy between a model's deployed reasoning strategy ($d_\theta$) and its rational counterpart ($d^\circ_\theta$).
  $$R(d_\theta) = \mathbb{E}_{x \sim \rho, o \sim P(\cdot|x,y^\circ)}[U(o)] - \mathbb{E}_{x \sim \rho, y \sim d_\theta, o \sim P(\cdot|x,y)}[U(o)]$$
  Where $y^\circ$ is the rational answer and $y$ is the answer generated by the deployed strategy.

Compute-Bounded Estimation

Since perfectly rational reasoning is infeasible to compute due to finite resources, the paper introduces a compute-bounded rational reasoning framework.

1. Candidate Set: For a given prompt, $K$ independent samples are drawn from the model.
2. Empirical Utility: The utility of each candidate is estimated via $L$ verifier calls (or deterministic checks).
3. Estimator: The Empirical Rational Value Risk ($\hat{R}_{M,K,L}$) is calculated as the difference between the maximum empirical utility found in the $K$ candidates (representing the best possible deployment) and the average empirical utility of the deployed strategy.

Error Decomposition

The estimation error of RVR is decomposed into three distinct components:

1. Candidate Approximation Error: The risk that the finite set of $K$ samples does not contain a high-utility answer.
2. Prompt Sampling Error: Statistical error arising from evaluating on a finite number of prompts ($M$).
3. Verification Error: Error introduced by stochastic or imperfect verifiers ($L$).

The paper provides theoretical bounds (Theorem 1) showing that the total error is bounded by the candidate approximation error plus a term dependent on the number of prompts and verifier confidence.

Experimental Setup

The study evaluates a wide range of models (7B to 72B parameters and proprietary APIs) across diverse benchmarks:

• Conversational Tasks: UltraFeedback, AlpacaEval (using stochastic verifiers).
• Verifiable Reasoning: GSM8K, MATH, HumanEval, and MathArena (using deterministic or near-deterministic verifiers).
• Models: Llama-3.1, Qwen-2.5, Tülu-3 families, and proprietary models (GPT-5.2, GPT-5.5, DeepSeek-V4).
• Variables: The experiments vary sampling temperature, self-consistency voting budgets, and reasoning lengths (token budgets).

Key Results

The experiments validate four primary hypotheses:

1. H1: Rational Value Risk is Widespread.
   Across all models and benchmarks, RVR is consistently positive. Models frequently generate high-utility candidates (high Rational Expected Utility, REU) but fail to deploy them consistently (lower Actual Expected Utility, AEU). For instance, on the MathArena benchmark, even strong models like GPT-5.2 exhibit an RVR of 0.360, accounting for 70.6% of the total utility discrepancy.

2. H2: Value Alignment Reduces but Does Not Eliminate RVR.
   Post-training alignment stages (SFT $\to$ DPO $\to$ RLVR) generally improve AEU and reduce RVR. However, a significant risk remains even after RLVR. In some cases (e.g., Tülu-3-70B on MathArena), alignment improves the model's ability to sample high-utility answers (increasing REU), but the RVR actually increases as a proportion of total error, indicating that the deployed strategy still fails to select the best candidates.

3. H3: RVR is Highly Sensitive to Inference-Time Strategy.

   • Temperature: Increasing sampling temperature often improves REU (by finding better candidates) but increases RVR because the model fails to select the best one consistently.
   • Self-Consistency: Increasing the voting budget (self-consistency) significantly reduces RVR by improving AEU while REU remains stable.

4. H4: Longer Reasoning Improves Rationality with Diminishing Returns.
   Extending reasoning length (token budget) can reduce RVR, but the benefits are non-monotonic and diminish after a certain threshold (e.g., $T \ge 256$ or $1024$). On difficult deployment benchmarks like MathArena, longer reasoning improves REU slightly but fails to improve AEU, leaving RVR high.

Significance and Claims

The paper claims to be the first to formally define and empirically measure Rational Value Risk in LLM reasoning, explicitly separating inference-time irrationality from value misalignment.

• Distinct Evaluation Dimension: The authors posit that rationality is a distinct dimension complementary to value alignment. A model can be perfectly aligned (knowing what is valuable) yet irrational (failing to act on that knowledge during inference).
• Diagnostic Tool: The framework serves as a diagnostic tool to identify whether performance gaps are due to the inability to generate high-quality answers (sampling failure) or the inability to select them (deployment failure).
• Limitations: The work focuses on utility defined over final answers, not reasoning trajectories. It does not propose new algorithms to fix RVR but provides the theoretical and empirical foundation for future inference-time algorithms (e.g., verifier-guided search) to minimize unrealized utility.

In conclusion, the paper demonstrates that "irrationality on top of value misalignment" is a pervasive issue in current LLMs, suggesting that future improvements in reasoning performance must address not just what the model learns, but how it reasons and selects answers at inference time.