Technical Summary: Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth

Problem Statement

As Large Language Models (LLMs) increasingly generate long-form outputs (e.g., code repositories, complex reports), effective uncertainty estimation must shift from discarding entire responses to identifying errors at fine-grained levels. However, evaluating uncertainty at such resolutions (token to atomic unit) is currently hindered by the lack of "zero-noise" benchmarks. Existing long-form benchmarks rely on fallible labels, often generated by other LLMs or human judges, which introduces label noise and subjectivity. This noise severely skews uncertainty metrics, particularly when correctness-label errors share biases with uncertainty scores, making it difficult to distinguish between a model's inability to rank errors and the noise inherent in the evaluation itself.

Methodology: The SALT Benchmark

To address these limitations, the authors introduce Single-answer Atomic Long-form Target (SALT), a benchmark designed for zero-noise uncertainty evaluation in long-form generation.

Design Principles

SALT consists of six procedurally generated tasks spanning logic, mathematics, code, translation, and "multi-needles-in-a-haystack" (information retrieval). Key design objectives include:

1. Deterministic Ground Truth: Every task instance is algorithmically constructed to have a single, unique, and deterministic ground truth known a priori. This eliminates reliance on external judges.
2. Unit-Level Decomposition: Generations are decomposed into "evaluation units" (atomic units like matrix elements or logical deductions, and line units like matrix rows). Correctness is determined via strict string equality between generated units and ground-truth units.
3. Unbounded Generation: Tasks are procedurally generated with tunable parameters, allowing for effectively infinite instance creation to prevent test-set memorization.
4. Granular Metrics: The benchmark evaluates accuracy (Precision), calibration (Expected Calibration Error - ECE), and ranking (Area Under the Receiver Operating Characteristic - AUROC) at multiple resolutions: atomic, line, and generation levels.

Experimental Setup

The authors evaluated over 50 contemporary LLMs (including GPT-5.1, Gemini-3-Pro, Qwen-3, and various reasoning models) across these tasks. They analyzed various confidence functions (logit-based and verbalized) and post-hoc calibration methods (Z-Sigmoid, Min-Max, Binned) to assess their performance in measuring calibration and ranking.

Key Findings

1. Correctness Dynamics in Long-Form Generation

• Path Dependence: Future correctness is strongly path-dependent. The quality of the generated prefix dictates future accuracy. A small fraction of prior errors (e.g., 15% incorrect atoms) correlates with a disproportionate degradation in subsequent correctness (dropping to ~50%).
• Global vs. Local Context: Global prefix correctness is the dominant driver of future errors, carrying a signal roughly twice as strong as local (last 10 atoms) correctness.
• Bounded Length Degradation: While increasing answer-context length contributes to error, this effect is bounded and saturates quickly (around 66 atoms), independent of the correctness of the prefix.

2. The High-Resolution Ranking Gap

• Granularity Failure: There is a significant gap in confidence ranking performance between coarse and fine resolutions. While models often show informative ranking signals at the line-unit level (Macro-AUROC), they frequently fail to distinguish correct from incorrect atomic units (Micro-AUROC), often performing barely better than random guessing.
• Implication: Raw confidence signals are insufficient for isolating localized errors in long-form generation, even when they work for coarser units.

3. The Reasoning Trade-off

• Accuracy vs. Ranking: Reasoning mechanisms (Chain-of-Thought prompting or specialized reasoning models) introduce a distinct trade-off. While they significantly improve Precision and Calibration (ECE), they systematically degrade Confidence Ranking (AUROC).
• Mechanism: Reasoning makes models more accurate but less able to distinguish their own errors, effectively making them more "convincing" but less reliable in self-assessment. This degradation is a distinct side effect, not merely a byproduct of precision changes.

4. Confidence Functions and Calibration

• Function Dominance: Logit-based confidence functions (e.g., Logprobs Sum, Mean Probability) consistently outperform verbalized methods across both calibration and ranking metrics.
• Calibration-Ranking Trade-off: Post-hoc calibration methods that minimize ECE (e.g., Binned calibration) often harm confidence ranking (AUROC). Conversely, methods that preserve ranking structure may yield poor calibration. There is no single method that optimizes both simultaneously.

Significance and Claims

The paper claims that SALT provides a necessary bedrock for future research by shifting uncertainty evaluation from coarse, noisy judges to high-resolution, deterministic ground truth. The authors assert that their findings reveal a critical decoupling between capability and reliability in current scaling paradigms:

• Risk-Critical Applications: For domains like healthcare, high accuracy is insufficient if accompanied by brittle confidence estimates that cannot identify specific errors.
• Reasoning Limitations: The observed "reasoning trade-off" suggests that current reasoning mechanisms may inadvertently degrade a model's ability to signal uncertainty, a crucial factor for safe deployment.
• Future Directions: The authors suggest that the dense, unit-level feedback provided by SALT is uniquely suited for training granular confidence classifiers and for Reinforcement Learning (RL) post-training to mitigate the reasoning-ranking trade-off.

The work concludes that as models scale in intelligence, they must also scale in the awareness of their own limitations, a goal currently hindered by the lack of zero-noise evaluation frameworks like SALT.