Always Tell Me The Odds: Fine-grained Conditional Probability Estimation

Technical Summary: Always Tell Me The Odds: Fine-grained Conditional Probability Estimation

Published at: COLM 2025

1. Problem Statement

While Large Language Models (LLMs) excel at deterministic reasoning, they struggle with probabilistic reasoning under uncertainty. Real-world knowledge is inherently probabilistic, yet current LLM approaches to uncertainty estimation suffer from three primary flaws:

• Coarseness: Verbalized confidence often sticks to a few discrete values (e.g., "75%" or "50%").
• Miscalibration: Logit-based probabilities are often overconfident or biased.
• Lack of Fine-grained Precision: Existing models often rely on small encoder-based architectures that fail to leverage the deep world knowledge and reasoning capabilities of modern large-scale decoder-only LLMs.

The authors aim to build models capable of estimating the fine-grained conditional probability $P(\text{proposition} \mid \text{context})$—providing precise, continuous-like estimates rather than simple categorical labels.

2. Methodology

The authors propose a multi-stage pipeline to train modern decoder-based LLMs for high-precision regression.

A. Synthetic Data Creation (Pseudo-labeling):
To overcome the scarcity of high-quality probabilistic human annotations, the authors use a sophisticated synthetic pipeline:

1. Reasoning-Augmented Prompting: Multiple LLMs generate probability estimates accompanied by reasoning chains.
2. Agreement-Based Filtering: Instances where different LLMs show high discrepancy are flagged.
3. LLM-as-a-Judge: For high-discrepancy samples, a powerful LLM judge evaluates the quality of the reasoning chains and assigns a confidence score to each.
4. Distribution Aggregation: These scores are used to create a weighted mixture distribution, providing a more robust "pseudo-label" than any single LLM could provide.

B. Model Fine-tuning (Decoder-based Regression):
Instead of standard classification, the authors treat the problem as a specialized regression task:

• Expected Label Scoring Rule: The interval $[0, 1]$ is divided into $N$ discrete bins, each assigned a special token. To achieve fine-grained precision, the model is trained to predict a quantized Gaussian distribution centered at the target probability. The final estimate is the expectation over these bins: $\hat{y} = \sum f(b_j) \cdot p(t_j \mid x)$.
• Hybrid Training Objectives: The model is optimized using a composite loss function:
  $$\mathcal{L} = \mathcal{L}_{\text{Direct}}(\text{Human}) + \beta_1 \mathcal{L}_{\text{Direct}}(\text{Synthetic}) + \beta_2 \mathcal{L}_{\text{Rank}}(\text{Pairwise})$$
  • $\mathcal{L}_{\text{Direct}}$: Forward KL-divergence to match the target distribution.
  • $\mathcal{L}_{\text{Rank}}$: A margin loss to ensure the model maintains consistent relative rankings between pairs of instances (e.g., if $A$ is more likely than $B$, the model's predicted scores must reflect this).

3. Key Contributions

• Novel Training Framework: A method to transform decoder-only LLMs into precise continuous regressors using the expected label scoring rule.
• Advanced Data Pipeline: A scalable approach to generating high-quality probabilistic supervision using LLM ensembles and "LLM-as-a-judge" refinement.
• Comprehensive Evaluation Suite: A three-tier evaluation framework covering Intrinsic (alignment with labels), Comparison (plausibility ranking), and Structural (uncertainty propagation in complex reasoning trees like Maieutic and BIRD) tasks.

4. Results

• Superior Performance: The proposed models consistently outperform existing fine-tuned encoder models (like RoBERTa) and zero-shot prompting methods (like GPT-4o) across almost all benchmarks.
• Scaling Laws: Performance improves significantly as the model size increases (e.g., moving from 7B to 14B parameters).
• Generalization: The inclusion of synthetic data significantly improves the model's ability to generalize to distant domains (e.g., medical texts or blog posts).
• Human Alignment: The models exhibit "human-like" uncertainty, producing multi-modal distributions in cases where human opinions are divided (as shown in the ChaosNLI analysis).
• Decision Support: The models significantly enhance structured reasoning frameworks (Maieutic Prompting and BIRD), showing that better local probability estimation leads to better global reasoning accuracy.

5. Significance

This work shifts the paradigm of uncertainty in LLMs from "verbalized confidence" to "fine-grained probabilistic estimation." By enabling LLMs to "tell us the odds" with precision, this research paves the way for more reliable AI agents capable of making nuanced decisions in high-stakes, uncertain, and ambiguous real-world environments.