Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models

1. Problem Statement

Current Large Language Model (LLM) benchmarks primarily focus on factual reading comprehension, where models retrieve specific textual evidence (e.g., "Who directed this movie?") or reason over localized facts. However, many real-world applications (e.g., market research, public opinion analysis) require distributional reading comprehension: the ability to infer population-level trends, preferences, and statistics by aggregating patterns across a collection of unstructured text.

Existing benchmarks for distributional knowledge are limited because they either:

• Probe pre-trained knowledge (e.g., survey-style questions about human values).
• Provide explicit numerical distributions (e.g., Poisson distributions) rather than requiring inference from natural language.

There is a gap in evaluating whether LLMs can derive population-level distributions (e.g., sentiment proportions, topic prevalence) solely by reading and aggregating information from natural language text.

2. Methodology: TEXT2DISTBENCH

The authors introduce TEXT2DISTBENCH, a fully automated, continuously updatable benchmark designed to evaluate this capability.

Data Construction Pipeline

The benchmark is constructed using a four-stage automated pipeline to ensure data freshness and prevent data leakage (models seeing training data):

1. Entity Selection: Selects movies and music released after the model's knowledge cutoff date (at least 6 months post-cutoff) to ensure the model has no prior exposure.
2. Comment Annotation: Uses multiple state-of-the-art LLMs to annotate YouTube comments for two attributes: Sentiment (Positive/Negative) and Topic (Actor, Storyline, Visual, Audio). Disagreements are resolved via majority voting.
3. Empirical Distribution Estimation: Aggregates annotations to create ground-truth discrete joint distributions $P(S, T)$ for each entity.
4. QA Generation: Generates questions based on these distributions.

Benchmark Structure

Each instance consists of entity metadata, a set of user comments, and distributional QA pairs. The evaluation is organized along two axes:

• Distribution Types:
  • Marginal: Overall distribution (e.g., $P(S)$: What is the overall sentiment?).
  • Conditional: Distribution given a specific value (e.g., $P(S|T)$: Sentiment specifically regarding the "Storyline").
  • Joint: Co-occurrence patterns (e.g., $P(S, T)$: Percentage of comments that are both "Positive" and about "Visuals").
• Task Types:
  • Estimation: Predict the percentage for every category (evaluated via Total Variation Distance, TVD).
  • Most Frequent: Identify the dominant category (evaluated via Accuracy).
  • Second Frequent: Identify the runner-up category (evaluated via Accuracy).

3. Key Contributions

1. New Benchmark (TEXT2DISTBENCH): The first reading comprehension benchmark specifically designed to test the inference of distributional knowledge from natural language text, distinct from factual retrieval or pre-trained knowledge.
2. Automated & Scalable Pipeline: A fully automated construction process that allows for continuous updates with newly emerging entities, ensuring long-term reliability and preventing data contamination.
3. Comprehensive Evaluation Framework: A systematic taxonomy covering marginal, conditional, and joint distributions across estimation and ranking tasks.

4. Experimental Results

The authors evaluated six state-of-the-art LLMs (including GPT-5.1, Gemini-2.5-Pro, Grok-4-Fast, Claude-Sonnet-4.5, Llama-3.3-70B, and Qwen3-32B).

• Overall Performance: All models significantly outperformed random baselines, proving LLMs can infer distributional patterns from text.
• Difficulty Hierarchy: Performance follows a consistent difficulty order:
  • Easiest: Marginal distributions.
  • Medium: Conditional distributions.
  • Hardest: Joint distributions (performance drops sharply here).
• Task Sensitivity:
  • Models generally perform better on Estimation and Most Frequent tasks.
  • Second Frequent tasks are notably harder, with significant performance variance across models.
• Distribution Characteristics:
  • Models perform better when the target distribution has a high Top-1 probability mass (clear dominance) and a large Top-2 margin (clear separation between top categories).
  • Performance decreases as the support size (number of non-zero categories) increases.
• Reasoning Scaling: Increasing reasoning effort (e.g., Chain-of-Thought) improves performance, particularly for joint distributions, though with diminishing returns at very high effort levels.
• Prior Beliefs: Models can form informative prior beliefs using only entity metadata (without comments). These priors often approximate the true distribution, and models with stronger priors tend to benefit more when comments are added.

5. Significance and Implications

• Beyond Factual Retrieval: The study highlights that current LLMs possess the capability to aggregate unstructured text into statistical insights, a crucial skill for real-world decision-making and opinion analysis.
• Limitations Identified: While capable, models struggle with complex joint distributions and subtle ranking tasks (identifying the second most frequent item), suggesting limitations in fine-grained aggregation.
• Future Research: TEXT2DISTBENCH provides a scalable, sustainable testbed for developing models that can better understand population-level trends, moving beyond simple fact-checking to holistic text analysis.

In conclusion, the paper demonstrates that while LLMs are effective at distributional reading comprehension, their performance is highly dependent on the complexity of the distribution and the specific task, necessitating careful model selection for different analytical needs.