Technical Summary: HypoArena – Benchmarking Prospective Hypothesis Discovery in LLMs

1. Problem Definition: The Gap in Investigative Reasoning

Current Large Language Model (LLM) benchmarks predominantly evaluate Reactive Question Answering (QA), where models retrieve answers to well-posed questions or complete predefined tasks. However, real-world discovery in scientific research, accident investigation, and financial analysis often begins not with a clear question, but with "real-world chaos": a tangle of anomalous observations, fragmented facts, and incomplete records.

The critical capability missing from current evaluations is Prospective Hypothesis Discovery (PHD). PHD requires a model to autonomously construct a grounded, discriminative, and testable hypothesis space from inconclusive evidence before a conclusion is reached. Existing benchmarks fail to measure this because:

• Data Limitations: Expert documents (papers, reports) are written post-conclusion, containing explicit causal links and final claims that collapse PHD into simple restatement.
• Metric Limitations: High-quality PHD outputs are open-ended hypothesis spaces, not single reference answers. Traditional metrics (exact match or absolute rubric scoring) underestimate valid hypotheses that fall outside a "gold set" or struggle to discriminate between open-ended outputs of similar surface quality.

2. Methodology: The HypoArena Framework

The authors introduce HypoArena, a benchmark and evaluation framework designed to isolate and measure PHD. The framework consists of two core components: HypoData (the dataset) and HypoEval (the evaluation protocol).

2.1 HypoData: Retrospective Context Regression

To create valid test cases without leaking answers, the authors propose a scalable Forge–Audit pipeline utilizing Retrospective Context Regression.

• Source Material: 988 cases are curated from expert documents across six domains: Biomedical Science, Machine Learning, Social Science, Financial Analysis, IT Operations, and Safety Investigation.
• Context Construction (The "Forge"): The pipeline reconstructs a "pre-conclusion" context ($C$) from source documents. It preserves factual substrates (timestamps, observations, fragmented records) while strictly filtering out explicit conclusions, target hypotheses, and retrospective causal attributions.
  • Scientific Domains: Uses document merging of pre-discovery literature to synthesize a factual substrate.
  • Analytical Domains: Employs "Structural De-conclusion" to strip expert verdicts while retaining granular facts.
• Reference Construction: Hidden from the model during generation, the reference side contains the original hypothesis-evidence pairs ($H, E$) derived from the source, serving as a ground truth for verification and calibration.
• Audit Loop: An automated Audit Agent validates candidates against three constraints: Leakage (no answer-side info in context), Faithfulness (hypothesis supported by source), and Supportability (evidence grounds the hypothesis). A human quality audit confirmed a 92% pass rate on sampled cases.

2.2 HypoEval: Arena-Based Evaluation

Recognizing that multiple valid hypotheses can exist for a single context, HypoEval moves away from single-reference matching.

• Primary Metric (Arena Protocol): Models compete in pairwise matchups. An LLM-as-a-judge compares two hypothesis sets generated from the same context.
  • Judgment Dimensions: Six dimensions are evaluated: Contextual Grounding, Inferential Insight, Evidential Justification, Hypothesis-Space Breadth, Directional Distinctness, and Analytical Utility.
  • Aggregation: Verdicts (ranging from $A \gg B$ to $B \gg A$) are aggregated using the Bradley–Terry–Davidson (BTD) model to produce a robust global leaderboard, explicitly modeling ties.
• Secondary Metric (Rubric Scoring): A 1–5 scale scoring track is retained for diagnostic profiling, allowing analysis of specific strengths and weaknesses (e.g., strong grounding but weak insight).

3. Experimental Setup and Results

The authors evaluated 15 frontier LLMs (including claude-sonnet-4.6, gpt-5.4, kimi-k2.6, and others) across both Baseline Mode (direct generation) and Agent Mode (using a library of 12 structured analytic skills like Analysis of Competing Hypotheses and Chronology Analysis).

3.1 Key Findings

• Capability Stratification: The arena evaluation revealed clear performance stratification, with a spread of over 360 BTD points across models. Top-tier models (claude-sonnet-4.6, claude-opus-4.6, gpt-5.4) significantly outperformed others.
• Resolution of Arena vs. Rubric:
  • Score Compression: Rubric scores exhibited extreme ceiling effects, with 95–98% of assessments landing at or above 4.0/5.0, compressing model differences into a narrow band.
  • Discriminative Power: The arena protocol provided high-resolution separation (345–490 point spread per domain), resolving fine-grained differences that rubric scoring missed.
  • Ranking Instability: Protocol-dependent reordering was observed; models ranked highly by rubric scores often dropped in arena rankings, and vice versa.
• Model-Dependent Effects of Analytic Skills: The use of structured analytic skills (Agent Mode) did not yield uniform gains. Improvements were heterogeneous and model-dependent; for instance, kimi-k2.5 gained ~88 points, while claude-opus-4.6 declined by ~60 points.
• Failure Modes: Qualitative analysis identified candidate compression as a key failure mode, where models internally generated diverse reasoning paths but compressed them into overly narrow final submissions.
• Alignment: The arena rankings showed strong alignment with human expert judgments (Kendall's $\tau = 0.90$) and correlated with external signals (ICLR 2026 acceptance outcomes for the ML subset), validating the protocol's fidelity.

4. Contributions and Significance

The paper claims the following contributions:

1. Task Definition & Benchmark: The formalization of Prospective Hypothesis Discovery (PHD) and the introduction of HypoArena, a benchmark of 988 cases across six diverse domains.
2. HypoData Construction: A scalable Forge–Audit method using Retrospective Context Regression to reconstruct pre-conclusion contexts from completed expert documents, solving the data leakage problem inherent in using raw scientific literature.
3. HypoEval Framework: A dual-track evaluation system combining pairwise arena ranking (via Bradley–Terry–Davidson aggregation) with rubric scoring for diagnostics. This addresses the limitations of reference-matching and absolute scoring in open-ended discovery tasks.
4. Empirical Insights: A systematic analysis of 15 LLMs demonstrating that arena evaluation provides finer-grained model separation than traditional rubrics and that structured analytic skills act as model-dependent interventions rather than universal performance boosters.

Significance: The authors argue that PHD represents a distinct, largely unmeasured competence of LLMs essential for real-world investigative reasoning. By treating PHD as a specific target for evaluation, HypoArena offers a more realistic and rigorous assessment of a model's ability to navigate "real-world chaos" and propose actionable, grounded directions for future investigation, moving beyond the current paradigm of reactive question answering.