ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition

1. Problem Statement

While Large Language Models (LLMs) show promise as research copilots, their specific capability to discover high-quality scientific hypotheses remains unexamined due to a lack of dedicated benchmarks. Existing benchmarks focus on general intelligence or narrow tasks (e.g., code generation, biomedical idea generation) but fail to evaluate the full scientific discovery process.

The core challenge is that scientific discovery is often viewed as an "intractable" black box. The authors argue that this process can be decomposed into a sufficient set of sub-tasks based on cognitive science principles: creative ideas often arise from the cohesive association of seemingly unrelated knowledge. Specifically, a hypothesis ($h$) is formed by combining a research background ($b$) with inspirations ($i$).

2. Methodology

A. Theoretical Framework: Sufficient Task Decomposition

The authors adopt a decomposition strategy where solving three sub-tasks perfectly guarantees solving the overall discovery task:

1. Inspiration Retrieval: Retrieving relevant knowledge/papers ($i$) based on a research question/background ($b$). This is an Out-of-Distribution (OOD) task because true inspirations are often not semantically obvious to the query.
2. Hypothesis Composition: Synthesizing the background and retrieved inspirations to formulate a novel hypothesis ($h = f(b, i)$).
3. Hypothesis Ranking: Evaluating and ranking the generated hypotheses to identify the most valid and novel ones.

B. Benchmark Construction (ResearchBench)

• Data Source: 1,386 high-quality papers published in 2024 or later across 12 disciplines (Cell Biology, Chemistry, Earth Science, Materials Science, Physics, Energy Science, Environmental Science, Biology, Business, Law, Math, Astronomy).
• Anti-Contamination: By selecting only post-2024 papers, the benchmark minimizes overlap with LLM pretraining data. The framework is designed to automatically extract newer papers as model cutoffs advance.
• Automated Agentic Framework: An LLM-based agent extracts four components from each paper:
  • Research Question
  • Background Survey
  • Inspirations (cited papers that motivated the work)
  • Main Hypothesis
  • Validation: The framework includes "Necessary" and "Sufficient" checkers to ensure extracted inspirations are non-redundant and collectively sufficient to form the hypothesis.
• Expert Validation: Five domain experts (PhD students) validated the decomposition on a random sample of 62 papers, achieving 91.9% accuracy (major issues only) and 82.3% (including minor issues).

C. Evaluation Protocol

• Inspiration Retrieval: Models are given a research question and a pool of 75 candidate papers (2–3 ground-truth inspirations + 72 negatives). Negatives are selected at three "distance levels" (cited papers, same discipline, different disciplines) to test robustness. Performance is measured by Hit Ratio (fraction of ground-truth inspirations in the top-k selection).
• Hypothesis Composition: Models generate hypotheses using the background and retrieved inspirations. Quality is scored on a 6-point Likert scale against ground-truth hypotheses based on key point coverage.
• Hypothesis Ranking: Models perform pairwise comparisons between a ground-truth hypothesis and negative hypotheses. To mitigate position bias, pairs are compared twice with reversed order.

3. Key Contributions

1. First Large-Scale Benchmark: Introduces ResearchBench, the first benchmark specifically designed to evaluate LLMs on the sufficient sub-tasks of scientific discovery across 12 diverse disciplines.
2. Automated, Contamination-Resistant Pipeline: Develops an agentic framework that extracts research components with high accuracy, enabling scalable benchmark updates without manual annotation or data contamination.
3. Comprehensive OOD Analysis: Provides the first large-scale study on Out-of-Distribution (OOD) inspiration retrieval, demonstrating that LLMs can surface novel knowledge associations beyond explicit semantic similarity.
4. "Research Hypothesis Mines" Concept: Proposes a new paradigm where LLMs act as mines for hypotheses, where model capability represents the richness of the mine and inference compute represents the number of miners.

4. Key Results

A. Inspiration Retrieval (OOD Task)

• High Performance: LLMs excel at retrieving inspirations, even when they are not semantically obvious.
  • When selecting the top 4% of candidates, GPT-4o achieved a 45.65% Hit Ratio (ground-truth inspirations included).
  • Even smaller models (e.g., Llama-3.1-8B) achieved ~37.87%.
• Scaling Law: Performance scales rapidly up to 8B parameters but hits a bottleneck around 70B parameters, suggesting a limit in current architectures for this specific OOD retrieval task.
• Bias: Models tend to prefer papers closer in semantic distance (cited or same discipline), but still successfully retrieve distant, cross-disciplinary inspirations.

B. Hypothesis Composition

• Moderate Performance: Models can associate background and inspirations but struggle with deep integration.
  • Top models (e.g., Qwen Plus, GPT-4o) achieved normalized scores around 53–57%.
  • Errors often stem from missing key elements or superficial combinations of inspirations.

C. Hypothesis Ranking

• Variable Performance: Ranking accuracy varies significantly.
  • DeepSeek-V3 and Claude 3.5 Sonnet achieved the highest accuracy (~80%).
  • Position Bias: Many models (e.g., Llama-3.1-8B) showed strong position bias, preferring the first hypothesis in a pair, leading to self-contradictory results in ~90% of cases when not averaged.
• Scaling: Unlike retrieval, ranking benefits significantly from larger parameter counts and better pretraining strategies.

5. Significance and Implications

• LLMs as Scientific Tools: The results suggest LLMs are not just text generators but effective "hypothesis mines." They can autonomously screen vast literature for novel associations, synthesize them into hypotheses, and rank them, potentially accelerating scientific discovery.
• Limitations Identified:
  • Retrieval: Limited by pretraining data distribution (dense intra-domain, sparse inter-domain links).
  • Composition: Limited by next-token prediction objectives that favor frequent patterns over novel, low-probability concept combinations.
  • Ranking: Constrained by autoregressive architecture causing positional dependencies.
• Future Directions: The benchmark provides a foundation for training models specifically for scientific discovery, moving beyond general chat capabilities to specialized "copilot" agents that can autonomously generate and validate scientific insights with minimal human intervention.

In conclusion, ResearchBench establishes that while current LLMs are highly effective at finding the "ingredients" (inspirations) for scientific breakthroughs, the "cooking" (composition) and "tasting" (ranking) still require refinement. However, the potential for LLMs to act as autonomous discovery engines is strongly supported by the data.