Technical Summary: SCIPATHS: Forecasting Pathways to Scientific Discovery

Problem Definition

Scientific progress relies on sequences of enabling contributions, yet existing AI4Science benchmarks predominantly focus on citation prediction, literature retrieval, or idea generation. These tasks often treat scientific outputs as standalone entities, failing to model the specific dependencies that make progress possible. Current citation-based forecasting operates at the paper level, which can conflate heterogeneous contributions within a single publication (e.g., distinguishing between a dataset and a model released in the same paper). Furthermore, citation-based supervision relies on influence proxies that may not accurately reflect the functional necessity of prior work.

This paper introduces discovery pathway forecasting: given a target scientific contribution and the prior literature available at a specific time, the task is to (1) identify the enabling contributions required to realize the target and (2) ground each enabling contribution in representative prior work when such work exists, or mark it as unmapped. The core challenge is reasoning backward from a target to the specific functional building blocks and prior-work dependencies that render it feasible.

Methodology

The SCIPATHS Benchmark

The authors present SCIPATHS, a benchmark constructed from machine learning and natural language processing papers (NeurIPS, ICML, ACL, EMNLP, 2023–2025). The dataset consists of:

• 262 Expert-Annotated Gold Pathways: These serve as the evaluation standard.
• 2,444 Silver Pathways: Generated automatically in a hindsight setting for training and large-scale analysis.

Construction Process:

1. Target Selection: Target contributions are selected based on evidence of downstream reuse. A citation-intent classifier identifies "USES" and "EXTENDS" contexts, which are then verified by LLMs to ensure functional dependence rather than mere mention.
2. Contribution Clustering: Downstream usage contexts are clustered by semantic similarity to consolidate repeated uses across independent papers, forming candidate target contributions.
3. Expert Annotation: Annotators decompose each target into a pathway under a constructive necessity criterion: an enabling contribution is included only if its removal would prevent the target from being realized in its claimed form.
4. Annotation Schema: Each pathway records:
   • Enabling Contributions: Functional components (e.g., model initializations, data sources, core methods).
   • Roles: Functional roles such as Core Method, Conceptual Framework, Data Source, etc.
   • Groundings: Representative prior papers or an "unmapped" decision if no single prior work cleanly realizes the contribution.
   • Rationales: Evidence-backed explanations for necessity and grounding decisions.

Task Formulation

The benchmark defines two tasks:

• Task A (Enabling Contribution Generation): Given a target contribution claim, the model predicts the set of enabling contributions, their functional roles, and rationales.
• Task B (Prior-Work Grounding): Given the enabling contributions (either gold or predicted), the model must identify prior papers that realize them or mark them as unmapped. This is evaluated under a fixed-budget literature search (Semantic Scholar) with retrieval and reranking.

Evaluation Metrics

• Task A: Evaluated using strict semantic one-to-one matching via an LLM judge (Gemini 3.1 Pro). Precision, recall, and F1 are computed using maximum bipartite matching to prevent broad predictions from receiving credit for multiple distinct contributions.
• Task B: Evaluated using paper-level precision@K, recall@K, and F1@K. Additionally, contribution coverage measures the fraction of mapped gold enabling contributions for which at least one acceptable grounding appears in the top-K results.

Key Results

Task A: Enabling Contribution Generation

Current frontier and open-weight language models recover only a limited fraction of expert pathways:

• Performance: The best model (Gemini 3.1 Pro) achieves an F1 score of 0.189 under strict semantic matching, with a recall of 0.246. Open-weight baselines perform significantly worse (F1 < 0.06).
• Decomposition Difficulty: Models struggle most with core methodological dependencies. While they can reliably identify concrete resources like model initializations (Recall ~0.46) and data sources (Recall ~0.34), they fail to infer specific methodological mechanisms (Recall ~0.12).
• Contextual Gains: Providing citation contexts or related work improves performance slightly but does not close the gap to expert-level recovery.
• Overgeneration: Models that predict a high number of contributions (e.g., GPT-5 Mini with ~9.5 per target) do not achieve higher recall, indicating that quantity does not compensate for the inability to identify necessary functional requirements.

Task B: Prior-Work Grounding

• Decomposition Bottleneck: Grounding performance improves substantially when gold enabling contributions are provided (e.g., Coverage@5 rises from 0.083 to 0.357 for the Gemini agent). This demonstrates that knowing what to search for is the primary bottleneck.
• End-to-End Failure: When using model-predicted contributions from Task A, grounding performance does not improve over claim-only retrieval. Current models do not yet generate enabling contributions that serve as effective search targets.
• Grounding Difficulty: Even with gold contributions, grounding remains challenging. In a diagnostic setting, the best model grounds only 26.8% of groundable contributions, though it correctly identifies unmapped cases with high accuracy (82.4%).

Significance and Claims

The paper posits that scientific dependency reasoning is a distinct capability from retrieving related papers or generating plausible ideas. The results suggest that:

1. Decomposition is the Primary Bottleneck: The inability to identify the correct scientific building blocks (enabling contributions) is a more significant barrier to pathway recovery than the ability to rank or retrieve papers.
2. Need for Structured Reasoning: Scientific forecasting requires models that can represent research trajectories as structured dependency pathways, reasoning backward from a target to its prerequisites, rather than simply aggregating citation lists.
3. Limitations of Current AI4Science: Current language models, even frontier ones, struggle to reconstruct the intermediate functional components that make a scientific contribution feasible, even for domains likely present in their pretraining data.

SCIPATHS shifts the evaluation focus toward this missing capability, providing a resource for modeling research trajectories and analyzing the structure of scientific progress. The authors emphasize that the benchmark captures evidence-grounded pathways rather than a unique historical account, acknowledging that multiple valid decompositions may exist.