Contemporary AI lacks the imagination to diverge or negate in science

Technical Summary: Contemporary AI lacks the imagination to diverge or negate in science

Problem Statement
Despite bold projections that artificial intelligence (AI) will accelerate scientific discovery, the field lacks large-scale, scientist-in-the-loop evidence to validate these claims. Current automated benchmarks often fail to capture the nuances of expert judgment, and there is a risk that fluent text substitutes for genuine insight. The central problem addressed is whether current Large Language Models (LLMs) can genuinely diverge from established knowledge to propose novel, feasible, and probable scientific hypotheses, or if they merely recombine existing patterns. Furthermore, it remains unclear whether automated evaluators (LLM-as-a-judge, metrics) can reliably substitute for human expert assessment in scientific idea generation.

Methodology
The authors conducted the largest expert audit of AI for science to date, involving a "human-in-the-loop" evaluation pipeline:

1. Data Collection: The study assembled 121,640 full-text empirical preprints published after 2023 from six non-arXiv platforms (BioRxiv, MedRxiv, ChemRxiv, SocArXiv, PsyArXiv, EdArXiv). arXiv was deliberately excluded to avoid data contamination, as its content dominates LLM pretraining corpora.
2. Context Extraction: Using the reasoning model o3-mini, the authors extracted the core scientific puzzle, surrounding factual context, and the authors' own hypotheses from each paper. A paraphrase-based leakage detector ensured that the extracted context did not contain the authors' original hypotheses (99.7% author confirmation rate).
3. Hypothesis Generation: 26 diverse LLMs (including 19 non-reasoning chat models, 5 reasoning models, and 2 agentic "deep research" systems) were prompted to generate follow-up hypotheses based solely on the extracted context and puzzle.
4. Expert Evaluation: 6,749 scientists (authors of the original papers) rated 25,139 AI-generated hypotheses. Each hypothesis was scored on four dimensions: Conceptual Novelty, Empirical Feasibility, Probability of Being True, and Favorability of Adoption.
5. Automated Benchmarking: The study evaluated various automated evaluators (LLM-as-a-judge, n-gram novelty, semantic distance, perplexity, and state-of-the-art reward models) against the human ground truth.
6. Reward Model Training: A Qwen3-14B model was post-trained on human pairwise preferences using a Bradley-Terry objective with a margin term to capture field-specific nuances.

Key Results

• Hypothesis Space and Divergence: Non-reasoning LLMs collapse into a narrow "hivemind," generating highly similar ideas (high pairwise cosine similarity). Reasoning models explore a wider hypothesis space but still fail to spontaneously propose null hypotheses (claims of no effect/relationship). While humans formulate null hypotheses infrequently due to publication bias, LLMs formulate them even less often, indicating a structural inability to reason about absence or negation.
• Expert Judgment Biases:
  • Self-Similarity: Scientists strongly prefer AI ideas that resemble their own work (coefficient = 1.28). High similarity correlates with higher perceived feasibility and probability but lower perceived novelty.
  • Seniority and Field: Senior scientists are harsher critics of AI ideas, particularly in the social sciences. Social scientists tolerate risk (lower probability) more readily than life scientists, who prioritize feasibility and probability due to the high cost of failed experiments.
  • Field-Specific Performance: AI ideas were rated lowest in the social sciences across all dimensions (novelty, feasibility, probability), likely because pluralistic fields with contested interpretations are difficult for models optimized to reproduce modal patterns.
• Failure of Automated Evaluators: Existing automated judges (including SOTA models and LLM-as-a-judge) show weak correlation with human expert judgment ($r < 0.35$). They exhibit a strong central tendency, clustering ratings in the middle of the scale and failing to assign extreme scores. Standard novelty metrics (semantic distance, perplexity, n-grams) perform near chance.
• Reward Model Success: A reward model post-trained on human ratings (Qwen3-14B) significantly outperformed SOTA baselines (up to 27% improvement) and achieved pairwise accuracy (61–69%) that exceeds the consistency of independent human peer reviewers (61.0% ± 0.1%). Domain-specific models performed better than general models when sufficient in-domain data was available.

Key Contributions

1. Large-Scale Empirical Audit: Provided the first large-scale, scientist-in-the-loop evaluation of AI-generated scientific hypotheses across multiple disciplines, moving beyond small-scale or synthetic benchmarks.
2. Identification of Structural Limitations: Demonstrated that current AI lacks the capacity for "subtractive change" (negation/null hypotheses) and struggles in pluralistic fields requiring context-aware interpretation.
3. Evaluation of Automated Judges: Systematically showed that current automated evaluation methods (LLM-as-a-judge, metrics) are insufficient proxies for expert scientific judgment.
4. Distillation of Expert Taste: Successfully trained a reward model that captures field-specific "taste" and achieves consistency levels comparable to human peer review, suggesting that expert judgment can be partially distilled into machinery.

Significance and Claims
The paper argues that while AI can expand the search space and surface candidate directions, it currently lacks the "imagination" to diverge from established patterns or to negate existing knowledge. The authors claim that:

• AI is a collaborator, not a replacement: Today's scientific AI benefits from human grounding; it cannot yet independently frame questions, recognize meaningful results, or contest categories.
• The "Null" Gap is Structural: The inability to generate null hypotheses is not a minor bug but a structural feature of how LLMs learn from corpora dominated by positive claims and suppressed null results.
• Curiosity is Missing: Current models are optimized to predict, not to wonder. True scientific acceleration requires encoding "computational curiosity"—an intrinsic drive to probe anomalies, contradictions, and absences.
• Evaluation Reform: Claims of AI-driven acceleration cannot be adjudicated by fluency or automated benchmarks alone; they require validation against domain experts who define the contribution.

The study concludes that until AI systems are directed toward violations of expectations and can actively reach into the world to accumulate data on "what does not work," they remain limited to recombining the published literature rather than extending it.