💬 NLP

Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context

This paper introduces LiveIdeaBench, a novel benchmark that evaluates LLMs' divergent thinking capabilities for scientific idea generation using minimal context, revealing that creative performance is poorly predicted by general intelligence metrics and suggesting a need for specialized training strategies to enhance scientific creativity.

Original authors: Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu, Hao Sun

Published 2026-02-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu, Hao Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a talent scout looking for the next great inventor. You have a room full of brilliant robots (Large Language Models, or LLMs). Traditionally, to test them, you'd give them a massive, detailed blueprint of a machine and ask them to fix a specific broken part. This tests their ability to follow instructions and solve known problems (Convergent Thinking).

But this paper asks a different question: Can these robots come up with a new invention from scratch, just by hearing a single word? This is Divergent Thinking—the spark of creativity.

Here is the paper "LiveIdeaBench" explained through simple analogies:

1. The Problem: The "Library Test" vs. The "Blank Canvas"

Most current tests for AI are like giving a student a thick textbook and asking, "What is the answer to question 5?" If the student has read the book, they get an A. This measures how well they can find and synthesize existing information.

However, real scientific breakthroughs often happen when someone looks at a single concept (like "gravity" or "bacteria") and says, "What if we tried this weird thing?"

The Paper's Goal: The authors built a new test called LiveIdeaBench. Instead of giving the AI a textbook, they give it a single keyword (like "weather forecasting" or "symbiosis") and ask: "Give me a brand new, useful scientific idea based on just this word."

2. The Judges: A Panel of Critics, Not One Teacher

In the past, we might have asked one teacher to grade the ideas. But what if that teacher is biased or tired?

The Solution: The authors created a "jury" of the top 10 smartest AI models available. When one robot generates an idea, a different set of top robots acts as judges.
The Grading Rubric: They don't just look for "correctness." They grade on five creative traits (based on an old psychology theory by Guilford):
- Originality: Is this idea weird and new, or just a copy of something we already know?
- Feasibility: Could a human actually build this, or is it pure magic?
- Clarity: Is the idea explained clearly, or is it a confusing mess?
- Fluency: Can the robot generate many different ideas, or just one?
- Flexibility: Can it handle any topic (from physics to poetry), or does it only work on math?

3. The Big Surprise: "Smart" Doesn't Always Mean "Creative"

This is the most exciting part of the paper.

The Expectation: We usually think the "smartest" robot (the one that wins all the math and logic tests) will also be the most creative.
The Reality: The paper found that being good at math doesn't mean you're good at brainstorming.
- Analogy: Imagine a Formula 1 car (high general intelligence). It's incredibly fast and precise on a track. But if you put it in a muddy field and ask it to find a new path through the trees, it might get stuck. Meanwhile, a rugged off-road truck (a model with lower "general intelligence" scores) might navigate the trees brilliantly.
- The Data: Some models that ranked low on general intelligence tests (like QwQ-32B-preview) performed just as well, or even better, at generating scientific ideas than the "super-smart" models (like Claude-3.7).

4. The "Thinking" Trap: Length $\neq$ Quality

Many people assume that if an AI writes a long, complex explanation with lots of "thinking steps," the idea must be better.

The Finding: The paper analyzed this and found no real connection.
Analogy: It's like a chef writing a 10-page essay about how to make a sandwich. The essay might be long and logical, but the sandwich itself could still be terrible. The paper found that the length of the AI's reasoning didn't predict how good the final idea was. Sometimes the best ideas came from short, punchy responses.

5. Why This Matters

The authors argue that we need to stop treating AI like a giant encyclopedia that just answers questions. To help scientists discover new cures or materials, we need AI that can dream up possibilities.

The Takeaway: We need to train AI differently. Instead of just teaching them to solve puzzles (convergent thinking), we need to teach them to play with concepts and make wild connections (divergent thinking).
The Future: Imagine a future where you have a "Math Bot" to check your calculations and a separate "Dream Bot" to help you come up with the next big hypothesis. They work together, but they need different skills.

Summary in One Sentence

LiveIdeaBench is a new test that proves the smartest AI isn't always the most creative one, showing us that to truly help science, we need to teach our robots how to brainstorm from a blank page, not just memorize the textbook.

1. Problem Statement

Current Large Language Model (LLM) benchmarks primarily evaluate convergent thinking—the ability to derive a single correct answer from rich contextual inputs (e.g., solving math problems, coding tasks, or summarizing provided articles). However, scientific discovery often relies on divergent thinking: the capacity to generate novel, diverse, and feasible hypotheses from minimal cues (e.g., a single keyword).

Existing evaluation methods fail to isolate this specific capability because:

They rely heavily on extensive context, which allows models to synthesize existing knowledge rather than generate new concepts.
They assume a strong correlation between general intelligence (reasoning/math/coding scores) and creative scientific ideation, a hypothesis this paper challenges.
They lack standardized metrics for scientific creativity, which requires balancing originality, feasibility, and clarity simultaneously.

2. Methodology: LiveIdeaBench

The authors introduce LiveIdeaBench, a dynamic benchmark designed to evaluate LLMs' divergent thinking in scientific contexts under "minimal context" constraints.

A. Experimental Design

Input: The benchmark uses 1,180 high-impact scientific keywords spanning 22 distinct domains (e.g., Physics, Biology, Finance, Anthropology). These keywords are updated monthly to prevent data contamination and ensure relevance to current research frontiers.
Prompt Constraint: Models are asked to generate a scientific idea based only on a single keyword, with a strict 100-word limit. This forces the model to rely on internal knowledge representations rather than retrieving and synthesizing provided text.
Evaluation Panel: A dynamic "jury" of the top 10 state-of-the-art LLMs (selected from LiveBench) acts as judges. To ensure robustness and reduce bias:
- Each idea is evaluated by a random subset of 3 judges.
- The panel composition is diverse (max 20% from any single organization).
- Self-evaluation is strictly prohibited (a model cannot judge its own output).

B. Evaluation Dimensions

Drawing from Guilford's theory of divergent thinking, the benchmark assesses five dimensions:

Originality: Novelty and uniqueness of the idea.
Feasibility: Technical implementability and alignment with scientific principles.
Clarity: Coherence and articulation of the idea.
Fluency: The diversity and non-redundancy of ideas generated for the same keyword (measured by comparing multiple outputs from the same model).
Flexibility: The model's ability to maintain performance across different scientific domains (calculated as the 30th percentile of scores across domains to identify the "performance floor").

C. Statistical Analysis

The study employs Pearson and Spearman correlations to analyze relationships between LiveIdeaBench scores and general intelligence benchmarks (LiveBench). It also includes human expert validation on a subset of ideas (Partial Differential Equations) to verify the reliability of the LLM-as-a-judge approach.

3. Key Contributions

Novel Benchmark (LiveIdeaBench): The first comprehensive benchmark specifically targeting scientific idea generation from minimal context, distinguishing it from standard reasoning or knowledge-retrieval tasks.
Decoupling Intelligence and Creativity: The study provides empirical evidence that general intelligence does not strongly predict scientific divergent thinking. High general intelligence scores (e.g., in math or logic) do not guarantee high performance in generating novel scientific hypotheses.
Dynamic Evaluation Framework: Unlike static benchmarks, LiveIdeaBench uses a self-updating panel of judges and a monthly-refreshed keyword database to mitigate overfitting and data contamination.
Comprehensive Dataset: The release of a dataset containing ~7,000 generated ideas, multi-dimensional scores, and reasoning traces for over 40 models.

4. Key Results

Weak Correlation with General Intelligence: There is only a weak positive correlation ( $r \approx 0.357$ $r \approx 0.357$ ) between general intelligence scores (LiveBench) and scientific idea generation scores (LiveIdeaBench).
- Example: QwQ-32B-preview (low general intelligence rank) achieved creative performance comparable to Claude-3.7-Sonnet:Thinking (high general intelligence rank).
- Example: o3-mini-high (high general intelligence) showed mediocre performance in scientific ideation.
Model Architecture & Training Matter More Than Scale:
- Mistral-small (smaller parameter count) outperformed Mistral-large-2411 (123B parameters) in divergent thinking, suggesting that parameter count is not the primary driver of creativity.
- Models within the same family but with different training objectives (e.g., reasoning-focused vs. instruction-focused) showed divergent creative profiles.
Trade-offs Exist: There is a clear Pareto frontier between Originality and Feasibility.
- Claude-3.7-Sonnet:Thinking excelled in Originality but had moderate Feasibility.
- Nova-pro-v1 showed the opposite pattern (high Feasibility, lower Originality).
- DeepSeek-R1 demonstrated a rare balanced performance across all dimensions.
Length vs. Quality: The length of the generated idea (or the reasoning chain) has a statistically significant but negligible correlation with idea quality ( $R^2 \approx 0.008$ ). Longer reasoning does not necessarily equate to better scientific ideas.
Human Validation: Human expert evaluation on Partial Differential Equations (PDE) ideas showed strong alignment with LLM judges for Originality and Clarity, but lower agreement on Feasibility (due to the inherent subjectivity of judging technical viability from text alone).

5. Significance and Implications

Redefining AI for Science: The findings suggest that developing AI for scientific discovery requires specialized training strategies distinct from those used for general problem-solving. Models should be optimized for diverse conceptual associations rather than just convergent accuracy.
Human-AI Hybrid Systems: The results support the creation of hybrid frameworks where different models are paired for specific stages of the scientific process (e.g., using high-originality models for hypothesis generation and high-feasibility models for experimental design).
Benchmarking Evolution: The paper highlights the limitations of using general intelligence benchmarks to predict creative potential, urging the community to adopt domain-specific, dynamic benchmarks for evaluating AI in scientific innovation.
Environmental Awareness: The study quantifies the carbon footprint of the benchmarking process (~3,074 kgCO2eq), highlighting the environmental cost of large-scale LLM evaluation.

In conclusion, LiveIdeaBench establishes that scientific creativity in LLMs is a distinct capability from general reasoning. It provides a critical tool for identifying models capable of genuine scientific ideation, paving the way for more effective AI-assisted scientific discovery.