🤖 AI

LLM Jaggedness Unlocks Scientific Creativity

This paper introduces the SciAidanBench benchmark to demonstrate that the uneven, "jagged" progression of large language model capabilities across scientific domains can be strategically leveraged through ensemble methods to significantly enhance AI-driven scientific creativity beyond the limits of any single model.

Original authors: Shray Mathur, J. Anibal Boscoboinik, Esther H. R. Tsai, Kevin G. Yager

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shray Mathur, J. Anibal Boscoboinik, Esther H. R. Tsai, Kevin G. Yager

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the ultimate "Idea Machine" to help scientists come up with new experiments and discoveries. You might assume that as these machines (AI models) get bigger and smarter, they get better at everything in a smooth, predictable way.

This paper argues that's not how it works. Instead, AI progress is "jagged."

Think of an AI model not as a smooth, flat road, but as a rocky mountain range. Some parts of the mountain are towering peaks where the AI is a genius; other parts are deep valleys where it stumbles and fails. Sometimes, making the mountain taller (increasing the model size) doesn't help the valleys; it might even make the cliffs steeper.

Here is a breakdown of what the researchers found, using simple analogies:

1. The "Jagged" Landscape

The researchers built a test called SciAidanBench. Imagine a giant box of 155 open-ended science questions, ranging from "How do we detect a new force of nature?" to "How do we deliver medicine to specific cells?"

They asked 30 different AI models to answer these questions. They didn't just want one right answer; they wanted the models to generate as many unique and coherent ideas as possible.

The Discovery:

General vs. Science: An AI that is great at general creative writing (like writing a story) isn't necessarily great at scientific creativity. It's like a chef who is amazing at baking cakes but terrible at grilling steaks. The skills don't transfer smoothly.
The "Burst" Effect: Even the smartest AI models are inconsistent. On some questions, they have a "burst" of genius and generate 50 great ideas. On the next question, they might only generate 2. Stronger models don't just do more of everything; they just have wider swings between "okay" and "amazing."
The "Spiky" Expert: Within a single model, the AI might be a genius at Physics but struggle with Biology. Its knowledge isn't a smooth blanket; it's a collection of sharp spikes.

2. Turning the "Jaggedness" into a Superpower

Usually, people think these uneven skills are a bug. The researchers say: No, it's a feature.

If you have a team of people where everyone is good at different things, you can build a super-team. The researchers tried three ways to combine these "jagged" AI models to create a Meta-Model (a team of AIs working together):

Thinking Longer (Inference-Time Compute):
- The Analogy: Imagine asking a student to solve a math problem. If you let them just blurt out the first answer, they might be wrong. If you tell them, "Take 5 minutes to think, write down your steps, and check your work," they get much better.
- The Result: When the AI was allowed to "think" longer (generate more internal reasoning steps before answering), it produced significantly more creative scientific ideas.
Pooling Knowledge (The Router):
- The Analogy: Imagine a hospital. You don't ask the heart surgeon to fix a broken leg. You route the patient to the specialist.
- The Result: The researchers built a system that looked at the question. If it was about Physics, it sent the question to the AI that was best at Physics. If it was about Biology, it sent it to the Biology expert. This "Router" team beat any single AI model because it always used the right expert for the job.
Brainstorming Together:
- The Analogy: Imagine a group of artists in a room. Artist A paints a line. Artist B sees that line and adds a shape. Artist C sees the shape and adds color. They build on each other's work.
- The Result: The researchers let five different AI models take turns adding ideas to a single list. One model would suggest an idea, and the next model would read it and try to improve or expand on it. This "chain reaction" of ideas created a much richer set of solutions than any single model could do alone.

3. The Ultimate Team (Top-5-Parallel)

The researchers combined all three methods. They took the top five AI models, let them think deeply, and had them work together in a brainstorming session where they could see each other's ideas.

The Result: This "Super-Team" outperformed every single AI model on its own. It didn't just do a little bit better; it smoothed out the jagged edges. Where one model had a "valley" (a weak spot), another model had a "peak," and together they covered the whole mountain.

The Bottom Line

The paper concludes that we shouldn't just try to build one "perfect" AI that is good at everything. Instead, we should embrace the fact that AIs are uneven. By recognizing their specific strengths and weaknesses, and by having them work together in teams, we can unlock a level of scientific creativity that no single machine could ever achieve alone.

In short: AI isn't a smooth, perfect machine. It's a jagged, rocky landscape. But if you know how to navigate the rocks and bring the right climbers together, you can reach the highest peaks.

Technical Summary: LLM Jaggedness Unlocks Scientific Creativity

Problem Statement

As artificial intelligence advances, model capabilities do not improve uniformly across tasks, domains, or scales. Instead, progress follows a "jagged frontier," where competence is uneven and irregular. While previous literature has identified jaggedness across tasks, model scales, reasoning depths, and problem complexities, a systematic characterization of this phenomenon specifically within the context of scientific idea generation remains underexplored. The authors posit that scientific creativity, being open-ended and generative, offers an ideal lens to reveal these capability disparities. Furthermore, while prior efforts to improve LLM diversity often rely on computationally intensive training interventions or structured multi-LLM workflows that limit open-ended generation, there is a need to understand how to harness, rather than merely smooth out, these jagged edges to enhance AI-driven scientific discovery.

Methodology

To investigate this, the authors introduce SciAidanBench, a benchmark dataset comprising 155 open-ended scientific questions curated by researchers at Brookhaven National Laboratory. The questions span six scientific domains: Physics (subdivided into Fundamental, Astrophysics, Synchrotron, and Condensed Matter), Chemistry, Biology, Neuroscience, Nanoscience, and Environmental Science. The questions are phrased to encourage concept generation (e.g., "Propose a method for...") rather than factual retrieval.

The study evaluates 30 variants of 19 base models from 8 providers (including OpenAI, Anthropic, Google, and local models like LLaMA 3.3 and Mistral). The evaluation protocol follows the AidanBench methodology:

Iterative Generation: For each question, a model is prompted to generate an initial idea ( $r_1$ ) and then iteratively re-prompted to generate subsequent ideas ( $r_2, r_3, \dots$ ) while maintaining context of previous outputs.
Dual-Constraint Evaluation: Each new idea must be both coherent (logically consistent and answering the prompt) and novel (distinct from all prior responses).
Scoring Metrics:
- Embedding-based Novelty: Cosine distance between the new response and previous responses using a pretrained embedding model.
- LLM-based Novelty: A judge LLM (o1-mini) evaluates similarity scores against previous responses.
- Coherence: A judge LLM rates logical consistency on a 0–100 scale.
- Generation stops when a response falls below novelty (<0.15) or coherence (<15) thresholds. The total count of valid responses serves as the proxy for creative potential.

Key Contributions

SciAidanBench: The first benchmark specifically designed to measure the scientific creativity of LLMs via sustained, open-ended idea generation across diverse scientific subfields.
Characterization of Jaggedness: A systematic demonstration that jaggedness manifests at three distinct levels:
- Cross-Task: General creativity improvements do not uniformly translate to scientific creativity.
- Prompt-Level: Stronger models exhibit high variability, showing "bursts" of creativity on specific questions while struggling with others.
- Domain-Level: Individual models display fragmented internal profiles, with uneven strengths across scientific subfields.
Meta-Model Ensembles: The proposal and evaluation of ensemble strategies that treat jaggedness as a resource. The authors introduce mechanisms to combine models:
- Inference-Time Compute: Leveraging reasoning tokens to expand the solution space.
- Knowledge Pooling: Routing questions to models with specific domain strengths.
- Brainstorming: Enabling cross-model idea refinement where models build upon each other's outputs.
- Top-5-Parallel: A configuration integrating all three mechanisms.

Results

Cross-Task Divergence: While general and scientific creativity are positively correlated ( $r=0.76$ ), the relationship is not uniform. Models ranking high in general creativity do not necessarily rank high in scientific creativity. For instance, OpenAI models showed strong general creativity but plateaued in scientific tasks, whereas Anthropic models outperformed in scientific creativity relative to their general performance.
Prompt-Level Variability: Stronger models (e.g., o3-high, Gemini-2.5-Pro) exhibit wider response-count distributions, indicating that their higher average scores are driven by "bursts" of high creativity on specific prompts rather than uniform improvement. Weaker models show consistently low, narrow distributions.
Domain Fragmentation: Top-performing models show distinct spikes in performance across subfields. For example, Gemini-2.5-Pro excelled in Condensed Matter Physics but was weaker in Nanoscience, while Claude-3.7-Sonnet-Thinking-16k was strongest in Neuroscience but weaker in Fundamental Physics.
Ensemble Superiority:
- Inference-Time Compute: Reasoning models (e.g., Claude-3.7-Sonnet-Thinking) significantly outperformed non-reasoning variants, with performance tracking actual token consumption rather than allocated budget.
- Knowledge Pooling: A "Router" ensemble, which directed questions to the best-performing model for each specific domain, surpassed the strongest individual model in every domain.
- Brainstorming: A "Top-5" ensemble (sampling from the five best individual models) outperformed all individual models. However, a "Top-5 Vendors" ensemble (diversifying by provider but not necessarily by capability) failed to outperform the strongest individual model, suggesting that capability level is more critical than vendor diversity alone.
- Top-5-Parallel: The ensemble combining inference-time compute, knowledge pooling, and brainstorming achieved the highest scores on both general and scientific creativity benchmarks, outperforming any single model or single-mechanism ensemble.

Significance and Claims

The paper argues that jaggedness is not merely a limitation but a structural resource of AI progress. Rather than viewing uneven capability profiles as defects to be smoothed away through scaling or alignment, the authors demonstrate that these irregularities can be leveraged. By orchestrating heterogeneous models through meta-model ensembles, it is possible to transform fragmentation into synergy, amplifying scientific idea generation beyond the limits of any single model.

The authors conclude that future gains in open-ended scientific idea generation may increasingly arise from systems that explicitly recognize, measure, and orchestrate jagged intelligence. They suggest that future research should explore deliberate prompt perturbations (e.g., cross-domain seeding) to access underused creative modes and investigate optimal team structures (hierarchical vs. horizontal) for collaborative ensembles. The work positions SciAidanBench as both a benchmark for scientific ideation and a framework for revealing and harnessing structural heterogeneity in model capabilities.