💬 NLP

IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs

The paper introduces ISOSCI, a benchmark of isomorphic cross-domain science problems demonstrating that most reasoning improvements in large language models are actually driven by domain knowledge retrieval rather than structural reasoning capabilities, thereby challenging the assumption that chain-of-thought reasoning inherently enhances short-horizon scientific problem-solving.

Original authors: Samir Abdaljalil, Erchin Serpedin, Hasan Kurban

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Samir Abdaljalil, Erchin Serpedin, Hasan Kurban

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a student is smart because they are good at logic, or just because they have a really good memory.

Usually, when we test AI models (like the ones that power chatbots) on science questions, we can't tell the difference. If the AI gets a chemistry problem right, did it use clever reasoning to solve it? Or did it just remember the answer from its training data?

The authors of this paper, ISOSCI, created a special test to solve this mystery. Here is how they did it, explained simply:

The "Twin" Test (Isomorphic Pairs)

The researchers created pairs of "twin" problems. These twins are identical in their structure (the steps you have to take to solve them) but completely different in their content (the specific facts you need to know).

Think of it like two different recipes:

Recipe A (Physics): "Take 2 cups of flour, add 1 egg, and bake at 350 degrees."
Recipe B (Chemistry): "Take 2 moles of gas, add 1 constant, and calculate the pressure."

Both recipes require the exact same logic: Take a number, multiply it by a constant, and get a result. But Recipe A needs you to know about flour and eggs, while Recipe B needs you to know about gas laws.

If an AI is truly "reasoning," it should be able to solve both twins equally well because the logic is the same. If it only solves one, it's just relying on its memory of that specific subject.

The Big Discovery: Memory vs. Logic

The researchers tested several top-tier AI models using these twin pairs. They turned a "reasoning mode" switch on and off to see if it helped.

Here is what they found, using a simple analogy:

The "Flashlight" Analogy
Imagine the AI is in a dark room trying to find a specific tool to fix a machine.

Reasoning Mode is like turning on a bright flashlight.
Knowledge is the tool sitting on the shelf.

The paper found that for short, standard science problems, turning on the flashlight (reasoning) didn't help the AI find the tool any faster. The AI only got better at solving the problem if it already knew where the tool was (the specific science fact).

In fact, 91.3% of the time the AI got better at solving a problem, it was because it remembered the specific fact, not because it got better at the logical steps.

The "O3-Mini" Surprise

One of the most interesting parts of the paper involves a specific AI model called o3-mini.

On a famous test called GPQA (which has very hard, deep conceptual questions), o3-mini was a superstar, beating other models by a huge margin. People thought, "Wow, this model is a reasoning genius!"
But when the researchers put o3-mini on their new ISOSCI test (the twin logic test), it actually performed worse than a standard model.

The Lesson: The "genius" label depends entirely on which test you give the AI. If the test requires deep, abstract thinking, the reasoning model shines. If the test requires recalling specific facts and plugging them into a formula, the reasoning model doesn't help much—and might even get in the way.

The "Toggle" Experiment

The researchers also tested models where they could literally flip a switch to turn "reasoning" on or off without changing the model itself.

Result: Flipping the switch made almost no difference (less than 5% improvement) on these short science problems.
It's like giving a calculator a "super-calculation" mode, but the problems are so simple that the calculator doesn't need the extra power. It just needs to know the numbers.

Summary

The paper concludes that for short, step-by-step science problems:

Reasoning isn't magic: It doesn't automatically make an AI smarter at logic.
It's mostly about memory: When AI seems to "reason" better, it's usually just because it successfully retrieved a specific fact it needed.
One size doesn't fit all: A model that looks like a reasoning expert on one test might look average on another, depending on whether the test needs deep thinking or just good memory.

The authors released their "Twin Test" (ISOSCI) so others can use it to stop guessing whether an AI is actually reasoning or just memorizing.

Technical Summary: ISOSCI

Problem Statement
Recent advances in Large Language Models (LLMs) have highlighted "reasoning" as a primary driver of performance on complex scientific tasks, often validated by benchmarks like GPQA, SciBench, and MMLU-STEM. However, these benchmarks conflate two distinct capabilities: the retrieval of domain-specific knowledge and the application of logical reasoning procedures. When a model fails a science problem, it is ambiguous whether the failure stems from a lack of factual recall or an inability to execute the necessary reasoning steps. Consequently, it remains unclear whether reasoning mechanisms (e.g., chain-of-thought prompting, reasoning-specific training) genuinely improve logical inference or primarily serve to surface relevant domain knowledge. Existing benchmarks lack the structural controls to disentangle these factors.

Methodology: The ISOSCI Benchmark
To address this, the authors introduce ISOSCI, a benchmark of isomorphic cross-domain science problem pairs. The core design principle is to hold the reasoning structure constant while varying the domain knowledge required.

Isomorphic Pairs: Each problem $q$ $q$ is paired with a counterpart $q'$ $q^{'}$ from a different scientific domain (Physics, Chemistry, Biology, or Earth Science). The pairs satisfy four conditions:
1. They belong to different domains.
2. They share the exact same logical structure type (e.g., formula recall, unit conversion, conservation law application).
3. The solution procedure for $q'$ is a bijection of the procedure for $q$ , replacing domain-specific atoms (formulas, constants) with their counterparts in the new domain.
4. The knowledge sets required for each problem are disjoint.
Dataset Construction: The benchmark consists of 144 pairs (288 problems) derived from seed problems in GPQA, SciBench, and MMLU-STEM, with synthetic generation for Earth Science. A three-stage pipeline (Seed collection, Partner generation, Automated verification) ensures structural equivalence and knowledge independence. Verification involves a panel of LLM judges and human expert audits, achieving high agreement with human consensus.
Evaluation Protocol: The study evaluates five model pairs across four model families (OpenAI, Google, Qwen, DeepSeek). Comparisons include "traditional" pairs (distinct models with different training objectives) and "toggle" pairs (the same model with reasoning enabled vs. disabled). Models are evaluated on ISOSCI and three other benchmarks (GPQA, MMLU-STEM, SciBench).
Decoupling Metric ( $p_{know}$ ): The authors propose a metric to attribute reasoning-mode gains. A gain is knowledge-dependent if the reasoning configuration improves performance on one member of an isomorphic pair but not the other. A gain is structure-invariant if it improves performance on both. The metric $p_{know}$ calculates the ratio of knowledge-dependent gains to total gains.

Key Contributions

Construction Methodology: A scalable framework for generating isomorphic cross-domain problem pairs that isolate reasoning structure from domain knowledge.
The $p_{know}$ Metric: A quantitative method to decompose reasoning gains into knowledge-dependent and structure-invariant components.
ISOSCI Benchmark: A publicly released dataset of 144 isomorphic pairs spanning four scientific domains, accompanied by empirical findings on the nature of reasoning gains.

Results

Dominance of Knowledge Dependence: Across five model pairs and 69 observed reasoning-mode gains, 91.3% (63/69) were knowledge-dependent rather than structure-invariant. This indicates that when reasoning modes improve performance, they do so by helping models retrieve or apply specific domain facts, not by improving the execution of the logical procedure itself.
Minimal Toggle Benefits: For high-capability models, enabling reasoning toggles provided negligible accuracy gains (less than 5 percentage points) across domains. For the Gemini 2.0 Flash model, the toggle had no statistically significant effect.
Benchmark-Dependent Conclusions: The study highlights a stark reversal in model performance based on the benchmark. The o3-mini model, which outperforms GPT-4o-mini by +19.2pp on GPQA Diamond (a benchmark favoring deep conceptual reasoning), underperforms GPT-4o-mini by −24.7pp on ISOSCI. This demonstrates that conclusions about the utility of reasoning models are entirely dependent on the benchmark's design.
Nature of Reasoning Gains: The data suggests that on short-horizon procedural science tasks, reasoning mechanisms function primarily as extended knowledge retrieval, increasing the probability that relevant facts are surfaced during generation, rather than enhancing logical inference capabilities.

Significance and Claims
The paper claims that the assumption that chain-of-thought reasoning inherently improves short-horizon procedural scientific problem-solving is challenged by these findings. The authors argue that for the problem types covered by ISOSCI, reasoning improvements are largely a proxy for better knowledge utilization. The study asserts that no single benchmark provides a complete picture of scientific capability; a model strong on conceptual derivation (GPQA) may be weak on structured knowledge retrieval (ISOSCI). The authors position ISOSCI as a diagnostic tool to determine where and how a model's scientific capabilities break down, distinguishing between knowledge gaps and reasoning deficits. They conclude that while the isomorphic-pair methodology and $p_{know}$ metric support more precise evaluation, the current findings are specific to short-horizon procedural problems, leaving the applicability to long-horizon or open-ended reasoning as an open question.