💬 NLP

SciCoQA: Quality Assurance for Scientific Paper--Code Alignment

The paper introduces SciCoQA, a dataset comprising 635 paper-code discrepancies derived from real-world issues and synthetic generation to evaluate and highlight the limitations of large language models in ensuring faithful implementations of scientific publications.

Original authors: Tim Baumgärtner, Iryna Gurevych

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tim Baumgärtner, Iryna Gurevych

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef who just published a famous cookbook. In the book, you describe a "Secret Chocolate Cake" recipe in detail: "Mix flour, sugar, and cocoa. Bake at 350°F for 30 minutes."

Now, imagine a fan tries to bake the cake using your instructions but also looks at the video you posted online showing how you made it. They notice something weird: in the video, you secretly added a cup of espresso and baked it for 45 minutes, even though the book didn't say that.

If the cake turns out amazing, the fan might think, "Great, the book works!" But if the cake turns out terrible, or if another baker tries to follow the book exactly and fails because they missed the secret espresso, you have a reproducibility problem. The "book" (the scientific paper) and the "video" (the computer code) don't match.

This is exactly the problem the paper SCICOQA is trying to solve.

The Big Problem: The "Reproducibility Crisis"

In the world of science (especially AI and computer science), researchers publish papers describing their discoveries. To prove they aren't lying, they also share their computer code. The goal is for anyone else to read the paper, run the code, and get the exact same result.

But often, the code doesn't actually do what the paper says it does.

Maybe the paper says "use a specific math formula," but the code uses a slightly different one.
Maybe the paper forgets to mention a crucial step, but the code includes it.
Maybe the code has a hidden "cheat" that makes the results look better than they really are.

For years, humans had to manually check the code against the paper to find these mismatches. But with millions of papers and code repositories, humans can't keep up. It's like trying to proofread a library of a million books by reading every single word yourself.

The Solution: SCICOQA (The "Code-Book Detective")

The authors created a new tool called SCICOQA. Think of it as a super-smart detective designed specifically to compare a scientific paper (the recipe book) with its code (the video tutorial) to find the differences.

To train this detective, they needed a massive "training manual" full of examples of mismatches. They built this manual in two ways:

Real-World Cases (The "True Crime" Files): They looked at real GitHub issues (where people complain about bugs) and reproducibility reports (where people try to copy a study and fail). They found about 92 real examples where the code and paper didn't match.
Synthetic Cases (The "Simulation" Files): Since real mismatches are rare and hard to find, they used an AI to invent new mismatches. They took real code, made tiny, realistic changes (like swapping a math formula), and created 543 fake mismatches. This is like a driving school using a simulator to teach students how to handle crashes they haven't actually experienced yet.

How They Tested the Detectives

The authors took 22 of the world's most advanced AI models (like GPT-5, Gemini, and others) and asked them to act as the "Code-Book Detective." They gave the AI a paper and its code and asked: "What is different between these two?"

The Results: The AI is Good, But Not Good Enough
The results were a bit of a wake-up call:

The Best AI: The top-performing models (Gemini 3.1 Pro and GPT-5 Mini) could only find about 47% of the real mismatches.
The Missed Clues: The AI was great at spotting obvious differences (like "The code uses a red button, but the paper says blue"). However, it was terrible at spotting missing information. If the paper forgot to mention a step, but the code included it, the AI often missed it.
The "Long Story" Problem: Scientific papers and code can be huge (thousands of pages of text). When the AI had to read a very long document, its performance dropped significantly. It's like trying to remember every detail of a 10-hour movie after watching it once; you start forgetting the middle parts.

Why This Matters

The authors argue that as science gets faster and more automated (with "AI Scientists" writing their own papers and code), we need a way to automatically check if the AI is being honest.

Currently, we can't fully trust AI to be the final judge of scientific truth. If an AI writes a paper and code, and we don't have a tool that can reliably spot the differences, we might end up with a lot of "fake" science that looks real but doesn't work.

The Takeaway

SCICOQA is a new benchmark (a test) that shows us how far we have to go. It proves that while AI is getting smarter, it still struggles to be a perfect "fact-checker" for science. It's not ready to replace human reviewers yet, but it's a crucial first step toward building a future where science is transparent, reliable, and reproducible.

In short: We built a test to see if AI can catch scientists (or other AIs) when their code doesn't match their story. The AI is trying hard, but it's still missing about half the clues. We need to teach it better before we let it run the whole science lab.

1. Problem Statement

The scientific community faces a "reproducibility crisis," where the code accompanying scientific publications often fails to faithfully implement the methods described in the text. These paper-code discrepancies manifest as:

Differences: The code implements logic distinct from the paper (e.g., using L1 norm instead of L2 norm).
Paper Omissions: The code includes critical components missing from the text.
Code Omissions: Steps described in the paper are absent from the repository.

Current manual review processes are insufficient due to the volume of submissions and the complexity of code. Furthermore, the rise of "AI Scientists" (autonomous agents generating papers and code) necessitates automated tools to verify the faithfulness of generated outputs. Existing benchmarks focus either on errors within the text or local code-comment inconsistencies, lacking a dataset for global alignment between dense scientific papers and multi-file code repositories.

2. Methodology: SCICOQA Dataset Construction

The authors introduce SCICOQA, a dataset of 635 paper-code discrepancies designed to evaluate Large Language Models (LLMs) on this alignment task. The dataset construction involves a hybrid approach of real-world data collection and synthetic generation.

A. Data Sources

Real-World Data (92 instances):
- GitHub Issues: Crawled repositories linking to papers (2020–2025). Issues were filtered using Qwen3 and manually verified to ensure they reported specific paper-code mismatches.
- Reproducibility Papers: Extracted from ML Reproducibility Challenges and conference reproducibility tracks. Discrepancies were extracted using GPT-5 and manually verified.
- Domains: Primarily Computer Science (AI, ML, CV, NLP).
Synthetic Data (543 instances):
- To scale the dataset and cover non-CS domains (Physics, Quantitative Biology, Math, EE), the authors used GPT-5 to generate code modifications in real scientific repositories.
- Generation Process: The model was prompted to introduce "small, conceptually meaningful" changes that create a semantic conflict between the paper and code (e.g., altering a loss function or adding an undocumented layer) without introducing simple bugs or hyperparameter mismatches.
- Domains: Covers CS, Electrical Engineering, Physics, Statistics, Quantitative Biology, and Mathematics.

B. Discrepancy Taxonomy

The dataset categorizes discrepancies by:

Type: Difference, Paper Omission, Code Omission.
Category: Algorithm, Model, Loss, Evaluation, Data, Training.

C. Evaluation Framework

Task: Given a paper (text) and its codebase, the model must generate a list of discrepancies.
Judging: An LLM-as-a-Judge (GPT-OSS 20B) evaluates whether predicted discrepancies match the ground truth.
Metrics: Primarily Recall (detecting all annotated discrepancies), as precision is less critical for initial screening, though Precision-Recall trade-offs are analyzed.

3. Key Contributions

SCICOQA Dataset: The first large-scale benchmark specifically for detecting semantic misalignments between scientific papers and their code implementations, covering 635 instances across multiple scientific domains.
Synthetic Data Pipeline: A robust method for generating realistic paper-code discrepancies in domains where real-world examples are scarce, validated by a strong correlation ( $r=0.94$ ) between synthetic and real-world model performance.
Comprehensive Analysis: A detailed taxonomy of discrepancy types and categories, providing a structured understanding of how implementations diverge from theory.
Benchmarking State-of-the-Art: Evaluation of 22 LLMs (including GPT-5, Gemini 3.1 Pro, GPT-OSS, Qwen3, and DeepSeek) on long-context, cross-modal reasoning tasks.

4. Results

The evaluation reveals that current LLMs struggle significantly with this task, highlighting a gap in automated scientific quality assurance.

Overall Performance: The best-performing models (Gemini 3.1 Pro and GPT-5 Mini) achieved a recall of only 46.7% on real-world data. This means they miss more than half of the actual discrepancies.
Synthetic vs. Real: There is a strong correlation ( $r=0.94$ ) between performance on synthetic and real data, validating the synthetic subset as a reliable proxy for scaling.
Discrepancy Difficulty:
- Code Omissions are the easiest to detect.
- Differences are moderately difficult.
- Paper Omissions (where code has features not mentioned in the paper) are the most challenging, as models must distinguish between valid engineering artifacts and undocumented scientific logic without a textual reference.
Context Length: Performance degrades as input token count increases. Models struggle significantly with inputs exceeding 100k tokens (median combined prompt size is ~57k tokens).
Ablation (Code Only): Removing the paper from the input causes a massive drop in performance (average recall drop of ~19.2 percentage points), confirming that cross-modal reasoning is essential and cannot be replaced by code-only analysis or parametric knowledge.
Domain Bias: Models perform worse on recent (2025) data, suggesting they rely on pre-training data contamination rather than true reasoning.

5. Significance and Implications

Limitations of Current AI: The results demonstrate that even the most advanced LLMs cannot yet serve as autonomous arbiters of scientific validity. Their low recall makes them unreliable for ensuring reproducibility without human oversight.
Future of Science: As "AI Scientists" become more prevalent, tools like SCICOQA are critical for verifying that generated code faithfully implements the scientific method.
Quality Assurance: The paper argues for the integration of automated discrepancy detection into peer review and reproducibility workflows to scale scientific integrity.
Call to Action: The authors encourage the broader adoption of reproducibility tracks to systematically capture real-world discrepancies and expand the dataset beyond synthetic generation.

In conclusion, SCICOQA exposes a critical bottleneck in the automation of scientific verification. While LLMs show promise in precision, their inability to consistently recall all discrepancies indicates that human-in-the-loop verification remains essential for the foreseeable future.