💬 NLP

MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts

The paper introduces MedConclusion, a large-scale dataset of 5.7 million structured PubMed abstracts designed to benchmark and advance the ability of large language models to generate scientific conclusions from biomedical evidence, while highlighting the distinct challenges of conclusion generation compared to summarization.

Original authors: Weiyue Li, Ruizhi Qian, Yi Li, Yongce Li, Yunfan Long, Jiahui Cai, Yan Luo, Mengyu Wang

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weiyue Li, Ruizhi Qian, Yi Li, Yongce Li, Yunfan Long, Jiahui Cai, Yan Luo, Mengyu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. You have a pile of clues: the Background (what we knew before), the Methods (how you investigated), and the Results (what you found). Your job is to write the final report: the Conclusion.

For a long time, we've been teaching AI (Large Language Models) to be great detectives. But we haven't had a good way to test if they can actually deduce the final answer just from the clues, without just guessing or making things up.

This paper introduces MedConclusion, a massive new training ground and testing kit for AI, specifically for the world of medicine.

Here is the breakdown in simple terms:

1. The Massive Library (The Dataset)

Think of the authors as librarians who went into the world's biggest medical library (PubMed) and pulled out 5.7 million research papers.

The Trick: They took every paper and cut out the "Conclusion" section.
The Game: They gave the AI the rest of the paper (Background, Methods, Results) and asked, "Based only on this, what is the conclusion?"
The Answer Key: They kept the original conclusion written by the human scientists to check if the AI got it right.

It's like giving a student a math problem with all the steps shown, but hiding the final answer, and then grading them on whether they can figure out that final number.

2. The "Summary" vs. "Conclusion" Trap

One of the biggest discoveries in the paper is that summarizing and concluding are two very different brain muscles.

Summary: Imagine you are a news reporter. You just want to tell the story of what happened. "We looked at 100 people, found X, and Y." It's a retelling.
Conclusion: Imagine you are a judge. You have to look at the evidence and make a specific, decisive ruling. "Because of X and Y, we know for sure that Z is true."

The paper found that AI models are great at being reporters (summarizing), but they often struggle to be judges (concluding). When asked to write a conclusion, they sometimes just write a summary instead, missing the specific "verdict" the human author intended.

3. The Grading Problem (The "Judge" Issue)

How do you grade an AI's conclusion?

The Old Way: You check if the AI used the same words as the human. (e.g., Did they both use the word "significant"?) This is like grading an essay just by counting how many times the student used the word "the." It doesn't tell you if the essay makes sense.
The New Way: The authors used other AIs as "judges." These judges read the AI's conclusion and the human's conclusion and gave scores on things like:
- Did they mean the same thing?
- Did they sound like a scientist?
- Did they mess up the numbers?
- Did they contradict the evidence?

The Surprise: The paper found that which AI you use as the judge changes the score! If you use AI "A" to grade the work, the AI might get an A+. If you use AI "B," it might get a C. It's like having two different teachers grade the same essay and giving it totally different grades. This means we need to be very careful about how we measure success.

4. The "Prestige" Factor

The authors also looked at whether the "status" of the journal (how famous or expensive it is) made the task harder or easier.

They found that papers from "fancy" journals were slightly easier for AI to mimic in terms of word choice and style.
However, the logic and accuracy didn't get easier just because the journal was famous. A difficult medical problem is hard to solve, no matter how fancy the paper looks.

Why Does This Matter?

Right now, we are trying to use AI to help doctors and scientists. If an AI can read a study and tell a doctor, "Based on this evidence, the drug works," that's huge. But if the AI just makes up a conclusion or confuses a summary with a verdict, it could be dangerous.

MedConclusion is a new, giant "driver's license test" for medical AI. It forces the AI to prove it can look at the evidence and draw the right logical line, rather than just repeating what it heard. It helps us understand where AI is ready to help us and where it still needs to go to school.

In short: We built a giant gym for AI to practice its "medical logic" muscles, and we found out that while the AI is getting stronger, it still confuses "telling a story" with "making a verdict," and we need better ways to grade its homework.

1. Problem Statement

Large Language Models (LLMs) have demonstrated strong reasoning capabilities, yet there is a lack of large-scale resources to evaluate their ability to infer scientific conclusions from structured evidence. Existing biomedical datasets are often limited in scope (focusing on specific study types like RCTs or echocardiography notes), lack journal-level metadata for subgroup analysis, or treat conclusion generation merely as a proxy for premise-conclusion alignment rather than a standalone reasoning task. Furthermore, evaluating open-ended scientific reasoning is challenging due to the inadequacy of traditional lexical overlap metrics (e.g., ROUGE) and the sensitivity of LLM-as-a-judge evaluations to the identity of the judging model.

2. Methodology

Dataset Construction (MedConclusion)

The authors curated MedConclusion, a large-scale dataset comprising 5.7 million structured PubMed abstracts published between 2000 and 2025.

Structure: Each instance consists of the non-conclusion sections of an abstract (Background, Methods, Results) as the input ( $x$ ) and the original author-written conclusion as the ground-truth target ( $y^*$ ).
Metadata: The dataset includes journal-level metadata, specifically biomedical subject categories (141 categories) and SJR (SCImago Journal Rank) scores, allowing for analysis across different subfields and venue prestige levels.
Curation Pipeline: The pipeline involved querying PubMed for structured abstracts, parsing XML records, filtering for English language and valid conclusion sections, and deduplicating based on PMID, DOI, and normalized titles.

Experimental Setup

The study evaluated diverse LLMs (including frontier closed-source models, open-source instruction-tuned models, reasoning models, and multimodal models) under four prompting modes:

Mode A: Generate a formal academic conclusion (unconstrained).
Mode B: Generate a formal academic summary (unconstrained).
Mode C: Generate a conclusion with explicit constraints on sentence/word count and style matching.
Mode D: Generate a summary with explicit constraints.

Evaluation Protocol

The authors employed a hybrid evaluation strategy to address the limitations of single-metric approaches:

LLM-as-a-Judge: A multi-dimensional scoring system (0–100) assessing:
- Semantic Similarity
- Writing Style Similarity
- Non-Contradiction Rate
- Numeric Consistency
- Formality Similarity
- Robustness Check: Scores were compared using two different judge backbones (GPT-5.4-mini and Gemini 3 Flash) to measure calibration shifts.
Reference-Based Metrics: Lightweight diagnostics including Word/Sentence Count Ratios, Embedding Cosine Similarity, ROUGE-1/2/L, BLEU, and Perplexity.

3. Key Contributions

MedConclusion Dataset: The introduction of the first large-scale (5.7M examples) benchmark specifically for biomedical conclusion generation, enriched with journal metadata for granular analysis.
Discourse Distinction: Empirical evidence demonstrating that conclusion generation is behaviorally distinct from summary generation, even when models are given the same source text.
Evaluation Insights: A comprehensive analysis showing that current automatic metrics struggle to separate top-tier models (score compression) and that absolute scores are highly sensitive to the identity of the judging LLM.
Subfield Analysis: Discovery that task difficulty varies by biomedical category and that lexical overlap (ROUGE) is a poor proxy for overall conclusion quality compared to semantic and stylistic alignment.

4. Key Results

Model Performance

Top Performers: GPT-5.4 emerged as the strongest model across all judge dimensions, followed closely by Gemini 3.1 Pro, Gemini 3 Flash, and DeepSeek-V3.2.
Score Compression: Strong models were closely clustered in performance under automatic metrics, suggesting that current benchmarks struggle to differentiate between frontier models in this specific task.
Reasoning Models: Dedicated reasoning models (e.g., DeepSeek-R1) did not necessarily outperform general-purpose models, and in some cases (like writing style similarity), performed significantly worse.

Conclusion vs. Summary

Behavioral Distinction: When prompted to write a summary (Mode B) instead of a conclusion (Mode A), models maintained high semantic similarity but suffered drastic drops in writing style similarity (approx. -8 points) and numeric consistency (approx. -22 to -28 points).
Implication: This confirms that "conclusion writing" is a specific discourse function requiring the selection of specific details and numeric qualifiers, distinct from general summarization.

Evaluation Sensitivity

Judge Identity: Switching the judge from GPT-5.4-mini to Gemini 3 Flash caused significant shifts in absolute scores (e.g., semantic similarity increased by ~11 points for the same outputs), though model rankings remained relatively stable.
Metric Divergence: High lexical overlap (ROUGE) did not correlate with high semantic or stylistic quality. For instance, some categories had high ROUGE-L scores but low semantic similarity, indicating that surface-level n-gram matching fails to capture rhetorical structure.

Subgroup Analysis

SJR Correlation: Higher journal prestige (SJR) showed a weak positive correlation with lexical and stylistic metrics but no significant correlation with factual consistency (non-contradiction) or numeric consistency.
Category Difficulty: Interdisciplinary, non-clinical fields (e.g., Software, Computer Science Applications) were consistently the hardest for conclusion generation, exhibiting irregular performance profiles across different metrics.

5. Significance

MedConclusion provides a critical resource for advancing scientific reasoning in AI. By isolating the "evidence-to-conclusion" reasoning step, it moves beyond simple retrieval or summarization tasks. The findings highlight that:

Scientific reasoning requires specific discourse training: Models cannot simply summarize; they must learn the specific rhetorical constraints of scientific conclusions.
Evaluation requires nuance: Relying solely on ROUGE or a single LLM judge is insufficient. A hybrid approach combining reference-based metrics with multi-dimensional LLM judging is necessary to capture the complexity of scientific writing.
Future Directions: The dataset enables future research into how LLMs handle scientific uncertainty, numeric precision, and domain-specific rhetorical structures, serving as a reusable foundation for improving AI in biomedical discovery.

Code and Data Availability: The dataset and code are available at https://github.com/Harvard-AI-and-Robotics-Lab/MedConclusion.