💻 bioinformatics

Systematic evaluation and benchmarking of text summarization methods for biomedical literature: From word-frequency methods to language models

This paper benchmarks 62 text summarization methods on 1,000 biomedical abstracts, revealing that general-purpose, medium-sized language models outperform both statistical extractive methods and specialized or frontier-scale models in generating accurate and semantically coherent scientific summaries.

Original authors: Baumgärtel, F., Bono, E., Fillinger, L., Galou, L., Keska-Izworska, K., Walter, S., Andorfer, P., Kratochwill, K., Perco, P., Ley, M.

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Baumgärtel, F., Bono, E., Fillinger, L., Galou, L., Keska-Izworska, K., Walter, S., Andorfer, P., Kratochwill, K., Perco, P., Ley, M.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine the world of science as a massive, never-ending library where new books are added every single second. In the field of medicine and biology, this library is growing so fast that even the smartest researchers can't read every new article about drugs, genes, or diseases. They need a way to quickly understand the main points without reading hundreds of pages. This is where "text summarization" comes in. Think of it as a super-powered assistant that reads a long, complicated story and writes a short, clear note on the back of the book telling you exactly what happened. For a long time, computers tried to do this by just counting how often words appeared, like a child highlighting every time they see the word "dog" in a story. But recently, computers have gotten much smarter, using "Large Language Models" (LLMs). These are like digital brains trained on huge amounts of text that can understand context, nuance, and even joke, rather than just counting words. The big question is: when it comes to the tricky, complex language of medical science, which type of computer brain is actually the best at writing these summaries?

A team of researchers decided to put 62 different summarization tools to the test in a giant showdown. They gathered 1,000 real scientific articles from top medical journals and asked every single tool to write a summary of each one. To see who won, they compared the computer's summaries against the "gold standard": the actual highlights written by the human authors of the papers. They didn't just look at how many words matched; they used a complex scoring system to check if the summary sounded natural, if it kept the correct meaning, and most importantly, if it made up any facts (a problem known as "hallucination").

The results were surprising and shifted the usual rules of the game. The researchers found that the "general-purpose" models—those big, broad AI brains trained on everything from cooking recipes to history books—were the clear champions. They outperformed the specialized models that were specifically trained only on medical texts. It turns out that for summarizing science, having a wide, diverse knowledge base is better than having a narrow, deep one. The study suggests that these general models are so good at understanding context that they don't need to be "specialized" to handle medical jargon effectively.

Another interesting twist was the size of the models. The researchers discovered that medium-sized models actually performed better than the massive, super-large ones. It's as if the biggest, heaviest brains were getting a bit clumsy, while the medium-sized ones found the perfect balance between smarts and speed. The specialized medical models, which you might expect to be the experts, often stumbled, sometimes failing to capture the main point or getting confused by the complex language. Meanwhile, the old-school methods that just counted words fell far behind, proving that in the modern era, understanding the story is far more important than just counting the words.

The study also checked to make sure the computers weren't cheating by memorizing the answers from their training data. They found that for the vast majority of the articles, the models hadn't seen them before, so their performance was genuine. When human experts stepped in to grade the top and bottom performers, they agreed with the computer scores: the general-purpose models wrote the most coherent, relevant, and factually accurate summaries.

In the end, this research suggests that if you want a computer to summarize a medical paper, you don't need to hire a specialist robot. Instead, you should use a smart, general-purpose AI that has read a little bit of everything. These broad, flexible models seem to be the most reliable tools for turning complex scientific research into clear, concise knowledge, offering a better balance of quality and efficiency than the specialized alternatives.

Technical Summary: A Systematic Evaluation and Benchmarking of Text Summarization Methods for Biomedical Literature

Problem Statement
The exponential growth of biomedical literature necessitates automated text summarization (ATS) tools capable of condensing research articles into concise, accurate summaries. While ATS methods have evolved from statistical approaches to deep learning models, a systematic comparison spanning the entire spectrum of methods—from traditional word-frequency algorithms to state-of-the-art Large Language Models (LLMs)—has been lacking. Existing benchmarks often focus on limited model sets or abstract-based references, leaving a gap in guidance for selecting appropriate tools for biomedical knowledge discovery.

Methodology
This study presents a systematic evaluation and comparative benchmark of 62 text summarization methods. The evaluation framework adheres to the TRIPOD-LLM reporting guideline and includes the following components:

Dataset: A gold-standard dataset comprising 1,000 biomedical peer-reviewed articles from 20 journals across ScienceDirect and Cell Press. The input consists of titles and abstracts, while the reference summaries are the author-written "highlights" sections, ensuring standardized and credible ground truth.
Models Evaluated: The 62 models were categorized into five groups:
1. Traditional models (frequency-based, TextRank).
2. General-purpose Encoder-Decoder Models (EDMs) (e.g., BART, T5, PEGASUS).
3. Domain-specific EDMs (e.g., PubMed-optimized PEGASUS, BioGPT).
4. General-purpose Small Language Models (SLMs) and LLMs (e.g., Mistral, Llama, GPT, Gemma, Granite).
5. Reasoning-oriented and Domain-specific SLMs/LLMs (e.g., DeepSeek-R1, OpenBioLLM, BioMistral).
  Note: Models were restricted to sizes reproducible on single-GPU setups, excluding exceptionally large models like LLaMA 3.1 405B.
Prompting: All prompt-based models received an identical prompt requesting concise summaries (15–100 words) focusing on main findings, with an explicit instruction to return a specific token if no substantive findings existed.
Evaluation Metrics: A composite suite of metrics was employed, normalized via z-score to compute an Overall Performance Score (OPS):
- Lexical: ROUGE-1, ROUGE-2, ROUGE-L, BLEU, METEOR.
- Semantic: BERTScore (RoBERTa, DeBERTa), all-mpnet-base-v2.
- Factual Consistency: AlignScore, SummaC, MiniCheck (FT5 and 7B variants). These metrics compare generated summaries against the source abstract rather than the reference highlights to detect hallucinations.
Validation:
- Data Leakage Analysis: A two-stage analysis compared article publication dates against model training cutoffs and performed abstract completion probes to rule out memorization of benchmark articles.
- Expert Assessment: Eight biomedical experts evaluated a subset of models (two top-ranked, two bottom-ranked) across coherence, fluency, relevance, and consistency.
- LLM-as-a-Judge: A panel of three independent LLMs (Anthropic, OpenAI, Mistral) evaluated the same subset to assess alignment with human ratings.

Key Results

Overall Performance: General-purpose LLMs achieved the highest overall performance across lexical, semantic, and factual dimensions. The top-ranked model was mistral-small3.2:24b, followed by other Mistral family members. General-purpose SLMs followed closely.
Model Categories:
- General-purpose vs. Domain-specific: Broad pretraining outperformed narrow domain adaptation. Domain-specific models (both SLMs and EDMs) generally performed significantly worse than general-purpose counterparts, suggesting that fine-tuning on biomedical text alone does not necessarily improve summarization quality and may lead to "catastrophic forgetting" of general context.
- Reasoning-oriented Models: Models designed for chain-of-thought reasoning (e.g., DeepSeek-R1) ranked in the middle, indicating that multi-step reasoning is less suited for the semantic compression required in summarization.
- Traditional and EDMs: Classical extractive methods and traditional encoder-decoder architectures (e.g., BART, PEGASUS) lagged behind neural approaches, particularly in semantic and relevance dimensions.
Size vs. Performance: Medium-sized models (e.g., 24B parameters) often outperformed larger proprietary models, suggesting an optimal balance between capacity and efficiency. Some small models (e.g., granite4:tiny-h, mistral:7b) outperformed several larger LLMs.
Factual Consistency: Traditional extractive models ranked highest on factual metrics, likely because their extractive nature inherently aligns with the source text, whereas generative models faced higher risks of hallucination, though top LLMs still maintained high factual scores.
Validation: Expert assessments confirmed the automated rankings, with top Mistral models significantly outperforming bottom-ranked EDMs across all human-rated dimensions. The LLM-as-a-judge panel showed strong agreement with expert consensus (Spearman's $\rho \approx 0.87$ ), though it exhibited a slight self-preference bias for its own model family.
Data Leakage: Analysis confirmed minimal data leakage; abstract completion probes showed no significant difference in similarity scores between pre-cutoff and post-cutoff articles, indicating models were not reproducing memorized benchmark content.

Significance and Claims
The paper claims to provide the first systematic reference for selecting biomedical summarization tools across a comprehensive range of architectures. Its primary contributions are:

Demonstration of Generalization: The study highlights that broad, general-purpose pretraining is more advantageous than narrow domain adaptation for biomedical summarization tasks, challenging the assumption that domain-specific fine-tuning is always superior.
Efficiency-Performance Trade-off: It identifies that medium-sized models often achieve the best balance between computational cost and output quality, making them highly attractive for resource-constrained biomedical research groups.
Benchmarking Framework: The study establishes a robust, multi-dimensional evaluation framework (lexical, semantic, factual) that aligns well with human judgment, offering a reproducible standard for future ATS research.
Practical Guidance: The findings offer actionable recommendations for researchers, suggesting that general-purpose LLMs (particularly medium-sized variants) are currently the most reliable choice for condensing biomedical literature, while cautioning against the uncritical adoption of domain-specific or reasoning-oriented models for this specific task.

The authors maintain a modest stance, acknowledging limitations such as the focus on abstract-to-highlights summarization (which may not generalize to full-text or clinical trial data), the subjectivity of reference summaries, and the snapshot nature of the results given the rapid evolution of LLMs.

More like this