Performance Evaluation of Open-Source Large Language Models for Assisting Pathology Report Writing in Japanese

This study evaluates seven open-source large language models for assisting Japanese pathology report writing, finding that while thinking and medical-specialized models excel in structured reporting and typo correction, their utility varies by task and rater preference, suggesting they are beneficial in specific, clinically relevant scenarios.

The Gist

Imagine a busy hospital where pathologists are like detectives. Their job is to examine tiny tissue samples under a microscope and write detailed reports explaining what's wrong with a patient. These reports are crucial, but they are also tedious, full of strict formatting rules, and prone to tiny typos that can cause big confusion.

This paper is like a taste test for seven different "AI assistants" (Open-Source Large Language Models) to see which ones are good enough to help these detectives write their reports in Japanese.

Here is the breakdown of the experiment using simple analogies:

1. The Contestants (The AI Models)

The researchers didn't just pick random robots. They picked seven specific "brains" available in early 2026.

• The Generalists: Some are like smart, all-purpose students (e.g., Gemma, Qwen).
• The Specialists: Some are like students who went to medical school specifically (e.g., MedGemma, SIP-jmed).
• The Thinkers: Some are designed to "think before they speak," taking extra time to reason through a problem (e.g., Qwen-Thinking, gpt-oss).

They ran all these models on a powerful local computer (a Mac Studio), simulating a hospital setting where patient data can't leave the building for privacy reasons.

2. The Three Challenges

The AI models were put through three distinct tests, like a triathlon:

Challenge A: The "Fill-in-the-Blank" Test (Formatting & Extraction)

• The Task: Imagine a doctor gives the AI a messy list of facts (JSON data) and says, "Turn this into a perfect, official hospital report." Or, "Read this official report and pull the facts back out into a list."
• The Result:
  • The Generalists were great at copying the format perfectly. They were like fast typists who never miss a comma.
  • However, when the task required math or logic (like calculating cancer stages based on tumor size), the regular models got confused. They guessed.
  • The "Thinker" models shined here. Because they pause to reason, they got the math right almost 100% of the time. They were like the students who actually understood the lesson, not just memorized the answers.

Challenge B: The "Proofreader" Test (Typo Correction)

• The Task: The researchers took real reports and intentionally added typos (swapping letters, deleting words, using the wrong Japanese characters). The AI had to find and fix them without changing the meaning.
• The Result:
  • This was tricky. Some AIs were too aggressive and deleted whole sentences (like a proofreader who throws away the page because they found one typo).
  • The Medical Specialists and the Thinkers did the best job. They understood the context of medical terms better than the general models. One model (Qwen) was the most balanced "proofreader," fixing errors without breaking the text.

Challenge C: The "Teacher" Test (Explaining to Humans)

• The Task: The AI had to write a simple explanation of a complex cancer case, as if teaching a new medical resident.
• The Result: This was the most surprising part.
  • Human taste is subjective. Just like how one person loves spicy food and another hates it, the doctors and clinicians who graded the AI had very different opinions.
  • One doctor might say, "This explanation is perfect!" while another says, "This is confusing."
  • There was no single "best" AI. The "Thinker" models were liked more by pathologists, while others were preferred by clinicians. It showed that what sounds good to a doctor depends entirely on who you ask.

3. The Big Takeaway

The paper concludes that there is no "one-size-fits-all" robot for this job yet.

• If you need speed and perfect formatting: Use a standard, fast model.
• If you need complex logic or math: Use a "Thinking" model, even if it's slower.
• If you need to fix typos or explain things: A medical-specialized model is your best bet.

The "Local" Advantage:
The authors emphasize that using these open-source models is like having a private library in your hospital. You don't have to send sensitive patient data to a big tech company (like Google or OpenAI) over the internet. You can run the AI right there in the hospital, keeping secrets safe.

The Bottom Line

Open-source AI is ready to be a valuable intern for Japanese pathologists, but it's not ready to be the boss yet. It needs to be assigned the right tasks (logic vs. typing vs. explaining) and needs a human to double-check its work, especially because different doctors have different styles.

In short: The tools are here, but we still need to learn how to use them wisely.

Technical Summary

1. Problem Statement

The rapid adoption of Large Language Models (LLMs) in medicine has raised the question of their utility in Japanese pathology reporting. While commercial models (e.g., ChatGPT, Gemini) offer high performance, their use with real patient data is often restricted by privacy, governance, and contractual issues. Consequently, open-source LLMs offer a viable alternative for local deployment. However, there is a lack of comprehensive benchmarks evaluating these models specifically for Japanese medical text, which presents unique challenges in morphology, terminology, and formatting compared to English. The study aims to fill this gap by evaluating open-source models across three critical clinical tasks: structured reporting, error correction, and explanatory writing.

2. Methodology

The authors benchmarked seven open-source LLMs available as of January 2026, running them locally on a Mac Studio (M2 Ultra, 196 GB RAM) using llama.cpp.

Models Evaluated:

• General Purpose: Gemma 3-27b-it, Qwen3-Next-80B-A3B-Instruct, gpt-oss-20b, gpt-oss-120b.
• Medical Specialized: MedGemma-27b-text-it, SIP-jmed-llm-3-8x13b-AC-32k-instruct.
• Reasoning/Thinking: Qwen3-Next-80B-A3B-Thinking.
• Note: Models were quantized (Q4/Q8 or MXFP4) to fit local hardware constraints.

Three Benchmark Tasks:

• Benchmark A: Structured Reporting & Information Extraction

  • Context: Based on the 19th edition of the Japanese Breast Cancer Society reporting rules (Breast Kiyaku).
  • Tasks:
    • A1: Convert JSON data to a hospital-specific template.
    • A2: Convert JSON to a template while inferring missing clinical scores (pT, Nuclear Grade, Histological Grade) from raw data.
    • A3: Convert JSON to a guideline-compliant template (few-shot learning).
    • A4: Extract structured JSON items from unstructured report text.
  • Metrics: Character-level 3-gram F1, Jaccard scores, and specific accuracy for clinical judgments.

• Benchmark B: Typo Correction

  • Data: 31 real pathology reports (100–400 Japanese characters) from the National Cancer Center Hospital East.
  • Method: Synthetic typos (deletion, insertion, transposition, kanji conversion errors) were injected with a 0.1 probability.
  • Evaluation: A pathologist manually annotated True Positives (TP), False Positives (FP), False Negatives (FN), and Large Deletions (LD).
  • Metrics: Precision, Recall, and F1 scores.

• Benchmark C: Subjective Evaluation of Explanatory Text

  • Task: Generate explanations for 23 complex cases suitable for a "newby resident."
  • Raters: 5 board-certified pathologists and 3 clinicians (5+ years experience).
  • Scoring: 1–5 scale (5 = usable without revision; 1 = non-functional).
  • Metrics: Intraclass Correlation Coefficients (ICC) to measure inter-rater reliability.

3. Key Results

A. Structured Reporting & Reasoning

• Formatting (A1, A3): Most models achieved near-perfect text overlap (F1 > 0.99) for simple template conversions.
• Reasoning (A2): This was the differentiator.
  • Non-thinking models (e.g., Gemma 3, MedGemma) achieved high string overlap but failed at clinical reasoning, with accuracy for pT and grade calculation close to chance (e.g., 0.18 for pT).
  • Thinking/Reasoning models (Qwen3-Next-Thinking, gpt-oss-120b) achieved near-perfect reasoning accuracy (1.00 for pT, NG, HG).
• Extraction (A4): High performance across most models, though the specialized SIP-jmed model struggled significantly (Match ratio ~0.53).

B. Typo Correction

• Best Performer: Qwen3-Next-80B-A3B-Instruct achieved the best balance (Macro F1: 0.697).
• Specialized Models: MedGemma (MG-27b) and SIP-jmed showed competitive performance, with SIP-jmed catching some domain-specific errors others missed but suffering from high rates of "Large Deletions" (over-correction).
• Worst Performer: gpt-oss-20b had the lowest overall performance in this task.

C. Subjective Evaluation (Explanatory Text)

• Utility: Roughly 25–33% of outputs were rated 4 or 5 (usable with little/no revision).
• Model Preferences:
  • MedGemma-27b received high scores from both groups.
  • Qwen3-Next-Thinking was rated higher by pathologists than clinicians.
• Reliability: Inter-rater agreement was low (ICC(2,1) < 0.23), indicating high variability in human preference. Reliability improved when averaging scores across raters (ICC(2,k) up to ~0.46), suggesting that "quality" is subjective and dependent on the specific rater's style.

4. Key Contributions

1. First Japanese Pathology Benchmark: Provides the first comprehensive evaluation of open-source LLMs specifically for Japanese pathology report generation, addressing a gap in non-English medical AI benchmarks.
2. Task-Specific Model Selection: Demonstrates that no single model dominates all tasks.
   • Thinking models are essential for tasks requiring clinical inference (e.g., calculating pT stages).
   • Medical-specialized models show advantages in typo correction and domain-specific phrasing.
   • Standard models suffice for deterministic formatting tasks.
3. Quantification of Human Variability: Highlights that subjective evaluation of medical explanations varies significantly among experts, suggesting that future systems must be personalized rather than optimized for a single "gold standard."
4. Local Deployment Feasibility: Proves that high-performance open-source models can run on local hardware (Mac Studio) with quantization, offering a privacy-preserving alternative to cloud-based commercial APIs.

5. Significance and Limitations

Significance:
The study validates that open-source LLMs are clinically relevant for specific, limited scenarios in Japanese pathology, particularly where data privacy is paramount. It provides a roadmap for hospitals to select models based on specific workflow needs (e.g., using "Thinking" models for diagnostic inference and "Specialized" models for editing).

Limitations:

• No Direct Commercial Comparison: The study did not compare open-source models against top-tier closed commercial models (e.g., GPT-5), which may still outperform open-source counterparts.
• Hardware Constraints: Evaluation was limited to a specific local setup; performance may vary on different hardware or with different quantization levels.
• Prompt Sensitivity: Results may be influenced by specific prompt engineering and the stochastic nature of sampling.
• Scope: Focused primarily on breast pathology; generalizability to other organ systems requires further study.

Conclusion

Open-source LLMs offer limited but meaningful utility for assisting Japanese pathology report writing. The optimal approach involves a hybrid strategy: utilizing reasoning-capable models for diagnostic inference, specialized models for error correction, and acknowledging that human preferences for explanatory text are highly variable, necessitating local validation and potential personalization before clinical deployment.