💬 NLP

CQA-Eval: Designing Reliable Evaluations of Multi-paragraph Clinical QA under Resource Constraints

This paper introduces CQA-Eval, a framework and set of recommendations for efficiently evaluating multi-paragraph clinical QA systems under resource constraints, demonstrating through physician annotations that fine-grained sentence-level analysis improves correctness agreement while coarse-level analysis better captures relevance, and that annotating a small subset of sentences can achieve reliability comparable to full coarse annotations.

Original authors: Federica Bologna, Tiffany Pan, Matthew Wilkens, Yue Guo, Lucy Lu Wang

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Federica Bologna, Tiffany Pan, Matthew Wilkens, Yue Guo, Lucy Lu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor trying to figure out if a new AI assistant is good at answering patients' health questions. The problem? Checking the AI's work is hard, expensive, and tiring. You need real medical experts to read long, multi-paragraph answers and decide if they are safe, accurate, and helpful. But experts are busy, and they often disagree with each other.

This paper introduces a new "rulebook" called CQA-EVAL to help researchers and developers evaluate these AI doctors more reliably, even when they have limited time and money.

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Whole Cake" vs. "Slicing the Cake"

When checking an AI's answer, researchers usually have two ways to look at it:

The "Whole Cake" Approach (Coarse): You read the entire answer and give it one overall grade. It's fast, but you might miss a tiny, dangerous ingredient hidden in the middle.
The "Slicing the Cake" Approach (Fine-Grained): You cut the answer into individual sentences and grade each one. It's slower and more detailed, but you catch every tiny mistake.

The Big Discovery: The paper found that which method you choose depends on what you are grading.

For Facts (Correctness): Think of this like checking a math problem. If the answer says "2+2=5," it's wrong. Slicing the cake (sentence-by-sentence) is much better here. It helps experts agree more because they are looking at specific facts, not the whole vibe.
For Context (Relevance): Think of this like judging a story. Does the story answer the question? Sometimes, you need to see the whole picture to know if it makes sense. Eating the whole cake (coarse) actually works better here because experts agree more when they see the big picture.
For Safety (Risk Disclosure): This is the hardest part. It's like checking if a recipe forgot to mention "allergies." Even with slicing or whole-cake methods, experts struggled to agree. It seems this is just a very hard human task to standardize.

2. The "Taste Test" Shortcut: Partial Slicing

Grading every single sentence is exhausting and expensive. The researchers asked: "Do we really need to taste every single crumb?"

They found that tasting just three sentences out of a long answer gave them almost the same reliable results as tasting the whole thing.

The Analogy: Imagine you are buying a bag of mixed nuts. You don't need to eat every single nut to know if the bag is good. If you grab a few random ones and they taste great, you can be confident the whole bag is good.
The Benefit: This saves about 50% of the time and money while keeping the quality of the evaluation high.

3. The "Length Bias" Trap

The researchers noticed something funny: AI models tend to write long, wordy answers, while human doctors write short, punchy answers.

The Bias: When experts read the whole "cake" (coarse method), they often gave higher scores to the long, fancy AI answers, even if the human answer was just as accurate. They were tricked by the length, thinking "more words = more information."
The Fix: By forcing experts to look at one sentence at a time (fine-grained), the "length bias" disappeared. The experts stopped being impressed by the word count and started focusing on whether the actual facts were right. This made the comparison between AI and humans much fairer.

4. Can AI Judge AI? (The "Robot Judge")

Finally, they tried using a super-smart AI (GPT-4) to grade the other AI's answers, hoping to save money on human experts.

The Result: The Robot Judge was okay, but it wasn't perfect. It agreed with human experts about as well as two humans agreed with each other.
The Lesson: You can use the Robot Judge as a helper when you don't have enough human experts, but you shouldn't replace the humans entirely. Also, giving the Robot Judge the "sentence-by-sentence" instructions helped it get better at checking facts, but not necessarily at checking relevance.

Summary: The New Rulebook

If you want to build or test an AI doctor, here is the paper's advice:

Don't use one size fits all. If you are checking if the AI knows the facts, grade it sentence-by-sentence. If you are checking if it answers the right question, grade the whole answer.
Don't grade everything. Just pick a small sample of sentences to check. It saves money and time without losing accuracy.
Watch out for long answers. If you only look at the whole answer, you might unfairly favor the AI just because it talks too much. Look at the sentences individually to be fair.
Safety is hard. Checking if an AI mentions all the risks is still very difficult for both humans and machines. We need new ways to solve this.

In short: Be smart about how you grade. Use the right tool for the right job, and you'll get reliable results without burning out your experts or your budget.

1. Problem Statement

Evaluating Large Language Model (LLM) outputs for multi-paragraph clinical question answering (QA) is resource-intensive and prone to inconsistency.

Expertise Requirement: Accurate evaluation requires medical expertise, which is costly and scarce.
Annotation Inconsistency: Expert annotators often disagree on what constitutes a "good" answer, leading to low Inter-Annotator Agreement (IAA).
Granularity Dilemma: It is unclear whether fine-grained (sentence-level) or coarse-grained (answer-level) annotation designs yield better reliability across different evaluation dimensions (e.g., factual correctness vs. contextual relevance) in high-stakes medical domains.
Resource Constraints: Full fine-grained annotation is prohibitively expensive and time-consuming, necessitating strategies that reduce cost without sacrificing reliability.

2. Methodology

The authors introduce CQA-EVAL, a framework and dataset designed to compare annotation strategies under resource constraints.

Dataset Construction

Source: 300 Question-Answer (QA) pairs derived from the K-QA dataset (real patient questions from the K Health platform).
Answer Types: Each of the 100 unique patient questions was paired with three answers:
1. Ground Truth: Curated by medical doctors.
2. LLM 1: Generated by GPT-4.
3. LLM 2: Generated by Llama-3.1-Instruct-405B.
Annotators: Six independent physicians recruited via Upwork, with 3–15 years of clinical experience.

Evaluation Dimensions

The study evaluates answers across three specific dimensions:

Correctness: Alignment with current medical knowledge.
Relevance: Whether the answer addresses the specific medical question.
Communicates-risks: Whether the answer discloses contraindications or potential risks.

Experimental Design

The study employed a randomized human annotation study comparing two designs:

Coarse-grained: Annotators rate the entire answer as a whole on a 5-point Likert scale.
Fine-grained: Annotators evaluate individual sentences (randomly sampled 6 sentences per answer) within the context of the whole answer.
Partial Annotation: A subset of the study tested evaluating only 3 sentences per answer to assess cost-efficiency.
LLM-as-Judge: GPT-4o was used as an automated judge to compare agreement with human experts under both coarse and fine-grained prompting strategies.

Metrics

Reliability: Measured using Randolph's $\kappa$ (Inter-Annotator Agreement) and Intra-Rater Reliability (IRR).
Effort: Measured via time per QA pair, NASA-TLX (mental/physical demand), and self-reported confidence.
System Performance: Comparison of system-level rankings and scores derived from different annotation granularities.

3. Key Contributions

CQA-EVAL Framework: A structured evaluation framework with explicit codebooks for coarse and fine-grained assessment of clinical QA.
New Dataset: A dataset of 300 QA pairs with multi-dimensional annotations (Correctness, Relevance, Risk) from six medical experts.
Granularity Analysis: A comprehensive comparison showing that the optimal annotation design depends on the specific dimension being evaluated.
Cost-Effective Strategy: Demonstration that partial fine-grained annotation (evaluating only 3 sentences) achieves reliability comparable to full fine-grained annotation at half the cost.
LLM-as-Judge Guidelines: Insights into how LLM judges perform relative to human experts under different prompting strategies.

4. Key Results

Inter-Annotator Agreement (IAA) by Dimension

Correctness: Fine-grained annotation significantly improved IAA ( $\kappa \approx 0.88\text{--}0.90$ ) compared to coarse ( $\kappa \approx 0.74$ ). Factual claims are easier to verify in isolation.
Relevance: Coarse-grained annotation yielded higher IAA ( $\kappa \approx 0.71$ ) than fine-grained ( $\kappa \approx 0.29\text{--}0.32$ ). Relevance is a holistic property that is difficult to judge sentence-by-sentence.
Communicates-risks: IAA remained low in both settings ( $\kappa < 0.45$ ), suggesting this dimension is intrinsically difficult to evaluate due to the need to identify missing information rather than verifying existing text.

Cost and Efficiency

Time: Coarse annotation took ~239 seconds per answer; full fine-grained took ~460 seconds.
Partial Fine-Grained: Evaluating only 3 sentences reduced time/cost to levels comparable to coarse annotation while maintaining high correlation ( $>0.8$ ) with full fine-grained ratings and lower variance.
Annotator Feedback: Annotators found coarse tasks "easier" (lower mental demand) but felt more "successful" and less discouraged with fine-grained instructions.

System-Level Bias Mitigation

Length Bias: Coarse annotations favored longer, more elaborated LLM answers over shorter, concise physician answers, conflating length with quality.
Correction: Fine-grained annotations mitigated this bias, resulting in fairer system rankings where physician answers received higher correctness scores (0.99) compared to coarse settings (0.78).

LLM-as-Judge Performance

Fine-grained instructions improved LLM-expert agreement for Correctness and Communicates-risks but decreased it for Relevance.
With appropriate prompting, LLMs achieved agreement with experts comparable to (or exceeding) expert-expert agreement for Correctness and Relevance, suggesting they are viable supplements when human resources are constrained.

5. Significance and Recommendations

The paper provides actionable guidelines for developing reliable clinical QA evaluation systems under resource constraints:

Tailor Granularity to Dimension:
- Use Fine-grained (or partial fine-grained) schemes for factual dimensions (Correctness) to maximize reliability and reduce length bias.
- Use Coarse-grained schemes for context-dependent dimensions (Relevance) where holistic judgment is superior.
Adopt Partial Sampling: For resource-constrained studies, evaluating a random subset of sentences (e.g., 3 per answer) offers the best trade-off between cost, effort, and statistical reliability.
Address Risk Disclosure: Current annotation designs fail to reliably capture "risk communication." Future work requires new strategies (e.g., counterfactual prompts or risk checklists) rather than just changing granularity.
LLM Integration: LLMs can serve as effective "judges" for factual correctness and relevance if prompted with fine-grained or coarse instructions respectively, but human oversight remains critical for safety and risk assessment.

Conclusion: CQA-EVAL demonstrates that a "one-size-fits-all" annotation strategy is suboptimal. By matching the annotation design (coarse vs. fine) to the specific evaluation dimension, researchers can achieve high reliability, reduce annotator fatigue, and ensure fairer comparisons between LLMs and medical professionals.