Discordance in pleural mesothelioma response classification and modelling of impact on clinical trials

This study demonstrates that the high rate of discordance in radiological response classification for pleural mesothelioma, driven largely by interpretation and measurement inconsistencies, significantly undermines statistical power and endpoint precision in clinical trials.

The Gist

The Big Picture: A Game of "Telephone" with Cancer Scans

Imagine you are playing a game of "Telephone" (or "Broken Telephone") with a group of friends. One person whispers a message, and it gets passed down the line. By the time it reaches the last person, the message has often changed completely.

In the world of Pleural Mesothelioma (a tough type of lung cancer caused by asbestos), doctors play a similar game, but instead of whispers, they are looking at CT scans (3D X-rays) of patients' lungs. They need to decide: Is the cancer getting better, staying the same, or getting worse?

To make this decision, they use a strict rulebook called mRECIST. It's like a ruler that tells them exactly how to measure the tumor. If the tumor shrinks by 30%, it's a "win." If it grows by 20%, it's a "loss."

The Problem: Even when two expert radiologists (the "super-players" of this game) look at the exact same scan using the exact same ruler, they often disagree. This paper found that they disagreed 35% of the time. That's like flipping a coin and getting a different result every time you ask two people to call heads or tails.

---

The Study: Two Parts to the Puzzle

The researchers did two main things to figure out why this happens and what it means.

1. The Detective Work (Looking at Real Patients)

They gathered 172 patients who were being treated with chemotherapy. They took the CT scans and had two different expert radiologists measure the tumors independently, without talking to each other.

• The Result: In 60 out of 172 cases, the two experts gave different answers. One said "The cancer is shrinking," and the other said "The cancer is growing."
• Why? It wasn't usually because they were careless. It was because the tumor looks like a weird, lumpy blanket draped over the lung. Measuring a lumpy blanket with a straight ruler is incredibly hard. A tiny difference in where you put the ruler (even by a millimeter) can change the result from "shrinking" to "growing."
• The "Human Error" Factor: Sometimes, they just picked the wrong scan or made a math mistake, but most of the time, it was just the difficulty of the measurement itself.

2. The Crystal Ball (Computer Simulations)

Since they couldn't run a real clinical trial with 10,000 patients to see what happens when doctors disagree, they built a computer simulation.

Imagine you are baking a cake (a clinical trial) to prove a new recipe works. You want to be 80% sure your cake is delicious.

• Scenario A (Perfect World): You taste the cake perfectly. You are 80% confident it's a success.
• Scenario B (The Messy Kitchen): Now, imagine your taste testers are confused. Sometimes they say "Delicious!" when it's actually "Burnt," and vice versa.

The computer simulation showed that as the confusion (misclassification) goes up, the confidence in the results crashes.

• If the doctors are wrong 17% of the time (which matches the real-world data), the trial's ability to prove the drug works drops from 80% down to about 55%.
• It's like trying to hit a target with a bow and arrow, but your eyesight is blurry. You might still hit the target, but you can't be sure if you actually did, or if the wind just pushed the arrow.

---

Why Does This Matter?

1. For Clinical Trials (The "Science" Part)

If a new drug is actually amazing, but the doctors measuring the results keep disagreeing, the trial might fail.

• The Analogy: Imagine a new car that gets 50 miles per gallon. But the speedometers in the test cars are broken and give random readings. The test might conclude, "This car is just average," and the amazing car never gets approved.
• The Consequence: Good drugs might get rejected, or bad drugs might get approved because the data is too "noisy" to tell the truth.

2. For Real Patients (The "Human" Part)

In the real world, a patient usually only has one doctor looking at their scan.

• If that one doctor makes a mistake and says "The cancer is growing," the patient might stop a drug that is actually working.
• If the doctor says "The cancer is shrinking" when it's actually growing, the patient might keep taking a toxic drug that isn't helping, wasting precious time.

The Solution?

The paper suggests we need better tools.

• The "Smart Ruler": Instead of humans trying to measure a lumpy blanket with a straight ruler, we need Artificial Intelligence (AI). AI can measure the entire volume of the tumor (like measuring the whole blanket's weight) rather than just a few lines.
• Standardized Templates: Using a checklist to make sure no one picks the wrong scan or forgets a step.

The Bottom Line

This study is a wake-up call. It shows that our current way of measuring cancer growth is as shaky as a house of cards. We are losing the ability to tell if new treatments work because our "rulers" are too wobbly. To save lives and find cures, we need to upgrade our measuring tools to be as precise as the science behind the drugs.

Technical Summary

1. Problem Statement

Pleural Mesothelioma (PM) is an aggressive malignancy where clinical trials are critical for developing new therapies. However, the assessment of treatment response in PM is notoriously difficult due to the tumor's unique growth pattern (diffuse pleural thickening rather than discrete nodules).

• Current Standard: The field relies on modified Response Evaluation Criteria In Solid Tumours (mRECIST v1.1), which requires radiologists to manually measure six unidimensional points on CT scans.
• The Issue: Previous studies suggest high inter-observer variability in measurements, but the frequency of discordant response classifications (where two experts disagree on whether a patient is responding, stable, or progressing) had not been quantified.
• The Gap: It was unknown how this classification discordance impacts the statistical power and precision of clinical trials. In routine practice, a single inaccurate assessment can lead to the continuation of toxic, ineffective drugs or the premature withdrawal of effective ones.

2. Methodology

The authors employed a mixed-methods approach combining a retrospective multicentre cohort study with in silico (computer simulation) modelling.

A. Retrospective Cohort Study

• Data Source: 172 chemotherapy-treated PM patients from four UK centers (Glasgow, Wythenshawe, Leicester).
• Protocol: Two expert thoracic radiologists (blinded to clinical data) independently reviewed baseline and follow-up CT scans using mRECIST v1.1.
• Classification: Patients were categorized as Partial Response (PR), Stable Disease (SD), or Progressive Disease (PD).
• Analysis:
  • Discordance Rate: Calculated as the percentage of cases where the two reviewers disagreed.
  • Root Cause Analysis: Discordant cases were categorized into:
    • Adjacent group swaps (e.g., PR vs. SD).
    • Non-adjacent swaps (e.g., PR vs. PD).
    • Causes: Subjective interpretation differences, human error (wrong scan, measurement error), or intrinsic mRECIST measurement variances.
  • Volumetric Correlation: A subset of cases underwent manual volumetric analysis to compare mRECIST results against actual tumor volume changes.

B. In Silico Modelling

• Objective: To simulate the impact of response misclassification on clinical trial outcomes.
• Design: Simulated two-arm trials for an active therapy with an intended 80% statistical power and 95% confidence interval (CI) coverage.
• Endpoints Modeled:
  1. Objective Response Rate (ORR)
  2. Disease Control Rate (DCR)
  3. Progression-Free Survival (PFS)
  4. Overall Survival (OS)
• Variables: The misclassification rate (equivalent to a single-reporter error rate) was varied from 0% to 100% in 1% increments.
• Simulation Scale: Each scenario was repeated 10,000 times to generate robust statistical inferences.
• Assumptions: Included specific PM parameters (e.g., median OS post-progression, hazard ratios) and assumed non-differential misclassification across response classes.

3. Key Contributions

• Quantification of Discordance: First systematic multicentre study to quantify the rate of discordant mRECIST classifications in PM, finding a 35% discordance rate between experts.
• Root Cause Identification: Detailed breakdown showing that 83% of discordances stem from intrinsic measurement/interpretation variances rather than gross human error.
• Impact Modelling: First in silico modelling to demonstrate the quantitative degradation of statistical power and endpoint precision in clinical trials due to response misclassification.
• Volume vs. mRECIST: Evidence that mRECIST classifications often fail to align with actual volumetric tumor changes, highlighting the limitations of unidimensional measurements for PM.

4. Key Results

Cohort Findings

• Discordance Rate: 35% (60 out of 172 cases) showed disagreement between the two radiologists.
• Agreement: Inter-observer agreement was only moderate (Cohen's kappa = 0.456).
• Nature of Discordance:
  • 83% (50/60) were "adjacent" swaps (e.g., PR vs. SD or SD vs. PD), driven by subjective interpretation or minor measurement differences (<10% variance).
  • 17% (10/60) were "non-adjacent" swaps (PR vs. PD), driven by human errors (wrong scan reviewed, missed findings).
• Tumor Volume: Discordance was not associated with baseline tumor volume.
• Volumetric Correlation: Poor agreement was observed between volumetric response and mRECIST classification.

Modelling Findings

The study modeled the impact of a 17% misclassification rate (derived from the observed 35% discordance, assuming at least one of two discordant reports is wrong).

• Statistical Power Reduction:
  • ORR: Dropped from 80% to 55%.
  • DCR: Dropped from 80% to 53%.
  • PFS: Dropped from 80% to 65%.
  • OS: Dropped from 80% to 66%.
• Endpoint Precision (Confidence Interval Coverage):
  • The proportion of trials where the 95% CI actually contained the true treatment effect dropped significantly:
    • ORR: 95% $\rightarrow$ 88%
    • DCR: 95% $\rightarrow$ 89%
    • PFS/OS: 95% $\rightarrow$ 92%

5. Significance and Implications

• Clinical Trial Design: The findings suggest that current PM clinical trials may be severely underpowered due to measurement noise. A trial designed for 80% power may effectively operate at ~55% power if misclassification rates are high, leading to false negatives (missing effective drugs).
• Routine Practice: Inconsistent assessments in real-world settings could lead to inappropriate treatment decisions, exposing patients to toxicity or denying them effective therapy.
• Future Directions:
  • AI Integration: The authors argue for the adoption of AI-assisted automated tumor segmentation to eliminate human measurement variability.
  • Alternative Endpoints: There is a strong case for moving away from mRECIST-based endpoints toward more robust metrics (e.g., volumetric or AI-derived PFS/OS) to ensure trial validity.
  • Standardization: The use of dedicated reporting templates combined with AI could mitigate the "subjective difference" and "human error" identified in the root cause analysis.

Conclusion: The study provides compelling evidence that the current standard for assessing PM treatment response is unreliable, leading to substantial statistical inefficiencies in clinical trials. It advocates for an urgent shift toward automated, volumetric, and AI-driven assessment methods to improve the accuracy of both research and clinical care.