🤖 machine learning

Auditing Sybil: Explaining Deep Lung Cancer Risk Prediction Through Generative Interventional Attributions

This paper introduces S(H)NAP, a model-agnostic auditing framework that uses generative interventional attributions to reveal that while the Sybil deep learning model accurately predicts lung cancer risk, it suffers from critical failure modes such as sensitivity to clinically unjustified artifacts and a distinct radial bias.

Original authors: Bartlomiej Sobieski, Jakub Grzywaczewski, Karol Dobiczek, Mateusz Wójcik, Tomasz Bartczak, Patryk Szatkowski, Przemysław Bombiński, Matthew Tivnan, Przemyslaw Biecek

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bartlomiej Sobieski, Jakub Grzywaczewski, Karol Dobiczek, Mateusz Wójcik, Tomasz Bartczak, Patryk Szatkowski, Przemysław Bombiński, Matthew Tivnan, Przemyslaw Biecek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, automated radiologist named Sybil. Sybil looks at a 3D scan of a person's lungs (a CT scan) and predicts how likely they are to get lung cancer in the next six years. Sybil is incredibly accurate and has been tested on thousands of people.

However, there's a problem: We don't know exactly how Sybil makes its decisions. It's a "black box." We know it gets the right answer often, but we don't know if it's looking at the right things (like a suspicious lump) or if it's secretly cheating by looking at the wrong things (like a metal snap on a hospital gown).

This paper introduces a new tool called S(H)NAP to "audit" Sybil. Think of S(H)NAP as a digital detective that uses a special kind of magic to poke and prod Sybil's brain to see what's really going on.

Here is how the paper explains it, using simple analogies:

1. The Problem: Correlation vs. Causation

Currently, we check if Sybil is good by looking at the results (did it predict cancer correctly?). This is like judging a chef only by whether the food tastes good, without knowing if they used fresh ingredients or if they accidentally put poison in it. The paper argues we need to move from just watching what happens to testing why it happens.

2. The Solution: The "Magic Eraser" and "Magic Paste"

To understand Sybil, the researchers needed to change the CT scans and see how Sybil reacted. But you can't just delete a tumor from a real patient's scan; that's impossible and dangerous.

So, they built a Generative AI (a type of artificial intelligence that creates images) that acts like a 3D Photoshop with a brain.

The Magic Eraser (Nodule Removal): They used this AI to find a suspicious lump (nodule) in a scan and "erase" it, replacing it with perfectly healthy lung tissue. It's so realistic that even expert human radiologists couldn't tell the difference between the real scan and the edited one.
The Magic Paste (Nodule Insertion): They took a known cancerous lump from one patient and "pasted" it into a different patient's healthy lung. The AI blended it so perfectly that it looked like it had always been there.

3. The Audit: Two New Tools

Using these magic tools, the researchers created two ways to test Sybil:

SHNAP (The "What If" Detective):
- How it works: They systematically erased different lumps from a scan, one by one, and in groups.
- The Analogy: Imagine a jury deciding a verdict. SHNAP asks, "If we remove this specific piece of evidence, does the verdict change?" If the verdict stays the same, that piece of evidence didn't matter. If the verdict flips, that piece was crucial.
- What they found: Sybil mostly works like a smart human doctor, focusing on the lumps. However, sometimes Sybil gets distracted. In some cases, it ignored a dangerous lump and focused on the background of the lungs instead. In other cases, it gave a high-risk warning because of a harmless lump, just to be safe.
SNAP (The "Sensitivity Map"):
- How it works: They took a known cancerous lump and dropped it into thousands of different spots in a healthy lung to see if Sybil noticed it.
- The Analogy: Imagine sprinkling glitter (cancer) all over a room and seeing where the security guard (Sybil) looks.
- What they found: Sybil has a blind spot. It is very good at spotting lumps in the center of the lungs, but it gets "lazy" or confused when lumps are right against the edge of the lung (the pleura). This is dangerous because the most common type of lung cancer grows right on that edge.

4. The Shocking Discoveries

The audit revealed some scary flaws that pure observation missed:

The "Hospital Gown" Mistake: Sybil sometimes thinks a metal snap on a hospital gown (visible in the scan) is a sign of cancer. It's like a security guard arresting someone because they are wearing a specific color shirt, not because they committed a crime.
The "Chin" Mistake: Sybil sometimes mistakes a patient's chin (visible in the scan) for a large, suspicious lump.
The "Edge" Blind Spot: As mentioned, Sybil struggles to see cancer growing on the very edge of the lung, which is a critical failure for a life-saving tool.

5. The Conclusion

The paper concludes that while Sybil is a powerful tool, we cannot trust it blindly yet. It makes mistakes not just because it's "wrong," but because it has learned bad habits (like looking at metal snaps) and has structural blind spots (like the lung edges).

The authors argue that before we let AI like Sybil make life-or-death decisions in hospitals, we must use tools like S(H)NAP to audit them. We need to know why it makes a decision, not just that it makes a decision, to ensure it isn't accidentally harming patients.

Technical Summary: Auditing Sybil: Explaining Deep Lung Cancer Risk Prediction Through Generative Interventional Attributions

1. Problem Statement

Lung cancer remains the leading cause of cancer mortality globally, driving the development of automated screening tools like Sybil, a deep learning model capable of predicting 6-year lung cancer risk from a single low-dose computed tomography (LDCT) scan with high precision. While Sybil has undergone extensive clinical validation demonstrating robustness across diverse populations, current assessments rely purely on observational metrics. This correlation-based approach confirms that the model works but fails to explain why it works or when it might fail. This lack of causal verification creates a critical oversight for high-stakes clinical deployment, as it overlooks the model's actual reasoning mechanisms, including potential reliance on spurious correlations or artifacts.

2. Methodology: The S(H)NAP Framework

To address the limitations of observational validation, the authors propose S(H)NAP, a model-agnostic auditing framework that shifts the paradigm from observational validation to interventional auditing. The framework leverages System-Embedded Diffusion Bridges (SDB) to perform realistic 3D generative interventions on pulmonary nodules, allowing for the isolation of object-specific causal contributions to risk scores.

The methodology is grounded in Hypothesis 1: Sybil's decision function can be effectively approximated by a Linear Model with Pairwise Interactions (LMPI) over pulmonary nodules, consisting of a sample-specific background term and main/pairwise effects of nodules.

The framework comprises two primary attribution methods:

A. SHNAP (SHapley Nodule Attribution Profiles)

SHNAP is designed to decompose the model's prediction into individual nodule contributions and inter-nodule interactions.

Mechanism: It utilizes nodule removal via diffusion bridges. By systematically replacing specific nodules with healthy tissue (using a "do-operator" approach), the framework generates a dataset of counterfactual scans representing all possible coalitions of nodules.
Attribution: It employs n-Shapley Values (nSV) with $n=2$ to approximate the model as an LMPI. This calculates main effects (individual nodule impact) and pairwise interactions (how nodules influence each other) while adhering to the data manifold.
Goal: To verify if Sybil processes nodules as modular semantic units independent of the background context.

B. SNAP (Substitutive Nodule Attribution Probing)

SNAP is designed to probe the model's volumetric sensitivity and spatial biases.

Mechanism: It utilizes nodule insertion. Realistic nodules of known malignancy are extracted from one patient and inserted into specific coordinates of another patient's scan using the diffusion bridge to align the nodule with the new anatomical context.
Attribution: It measures the shift in the model's logit output (log-odds ratio) resulting from the insertion.
Goal: To reveal spatial biases (e.g., sensitivity to location, size, or surrounding tissue) and test the model's reasoning under controlled "what-if" scenarios.

C. Generative Intervention via Diffusion Bridges

The core technical innovation is the use of System-Embedded Diffusion Bridges (SDB) to ensure interventions remain in-distribution.

Nodule Removal: The SDB treats the masked nodule region as missing data and reconstructs it using the score model trained on healthy tissue distributions, effectively erasing the nodule while preserving the surrounding anatomy.
Nodule Insertion: A nodule from a source patient is "copy-pasted" into a target scan and diffused. The SDB ensures the inserted nodule blends seamlessly with the new context, avoiding the artifacts common in naive copy-paste methods.
Theoretical Basis: The approach relies on the Mismatched Estimation Theorem (Verdú, 2009), which guarantees that diffused samples from different distributions become statistically indistinguishable after a specific timestep, validating the use of the score model for reconstruction.

3. Key Results

A. Validity of Synthetic Perturbations

Expert radiologists evaluated the realism of the generated interventions.

Removal: Radiologists could not distinguish between real healthy tissue and synthetically removed nodules (accuracy statistically indistinguishable from random guessing), confirming high fidelity.
Insertion: An optimal blending timestep ( $\tau = 0.3$ ) was identified, where inserted nodules preserved source properties without visual artifacts or excessive deviation from the new context.

B. Model Structure Verification

LMPI Approximation: SHNAP experiments confirmed that Sybil functions effectively as a linear model with pairwise interactions over pulmonary nodules. The approximation achieved a high coefficient of determination ( $R^2 \approx 1$ ) for the majority of cases, supporting Hypothesis 1.
Stability: SHNAP attributions showed low variance across independent runs, indicating robustness against the stochasticity of the reverse diffusion process.

C. Critical Failure Modes and Misalignments

The interventional audit revealed several critical issues not detectable via observational metrics:

Radial Sensitivity Bias: SNAP revealed a systematic failure to detect peripheral nodules near the lung boundary (pleura). A linear regression showed a significant positive correlation between distance-to-pleura and attribution, indicating sensitivity drops near the boundary. The authors hypothesize this stems from zero-padding in 3D convolutions, a structural vulnerability clinically concerning given that adenocarcinoma often arises in the periphery.
Lobar Bias: While Sybil correctly identified upper lobes as higher risk (aligning with clinical standards), it exhibited a global bias where sensitivity varied distinctly by lobe, independent of patient-specific factors.
Spurious Correlations and Artifacts:
- False Positives: The model occasionally attributed high risk to benign findings or artifacts, such as ECG electrodes on the chest or metal snaps from hospital gowns.
- False Negatives: In some malignant cases, the model ignored actual nodules, assigning negative attribution to them, and instead relied on background features or interactions to make a correct (but for the wrong reasons) prediction.
- Inconsistency: Longitudinal analysis showed the model could shift its focus between nodules in follow-up scans, rendering its reasoning unstable despite potentially defensible initial predictions.

4. Significance and Claims

The paper claims to provide the first interventional audit of the Sybil model, demonstrating that while the model often mimics expert radiologist behavior, it suffers from critical failure modes that correlation-based validation misses.

Paradigm Shift: The work argues for a shift from purely observational validation to causal verification via interventional auditing to ensure robust decision-making before clinical deployment.
Methodological Contribution: It introduces a novel framework (S(H)NAP) that combines game-theoretic attributions, generative modeling, and radiological expertise to audit models in a way that respects the data manifold.
Safety and Trust: By uncovering reliance on clinically unjustified artifacts (e.g., ECG leads) and structural biases (e.g., radial sensitivity), the paper highlights the necessity of rigorous auditing to prevent unreliable predictions in high-stakes medical settings.
Model Agnosticism: The framework is model-agnostic, relying only on input-output pairs, making it applicable to proprietary systems (e.g., Optellum) and other medical imaging models.

The authors conclude that while Sybil demonstrates significant discriminative power, the identified reasoning misalignments underscore the need for strict model verification beyond standard performance metrics to ensure the ethical and safe development of AI in medicine.