💻 computer science

Does Model Compression Amplify Clinical Bias? A Subgroup Fairness Audit of Quantized ICU Risk Models on MIMIC-IV

This study audits the impact of model compression techniques on subgroup fairness in ICU AKI risk prediction using MIMIC-IV, revealing that while naive quantization can catastrophically degrade performance and pruning significantly harms subgroup accuracy, proper methodological controls are essential to avoid spurious fairness claims and address inherent disparities driven by small subgroup sizes.

Original authors: Aman Chandra H, Stuti Shivhare

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aman Chandra H, Stuti Shivhare

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a brilliant, super-smart robot doctor. This robot can look at a patient's vital signs and predict if they might get a serious kidney problem in the next three days. It's amazing, but it's also a giant, heavy, and hungry machine that needs a massive server room to run. Now, imagine you want to take this robot doctor to a small, remote clinic where there's no big server, just a tiny, battery-powered computer. To make the robot fit, you have to shrink it down. You might try to make it speak in shorter sentences (quantization), cut out its less important memories (pruning), or teach a smaller, younger robot to copy the big one's thinking (distillation).

But here's the tricky part: when you shrink a smart robot, does it become dumber for everyone equally? Or does it start making mistakes only for specific groups of people? In the world of medical AI, this is a huge deal. If a shrinkage method makes the robot slightly less accurate for older patients but totally fine for younger ones, that's unfair and dangerous. This paper dives into that exact question: when we squeeze these medical models to fit on small devices, do we accidentally make them biased against certain groups of people, or do they stay fair?

The Great Shrink-Down Experiment

The researchers in this study decided to play a game of "squeeze and see." They took a model designed to predict Acute Kidney Injury (AKI) in ICU patients and tried to shrink it using three different methods:

Quantization: Turning the model's precise math into simpler, whole numbers (like changing a recipe from "1/3 cup" to "1/4 cup").
Pruning: Cutting out 30% or 50% of the model's connections, like trimming branches off a tree.
Distillation: Training a tiny "student" model to learn from the big "teacher" model.

They tested these shrunken models on a massive dataset of over 52,000 patient records from the MIMIC-IV database, checking if the models treated men and women, different age groups, and people with different types of health insurance fairly.

The "Outlier" Surprise: A Glitch in the Matrix

Before they could even talk about fairness, the team hit a massive, funny, and scary snag. When they first tried to shrink the model using quantization, the robot doctor completely lost its mind. Its prediction accuracy dropped to the level of a coin flip (50%).

Why? Because the raw data from the hospital had a few "glitchy" numbers. Imagine a patient's oxygen level being recorded as "5,000%" instead of "98%." These were just typos or sensor errors. When the model tried to shrink its math to fit these numbers, those few crazy outliers took over the whole system, confusing the robot so badly it couldn't tell a healthy patient from a sick one.

The team had to fix this by adding a "plausibility filter"—a simple rule that says, "Hey, oxygen can't be more than 100%!" Once they cleaned up the data, the quantized model started working again, almost as well as the original. This was a crucial discovery: sometimes, when a model breaks after shrinking, it's not the shrinking's fault; it's the data's fault.

The "Fairness" Trap: Don't Be Fooled by the Scoreboard

The researchers also found a sneaky trap in how people usually test these models. If you compare a "perfect" test of the original model against a "slightly flawed" test of the shrunken model, you might accidentally think the shrunken model is better at being fair. It's like comparing a runner who ran on a track with a tailwind to a runner on a windy day and saying the second runner is faster because they finished in a better time relative to the wind.

The team realized they had to test both models using the exact same rules (the "same-protocol" method) to get a true answer. Once they did this, the magic disappeared. The shrunken models didn't suddenly become fairer; they just got slightly worse at their job.

The Results: Everyone Gets a Little Dumber, But Fairness Stays the Same

So, what happened when they actually shrunk the models?

Accuracy Dropped (A Little): Every single method made the model slightly less accurate. The "pruning" method (cutting 50% of the connections) hurt the most, dropping accuracy by about 3 points. The "quantization" method hurt the least, dropping only about 0.5 to 0.9 points.
Fairness Didn't Get Worse: Here is the good news. The models didn't start treating specific groups (like women or older people) unfairly because they were shrunk. The gap in performance between different groups stayed roughly the same.
The "Fake" Fairness Improvement: The researchers noticed something weird. In some cases, the "fairness gap" (the difference in error rates between groups) actually got smaller after shrinking. But this wasn't because the model got better at helping the disadvantaged group. It was because the model got worse at helping the advantaged group, and the two groups' scores just met in the middle. It's like if a teacher stops giving the top students A's and starts giving them C's, while the struggling students still get D's. The gap between A and D is smaller, but the class isn't actually doing better; everyone just got a lower grade.

The Bottom Line

The paper concludes that shrinking medical AI models is possible, but it comes with a cost: the model becomes slightly less accurate for everyone. However, it doesn't seem to amplify existing biases or create new ones. The biggest lesson? If you are a doctor or a developer trying to put these models on small devices, you need to check the data for "glitchy" numbers first, and you must test the shrunken model using the exact same rules as the original.

Most importantly, don't just look at the "fairness score" and celebrate if it goes down. You have to look at why it went down. If the score improved because the model got worse for everyone, that's not a win for fairness; it's just a loss for accuracy. The researchers suggest that for real-world medical use, we should always report how much accuracy each specific group lost, rather than just looking at a single summary number.

Technical Summary: Does Model Compression Amplify Clinical Bias?

Problem Statement

Deploying clinical risk-prediction models on resource-constrained hardware (e.g., embedded or point-of-care devices) necessitates model compression techniques such as quantization, pruning, and knowledge distillation. While extensive literature evaluates the impact of these methods on aggregate accuracy, there is a critical gap in understanding their effects on subgroup fairness. A model may maintain high overall performance while introducing or amplifying performance disparities across patient subgroups (e.g., by gender, insurance type, or age). This paper addresses the lack of systematic audits regarding whether compression methods inadvertently degrade fairness in clinical settings, specifically for Acute Kidney Injury (AKI) prediction.

Methodology

Dataset and Task

The study utilizes the MIMIC-IV dataset, focusing on 52,178 adult first ICU stays with a length of stay ≥24 hours. The prediction task is the onset of AKI within 72 hours of admission, defined by KDIGO staging criteria. The positive class rate is 68.9%.

Features: 46 summary statistics (mean, std, min, max) derived from 11 vital signs and 10 laboratory values over the first 24 hours, plus demographics and comorbidity counts.
Subgroups: Gender, Insurance Type (Medicaid, Medicare, Private, Other, No charge), and Age Group (18-40, 41-65, 65+). Race/ethnicity was deliberately excluded from the scope.

Baseline Model

A compact Multilayer Perceptron (MLP) with 5,313 parameters was trained using 5-fold stratified cross-validation. The architecture consists of two hidden layers (64 and 32 units) with ReLU, BatchNorm, and Dropout. The model was optimized for 80% sensitivity, a clinically relevant target for early warning systems.

Compression Variants

Three standard compression methods were applied to the baseline:

INT8 Dynamic Quantization: Post-training quantization of all linear layers to 8-bit integers.
Magnitude Pruning: Unstructured L1 pruning at 30% and 50% of weights per layer. Note: Weights were zeroed but the tensor remained dense, meaning storage size did not decrease, serving as a probe for fairness sensitivity at specific sparsity levels.
Knowledge Distillation: Training a smaller student model (32 and 16 hidden units; ~1/4 parameters) using the teacher's soft outputs (temperature $T=3.0$ ) combined with hard labels.

Critical Methodological Corrections

The authors identified and corrected two significant confounding factors that could distort fairness audits:

Physiological Plausibility Filter: Initial runs of INT8 quantization collapsed the AUROC to chance level (0.500). Diagnosis revealed that a tiny fraction of physiologically impossible raw EHR values (e.g., oxygen saturation >100%) dominated the activation quantization range, collapsing the signal. The authors implemented a source-level filter to clip raw vitals/labs to plausible ranges (e.g., $O_2 \in [0, 100]\%$ ) before aggregation, restoring performance to 0.699.
Same-Protocol Baseline: An initial evaluation compared an honest out-of-fold baseline against compressed models evaluated in-sample (on the full cohort), creating a spurious artifact where compression appeared to improve fairness. The authors established a "same-protocol" baseline (evaluating the best-fold checkpoint on the full cohort) to isolate the true effect of compression from train/test contamination.

Evaluation Metrics

Performance: Subgroup-level AUROC, AUPRC, Sensitivity, Specificity, and PPV.
Fairness: Equalized Odds Difference (EOD) and Demographic Parity Difference, computed via Fairlearn.
Statistical Significance: Bootstrap testing (1,000 resamples) was used to determine if performance deltas between compressed variants and the baseline were distinguishable from noise.

Key Results

1. Subgroup Accuracy Degradation

All compression methods resulted in statistically significant (mostly $p < 0.001$ ) but small absolute decreases in AUROC across nearly all subgroups.

50% Pruning caused the largest degradation (approx. -0.03 AUROC).
Distillation and 30% Pruning showed moderate degradation.
INT8 Quantization showed the mildest degradation (-0.005 to -0.009 AUROC) once the outlier filter was applied.
The degradation was relatively uniform across subgroups within an axis (e.g., gender), suggesting compression did not concentrate harm on a single specific group in this cohort.

2. Fairness Metrics (Equalized Odds)

Gender: EOD was small (0.02–0.04) and largely unaffected by compression.
Insurance and Age: Baseline EOD was large (≥0.25). Compression reduced these gaps, but not through genuine equity improvement.
- The reduction in the gap was driven by the degradation of the better-performing subgroups (e.g., Medicare, younger patients) converging toward the performance of worse-performing subgroups, rather than the improvement of disadvantaged groups.
- The "No charge" insurance subgroup ( $n=6$ ) exhibited extreme sensitivity (0% sensitivity vs. 89% for Medicare), driving the insurance-axis EOD. This small sample size makes these metrics statistically fragile.

3. Calibration and Latency

Calibration: Pruning (specifically 30%) degraded calibration, shifting predicted probabilities further from observed outcomes, particularly at low-to-mid risk levels.
Latency: Counter-intuitively, INT8 quantization and distillation showed higher mean CPU latency than the FP32 baseline on cloud hardware. This is attributed to quantization/dequantization overhead and interpreter overhead on small models, which may not reflect performance on dedicated integer-inference embedded hardware.

Contributions and Significance

The paper makes four primary contributions:

Systematic Fairness Audit: A same-protocol comparison of three compression families on a real clinical cohort with bootstrap significance testing for every subgroup-metric combination.
Quantization Failure Diagnosis: Identification of a specific failure mode where naive per-tensor dynamic quantization collapses due to data-quality outliers (physiologically impossible values) dominating the activation range. This highlights a critical pre-processing requirement for quantized models trained on raw EHR data.
Evaluation Protocol Correction: Demonstration that comparing out-of-fold baselines against in-sample compressed models creates a "compression improves fairness" artifact. The paper advocates for a same-protocol comparator to isolate true compression effects.
Trade-off Analysis: A joint analysis of latency, memory, and fairness under CPU-only inference, revealing that unstructured pruning on dense tensors yields no true latency benefit proportional to sparsity at this scale.

Conclusion and Claims

The authors conclude that while model compression introduces small, statistically robust accuracy degradations, it does not necessarily amplify subgroup bias in terms of widening gaps. However, the observed narrowing of fairness gaps (EOD) is a degradation-driven convergence, not an equity improvement.

The paper emphasizes that a naive monitoring of axis-level summary metrics (like EOD) could lead to the false conclusion that compression improved fairness. Therefore, the authors recommend that per-subgroup delta reporting be standard practice when validating compressed clinical models for deployment, rather than relying solely on aggregate fairness metrics. The study also serves as a cautionary tale regarding data quality (outlier filtering) and evaluation protocols in the context of compressed clinical AI.