🤖 machine learning

Quantization Effects on Biomedical LLM Reliability

This study demonstrates that for biomedical decoder language models, prompt template design and probability scoring protocols exert a dominant influence on calibration and accuracy—often surpassing the effects of model architecture or quantization—highlighting the critical need to standardize these implementation choices in experimental evaluations.

Original authors: Anton Rasmussen, Hong Qin

Published 2026-08-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Anton Rasmussen, Hong Qin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to read medical journals. These journals are full of important information about new treatments and diseases, but they are written in a confusing mix of paragraphs. Doctors need to quickly sort these sentences into neat categories like "Background," "Methods," or "Results" to find the truth. This is a huge job for humans, so scientists are trying to use Artificial Intelligence (AI) to do it instead.

But here's the tricky part: AI isn't just about getting the right answer; it's about knowing how sure it is. If a robot says, "I am 100% certain this drug cures cancer," but it's actually wrong, that's dangerous. It creates a false sense of security. Scientists call this "calibration." A well-calibrated robot is honest: if it's only 60% sure, it should say so. However, measuring this honesty is like trying to weigh a feather on a scale that changes its own rules every time you press a button. The way you ask the robot a question, the way you listen to its answer, and even how you crunch the numbers to get a score can completely change whether the robot looks honest or crazy. This paper dives deep into that messy reality to see if we can trust these medical robots when they are running on cheaper, faster hardware.

The Great Robot Calibration Heist

In this study, the researchers treated three different versions of a powerful AI brain (called Mistral-7B) like contestants in a game show. They wanted to see which one was the most honest and accurate at sorting medical sentences. The contestants were:

The Base Model: A raw, untrained brain.
The BioMistral: A brain that read millions of medical papers to learn the lingo.
The Instruct Model: A brain that was taught how to follow commands and chat.

They tested these brains under three different "energy modes": the standard high-power mode (FP16), a medium-power mode that saves memory (INT8), and a super-squeezed mode (INT4) that fits on tiny computers.

The "How You Ask" Trap

The biggest surprise in the paper is that the robot's "honesty score" depends entirely on how you ask the question. The researchers tried two main ways to score the answers:

The "Total Points" Method: You add up every little piece of confidence the robot gives for the whole answer.
The "Average Points" Method: You take the average confidence per word.

Here is the twist: Switching between these two methods completely flipped the leaderboard.
When using the "Total Points" method, the Instruct model looked terrible at being honest (it was overconfident), while BioMistral looked great. But when they switched to the "Average Points" method, the Instruct model suddenly looked like the most honest one, and BioMistral looked overconfident.

It's like judging a student's test score by adding up every point they got (Total) versus averaging their score per question (Average). One student might have a few perfect answers and many blanks, while another has consistent B-pluses. Depending on which math rule you use, you might declare a different winner. The paper shows that for these medical robots, the math rule you choose matters more than which robot you picked.

The "Prompt" Rollercoaster

The researchers also found that the style of the question (the "prompt") caused wild swings in accuracy. They tried four different ways to frame the task, from very structured with fancy borders to bare-bones text.

For the Base Model, changing the prompt style caused accuracy to jump by 7 to 24 percentage points. That's a huge difference! It's like if a student got an 80% on a test when the teacher wrote the instructions in blue ink, but only a 55% when the instructions were in red ink.
Sometimes, the BioMistral model was the best with one prompt, but the Instruct model was the best with another. There was no single "best" robot; the winner changed based on the outfit the robot was wearing.

The "Squeezed" Hardware Test

Since hospitals and clinics might not have super-expensive computers, the researchers tested what happens when they squeeze the robots into smaller, compressed formats (INT8 and INT4).

Good News: For the specialized medical robots (BioMistral and Instruct), squeezing them into INT8 mode barely changed their accuracy or honesty. They stayed within 1–2 percentage points of their full-power selves. It's like putting a heavy backpack on a runner; they might slow down a tiny bit, but they can still finish the race just as well.
Mixed News: When they squeezed them even harder into INT4 mode, the results were a bit chaotic. Sometimes the accuracy went up slightly, sometimes it went down, but it didn't cause a total disaster. The base model, however, was more sensitive and showed bigger changes.

The "One-Letter" Disaster

Before settling on the final method, the researchers tried a shortcut. They asked the robots to just answer with a single letter (A, B, C, D, or E) instead of writing out the full word (like "Background" or "Methods").
This failed miserably. The robots started guessing the same letter over and over again, ignoring the actual medical content. It was as if the robot was just guessing "A" because it saw the letter "A" most often in its training data. This proved that you can't just use short codes; you have to let the robot write out the full answer to get a real reading of its brain.

The Bottom Line

The main takeaway from this paper is that when we try to measure how reliable these medical AI robots are, we have to be very careful about the rules we use.

The "honesty" of the robot isn't just a fixed trait inside the machine; it's a result of how we measure it. If you change the scoring math or the question style, you might think one robot is a genius and another is a liar, when in reality, they might be the same. The paper suggests that before we trust these robots to help doctors, we need to test them with many different question styles and scoring methods, not just one.

Also, while squeezing these robots onto smaller computers (INT8) seems safe for the specialized medical versions, we can't just assume it works for everyone. We have to check each specific setup. The researchers didn't find a magic bullet that fixes everything, but they did find a map of all the traps we need to avoid so we don't get fooled by a robot that looks confident but is actually just guessing.

Technical Summary: Quantization Effects on Biomedical LLM Reliability

Problem Statement

Systematic reviews are critical to evidence-based medicine, yet manual screening of research abstracts remains a bottleneck. Automated sentence classification—assigning functional roles (e.g., BACKGROUND, METHODS, RESULTS) to sentences in medical abstracts—offers a solution. While Large Language Models (LLMs) are natural candidates for this task, their deployment in healthcare-adjacent pipelines requires not only accuracy but also well-calibrated confidence estimates. A model that is highly accurate but overconfident in its errors poses significant risks to downstream meta-analyses and clinical guidelines.

Furthermore, resource-constrained environments necessitate model compression techniques like INT8 and INT4 quantization. However, the impact of these quantization methods on calibration quality, particularly for biomedical LLMs, is poorly understood. Crucially, apparent calibration metrics are often entangled with implementation choices—specifically the prompt template, verbalizer (label-to-token mapping), and scoring rule—which are rarely treated as experimental variables. This paper investigates how these measurement choices, alongside quantization, influence the reliability and calibration of biomedical decoders.

Methodology

Models and Dataset

The study evaluates three 7B-parameter decoder variants of Mistral on the PubMed RCT sentence classification task (5 classes: BACKGROUND, OBJECTIVE, METHODS, RESULTS, CONCLUSIONS) using a stratified sample of $n=2,000$ test examples:

Mistral-7B-v0.3 (Base): A generic decoder with no task-specific or instruction tuning.
BioMistral-7B: A model continued-pretrained on PubMed Central articles, inheriting instruction tuning from its parent (Mistral-7B-Instruct-v0.1).
Mistral-7B-Instruct-v0.3 (Instruct): The instruction-tuned variant of the base model.

A fine-tuned encoder, PubMedBERT, serves as a supervised reference point (trained on $\sim$ 176k examples), though results are noted as not directly comparable due to differences in training data and evaluation distribution.

Experimental Variables

The evaluation systematically varies three factors:

Precision: FP16 (baseline), INT8 (LLM.int8() mixed-precision), and INT4 (NF4 with double quantization).
Prompt Templates: Four answer-text templates were used:
- Struct-Delim and Bare-Cont: Discovered via evolutionary prompt search on a development set.
- Min-Inst and Alt-Frame: Independently designed to test sensitivity to instruction verbosity and lexical framing.
Scoring Protocols: Two methods for extracting probabilities from the decoder:
- Summed Log-Likelihood: Sum of log-probabilities over the label tokens.
- Mean-Token Log-Likelihood: Average log-probability per token (length-normalized).

Calibration and Metrics

Calibration was assessed using Expected Calibration Error (ECE), Brier score, and Negative Log-Likelihood (NLL). Post-hoc temperature scaling was applied to a held-out validation set ( $n=200$ ) to minimize NLL. The study also evaluated the failure modes of single-token class-code scoring versus full answer-text scoring.

Key Contributions

Scoring-Rule Sensitivity: The paper demonstrates that the choice of scoring protocol (summed vs. mean-token log-likelihood) dominates apparent calibration rankings. Switching protocols reverses the calibration hierarchy between BioMistral and Instruct models while leaving accuracy largely unchanged.
Template Variation Impact: Prompt template choice induces accuracy swings of 7–24 percentage points, a magnitude comparable to or exceeding the differences between model variants. This suggests template design is a first-order experimental decision, not an implementation detail.
Quantization Deltas: For specialized models (BioMistral and Instruct), INT8 quantization results in accuracy and F1 changes within 1–2 percentage points of FP16. INT4 (NF4) shows heterogeneous but non-catastrophic effects, with specific improvements in ECE for BioMistral on the Bare-Cont template. The base model exhibits larger sensitivity to quantization on specific templates.

Results

Calibration and Accuracy Tradeoffs

Under summed log-likelihood scoring, the Instruct model appeared significantly overconfident (Mean ECE = 0.237) compared to BioMistral (Mean ECE = 0.097). However, under mean-token scoring, this ranking reversed: Instruct's ECE dropped to 0.096, while BioMistral's rose to 0.289. Accuracy remained stable across these scoring changes for specialized models (<1% difference), but the base model lost 4–6 percentage points under mean scoring.

Quantization Effects

Specialized Models (BioMistral/Instruct): INT8 quantization produced accuracy deltas $\le$ 1 percentage point and F1 deltas $\le$ 2.1 percentage points. In several cases, INT8 marginally improved performance over FP16.
Base Model: Showed larger sensitivity, with INT8 accuracy shifts up to +4.2 percentage points on the Struct-Delim template.
INT4 (NF4): Effects were heterogeneous. For BioMistral, INT4 improved ECE specifically on the Bare-Cont template (0.063 vs. 0.104 at FP16) while maintaining accuracy; on Struct-Delim, effects were mixed. For Instruct, INT4 caused a modest accuracy drop on Bare-Cont with slightly worse ECE. No catastrophic degradation was observed in the tested configurations.

Temperature Scaling

Post-hoc temperature scaling reduced ECE under summed scoring for both models. Fitted temperatures were higher for Instruct ( $T = 2.2–4.6$ ) than BioMistral ( $T = 1.3–1.7$ ). However, the authors note this finding is specific to the summed scoring rule and does not generalize to mean-token scoring.

Failure of Single-Token Scoring

Initial attempts to use single-token class codes (A–E) resulted in label collapse (macro-F1 near chance) for base and continued-pretrained models, driven by positional and token-frequency biases. Switching to answer-text scoring (generating full label names) rescued classification performance, highlighting the critical role of the verbalizer.

Significance and Claims

The paper argues that probability-extraction protocols are the primary drivers of apparent calibration differences in decoder-based biomedical NLP, often overshadowing model architecture or training history.

Methodological Implication: Calibration comparisons between models are not interpretable without explicitly specifying and varying the scoring protocol. A model deemed "overconfident" under one scoring rule may appear well-calibrated under another.
Experimental Design: Prompt-template design must be treated as a primary experimental variable. Accuracy variations caused by template choice can exceed the performance gains from model selection or instruction tuning.
Deployment Safety: While INT8 quantization appears safe for specialized biomedical decoders (with minimal accuracy loss), the heterogeneous effects of INT4 and the sensitivity of the base model suggest that equivalence testing with predeclared margins is necessary before deployment.
Limitations: The authors caution that their findings are specific to the PubMed RCT task and the tested quantization methods (bitsandbytes). The comparison between BioMistral and Instruct is confounded by differences in base versions and pretraining history, and the temperature scaling results are tied to a specific scoring rule.

In conclusion, the study underscores that reliable evaluation of biomedical LLMs requires rigorous control over prompt templates, verbalizers, and scoring normalization, rather than relying on a single protocol to judge model calibration.