💬 NLP

Quantisation Reshapes the Metacognitive Geometry of Language Models

This paper demonstrates that model quantization fundamentally reshapes domain-level metacognitive efficiency profiles in LLMs by altering M-ratio normalizations without affecting underlying discrimination signals, thereby revealing a critical dependency of confidence-based diagnostics on inference format that cannot be resolved through domain-conditional training.

Original authors: Jon-Paul Cacioli

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jon-Paul Cacioli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: A Case of "Wrong Glasses"

Imagine you are a coach trying to help a student improve at different subjects: Math, History, Art, and Geography.

You want to know which subject the student is bad at monitoring. In other words, when they get a question wrong, do they realize they are wrong? Or do they confidently say the wrong answer?

The researcher in this paper tried to fix the student's "bad monitoring" by giving them extra practice specifically on that subject. But before they could start, they realized something strange: The student looked like a genius in one pair of glasses, but a disaster in another pair of glasses.

The "glasses" in this story are Quantisation. In AI, this is a way of shrinking a model's brain to make it run faster and use less memory. It's like compressing a high-definition movie into a lower-quality stream. Usually, we think this just makes the picture slightly grainier. This paper discovered that for AI confidence, it actually changes the plot of the movie entirely.

The Experiment: The "Prescribe-Don't-Average" Plan

The researcher had a brilliant idea called "Prescribe-Don't-Average."

The Old Way: Give the AI generic training to fix its confidence everywhere.
The New Way: Diagnose exactly which subject is broken, then prescribe a specific "medicine" (training) just for that subject.

The Plan:

Diagnose: Check the AI's confidence on 3,000 questions across four topics (Science, Geography, History, Arts).
Prescribe: The AI was "worst" at Science (it was confident but wrong). So, the researcher gave it extra training specifically on Science questions to make it more humble and accurate.
Evaluate: Test the AI again to see if the training worked.

The Result: The training failed. The AI didn't get better at monitoring its own Science answers.

Why? The researcher realized they made a fatal mistake in the measurement. They diagnosed the problem using Glasses A (a compressed, low-precision format called Q5_K_M), but they tested the results using Glasses B (a full-precision format called f16).

The Shocking Discovery: The "Metacognitive Geometry" Shift

When the researcher compared the two sets of glasses on the exact same 3,000 questions, they found something bizarre:

In Glasses A (Compressed): The AI was terrible at monitoring Arts (it was overconfident and wrong) but good at Geography.
In Glasses B (Full Precision): The AI was suddenly terrible at Geography but amazing at Arts.

The Ranking Swapped Completely.
It's as if you asked a person, "Who is the fastest runner?"

In the morning, they say: "Alice is the fastest."
In the afternoon, they say: "Bob is the fastest."
But when you actually time them running, Alice is still faster than Bob. The order didn't change, but the score they gave themselves flipped upside down.

The Technical Explanation (Simplified):
The paper found that the AI's raw ability to tell right from wrong (called AUROC) stayed exactly the same. The AI knew the answers just as well in both formats.
However, the metric used to measure "confidence efficiency" (called M-ratio) depends on a mathematical division. The compressed format changed the "difficulty score" of the questions differently for each subject.

Arts became "easier" in the compressed format, so the AI looked overconfident.
Science became "harder," so the AI looked under-confident.

Because the math divides confidence by difficulty, the ranking of the subjects flipped. The "shape" of the AI's confidence (its geometry) was reshaped by the compression.

The Lesson: Don't Trust the Scoreboard

The researcher tried to fix the AI by training it on Science because the "compressed glasses" said Science was the weak link. But when they tested it in "full precision," the weak link was actually Geography!

The training failed because the diagnosis was taken from the wrong reality.

Key Takeaways for Everyone

Compression Changes the Story: When we shrink AI models to make them faster, we aren't just making them "grainy." We are subtly changing how they perceive the difficulty of different topics.
The Metric Matters: If you use a specific math formula (M-ratio) to check if an AI is honest, you must use the exact same format for checking and for using the AI. If you switch formats, the scoreboard lies.
Better Tools Exist: There is a different way to measure confidence (AUROC) that is like a "rock-solid" ruler. It doesn't change when you switch glasses. The paper suggests we should use this "rock-solid" ruler for diagnosing AI problems.

The "Honest" Conclusion

The researcher didn't find a way to make the AI smarter. Instead, they found a trap. They showed us that if we rely on certain measurements to fix AI, we might be fixing the wrong problem because the measurement itself is unstable.

It's a reminder that in the world of AI, how you measure something is just as important as the thing you are measuring. If you measure with a ruler that stretches and shrinks, you'll never know if your house is actually growing or just the ruler that's changing.

1. Problem Statement

The paper addresses a critical gap in Large Language Model (LLM) metacognition research: the assumption that diagnostic metrics for model confidence are invariant across different inference formats.

Context: Previous work (Cacioli, 2026a) established that metacognitive efficiency (the ability of a model to distinguish correct from incorrect answers based on confidence) varies significantly across knowledge domains within a single model.
The Gap: While model quantisation (reducing numerical precision to save memory/compute) is known to affect accuracy and calibration levels, it was unknown whether quantisation alters the structural ordering of metacognitive efficiency across domains.
The Hypothesis: The author attempted to apply the "Prescribe-Don't-Average" principle: diagnose the weakest domain, train specifically on it, and improve metacognition. However, the intervention failed, leading to the discovery that the diagnostic metric itself was unstable across quantisation formats.

2. Methodology

The study employed a pre-registered experimental design involving the Llama-3-8B-Instruct model.

A. Experimental Design

Pre-registration: Two pre-registrations were filed on OSF before training.
Intervention: The authors attempted to improve metacognition in the "Science" domain (diagnosed as weak in a pilot) using Domain-Conditional Supervised Fine-Tuning (SFT) with confidence amplification.
Control Conditions:
1. Baseline: No intervention.
2. Domain-Agnostic SFT: Trained on all domains with matched data budget.
3. Wrong-Prescription SFT: Trained on "Geography" (a strong domain) to test causal specificity.
4. Low-Learning-Rate SFT: A variant of the target intervention.
Data: 3,000 TriviaQA questions (Set C1) used for evaluation, classified into four domains: Arts, Geography, History, and Science.

B. The Quantisation Mismatch (The Critical Flaw)

Diagnosis (M1): Metacognitive profiles were initially computed using Q5_K_M (5-bit) quantisation.
Evaluation (M2): Due to technical constraints in merging LoRA adapters, the adapted models were evaluated in f16 (16-bit float) precision.
Post-Hoc Analysis: To isolate the cause of the null results, the authors evaluated the unadapted baseline model on the exact same 3,000 questions at both Q5_K_M and f16 to compare metrics directly.

C. Metrics

Type-2 Signal Detection Theory: Used to compute Meta-d′ (metacognitive sensitivity) and M-ratio (Meta-d′ / d′, normalizing for task difficulty).
AUROC2: Type-2 Area Under the Receiver Operating Characteristic curve, measuring raw discrimination ability without normalizing for difficulty.
NLP Gap: Mean log-probability difference between correct and incorrect answers.

3. Key Results

A. The Null Intervention

All four confirmatory hypotheses regarding the improvement of meta-d′ in the Science domain were null.

The targeted SFT (Condition 2) failed to improve meta-d′ compared to the baseline or controls.
Interestingly, the training did successfully double the NLP gap in Science (from 0.076 to 0.152), indicating the model learned to separate confidence distributions, but this did not translate to improved metacognitive sensitivity (meta-d′).

B. The Core Discovery: Metacognitive Geometry Restructuring

The post-hoc analysis on the baseline model revealed a profound effect of quantisation on metacognitive profiles:

M-ratio Instability: The ranking of domains by metacognitive efficiency completely inverted between Q5_K_M and f16.
- Arts: Ranked 4th (worst) at Q5_K_M (M-ratio = 0.606) $\rightarrow$ Ranked 1st (best) at f16 (M-ratio = 1.542).
- Geography: Ranked 2nd at Q5_K_M (M-ratio = 1.210) $\rightarrow$ Ranked 3rd at f16 (M-ratio = 0.798).
- Correlation: The Spearman rank correlation between M-ratio profiles across formats is $\rho = 0.00$ .
AUROC2 Stability: In contrast, the raw discrimination signal (AUROC2) remained perfectly stable.
- The ranking of domains (Arts > History > Geography > Science) was identical at both quantisation levels.
- Correlation: $\rho = 1.00$ .

C. Mechanism of Failure

The "restructuring" is an artifact of the normalization process in the M-ratio calculation ( $M = \text{meta-d}' / d'$ ).

Quantisation shifts the d′ (task performance) non-proportionally across domains.
- Example: Arts d′ dropped by 37% (0.891 $\to$ 0.559) when moving to f16, while Science d′ increased by 26% (0.365 $\to$ 0.461).
Because M-ratio divides by d′, these differential shifts in task performance amplify into a complete restructuring of the efficiency profile, even though the underlying ability to discriminate correct from incorrect answers (AUROC2) remained unchanged.

4. Key Contributions

Discovery of Format-Dependent Metacognitive Geometry: The paper demonstrates that the "shape" of metacognitive efficiency across domains is not an intrinsic property of the model but is highly dependent on the inference format (quantisation level).
Dissociation of Discrimination vs. Normalization: It establishes a clear dissociation between Type-2 discrimination (stable across formats, measured by AUROC2) and Metacognitive Efficiency (unstable, measured by M-ratio).
Failure of Prescribe-Don't-Average in Mixed Formats: It shows that domain-specific interventions fail if the diagnosis and evaluation formats differ, as the "weak" domain identified in one format may be a "strong" domain in another.
NLP Gap vs. Meta-d′ Dissociation: The study proves that increasing the separation of confidence distributions (NLP gap) does not necessarily improve metacognitive sensitivity (meta-d′), highlighting the complexity of calibration training.

5. Significance and Implications

Metric Selection: For systems relying on domain-level diagnostics, M-ratio is unsafe if diagnosis and deployment use different quantisation levels. AUROC2 is recommended as a format-stable alternative for ranking domain performance.
Operational Requirements: If M-ratio must be used for theoretical reasons, strict format consistency between the diagnostic phase and the deployment phase is mandatory.
Training Strategy: The "Prescribe-Don't-Average" framework is theoretically sound but practically fragile; it requires that the diagnostic metric used to select the training target is invariant to the deployment environment.
Future Research: The mechanism behind domain-differential d′ shifts under quantisation (e.g., attention head sensitivity or token distributions) remains an open question.

Conclusion

The paper concludes that quantisation does not merely degrade model performance uniformly; it reshapes the metacognitive geometry of the model. The apparent "weakness" of a domain in one format can be an artifact of the quantisation-induced shift in task difficulty (d′), which distorts the normalized efficiency metric (M-ratio). Consequently, any system diagnosing and acting on domain-specific metacognition must ensure metric stability across inference formats to avoid misdirected interventions.