Distilling Self-Consistency into Verbal Confidence: A Pre-Registered Negative Result and Post-Hoc Rescue on Gemma 3 4B

1. Problem Statement

Large Language Models (LLMs), particularly those in the 3–9B parameter range, exhibit a "degenerate verbal confidence" problem. When asked to state their confidence as a percentage, these models consistently report near-maximum confidence (ceiling rates >90%) regardless of whether their answer is correct.

• The Gap: While internal representations (hidden states) and single-pass logit entropy contain significant information about correctness (AUROC2 ~0.6–0.8), the verbal channel fails to transmit this information (AUROC2 ~0.5, near chance).
• The Goal: The study investigates whether Confidence-Conditioned Supervised Fine-Tuning (CSFT) can distill the information from multi-sample self-consistency (a robust but computationally expensive method) into a reliable, single-pass verbal confidence signal.

2. Methodology

The study utilized Gemma 3 4B-it and followed a rigorous psychometric evaluation pipeline based on the CMM programme (Type-2 Signal Detection Theory).

Experimental Design

• Data: Items drawn from TriviaQA (calibration/training and held-out evaluation) and MMLU (cross-benchmark transfer).
• Target Derivation: For each training item, 10 samples were generated at temperature $T=0.7$. The number of correct samples ($n_{correct}$) was mapped to a confidence target (e.g., $n_{correct}=10 \rightarrow 95\%$, $n_{correct}=0 \rightarrow 5\%$).
• Training: LoRA fine-tuning (rank 16) on the chat prompt, training the model to output the answer followed by the derived confidence percentage.
• Controls: A shuffled-target control was trained with identical architecture but randomly permuted confidence labels to distinguish between learning the format vs. learning the signal.

Two Phases

1. Pre-Registered Protocol (Phase 0): Included a modal filter, restricting training data to items where the most frequent answer across 10 samples was correct.
2. Post-Hoc Rescue: The modal filter was removed, allowing the model to train on all 2,000 calibration items, including those with incorrect modal answers (bimodal distribution).

3. Key Results

A. Pre-Registered Result (Negative)

• Outcome: The protocol failed.
• Metrics: Verbal AUROC2 dropped from 0.554 (baseline) to 0.509. Accuracy degraded by 7.6%. The Validity Rating Scale (VRS) remained "Invalid."
• Diagnosis: Label-Entropy Collapse. The modal filter removed all items where the model was uncertain or wrong ($n_{correct} < 10$). The resulting training set had near-uniform high-confidence targets (mostly 95%). The model learned to ignore the input and output the mode (95%), failing to learn when to express low confidence.

B. Post-Hoc Rescue (Exploratory Positive)

By removing the modal filter, the training set included a bimodal distribution: 48.2% of items had $n_{correct}=10$ (target 95%) and 36.5% had $n_{correct}=0$ (target 5%).

• TriviaQA Performance:
  • AUROC2: Improved significantly from 0.554 (baseline) to 0.774.
  • Ceiling Rate: Collapsed from 97.7% to 49.8%, indicating the model learned to vary its confidence.
  • Nature of Signal: The model learned a binary discriminator (outputting either 5% or 95%) rather than continuous calibration. It successfully compressed a 10-sample self-consistency signal (AUROC2 = 0.999) into a single-pass verbal readout, recovering ~77.5% of the discrimination power.
  • Comparison: The distilled verbal signal (0.774) outperformed single-pass logit entropy (0.701).
• MMLU Cross-Benchmark Transfer:
  • Accuracy: Improved dramatically from 54.2% to 77.4%.
  • AUROC2: Improved from 0.535 to 0.616.
  • Control: The shuffled-target model showed no improvement (accuracy 56.1%, AUROC2 0.523) and produced verbose, unparseable outputs, confirming the gains were due to the content of the confidence targets, not just the LoRA procedure.

4. Key Contributions & Design Principles

1. Label Entropy is Critical: The study identifies label-entropy collapse as a primary failure mode in confidence training. To train a model to express uncertainty, the training set must include examples of uncertainty (low-confidence targets). Filtering out "hard" or "wrong" examples guarantees failure.
2. Distillation of Self-Consistency: It is possible to distill multi-sample self-consistency information (usually requiring 10 forward passes) into a single-pass verbal signal. While there is a compression loss (~22.5%), the resulting signal is more informative than standard logit entropy and accessible via standard APIs.
3. Format Regularization: Training on correct confidence targets appeared to regularize the output format. The real-target model produced concise, parseable responses, whereas the shuffled-target model produced verbose, unstructured text. This suggests confidence targets may act as a regularizer for output structure.
4. Accuracy-Confidence Trade-off: The intervention improved discrimination but caused a 7.5% drop in TriviaQA accuracy. This suggests the model adopted a new "answer-confidence policy" that prioritized discriminative signaling over raw answer generation, or that the confidence signal helped filter out confabulations on other benchmarks (MMLU).

5. Significance and Limitations

• Significance: This work provides a practical pathway to enable LLMs to "speak" their uncertainty reliably without requiring access to internal logits or multiple sampling passes. It highlights that the "verbal channel" is trainable but requires specific data distributions (high entropy) to function.
• Limitations:
  • Binary Output: The model learned a binary classifier (5% vs. 95%) rather than continuous calibration (e.g., 60% confidence).
  • Post-Hoc Nature: The positive result was not pre-registered and requires replication.
  • Single Model/Seed: Results are specific to Gemma 3 4B and a single random seed.
  • Accuracy Drop: The improvement in MMLU accuracy and the drop in TriviaQA accuracy suggest a complex interaction between confidence training and answer generation policies that requires further ablation studies.

Conclusion

The paper demonstrates that while small LLMs naturally fail to report uncertainty, this can be remedied via CSFT if the training data preserves label entropy. By training on the full spectrum of self-consistency outcomes (including failures), the model can distill complex multi-sample reasoning into a reliable, single-pass verbal confidence signal that outperforms standard entropy baselines and improves performance on cross-benchmark tasks like MMLU.