💬 NLP

Shared Doubt: Zero-shot Cross-Lingual Confidence Estimation for Language Models

This paper demonstrates that a linear probe trained monolingually can effectively estimate confidence in large language models across diverse, unseen languages without retraining, revealing that multilingual LLMs encode shared, transferable confidence features concentrated in their middle layers.

Original authors: Athina Kyriakou, Dennis Ulmer, Ivan Titov

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Athina Kyriakou, Dennis Ulmer, Ivan Titov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, multilingual robot that can answer questions in dozens of languages. But like any human, this robot sometimes makes things up (a problem called "hallucination") without realizing it. We need a way to know when the robot is confident and when it's just guessing.

This paper is about teaching the robot to "trust its gut" across different languages, even if we only trained it to do so in one.

The Core Problem: The "English-Only" Blind Spot

Most research on making AI confident has only looked at English. But the real world is multilingual. The authors noticed a worrying gap: if we don't know how confident the robot is in, say, Japanese or Russian, we might trust a wrong answer just as much as a right one.

Existing methods often fail when you switch languages. It's like having a lie detector that works perfectly on English speakers but breaks completely when you speak French.

The Solution: The "Shared Confidence Room"

The researchers had a hunch based on previous studies: inside these big AI models, there is a "shared room" where different languages meet. Even though the robot speaks French, Russian, and Japanese, the middle part of its brain processes the meaning of words in a similar way, regardless of the language.

They asked: If the robot knows it's confident in French, does it also know it's confident in Russian, even if we never taught it Russian confidence?

The Experiment: The "Lightweight Translator"

To test this, they built a tiny, simple tool called a linear probe. Think of this probe as a very simple translator or a "confidence meter."

Training: They taught this meter using only French data. They showed the robot French questions and answers, and the meter learned to look at the robot's internal "thoughts" (hidden states) to predict: Is this answer right or wrong?
The Test: Then, they turned off the French training and asked the robot to answer questions in Spanish, Polish, Russian, and Japanese.
The Result: The meter, which had never seen Russian or Japanese data, still worked surprisingly well! It could look at the robot's internal thoughts in these new languages and say, "Hey, this answer looks shaky," or "This one looks solid."

The "Middle Layer" Discovery

Where in the robot's brain does this magic happen?
The researchers found that the "confidence signal" lives in the middle layers of the AI.

Analogy: Imagine the AI is a multi-story office building.
- The bottom floors are where the raw words (the alphabet, the script) are processed.
- The top floors are where the final answer is written out.
- The middle floors are the "shared conference room." This is where the robot stops thinking in "French" or "Japanese" and starts thinking in pure "concepts."
- The researchers found that the confidence meter works best when it listens to the people in this middle conference room.

How Well Did It Work?

It's not perfect: The meter worked best on languages similar to French (like Spanish) and got a bit worse on very different ones (like Japanese). This is expected, like how a French speaker might understand Spanish better than Japanese.
It beats the alternatives: Even with this drop-off, their simple "French-trained" meter performed better than many other complex methods that try to guess confidence without retraining.
Calibration vs. Discrimination:
- Discrimination: The meter was good at telling the difference between a right and a wrong answer (like a good lie detector).
- Calibration: It was also good at knowing how confident it should be (e.g., saying "I'm 90% sure" when it's actually 90% right).

The Takeaway

The paper proves that confidence is a universal feature in these AI models. You don't need to retrain the whole robot for every new language to know if it's lying. You just need to tap into that "shared middle room" where all languages overlap.

This is a big step toward making AI safer and more reliable for everyone, not just English speakers, by giving us a way to check the robot's confidence in languages we might not even speak.

Technical Summary: Shared Doubt: Zero-shot Cross-Lingual Confidence Estimation for Language Models

Problem Statement
Confidence estimation (CE) in Large Language Models (LLMs) aims to quantify the reliability of a model's prediction, a critical capability for mitigating hallucinations and maintaining user trust. While CE has garnered significant attention, the vast majority of existing research focuses exclusively on English. This oversight is problematic because LLMs often exhibit reduced reliability in languages with fewer training resources, making accurate confidence estimation even more vital in those contexts. Furthermore, many existing CE methods degrade in performance or require retraining when applied to unseen languages, failing to address the multilingual reality of LLM deployment. The central research question is whether multilingual LLMs encode shared, language-transferable confidence features that can be exploited without target-language supervision.

Methodology
The authors propose a lightweight linear probe designed to predict answer correctness directly from the intermediate hidden representations of an LLM. The methodology relies on the hypothesis that multilingual LLMs possess a shared, language-agnostic subspace where confidence signals are encoded.

Representations: The study utilizes decoder-only transformer-based LLMs. It extracts hidden states from two specific time steps: the last token of the input prompt (serving as a proxy for the entire input) and the last token of the generated answer (representing the final state).
Probe Architecture: The probe learns a weighted average of layer hidden representations. It introduces a learnable parameter vector $w$ to compute layer weights $\alpha_\ell$ via a softmax function, creating a pooled representation $\bar{h}_t$ . This pooled representation is then passed through a linear layer with a sigmoid activation to output a confidence score.
Training and Evaluation Setup: The probe is trained monolingually on a source language (French in the primary experiments, with English as a secondary source) using datasets MKQA (open-ended QA) and Global-MMLU (converted to open-ended). It is then evaluated in a zero-shot setting on typologically diverse target languages (English, Spanish, Polish, Russian, Japanese) without any target-language supervision.
Baselines: The approach is compared against majority class and prior probability baselines, as well as other CE methods including length-normalized sequence likelihood, P(True), verbalized confidence, and the mass-mean probe.

Key Results

Zero-Shot Transfer: The probe, trained solely on French, successfully generalizes to unseen languages across different language families and scripts. While discrimination metrics (AUROC, AUPR) degrade as the typological distance from the source language increases, calibration metrics (Brier score, ECE) often remain stable or even outperform oracle probes trained on the target language in specific instances.
Layer Importance: Analysis of learned layer weights and ablation studies reveals that confidence features are concentrated in the upper-middle layers of the network. This aligns with prior findings that these layers map inputs into a shared representational space. Removing these middle layers significantly impacts performance, particularly for zero-shot calibration.
Token Timing: The study finds a distinction between the utility of the last query token and the last answer token. The last query token yields better calibration across languages, while the last answer token provides superior discrimination between correct and incorrect responses.
Comparison: The cross-lingual probe outperforms several popular CE baselines (such as P(True) and verbalized confidence) in zero-shot settings, particularly in discrimination, though it sometimes trails the mass-mean probe in calibration for highly dissimilar languages.

Significance and Contributions
The paper claims to be the first to demonstrate zero-shot cross-lingual transfer of confidence estimation through the probing of intermediate LLM representations. Its primary contributions are:

Demonstration of Shared Subspace: It provides empirical evidence that confidence signals learned in a high-resource language can transfer to low-resource or typologically distant languages without retraining, suggesting the existence of a shared confidence subspace in multilingual LLMs.
Layer Localization: It identifies that the most salient information for zero-shot confidence prediction is localized in the middle-to-upper layers, reinforcing the theory that these layers contain language-agnostic representations.
Practical Baseline: The proposed method offers a strong, retraining-free baseline for multilingual confidence estimation, which is particularly valuable for underserved languages where labeled data is scarce.

The authors maintain a modest tone, acknowledging that performance degrades with increasing typological distance and that the approach currently focuses on short-form QA. They suggest that future work could involve training on multiple source languages, identifying critical latent representations more precisely, and applying post-hoc calibration techniques to further improve robustness.