💬 NLP

Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation

This paper introduces and evaluates five verbalized methods for extracting per-token confidence from large language models in machine translation, finding that they perform comparably to internal signals in error detection and calibration despite showing little correlation with them.

Original authors: Ali Marashian, Alexis Palmer, Katharina von der Wense

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Marashian, Alexis Palmer, Katharina von der Wense

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, multilingual robot translator. You ask it to translate a sentence, and it does so instantly. But how do you know if it got it right? Does the robot know if it made a mistake?

This paper investigates a specific question: Can we ask the robot, "How sure are you about this specific word?" and trust its answer?

The researchers compared two ways of getting the robot's "confidence level":

The "Internal" Way: Looking at the robot's secret math (its internal probabilities). This is like peeking at the robot's brain to see how many options it was considering before it picked a word.
The "Verbalized" Way: Simply asking the robot, "On a scale of 0 to 1, how confident are you in this word?" and having it type out the answer.

Here is a breakdown of their findings using simple analogies:

1. The Problem with "Peeking at the Brain"

Usually, when we want to know if a translation is good, we look at the robot's internal math. If the robot was 99% sure it picked the right word, we assume it's correct.

However, the paper points out a flaw in this method called "Surface Form Competition."

The Analogy: Imagine you are translating "The cat sat on the mat." You could also say "The feline sat on the rug." Both are perfect translations.
The Issue: If the robot is torn between "cat" and "feline," its internal math might show it's only 50% sure about "cat" because it's also thinking about "feline." But "cat" is actually a correct translation!
The Result: The robot's internal math says, "I'm unsure," but the output is actually perfect. The internal signal is misleading because it measures confusion between options, not correctness.

2. The New Idea: Just Ask It

Since the robot can talk, the researchers tried a different approach: Verbalized Confidence. Instead of peeking at the math, they prompted the robot to say, "I am 90% sure this word is right."

They tested five different ways of asking:

The List: "Tell me which parts of the sentence are wrong."
Word Numbers: "Give every word a score from 0 to 1."
Word Words: "Give every word a label like 'Very Certain' or 'Not Sure'."
Token Numbers/Words: Doing the same thing but for tiny pieces of words (tokens) instead of whole words.

3. The Big Surprise: They Don't Agree

The researchers expected that if the robot's internal math said "I'm unsure," and they asked the robot "Are you unsure?", the robot would say "Yes."

They were wrong.

The Analogy: It's like a weather forecaster who looks at their complex radar data (Internal) and says, "There's a 50% chance of rain," but when you ask them directly, "Do you think it will rain?", they say, "I'm 90% sure it won't."
The Finding: There was almost no correlation between the robot's internal math and what it said out loud. They were essentially two different strategies that didn't really talk to each other.

4. Did Asking Help?

So, if they don't agree, which one is better at spotting actual translation errors?

The Results: Surprisingly, asking the robot (Verbalized) worked just as well as, and sometimes better than, peeking at the math (Internal).
The Catch: It depends on which robot you use.
- For one model (Llama3), asking it to give a "Likert" score (e.g., "Very Certain") worked best.
- For another model (Aya23), looking at the internal math (Entropy) worked best.
The Reality Check: Even the best method wasn't perfect. The robots were still only moderately good at spotting their own mistakes. It's like a student taking a test who is pretty good at knowing which answers are wrong, but still makes a few mistakes they don't realize.

5. Why This Matters (According to the Paper)

The paper concludes that we don't need to be "hackers" looking inside the robot's code to get a confidence score. We can just ask it.

Accessibility: You don't need access to the robot's internal "brain" (which is often hidden in closed-source models). You just need to be able to chat with it.
Granularity: The researchers found that even when they asked for a precise number (like 0.83), the robots tended to round their answers to simple steps (like 0.8 or 0.9), acting more like they were using a simple checklist than a high-precision calculator.

Summary

The paper is a test of whether we can trust a Large Language Model's own words about its confidence.

Old Way: Look at the math (Internal).
New Way: Ask the model (Verbalized).
Verdict: They are totally different things, but asking the model works just as well as looking at the math for finding translation errors. It's a simpler, more accessible way to check if a translation is likely to be bad, without needing to see the model's secret code.

Technical Summary: Speaking in Self-Assessing Tongues

Problem Statement

The rapid adoption of Large Language Models (LLMs) for Machine Translation (MT) has highlighted a critical reliability issue: LLMs are poorly self-calibrated for translation tasks. Unlike classification tasks where a single correct answer exists, translation involves "surface form competition," where multiple valid syntactic or semantic realizations compete. Consequently, internal signals such as token probabilities or entropy often reflect uncertainty among competing valid alternatives rather than the correctness of the selected output. Furthermore, reliance on internal signals requires access to model internals, which is often unavailable for closed-source models. There is a need for robust, unsupervised methods to estimate per-token confidence that do not suffer from these limitations.

Methodology

The authors propose and evaluate five verbalized methods for extracting per-token confidence directly from the model's generated text, avoiding the need for internal access. These methods are compared against two standard internal uncertainty measures: token-level probability and entropy.

Experimental Setup

Datasets: The study utilizes WMT2024 General Machine Translation data (English to Chinese, Czech, Hindi, Japanese, and Russian) paired with Error Span Annotations (ESA) from Kocmi et al. (2024a) as ground truth.
Models: Two open-source LLMs were selected: Aya23 and Llama3-70B.
Verbalized Approaches:
1. List: The model returns a list of spans it is not confident about.
2. Word_Numeric: The model assigns a numerical confidence score $[0, 1]$ to each word via separate prompts.
3. Word_Likert: The model assigns qualitative labels (e.g., "very certain" to "very uncertain") converted to numerical values.
4. Token_Numeric: Similar to Word_Numeric but applied at the token level.
5. Token_Likert: Similar to Word_Likert but applied at the token level.
Evaluation Metrics:
- Binary Error Detection: Measured via F1 score (specifically for the incorrect class), AUROC, and AUPRC.
- Calibration: Measured via Expected Calibration Error (ECE) and reliability diagrams.
- Alignment: Spearman's rank correlation was used to measure the relationship between verbalized confidence and internal signals.

Key Results

1. Error Detection Performance

Overall Performance: All methods yielded low F1 scores (ranging from 0.02 to 0.23), indicating the inherent difficulty of the task, though all performed better than random baselines.
Model Dependency: No single method consistently outperformed others across all settings. Performance varied significantly by language pair and model.
- For Aya23, internal measures (Entropy) and Word_Numeric performed best on average.
- For Llama3-70B, Word_Likert achieved the best performance in 4 out of 5 language pairs.
Comparison: Verbalized methods generally performed comparably to or better than internal methods for Llama3-70B, but results were mixed for Aya23. Notably, in English-to-Japanese translation with Llama3-70B, verbalized methods outperformed internal methods by 0.09 F1.

2. Calibration

Internal Superiority: For calibration (ECE), normalized Entropy consistently yielded the best results for both models.
Verbalized Performance: Verbalized methods performed similarly to model probability but lagged behind model entropy.
Likert vs. Numeric: Likert-based methods consistently outperformed their numeric counterparts in calibration. The authors attribute the poor calibration of numeric methods to known issues where LLMs struggle with numerical contexts.

3. Alignment Between Methods

Low Correlation: A primary finding is the lack of significant correlation between verbalized confidence and internal uncertainty measures (probability and entropy). This suggests that the two strategies (internal calculation vs. verbal self-assessment) operate on different principles or capture different aspects of the model's state.
Granularity: Despite the ability to return arbitrary values, numeric methods exhibited limited granularity, with outputs often restricted to multiples of 0.1 or 0.2, effectively behaving similarly to Likert scales.

4. Granularity and Syntax

Token vs. Word: Token-level methods showed different behaviors compared to word-level methods, particularly regarding recall and precision trade-offs.
Syntactic Categories: Verbalized methods showed higher recall for open-class words (nouns, verbs) compared to closed-class words for Llama3-70B, a pattern not observed in internal methods.

Significance and Claims

The paper claims that verbalized confidence is a viable, accessible alternative to internal uncertainty signals for MT error detection, particularly for users without access to model internals.

Accessibility: Verbalized methods provide a "black-box" solution for confidence estimation, requiring only the model's output text.
Complementarity: The finding of little to no correlation between verbalized and internal methods implies that they are not redundant; they may capture distinct signals of model behavior.
Limitations: The authors modestly note that the absolute performance of all methods remains relatively low, partly due to the high quality of translations in high-resource languages and the difficulty of the task. They emphasize that these unsupervised methods are not replacements for human annotation but serve as a supplementary quality control mechanism.
Future Directions: The authors suggest future work should investigate the factors driving the alignment (or lack thereof) between internal and verbalized signals and explore the generalizability of these findings to low-resource language scenarios where human annotation is scarce.