💬 NLP

When Readability and Source Retention Diverge: An Evaluability Gap in AI Translation

This study reveals that while readable AI translations often create an "evaluability gap" where users fail to recognize fidelity differences even when the source text is visible, their trust and willingness to disclose personal information are more strongly driven by perceived task performance and anthropomorphic attribution than by objective content retention.

Original authors: Chenchen Mao, Hanjing Shi, Haiyan Jia, Emily Wegrzyn, Dominic DiFranzo

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenchen Mao, Hanjing Shi, Haiyan Jia, Emily Wegrzyn, Dominic DiFranzo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet corners of our digital lives, we constantly hand over pieces of ourselves to machines. We ask translation tools to carry our private messages across language barriers, trusting them to keep our meaning intact while making it understandable to someone else. For years, the goal of these tools was simply to produce text that flowed smoothly, sounding natural and easy to read. But as artificial intelligence has become more powerful, a new tension has emerged. A machine can now rewrite a difficult text to make it simpler and more readable, but in doing so, it might accidentally strip away the original meaning or leave out crucial details. This creates a tricky situation for the person using the tool: they can see the original text on the screen, yet they might not be able to tell if the machine's version has changed the story. The question researchers are now asking is whether simply showing the original text is enough to help us judge if a translation is good, or if there is a gap between what we can see and what we can actually understand.

A team of researchers at Lehigh University set out to explore this gap by testing how people judge translations when the machine prioritizes either easy reading or strict accuracy. They built a simple, text-only interface called TransLingo, designed to look like a standard utility without any friendly avatars or voices that might trick a user into thinking the machine is human. They invited over three hundred people to use this tool with two different types of source text. The first type was simple, everyday narrative prose, similar to a story a child might read. The second type was complex, literary-philosophical writing, drawn from ancient texts that require a college-level education to fully grasp. For each type of text, the researchers presented two versions of the translation. One version was generated by an artificial intelligence to be as readable and smooth as possible, even if it meant simplifying the content. The other version was carefully revised by a human researcher to stay as faithful as possible to the original words and ideas, even if it remained slightly more difficult to read.

The researchers wanted to see if people could spot the difference between these two versions and if their judgment changed depending on how hard the original text was to understand. When the source text was simple, the participants were sharp. They could clearly see that the version which stayed true to the original content was better. They rated the faithful translation higher in quality, and they also felt that the machine itself was smarter and more trustworthy when it produced that accurate version. However, the results took a surprising turn when the source text was complex. Even though the researchers had verified that the faithful version still contained more of the original meaning, the participants could not tell the difference. They rated the easy-to-read version and the faithful version as equally good. In fact, for the difficult texts, the machine's ability to make the output readable seemed to blind the users to the fact that the faithful version was preserving more of the source material. The source text was right there on the screen, visible for comparison, yet the users' overall judgment did not reflect the difference in what was actually retained.

This phenomenon, which the authors call an "evaluability gap," suggests that showing a user the original text is not enough to guarantee they can evaluate the translation correctly. When the material is complex, the effort required to compare the two texts seems to push users toward relying on how easy the output feels to read, rather than checking if the meaning has been preserved. The study also looked at how these judgments affected trust and the willingness to share personal information. The researchers found that when people trusted the machine to perform its task accurately, they were more willing to say they would share personal details like their political views, financial status, or family relationships with it. However, the type of translation they saw did not change their willingness to share. Whether the machine produced a readable or a faithful translation, the participants' stated comfort with disclosing personal information remained the same. This indicates that judging the quality of a translation and deciding whether to entrust a machine with private data are two separate mental processes.

The findings challenge the idea that a transparent interface, where the original text is always visible, solves the problem of trust in artificial intelligence. The study shows that for complex tasks, visibility does not equal understanding. Users may look at the source and the output side by side, but if the output is made to feel smooth and easy, they may overlook the fact that the machine has altered the content. This has important implications for how we design tools that handle our personal information. It suggests that we cannot rely on users to figure out if a machine is being honest just by giving them the source text. Instead, designers may need to provide specific help, such as highlighting exactly what was changed or simplified, to bridge the gap between seeing the text and truly understanding what the machine has done. The research confirms that while we can build machines that read well, ensuring that we can also judge what they have kept is a much harder problem to solve.

Technical Summary: When Readability and Source Retention Diverge: An Evaluability Gap in AI Translation

Problem Statement
The paper addresses a critical tension in Large Language Model (LLM)-based translation: the potential divergence between output readability and source fidelity. While early machine translation was often criticized for disfluency, contemporary LLMs generate highly fluent text, making fluency a less diagnostic cue for quality. The authors posit that "readability-oriented" generation (simplifying text for ease of reading) may come at the cost of "faithfulness" (preserving the original meaning). This creates an "evaluability gap": even when the source text is visible to the user, a holistic quality judgment may fail to reflect the extent to which the output has retained or altered the source content. This gap is particularly concerning because users often submit personal text to translation systems, turning the interface into a disclosure mechanism where trust and willingness to entrust data are paramount.

Methodology
The study employed a focal 2×2 between-subjects experimental design ( $N=306$ ) using a prototype web interface named "TransLingo."

Factors:
1. Source-Text Condition: Simple (contemporary LLM-generated narrative prose) vs. Complex (19th-century literary-philosophical prose from Meditations).
2. Output Rendering: Readability-oriented (LLM-generated to prioritize ease of reading) vs. Fidelity-oriented (researcher-revised to prioritize source retention while maintaining readability).
Procedure: Participants viewed English source texts and their corresponding English translations (via a Chinese intermediate step). They rated perceived quality, perceived system intelligence, agency-oriented anthropomorphism, task-performance trust, and willingness to disclose personal information across six domains (e.g., tastes, work, economic status).
Stimulus Audit: Before the user study, the authors conducted a quantitative audit of the prepared outputs using metrics including Flesch reading-ease gain ( $\Delta$ Flesch), content-word Jaccard overlap, BERTScore, and COMET. This audit confirmed that fidelity-oriented outputs retained significantly more source content than readability-oriented outputs in both conditions.
Analysis: The study utilized 2×2 factorial ANOVAs to test for interaction effects between rendering and source complexity. Additionally, a theory-ordered Structural Equation Model (SEM) was estimated to characterize the concurrent associations among perceived quality, intelligence, anthropomorphism, trust, and disclosure willingness.

Key Results

The Evaluability Gap: A significant interaction was found between output rendering and source-text condition for perceived quality.
- Simple Sources: Participants rated fidelity-oriented outputs significantly higher than readability-oriented outputs. The holistic rating successfully reflected the audited difference in source retention.
- Complex Sources: No reliable difference in perceived quality emerged between the two renderings, despite the audit confirming that fidelity-oriented outputs retained significantly more source content. The visibility of the source text did not ensure that users' overall quality judgments reflected the difference in retained content.
System Appraisals: The same interaction pattern extended to system-level appraisals. For simple sources, fidelity-oriented outputs received higher ratings for perceived intelligence, agency-oriented anthropomorphism, and task-performance trust. For complex sources, no reliable rendering differences were observed for these constructs.
Disclosure Willingness: Crucially, no reliable rendering-by-source-condition interaction was found for stated disclosure willingness in any of the six personal-information domains. While fidelity-oriented outputs were rated higher on quality and trust for simple texts, this did not translate into a measurable increase in willingness to disclose personal data.
Structural Associations (SEM): The appraisal-structure model revealed a specific pathway: Perceived Quality $\rightarrow$ Perceived Intelligence $\rightarrow$ (Anthropomorphism & Trust) $\rightarrow$ Disclosure Willingness. Trust was the proximal correlate of disclosure willingness across all domains. However, the model-implied indirect association between output rendering and disclosure willingness was small for simple sources and negligible for complex sources.

Key Contributions

Distinguishing Source Visibility from Source Evaluability: The study demonstrates that simply displaying the source text alongside the output is insufficient to ensure users can evaluate what the output has preserved, particularly when the source text is complex. There is a gap between access to the source and the evaluability of the translation's fidelity.
Theory-Ordered Appraisal Structure: The paper provides a structural account of how users evaluate translation systems in a plain-text interface (lacking avatars or voices). It establishes that perceived quality drives perceived intelligence, which in turn influences anthropomorphic attribution and trust, with trust serving as the primary driver of disclosure willingness.
Separation of Output Evaluation and Information Entrustment: The findings suggest that evaluating translation output and deciding what personal text to entrust to a system are distinct problems. Improvements in perceived quality or trust for specific renderings (e.g., fidelity-oriented) did not reliably alter users' stated willingness to disclose personal information, indicating that interface support for quality evaluation and data-handling transparency require separate design interventions.

Significance and Claims
The authors claim that their findings expose a limit of current source-visible translation interfaces: placing source evidence on the screen does not guarantee that users' holistic judgments will reflect differences in content retention, especially for complex texts. Consequently, the paper argues that "source-linked change information" (helping users see what changed) and "data-handling information" (helping users decide what to submit) address different user decisions and should be evaluated as separate interface-design problems. The study cautions that relying on holistic preference or readability in human-in-the-loop evaluation or LLM optimization may inadvertently favor outputs that sacrifice source fidelity, as users may not detect these losses when source texts are complex.

Technical Summary: When Readability and Source Retention Diverge: An Evaluability Gap in AI Translation

More like this