Fluency and Faithfulness in Human and Machine Literary Translation

Technical Summary: Fluency and Faithfulness in Human and Machine Literary Translation

Problem Statement
Literary translation requires balancing two often competing objectives: producing text that is fluent and natural in the target language (fluency) while preserving the semantic and stylistic content of the source text (faithfulness). While translation theory, notably Friedrich Schleiermacher's formulation, posits these as opposing poles, this tension has rarely been examined empirically at scale. Recent Large Language Models (LLMs) demonstrate high fluency in literary machine translation, yet it remains unclear whether this fluency corresponds to semantic preservation. Furthermore, reference-free quality estimation metrics (like COMET-KIWI) are increasingly used to evaluate translation without gold-standard references, but their relationship with syntactic fluency in literary contexts is not well understood.

Methodology
The authors analyze a corpus of 130,486 translated paragraphs from 106 novels spanning 16 source languages. The dataset includes human translations, Google Translate outputs, and translations generated by TranslateGemma (a fine-tuned model based on the Gemma3 architecture).

To measure the relationship between fluency and faithfulness, the study employs two distinct, independent metrics:

1. Fluency (Original-Likeness): Fluency is operationalized as the degree to which a translated paragraph resembles original English text. To isolate syntactic structure from semantic content, the authors anonymize text by replacing tokens with fine-grained Part-of-Speech (POS) tags. A logistic regression classifier, trained on POS n-grams (unigrams, bigrams, trigrams), distinguishes between original English paragraphs (from a corpus of 1800–1930 novels) and translated paragraphs. The classifier is trained with book-level group cross-validation and length-stratified downsampling to control for segment length. Fluency is defined as $1 - P(\text{translation} | x)$, where higher scores indicate greater similarity to original English syntax.
2. Faithfulness (Semantic Adequacy): Faithfulness is measured using COMET-KIWI, a reference-free neural metric that predicts translation quality based on source and target text pairs. The authors validate COMET-KIWI as a proxy for faithfulness by demonstrating a strong positive correlation between its scores and roundtrip semantic similarity (using BGE-M3 embeddings).

Key Contributions

• Large-Scale Empirical Analysis: The study provides a paragraph-level analysis of over 130,000 translations across human, Google Translate, and LLM outputs, examining the interplay between fluency and faithfulness.
• POS-Based Fluency Framework: The authors propose a scalable method for measuring syntactic fluency independent of semantic fidelity using POS-based translationese classification with strict book-level held-out evaluation.
• Reference-Free Evaluation Application: The work applies COMET-KIWI to literary translation to assess semantic faithfulness without gold references, correlating it with syntactic fluency signals.
• Empirical Evidence of Trade-offs: The study provides evidence of a negative relationship between fluency and faithfulness signals, particularly in human translations of similar lengths.

Results

• Length Confounding: The authors found that paragraph length significantly confounds both metrics. Longer paragraphs correlate negatively with both COMET-KIWI scores and fluency scores.
• Negative Correlation: After controlling for paragraph length using partial Spearman correlation, a consistent negative relationship emerges between fluency and faithfulness. This indicates that as a translation becomes more syntactically "original-like" (fluent), its predicted semantic adequacy tends to decrease.
• Source-Specific Variations:
  • Human Translations: Exhibit a statistically significant negative correlation ($\rho \approx -0.078$), suggesting a distinct trade-off where achieving native-like syntax may involve semantic shifts.
  • Google Translate: Also shows a negative correlation ($\rho \approx -0.051$), though slightly weaker than human translations.
  • TranslateGemma (LLM): Shows a weak, often non-significant, and sometimes positive correlation. This suggests that LLMs may not exhibit the same tension between fluency and faithfulness as human translators or traditional MT systems, or that the metrics capture different aspects of their output.
• Metric Divergence: The results suggest that COMET-KIWI does not align with POS-based signals of fluency. High fluency (in the sense of original-likeness) does not necessarily equate to high semantic faithfulness as measured by COMET-KIWI.

Significance and Claims
The paper claims that these findings provide empirical evidence for a systematic, albeit modest, tension between producing syntactically native-like English and preserving semantic correspondence in literary translation. The authors argue that this trade-off is more distinctly observed in human literary translations of similar lengths.

The study suggests that automatic evaluation metrics like COMET-KIWI may implicitly favor translations that preserve structural patterns aligned with the source text, potentially rewarding "translationese" signals when they correlate with surface-level fidelity. Conversely, the POS-based fluency metric captures syntactic naturalness that may diverge from semantic adequacy. The authors conclude that while the magnitude of the relationship is modest, the consistency of the pattern across analytical approaches indicates that fluency and faithfulness are not entirely independent signals in literary translation. They also note that the behavior of the fluency classifier offers a lens to examine stylistic differences between historical human translations (trained on 1800–1930 texts) and modern machine outputs optimized for contemporary standard English.