Can linguistic complexity predict machine translation quality?- a corpus-based study of complexity-quality nexus in philosophical texts

Technical Summary: Can Linguistic Complexity Predict Machine Translation Quality? A Corpus-Based Study of Complexity-Quality Nexus in Philosophical Texts

Problem Statement
While Artificial Intelligence and Neural Machine Translation (NMT) have significantly advanced translation quality in high-resource domains (e.g., legal, journalistic, academic texts), there is a notable gap in understanding how these systems perform in low-resource, cognitively demanding domains such as philosophical texts. Specifically, existing literature lacks comprehensive comparisons across diverse machine translation (MT) systems—including traditional NMT, Large Language Model (LLM)-based systems like DeepSeek, and deep learning-driven systems like DeepL—when applied to complex, abstract source material. Furthermore, prior studies linking linguistic complexity to translation errors have been limited to specific language pairs (e.g., inflectional languages) or genres (e.g., journalism), often failing to explore the "complexity-quality nexus" in low-resource literary and philosophical contexts. This study addresses the question of whether lexical and syntactic complexity indices can predict machine translation quality in German-to-Chinese philosophical texts.

Methodology
The research employs a corpus-based experimental design utilizing Nachgelassene Schriften (Posthumous Writings) by the German philosopher Theodor W. Adorno. To eliminate bias from pre-existing training data, the authors selected a text with no prior Chinese translation. A reference translation was produced by a senior academic translator and validated through back-translation (yielding a BLEU score of 0.72 against the original German) and terminological alignment.

The corpus consists of 254 text segments (approx. 59,564 words) translated into Chinese by three distinct systems:

1. DeepL (Deep learning-driven)
2. DeepSeek (LLM-based)
3. NMT (Standard Neural Machine Translation model)

Translation quality was quantified using BLEU scores (Bilingual Evaluation Understudy) calculated against the human reference translation. Linguistic complexity was measured using a suite of Chinese-specific indices adapted from prior research on translationese and second-language writing, categorized into:

• Lexical Indices: Type-Token Ratios (TTR1, TTR2, TTR1C, TTR2C), Mean Word Rank (MWR), and Mean Character Rank (MCR).
• Syntactic Indices: Mean Length of Sentences (MLS), Mean Length of Clauses (MLC), Mean Length of T-units (MLTU), T-units per sentence (T/S), and Clauses per sentence (C/S).

Three multiple regression models were constructed to predict BLEU scores:

• Model 1: Based solely on lexical complexity.
• Model 2: Based solely on syntactic complexity.
• Model 3: A combined model integrating both lexical and syntactic features.

Key Results
The analysis yielded the following findings regarding the predictive power of linguistic complexity:

• System Performance: DeepL consistently achieved higher BLEU scores compared to DeepSeek and the standard NMT model across the corpus. This finding challenges the assumption that LLM-based systems (like DeepSeek) inherently serve as superior alternatives to traditional NMT models in this specific domain.
• Model 1 (Lexical Complexity): Demonstrated strong explanatory power with an R-squared of 0.849. Lexical richness and diversity (indicated by positive coefficients for TTR1, TTR1C, TTR2, MWR, and MCR) were significant positive predictors of translation quality. Conversely, TTR2C showed a negative coefficient, suggesting that excessive lexical repetition or redundancy detracts from quality.
• Model 2 (Syntactic Complexity): Showed moderate explanatory power with an R-squared of 0.395. While longer sentence structures (MLS) and clauses (MLC) correlated positively with quality, other syntactic features like clause density (C/S) were not statistically significant. This indicates that syntactic complexity alone is a weaker predictor than lexical complexity.
• Model 3 (Combined Complexity): Achieved the highest explanatory power with an R-squared of 0.86. This model confirmed that a holistic approach integrating both lexical and syntactic features provides the most comprehensive prediction of translation outcomes. Notably, in the combined model, the ratio of subordinate to main clauses (T/S) exhibited a negative coefficient, suggesting that high structural subordination may increase translation difficulty and reduce quality.

Significance and Contributions
The paper claims to contribute to the field of Machine Translation Quality Assessment (MTQA) in several specific ways:

1. Domain Expansion: It fills a research gap by examining the complexity-quality relationship in low-resource, abstract domains (philosophical texts) rather than high-resource or journalistic genres.
2. System Comparison: It provides empirical data comparing the performance of DeepL, DeepSeek (LLM), and standard NMT, offering insights into how different architectural approaches handle complex semantic and syntactic structures.
3. Predictive Modeling: The study validates the utility of regression models in predicting translation quality based on linguistic complexity. It establishes that lexical richness is a more dominant factor than syntactic complexity, though the integration of both yields the most accurate predictions.
4. Methodological Framework: By utilizing a rigorously constructed corpus with a verified reference translation and specific complexity indices, the study offers a replicable framework for assessing MT performance in challenging, non-standardized domains.

The authors conclude that while current MT systems show promise, their performance in philosophical texts is heavily dependent on the source text's lexical diversity and syntactic structure. The research underscores the need for a balanced integration of lexical and syntactic considerations in assessing and improving MT systems, particularly for low-resource and high-complexity applications.