Evaluative Judgement in Teaching AI-based Translation: A Class-room Case Study of AI-Mediated Translation and Post-Editing

Technical Summary: Evaluative Judgement in Teaching AI-based Translation

Problem Statement
Translation technology pedagogy has increasingly shifted from procedural tool instruction toward models requiring critical AI literacy, including human agency, data awareness, and ethical decision-making. A central challenge in this domain is that Generative AI (GenAI) and Large Language Models (LLMs) often produce fluent, coherent outputs that may still harbor terminological inaccuracies, adequacy issues, or stylistic flaws. Consequently, translator education must move beyond teaching students how to generate output to training them to act as "arbiters of quality." The specific pedagogical problem addressed is how to foster "evaluative judgement"—the capacity to make informed, evidence-based decisions about translation quality and to justify those decisions—within a classroom setting that compares LLMs, Neural Machine Translation (NMT) systems, and human translation.

Methodology
The study employs an exploratory qualitative case-study design with descriptive quantitative support, analyzing 23 anonymized student projects from a fourth-year undergraduate Machine Translation and Post-editing course at Universitat Pompeu Fabra (2025–2026 academic year).

• Assignment Design: Students completed a multi-stage workflow:
  1. Produced a human translation of a specialized English Wikipedia text (approx. 300 words) into Catalan or Spanish.
  2. Generated four machine translation outputs: two using LLM-based systems and two using NMT systems.
  3. Evaluated outputs using automatic metrics (COMET, BLEU, TER, ChrF) via the MATEO environment.
  4. Conducted human evaluation focusing on adequacy and fluency (4-point scale) with segment-level error annotation.
  5. Selected one output for post-editing and justified the choice in a written report.
  6. Performed post-editing on the selected version.
• Data Analysis: The analysis focused on the 22 cases accompanied by written reports. Data sources included the master coding table (aggregating metrics, system choices, and error categories) and the qualitative content of the reports.
• Coding Strategy: A hybrid approach was used. Deductive categories (adequacy, fluency, post-editing effort) were derived from the assignment design, while inductive subthemes (terminology instability, syntactic naturalness) emerged from the reports. Reports were rated on an ordinal scale of evaluative judgement (Low to High) based on the triangulation of evidence (metrics, human assessment, error analysis) used to justify decisions.

Key Results

• Divergence of Metrics and Human Choice: Automatic metrics did not dictate final selections. In 10 out of 22 report-backed cases, the system ranked best by automatic metrics was not the one selected for post-editing. For instance, while ChatGPT was the most frequent "automatic winner" (9 cases), DeepL was the most frequent final selection (10 cases).
• Nature of Evaluative Judgement: The majority of reports (16 out of 22) demonstrated "High" or "Medium to High" evaluative judgement. Students did not rely on metric scores alone; instead, they justified choices based on a multidimensional assessment including adequacy, fluency, terminology, naturalness, and expected post-editing effort.
• Error Patterns: Students' attention was concentrated on specialized issues often missed by surface-level fluency. The most frequent error categories identified were terminology/lexis (19 reports), adequacy/meaning transfer (15), syntax/calques (14), and naturalness/style (14).
• Post-Editing Effort as a Decision Criterion: A significant finding is that students frequently selected the output that required the least severe reformulation or the most localized revision, even if another system scored higher on automatic metrics. This indicates a shift toward workflow-aware decision-making.
• AI Literacy Development: Students demonstrated a conditional approach to AI, recognizing that different systems offered partial strengths. Some reports explicitly noted that while one system was selected for post-editing, insights from other systems contributed to the final understanding of the text.

Key Contributions

1. Pedagogical Design: The paper presents an integrated classroom workflow that combines human translation, multi-system comparison (LLM and NMT), automatic and human evaluation, and consequential post-editing decisions. This design forces students to reconcile different forms of evidence rather than treating AI output as a finished product.
2. Empirical Evidence of Reasoning: The study provides qualitative evidence of how students justify system selection. It demonstrates that students can be trained to override automatic metrics in favor of domain-specific adequacy and post-editing efficiency.
3. Operationalization of Evaluative Judgement: The paper argues for treating evaluative judgement not as an abstract concept but as a practical component of AI literacy, where students learn to calibrate reliance on AI tools based on text type, domain, and quality requirements.

Significance and Claims
The paper modestly claims that its primary contribution is pedagogical rather than technical benchmarking. It does not assert that LLMs or NMT systems are categorically superior; rather, it shows that student preferences are text-dependent and justified by specific criteria (terminology, naturalness, effort).

The significance lies in demonstrating that structured comparison tasks can effectively foster critical, reflective, and professionally relevant judgement in translator education. By requiring students to leave a "trace of their reasoning" through metric interpretation, error annotation, and justification reports, the assignment positions students as active arbiters of quality. This supports the broader educational goal of moving students from passive tool users to critical comparators who can make accountable decisions about when and how to intervene in AI-mediated translation. The study concludes that such tasks are essential for developing the "human agency" required to navigate the risks and affordances of GenAI in professional translation workflows.