Technical Summary: Hermes the Polyglot

Problem Definition

Interlingual subtitling involves translating subtitles from visual media (films, TV series) into a target language. While Large Language Models (LLMs) have advanced general machine translation, they struggle with the specific constraints of subtitle translation. The paper identifies four primary challenges:

1. Semantic Coherence: LLMs have limited input token capacities, making it difficult to process entire programs at once. Improper segmentation of subtitle lines disrupts context, leading to incoherent translations.
2. Pronoun Reference: Cultural and linguistic differences in pronoun systems (e.g., honorifics in Japanese/Korean or pronoun omission) cause errors when the speaker identity is ambiguous.
3. Terminology Translation: Visual media often contains specialized proper nouns (names, locations, items) that require accurate, consistent, and context-aware translation, which standard LLMs often fail to maintain.
4. Translation Expressiveness: Beyond accuracy, audiences require translations that are natural (conforming to target language grammar/culture) and vivid (conveying emotion, tone, and atmosphere). Standard models often produce literal, flat translations.

Methodology: The Hermes Framework

The authors propose Hermes, a unified, LLM-based framework designed to address these challenges through three integrated modules and a specific training strategy.

1. Speaker Diarization (SD)

This module annotates speakers for each subtitle line to resolve pronoun ambiguity and guide context-aware segmentation.

• Multimodal Integration: It combines visual and speech modalities. Visual features (face embeddings via CurricularFace) and audio features (timbre embeddings via ERes2NetV2) are extracted.
• Clustering & Registration: Visual clustering serves as the anchor for speaker registration. Audio clusters are mapped to visual clusters via voting to create speaker prototypes.
• Supplementation: To handle audio-visual asynchrony (where speakers are not visible), the system compares timbre embeddings of undetected lines against registered prototypes. A novel "speaker supplementation" strategy groups adjacent lines based on similarity thresholds; if a group lacks a detected speaker, it is registered as a new speaker based on average timbre embedding.

2. Terminology Identification (TI)

This module ensures the accurate and consistent translation of proper nouns.

• Knowledge Distillation Approach: An off-the-shelf LLM (Qwen-Max) is used in a one-shot setting to detect terms, their types, and translations from bilingual parallel corpora.
• Filtering & Voting: Raw candidate results are filtered and voted upon to determine the canonical type and translation for each term.
• Specialized Training: A dedicated Terminology LLM ($\pi_{term}$) is trained on a constructed dataset to retrieve and translate these terms during inference, ensuring consistency across the program.

3. Expressiveness Enhancement (EE)

This module optimizes the translation model for naturalness and vividness using a novel technique called Segment-wise Adaptive Preference Optimization (SAPO).

• Local Preference Optimization: Unlike standard preference alignment (e.g., DPO) which treats the entire response as a single unit, SAPO treats each subtitle line as a segment. This is necessary because the translation of a line depends on its specific context within the sequence.
• Sampling Strategy: For each source line, the model samples multiple candidate translations. An evaluator LLM (LLM-as-a-Judge) scores these candidates on expressiveness.
• Adaptive Weighting: The training loss applies adaptive weights to segments. A gating function excludes low-diversity or low-quality segments (noise), while an importance score prioritizes segments with rich translation variations.
• Training: The final translation model ($\pi_{st}$) is trained using a preference loss (e.g., DPO) on these segment-level pairs, guided by the SAPO sampling process.

Key Contributions

1. Task Definition: The paper formally defines interlingual subtitling as a multimodal machine translation task, systematically identifying unique challenges regarding coherence, pronouns, terminology, and expressiveness.
2. Unified Framework: The proposal of Hermes, which integrates visual/audio speaker diarization, terminology identification via knowledge distillation, and a segment-wise preference optimization technique.
3. Evaluation Framework: The construction of a multidimensional evaluation framework based on "LLM-as-a-Judge," assessing accuracy (pronoun accuracy, terminology consistency, translation accuracy) and expressiveness (naturalness, vividness).
4. Performance: Experimental results demonstrating that Hermes achieves state-of-the-art performance in speaker diarization and generates translations that are more expressive and contextually coherent than existing baselines.

Experimental Results

The framework was evaluated on a multilingual dataset (Youku platform) covering directions such as en$\Rightarrow$de, en$\Rightarrow$fr, en$\Rightarrow$zh, ko$\Rightarrow$zh, zh$\Rightarrow$en, and zh$\Rightarrow$th.

• Translation Quality: Hermes (specifically the 14B model trained with SAPO) significantly outperformed traditional models (NLLB, MADLAD, Google Translate) and competitive baselines (VideoDubber).
  • It achieved higher scores in Accuracy (Pronoun Accuracy and Terminology Consistency) and Naturalness compared to frontier chat LLMs (GPT-4o, Qwen-Max) and reasoning models (DeepSeek-R1, GPT-5).
  • Crucially, Hermes achieved the highest Vividness scores across all directions, occasionally surpassing frontier LLMs, indicating superior ability to convey emotion and tone.
  • In low-resource directions (e.g., ko$\Rightarrow$zh, zh$\Rightarrow$th), Hermes excelled across all dimensions.
• Speaker Diarization: The SD module achieved superior performance (lower DER and JER) compared to audio-only baselines (VBx, Spectral Clustering) and the tri-modal SOTA method (E2CP), particularly in challenging scenarios like dialectal shows.
• Human Evaluation: Pairwise comparisons with human references and other baselines confirmed the reliability of the LLM-as-a-Judge framework, with Hermes showing high win rates in naturalness and vividness.
• Ablation Studies:
  • The adaptive gating function in SAPO was identified as the most critical component for reducing noise.
  • Larger backbone models (14B) performed better than smaller ones (8B), though SAPO improved all backbones over standard SFT.
  • DPO was found to be the most stable and effective alignment loss among those tested (DPO, SimPO, GRPO).

Significance

The paper claims that Hermes advances the research in interlingual subtitling by moving beyond simple translation accuracy to address the holistic requirements of localization. By integrating multimodal cues (visual/audio) and employing a segment-wise optimization strategy, the framework successfully bridges the gap between the rigid constraints of subtitle formats and the nuanced, expressive nature of human dialogue. The work demonstrates that with specific architectural adaptations and preference optimization, LLMs can surpass both traditional machine translation systems and general-purpose frontier models in the specialized domain of visual media localization.