Technical Summary: Pun Intended: Multi-Agent Translation of Wordplay with Contrastive Learning and Phonetic-Semantic Embeddings

Problem Statement

Translating wordplay, particularly puns, across languages remains a significant challenge in machine translation. Unlike standard translation tasks, pun translation requires preserving both semantic meaning and linguistic incongruity (humor) simultaneously. Because puns rely on language-specific homonyms and phonetic ambiguities, a literal translation often fails to reproduce the joke in the target language. The core difficulty lies in generating entirely new wordplay in the target language that maintains the "script opposition" and cognitive surprise of the source, rather than simply reproducing the source vocabulary. Current automatic evaluation metrics (e.g., BLEU, BERTScore) often fail in this domain because they reward lexical similarity, which is frequently incompatible with successful pun translation.

Methodology

The authors investigated three progressively sophisticated approaches to translate English puns into French, utilizing Large Language Models (LLMs) combined with linguistic constraints. The study utilized the CLEF JOKER 2025 Task 2 English–French dataset (1,405 English puns paired with 5,838 French translations for training, and a hidden test set of 376 puns).

1. Baseline: Discriminator-Guided Generation

This approach uses an LLM to generate French puns based on the source sentence's semantic content and humorous effect, rather than literal translation.

• Process: The generator is prompted to create a French pun preserving the source's humor.
• Feedback Loop: A discriminator model (Gemini 2.5 Flash) evaluates candidates using few-shot prompting with 25 positive (puns) and 25 negative (non-puns) examples.
• Iteration: Candidates classified as non-puns are regenerated for up to ten iterations.

2. Guided Chain-of-Thought (CoT) with Phonetic-Semantic Embeddings

This pipeline introduces explicit linguistic reasoning and retrieval to guide generation.

• Pun Identification: An LLM identifies the pun word, classifies it (homographic vs. homophonic), and generates synonym lists for its two intended meanings.
• Translation: The pun word and synonym lists are translated into French.
• Phonetic-Semantic Retrieval: The system retrieves French lexical candidates that are semantically similar to one meaning and phonetically similar to the other.
  • Embeddings: French phonetic embeddings were trained using Lexique pronunciation data and PanPhon articulatory features (converted to IPA bigrams). These were concatenated with FastText semantic embeddings.
  • Similarity: Jaccard similarity was used for phoneme bigrams; cosine similarity was used for semantic vectors. Candidates were retained if they exceeded a threshold (cosine > 0.75) in both dimensions.
• Guided Generation: The LLM is instructed to construct the translation based on the retrieved candidates, with specific strategies for homographic vs. homophonic puns.

3. Multi-Agent Evaluation and Refinement

This framework replaces direct selection with an iterative refinement loop using specialized evaluators.

• Process: Specialized LLM agents iteratively assess candidate translations based on four dimensions derived from MMTE: Equivalence, Quality, Emotion, and Authenticity.
• Scoring: Evaluators assign numerical scores (e.g., 0–2 for equivalence/quality, binary for emotion, 0–4 for authenticity) and provide textual feedback.
• Refinement: The feedback is used to generate a revised translation. This loop continues until the average score reaches 2.0 or five iterations are completed.

Key Contributions

1. Functional Equivalence over Lexical Correspondence: The paper demonstrates that prioritizing the recreation of the joke's humorous effect (functional equivalence) yields superior human evaluation results compared to strategies focused on literal word-for-word translation.
2. Novel Architectures for Wordplay: The authors propose and evaluate three distinct LLM-based architectures:
   • A contrastive discriminator-guided baseline.
   • A guided reasoning pipeline utilizing explicit phonetic-semantic embeddings for lexical retrieval.
   • An iterative multi-agent framework that refines translations through specialized feedback.
3. Empirical Validation of Evaluation Metrics: The study highlights the disconnect between standard automatic metrics and human judgment in humor translation, showing that high-performing systems in human evaluation can rank poorly on BLEU and BERTScore.

Results

The systems were evaluated on the official CLEF JOKER 2025 Task 2 test set using BLEU, BERTScore, a pun location metric, and manual human evaluation by a native French speaker with expertise in wordplay.

• Human Evaluation: The Multi-Agent system ranked 1st (88.09% success rate), and the Guided CoT system ranked 2nd (85.71% success rate). The Baseline system ranked 25th (47.62%).
• Pun Location Metric: The Multi-Agent system ranked 1st (9.27% error rate), followed by the Guided CoT system (7.85%), with the Baseline ranking 45th (3.57%). This metric was calculated across 1,682 evaluated translations.
• Automatic Metrics (BLEU/BERTScore): Despite their high human evaluation scores, the Multi-Agent and Guided systems ranked near the bottom (41st and 45th out of 51 submissions, respectively) on BLEU and BERTScore. This inversion confirms that lexical overlap metrics systematically undervalue successful pun translation.

Significance and Claims

The paper claims that successful computational pun translation is achievable but requires a fundamental shift in objective: moving from reproducing source text to recreating humorous effects.

• Reinforcement of Translation Theory: The results empirically support long-standing translation theories (e.g., Delabastita, Low) which argue that preserving the "script opposition" and pragmatic function of a joke is more critical than lexical fidelity.
• Limitations of Current Metrics: The authors assert that automatic metrics like BLEU are fundamentally misaligned with the goals of humor translation, as they penalize the necessary lexical divergence required to create a pun in a new language.
• Methodological Insights:
  • Creativity before Constraint: The most successful systems allowed for creative exploration before applying constraints, rather than dictating translation strategies from the outset.
  • Architecture Influence: System architecture dictates translation strategy; the multi-agent approach favored iterative refinement of structure and humor, while the guided approach remained more anchored to English linguistic reasoning.
  • Evaluator Efficacy: Simple LLM-based evaluators proved surprisingly effective, potentially offering a viable alternative to traditional metrics for assessing creative language tasks.
• Future Direction: The authors suggest that Low's "polygonal algorithm" (an iterative search for semantic and phonetic alternatives) provides a robust roadmap for future computational pun translation, noting that their implementation only scratched the surface of this framework.