ATD-Trans: A Geographically Grounded Japanese-English Travelogue Translation Dataset

Technical Summary of ATD-Trans: A Geographically Grounded Japanese–English Travelogue Translation Dataset

Problem Statement
Geographic text, characterized by implicit location information conveyed through geo-entity mentions (e.g., place and facility names) rather than explicit coordinates, is a critical resource for applications like tourism and disaster management. However, equitable access to this information across languages is hindered by variations in Machine Translation (MT) quality. Existing MT systems often exhibit biases based on training data, leading to disparities in how well they translate geo-entities depending on the language focus of the model and the geographic region described. While entity-aware MT and document-level MT have been studied separately, the specific challenges of translating geographic text—particularly regarding the preservation of geo-entity accuracy across different regions (domestic vs. overseas) and model architectures—remain underexplored.

Methodology
To address these gaps, the authors constructed ATD-Trans, a geographically grounded Japanese–English travelogue translation dataset. The dataset is derived from the NAIST Academic Travelogue Dataset (ATD), comprising 90 translated articles (42 domestic and 48 overseas) with corresponding source texts. Key features of the dataset include:

• Geo-Annotations: Both source and target texts are annotated with geo-entity mentions and linked to OpenStreetMap (OSM) entry IDs, enabling evaluation at both the document and entity levels.
• Experimental Design: The authors evaluated eight existing Large Language Models (LLMs), categorized by language focus (English-centric: Llama 3.1, Gemma 2; Japanese-enhanced: Llama-3.1-Swallow, Gemma-2-Llama Swallow) and size (8B/9B vs. 70B/27B).
• Evaluation Conditions: Experiments were conducted under varying inference settings:
  • Context: Paragraph-level vs. Document-level context.
  • Term Specification: Basic prompts vs. prompts incorporating term specifications derived from OSM entries (predicted or gold) and oracle terms from reference translations.
• Metrics: Translation quality was assessed using d-BLEU and COMET for overall performance, and Term Accuracy (fuzzy matching with a 0.9 threshold) for geo-entity preservation. Additional experiments included synthetic ID-replacement tasks to isolate entity translation difficulty and geocoding/mention recognition tasks.

Key Results
The experiments yielded several significant findings regarding model performance and translation difficulty:

1. Language Focus: Japanese-enhanced models consistently outperformed their English-centric counterparts, particularly in Term Accuracy. This suggests that continual pre-training on Japanese data enhances the models' knowledge of Japanese geo-entities and their English equivalents.
2. Geographic Region: Translation quality was significantly higher for overseas travel blogs compared to domestic (Japan) blogs. Domestic texts presented greater difficulty, largely attributed to the complexity of translating specific, locally-oriented geo-entities. Synthetic experiments replacing geo-entities with IDs confirmed that entity mentions are a primary source of translation difficulty, especially for domestic data.
3. Context and Size: Document-level context generally improved overall translation quality (d-BLEU, COMET), though its effect on Term Accuracy varied. Larger models (70B/27B) consistently outperformed smaller ones (8B/9B).
4. Term Specification:
   • Oracle Terms: Providing correct terms from reference translations significantly improved Term Accuracy (reaching ~100%) without degrading overall quality.
   • KB Terms: Using terms derived from OpenStreetMap (KB gold or KB predicted) often degraded overall translation quality (d-BLEU, COMET). This was attributed to mismatches between KB terms and reference translations, as well as negative interference on mentions where no KB entry existed.
   • Model Sensitivity: Larger models (70B) were more adversely affected by noisy term information than smaller models (8B), suggesting that models with richer internal knowledge benefit less from external term specifications if those specifications are imperfect.

Significance and Claims
The paper claims that ATD-Trans provides a necessary resource for the in-depth analysis of MT for geographic text, enabling evaluation at both the document and geo-entity levels. The study highlights that accurate translation of geographic text is not merely a general language task but is heavily dependent on the model's specific knowledge of the region and the entities involved.

The authors emphasize that while term specification can be beneficial when it provides accurate knowledge the model lacks, it carries risks when the provided terms are inappropriate or mismatched. Consequently, the paper suggests that future systems must carefully balance the integration of external geographic knowledge bases with the model's internal knowledge to avoid degrading translation quality. The work serves as a case study demonstrating that equitable geo-information access requires addressing specific biases in MT systems regarding language focus and geographic specificity.