XITE: Cross-lingual Interpolation for Transfer using Embeddings

Technical Summary: XITE (Cross-lingual Interpolation for Transfer using Embeddings)

1. Problem Statement

Multilingual Language Models (MLMs) like XLM-R and mBERT are capable of zero-shot cross-lingual transfer, but their effectiveness is often hindered by representation misalignment. This misalignment arises from differences in linguistic properties, lexical variations, or domain shifts between high-resource source languages (typically English) and low-resource target languages. Existing methods to fix this often rely on expensive machine translation (Translate-Train) or complex adversarial learning to force language-invariant features. There is a need for a more scalable, efficient method to bridge the gap between source and target embedding spaces without relying on explicit translation.

2. Methodology

The authors propose XITE, an embedding-based data augmentation framework that uses interpolation to mix source and target representations. The process follows three core stages:

• Stage 1: Target-to-Source Mapping:
  Starting with unlabeled text in a target language, the system identifies the most semantically similar instances in a labeled English (source) corpus. This is achieved by computing cosine similarities between the embeddings of the target text and the source corpus. The labels from these matched English instances are then "projected" onto the target text to create pseudo-labeled data.

• Stage 2: Deriving a Language-Rich Basis (LDA-based Projection):
  To prevent the loss of important linguistic nuances during mixing, the authors introduce a projection step. They use Linear Discriminant Analysis (LDA) to identify axes that maximize the separability between the source and target languages. By projecting target embeddings onto this "language-rich" basis before interpolation, the model can isolate language-specific features, ensuring that the semantic richness of the source is balanced with the linguistic identity of the target.

• Stage 3: Interpolation during Fine-tuning:
  The framework employs two interpolation strategies:

  • XITE-reg-reg (Simple Interpolation): The embeddings of the target text and its mapped source counterpart are simply added together.
  • XITE-reg-lda (Projection + Interpolation): The target embedding is first projected onto the LDA-derived language-rich basis and then added to the source embedding. This mixed representation is then fed into the task-specific classifier.

3. Key Contributions

• Novel Interpolation Framework: Introduces a way to bridge the representation gap by mixing embeddings rather than relying on text-level translation.
• LDA-driven Subspace Projection: A new strategy to isolate and preserve language-specific linguistic features during the augmentation process.
• Scalability: Unlike "Translate-Train" methods, XITE does not require translating large corpora, making it highly suitable for low-resource languages.
• Mitigation of Catastrophic Forgetting: The method acts as a regularizer, helping the model maintain high performance on the source language (English) while adapting to the target.

4. Experimental Results

The authors evaluated XITE on Sentiment Analysis (SA) and Natural Language Inference (NLI) tasks using XLM-R large and multilingual-e5 models across languages including Korean, Arabic, Hindi, and Urdu.

• Significant Performance Gains: XITE yielded improvements of up to 35.91% for SA and up to 81.16% for NLI compared to standard cross-lingual baselines (Baseline-PS).
• Superiority of LDA: The XITE-reg-lda variant consistently outperformed the simple XITE-reg-reg approach, proving that projecting onto a language-separable basis is beneficial.
• Robustness to Imperfect Labels: Even when the initial label projection (mapping) was imperfect, the interpolation process allowed the model to learn nuanced classification boundaries by leveraging the semantic cues from both languages.
• Comparison to Baselines: While "Translate-Train" (Baseline-TT) can achieve high scores, it is computationally expensive. XITE provides a competitive and much more efficient alternative.

5. Significance

XITE represents a significant step forward in efficient cross-lingual transfer. By moving the augmentation process from the text level (translation) to the embedding level (interpolation), it provides a computationally lightweight way to improve NLU performance in low-resource settings. Furthermore, its ability to safeguard against "forgetting" the source language makes it a more stable method for continuous multilingual adaptation.