On the limited utility of parallel data for learning shared multilingual representations

1. Problem Statement

Multilingual Language Models (MLMs) are expected to develop shared cross-lingual representations, enabling knowledge transfer between languages (e.g., zero-shot translation or cross-lingual task performance). A prevailing hypothesis in the field is that including parallel data (translated sentence pairs) in the pretraining corpus acts as a crucial signal to align these representations.

However, it remains unclear to what extent parallel data actually drives this alignment versus other factors (such as architectural constraints or shared vocabulary). This study investigates whether varying the proportion of parallel data in pretraining significantly impacts the degree of cross-lingual representation sharing in fully trained models.

2. Methodology

Experimental Setup

• Models: The authors trained four 1.4 billion-parameter decoder-only Transformer models from scratch, based on the OLMo architecture (24 layers, 2048 hidden dimension, 16 attention heads).
• Corpus: A total of 200 billion tokens consisting of English and Finnish (a typologically distant pair to ensure robustness).
• Variable: The proportion of parallel data was systematically varied across the four models:
  • lm-0: 0% parallel data.
  • lm-1: 1% parallel data.
  • lm-2: 2% parallel data.
  • lm-5: 5% parallel data.
• Data Sources: Monolingual data came from FineWeb/FineWeb2. Parallel data was sourced from OPUS collections (HPLT2, CCMatrix, OpenSubtitles). To prevent format overfitting, parallel pairs were randomized across 60 different instruction/completion formats.
• Training: Models were trained on the LUMI supercomputer with a 2048-token context window. Sequences were not concatenated across languages within chunks to avoid unintended signals, though chunks were shuffled in batches.

Evaluation Metrics

The authors employed five distinct methods to assess representation sharing:

1. Principal Component Projections (PCA): Visualizing sentence representations (mean-pooled from residual streams) projected onto the first two principal components to observe overlap between English and Finnish.
2. Cosine Similarity: Measuring the angular distance between parallel sentence pairs versus random sentence pairs across layers.
3. Projection Weighted Canonical Correlation Analysis (PWCCA): A metric invariant to linear transformations, used to quantify the correlation between the subspaces of the two languages.
4. Language-Specific Neuron Analysis: Identifying neurons that strongly activate for one language but not the other (using Average Precision). The study tracks the distribution of these neurons and the proportion of "language-agnostic" neurons.
5. Language Control Vectors: Testing the ability to steer model outputs from English to Finnish using control vectors derived from parallel data, measuring fluency and coherence.

3. Key Contributions & Results

A. Cross-Lingual Alignment Emerges Without Parallel Data

The most significant finding is that parallel data has little to no effect on the final level of cross-lingual alignment in fully trained models.

• PCA & PWCCA: All models, including lm-0 (0% parallel data), exhibited strong cross-lingual alignment in the middle layers (approx. layers 10–20). The overlap in PCA plots and PWCCA scores were nearly identical across all four models.
• Cosine Similarity: While translation pairs showed higher similarity than random pairs, this was attributed to anisotropy (the model pushing all representations into a small region of the vector space) rather than specific alignment driven by parallel data.

B. Alignment is Layer-Dependent

Consistent with prior research, the study confirmed that shared representations are concentrated in the middle layers of the network.

• Early/Late Layers: Contain more language-specific features.
• Middle Layers: Contain semantic, language-agnostic representations. This structure emerged regardless of the presence of parallel data.

C. Limited Utility of Parallel Data

• Final State: Increasing parallel data from 0% to 5% did not result in statistically significant improvements in alignment metrics (PWCCA, neuron distribution) in the final checkpoints.
• Neuron Analysis: While parallel data slightly reduced the total number of highly language-specific neurons, it did not significantly increase the proportion of shared (language-agnostic) neurons compared to the 0% baseline.

D. Nuanced Effects: Training Dynamics and Neuron Specificity

Despite the lack of impact on the final state, the study identified two subtle effects of parallel data:

1. Acceleration in Early Training: During the initial phases of pretraining (first 20,000 steps), models with parallel data (specifically lm-5) showed slightly faster emergence of cross-lingual alignment (measured via coherence differences in control vector experiments).
2. Reduction of Language-Specific Neurons: Models with parallel data tended to have fewer highly language-specific neurons compared to the 0% model, suggesting parallel data helps "prune" language-specificity, even if it doesn't drastically alter the shared space.

E. Control Vector Effectiveness

Language control vectors successfully shifted output language for all models. However, there was no positive correlation between the amount of parallel data and the fluency or coherence of the generated Finnish text, further supporting the conclusion that alignment is not strictly dependent on parallel data volume.

4. Significance and Implications

• Challenging the "Parallel Data" Dogma: The paper provides strong empirical evidence that parallel data is not a necessary condition for learning shared multilingual representations. Models can achieve high levels of cross-lingual alignment using only monolingual data.
• Alternative Explanations: The authors suggest that shared representations may emerge due to parameter capacity constraints (the model is forced to share weights to learn multiple languages) or hidden parallel structures within monolingual corpora (e.g., code-switching or shared named entities).
• Resource Efficiency: For low-resource language pairs where high-quality parallel data is scarce or expensive, this study suggests that pretraining on monolingual data alone may be sufficient to achieve robust cross-lingual transfer capabilities, potentially saving computational resources.
• Training Dynamics: While parallel data may not change the final outcome, it may serve as a "warm start" mechanism, accelerating the convergence of shared representations in the early stages of training.

Conclusion

The study concludes that while including parallel data is intuitively attractive, its utility for improving the final cross-lingual alignment of multilingual language models is limited. The shared representation space appears to be an emergent property of the multilingual training objective itself, particularly in the middle layers of the transformer, rather than a direct result of explicit parallel sentence alignment.