Lost but not erased: Finding traces of a forgotten language in neural speech models

Technical Summary: Lost but not erased: Finding traces of a forgotten language in neural speech models

Problem Statement
International adoptees provide a natural experiment for studying early language exposure: they lose access to their birth language (Lpre) early in life, rapidly acquire a new language (Lpost), and eventually exhibit native-like proficiency in Lpost with no conscious memory of Lpre. However, behavioral studies reveal that traces of Lpre persist, evidenced by accelerated relearning of Lpre phonological contrasts and specific brain activation patterns. While these phenomena are often attributed to a biologically-timed critical period involving heightened neural plasticity, it remains unclear whether these effects stem from biological maturation or the ordinary dynamics of learning. Existing human studies cannot isolate biological maturation from learning dynamics, and prior artificial intelligence studies have focused on high-level text or computer vision tasks without localizing where in a network such "critical period" effects occur. This study asks whether the persistence of lost language traces and the associated relearning advantage can be explained by the hierarchical dynamics of learning in deep neural networks, independent of biological maturation.

Methodology
The authors employed state-of-the-art automatic speech recognition (ASR) models (Conformer architecture with 12 encoder and 6 decoder layers, ~109M parameters) trained on raw speech spectrograms from the CommonVoice dataset (English, French, German).

1. Simulation of Adoption: To mimic the international adoptee experience, models ($M_a$) were trained on a pre-adoption language ($L_{pre}$) for a specific duration (3.4k to 8.4k updates) and then abruptly switched to a post-adoption language ($L_{post}$). Crucially, the learning rate was kept constant (no cooldown) to prevent "freezing" effects, ensuring that any persistence of traces was due to learning dynamics rather than optimization schedules.
2. Control Conditions: The study compared $M_a$ against:
   • $M_{pre}$: Models trained only on $L_{pre}$.
   • $M_{post}$: Models trained only on $L_{post}$.
   • $M_{ctrl}$: Adoptee models switched from a different language (e.g., English $\to$ French) to test for language-family-specific transfer versus specific $L_{pre}$ traces.
3. Analytical Pipeline:
   • Behavioral Tracking: Monitored next-token prediction accuracy and word error rate (WER) to confirm the loss of $L_{pre}$ proficiency and acquisition of $L_{post}$.
   • Phonemic Linear Probes: Trained linear classifiers on frozen encoder layers to predict aligned phonemes, testing for retained phonological discrimination ability.
   • Representational Similarity Analysis (RSA): Computed cosine similarity matrices of internal representations for held-out $L_{pre}$ samples across layers. RSA compared the geometry of $M_a$ against $M_{pre}$ and $M_{post}$ to detect persistent structural traces.
   • Relearning Experiments: Measured the number of steps required for models to relearn $L_{pre}$ to a 70% accuracy threshold.
   • Mechanistic Localization: Conducted layer-splicing experiments where early (pre-phonemic) or late (phonemic/post-phonemic) layers of $M_a$ were swapped with those of $M_{post}$ to isolate the source of the relearning advantage.

Key Results

1. Behavioral Mimicry: The adoptee models ($M_a$) exhibited a sharp decline in $L_{pre}$ accuracy while rapidly acquiring $L_{post}$, eventually reaching proficiency levels indistinguishable from monolingual $M_{post}$ models. This mirrors the behavioral profile of human adoptees.
2. Localization of Traces:
   • Phonemic Probes: $M_a$ showed a significant advantage in discriminating $L_{pre}$ phonemes compared to $M_{post}$, but this advantage was concentrated in the lowest, pre-phonemic layers (1–4) and the early phonemic layers. The advantage vanished in higher-level layers.
   • RSA: Representational traces of $L_{pre}$ persisted in $M_a$ even after thousands of updates of $L_{post}$ training. These traces were strictly localized to the earliest encoder layers (1–4). The similarity of $M_a$ to $M_{pre}$ stabilized at a level significantly above the cross-language baseline, indicating a persistent geometric trace of the lost language.
   • Optimal Window: The magnitude of these traces did not scale linearly with pre-training duration; they peaked after a specific window of exposure (approx. 5.0k–6.7k updates) and then slowly declined, suggesting an optimal period for foundational entrenchment.
3. Functional Savings Effect: $M_a$ relearned $L_{pre}$ 14.3% faster than $M_{post}$ models and 12.8% faster than control models ($M_{ctrl}$) that lacked specific $L_{pre}$ exposure. This confirms a "savings effect" driven by retained traces.
4. Mechanistic Causality: When the earliest (pre-phonemic) layers of $M_a$ were replaced with those from $M_{post}$, the relearning advantage was significantly reduced. This confirms that the low-level, pre-phonemic representations are the primary drivers of the accelerated relearning.

Significance and Claims
The paper argues that the "critical period" effects observed in international adoptees do not necessarily require a biologically-timed window of heightened plasticity. Instead, these effects can arise from the entrenchment of foundational representations within a hierarchical learning system.

• Hierarchical Entrenchment: The authors propose that early learning establishes low-level, pre-phonemic representations that become the statistical foundation for higher-level categories. Because revising these foundational structures risks destabilizing the entire hierarchy, they are resistant to change (entrenched), while higher-level representations are more easily overwritten.
• Experience over Biology: The study demonstrates that a system with fixed plasticity (no biological maturation) reproduces the key phenomena of critical periods: the loss of conscious access, the persistence of low-level traces, and the savings effect in relearning.
• Reframing Critical Periods: The authors suggest that critical period effects may be a general property of how learning systems build representations from the bottom up, rather than a unique biological constraint. Consequently, the experiential account should serve as the null hypothesis for critical period phenomena; biological maturation claims must demonstrate effects that exceed what learning dynamics alone can produce.
• Implications: This mechanistic understanding implies that the order of experience (curriculum) plays a central role in shaping durable latent structures, offering potential avenues for educational strategies like early immersion or heritage language maintenance, even if overt proficiency is lost.

The paper concludes that while it does not rule out biological factors in human development, it provides a computational proof-of-concept that learning dynamics alone are sufficient to generate the lasting, consequential traces of early language exposure observed in international adoptees.