Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processing

1. Problem Statement

While Large Language Models (LLMs) have achieved human-like performance in grammar and sentence comprehension for English, their bilingual and multilingual cognitive capabilities remain under-explored. Specifically, it is unclear how these models handle the complex interplay between orthography (spelling), phonology, and semantics when processing words across languages.

The paper focuses on three critical linguistic phenomena that challenge bilingual processing:

• Cognates: Words sharing similar orthography and meaning across languages (e.g., English blind / German blind).
• Non-cognates: Words sharing meaning but having different forms (e.g., English pot / Spanish olla).
• Interlingual Homographs: Words sharing orthography but possessing distinct meanings in different languages (e.g., English gift [present] vs. German gift [poison]).

The core research question is: Do multilingual LLMs process these word types similarly to human bilinguals, or do they rely heavily on orthographic patterns at the expense of semantic understanding?

2. Methodology

2.1 Datasets

The authors constructed three novel datasets covering three language pairs: English-Spanish, English-French, and English-German.

• Word Lists: Each dataset contains 420 word pairs (nouns and adjectives) categorized as cognates, non-cognates, and interlingual homographs.
• Semantic Constraint Sentences: The authors generated 1,680 sentences (low and high semantic constraint) where target homographs were embedded in incongruent contexts (e.g., a Spanish word in an English sentence).
• Validation: All data was curated and verified by linguists using authoritative dictionaries (Merriam-Webster, Real Academia, etc.).

2.2 Models Evaluated

Five publicly available multilingual LLMs were tested:

• BLOOMZ
• LLaMA-2
• LLaMA-3
• LLaMA-3.1
• Mistral

2.3 Experimental Tasks

The study adapted psycholinguistic tasks (specifically Lexical Decision Tasks) for LLMs:

1. Word Pair Disambiguation: The model is given a pair of words and must label them as "True" (same meaning) or "False" (different meaning). This tests the ability to distinguish cognates from homographs.
2. Semantic Judgment: The model is given a word and two meaning options (one correct, one incorrect) and must select the correct definition. This tests semantic retrieval independent of orthography.
3. Sentence Comprehension (Incongruent Contexts): The model analyzes code-mixed sentences containing homographs. It must identify the word's language, its intended meaning, and whether the sentence makes sense. This tests the ability to use context to resolve ambiguity.

2.4 Analysis Metrics

• Token Overlap Analysis: Measured sub-word token overlap between word pairs to quantify orthographic similarity.
• Conditional Entropy: Used to determine if semantic judgment performance is correlated with word pair disambiguation performance (testing if the two tasks rely on the same internal representations).
• Human Evaluation: For the sentence task, human linguists evaluated the free-text outputs of LLaMA-3.1 to categorize model behavior (e.g., "Correction Type 1," "Correction Type 2," "Confusion").

3. Key Contributions

1. Novel Benchmarking: Introduced a comprehensive framework to evaluate bilingual lexical access in LLMs using cognates, non-cognates, and interlingual homographs across three major language pairs.
2. Orthography vs. Semantics Decoupling: Demonstrated that LLMs rely heavily on orthographic cues for disambiguation but fail to link these cues to semantic meaning, a finding contrary to human bilingual processing where orthography and semantics are integrated.
3. Contextual Asymmetry: Revealed a significant asymmetry in how models process English vs. non-English words. Models tend to default to English semantics even when the context is non-English, whereas they struggle to utilize non-English context to interpret English words.
4. Theoretical Grounding: Applied the Bilingual Interactive Activation Plus (BIA+) Model from psycholinguistics to interpret LLM failures, suggesting that LLMs lack the "grounded" word-to-world connection necessary for true semantic understanding.

4. Key Results

4.1 Word Pair Disambiguation

• Cognate Facilitation: Models performed exceptionally well on cognates (e.g., LLaMA-3.1: ~94% accuracy) and non-cognates, showing they can recognize shared orthography.
• Homograph Failure: Models struggled significantly with interlingual homographs. Most models (LLaMA-2, LLaMA-3, LLaMA-3.1, Mistral) performed below random baselines (accuracy ~25-37%), often defaulting to "False" or failing to distinguish meanings.
• Orthographic Bias: Performance on homographs was inversely related to orthographic similarity. Models performed 15x better on partially identical homographs than completely identical ones, indicating that high orthographic overlap causes severe interference.
• BLOOMZ Anomaly: BLOOMZ showed high accuracy on homographs (90%) but failed on cognates, attributed to a bias toward predicting "False" (semantically dissimilar).

4.2 Semantic Judgment

• Lack of Grounding: Models performed poorly on semantic retrieval tasks overall (average accuracy ~50-65%), with no significant difference in performance between cognates, non-cognates, and homographs.
• Independence of Tasks: Conditional entropy analysis revealed zero correlation between a model's ability to disambiguate word pairs and its ability to retrieve word meanings. This confirms that models use orthographic signals for disambiguation but do not possess a robust semantic representation of the words.

4.3 Sentence Context and Ambiguity

• Contextual Reliance: In incongruent sentences, models frequently defaulted to the English meaning of a homograph, even when the sentence was in Spanish, French, or German.
• Correction Strategies:
  • Non-English Homographs: The model successfully resolved ambiguity in ~70% of cases, primarily using Correction Type 2 (identifying the word as foreign but using the English sentence context to infer the correct meaning).
  • English Homographs: The model failed to correct itself in 51% of cases, ignoring non-English context cues and relying solely on the word's English meaning.
• Confusion: Models frequently misidentified the language of non-English words (e.g., labeling German words as Dutch) and often marked semantically nonsensical sentences as "meaningful."

5. Significance and Conclusion

Theoretical Implications:
The study challenges the assumption that multilingual LLMs possess an integrated mental lexicon similar to human bilinguals. While humans exhibit "cognate facilitation" (faster processing of cognates) and "homograph interference" (slower processing of homographs) driven by semantic competition, LLMs exhibit orthographic facilitation but semantic blindness. The lack of correlation between orthographic disambiguation and semantic retrieval suggests that LLMs process words as statistical patterns of tokens rather than grounded concepts.

Practical Implications:

• AI Safety & Reliability: The inability to disambiguate interlingual homographs in context poses risks for translation services, legal analysis, and medical applications where precise meaning is critical.
• Model Design: Future multilingual models need to be trained with stronger referential grounding to link orthographic forms to distinct semantic representations across languages, rather than relying on shared sub-word tokens.
• Prompting Strategies: The findings suggest that current prompting strategies may inadvertently reinforce English-centric biases, as models struggle to reason within non-English contexts.

Conclusion:
Multilingual LLMs currently function as "stochastic parrots" regarding bilingual processing. They excel at recognizing surface-level orthographic similarities (cognates) but fail to resolve deep semantic ambiguities (homographs) without explicit, strong contextual grounding. This highlights a fundamental gap between current LLM architectures and the integrated, non-selective lexical access mechanisms observed in human bilingual cognition.