💻 computer science

Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMs

This paper introduces Code-Switching In-Context Learning (CSICL), an inference-time mechanism that gradually transitions from target languages to English to align non-English inputs with English-centric reasoning spaces, thereby significantly improving multilingual LLM performance across diverse languages and low-resource settings.

Original authors: Haneul Yoo, Jiho Jin, Kyunghyun Cho, Alice Oh

Published 2026-08-14

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Haneul Yoo, Jiho Jin, Kyunghyun Cho, Alice Oh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant, multilingual robot how to solve a tricky puzzle. You speak to it in Spanish, French, or Swahili, expecting it to think and answer in that same language. But here's the catch: deep inside its "brain," this robot has a secret habit. No matter what language you speak to it, it secretly translates everything into English first, does its thinking in English, and then translates the answer back. This works great if the robot is fluent in English, but if the language you speak is rare or complex, that secret translation step can get messy. The robot might get confused, lose the meaning, or give a wrong answer because the "bridge" between your language and its English brain is shaky. This is a big problem for making AI helpful for everyone, everywhere, not just English speakers. Scientists call this "cross-lingual misalignment." To fix it, researchers have tried showing the robot examples in different languages, but often those examples are just abrupt jumps—like asking the robot to suddenly switch from Spanish to English without any warning. It's like asking someone to jump off a diving board without a ladder; they might panic and miss the water.

This paper introduces a clever new trick called Gradual Code-Switching In-Context Learning (CSICL). Instead of forcing the robot to jump languages, the researchers teach it to walk across a bridge, step by step. They show the robot a series of examples where the language slowly melts from the target language (like Korean) into English. Imagine a sentence that starts 100% Korean, then becomes 75% Korean and 25% English, then 50/50, then 25% Korean, and finally 100% English. By watching this smooth transition, the robot learns to align its internal "English thinking" with the non-English input without getting shocked. The researchers tested this on four different large language models across ten languages and six different types of tasks, from answering trivia to solving math problems. They found that this gentle, gradual approach consistently helped the robots perform better, especially in languages they hadn't seen much before. In fact, for low-resource languages (the ones the robots know the least about), this method boosted performance by a massive 14.7 percentage points. The paper suggests that by scaffolding the reasoning process with these gradual transitions, we can help AI models become more fair and effective for speakers of all languages, turning a shaky bridge into a sturdy highway.

Technical Summary: Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMs

Problem Statement

Despite significant progress in multilingual capabilities, Large Language Models (LLMs) exhibit uneven performance across languages. This disparity stems from an over-reliance on English-centric latent representations. Recent studies indicate that LLMs often internally translate non-English inputs into English before reasoning, creating a "translation barrier." If this initial internal translation fails, the quality of the final output degrades sharply.

Existing approaches to mitigate this, such as Cross-Lingual In-Context Learning (X-ICL), typically rely on an abrupt, single-step pivot. For instance, a model might be prompted with English demonstrations to solve a task in a target language, or a query might be translated entirely into English before reasoning. These methods often transfer superficial task patterns (e.g., output formats) without effectively bridging the underlying representational gap between the target language and the model's English-centric latent space. Consequently, performance in low-resource and unseen languages remains suboptimal.

Methodology: Code-Switching In-Context Learning (CSICL)

The authors propose CSICL, a training-free, inference-time mechanism designed to align cross-lingual representations by explicitly scaffolding the reasoning trajectory. Instead of an abrupt transition, CSICL employs a gradual code-switching strategy that transitions from the target language to English.

Core Mechanism

CSICL operates through two primary components:

Gradual Translation Instruction: The model is instructed to process a non-English input by gradually translating it into English, thinking in English, and then generating the final answer in the target language.
Gradual Code-Switching Demonstrations: The few-shot examples provided to the model do not switch languages abruptly. Instead, they follow a stepwise progression:
- 0% (Target Language): The query is entirely in the target language.
- 25% - 75% (Intermediate Steps): The query progressively incorporates English tokens while retaining the grammatical structure of the target language (Matrix Language Frame model).
- 100% (English): The query is fully translated into English.

This gradual shift acts as a "linguistic bridge," nudging the LLM to dynamically align non-English inputs with its English-centric reasoning space before the actual reasoning step occurs.

Implementation Details

Demonstration Construction: For each target language, the authors generate gradual code-switching sequences. While the default implementation uses GPT-5 for demonstration construction, CSICL is not fundamentally restricted to an external oracle. The method supports self-generation (using the same model) or rule-based generation. The method is not restricted to using GPT-5 as the inference model; the paper evaluates CSICL on four distinct multilingual LLMs: Qwen3-32B, deepseek-chat-v3.1, grok-4-fast, and Gemini 2.5 Flash.
Directionality: The method specifically transitions from Target Language $\to$ English. Ablation studies suggest this direction is superior to the reverse (English $\to$ Target), as it aligns the input with the model's latent space rather than diverging from it.
Granularity: The authors tested various step counts (e.g., 3-step, 5-step, 7-step transitions) and found that a 5-step linear transition generally offers the best balance between performance and token efficiency.

Key Contributions

Proposal of CSICL: A novel inference-time mechanism that bridges target languages and English representations through structured, gradual code-switching, avoiding the pitfalls of abrupt pivots.
Systematic Empirical Evidence: Comprehensive evaluation across 4 multilingual LLMs (Qwen3, DeepSeek, Grok, Gemini), 6 datasets (including Global MMLU, MedExpQA, PolyMath), and 10 languages (spanning high, mid, and low-resource settings).
Latent Space Visualization: The paper provides visual evidence (via PCA of hidden states) showing that CSICL systematically shifts non-English input representations closer to the English anchor in the latent space, validating the alignment hypothesis.
Design Factor Analysis: The authors analyze critical factors including directionality, granularity, and the source of demonstration generation, demonstrating that CSICL remains effective even when using self-generated or rule-based demonstrations.

Experimental Results

CSICL consistently outperforms standard X-ICL baselines (including monolingual, parallel, and translation-based prompting) across all evaluated settings.

Performance Gains:
- Target Languages: Average gain of 6.0 percentage points (pp) over monolingual baselines.
- Unseen Languages: Average gain of 4.8 pp.
- Low-Resource Settings: The improvements are most pronounced in low-resource scenarios, with gains of 14.7 pp in target languages and 5.3 pp in unseen languages.
Task Specificity:
- Translation: CSICL achieved the largest gains (+6.8 pp in target languages), as the gradual transition directly scaffolds the latent translation process.
- Reasoning: Significant improvements were observed in reasoning-oriented tasks (+5.4 pp), suggesting that "thinking in English" via gradual alignment mitigates representational interference.
- Knowledge: Modest but consistent gains were found in cultural knowledge and social bias tasks.
Efficiency: While the full CSICL configuration (5 shots, 5 steps) uses more tokens than simpler baselines, the overhead is manageable within standard context windows. Notably, even 0-shot and 1-shot versions of CSICL outperform strong baselines while using fewer tokens than many existing X-ICL alternatives.

Significance and Claims

The paper positions CSICL as a robust and effective approach for reducing cross-lingual misalignment during inference without requiring model retraining or fine-tuning.

Equitable Deployment: By improving performance in low-resource and unseen languages, CSICL moves LLMs toward more equitable multilingual systems, addressing the current limitation where competence is heavily skewed toward English.
Mechanism Insight: The work establishes that gradual code-switching is not merely a stylistic choice but a functional mechanism that controls the reasoning trajectory, allowing models to leverage their strongest English reasoning capabilities while maintaining target language fidelity.
Practical Utility: The method is presented as a lightweight, tunable solution that offers a favorable efficiency–accuracy trade-off, making it suitable for both budget-constrained settings (via 0/1-shot variants) and high-accuracy requirements.

The authors conclude that CSICL represents a new lens for cross-lingual alignment, demonstrating that explicitly scaffolding the transition from target language to English can significantly enhance the multilingual capabilities of existing LLMs.