💬 NLP

RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer

The paper proposes RP-OPSD, a novel method that enhances multilingual reasoning transfer by guiding on-policy self-distillation to prioritize "reasoning pivots" over surface text, thereby outperforming existing baselines across 17 languages.

Original authors: Xinye Wang, Junxiao Liu, Shujian Huang

Published 2026-08-07

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinye Wang, Junxiao Liu, Shujian Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant but multilingual robot how to solve a tricky math puzzle. You speak English, but the robot needs to learn to think in Swahili, French, or Japanese. The big challenge isn't just translating the words; it's translating the thinking. If the robot gets the math right in English but stumbles when it tries to explain the steps in another language, it's like a student who knows the answer but can't write the essay. This is the world of "multilingual reasoning," where scientists try to help large language models (LLMs) use their smart logic skills in languages they don't know as well as English.

To do this, researchers often use a trick called "self-distillation." Think of it as the robot practicing alone: it tries to solve a problem, and then a "teacher" version of itself (which has a secret reference sheet in English) checks the work and says, "Good job!" or "Try again!" The robot learns from these corrections. However, a common problem with this method is that the teacher might be too picky about every single word the robot writes. It might get annoyed if the robot uses a slightly different way to say "therefore" or "equals," even if the math logic is perfect. This can confuse the robot, making it focus on sounding fluent rather than thinking correctly.

This paper introduces a new, smarter way to teach the robot, called RP-OPSD. The authors, Xinye Wang, Junxiao Liu, and Shujian Huang from Nanjing University, suggest that we shouldn't treat every word in a reasoning chain the same. Instead, we should identify the "pivots"—the critical moments where the robot makes a big decision, like choosing to multiply instead of add, or realizing a variable needs to change. These are the "aha!" moments of reasoning. The rest of the words are just "surface" details, like the grammar or the specific way to say "the cake is cut."

The researchers found that by using a special "gate" to decide which words need the teacher's strict attention and which words should just be left to the robot's natural language flow, the robot learns much faster. They tested this on 17 different languages, including many African languages that usually get less attention from AI. The results showed that their method helped the robot solve math problems better than previous methods, especially in those harder languages. It's like giving the robot a highlighter: it only highlights the most important logic steps for the teacher to check, while letting the robot write the rest of the story in its own voice. This suggests that to teach a robot to think in a new language, you don't need to micromanage every sentence; you just need to make sure the critical thinking steps are solid.

The Core Idea: Finding the "Pivots"

The paper argues that when a model reasons in a new language, it's doing two things at once:

Surface Realization: Writing the words in the correct language (e.g., saying "multiplie" in French instead of "multiply").
Reasoning Pivots: Making the actual logical decisions (e.g., deciding to multiply 4 by 3).

The authors noticed that previous methods treated these two things equally. They tried to force the robot to copy the teacher's English logic and the teacher's English word choices perfectly. But this is like a music teacher correcting a student's finger placement on the piano while also yelling about the student's accent. It's too much pressure.

The paper proposes that surface variants (different ways to say the same thing) are fine, but pivot drift (changing the logic) is dangerous. If the robot decides to add instead of multiply, the whole answer is wrong, no matter how fluent the French sounds.

How RP-OPSD Works: The "Gate"

To fix this, the authors created a system called RP-OPSD (Reasoning-Pivot-Guided On-Policy Self-Distillation). Here is the analogy of how it works:

Imagine the robot is writing a story about solving a math problem.

The Teacher View 1 (The Reference Sheet): The teacher sees the problem, the English solution, and the robot's current sentence.
The Teacher View 2 (No Reference Sheet): The teacher sees the problem and the robot's sentence, but without the English solution.

The system compares these two views.

If the teacher changes its mind about what word to say next only because it has the English solution, that word is a Pivot. It means the English logic is crucial here. The "Gate" opens wide, and the robot gets a strong lesson: "Copy the logic here!"
If the teacher says the same thing whether it has the English solution or not, that word is just Surface. It's probably just grammar or a common phrase. The "Gate" stays closed, and the robot is allowed to write naturally in its target language without being forced to copy the English style.

This "gate" is a mathematical filter that decides, for every single word the robot generates, whether it needs a heavy dose of English logic or just a gentle nudge to stay in the target language.

What They Found

The team tested this on 17 languages, ranging from high-resource ones like French and Spanish to low-resource African languages like Swahili and Yoruba. They used two main math benchmarks: AfriMGSM (for African languages) and PolyMath (a harder, multi-difficulty math test).

The results were clear:

Better Scores: RP-OPSD beat all the other methods they compared it to, including standard "Self-Distillation" and other advanced reasoning techniques. For example, on the Qwen3-1.7B model, it improved the average score on African languages from about 16.7% to 19.07%, and on the harder PolyMath test, it reached 17.97% accuracy.
Smarter Learning: The analysis showed that the "Gate" correctly identified the important moments. It focused on words like "therefore," "so," "but," and specific math operations (like "square root" or "trapezoid"). It ignored the boring stuff like variable names ( $x$ , $y$ ) or common connectors.
No "Language Leakage": A big fear in this field is that the robot will just start thinking in English and only translate the final answer. The paper shows that RP-OPSD actually keeps the robot thinking in the target language. It improved the robot's ability to reason in the target language without making it sound more like English. In fact, the robot's "language consistency" (how well it stuck to the target language) actually went up while its math scores went up.

What They Ruled Out

The paper explicitly argues against a few common ideas:

Uniform Weighting: They show that treating every word the same (giving every word the same importance) is a mistake. It dilutes the important logic signals with too much noise from surface-level words.
Just "Disagreement": They found that simply looking at where the teacher and student disagree isn't enough. Sometimes they disagree because of a typo or a grammar choice, not because of a logic error. The "Pivot" method is better because it specifically looks for where the English solution changes the outcome.
Translation Artifacts: They argue that just translating English reasoning into another language (Supervised Fine-Tuning) isn't enough because the translation might not match how the robot naturally thinks in that language.

The Bottom Line

The paper suggests that to teach a robot to reason in a new language, you don't need to force it to speak like an English speaker. You just need to make sure it understands the logic of the English solution at the exact moments where the logic matters. By using a "gate" to separate the "thinking" words from the "talking" words, the robot learns to think deeply in its new language without losing its natural voice.

The authors are confident in these results based on their experiments across 17 languages and multiple model sizes. They suggest that this approach could be a key step toward making AI truly multilingual, capable of solving complex problems in any language, not just the ones with the most data.

Technical Summary: RP-OPSD

Problem Statement

Multilingual reasoning transfer aims to extend the reasoning capabilities of Large Language Models (LLMs) from high-resource languages (e.g., English) to low-resource languages. While On-Policy Self-Distillation (OPSD) has emerged as a promising paradigm by providing dense token-level supervision on student-generated rollouts, existing variants suffer from a critical limitation: they assign uniform weight to all generated tokens. This approach fails to distinguish between surface realization tokens (language-specific syntax and fluency) and reasoning pivots (decisions that advance or redirect the inference process).

The authors argue that in multilingual settings, surface variants can preserve the same reasoning state, whereas "pivot drift"—errors in critical decision-making tokens—can corrupt downstream reasoning even if the continuation remains fluent. Existing methods either rely on teacher confidence (which may not reflect cross-lingual reasoning relevance) or require difficult-to-obtain annotations for key spans. Furthermore, simple teacher-student disagreement (KL divergence) is a coarse signal that often highlights formatting or lexical differences rather than reasoning-critical positions.

Methodology: RP-OPSD

The paper proposes RP-OPSD (Reasoning-Pivot-Guided On-Policy Self-Distillation), a method that selectively routes privileged distillation to reasoning-sensitive tokens while anchoring surface tokens to the target language.

1. Teacher Views and Privileged Reasoning Sensitivity (PRS)

Instead of using a single teacher view, RP-OPSD contrasts two views of the same policy ( $\pi_\theta$ ) under different information states:

Solution-Conditioned View ( $q^+_t$ ): The policy conditioned on the low-resource question, its English translation, the rollout prefix, and the English reference reasoning trace.
Ablated View ( $q^-_t$ ): The policy conditioned on the same inputs but without the English reference trace.

The Privileged Reasoning Sensitivity (PRS) score, denoted as $a_t$ , is calculated as the KL divergence between these two views:
$a_t = D_{KL}(q^+_t \parallel q^-_t)$
A high $a_t$ indicates that the presence of the English solution significantly alters the model's next-token prediction, suggesting the token is a reasoning pivot where privileged evidence is critical. Conversely, low $a_t$ suggests the token is governed by target-language realization or generic priors.

2. Reasoning-Pivot Transfer (RPT) Gate

The PRS score is normalized and passed through a sigmoid function to generate a continuous gate $g_t \in [0, 1]$ :
$g_t = \text{sg}[g_{min} + (1 - g_{min})\sigma(\beta(\tilde{a}_t - \tau))]$
This gate acts as a routing coefficient:

High $g_t$ : The token is treated as a reasoning pivot.
Low $g_t$ : The token is treated as surface realization.

3. Routed Training Objective

The training objective combines two full-vocabulary forward-KL terms, routed by the gate:

Pivot-Transfer Term ( $L_{pivot}$ ): Aligns the student distribution ( $p_t$ ) with the solution-conditioned teacher ( $q^+_t$ ) at high-gate positions. This transfers the privileged reasoning logic.
$L_{pivot} = \frac{1}{N} \sum_{t=1}^T m_t g_t D_{KL}(q^+_t \parallel p_t)$
Anchoring Term ( $L_{anchor}$ ): Aligns the student with a frozen reference policy ( $\pi_{ref}$ ) at low-gate positions. This preserves the target-language surface realization and prevents the model from overfitting to the English solution's syntax.
$L_{anchor} = \frac{1}{N} \sum_{t=1}^T m_t (1 - g_t) D_{KL}(r_t \parallel p_t)$

The final loss is $L_{RP-OPSD} = L_{pivot} + \lambda L_{anchor}$ .

Key Contributions

Identification of Reasoning Pivots: The paper characterizes multilingual reasoning as an interleaving of surface text and reasoning pivots, arguing that transfer efficiency depends on isolating the latter.
Distributional Shift Proxy: It introduces a novel, annotation-free mechanism to identify pivots by measuring the distributional shift between a teacher view with and without privileged reasoning evidence. This avoids the pitfalls of using raw teacher-student disagreement.
Dual-Objective Routing: The method simultaneously performs privileged distillation on reasoning-critical tokens and reference anchoring on surface tokens, balancing reasoning transfer with language fidelity.

Experimental Results

The method was evaluated on 17 languages (including 12 African languages) across two mathematical reasoning benchmarks: AfriMGSM and PolyMath.

Performance: RP-OPSD outperformed strong baselines, including SFT, GRPO, MAPO-DPO, M-Thinker, and existing OPSD variants (COPSD, EGRSD).
- On AfriMGSM (Qwen3-1.7B), RP-OPSD achieved an average pass@12 of 19.07%, surpassing COPSD (16.70%) and M-Thinker (14.87%).
- On PolyMath (Qwen3-4B), it achieved a difficulty-weighted accuracy (DW-ACC) of 31.87%, outperforming COPSD (29.94%) and M-Thinker (29.74%).
Ablation Studies:
- Gate Localization: Selecting the top 20% of tokens based on the RPT gate (TG) significantly outperformed random (RG) or bottom-gate (BG) selection, confirming that the gate correctly identifies transferable reasoning signals.
- Anchoring: Removing the reference anchoring term degraded performance, particularly in languages with stronger baseline capabilities (e.g., French), indicating that anchoring is crucial for preserving target-language fluency.
Analysis:
- Token Types: High-gate tokens were found to correspond to reasoning-control transitions (e.g., "therefore," "but") and problem-conditioned state updates (e.g., specific variables, operators), while low-gate tokens were mostly surface realizations (variables, syntax).
- Reachability: RP-OPSD not only transferred English-reachable solutions but also extended reasoning capabilities beyond the base model's English reachability, doubling the gain in "beyond-English" solutions compared to other baselines.
- Language Consistency: Unlike methods that improve accuracy by drifting into English reasoning patterns, RP-OPSD improved reasoning accuracy while maintaining or increasing language consistency scores.

Significance and Claims

The paper claims that RP-OPSD offers a more efficient and effective paradigm for multilingual reasoning transfer by concentrating privileged supervision where it matters most. By distinguishing between the "what" (surface realization) and the "how" (reasoning logic) of CoT generation, the method avoids diluting the reasoning signal across fluent but non-critical tokens.

The authors assert that their approach enables models to share computational structures across languages while retaining natural, language-specific forms of expression. The method is presented as a step toward multilingual reasoners that do not merely translate English reasoning but adapt the underlying reasoning logic to the target language's constraints, achieving superior performance without requiring translated reasoning traces or complex reward engineering.