What Makes Good Multilingual Reasoning? Disentangling Reasoning Traces with Measurable Features

1. Problem Statement

Large Reasoning Models (LRMs) exhibit significant performance gaps between English and non-English languages. Current approaches to bridging this gap often rely on English-centric assumptions, such as translating queries into English before reasoning or rewarding reasoning traces that are structurally or semantically similar to English references.

The authors challenge the assumption that "good" reasoning in English transfers unchanged to other languages. They posit that:

• Models may reason more effectively in their native language.
• Typologically distant languages may follow distinct but equally valid reasoning trajectories.
• Blindly rewarding English-like patterns risks penalizing language-specific behaviors that lead to correct answers.

The core research question is: What actually characterizes effective reasoning in multilingual settings, and to what extent do English-derived reasoning features generalize?

2. Methodology

The study employs a three-pronged approach to analyze reasoning traces across 10 languages, 4 LRMs, and 2 mathematical benchmarks (MGSM-Rev2 and AIME 2024–25).

A. Defining Measurable Reasoning Features

The authors curated a suite of 16 human-interpretable features categorized into three dimensions (Table 1):

1. Multilingual Alignment (3 features): Measures how well non-English traces align with English counterparts.
   • COMET-QE: Translation quality of the query.
   • Structural Similarity: Sequence alignment of reasoning flow tags (Smith-Waterman algorithm).
   • Semantic Similarity: Cosine similarity of embeddings (LABSE) between English and non-English traces.
2. Reasoning Step (5 features): Step-level quality metrics.
   • Num. Steps: Trace length.
   • Validity: Logical consistency (NLI-based entailment).
   • Direct/Indirect Utility: Proportion of steps contributing to the final answer.
   • V-Information: How much the trace increases model confidence in the gold answer.
3. Reasoning Flow (8 features): High-level cognitive patterns (adapted from Bogdan et al., 2025).
   • Includes Self-Checking, Active Computation, Plan Generation, Uncertainty Management, Final Answer Emission, etc.
   • Annotated using GPT-4O to tag steps and build dependency graphs.

B. Feature Analysis (Regression)

The authors use univariate logistic regression to quantify the association ($\Delta Acc$) between each feature and final answer accuracy per language.

• They calculate the change in predicted accuracy when a feature moves from -1 to +1 standard deviation.
• They test for significant differences between English and non-English associations using interaction terms.

C. Sparse Autoencoder (SAE) Analysis

To move beyond hand-designed features, they train Sparse Autoencoders (SAEs) on multilingual reasoning traces.

• Traces are chunked, embedded (LABSE), and fed into a Batch TopK SAE.
• Predictive neurons (correlated with accuracy) are identified and interpreted by prompting GPT-4O to describe the latent concept (e.g., "repeating phrases," "using logarithmic identities").
• This serves to validate hand-designed features and discover new, latent reasoning patterns.

D. Test-Time Selection

The authors treat feature values as selection policies in a "best-of-n" setting.

• For each query, 32 candidate traces are generated.
• Traces are re-ranked based on specific feature values (e.g., highest Direct Utility).
• The top-ranked trace is selected, and the resulting pass@1 accuracy is measured to see if the feature can steer the model toward better performance.

3. Key Results

RQ1: What Characterizes Effective Multilingual Reasoning?

• General Trends: Most features (e.g., Validity, Utility, Active Computation) show positive associations with accuracy in both English and non-English languages.
• Language-Specific Variance: The strength of these associations varies significantly.
  • Conflict: Some features beneficial in English are detrimental in specific languages. For example, Self-Checking improves English accuracy but correlates negatively with accuracy in Swahili and Telugu.
  • Reversal: On the harder AIME benchmark, Validity is strongly positive for English but can be negative for other languages.
• Benchmark Differences: On easy benchmarks (MGSM-Rev2), features often show near-zero effects (models solve via latent reasoning). On hard benchmarks (AIME), feature effects are more pronounced.
• Alignment: Structural Similarity to English traces is often a stronger predictor of accuracy than Semantic Similarity, especially for long traces.

RQ2: Can We Automatically Discover Features?

• SAE Validation: SAEs largely recovered patterns found in hand-designed features (e.g., "uncertainty management" concepts correlated with lower accuracy).
• New Discoveries: SAEs revealed granular behaviors not captured by the initial set, such as:
  • Negative: "Mixing multiple languages within a trace" (-36% separation score).
  • Positive: "Breaking down multi-step reasoning with ordinal adverbs" (+31%) or "Using logical deductions to verify calculations" (+38%).

RQ3: Can Features Steer Models?

• Selection Efficacy: Using features as selection policies significantly improves accuracy on hard benchmarks (AIME).
  • Selecting by Direct Utility or Result Consolidation yielded up to 10% higher accuracy than random selection.
• Limitations of English-Centric Rewards: Semantic Similarity to English traces was competitive but not universally best. For some models, alternative features (like Utility) outperformed semantic alignment.
• Negative Signals: Features like Uncertainty Management consistently lowered accuracy when used as selection criteria.

4. Key Contributions

1. Systematic Feature Disentanglement: Introduced a comprehensive, measurable framework for analyzing multilingual reasoning traces, moving beyond final answer accuracy to intermediate step behaviors.
2. Evidence of Language-Specific Reasoning: Demonstrated that "good" reasoning is not monolithic; features that predict success in English do not always predict success in other languages, and some English-centric behaviors (like excessive self-checking) can be harmful in specific linguistic contexts.
3. SAE for Reasoning Analysis: Successfully applied Sparse Autoencoders to multilingual reasoning traces to discover latent concepts, bridging the gap between hand-crafted metrics and representation learning.
4. Practical Steering Mechanism: Proved that reasoning features can be used as effective test-time selection policies to improve multilingual performance without retraining.

5. Significance and Implications

• Challenging English-Centricity: The findings argue against the "translate-to-English" or "mimic-English-reasoning" paradigm. Reward models and training objectives should not blindly favor English-like traces.
• Adaptive Objectives: The paper advocates for adaptive reward designs that can flexibly decide when to remain in the original language, when to translate, and which feature signals (e.g., utility vs. semantic similarity) to prioritize based on the target language.
• Benchmark Design: Highlights the need for high-quality, human-verified translations in multilingual benchmarks (as COMET-QE scores correlated with accuracy) and suggests that step-level evaluation metrics (like Utility) are transferable across languages.
• Future Directions: Points toward developing language-specific reasoning patterns and adaptive reasoning models (ARMs) that dynamically adjust their reasoning strategies based on the input language.

6. Limitations

• Scope: Limited to mathematical reasoning; findings may not transfer to domains like legal or commonsense reasoning.
• Annotation: Reliance on GPT-4O for feature annotation, though human verification showed high agreement.
• Models: Evaluated on four specific LRMs; patterns may vary with different architectures or training regimes.