Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap

Technical Summary: Mind the Cap

Problem Statement
Multilingual evaluations of Large Language Models (LLMs) frequently report a "reasoning gap," where models perform better on reasoning tasks when the problem is translated into English (e.g., via a "Translate-Act" strategy) compared to reasoning directly in the native language. However, these evaluations typically measure accuracy at a single, fixed output-token cap. The paper argues that this cap is a hidden experimental variable. Because different languages require different numbers of tokens to express the same content (tokenization premiums), a fixed cap disproportionately truncates native-language traces compared to English-translated ones. The authors posit that the measured "reasoning gap" may be an artifact of this token budget mismatch rather than a genuine deficit in multilingual reasoning ability.

Methodology
The study operationalizes the "token-budget artifact" by analyzing the MGSM (Multilingual Grade School Math) benchmark across three languages (German, Thai, Swahili) and two models (Qwen3-8B and Llama-3.1-8B-Instruct).

1. Experimental Design:

   • Strategies: Four prompting strategies were evaluated: NATIVE (reasoning and answering in the target language), TRANSLATE-ACT (translate to English, solve, answer in target language), PIVOT (reason in English, answer in target language), and CODE-SWITCHED.
   • The Ledger: The authors generated 48,000 stored generations (4096 tokens each) and scored every prefix of these generations. This "ledger" allows for retrospective evaluation at various budget levels ($B$) without re-generating data, eliminating between-budget sampling noise.
   • Independent Decoding: To validate findings, a separate sample of 540,000 decodes was generated with hard caps enforced at specific budgets, ensuring no shared trajectories between discovery and confirmation phases.

2. Estimand and Metrics:

   • The core metric is $\Delta_L(B) = acc_N(\lfloor r_{m,L} B \rfloor) - acc_N(B)$, where $acc_N$ is native accuracy, $B$ is the budget, and $r_{m,L}$ is the FLORES-200 token premium of language $L$ over English.
   • This metric isolates the gain in native accuracy when the budget is scaled to match the token cost of the English translation, effectively asking: "How much of the gap is due to the native trace being cut off?"
   • Falsifiable Hypothesis: If the gap is a budget artifact, scaling the native budget should close the gap. If the gap persists after scaling, it suggests a genuine reasoning deficit.

3. Statistical Rigor:

   • The study employs a "prospectively frozen" design. A specific budget ($B^*=1024$) and a set of six Holm-corrected tests were defined before data analysis.
   • Audits: Five ledger-computable audits were conducted: trace-language ID (GlotLID), translation quality (COMET), parser robustness, decoder parity, and normalizer sensitivity.

Key Results

1. Budget Dependence of the Gap:

   • The measured gap is highly sensitive to the output budget. Under tight caps, the gap can swing by up to 57 points.
   • Length Normalization: Applying length normalization (scaling the native budget by the token premium) moves the gap by up to 38.9 points in the "budget-binding regime" (where answers are still being emitted).
   • Strategy Reversal: At very tight caps, length normalization can reverse which strategy appears superior. For example, in Qwen German and Swahili, the NATIVE strategy outperforms TRANSLATE-ACT at low budgets (e.g., 128 tokens) because the translation preamble consumes too many tokens, leaving no room for the answer.

2. The Frozen Test at $B^*=1024$:

   • At the pre-specified budget of 1024 tokens, the study failed to reject the null hypothesis that the gap is zero for Qwen (and Llama).
   • Reason: Native accuracy had already saturated at this budget (e.g., Qwen German native accuracy plateaued at 79%). Once accuracy saturates, the residual difference between strategies is a "strategy-performance gap" (driven by translation quality, prompt reformulation, etc.) rather than a "reasoning deficit" caused by truncation.
   • The study concludes that the "reasoning gap" is not identified at saturation; it is only visible in the binding regime where traces are truncated.

3. Independent Confirmation:

   • A second family of tests using independently capped decodes confirmed that the large gaps observed at discovery peaks (e.g., 38.9 points for Thai at 256 tokens) were not artifacts of shared trajectories.
   • However, at the saturation point ($B=1024$), the independent tests also failed to reject the null, reinforcing that the gap vanishes when the budget is sufficient for the native trace to complete.

4. Announcement Effects:

   • The study found that announcing a budget to the model changes behavior even when the enforced cap remains fixed. Announcing a tighter budget (128 vs. 2048) increased Thai native accuracy by 5.1 points, suggesting accuracy is not solely a function of the enforced cap but also of the model's internal budget estimation.

5. Adaptation Ladder (Vocabulary Extension):

   • The authors tested if extending the vocabulary (adding target-language tokens) could close the gap.
   • Result: Vocabulary extension only closed the gap where the cap was still truncating traces. At saturation ($B=1024$), where no traces were truncated, the extension closed 0.00 points of the gap.
   • This confirms that the gap is driven by truncation, not by a lack of vocabulary or reasoning capability per se.

Significance and Claims
The paper claims a methodological shift in how multilingual reasoning should be evaluated:

• The Cap is a Variable: Output caps should be treated as independent variables, not background constants. Reporting accuracy at a single budget is insufficient and potentially misleading.
• Reframing the Gap: The "multilingual reasoning gap" is often a "budget-binding artifact." When native traces are allowed to finish (via length normalization or larger caps), the gap diminishes or disappears in the binding regime.
• Saturation vs. Deficit: Once native accuracy saturates, the remaining difference between native and translated strategies reflects execution factors (translation quality, prompt adherence) rather than a fundamental inability to reason in the native language.
• Practical Guidance: Before investing in costly adaptations (like fine-tuning or vocabulary extension), practitioners should measure the realized gain from simply extending the output cap. If the cap is non-binding, these interventions will yield no benefit.

The authors are modest in their scope, noting that their findings are specific to the MGSM benchmark, the tested models, and the specific "exact match" evaluation metric. They do not claim that multilingual reasoning is identical to English reasoning in all contexts, but rather that the measured gap in this specific setup is largely an artifact of token budgeting.