Technical Summary: Actions Speak Louder than Words

Problem Statement

Current multilingual evaluations of tool-using agents focus almost exclusively on final answer correctness, discarding the intermediate action traces. This approach assumes that if an agent produces the correct answer in multiple languages, its behavior is consistent. However, for an agent, the "route" (the sequence of tool calls and intermediate steps) is the actual product. Divergent routes lead to different costs, latencies, failure modes, and auditability issues. A safeguard validated on an English trace may not hold for a non-English trace if the underlying action policy differs, even if the final output is identical.

The core problem is the lack of a defensible metric to measure cross-lingual policy retention: the degree to which an agent executes the same sequence of actions for semantically identical tasks across different languages. Standard similarity measurements are unreliable due to five specific confounds:

1. No Baseline: Models are not perfectly self-consistent; a model asked the same question twice in the same language often produces different traces. Without a same-language baseline, cross-language differences cannot be isolated.
2. Trace Length Bias: Shorter traces artificially score higher on similarity metrics.
3. Empty Trace Bias: Traces with no actions score perfectly (1.0) on similarity, skewing results.
4. Ceiling Effect: The maximum possible agreement is bounded by a model's own reproducibility (self-consistency).
5. Chance Floor: Unrelated traces agree by chance more than 50% of the time, creating a non-zero baseline for similarity.

Methodology

The authors propose a protocol to measure Normalized Policy Retention ($\tilde{I}$) by explicitly correcting for the five confounds listed above.

• Experimental Setup:

  • Models: 8 instruction-tuned models (including Gemma-3, Sarvam-M, Qwen3, Llama-4, and GPT-OSS) ranging from 4B to 235B parameters, covering dense and mixture-of-experts architectures.
  • Tasks: 6 parallel benchmarks (FLORES-200, XQuAD, XNLI, Belebele, XCOPA, and a synthetic suite) rendered in 41 languages.
  • Scale: 2.38 million agent rollouts across 505 cells.
  • Scaffold: A fixed symbolic tool-use scaffold (Thought/Action/Finish) with a shared 5-tool alphabet. Traces are extracted via regex; tools are symbolic (parsed but not executed).

• Measurement Protocol:

  • Replicates: Every task-language pair is generated twice under identical conditions (same seed, task, budget) to establish a same-language baseline ($I_{within}$).
  • Cross-Language Comparison: Traces from different languages are compared ($I_{cross}$).
  • Normalization: The metric is defined as $\tilde{I} = I_{cross} / I_{within}$. This normalizes cross-lingual agreement against the model's own self-consistency ceiling.
  • Corrections:
    • Length-matching in both directions to eliminate trace length bias.
    • Exclusion of empty traces.
    • Measurement of the chance floor via permutation rather than assumption.
    • Correction for the ceiling effect (self-consistency).

• Interventions:

  • Temperature Ladder: Testing across temperatures $T \in \{0, 0.3, 0.5, 0.7, 1.0\}$ to distinguish sampling noise from structural divergence.
  • English Pivot Ablation: Removing the Translate tool or mandating it as the first step to test causal reliance on English.
  • Trace Length Manipulation: Artificially varying trace lengths to isolate length effects.

Key Results

1. The Effect Grows with Correction

Contrary to the intuition that correcting for confounds might reduce the observed effect, every correction applied (length-matching, empty trace removal, chance floor adjustment) increased the measured divergence. The raw gap between languages was a lower bound; the corrected gap was significantly larger.

2. Convergence at the Frontier

Under greedy decoding ($T=0$), four diverse frontier models (spanning different scales, architectures, and training mixtures) converged on a nearly identical policy retention rate: 71–73%.

• Model identity explained only 5.7% of the variance in this normalized metric.
• In contrast, at $T=0.5$ (sampling included), model identity explained 74.8% of the variance, and rankings inverted completely.
• This suggests that cross-lingual policy retention is a structural property of the models under greedy decoding, not an artifact of sampling noise.

3. The Boundary of Regularity

The 71–73% regularity breaks down for models below approximately 10B parameters.

• Smaller models (4B, 8B) showed high variability.
• The apparent ordering of small models based on raw metrics was largely an artifact of the "chance floor" (shorter traces in smaller models led to artificially high similarity scores). Once corrected, the ranking of small models reversed.

4. The English Pivot Mechanism

The study identifies a causal mechanism for the divergence: Agents route non-English tasks through an English pivot.

• Translate was the most used tool in every benchmark for compliant models.
• Reasoning text was ~99% ASCII even for non-Latin scripts.
• Causal Test: Removing the Translate tool reduced cross-lingual agreement proportionally to the model's reliance on it. Mandating translation improved retention for models with "head-room" (low baseline translation usage).
• Robustness: Instructing models to reason in the task's language failed (<1% compliance), indicating the English pivot is a structural preference, not a stylistic default that can be prompted away.

5. Measurement Pathology

The paper highlights a critical measurement failure: a single trace-extraction regex caused a competent model (GPT-OSS-120B) to appear broken.

• The model answered in prose ("We will use Translate") rather than the required syntax, resulting in 76.4% empty traces.
• Because empty traces score 1.0 on similarity, the model appeared to have high invariance but near-zero accuracy.
• Adding two few-shot exemplars restored parseability, increasing measured accuracy 26-fold without changing the model's actual capability, only its legibility.

Significance and Claims

The paper claims that answer-level parity is compatible with significant behavioral divergence, which has real-world implications for cost, latency, and safety.

• Metric Contribution: The authors provide a ceiling-corrected estimand ($\tilde{I}$) and a protocol that prices five specific confounds, two of which are established causally.
• Structural Insight: Cross-lingual divergence is not random noise but a structural property where models retain roughly 71–73% of their action policy when language changes, provided they are large enough and decoded greedily.
• Mechanism: The reliance on an English pivot is a load-bearing causal factor that models resist abandoning even when explicitly instructed otherwise.
• Validity: The study demonstrates that standard regex-based trace extraction can manufacture apparent multilingual failures, necessitating the reporting of parse-failure rates alongside accuracy.

The work bridges the gap between agentic benchmarks (which focus on outcomes) and cross-lingual evaluation (which focuses on final predictions), arguing that the action policy itself must be the measured object to truly understand multilingual agent behavior.