Technical Summary: STEEREVAL – Inference-time Interventions for Multilingual Neural Summarization Metrics

Problem Statement

The evaluation of Natural Language Generation (NLG) tasks, particularly summarization, faces a significant bottleneck in multilingual settings. While metrics like BLEU and ROUGE established automatic evaluation, recent trends favor model-based metrics (e.g., BERTScore, COMET) and Large Language Models (LLMs) as judges due to their stronger correlation with human judgments. However, empirical studies indicate that these multilingual metrics often fail to correlate well with human judgments across diverse languages.

The authors hypothesize that this performance gap stems from the internal architecture of multilingual LLMs, which frequently utilize English as a "pivot language" for processing. When a model processes a low-resource language, the internal representations may misalign with this English pivot, leading to degraded downstream performance. This misalignment, previously linked to issues in machine translation, is proposed here as a root cause for poor metric performance in non-English summarization evaluation.

Methodology

The paper proposes STEEREVAL, a framework that applies inference-time interventions to "steer" the internal activations of neural metrics toward an English pivot. This approach requires no model retraining. The methodology involves two primary intervention techniques applied at test time:

1. Vector-based Intervention: Based on the linear representation hypothesis, the authors compute a steering vector representing the direction from a source language ($h_s$) to English ($h_t$) using parallel data (FLORES dataset).

   • The vector is calculated as the difference between the mean representations of the target and source languages: $\bar{h}_v = \bar{h}_t - \bar{h}_s$.
   • At inference, this vector is added to the model's activations: $h^{mix}_{q,l} = h_s + \rho \bar{h}_v$, where $\rho$ controls the steering strength.

2. Map-based Intervention: Inspired by recent work on cross-lingual generation, this method learns a linear mapping matrix $W_l$ that projects source language representations into the target language space.

   • The mapping is learned by minimizing the distance between source and target representations on parallel data.
   • At inference, the source representation is projected and mixed with the original: $h^{mix}_{q,l} = (1 - \sigma)h_s + \sigma \hat{h}_t$, where $\sigma$ controls the degree of deviation.

The authors evaluate these interventions on two types of metrics:

• LLM-as-a-Judge: Direct prompting (scoring 1–5) and GPTScore (using conditional generation probabilities). Models tested include Llama-3-8B, Bloom-7B, and Aya-Expanse (8B and 32B).
• Encoder-based Metric: COMET (specifically the wmt22-comet-da model), which is typically used for machine translation but adapted here for summarization.

The evaluation is conducted on a meta-evaluation dataset covering eight typologically diverse languages (Arabic, Spanish, Hebrew, Japanese, Turkish, Ukrainian, Yoruba, Chinese) with human judgments on coherence and completeness.

Key Results

The study demonstrates that steering activations toward an English pivot consistently improves the correlation between neural metrics and human judgments:

• General Improvement: Steering methods (both Vector and Map) almost always improve correlation over unsteered baselines. In many cases, particularly where baseline performance was low, the relative improvement ($\Delta\%$) exceeded 100%, effectively doubling the correlation.
• Impact on Low-Resource Languages: Languages with weaker baseline performance (e.g., Hebrew, Turkish, Yoruba) benefited the most from steering. For instance, Bloom-7B's direct prompting on Japanese coherence improved from near-zero correlation to 0.18 after steering.
• Metric Specificity:
  • COMET: Showed high responsiveness to steering, with large improvements across nearly all languages and dimensions.
  • Direct Prompting: Exhibited highly variable behavior. While relative gains were often dramatic, they frequently occurred when baseline correlations were near zero, resulting in modest absolute correlations.
  • GPTScore: Benefited moderately from steering, with diminishing returns observed when baseline correlations were already strong.
• Vector vs. Map: Vector steering generally yielded larger gains than Map-based steering, particularly for COMET and low-performing language-metric pairs. However, the optimal steering strength ($\rho$ or $\sigma$) varied significantly by language, suggesting that a single global parameter is suboptimal.
• Pivot Language: Experiments using French as an alternative pivot language also resulted in significant improvements, supporting the hypothesis that steering toward a high-resource, well-aligned language enhances performance.

Key Contributions

1. Activation Steering for Metrics: The paper introduces and validates the application of activation steering techniques to multilingual summarization metrics. This is the first exploration of steering for encoder-based metrics (like COMET) and shows consistent improvements for LLM-based evaluators.
2. Multilingual Meta-Analysis: The authors provide a comprehensive meta-analysis of model-based summarization metrics across eight diverse languages, highlighting the inconsistency of current LLM-as-a-judge approaches in non-English settings.
3. Cross-Lingual Vector Similarity: The study empirically measures the similarity of language-to-English steering vectors across layers. It finds that these vectors are largely similar across languages (except for outliers like Yoruba), corroborating prior work on the shared geometry of language representations.

Significance and Claims

The paper claims that the internal reliance of multilingual LLMs on English as a pivot is a critical factor limiting the reliability of automatic evaluation in other languages. By explicitly correcting this misalignment at inference time, STEEREVAL offers a practical, training-free method to enhance metric performance.

The authors position this work as a step toward leveraging model-internal insights for practical evaluation scenarios. They note that while their focus is on summarization, the findings suggest that steering could be a generalizable strategy for improving multilingual generalization in neural metrics. The paper remains modest regarding future implications, suggesting that future work could investigate combining language-specific and dimension-specific interventions, but does not propose specific new applications beyond the scope of metric evaluation.