🤖 AI

Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast

This paper introduces Contribution-Contrast (CoCo), a novel response-level interpretation method for Mixture-of-Experts reward models that overcomes the limitations of routing-weight-based analysis by identifying experts' roles through chosen-rejected response pairs with the largest contribution contrasts, thereby delivering more faithful and specialized interpretations while maintaining competitive accuracy.

Original authors: Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Soyoung Oh, Isabel Valera, Vera Demberg

Published 2026-08-10

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Soyoung Oh, Isabel Valera, Vera Demberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to be helpful, kind, and smart. You can't just program it with a rigid rulebook because human preferences are messy and complicated. Instead, you show the robot thousands of examples of "good" answers versus "bad" answers, letting it learn what humans like. This is the world of Reinforcement Learning from Human Feedback (RLHF), a technique that helps giant AI chatbots align with our values. To do this, the AI uses a "Reward Model," which acts like a strict teacher grading the robot's homework. But here's the tricky part: sometimes this teacher is a "black box." We know it gives a high score to a good answer, but we don't know why it decided that. Is it because the answer was funny? Because it was polite? Or because it followed a specific format?

To solve this, researchers recently tried building a "Mixture-of-Experts" (MoE) reward model. Think of this not as one giant teacher, but as a classroom of 20 different specialists. When a question comes in, a "router" (like a principal) looks at the question and decides which specialist should grade it. Maybe one expert is great at math, another at creative writing, and a third at giving legal advice. The goal was to make these experts easy to understand by looking at which questions they received. But, as the paper we are about to explore suggests, just knowing who got the question doesn't tell you how they graded the answer.

This paper, titled "Beyond Routing Weights," introduces a new way to peek inside the mind of these AI specialists. The authors, a team from Saarland University and other institutions, argue that simply looking at which questions a specialist receives is like judging a chef only by the ingredients they were handed, without tasting the dish. Instead, they propose a method called Contribution-Contrast (CoCo). CoCo looks at the specific moments where a specialist's opinion made the biggest difference between a "good" answer and a "bad" answer. By comparing these pairs, CoCo reveals exactly what the expert cares about—whether it's "being concise" or "using step-by-step logic"—rather than just what topic they usually handle. The researchers tested this on two large datasets and found that CoCo gives a much clearer, more honest, and more detailed picture of what these AI experts are actually doing, without making the AI any less accurate at its job.

The Problem: The "Principal" vs. The "Chef"

In the world of AI reward models, the "Mixture-of-Experts" setup is like a school with a principal and many teachers. The principal (the router) reads a student's question and decides which teacher is best suited to grade it. If the question is about cooking, the principal sends it to the "Culinary Expert." If it's about coding, it goes to the "Programming Expert."

For a while, researchers thought they could understand these experts just by looking at the principal's notes. They would say, "Ah, the Culinary Expert mostly gets questions about baking, so they must be the 'baking expert'." This is called looking at routing weights.

But the authors of this paper realized this was a bit like judging a chef by the ingredients they were given, rather than the meal they cooked. Just because the Culinary Expert received a question about baking doesn't tell you how they judged the answer. Did they prefer detailed recipes? Did they hate long explanations? Did they care about safety warnings? The routing weights only tell you who got the work, not how they did it. It's a partial story that misses the most important part: the actual decision-making process.

The Solution: CoCo (Contribution-Contrast)

To fix this, the team invented CoCo, which stands for Contribution-Contrast. Instead of just asking, "Which expert got this question?", CoCo asks, "Which expert made the biggest difference in deciding that this answer was better than that one?"

Imagine you are a judge in a cooking competition. You have two dishes: one is a perfect lasagna, and the other is a slightly burnt one.

The Old Way (Routing Weights): You look at the scorecard and see that the "Italian Expert" was assigned to judge this round. You conclude, "The Italian Expert likes pasta." But you don't know if they liked the lasagna because it was cheesy, because it was hot, or because it had basil.
The CoCo Way: You look at the scorecard and see that the "Italian Expert" gave the lasagna a huge boost in points compared to the burnt dish, while the other experts were mostly neutral. CoCo then says, "Aha! The Italian Expert specifically loves cheesy, hot, basil-topped pasta."

CoCo works by finding the pairs of answers where an expert's opinion was the deciding factor. It multiplies two things together:

How likely the expert was to get the question (the routing weight).
How much the expert's score changed the final result (the difference between the "good" answer and the "bad" answer).

By focusing on these high-impact moments, CoCo can generate a description of the expert that is faithful to their actual behavior. It doesn't just say "This expert handles cooking questions"; it says "This expert prefers detailed, step-by-step cooking instructions over vague suggestions."

What They Found: A Clearer Picture

The researchers tested CoCo against other methods, including the old "routing weight" method and some other fancy techniques using something called "Sparse Autoencoders" (which are like trying to find hidden patterns in a giant pile of data). They tested these on two huge datasets: one with 700,000 examples and another from Reddit.

The results were quite clear. CoCo produced interpretations that were:

More Faithful: The descriptions matched what the experts actually did in the AI's decision-making process. When the researchers removed the expert from the AI, the AI's behavior changed exactly as CoCo predicted.
More Specialized: The experts described by CoCo had very distinct personalities. One might be the "strict grammar police," while another was the "creative storyteller." The other methods often produced vague or repetitive descriptions.
Just as Accurate: Crucially, using CoCo to understand the experts didn't make the AI worse at its job. The reward model remained just as good at picking the best answers.

In human tests, where people read the descriptions of the experts, they rated CoCo's descriptions as the most coherent and useful. They could actually understand what each "teacher" in the classroom was good at.

Why This Matters

This paper suggests that if we want to truly understand how AI makes decisions, we need to look beyond the surface. Just knowing which topic an AI expert handles isn't enough; we need to know how they evaluate the quality of a response. CoCo offers a way to do this without needing to retrain the AI or add complex new layers. It's a tool for "auditing" the AI, helping us see if our digital teachers are actually teaching what we think they are.

The authors are careful to note that this isn't a magic wand that reveals the "true" hidden thoughts of the AI. The quality of the explanation depends on how well the AI was trained in the first place. If the training data is messy, the experts might be messy too. However, within the limits of the current technology, CoCo provides the most honest and detailed window we have into the minds of these specialized AI reward models. It moves us from guessing what the experts do based on who they talk to, to understanding exactly how they judge the work they do.

Technical Summary: Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast

1. Problem Statement

Reward models are central to Reinforcement Learning from Human Feedback (RLHF) and aligning Large Language Models (LLMs) with human preferences. However, understanding the specific properties driving their predictions remains challenging. While recent sparse Mixture-of-Experts (MoE) reward models attempt to improve interpretability by routing prompts to specialized experts, current interpretation methods are insufficient.

Existing approaches, such as those by Wang et al. (2026), characterize experts using examples with the highest routing weights. This method reveals which prompts an expert receives (prompt-level domains) but fails to explain how the expert judges responses. Consequently, these interpretations provide only a partial account of expert behavior, capturing the context of input assignment rather than the response-level preference dimensions that directly explain the model's decisions. Furthermore, alternative methods like score-based ranking ignore routing weights, potentially highlighting examples where an expert has high score variance but low overall influence on the final reward.

2. Methodology: Contribution-Contrast (CoCo)

The authors propose Contribution-Contrast (CoCo), a response-level interpretation method designed to faithfully characterize the roles of experts in MoE reward models.

Core Concept

CoCo moves beyond routing weights or raw score differences alone. Instead, it uses contribution contrast as the primary interpretation signal. For a given preference pair $(x, y_w, y_l)$ and expert $k$ , the signed contribution contrast is defined as:
$\Delta c_k(x, y_w, y_l) = \pi_{\phi,k}(x) [r_{\theta_k}(x, y_w) - r_{\theta_k}(x, y_l)]$
where $\pi_{\phi,k}(x)$ is the routing weight and $[r_{\theta_k}(x, y_w) - r_{\theta_k}(x, y_l)]$ is the score difference between the chosen and rejected responses. The magnitude $c_k$ measures how strongly the expert distinguishes the responses, while the sign indicates which response the expert supports.

Interpretation Extraction

To interpret an expert, CoCo selects the top 10 validation examples with the largest contribution contrast magnitudes. An LLM is then prompted to summarize the shared contrastive patterns in these pairs, identifying whether the pattern relates to a shared context, a consistent preference dimension, or both.

Regularization

To encourage sparse and diverse contribution patterns, the authors adapt existing interpretability regularizers (local sparsity, global balance, and expert diversity) from Wang et al. (2026). Instead of applying these to routing weights, they apply them to the contribution contrast profiles of the preference pairs. This lightweight adaptation aligns the interpretability constraints with the contrastive contribution signal without altering the underlying MoE architecture.

Baselines

The paper compares CoCo against:

Router-based: Ranks examples by routing weight (captures prompt-level domains).
Score-based: Ranks examples by absolute expert score difference (captures response-level dimensions but ignores routing influence).
SAE-based: Methods like SARM and WIMHF that use Sparse Autoencoders to extract features.

3. Key Contributions

Novel Interpretation Signal: The paper introduces contribution contrast as a signal that jointly captures routing behavior and preference behavior, addressing the gap where routing weights alone fail to explain response-level decisions.
Response-Level Characterization: Unlike router-based methods that identify prompt domains, CoCo identifies response-level preference dimensions (e.g., "concise recommendations rather than lengthy lists") that are faithful to the model's actual decision-making process.
Systematic Study: This work presents the first systematic study of interpretation methods specifically for MoE reward models, comparing router-based, score-based, and SAE-based alternatives.
Regularization Adaptation: The authors demonstrate how to adapt sparsity and diversity regularizers to operate on contribution contrasts rather than routing weights, enhancing the distinctiveness of expert roles.

4. Experimental Results

The authors evaluated CoCo on two datasets: 700K (Dong et al., 2024) and Reddit (Ethayarajh et al., 2022), using a 3B parameter MoE reward model.

Automatic Evaluation

CoCo outperformed existing baselines (SARM, WIMHF, SMoE) and alternative signals (router-based, score-based) across multiple metrics:

Interpretation Quality: CoCo achieved the highest Fidelity (correlation between the signal and LLM judgments of whether examples follow the interpretation). While SMoE achieved lower Redundancy on the 700K dataset, CoCo demonstrated strong overall interpretability, balancing distinctness with high fidelity and faithfulness.
Decision Faithfulness: CoCo showed superior Expert-Model Agreement (proportion of top examples where the expert and full model agree) and Removal Flip Rate (proportion where removing the expert changes the model's prediction).
Expert Specialization: CoCo experts demonstrated higher Expert Accuracy and Relative Expert Advantage on their top-ranked examples compared to other methods.

Human Evaluation

Five annotators evaluated the top 10 examples for five experts per method on the Reddit dataset. CoCo received the highest ratings for both Pattern Coherence and Description Quality, indicating that its interpretations were more coherent and better described the expert's role than router-based or SAE-based methods.

Ablation Studies

Comparisons within the same MoE model trained with different regularization objectives confirmed that the Contribution Contrast signal is the most effective for selecting informative example sets. Even when applied to models trained with router-based or score-based objectives, CoCo interpretations remained more faithful and specialized than interpretations derived from the other signals.

5. Significance and Claims

The paper claims that CoCo provides a more coherent, faithful, and specialized interpretation of MoE reward model experts than existing alternatives. By jointly capturing routing and preference behavior, CoCo offers a more informative basis for understanding model decisions than routing weights or expert scores in isolation.

The authors maintain a modest stance regarding their claims:

They acknowledge that CoCo depends on the structure learned by the underlying MoE model and cannot guarantee that learned experts correspond to "clean" or semantically meaningful preference dimensions.
They note that as a data-centric approach, CoCo may fail to recover patterns absent or weakly represented in the training data.
They clarify that their metrics (fidelity, faithfulness, etc.) serve as evidence of behavioral coherence and decision alignment, not as proof of ground-truth disentanglement.
Ethically, they position CoCo as an auditing tool rather than a validator of desirable human values, noting that interpretations may reflect biases present in the underlying datasets.

In conclusion, the paper establishes contribution contrast as a superior signal for interpreting MoE reward models, offering a pathway to more transparent and actionable insights into how these models align with human preferences.