💻 computer science

Disagreement reliability routing for multimodal large language model evidence pipelines with imperfect counterfactual priors

This paper introduces EviRoute-DR, a disagreement-reliability routing method that effectively fuses corrected rule priors with direct factual outcome experts to improve evidence pipeline calibration and resolution costs in multimodal large language models, even when counterfactual rules are imperfect and only one factual outcome is observed per case.

Original authors: Nataraj Agaram Sundar, Tejas Morabia

Published 2026-10-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nataraj Agaram Sundar, Tejas Morabia

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern digital economy, vast amounts of information flow through automated systems that decide how to handle everything from a misplaced package to a disputed transaction. These systems often rely on artificial intelligence to read documents, check policies, and suggest fixes. However, a persistent challenge in this field is that while machines can generate many possible solutions, the real world only reveals the outcome of the single choice that was actually made. Imagine a doctor who can suggest ten different treatments for a patient but only gets to see the result of the one they prescribed; the other nine remain unknowns. In the world of automated decision-making, this creates a gap between what the system thinks might work based on its rules and what actually happened in reality. When these systems try to learn from their past, they often struggle to distinguish between a rule that was simply wrong and a rule that was right but applied to a case that turned out differently by chance.

Researchers at eBay, Nataraj Agaram Sundar and Tejas Morabia, tackled this specific problem in a new study focused on improving how these AI systems route evidence to the right solution. They developed a method called EviRoute-DR, which acts as a smart filter for decision-making. Instead of blindly trusting the system's initial rules or ignoring them entirely, this new approach learns to spot when a rule is likely to be mistaken. It does this by comparing the rule's prediction against the single, factual outcome that was recorded for a completed case. By analyzing thousands of these real-world examples, the system learns a pattern of "disagreement"—identifying the specific conditions under which the standard rules tend to fail. It then combines this knowledge with direct observations of what worked in the past to create a more reliable guide for future decisions.

The study examined over two thousand real cases involving seven different possible actions, such as retrying a search, asking a human for help, or simply waiting. For each case, the system had a set of pre-written rules suggesting what to do, but only one action was actually taken, and only that one action had a confirmed result. The researchers found that the standard rules were incorrect about the outcome in nearly 18 percent of the cases where they were applied. This meant that blindly following the rules would have led to the wrong decision almost one time in five. The new method, EviRoute-DR, learned to recognize these moments of error. It successfully identified when a rule was likely to be wrong and adjusted its confidence accordingly, effectively teaching itself to trust the rule only when it was reliable and to look elsewhere when it was not.

When tested on a large set of new, unseen cases, this approach proved more accurate than standard methods. The researchers measured the quality of the predictions using a score that reflects how close the system's guesses were to the actual truth. The new method achieved a score of 0.926, which was a clear improvement over the best traditional statistical model, which scored 0.921. While the difference might seem small, in a system handling millions of transactions, even a tiny increase in accuracy can prevent a significant number of errors. The method was particularly effective at handling difficult cases where information was missing or conflicting, showing that it could adapt its strategy based on the reliability of the available data.

Crucially, the researchers were careful to define the limits of what their study proved. They emphasized that their system learns from the single action that was taken, not from the other six possibilities that were considered but rejected. Because the system always chose the action it thought was best, there is no way to know for sure if the other options would have worked better or worse. Therefore, the study does not claim to have discovered the perfect solution for every possible scenario, nor does it claim to have solved the problem of predicting outcomes for actions that were never tried. Instead, it offers a practical way to make the best possible use of the limited factual feedback that is available in real-world operations. The results suggest that by acknowledging when rules might fail and adjusting accordingly, automated systems can become significantly more trustworthy without needing to rewrite their entire logic from scratch.

The study also explored how this method performs when costs are involved, such as the time it takes to resolve an issue or the effort required from a human reviewer. By balancing the likelihood of a successful resolution against the cost of the action, the system identified a range of operating points where it could achieve high success rates without incurring excessive costs. In a simulated replay of past decisions, the new method reached a resolution rate of about 81.5 percent at a moderate cost level, outperforming the standard rule-based approach. This indicates that the method can help organizations make decisions that are not only more accurate but also more efficient.

Ultimately, this work highlights a shift in how we think about artificial intelligence in complex environments. Rather than expecting machines to be perfect or to learn from every hypothetical outcome, the focus is on building systems that understand their own uncertainty. By treating rules as helpful but fallible guides rather than absolute truths, and by learning from the few concrete facts that are available, these systems can navigate the messy reality of production environments more effectively. The findings offer a blueprint for improving the reliability of automated decision-making in fields where data is incomplete and the cost of error is high, proving that a little bit of skepticism about one's own rules can lead to much better results.

Technical Summary: Disagreement Reliability Routing for Multimodal LLM Evidence Pipelines

Problem Statement
Multimodal Large Language Model (LLM) evidence pipelines often operate under an asymmetric feedback regime. While a system may generate dense, rule-derived counterfactual outcomes for multiple candidate remediation actions, production logs typically record only the factual outcome of the single executed action. Treating rule-derived outcomes as ground truth risks miscalibrating routing decisions, while discarding them entirely wastes structured operational knowledge. Furthermore, existing routing literature often assumes full-information supervision or bandit feedback with known behavior policies, conditions not met when production logs record only the argmax of a deterministic score vector without stochastic selection. This creates a challenge: how to leverage imperfect rule priors for training without conflating procedural knowledge with observed truth, particularly when missing data channels and conflicting signals affect rule reliability.

Methodology: EviRoute-DR
The authors propose EviRoute-DR (Disagreement-Reliability), a routing framework designed to learn when rule priors fail and to fuse corrected-rule experts with direct factual experts. The methodology consists of three core components:

Disagreement Correction ( $q_\theta$ ):
Instead of treating rule labels as truth, the system learns a "disagreement expert" that estimates the probability that a rule prior is incorrect given the context ( $x$ ) and action ( $a$ ).
- Target: $D_i = 1\{Y_i(A_i) \neq r_i(A_i)\}$ , where $Y$ is the factual outcome and $r$ is the rule prior.
- Correction: The rule-corrected success probability is calculated as $\hat{p}^R_\theta(x, a) = r(x, a)[1 - q_\theta(x, a)] + [1 - r(x, a)]q_\theta(x, a)$ . This effectively reverses the rule label if the disagreement probability is high and trusts it if low.
Reliability-Conditioned Stacking:
A meta-learner combines predictions from two branches:
- Direct Factual Experts: Models (Logistic Regression, Random Forest) trained strictly on the single factual outcome per case.
- Rule-Correction Experts: Models trained to predict the disagreement probability, which then generate corrected success probabilities.
- Reliability Features: The meta-learner ingests additional signals including missingness counts, conflict scores, logged production action scores, and action costs.
- The final score is a regularized logistic combination: $\hat{p}^{ER}(x, a) = \sigma(\beta_0 + \beta^\top z(x, a))$ .
Cost-Averse Routing:
The system selects the action $\pi_\lambda(x) = \arg\max_{a \in A} [\hat{p}^{ER}(x, a) - \lambda c(x, a)/20]$ , where $\lambda$ is a cost-aversion parameter and $c(x, a)$ is a composite utility cost (failure loss, review effort, latency, etc.).

Experimental Setup

Data: 2,100 production cases from eBay, each with 7 candidate actions (14,700 case-action records).
Feedback: Exactly one factual outcome per case; the remaining 6 actions have rule-derived counterfactual annotations used only for evaluation or correction, not as factual training labels.
Splits: 600 factual training cases, 200 validation cases (for meta-learner fitting), and 1,300 held-out test cases (including main, temporal, hard, and workflow-shift cohorts).
Baselines: Rule prior, direct Logistic Regression, direct Random Forest, and various rule-correction variants.
Constraints: The authors explicitly avoid Inverse Propensity Scoring (IPS) or Doubly Robust (DR) causal claims because the logged action is always the argmax of the score vector, implying no stochastic selection mechanism to identify behavior propensities.

Key Results

Factual Prediction: On the 1,300 held-out factual cases, EviRoute-DR achieved an AUC of 0.926 and a Brier score of 0.0910.
- This outperformed direct Logistic Regression (AUC 0.921, Brier 0.0971) with a statistically significant Brier improvement of 0.0060 (95% CI 0.0021–0.0092).
- It showed modest point-estimate improvements over Random Forest (AUC 0.924, Brier 0.0922), but the paired bootstrap intervals included zero, meaning superiority over Random Forest was not statistically established.
Ablation: Removing the "rule branch" (corrected experts) degraded performance, confirming that disagreement correction contributes significantly to probabilistic accuracy.
Counterfactual Replay: A secondary evaluation using the full 7-action ledger showed EviRoute-DR achieving a resolution rate of ~0.815 at a mean cost of 12 units, compared to ~0.786 for the raw rule prior. This suggests a better cost-resolution frontier at moderate operating costs.
Sensitivity: The replay gains were found to be sensitive to concentrated annotation errors in the unexecuted actions, highlighting that the replay results are descriptive and dependent on the quality of the supplied annotations.

Significance and Claims
The paper positions EviRoute-DR as a practical methodology for evidence routing under partial factual feedback. Its primary contributions are:

Formalizing the Asymmetry: Defining a routing problem where one factual outcome coexists with imperfect rule-derived priors.
Disagreement-Aware Fusion: Demonstrating that learning the error structure of rule priors (disagreement) allows for a "corrected" probability that improves probabilistic accuracy over direct learning.
Strict Causal Boundaries: The authors emphasize that their results are not causal policy evaluation (OPE). Because the logs do not support stochastic selection assumptions, they refrain from claiming causal intervention effects. Instead, they frame the work as improving factual prediction and providing a secondary, annotation-dependent replay analysis.

The authors conclude that disagreement-aware reliability fusion is a viable approach for multimodal systems where missing channels and signal conflicts make rule priors fallible, offering a way to utilize operational structure without treating it as ground truth.

Technical Summary: Disagreement Reliability Routing for Multimodal LLM Evidence Pipelines

More like this