💻 computer science

Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks

This paper audits offline root-cause analysis benchmarks to demonstrate that relying on pooled top-1 accuracy rankings is misleading because it obscures significant system-specific performance variations, often leading engineers to select suboptimal methods for their specific subsystems.

Original authors: Lining Hu, Ting Liu, Yuzhuo Fu

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lining Hu, Ting Liu, Yuzhuo Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a car mechanic trying to figure out which brand of spark plug is the "best" for your specific car.

The Current Situation: The "Average" Leaderboard
Right now, researchers who test root-cause analysis (RCA) tools—software that helps engineers figure out why a computer system crashed—act like a car magazine that tests 11 different car models (a tiny sedan, a heavy truck, a race car, etc.). They run all the spark plugs on all 11 cars, mix all the results together, and calculate one single "average score."

If Spark Plug A has an average score of 85% and Spark Plug B has 80%, the magazine declares Spark Plug A the winner.

Engineers reading this magazine then assume, "Okay, Spark Plug A is the best, so I'll buy it for my specific truck."

The Problem: The "One-Size-Fits-All" Trap
This paper argues that this "average score" is a dangerous lie. It's like saying a snowboard is the "best" winter sports equipment because it averaged out perfectly across skiing, surfing, and skateboarding.

The authors audited three major "magazines" (benchmarks) that cover 11 different computer systems (subsystems). They found that:

The winner changes depending on the car. In some systems (like a "Bank" system), Spark Plug A was great. In others (like a "Sock-Shop" system), Spark Plug A was terrible, and Spark Plug B was the clear winner.
The "Average" hides the chaos. When you mix the results, the bad performance on some systems cancels out the good performance on others, creating a single number that looks stable but is actually misleading.
Following the average leads to mistakes. If you pick the "average winner" for your specific system, you might end up picking the tool that performs the worst for your specific job. In one case, following the average recommendation led to a 24.8% drop in performance compared to picking the right tool for that specific job.

The Analogy: The "Universal Doctor"
Think of these RCA tools as doctors.

The Pooled Leaderboard says: "Dr. Smith is the best doctor overall because they cured 60% of patients across all 11 different diseases."
The Reality: Dr. Smith is amazing at treating the flu but terrible at treating broken bones. Dr. Jones is the opposite.
The Audit: The authors looked at the data and realized that if you have a broken bone (a specific subsystem), the "overall winner" (Dr. Smith) is actually the wrong choice. The "overall winner" only wins because they are good at the diseases that are easy to treat or very common in the test group.

What Did They Do?
The authors didn't invent a new spark plug or a new doctor. Instead, they built a reporting audit tool (a 320-line software module).

They took the existing test results and broke them down. Instead of just showing one big number, they showed:

How each tool performed on each specific system.
How much the performance varied (heterogeneity).
A "regret score": If you picked the "average winner" for a specific system, how much worse did you do?

The Results
They tested four different tools (including a popular one called BARO and some simple, basic rules). They found that no single tool was the winner for all 11 systems.

Sometimes Tool A beat Tool B.
Sometimes Tool B beat Tool A.
Sometimes the difference was huge (like 30% better or worse).
The "average" ranking was completely different from the "specific system" ranking.

The Conclusion
The paper concludes that we need to stop treating these "average" leaderboards as a recommendation for specific jobs.

For the Magazine Writers (Benchmark Authors): Don't just give one number. Show the breakdown for each system and admit when a tool is only good for specific types of problems.
For the Mechanics (Engineers): Don't just pick the tool with the highest average score. Look at the breakdown to see if that tool actually works for your specific system.

What They Did NOT Say

They did not say one tool is "bad" and another is "good." Both tools have their place; it just depends on the system.
They did not propose a new AI algorithm to fix this. They proposed a new way of reporting the results so people don't get fooled by averages.
They did not test this on live, real-time systems where the software is constantly talking to the internet (closed-loop agents); they only looked at offline test data where the results are already recorded.

In short: Averages are great for summarizing a group, but terrible for making decisions about individuals. If you want to fix a specific computer system, you need to know which tool works for that system, not which tool wins the "average" contest.

Technical Summary: Pooled Leaderboards Hide System-Specific Winners

Problem Statement
Offline root-cause-analysis (RCA) benchmarks currently rank methods using a single "pooled" top-1 accuracy score derived by averaging performance across multiple microservice subsystems. Engineers frequently interpret the "pooled winner" as a recommendation for their specific subsystem. This paper identifies a fundamental estimand mismatch: the pooled score answers how a method performs on average across a suite, whereas engineers need to know if a method works for their specific service graph, telemetry schema, and fault distribution. The authors argue that when method performance varies significantly across subsystems (heterogeneity), the pooled score becomes a misleading metric for subsystem-level selection, potentially hiding cases where a lower-ranked method is actually superior for a specific system.

Methodology and Scope
The authors conduct a reporting-protocol audit on three public offline RCA benchmark families: OpenRCA, RCAEval, and PetShop. The audit covers 11 subsystems and 778 matched scoring units.

To ensure valid pairwise comparisons, the study applies a strict full-coverage inclusion criterion: a method or comparator is included only if it produces a score for every single one of the 778 cases across all 11 subsystems. This prevents conclusions from being driven by missing data patterns. The four methods/comparators meeting this criterion are:

BARO: A published statistical RCA method based on multivariate Bayesian online change-point detection.
CD-1min: A deterministic adapter built on the OpenRCA pipeline, resampling telemetry to 1-minute resolution and ranking services by percentage change.
max-|Z|: A schema-agnostic heuristic selecting the service associated with the metric having the maximum absolute Z-score in the post-injection window.
Per-service alert-count: A heuristic counting metrics with $|Z| > 3$ per service and selecting the service with the highest count.

The analysis employs standard random-effects meta-analysis techniques (Cochran's Q, DerSimonian-Laird, Paule-Mandel $\tau^2$ ) and small-sample corrections (Hartung-Knapp-Sidik-Jonkman). Key diagnostic metrics include:

Paired Subsystem Effects ( $\Delta_s$ ): The difference in accuracy between two methods for a specific subsystem.
Prediction Intervals (PI): 95% intervals estimating the range of effects on a new, unseen subsystem.
Leave-One-System-Out (LOSO) Regret: The performance cost incurred when the pooled winner (selected from 10 systems) is applied to the held-out 11th system.
Interaction Tests: Likelihood-ratio tests to determine if method performance is exchangeable across systems.

Key Results
The audit reveals that no full-coverage method pair maintains a stable sign of superiority across all subsystems.

Sign Reversals: All six pairwise comparisons among the four methods exhibit both positive and negative effects across different subsystems. For example, BARO is the pooled loser against max-|Z| overall but outperforms it on the Bank, High-Traffic, and Online-Boutique subsystems. Conversely, BARO is the pooled winner against alert-count but underperforms on Market-1, Telecom, Sock-Shop, and Train-Ticket.
Heterogeneity: Random-effects heterogeneity ( $I^2$ ) is substantial across all pairs (ranging from 67.0% to 92.7%). Crucially, every 95% prediction interval crosses zero, indicating that the direction of superiority is unpredictable for a new subsystem.
Selection Regret: When applying the pooled recommendation to a held-out subsystem, the "wrong" method is selected in 3 to 5 out of 11 cases per comparison. The maximum regret reaches 24.8 percentage points (on RCAEval / Sock-Shop when selecting BARO over alert-count).
Non-Exchangeability: Case-level interaction tests reject the hypothesis that method effects are exchangeable across systems in 5 of the 6 pairs (with the 6th pair showing marginal significance but still exhibiting reversals).
Robustness: These findings persist even when removing specific benchmark families, subsystems, or altering pooling rules. The heterogeneity is not solely an artifact of cross-benchmark protocol differences; sign reversals also occur within the OpenRCA suite alone.

Contributions and Claims
The paper does not propose a new RCA algorithm or claim that existing methods are flawed. Instead, it contributes a reporting audit protocol and a 320-line implementation module that recomputes per-subsystem stability checks alongside pooled scores.

The authors claim that:

Pooled leaderboards are insufficient for subsystem-level selection. Without stability diagnostics, a pooled winner cannot be reliably recommended for a specific engineering context.
Current reporting practices hide critical information. The aggregation of scores into a single number obscures the fact that method rankings are highly dependent on the specific subsystem characteristics.
The proposed audit is a necessary diagnostic. The authors provide a tool to compute paired effects, prediction intervals, and selection regret, arguing that benchmark releases should include these metrics to qualify any "state-of-the-art" claims.

Significance and Limitations
The significance of the work lies in its application of meta-analytic rigor to the evaluation of offline RCA benchmarks, highlighting a structural flaw in how results are currently consumed by the community. The authors explicitly limit their scope to offline benchmarks with matched per-case scoring (OpenRCA, RCAEval, PetShop). They do not evaluate closed-loop agentic benchmarks or ad-hoc single-system testbeds, as these lack the standardized, matched scoring required for their paired-effect estimators. The paper concludes that until benchmark releases include per-subsystem reporting with heterogeneity statistics, pooled offline-RCA leaderboards alone do not justify subsystem-level method selection.

Technical Summary: Pooled Leaderboards Hide System-Specific Winners

More like this