The Abstention Protocol: RCA for Clos Fabrics
This paper introduces CoreSec, a production root cause analysis system for large-scale Clos fabrics that replaces unstable score-based fusion with a deterministic PAM-style abstention algebra to achieve stable, explainable, and monotonic failure attribution in noisy telemetry environments.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast, humming architecture of modern cloud computing, data does not flow through a single pipe but through a sprawling, multi-layered web of connections. Imagine a city where every building is connected to a local neighborhood switch, which in turn connects to a district hub, which finally links to a central spine. This structure, known as a Clos fabric, allows millions of servers to talk to each other with incredible speed and redundancy. If one path is blocked, traffic simply finds another. Because of this design, the system is incredibly resilient; it can absorb thousands of tiny, random glitches every day—like a loose cable, a flickering light, or a brief software hiccup—without the average user ever noticing. However, this constant background noise creates a profound problem for the engineers who keep the system running. When a specific service fails for a customer, the system lights up with hundreds of warnings. The challenge is not finding the broken parts, but figuring out which one of those many broken parts actually caused the specific outage.
For years, the standard way to solve this puzzle was to assign a score to every warning signal. If a cable had a high error count, it got a high score. If a switch rebooted, it got a score. The system would add these scores up, and the entity with the highest total was blamed for the trouble. This approach worked well enough when the network was quiet, but in a hyperscale environment, it often failed. Because there is always some background noise, the system frequently found a "culprit" even when nothing was truly wrong, or it blamed the wrong device because the scores were too close to call. The engineers found that trying to fine-tune these scores to fix one type of error would accidentally break the system's ability to catch another. The result was a cycle of uncertainty, where automated fixes were sometimes triggered by false alarms, making the situation worse rather than better.
To solve this, a team at Microsoft developed a new system called CoreSec, which changes the fundamental logic of how these decisions are made. Instead of adding up scores, the new system treats the investigation like a series of strict, independent checks, similar to how a security system might verify a person's identity. In a high-security building, a guard might require a password, a fingerprint, and a key card. If the password is missing, the guard does not guess; they simply deny entry and stop the process. CoreSec applies this same "abstention" logic to network failures. It assigns specific roles to different types of data. Some signals are mandatory: if a critical piece of evidence is missing or stale, the system refuses to make a decision. Other signals are sufficient on their own: if a specific, undeniable error is found, the system stops looking and names the cause immediately.
The system runs five different investigations in parallel, each looking at a different layer of the network, from the individual cables connecting servers to the massive spine switches that hold the whole fabric together. Each investigation uses its own set of rules to decide if it has enough proof to vote for a specific cause. If the evidence is clear, it votes. If the evidence is missing or contradictory, it abstains. This is a crucial shift. In the old system, the computer was forced to pick a winner even when it didn't know the answer. In the new system, admitting ignorance is a valid and useful outcome. When the system abstains, it tells the human engineers, "I cannot be sure yet," and hands the case over to them with a clear summary of what data was missing. This prevents the system from making confident but wrong guesses that could trigger unnecessary and potentially harmful automated repairs.
Once the five parallel investigations have finished, a second layer of logic steps in to combine their results. This logic understands the physical shape of the network. It knows that if a single switch fails, it might affect a few servers, but if a higher-level hub fails, it will cause a ripple effect that makes many switches look sick at the same time. The system uses simple, hard-wired rules to decide which layer is truly responsible. For example, if a high-level switch is suspected, the system checks if at least two-thirds of the smaller switches connected to it are also showing problems. If they are, the system concludes the high-level switch is the root cause and ignores the individual switches below it. This prevents the system from getting distracted by the symptoms and blaming the wrong level of the network.
The results of deploying this system across more than sixty regions of the Azure cloud have been striking. Over the course of three years, the system processed more than 700,000 incidents. The rate of false alarms, where the system blamed a healthy device, dropped from nearly twenty percent to less than one percent. At the same time, the number of times the system correctly identified the problem without needing human help increased significantly. Perhaps most importantly, the system eliminated the need for three full-time engineers to manually review and reconcile conflicting data for every incident. The engineers who used to spend hours untangling these knots now receive a clear, structured report that tells them exactly what the system found, what it could not decide, and where they should look next.
The success of CoreSec lies in its refusal to guess. By treating the fusion of different data sources as a composition problem rather than a scoring game, the system achieves a level of stability that was previously impossible. It does not rely on complex machine learning models that might change their behavior as the network evolves. Instead, it uses a fixed set of logical rules that have proven to work across different hardware, different traffic patterns, and different datacenter designs without needing to be retuned. The system has shown that in a world of noisy, incomplete information, the most powerful tool is often the ability to say, "I do not know," and wait for better evidence. This approach has transformed root cause analysis from a game of probability into a reliable, explainable process, allowing the cloud to remain stable even as it continues to grow larger and more complex.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.