A Practical Framework for Flaky Failure Triage in Distributed Database Continuous Integration

1. Problem Statement

In distributed database Continuous Integration (CI), failures are often caused by transient runtime states (e.g., lock contention, replication lag) rather than persistent code defects. When a failure occurs, operators must make an immediate decision: auto-rerun the job (if likely "flaky") or escalate it for debugging (if likely "persistent").

This decision must be made under strict constraints:

• Latency: CPU-only, millisecond-scale budgets.
• Causality: Decisions must be made before post-failure artifacts (logs, error strings) are available to avoid data leakage.
• Label Bias: Ground truth labels are generated by finite rerun policies. If a job fails the first $R$ reruns, it is labeled "persistent," even if it might have passed on the $(R+1)$-th attempt. This creates a selection bias where flaky failures are systematically under-observed.

Existing approaches often rely on post-failure logs (violating causality), produce poorly calibrated scores under distribution shifts, or fail to account for the bias introduced by limited rerun budgets.

2. Methodology: The SCOUT Framework

The authors propose SCOUT (State-aware Causal Online Uncertainty-calibrated Triage), a framework designed specifically for this deployment setting. It consists of an Online Triage Path and an Offline Adaptation Path.

A. Strict-Causal Feature Extraction

To ensure no data leakage, SCOUT constructs features using only information available at the moment of failure ($t=0$):

• Pre-failure Telemetry: Aggregated statistics (mean, max, std, p95) from a 120-second window prior to failure across 14 telemetry families (e.g., lock/transaction, Raft, storage, queue signals).
• Pre-run Metadata: Governed tokens describing the workload, version, and test identity (excluding raw identifiers that could leak future info).
• Historical Data: Statistics computed strictly from earlier primary runs of the same test.
• Exclusion: Post-failure logs, error strings, and future rerun data are strictly blocked.

B. Lightweight State-Aware Scoring

SCOUT employs lightweight logistic regression models to predict the probability of flakiness ($\hat{p}_i$) to meet millisecond latency requirements. It supports three variants:

1. State LR: Uses dense pre-failure telemetry.
2. Text LR: Uses sparse metadata tokens (TF-IDF).
3. SDJ-LR: A fusion of dense telemetry and sparse metadata.
   The system avoids heavy sequence models or multimodal LLMs, prioritizing speed and interpretability.

C. Decision-Portable Calibration (Offline)

Since raw model scores are often miscalibrated under temporal or cross-domain shifts (e.g., new database versions), SCOUT applies post-hoc calibration to ensure a single fixed threshold ($\tau^*$) remains effective across shifts.

• OA-Cal (Overlap-Aware Calibration): A novel calibration method that selects the best calibrator (e.g., Isotonic, Sigmoid, Beta) based on the source domain data and unlabeled target scores.
• It uses a domain classifier to estimate importance weights. If source-target overlap is poor, it disables weighting to prevent degradation.
• It optimizes for fixed-threshold decision cost rather than just probabilistic accuracy (PR-AUC), ensuring the threshold derived from cost models ($c_{auto}, c_{fp}, c_{fn}$) remains valid.

D. Rerun-Budget Correction (Offline)

To address the label bias caused by finite rerun budgets ($R$), SCOUT replaces hard binary labels with Posterior-Soft Targets.

• If a job passes within $R$ attempts, the label is 1.
• If it fails all $R$ attempts, instead of labeling it 0 (persistent), SCOUT calculates the posterior probability that it would have passed if given a larger oracle budget ($R'$).
• This uses a Beta-Binomial model to estimate the probability of a future pass, effectively "uncensoring" the label for training purposes without requiring actual infinite reruns.

3. Key Contributions

1. Formulation: Defined flaky-failure triage as a strict-causal online decision problem under millisecond CPU constraints.
2. Framework (SCOUT): Developed a practical framework combining strict-causal feature extraction, lightweight scoring, and decision-portable calibration.
3. Techniques:
   • OA-Cal: A calibration method that adapts to distribution shifts without target labels, optimizing for fixed-threshold decision costs.
   • Posterior-Soft Correction: A method to mitigate label bias induced by finite rerun policies, improving learning toward a larger-budget oracle.
4. Deployment: Demonstrated that lightweight models with rigorous calibration outperform heavier models in production settings.

4. Experimental Results

The authors evaluated SCOUT on a synthetic benchmark (3,680 runs), a real TiDB v7/v8 trace (1,600 runs), and a large GitHub Actions metadata trace.

• Feature Effectiveness (RQ1): Dense pre-failure telemetry is the primary source of signal. Simple window-aggregated statistics outperform complex sequence models (e.g., GRU/CNN) and history-only baselines.
• Calibration Importance (RQ2): Calibration is critical. Under temporal shifts, uncalibrated models incur high decision costs. Isotonic calibration reduced decision costs from 765.4 to 496.9. OA-Cal successfully transferred thresholds across versions and workloads without re-tuning.
• Label Correction (RQ3): Posterior-soft correction significantly improved calibration. It reduced the Expected Calibration Error (ECE) from 0.320 to 0.027 and Brier score from 0.213 to 0.096 compared to naive training on censored labels. It remained robust under correlated reruns and selective labeling.
• Deployment Practicality (RQ4):
  • Latency: Achieved an end-to-end P95 latency of 1.17 ms on CPU (Apple M4).
  • Robustness: Performance did not degrade when identifiers were removed or when strict causal windows were enforced.
  • Interpretability: SHAP analysis confirmed that features like lock-wait ratio and Raft latency drove predictions logically.
• Generalization (RQ5): SCOUT variants outperformed symptom-based baselines (TF-IDF, DistilBERT) on real TiDB traces and GitHub Actions metadata, proving the approach generalizes beyond synthetic data.

5. Significance

This paper addresses a critical gap in AIOps and CI/CD: the need for immediate, low-latency, and causally valid decisions in distributed systems.

• Practicality: It moves beyond theoretical accuracy (AUC) to optimize for operational decision costs (rerun vs. escalate).
• Robustness: By solving the label bias problem (finite reruns) and the distribution shift problem (calibration), SCOUT provides a deployable solution that works in production environments where data is noisy and policies change.
• Efficiency: It proves that complex deep learning models are unnecessary for this specific task; lightweight, well-calibrated logistic models with strict causal features are superior for real-time triage.

The framework has been deployed in the production environment of PingCAP (TiDB), demonstrating its viability for maintaining efficient and reliable continuous integration pipelines.