Technical Summary: ConformalFL

Problem Statement

Spectrum-Based Fault Localization (SBFL) generates a ranking of executable program lines based on test coverage but fails to provide a developer with a concrete stopping criterion. Developers must decide how many lines to inspect before abandoning the ranking, a decision complicated by vast differences in program size, test support, score concentration, and exact score ties. Existing approaches often rely on fixed Top-N policies or uncalibrated heuristics that ignore these operational realities and do not offer a formal guarantee regarding the risk of missing a fault.

The paper addresses the need for a method that translates a user-selected "marginal miss risk" (e.g., "I am willing to miss the fault 10% of the time") into a specific, bug-adaptive inspection set. The goal is to provide an auditable, tie-complete set of candidates that contains at least one mapped faulty line with a guaranteed probability, without assuming the method improves the underlying SBFL ranking quality.

Methodology

The proposed method, ConformalFL, applies Conformalized Quantile Regression (CQR) to predict the inspection cutoff for a specific bug based on its pre-fault spectrum features.

1. Target Formulation

The method defines the prediction target as the normalized entry position of the earliest faulty score group ($y_{entry}$).

• Operational Logic: Candidates are ordered by decreasing SBFL score. Exact ties form score groups. The method predicts a fraction $b_i$ and calculates a cutoff index $k_i$. The resulting candidate set includes all lines with scores greater than or equal to the score at position $k_i$.
• Tie Handling: To ensure the set is "tie-complete," if the cutoff lands within a tie group, the entire group is included. The target is the entry point of the faulty tie group, not the end, to avoid unnecessary double-counting of the tie expansion.

2. Prediction Pipeline

• Feature Vector: The model uses 20 features available before the fault is revealed, including code size, test/coverage metrics, and 12 distribution features of the Ochiai scores (moments, entropy, gaps, tie counts, etc.). Project identity and fault location are excluded from the primary model.
• Quantile Regression: A gradient-boosted regressor estimates the conditional upper quantile ($bq_{1-\alpha}$) of the target $y_{entry}$ given the features $x_i$.
• Conformal Correction: To achieve finite-sample marginal coverage guarantees, the method uses a held-out calibration set. It calculates one-sided residuals ($r_i = y_{entry, i} - bq_{1-\alpha}(x_i)$) and applies a correction based on the $(1-\alpha)$-quantile of these residuals.
• Final Cutoff: The final budget is the sum of the predicted quantile and the conformal correction, clipped to $[0, 1]$. If the required calibration sample size is insufficient (e.g., for very high coverage targets with small datasets), the method defaults to a "no-compression" outcome (full inspection), explicitly signaling that the requested risk level is unsupported.

3. Experimental Design

• Dataset: A pinned universe of 395 historical Defects4J bugs (Java), reduced to 248 "candidate-present" bugs (those with at least one mapped faulty executable line and both passing/failing tests).
• Protocol: 30 project-stratified splits (60% training, 20% calibration, 20% test).
• Comparators:
  • GBR: The uncalibrated gradient-boosted quantile regressor.
  • Global Conformal: A feature-independent constant cutoff derived from calibration targets.
  • Fixed Policies: Pre-registered Top-N and fixed-percentage policies (e.g., Top-10%).
• Stress Tests: Held-out project (Leave-One-Project-Out), chronological (temporal shift), and cross-language (transfer to Python/BugsInPy) evaluations.

Key Results

Evaluated at a nominal 90% coverage level:

1. Coverage vs. Workload:

   • ConformalFL (CQR): Achieved 91.8% mean coverage while inspecting 30.2% of executable candidates (median 508.5 lines).
   • Uncalibrated (GBR): Achieved 87.6% coverage at 15.6% inspection.
   • Global Conformal: Achieved 91.8% coverage but required 44.2% inspection.
   • Fixed Top-10%: Achieved 90.1% coverage at 28.3% inspection.

2. Calibration Value:

   • Calibration added approximately 4.2 percentage points of coverage over the uncalibrated GBR but at a cost of +14.7 percentage points in inspection effort.
   • Compared to the constant Global Conformal cutoff, CQR maintained the same mean coverage while reducing inspection by 14.0 percentage points, demonstrating the value of bug-adaptive cutoffs over static ones.

3. Workload Distribution:

   • The workload distribution is highly skewed. While the median inspection was ~508 lines, the mean was ~1,521 lines due to a heavy tail.
   • 18.9% of cases resulted in "no compression" (full inspection) due to score ties expanding the candidate set to the entire ranking. This occurred specifically when the cutoff fell on a zero-score boundary.

4. Stress Tests:

   • Project Shift: Coverage varied by project (85.7%–100%), and small calibration samples (e.g., 5 bugs) resulted in vacuous (100% inspection) outcomes.
   • Cross-Language: When transferred to Python (BugsInPy) without retraining, CQR maintained high empirical coverage (98.3%) but suffered severe workload degradation, inspecting 66.9% of statements on average, with 53.3% of cases requiring full inspection.

Significance and Claims

The paper makes modest, specific claims regarding the contribution of ConformalFL:

• Auditable Risk Control: The primary contribution is an auditable mapping from a requested miss risk to a bug-adaptive, tie-complete inspection set. It provides a formal guarantee of marginal coverage under the assumption of exchangeability between calibration and deployment bugs.
• No Universal Dominance: The paper explicitly states that ConformalFL does not empirically dominate well-chosen fixed policies (like Top-10%) on this benchmark. The fixed Top-10% policy performed comparably in terms of coverage and effort without requiring calibration.
• Limitations:
  • The method does not improve the underlying SBFL ranking quality.
  • It does not guarantee conditional coverage for specific projects or subpopulations.
  • It does not guarantee validity under distribution shifts (e.g., cross-language or new projects) where exchangeability fails.
  • It does not measure human debugging time, as line count does not equate to context-switching costs or comprehension effort.

Conclusion: ConformalFL offers a practical tool for teams with historical fault data to replace uncalibrated heuristics with a transparent rule that explicitly trades off miss risk against inspection effort. However, its utility is bounded by the requirement for representative calibration data and the potential for "no-compression" outcomes when score ties are prevalent or calibration samples are small.