Technical Summary: From Program Slices to Causal Clarity

Problem Statement

Large Language Model (LLM)-based debugging systems are increasingly capable of generating failure explanations. However, these explanations are often incomplete, vague, or incorrect, posing significant risks to downstream tasks such as bug triage and automated fixing. Current research predominantly treats failure explanations as an ad hoc by-product of debugging workflows, relying on generic prompts over undifferentiated artifacts (e.g., entire code files, full error logs) rather than treating explanations as a first-class output with dedicated quality assessment.

Consequently, existing approaches lack support for assessing whether explanations capture the underlying fault–error–failure mechanism or provide actionable next steps. Furthermore, the prevailing "more context is better" assumption in LLM-based software engineering often leads to semantic dilution, where excessive or irrelevant context (e.g., long prose, broad code dumps) obscures critical failure-relevant evidence, resulting in vague explanations. There is a critical gap in understanding how specific context compositions affect the causal clarity and actionability of LLM-generated explanations.

Methodology

The authors propose a systematic framework to evaluate how manipulating debugging context influences explanation quality and downstream repair success.

1. Context Partitioning and Modules

The study treats debugging evidence as composable context modules. For each defect, the system defines a fixed set of eight modules:

• Direct Artifacts: CODE (source), ERROR (error message), TEST (failing test), DOCSTRING (documentation).
• Slicing-Derived Modules: SLICE_BLOCK (enclosing AST block), SLICE_BACKWARD (dynamic backward slice), SLICE_FORWARD (forward slice), and SLICE_UNION (union of backward/forward).

The researchers generated 93 unique context configurations per defect, ranging from isolated modules to pairwise and three-way combinations, including a BASELINE containing all modules.

2. Experimental Setup

• Dataset: 12 localized, reproducible software defects originally from Defects4J (Java), manually ported to Python to mitigate pre-training contamination risks.
• Models: Three economically viable models were evaluated: gpt-5-mini, DeepSeek-V3.2, and Grok-4.1-fast.
• Procedure: For each defect and configuration, the LLM generated a failure explanation. The study also generated repairs in two conditions: (i) without an explanation (NO_EXPLANATION) and (ii) with the generated explanation appended (WITH_EXPLANATION).

3. Evaluation Framework

Explanation quality was operationalized via six binary criteria (C1–C6):

• C1 (Readability): Measured via Flesch–Kincaid grade level (deterministic).
• C2 (Problem Identification): Correct identification of the root cause (LLM-as-a-judge).
• C3 (Causal Chain): Coherent explanation of the fault-error-failure mechanism (LLM-as-a-judge).
• C4 (Actionability): Concrete, developer-usable next steps (LLM-as-a-judge).
• C5 (Grounding): Explicit references to provided artifacts (deterministic).
• C6 (Brevity): Conciseness (LLM-as-a-judge).

LLM-as-a-Judge Validation: To address the subjectivity of criteria C2–C4 and C6, the authors conducted a user study with 12 software engineering experts. They validated that LLM-as-a-judge scores align well with human ratings for causal and actionable criteria (C2–C4) but show lower agreement for brevity (C6).

4. Downstream Utility

The study quantified the utility of explanations by comparing repair pass rates and the similarity of generated fixes to reference minimal fixes (using AST-normalized diff metrics) between the NO_EXPLANATION and WITH_EXPLANATION conditions.

Key Contributions

1. Context-Module Catalog and Pipeline: A systematic pipeline that treats debugging context as composable modules, enabling the generation and evaluation of explanations and fixes under varied context compositions.
2. Failure-Explanation-Specific Evaluation Framework: A multi-dimensional quality framework (C1–C6) that moves beyond task success (patch correctness) to assess causal clarity and actionability, validated against human judgment.
3. Empirical Study of Context Partitioning: A large-scale study (93 configurations × 3 models × 12 defects) quantifying how context composition shapes explanation quality and its causal effect on downstream repair precision.

Results

1. LLM-as-a-Judge Validity (RQ1)

LLM-as-a-judge serves as a reliable proxy for human evaluation on causal (C2, C3) and action-oriented (C4) criteria, achieving high agreement (0.74–0.88). However, agreement drops for brevity (C6) (approx. 0.58), which humans rated as the most difficult criterion to judge. LLMs also tend to be more "pass-prone" than humans, particularly on brevity.

2. Criterion Discriminativeness (RQ2)

Evaluation criteria are not equally discriminative. Causal and action-oriented criteria (C2–C4) are strong predictors of downstream repair success. Explanations in the highest quality quartile (Q4) consistently yielded higher repair pass rates than those in the lowest quartile (Q1). Notably, low-quality explanations (Q1) could underperform the NO_EXPLANATION baseline in some configurations (e.g., DeepSeek-V3.2 in three-way contexts: Q1 pass rate 0.341 vs. baseline 0.500).

3. Sensitivity to Context Prompting (RQ3)

Explanation quality is highly sensitive to context composition:

• Evidence-Rich Artifacts: For DeepSeek-V3.2 and Grok-4.1-fast, adding CODE and TEST modules provided the strongest positive marginal effects on quality.
• Noise from Prose: Adding DOCSTRING often had negative marginal effects, suggesting it introduces noise relative to executable evidence.
• Model Dependence: gpt-5-mini performed poorly with broad contexts and benefited more from slice-derived modules (especially SLICE_UNION) which improved grounding.
• Composition Effects: For DeepSeek-V3.2 and Grok-4.1-fast, three-way context combinations generally yielded higher quality scores than two-way or isolated configurations.

4. Downstream Repair Precision (RQ4)

Higher explanation-quality quartiles are associated with:

• Higher Repair Pass Rates: Significant improvements in pass rates for Q4 vs. Q1 in composed settings (two-way and three-way).
• Reference-Like Fixes: For DeepSeek-V3.2 and Grok-4.1-fast, high-quality explanations correlated with fixes that were closer to the reference minimal fix (smaller line deviations and fewer spurious edits). However, this effect was model-dependent and not statistically significant across all models or batches.

Significance and Claims

The paper claims that explanation quality is causally affected by context composition, challenging the assumption that providing more context is inherently beneficial. The authors demonstrate that:

• Context Engineering Matters: Selecting specific, failure-relevant artifacts (like CODE and TEST) and using program slicing to isolate evidence significantly improves the causal clarity and actionability of LLM explanations.
• Quality Drives Utility: High-quality explanations are not just interpretive; they directly improve the success rate and precision of downstream automated repairs. Conversely, poor explanations can actively hinder the repair process.
• Evaluation Must Be Multi-Dimensional: Relying solely on task success (e.g., "did the patch pass tests?") is insufficient. A dedicated framework assessing causal understanding and actionability is necessary to build reliable LLM-based debugging pipelines.

The authors conclude that model-aware context-selection strategies are essential. Systems should prioritize executable artifacts and use slicing to constrain the context, rather than relying on generic, undifferentiated prompts.