Compressing Code Context for LLM-based Issue Resolution

1. Problem Statement

Large Language Models (LLMs) have shown promise in resolving real-world GitHub issues, but their scalability is hindered by context overapproximation.

• The Bottleneck: Current retrieval methods (e.g., RAG, heuristic ranking) return excessive code context (hundreds of lines across multiple files) to ensure no dependencies are missed.
• Consequences:
  1. High Cost: Inference costs scale linearly with context size.
  2. Low Effectiveness: Excessive irrelevant code ("noise") floods the context window, distracting the LLM from the specific "bug-fixing signal."
• Limitations of Existing Solutions:
  • Generic Compressors: Treat code as natural language, disrupting syntactic structure and semantic links (e.g., definition-use relations) required for patch generation.
  • Code-Specific Pruners: Rely on lexical similarity or shallow structural signals. They often discard "fix ingredients" (specific variables, types, or logic) that are semantically necessary for a fix but lexically distant from the bug description.

2. Methodology

The authors propose a two-stage framework: Oracle-guided Code Distillation (OCD) for data generation and SWEzze, a lightweight model trained on this data for inference.

A. Oracle-guided Code Distillation (OCD)

OCD is a search-based algorithm designed to generate training data by finding the Minimal Sufficient Context (MSC) for a given issue.

• Goal: Reduce an initial over-approximated context ($C_{init}$) to a 1-minimal subsequence ($\hat{C}$) where every retained element is functionally necessary for the LLM to generate a passing patch.
• Search Space Representation: Code is decomposed hierarchically into Structural Units:
  • File Level: Whole files.
  • Function Level: Method/function definitions.
  • Block Level: Contiguous sequences of statements within functions.
  • Placeholders: Omitted units are replaced with placeholders (e.g., # ... N lines omitted) to preserve syntactic structure.
• Two-Phase Search Strategy:
  1. Genetic Algorithm (GA): Explores the exponential space of context configurations to find any subset that enables a successful patch. It uses a priority scoring system based on:
     • Patch Overlap: Does the unit belong to files modified by the ground-truth patch?
     • Test Coverage: Does the unit cover lines executed by failing tests?
     • Symbol Overlap: Lexical overlap between the unit and identifiers in the ground-truth patch.
  2. Hierarchical Delta Debugging (HDD): Systematically minimizes the GA-found solution. It iteratively removes units (File $\to$ Function $\to$ Block) to ensure 1-minimality (removing any single unit breaks the fix).
• Data Construction: Applied to the SWESmith dataset (10k issues), producing 3,157 instances of (Issue, Fault Location, $C_{init}$, $\hat{C}$) tuples. Analysis revealed a severe class imbalance (only ~8.4% of segments are relevant) and that "Schema" and "Definition" roles have the highest relevance density.

B. SWEzze (The Compression Model)

SWEzze is a lightweight model trained to mimic the OCD oracle at inference time.

• Architecture: A cross-encoder fine-tuned from Qwen3-Reranker-0.6B using LoRA (Low-Rank Adaptation).
• Training Objective: Pointwise classification. Given a query (Issue + Fault Location) and a code segment, predict if the segment should be retained ($y=1$) or discarded ($y=0$).
• Handling Imbalance: Uses class-level weighting (upsampling positive segments) and role-aware sample weighting (prioritizing high-value semantic roles like Schemas) to prevent the model from learning a trivial "discard all" strategy.
• Inference:
  1. Parses the initial context into segments.
  2. Scores segments using the cross-encoder.
  3. Assembles the final context via budget-aware greedy selection, ensuring the token budget is respected while maximizing the inclusion of high-scoring segments.

3. Key Contributions

1. Oracle-guided Code Distillation (OCD): A novel framework that uses an execution oracle (LLM + Test Suite) to systematically distill code contexts into minimal sufficient subsequences, moving beyond static heuristics.
2. SWEzze: A specialized code context compression model that learns to preserve "fix ingredients" while filtering noise, achieving a stable compression rate without sacrificing repair logic.
3. Comprehensive Evaluation: Demonstrates that preserving sufficiency is more critical than maximizing compression ratio or similarity.

4. Experimental Results

Evaluated on SWE-bench Verified (500 instances) across three frontier LLMs: GPT-5.2, DeepSeek-V3.2, and Qwen3-Coder-Next.

• Resolution Rates (Effectiveness):
  • SWEzze improved issue resolution rates by 5.0% – 9.2% compared to the uncompressed baseline.
  • It covered 93.8% – 99.2% of the union of all instances resolved by any baseline, recovering cases missed by other compressors.
• Compression Efficiency:
  • Maintained a stable ~6× compression rate across all models.
  • Reduced total token budgets by 51.8% – 71.3% compared to uncompressed settings.
• Latency:
  • SWEzze is 4.0× faster than the hierarchical baseline (LongCodeZip) but slower than token-level compressors (LLMLingua-2). It offers the best balance between speed, cost, and effectiveness.
• Failure Analysis:
  • Baselines failed primarily due to: (1) retaining distracting code, (2) imprecise scoring of similar code segments, and (3) breaking syntactic structure via aggressive pruning. SWEzze mitigated these by preserving structural integrity and semantic dependencies.

5. Significance

• Paradigm Shift: The paper argues that for repository-level issue resolution, the goal of compression should not be maximizing token reduction or lexical similarity, but rather preserving repair-sufficient context.
• Cost-Performance Trade-off: SWEzze proves that a moderate compression rate (~6×) that retains critical "fix ingredients" yields better downstream performance than aggressive compression that strips away essential logic.
• Generalizability: The distilled "fix ingredients" are model-agnostic; a model trained on data distilled by one LLM (Qwen3-30B) successfully improves the performance of three different, unseen frontier models.
• Practical Impact: By reducing token costs by ~60% and improving success rates, SWEzze makes automated issue resolution more economically viable for industrial applications.