Technical Summary: Agentic Repository Mining: A Multi-Task Evaluation

Problem Statement

Mining software repositories (MSR) frequently requires classifying artifacts—such as commits, code lines, reviews, or entire repositories—into specific categories. Traditional human labeling is expensive, time-consuming, and prone to error. While Machine Learning (ML) and Large Language Models (LLMs) offer alternatives, they face a critical bottleneck: context engineering.

Standard LLM approaches require humans to pre-engineer the context (e.g., selecting specific code diffs, metadata, or file contents) to fit into the model's prompt. This process is labor-intensive and often leads to misclassifications or uncertainty when the provided context is insufficient or when the artifact size exceeds the model's context window. Conversely, Agentic Repository Mining proposes that LLM agents can autonomously explore repositories using standard development tools (e.g., bash, git) to dynamically retrieve the necessary context for classification, bypassing the need for human pre-engineering. However, it remains unclear whether these self-retrieved contexts can match the classification quality of human-curated inputs.

Methodology

The authors conducted a multi-task empirical evaluation comparing Agentic approaches against Simple LLM baselines across four distinct MSR classification tasks.

Experimental Setup

• Tasks:
  1. Herbold (Tangled Commits): Classifying whether a specific line in a bug-fix commit contributes to the fix or is unrelated (e.g., refactoring, whitespace).
  2. Härtel (Security Reviews): Determining if a Pull Request (PR) review comment discusses a potential security defect.
  3. Levin (Maintenance Activities): Classifying commits as corrective, adaptive, or perfective.
  4. Munaiah (Repository Classification): Distinguishing engineered software projects from homework, toys, or personal experiments.
• Approaches:
  • Simple LLMs: Receive pre-engineered context (e.g., full diffs, commit messages, READMEs) in a single turn. Variants include Chain-of-Thought (CoT), No-CoT, and a Memorization baseline (no context).
  • Agentic LLMs: Operate in a sandboxed Docker container with access to the full repository. They autonomously execute a sequence of bash commands (e.g., git checkout, grep, cat) to explore the repository and gather evidence before classifying. Two tool-calling variants were tested: Native (vendor-specific structured output) and Stop-Sequence (XML-tagged commands).
• Models: Experiments utilized Claude 3.7 Sonnet, Mistral Large 3, and Llama 3.3 70B via Amazon Bedrock.
• Metrics: Classification accuracy (using a hierarchical Bayesian model to account for sample difficulty), resource usage (tokens, time, cost), and failure modes (context overflow, tool errors).
• Scale: 4,943 valid classifications across 4943 experiments (excluding context overflow errors in simple LLMs for some metrics).

Key Contributions

1. Agentic Framework: Introduced a simple framework for agentic repository mining where agents use standard bash commands in isolated containers to dynamically retrieve context for classification.
2. Comparative Evaluation: Provided a direct comparison of accuracy, resource usage, and failure modes between agentic and simple LLM approaches across four diverse tasks.
3. Ground Truth Diagnosis: Manually diagnosed 100 cases where approaches disagreed with the ground truth, revealing ambiguities in original labeling guidelines and limitations of labels produced under limited context.

Results

1. Robustness and Scalability

The primary advantage of agents is robustness to context size.

• Context Overflow: Simple LLMs frequently encountered context-window overflow errors (e.g., 9 errors for Llama, 7 for Sonnet) when artifacts (like full repositories or large diffs) exceeded model limits. Agents encountered zero context overflow errors.
• Cost Scaling: Simple LLM costs correlated linearly with the size of the pre-engineered context. Agent costs showed near-zero correlation with artifact size because agents selectively read only necessary lines.
• Cost Efficiency: While agents were generally more expensive (factors of 1.2x to 3.2x higher) due to multiple tool calls and token accumulation, they became more cost-effective than Simple Sonnet when the engineered context exceeded approximately 30K tokens.

2. Classification Accuracy

• Competitive Performance: Agents achieved accuracy competitive with simple LLMs despite retrieving their own context rather than receiving pre-engineered input.
• Task-Specific Variance:
  • Herbold (Lines): All approaches struggled (46–56% accuracy), with no clear winner.
  • Härtel (Security): Agents performed comparably to Simple Sonnet. Notably, Simple Llama performed best when errors were excluded, likely due to its tendency to predict "unclear" (a valid strategy for this ambiguous task).
  • Levin (Commits): High accuracy across all models (87–93%), with no significant difference between agents and simple LLMs.
  • Munaiah (Repositories): Agents showed a slight edge, particularly Agent Mistral, which was 78% likely to be meaningfully better than Simple Mistral. This task showed the strongest signal for an agent advantage.
• Uncertainty Handling: Both agents and simple LLMs struggled to correctly identify "unclear" labels, often defaulting to a specific category. Agents, having broader context access, tended to under-express uncertainty.

3. Failure Modes and Disagreements

• Tool Errors: Agents had non-fatal tool-calling error rates of ~1.7–2.2% but recovered in subsequent steps.
• Ground Truth Disagreements: Manual diagnosis of 100 disagreement cases revealed:
  • Specification Ambiguities: Many disagreements stemmed from vague task definitions (e.g., distinguishing between whitespace cleanup and refactoring).
  • Context Limitations: Some ground truth labels were likely incorrect or "unclear" because the original human annotators lacked the broader context that agents accessed.
  • Conclusion: Accuracy metrics against existing ground truth may underestimate the performance of approaches with broader context access.

Significance and Claims

The paper claims that agentic repository mining is a viable and robust alternative to traditional context-engineered LLM classification.

• Independence from Artifact Size: Agents scale independently of artifact size, avoiding the context-window bottlenecks that plague simple LLMs.
• Competitive Accuracy: Despite the complexity of dynamic retrieval, agents match the accuracy of human-curated inputs, suggesting that the "context engineering" bottleneck can be automated without sacrificing quality.
• Re-evaluating Ground Truth: The study suggests that existing MSR ground truths, often derived from limited context, may contain inherent ambiguities or errors. Approaches with broader context access (like agents) may provide more accurate classifications than the current "gold standard" labels imply.

The authors conclude that while agents are currently more expensive for small artifacts, their ability to handle large, complex repositories without overflow makes them a superior choice for scalable software engineering research.