Technical Summary: CoSQA+ Enhancing Code Search Evaluation

Problem Statement

Semantic code search aims to retrieve code snippets that match natural language queries, a task critical for software engineering productivity. However, existing benchmarks and evaluation methodologies suffer from three primary limitations:

1. Scalability and Accuracy of Human Annotation: Traditional benchmarks (e.g., CoSQA, CodeSearchNet) rely on human annotators who assess code based on semantic understanding rather than functional verification. This leads to potential inaccuracies due to a lack of full understanding of syntax and behavioral correctness. Furthermore, manual annotation is labor-intensive, limiting dataset scalability.
2. Inadequacy of Single-Choice Paradigms: Existing benchmarks typically enforce a one-to-one query-code matching task, evaluated using Mean Reciprocal Rank (MRR). This contradicts real-world developer behavior, where a single query often yields multiple valid code snippets (multi-choice context). A survey of 200 Python developers revealed that 63.2% of queries commonly result in multiple valid solutions, and developers typically reference 2–3 examples per search.
3. Limitations of LLM-Based Annotation: While Large Language Models (LLMs) offer scalability, they struggle to determine functional correctness, often prioritizing semantic similarity over execution logic. Consequently, both human and LLM annotations suffer from accuracy issues regarding functional verification.

Methodology

The paper introduces CoSQA+, a new benchmark designed for multi-choice code search, constructed through an automated pipeline featuring a novel test-driven agent system.

1. Data Collection and Filtering

• Query Source: Real-world user queries from the CoSQA dataset (originally from Microsoft Bing search logs).
• Code Source: Python functions filtered from the CodeSearchNet corpus. A rule-based AST filter was applied to retain only functions with at least one input parameter and a valid return statement, ensuring testability and relevance (80.97% relevance vs. 16.25% for parameter-less functions).

2. Candidate Pair Construction

To address the multi-choice nature of the task, the authors employ a multi-model embedding approach to select candidates:

• Three embedding models (CodeBERT, UniXcoder, and CodeT5+) generate vector representations for queries and code.
• Cosine similarity is calculated and averaged across the three models to reduce bias toward any single model.
• The top 20 most similar code snippets are selected as candidates for each query.

3. Test-Driven Agent Annotation System

A fully automated pipeline, based on the DeepSeek-V3 model, annotates the query-code pairs. The system operates via Algorithm 1:

• Preliminary Screener: Categorizes pairs into "clear match," "clear non-match," or "unclear." Clear cases are labeled immediately.
• Test Program Generator: For unclear cases, the agent generates executable test suites, including dependency scaffolding and assertions based on the query's intent.
• Test Executor & Bug Fixer: Tests are executed in an isolated Docker environment. If errors occur (e.g., missing dependencies), a Bug Fixer agent generates terminal commands to resolve them iteratively.
• Final Arbiter: Evaluates the execution results and test outcomes to assign a final binary label (1 for exact match, 0 for non-match).

4. Evaluation Framework

The authors constructed two datasets:

• CoSQA+_all: 412,080 agent-annotated pairs.
• CoSQA+_verified: 1,000 human-verified pairs (ground truth) used for rigorous evaluation of the annotation method.
• Metrics: Mean Average Precision (MAP) and MRR are used, with MAP@10 prioritized to capture the retrieval of multiple relevant snippets.

Key Results

1. Annotation Accuracy

The test-driven agent achieved an accuracy of 93.9% against the human-verified ground truth. This significantly outperformed:

• Human annotators without test verification (89.1%).
• Single LLM annotators (DeepSeek-V3: 88.3%, O1-mini: 88.1%, Claude-3.5-Sonnet: 86.3%).
• The high accuracy is attributed to the concrete evidence provided by test execution, which reduces ambiguity and human bias. Additionally, 83.67% of the generated test programs were successfully executable.

2. Benchmark Quality and Model Training

• Superiority over CoSQA: In a comparison where 1,000 identical queries were matched against codes from CoSQA and CoSQA+, 60.3% of the "better" codes were from CoSQA+.
• Training Performance: Fine-tuning CodeBERT, UniXcoder, and CodeT5+ on CoSQA+ yielded consistently higher performance on the CSN99 Python benchmark compared to models trained on CoSQA. Notably, the improvement in MAP@10 was more significant than MRR, indicating better retrieval of multiple relevant codes.

3. Component Analysis (RQ4)

An ablation study revealed that the Bug Fixer component was the most critical for performance. Configurations without the Bug Fixer (relying on simulated execution or no execution) resulted in lower accuracy (~0.85), while the full pipeline with the Bug Fixer achieved 0.939. This highlights the necessity of real execution feedback over simulated reasoning.

4. Cross-Language Generalization (RQ5)

The test-driven agent demonstrated strong generalization to PHP, Java, and Go, achieving average accuracy gains of 10.33% over single LLM annotators across these languages.

Significance and Claims

The paper claims the following contributions:

1. First Multi-Choice Benchmark: CoSQA+ is presented as the first benchmark specifically designed to evaluate multi-choice code search, addressing the gap between single-choice evaluation paradigms and real-world developer needs.
2. High-Accuracy Automated Annotation: The authors developed the first multi-agent annotator for code search datasets, achieving 93.9% accuracy, surpassing both traditional human annotation (without tests) and single LLM approaches.
3. Enhanced Construction Process: The paper demonstrates an enhanced, fully automated construction process utilizing multi-model candidate selection and test-driven agent annotation to create high-quality, large-scale datasets.
4. Resource Availability: The authors publicly release CoSQA+_all (412,080 pairs) and CoSQA+_verified (1,000 pairs) to facilitate further research in semantic code search.

The authors conclude that CoSQA+ offers superior quality and training performance compared to existing benchmarks like CoSQA, and that the test-driven approach provides a scalable, reliable alternative to manual annotation for evaluating functional code search.