Technical Summary: Go-UT-Bench

Problem Statement

Training data imbalance presents a significant hurdle for Code Large Language Models (LLMs). While models excel at code autocompletion due to the abundance of raw open-source code, they struggle with complex software engineering workflows, particularly unit test generation. This gap is exacerbated in low-resource languages like Golang, where high-quality, task-specific datasets are scarce. Existing efforts often lack diversity, permissive licensing, or reproducibility. Furthermore, the closed-source data policies of leading GenAI organizations hinder open research. Writing unit tests in Go is inherently challenging due to the language's static typing, concurrency model (Goroutines, channels), and error handling mechanisms, creating a specific need for automation that current base models fail to adequately support.

Methodology

Dataset Construction: Go-UT-Bench

The authors introduce Go-UT-Bench, a fine-tuning dataset comprising 5,264 {code, unit test} pairs. The dataset is curated from 10 permissively licensed, industrial-scale Go repositories spanning diverse domains, including distributed systems (Kubernetes, TiDB), cloud infrastructure (Terraform, Moby), blockchain (Ethereum), and API development (Gin).

The curation process involves a scalable, two-stage pipeline:

1. File Extraction and Grouping: The pipeline recursively scans repositories to identify Go source files (.go) and their corresponding test files (_test.go). Files are grouped by base name to ensure a direct mapping between source and test.
2. Metadata Augmentation: Each pair is enriched with metadata to guarantee reproducibility and provenance tracking, including SHA256 identifiers, repository names, file paths, and specific Git commit hashes for both the code and the unit test.

The dataset exhibits heterogeneity in file sizes (ranging from 20 to over 1,500 lines) and domain diversity, reflecting real-world software engineering practices.

Experimental Workflow

To evaluate the dataset's efficacy, the authors employed a three-step workflow for unit test generation:

1. Code Chunking: To address the "lost in the middle" phenomenon and context window limitations in large Go files, the authors implemented an Abstract Syntax Tree (AST)-based chunking strategy. This partitions files into logically connected segments while preserving dependencies.
2. In-Context Learning (ICL) Generation: Unit tests are generated for each chunk using prompt engineering.
3. Compilation: Generated chunks are aggregated to form the final unit test file.

Evaluation Protocol

The study utilizes an LLM-as-a-Judge paradigm, employing GPT-4o-mini as an oracle to perform pairwise evaluations. This approach compares the outputs of a base model against a fine-tuned model for each input. The oracle assigns alignment scores based on the generated test's similarity to the ground truth. Performance is measured by win rates (the percentage of instances where the fine-tuned model outperforms the base model) and tie rates.

Two model families were tested:

• Mixture-of-Experts (MoE): DeepSeek-Coder-V2-Lite-Instruct (16B total parameters, 2.4B active).
• Dense Decoder: Llama-3.2-3B-Instruct.

Experiments utilized a stratified split (3,684 training, 790 validation, 790 testing) on NVIDIA H100 hardware.

Key Results

Fine-tuning on Go-UT-Bench yielded substantial improvements over base models:

• DeepSeek-Coder-V2-Lite-Instruct: After Low-Rank Adaptation (LoRA) fine-tuning, the win rate increased from 14.2% (base) to 81.9%. The model surpassed the mean win rate in 7 out of 10 repositories, demonstrating strong generalization, though performance dipped slightly on underrepresented repositories like moby/moby and gin-Gonic/gin.
• Llama-3.2-3B-Instruct: After full fine-tuning, the win rate rose from 22.0% (base) to 76.7%. This model also showed generalizability across 6 out of 10 repositories, with performance declines observed in moby/moby and hashicorp/terraform, potentially attributed to the model's smaller parameter count.

Overall, fine-tuned models outperformed their base counterparts on more than 75% of benchmark tasks, validating the dataset's utility for instruction adaptation.

Key Contributions

1. Go-UT-Bench Dataset: An extensible, reproducible fine-tuning dataset of 5,264 {code, unit test} pairs derived from real-world industrial projects, designed to support LLM adaptation for Go unit test generation.
2. Reproducible Curation Pipeline: A methodology for extracting and mapping code-test pairs from permissively licensed open-source repositories, complete with commit hashes and metadata for version tracking.
3. AST-Guided Chunking: A methodological contribution to handle long, monolithic Go files by leveraging AST parsing to create contextually coherent chunks, mitigating context window limitations.
4. Empirical Validation: Demonstration that fine-tuning on this specific dataset significantly enhances unit test generation quality across different model architectures (MoE and dense).

Significance and Limitations

The paper positions Go-UT-Bench as a critical resource for advancing AI-driven software engineering, specifically addressing the underexplored domain of statically typed, concurrency-heavy languages like Go. Unlike prior Python-centric benchmarks (e.g., SWE-bench, SWT-bench), this work explicitly targets the structural and semantic complexities of Go.

The authors acknowledge several limitations:

• Evaluation Bias: Reliance on a single LLM oracle (GPT-4o-mini) without human judgment or execution-based metrics (e.g., compilation success) may introduce bias or oversimplify performance assessment.
• Data Leakage: As the dataset is derived from public repositories, there is a risk that pre-trained models were exposed to this data, potentially inflating performance metrics.
• Representation Bias: Certain domains (e.g., blockchain, API development) are underrepresented compared to infrastructure projects, which may skew model performance.

The authors conclude that while Go-UT-Bench provides a robust foundation for research, future work should incorporate human expert evaluation, execution-based metrics, and strategies to mitigate data leakage and representation imbalances.