Technical Summary: Leveraging Language Models for Log Statement Generation in Multilingual Scenarios

Problem Statement

Log statements are critical for software maintenance activities such as testing, debugging, and failure analysis. However, designing effective log statements requires significant manual effort from developers to determine optimal insertion locations, assign appropriate severity levels, and formulate context-rich messages. While recent advances in Large Language Models (LLMs) have enabled end-to-end automated log statement generation, existing research has primarily evaluated these approaches within single-language environments, specifically Java. Consequently, the effectiveness of these methods in modern multilingual software development environments remains underexplored. Furthermore, it is unclear how training strategies (e.g., monolingual vs. multilingual adaptation) and language-specific logging characteristics influence performance across different programming languages.

Methodology

To address these gaps, the authors conducted a large-scale empirical study involving the construction of a multilingual benchmark and the comparative evaluation of various generation approaches.

Benchmark Construction

The authors constructed a comprehensive multilingual benchmark comprising 150,000 instances across five programming languages: Java, Python, JavaScript, TypeScript, and C#. The dataset was mined from public GitHub repositories, filtered for quality (e.g., minimum commits, contributors, and stars) and specific logging library usage (e.g., Log4j for Java, standard logging for Python).

• Data Processing: Methods were extracted, formatted using language-specific tools (e.g., Black for Python, Prettier for JS/TS), and split into training, validation, and test sets (8:1:1 ratio).
• Instance Generation: A "leave-one-out" strategy was employed where, for methods containing multiple log statements, one statement was removed to serve as the target output while others remained in the input to provide context.
• Balancing: To ensure fair comparison, the dataset was balanced to contain exactly 30,000 instances per language (24k train, 3k valid, 3k test).

Evaluated Approaches

The study evaluated three representative end-to-end approaches and five Large Language Models (LLMs):

1. Existing Approaches:
   • LANCE: A PLM-based approach (T5 architecture) using fine-tuning.
   • FastLog: A two-stage PLM approach predicting insertion locations and generating messages sequentially.
   • UniLog: An LLM-based approach combining retrieval-based few-shot prompting with a lightweight "warmup" (parameter tuning) strategy.
2. Large Language Models:
   • Self-hosted: Llama3.1-8B-Instruct, Qwen2.5-Coder-7B, and Mistral-7B-Instruct-v0.3.
   • API-based: GPT-4.1 mini and DeepSeek-V3.

Experimental Design

The study addressed three Research Questions (RQs):

• RQ1 (Performance): Evaluated all approaches across the five languages using metrics including Position Accuracy, Level Accuracy, Message Accuracy, All Accuracy (exact match of all three elements), BLEU, and ROUGE.
• RQ2 (Training Strategies): Investigated the impact of training strategies using LoRA fine-tuning. Comparisons included Monolingual LoRA (separate adapters per language) vs. Multilingual LoRA (shared adapter), and varying data scales (e.g., 24k vs. 120k total instances).
• RQ3 (Language Characteristics): Analyzed how language-specific factors (insertion location categories, log level distributions, and message diversity) contribute to performance disparities. This involved classifying log locations into six categories (e.g., Try-Catch, Looping Block) and using LLM-as-a-Judge for semantic message evaluation.

Key Results

RQ1: Performance in Multilingual Environments

• Best Overall Approach: UniLog achieved the highest overall performance among all evaluated methods, with an average All Accuracy of 20.35% across the five languages. It outperformed the best LLM (DeepSeek-V3, 10.96% All Accuracy) by a statistically significant margin.
• Language Variance: Significant performance disparities were observed across languages. JavaScript consistently yielded the highest performance (UniLog All Accuracy: 44.83%), while Python showed the lowest (UniLog All Accuracy: 8.30%).
• LLM Performance: Among LLMs, DeepSeek-V3 performed best, but self-hosted models (e.g., Llama3, Qwen2.5-Coder) showed competitive performance on specific subtasks (e.g., Level Accuracy) despite lower end-to-end accuracy.

RQ2: Impact of Training Strategies

• Monolingual vs. Multilingual: When the total training budget was controlled, monolingual training consistently outperformed multilingual joint training. For instance, Mono-LoRA (24k instances per language) achieved 15.13% All Accuracy, whereas Multi-LoRA-24k (4.8k instances per language) achieved only 13.10%.
• Data Efficiency: UniLog demonstrated exceptional data efficiency. The Mono-UniLog-L variant, trained with only 500 instances per language (warmup), achieved 16.24% All Accuracy, outperforming Mono-LoRA (15.13%) which used 24,000 instances per language.
• Metric Sensitivity: Increasing training data size had the most pronounced effect on Position Accuracy, while Level and Message accuracy were more strongly influenced by the specific approach design and adaptation strategy.

RQ3: Language-Specific Characteristics

• Position Difficulty: Prediction difficulty varied by structural context. Looping Blocks (Category 3) were the most difficult to predict (26.3% accuracy). Python had a higher density of loops compared to JavaScript, correlating with its lower position prediction accuracy.
• Level and Message: Log level distributions and message content varied significantly by language. Python exhibited higher message diversity (distinct2), making exact matching harder, whereas JavaScript messages were more formulaic in certain categories, aiding generation.
• Semantic Quality: An LLM-as-a-Judge analysis revealed that UniLog's semantic capability was underestimated by exact-match metrics. While Message Accuracy was 22.74%, 39.9% of generated messages were semantically equivalent to the target ("Same Information"), suggesting strict exact-match metrics penalize valid variations.

Key Contributions

1. Multilingual Benchmark: Release of a large-scale, balanced benchmark of 150,000 log statement instances across five programming languages (Java, Python, JavaScript, TypeScript, C#).
2. Empirical Evaluation: A comprehensive comparison of three state-of-the-art end-to-end approaches and five LLMs under unified experimental conditions, revealing significant cross-language performance gaps.
3. Training Strategy Analysis: Evidence that language-specific adaptation (monolingual training) is superior to joint multilingual training under limited data budgets, and that lightweight warmup strategies (UniLog) can outperform large-scale fine-tuning.
4. Characteristics Analysis: Identification of specific factors driving performance differences, including loop density, log level distribution, and message diversity, providing a basis for future language-tailored approaches.
5. Reproducibility: Public release of the dataset, experimental code, and results.

Significance and Claims

The paper claims that simply scaling model size or increasing the volume of training data is insufficient for robust multilingual log statement generation. Instead, the findings suggest that designing approaches tailored to the specific characteristics of target languages is crucial.

The study highlights that:

• UniLog's combination of retrieval-based prompting and lightweight warmup is currently the most effective strategy for multilingual settings.
• Language-specific adaptation yields better results than generic multilingual training when data is limited.
• Cross-language performance gaps are not uniform; they stem from structural differences (e.g., loop density) and logging idioms (e.g., message diversity).
• Evaluation metrics relying solely on exact string matching (Message Accuracy) may significantly underestimate the practical utility of generated logs, as semantic equivalence is often preserved even when phrasing differs.

The authors conclude that future automated logging techniques must explicitly account for language-specific logging characteristics and insertion-category distributions to achieve robust performance in multilingual software development environments.