Single-Language Evidence Is Insufficient for Automated Logging: A Multilingual Benchmark and Empirical Study with LLMs

Technical Summary: Single-Language Evidence Is Insufficient for Automated Logging

Problem Statement
Automated logging statement generation is a critical task for software maintenance, failure diagnosis, and production observability. While developers rely on logs for runtime behavior understanding, manually crafting high-quality logging statements is labor-intensive and context-dependent, requiring decisions on insertion location, API selection, severity levels, message content, and variable exposure. Although recent research has explored automated generation using Large Language Models (LLMs), existing evidence suffers from two primary limitations that restrict the generalizability of findings:

1. Single-Language Bias: Prior studies (e.g., LANCE, LEONID, SCLogger) are predominantly Java-centric. Automated logging depends heavily on language-specific factors such as syntax, type systems, control-flow idioms, and logging framework designs. Consequently, conclusions drawn from a single ecosystem may not transfer to others.
2. Repository-Snapshot Limitation: Most benchmarks rely on frozen repository snapshots where the model reconstructs an existing logging statement from static context. This fails to capture the dynamic nature of code evolution, where developers introduce logging statements while simultaneously modifying functionality or addressing failures.

Methodology: MultiLogBench
To address these gaps, the authors construct MultiLogBench, a multilingual benchmark and empirical study spanning six programming language ecosystems: Java, Python, Go, C++, JavaScript, and C#. The benchmark integrates three complementary data branches:

1. Repository-Snapshot Data: The primary controlled benchmark containing 63,965 production-code instances mined from frozen revisions. This serves as the baseline for cross-language comparison.
2. Revision-History Data: A complementary branch containing 744 instances extracted from authentic commits where developers introduced logging statements. This branch tests whether findings hold in realistic maintenance settings where logging decisions are interleaved with code evolution.
3. Transformed Revision-History Data: A robustness branch where the surrounding code of the revision-history instances is rewritten using semantics-preserving surface transformations (e.g., parenthesis insertion, condition neutralization, block normalization). This tests whether model performance relies on exact surface memorization of pretraining data rather than genuine reasoning.

The study evaluates seven contemporary LLMs (including frontier closed-source models and specialized code models) under a unified protocol. The evaluation decomposes the task into specific components:

• Logging-site localization: Measured by Position Accuracy (PA).
• Framework-anchor matching: Measured by Framework Accuracy (FA), assessing if the correct API/receiver is used.
• Severity prediction: Measured by Level Accuracy (LA) and Average Ordinal Distance (AOD).
• Message generation: Measured by BLEU-4 and ROUGE-L.
• Variable recovery: Measured by Precisely Matched Rate (PMR) and set-based F1.
• Overall quality: A Cascaded Composite Score (CCS) that gates content quality metrics on correct position and framework selection.

Key Results
The empirical study yields four primary findings across the research questions (RQ1–RQ4):

• RQ1: Cross-Language Variation: Performance varies substantially across languages. Java achieves the highest overall quality (CCS), followed by Go and C#, while C++ and JavaScript show lower scores. Crucially, framework-anchor matching (FA) is the most language-sensitive component, exhibiting the largest cross-language dispersion. For instance, C++ shows a significant weakness in framework matching compared to other languages.
• RQ2: Structural Difficulty: The difficulty of generation is unevenly distributed within functions. Loop and nested-callable contexts are the most challenging structural environments, requiring models to localize logging statements within iterative updates or competing inner scopes. In contrast, branch and exception-handling contexts are comparatively easier.
• RQ3: Stability in Maintenance Settings: When transitioning from repository-snapshot data to revision-history data, absolute performance declines across all languages and metrics. While coarse-grained cross-language differences persist, model rankings become significantly less stable. The top-tier models (Gemini-3.1-Pro and Claude-Sonnet-4.6) remain relatively stable, but middle- and lower-tier models reorder frequently across languages in the revision-history setting, rendering single-language leaderboards insufficient for fine-grained model selection.
• RQ4: Robustness to Transformations: Under semantics-preserving code transformations, the models do not exhibit a broad, same-direction performance collapse. The mean change in CCS is negligible (−0.005), suggesting that the observed performance patterns are not primarily driven by exact surface-form memorization of the benchmark instances.

Qualitative Failure Analysis
Qualitative analysis reveals two dominant failure patterns:

1. Framework Confusion: Models often correctly identify the intent to log but fail to align with project-specific API families or receiver conventions (e.g., using generic LOG instead of project-specific VLOG in C++, or instance receivers instead of static ones in C#).
2. Structural Ambiguity: In loop and nested-callable contexts, models struggle to localize the logging statement to the correct subscope, often attaching messages to broader operations or neighboring handlers rather than the specific iterative or callback context.

Significance and Contributions
The paper argues that single-language evidence is insufficient for drawing robust conclusions about automated logging statement generation. The key contributions are:

1. MultiLogBench: A multilingual benchmark integrating repository-snapshot, revision-history, and transformed data to evaluate logging generation across six ecosystems.
2. Unified Evaluation Protocol: A framework that decomposes generation quality into localization, framework matching, severity, message, and variable components, allowing for the isolation of specific failure sources.
3. Empirical Evidence of Instability: Systematic evidence showing that conclusions derived from one language ecosystem (e.g., Java) do not reliably transfer to others. Specifically, framework-anchor matching is a major bottleneck, and model rankings are unstable outside the frontier tier when evaluated across languages or realistic maintenance settings.
4. Validation of Robustness: Demonstration that while performance degrades in realistic maintenance scenarios, the core multilingual patterns persist and are not merely artifacts of data leakage or surface memorization.

The authors conclude that future work and tool development must prioritize multilingual evaluation and maintenance-oriented validation. They emphasize that deployment decisions should be language-aware rather than relying on global model-centric leaderboards, and that progress in automated logging requires addressing ecosystem-specific realization (framework matching) and structural localization (handling loops and nested scopes) rather than just improving message text generation.