Logging Like Humans for LLMs: Rethinking Logging via Execution and Runtime Feedback

1. Problem Statement

Current approaches to automatic logging statement generation face two critical limitations in the era of Large Language Models (LLMs):

• Static, Single-Pass Generation: Existing methods rely solely on static code analysis (e.g., ASTs, variable types) to generate logging statements in a single pass. They ignore actual program execution, which often leads to statements that are syntactically correct but diagnostically useless (e.g., logging the wrong variables or missing critical state changes).
• Flawed Evaluation Metrics: Prior work evaluates generated logs by measuring textual similarity to developer-written "gold standard" logs. This assumes developer-written logs are optimal, which is often not the case. Furthermore, it fails to account for the fact that modern logs are increasingly consumed by LLMs for downstream tasks (like defect localization and repair), where utility matters more than similarity.

The core problem is that static generation cannot capture the dynamic runtime behaviors necessary for effective debugging, and the evaluation methodology does not reflect the practical value of logs for automated agents.

2. Methodology: The ReLog Framework

The authors propose ReLog, an iterative, runtime feedback-driven framework designed to generate logging statements that are optimized for downstream LLM-based debugging tasks. Unlike static generators, ReLog treats logging generation as a closed-loop refinement process.

The framework operates in four stages:

1. Initial Logging Generation:

   • ReLog executes the input code to capture runtime signals (e.g., exceptions, return values).
   • It uses both the source code context and the observed execution outcome to generate an initial set of targeted logging statements.
   • Innovation: If an exception occurs, the generator focuses on fault-relevant code regions rather than blindly scanning the whole file.

2. Compilation Repair:

   • Inserting logging statements often introduces compilation errors (e.g., missing imports, unreachable code, type mismatches).
   • ReLog includes a dedicated repair module that uses compiler feedback to fix these errors without altering the original program logic. This ensures the instrumented code is always executable for the next stage.

3. Log Sufficiency Evaluation (The Critic):

   • An LLM-based "Critic" evaluates the runtime logs produced by the instrumented code.
   • Instead of checking similarity to human logs, the Critic assesses observability based on three dimensions:
     • Traceability: Does the log clearly show the execution path?
     • State Visibility: Are key variables and intermediate states recorded?
     • Causal Linkage: Does the log explain why a behavior occurred, not just that it occurred?
   • If the logs are insufficient, the Critor provides actionable feedback (e.g., "Log the value of variable X inside the loop").

4. Iterative Refinement:

   • A "Refiner" module updates the logging statements based on the Critic's feedback.
   • The loop (Generate $\to$ Compile/Repair $\to$ Execute $\to$ Evaluate $\to$ Refine) repeats until the logs are deemed sufficient or a maximum iteration limit is reached.

3. Key Contributions

• ReLog Framework: A novel, execution-aware framework that iteratively refines logging statements using runtime feedback and LLM agents, moving beyond static single-pass generation.
• New Evaluation Paradigm: The authors shift the evaluation metric from "textual similarity to human logs" to "downstream utility." They evaluate generated logs based on their ability to assist LLM agents in defect localization and program repair.
• New Benchmarks: Two new datasets derived from Defects4J were constructed:
  • Direct Debugging: The agent has access to both source code and runtime logs.
  • Indirect Debugging: The agent has no access to source code; it must rely solely on runtime logs and caller context (simulating production environments).
• Empirical Validation: Comprehensive experiments demonstrating that runtime-guided refinement significantly outperforms static baselines.

4. Experimental Results

The authors evaluated ReLog against state-of-the-art baselines (SCLogger, UniLog, LANCE, LANCE2, FastLog) using multiple LLMs (GPT-5-mini, DeepSeek-V3, Qwen3, GLM-4).

Key Findings:

• Superior Debugging Performance:
  • Direct Debugging: ReLog achieved an F1 score of 0.520 for defect localization (16.33% higher than the best baseline) and successfully repaired 97 defects.
  • Indirect Debugging: Even without source code, ReLog achieved an F1 score of 0.408, outperforming the best baseline by 16.57%.
• Robustness Across Models: The performance gains were consistent across different LLM backends, indicating the framework's design is the primary driver of success, not just the underlying model's capability.
• Compilation Stability: ReLog introduced zero compilation failures in the direct setting (and only 2 in indirect), whereas baselines like LANCE2 and FastLog failed to compile in 171 and 142 instances, respectively.
• Ablation Studies: Removing either the Compilation Repair module or the Iterative Refinement loop significantly degraded performance, proving both components are essential.
• Log Quality: ReLog generated more informative logs (avg. 5.52 logs/method) compared to static baselines, capturing necessary control flow and variable states without excessive verbosity.

5. Significance and Impact

• Paradigm Shift: The paper reframes logging generation from a static text prediction task to a runtime-guided engineering process. It acknowledges that effective logging requires observing the system in action.
• LLM-Centric Design: By optimizing logs for LLM consumption (downstream tasks) rather than human readability alone, ReLog bridges the gap between automated code generation and automated debugging.
• Practical Utility: The framework is particularly valuable for indirect debugging scenarios (e.g., production incidents where source code is unavailable), where high-quality runtime logs are the only diagnostic tool available.
• Future Direction: It advocates for evaluating software engineering tools based on their functional utility in downstream tasks rather than superficial similarity metrics.

In summary, ReLog demonstrates that by simulating the human process of "write $\to$ run $\to$ observe $\to$ refine," automated systems can generate logging statements that are significantly more effective for modern, LLM-driven software maintenance.