💻 computer science

Natural Language-Focused Software Engineering via Code-Documentation Equivalence

This paper introduces "Documentary," a novel approach that generates documentation-to-code equivalent descriptions, demonstrating that such high-quality, accurate documentation significantly improves large language model performance in code understanding and editing tasks while also being rated as more useful by human developers than existing documentation.

Original authors: Aryaz Eghbali, Zhongxin Liu, Michael Pradel

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aryaz Eghbali, Zhongxin Liu, Michael Pradel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Bad Map"

Imagine you are handed a complex machine, like a high-tech toaster, but the instruction manual is missing, outdated, or says things like "Push the button to toast." It doesn't tell you which button, how long to push it, or what happens if you push it twice.

In software development, this "bad manual" is called documentation. It's supposed to explain what the code (the machine) does. But often, the documentation is:

Incomplete: It leaves out important steps.
Outdated: It describes how the machine worked last year, not today.
Vague: It says "it does stuff" instead of "it heats the bread."

When humans try to fix the machine, they get confused. Now, we also have "AI robots" (Large Language Models or LLMs) trying to fix the machine. If the AI reads a bad manual, it will make mistakes, too.

The New Idea: "The Perfect Translation"

The authors of this paper introduce a new concept called Documentation-to-Code Equivalence.

Think of it like a perfect translation. If you translate a recipe from French to English, a "perfect" translation is one where, if you give the English recipe to a chef, they can cook the exact same dish as the French recipe intended. No missing ingredients, no wrong temperatures.

In this paper, "equivalence" means: If you give the documentation to an AI, can the AI write the exact same code that the original programmer wrote? If yes, the documentation is "equivalent." If no, the documentation is missing something important.

The Solution: "Documentary" (The Self-Correcting Editor)

The authors built a tool called Documentary. It's like a super-smart editor that keeps rewriting the manual until it's perfect. Here is how it works, step-by-step:

The Guess: Documentary looks at the code and the existing (bad) manual. It asks an AI: "Based on this manual, write the code."
The Comparison: The AI writes a new version of the code. Documentary then compares this new version with the original code.
- Analogy: Imagine the AI tries to draw a picture of a cat based on a description. Documentary then checks: "Does this drawing look exactly like the real cat?"
The Correction: If the drawing is wrong (e.g., the AI drew a dog because the manual said "it has fur"), Documentary tells the AI: "You got it wrong. The real code does X, but your code did Y. Fix the manual to explain X."
Repeat: The AI rewrites the manual, tries to draw the cat again, and checks the result. It keeps looping until the drawing matches the real cat perfectly.

What Did They Find?

The researchers tested this on real Python code (a popular programming language).

Success Rate: Documentary successfully created "perfect manuals" (equivalent documentation) for about 53% of the code snippets it tried. This is much better than just asking an AI to write a manual in one go (which only worked 34% of the time).
Helping AI: When the researchers used these new "perfect manuals" to ask an AI to predict what a piece of code would do, the AI got it right 12% to 24% more often than when using the old, human-written manuals.
Helping Humans: They also asked human developers to use these new manuals to understand and edit code. The humans said the new manuals were more useful and helped them understand the code better than the original ones.

Why Does This Matter?

The paper argues that as we rely more on AI to write and fix software, the quality of the "manuals" (documentation) becomes critical.

For AI: If the manual is vague, the AI guesses wrong. If the manual is "equivalent" (perfectly precise), the AI acts like a master mechanic.
For Humans: Even for humans, a manual that describes every side-effect and interaction (like the "Documentary" ones) is easier to understand than a vague summary.

The Bottom Line

The paper doesn't claim this fixes every piece of code (it worked on about half the cases). But it proves that if we treat documentation as a two-way street—where the text must be precise enough to rebuild the code—we can make software easier for both humans and AI to understand and fix.

In short: They built a tool that keeps rewriting code manuals until they are so accurate that an AI can rebuild the code from scratch just by reading them. This makes the AI smarter and the humans' jobs easier.

Technical Summary: Natural Language-Focused Software Engineering via Code-Documentation Equivalence

Problem Statement

Source code documentation is essential for understanding code and facilitating developer communication, yet existing documentation is frequently incomplete, outdated, or inaccurate. This deficiency hinders code comprehension and delays development. The problem is exacerbated in the era of Large Language Models (LLMs), where documentation serves as a critical context for model performance.

Prior work has focused on summarizing code or ensuring documentation consistency (soundness) with code changes. However, these approaches fail to guarantee completeness. A summary often omits details, and consistent documentation merely ensures the text does not contradict the code, without ensuring it fully describes the code's behavior, side effects, or interactions. Furthermore, evaluating documentation quality has historically relied on textual similarity to human-written documentation, a metric that is often vague and fails to capture semantic equivalence, especially since human documentation itself is often incomplete.

Methodology

Core Concept: Documentation-to-Code Equivalence

The paper introduces documentation-to-code equivalence, a novel property defining whether documentation accurately and completely describes the corresponding code. Formally, documentation $d$ is equivalent to code $c$ (given context $ctx$ and an LLM $L$ ) if $L$ can generate $c$ from $(d, ctx)$. This property ensures both soundness (no contradictions) and completeness (full description), evaluated by checking if an LLM can reconstruct the original code from the documentation.

The Documentary Approach

To automatically generate equivalent documentation, the authors propose Documentary, an iterative refinement approach. The process operates as follows:

Initialization: Start with an existing docstring (or an empty string) and the surrounding code context.
Code Generation: Prompt an LLM to generate the function body based on the current docstring and context.
Equivalence Check: Use an LLM as a "judge" to compare the generated code against the original code. The judge analyzes semantic differences and determines if the two snippets are equivalent, ignoring implicit errors where input validity is unknown.
Refinement: If the code is not equivalent, the LLM generates a description of the differences between the original and generated code. This description is used to prompt the LLM to generate a revised, more specific docstring that addresses the missing information.
Iteration: Steps 2–4 repeat until the generated code is judged equivalent to the original or a maximum iteration limit is reached. A final refinement step ensures the docstring does not contain information unsupported by the implementation.

Evaluation Tasks

The paper evaluates the utility of equivalent documentation in two software engineering tasks:

Code Understanding (Output Prediction): Given a function signature, documentation, and a test prefix, an LLM must predict the output of an assertion.
Code Editing: Human developers modify code based on a natural language description of a desired change, using either original or equivalent documentation.

Key Contributions

Documentation-to-Code Equivalence: A formal property capturing the semantic relationship where documentation contains sufficient information to re-implement the code.
Documentary: An iterative, LLM-driven approach to automatically generate equivalent documentation by round-tripping code generation and comparison.
Empirical Evidence: Demonstrations that equivalent documentation significantly improves performance in code understanding and editing tasks for both LLMs and human developers.

Results

Generation Effectiveness

Evaluated on 238 function-level snippets from real-world Python projects:

Success Rate: Documentary generated equivalent docstrings for 53.4% of the functions (compared to 34.0% for a single-step baseline).
Precision: When Documentary claimed equivalence, the generated docstrings were actually equivalent 66.4% of the time.
Iteration: 75% of successful generations occurred within five iterations, with a median of two.
Cost: The average cost per function was approximately $0.009, with an average runtime of 33.8 seconds.

Impact on LLMs (Code Understanding)

In the output prediction task, using equivalent documentation improved LLM performance significantly compared to original human-written documentation and baseline-generated documentation:

Accuracy: LLMs predicted function outputs with 12.8–24.5% higher accuracy.
Metrics: The number of correctly predicted assertions increased by 4.0–15.9, and the number of tests with all assertions correctly predicted increased by 5.6–19.0 percentage points.

Impact on Human Developers (Code Editing)

A user study with 11 participants showed:

Helpfulness: Participants rated equivalent documentation as significantly more helpful for understanding code (3.76 vs. 2.58, $p=0.0004$ ) and modifying code (2.85 vs. 2.15, $p=0.0425$ ).
Confidence & Difficulty: While participants reported higher confidence and lower difficulty with equivalent documentation, these differences were not statistically significant in this sample size.
Time: No statistically significant difference was found in the time taken to understand or modify code.

Model Independence

Equivalent documentation generated by one LLM (gpt-4.1-nano) remained effective when used by other models (gpt-5-nano and gemini-2.5-flash-lite) to generate code, with success rates of 89.6% and 74.7% respectively, demonstrating the portability of the generated documentation.

Significance and Claims

The paper posits that documentation-to-code equivalence is a crucial step toward natural language-focused software engineering, where tasks are performed based on natural language documentation as a primary information source. The authors claim that:

Equivalent documentation bridges the gap between natural language and formal code, providing a more objective and complete description than traditional summaries or consistency checks.
It enhances the capabilities of both LLMs and human developers, particularly in unfamiliar codebases where context is scarce.
The approach is practical, with low computational costs and high transferability across different LLMs, making it viable for adoption in large codebases.

The authors acknowledge limitations, including reliance on test suites for equivalence checking (which may not be comprehensive), the focus on Python, and the small sample size of the user study. However, they assert that the principles of their approach are language-agnostic and offer a robust framework for improving software engineering workflows.