CODENS: Transforming Code Changes into Living, Accessible, and Queryable Documentation
CODENS is a system that transforms pull requests into living, queryable documentation by incrementally building a typed software knowledge graph to enable agent-guided repository-level question answering, as demonstrated by its successful evaluation on a production Ruby on Rails project.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are walking through a massive, ancient library where the books are constantly being rewritten. Every time a new chapter is added, a note is scribbled in the margin, or a page is torn out and replaced, the story changes. In the world of computer software, this library is the "codebase," and the notes are "pull requests"—the digital tickets developers use to propose changes. The problem is that the "story" of how the software works (its documentation) often gets lost in the chaos of these changes. Developers end up guessing how things fit together because the instructions are scattered across thousands of files and old chat logs.
To solve this, scientists have started building "knowledge graphs." Think of these not as a simple list of facts, but as a giant, living map where every piece of code is a city, and the connections between them are roads. If you change one city, the map automatically updates the roads leading to it. Another key idea is "Retrieval-Augmented Generation" (RAG), which is like giving a super-smart librarian a specific set of books to read before they answer your question, ensuring they don't just guess but actually find the truth in the text. The big question researchers are asking is: Can we build a map that doesn't just show the library as it is today, but remembers every single change ever made, turning the history of code updates into a living, searchable guide?
Enter CODENS, a new system designed to do exactly that. Imagine CODENS as a magical, hyper-organized archivist that watches over a software project like a Ruby on Rails web platform. Instead of just reading the final version of the code, CODENS watches the "movie" of the project's history. It starts by scanning the entire codebase to build a skeleton map, identifying 1,739 different "rooms" (like controllers, models, and views) based on the standard rules of the framework.
Then, the magic happens. CODENS replays the project's history, one pull request at a time, like watching a time-lapse video of a building being constructed. For every change, it doesn't just look at the new bricks; it asks a powerful AI (an LLM) to explain why the change was made and how it fits with what was already there. Crucially, it doesn't just overwrite the old story; it merges the new details with the old ones. If a developer adds a new button to a page, CODENS updates the map to show the button exists, but it also keeps the history of what the page looked like before. This creates a "living documentation" layer that grows smarter and more detailed with every single update.
Once this map is built, CODENS offers three ways to explore it, like different modes of travel. You can do a quick "vector search," which is like asking the librarian for books with similar words. You can do "multi-hop retrieval," which is like following a trail of breadcrumbs from one room to the next to see how they connect. But the most powerful mode is "agent-guided traversal." Here, an AI agent acts like a detective with a flashlight. It doesn't just follow a fixed path; it decides which doors to open, which roads to take, and even writes its own custom queries to find the exact answer. It uses tools to peek inside specific nodes, check their neighbors, or run complex searches to piece together a complete answer.
The researchers tested this system on a real-world, industrial project with over 1,700 files and hundreds of pull requests. They asked the system 11 different questions that a real developer might have, such as "How does this feature work?" or "What happens when I click this button?" The results were promising. The system scored very high on accuracy and "faithfulness," meaning the answers were grounded in the actual code and didn't make things up. On a scale of 1 to 5, human experts rated the relevance of the answers at an average of 4.09 and the completeness at 4.45. The system was also able to track the cost and time of each query, showing that while the detective mode (agent) took more time and tokens, it provided the most thorough answers.
However, the paper also points out a small snag. While the system was great at finding the right information, the way it presented the answers sometimes felt a bit too technical. The human reviewers noted that some answers were "too detailed" or "too code-centric," listing line-by-line citations when a simpler summary would have been better. It's like the librarian gave you the exact page number and paragraph of a book but didn't summarize the story for you. The authors suggest that the next step isn't necessarily finding better information, but teaching the system to synthesize that information into clearer, more user-friendly explanations.
In short, CODENS suggests that we can turn the messy, scattered history of code changes into a structured, queryable memory for software. It proves that by treating pull requests as a continuous stream of knowledge rather than just temporary fixes, we can build a system that helps developers understand complex codebases much faster. While it's not a perfect, solved problem yet—especially regarding how to make the answers sound more like a helpful human and less like a robot reading a manual—it offers a vivid glimpse into a future where software documentation is alive, up-to-date, and always ready to answer your questions.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.