Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP
Codebase-Memory is an open-source system that leverages Tree-Sitter and the Model Context Protocol to construct persistent knowledge graphs of codebases, enabling LLM agents to explore complex repositories with significantly reduced token consumption and tool calls while maintaining competitive answer quality.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a detective trying to solve a mystery inside a massive, chaotic library. This library contains millions of books (code files), but they aren't organized on shelves; they are just piled in huge, unsorted heaps.
The Old Way (The "Explorer" Agent)
Currently, most AI coding assistants work like a detective who has to physically run to every single book, open it, read a few pages, close it, and then run to the next one to see if it's related.
- The Problem: If you ask, "What happens if I change this one sentence in Book A?", the AI has to run to Book A, find the sentence, see it mentions Book B, run to Book B, find a reference to Book C, and so on.
- The Cost: This takes forever. It burns through a massive amount of "energy" (computer tokens) and requires thousands of trips (tool calls) just to understand a simple connection. It's like trying to find a specific thread in a haystack by pulling out every single piece of hay one by one.
The New Way (Codebase-Memory)
The authors of this paper built a system called Codebase-Memory. Instead of treating the library as a pile of paper, they built a giant, interactive 3D map of the entire library before the detective even arrives.
Here is how it works, using simple analogies:
1. The "Instant Map" (Tree-Sitter & Knowledge Graph)
Imagine the system walks through the library once and draws a perfect map.
- It doesn't just write down the text; it draws lines connecting related things.
- It knows that "Function A" calls "Function B."
- It knows that "Class X" inherits from "Class Y."
- It groups books into neighborhoods (Communities) based on what they do together.
- The Magic: This map is stored in a tiny, super-fast database (SQLite). It's like having a GPS for code.
2. The "Super-Connector" (MCP)
The system uses a standard protocol called MCP (Model Context Protocol). Think of this as a universal remote control.
- Any AI agent (like Claude, Cursor, or Aider) can pick up this remote.
- Instead of asking the AI to "read the whole book," the agent just presses a button on the remote: "Show me all the lines connected to this button."
- The system instantly zooms in on the map and shows the answer in a split second.
3. The "Self-Updating" Feature
Libraries change; new books are added, and old ones are edited.
- Codebase-Memory has a "security guard" (a file watcher) that watches the library.
- If a single page is changed, the guard doesn't redraw the whole map. It just erases the tiny section that changed and redraws that specific corner. This keeps the map fresh without slowing things down.
4. The "Safety First" Approach
Because this system gives the AI deep access to your computer's files, the authors were very worried about security (like giving a stranger the keys to your house).
- They built an 8-layer security audit.
- They treat the software like a high-security vault. Before releasing it, they run it through dozens of antivirus scanners, check every line of code for hidden traps, and verify its "birth certificate" (provenance) to ensure no one tampered with it. It's like a car that comes with a factory-sealed engine and a guarantee that no one has ever touched the bolts.
The Results: Why It Matters
The researchers tested this new system against the old "run-and-read" method on 31 different programming languages (like Python, Java, C++, etc.).
- Speed: The new system is 1,000 times faster at answering structural questions.
- Efficiency: It uses 10 times less energy (tokens) to get the same answer.
- Accuracy: It gets the right answer 83% of the time (compared to 92% for the slow method), but for complex "connection" questions (like "Who uses this function?"), it actually beats the slow method.
The Bottom Line
Codebase-Memory is like upgrading from a detective who has to read every book in the library to a detective who has a magic, living map that shows every connection instantly.
It saves time, saves money (less computer power used), and makes AI coding assistants much smarter at understanding how code fits together, all while keeping your data safe with military-grade security checks. It turns the chaotic haystack into an organized, navigable city.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.