Ephemeral Subgraph Generation: Real-Time Knowledge GraphConstruction for Cross-System Investigation
This paper proposes Ephemeral Subgraph Generation (ESG), a real-time, LLM-guided approach that constructs temporary, question-specific knowledge graphs across heterogeneous engineering systems to outperform static retrieval and fixed-hop baselines in cross-system investigations, while also documenting and resolving specific software defects identified during evaluation.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern world of software engineering, critical information is rarely found in a single place. When a service fails or a bug appears, the truth is scattered across a dozen different digital silos: a ticketing system where the problem was first logged, a code repository where the fix was written, an incident management dashboard, a documentation wiki, and internal chat logs. Each of these systems holds a piece of the puzzle, but none holds the whole picture. Traditionally, engineers have tried to solve this by building a massive, permanent map that connects every piece of data from every system in advance. This approach is expensive to build, difficult to keep up to date as the underlying systems change, and often fails to capture the subtle, non-obvious connections that lead to a solution.
A new approach, described in recent research, suggests a different way to think about this problem. Instead of building a permanent map that tries to cover everything all the time, this method constructs a small, temporary map only when a specific question is asked. It is like sending a specialized team to investigate a single incident, gathering only the evidence needed for that specific case, and then packing up the evidence once the answer is found. This technique, called Ephemeral Subgraph Generation, allows an artificial intelligence system to hop between different software platforms, following clues that link a ticket to a line of code, or a chat message to a documentation page, without needing a pre-existing database of all possible connections. The goal is to find the root cause of a problem by actively exploring the relationships between scattered documents, rather than just searching for keywords in a static list.
The researcher behind this study, Saket Jain, set out to test whether this on-demand, temporary map could find answers that traditional methods miss. To do this, they created a synthetic environment mimicking a real engineering organization, complete with 154 records spread across five different systems: an incident manager, a ticketing system, a code host, a wiki, and a chat platform. They then posed 35 specific questions to the system, ranging from "why did the login service fail?" to "what changes were made recently that might be related?" The system was tasked with finding the correct documents to answer these questions. The researcher compared this new method against two simpler, cheaper approaches. The first was a standard search that looked at all documents at once without trying to link them together. The second was a slightly more advanced method that looked for direct identifiers, like ticket numbers, and then followed those numbers to the next document, but only for a fixed number of steps.
The results showed a clear advantage for the temporary map approach. When measuring how many of the correct documents the system found, the new method succeeded in nearly 97 percent of cases. In contrast, the simple search found only about 58 percent of the correct documents, and the method that followed identifiers for two steps found about 70 percent. The difference was not just a matter of finding one or two extra documents; the new method found connections that the other approaches could not reach at all. Specifically, it succeeded in cases where the documents shared no common names, ticket numbers, or obvious text links. These were "soft" connections, where the link between a problem and its cause existed only in the narrative flow of the text or in the metadata of a file, invisible to methods that relied on following a chain of known identifiers. The study demonstrated that simply following a chain of known links, no matter how long, hits a ceiling where it cannot go further, whereas the new method could jump across gaps by understanding the meaning of the content.
However, this improved ability to find answers came with a significant cost. The new method required much more time and money to run. While the simpler methods cost less than a dollar and took a few minutes to complete, the new method cost roughly six dollars per question and took about an hour to finish a full set of questions. This is because the system had to make many more calls to the artificial intelligence to decide if a document was relevant, to extract names from the text, and to verify connections. The researcher was transparent about this trade-off, noting that the higher cost was the price paid for the ability to find the elusive connections that the cheaper methods missed. The precision of the answers, or how many of the retrieved documents were actually useful, was slightly lower for the new method, but the researcher found that much of this was due to the system finding extra, correct documents that simply weren't on the original list of expected answers, rather than finding wrong information.
During the evaluation, the researcher also discovered and fixed several flaws in the system, treating the process as a rigorous scientific experiment rather than just a demonstration of success. One issue was that the system initially accepted documents just because they contained a searched name, even if the document was irrelevant. This was fixed by adding a relevance check. Another issue was that the system sometimes stopped processing a large list of potential answers because it ran out of space for the response, causing it to silently reject valid documents. This was fixed by increasing the space allowed for the response. A third, more subtle limitation was found where the system struggled to judge connections that existed only in the structured metadata of a file rather than in the text itself; for this specific case, the researcher decided to bypass the judgment step entirely. These corrections were verified by running the tests multiple times, confirming that the system's performance was stable and that the improvements were real.
The study concludes that for complex investigations where information is scattered and connections are not obvious, building a temporary, question-specific map is a powerful tool. It outperforms methods that rely on fixed chains of identifiers or simple keyword searches, particularly when the answer lies in the subtle relationships between documents. While the cost is higher, the ability to uncover the full picture of a problem, including the parts that no single system knows, suggests that this approach could be valuable for organizations facing complex, multi-system challenges. The researcher notes that this work was tested on a synthetic dataset and within the specific context of software engineering, leaving open the question of how well it would scale to real-world, messy data or other fields like fraud detection. Nevertheless, the findings provide a clear path forward for systems that need to investigate problems across a fragmented digital landscape, proving that sometimes the best way to find the answer is to build a new map for every question.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.