CTIFoundry: An Agent-Native Corpus Scaffold for Cyber Threat Intelligence
This paper introduces CTIFoundry, an agent-native corpus scaffold that materializes the latent structure of Cyber Threat Intelligence through a deterministic ontology graph and span-grounded reports, significantly enhancing agentic investigation performance and efficiency compared to traditional flat retrieval substrates.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the digital world, security experts constantly hunt for clues about who is attacking them and how. This work, known as cyber threat intelligence, relies on massive collections of reports written by different companies and organizations. For years, the standard way to use these reports with artificial intelligence has been to treat them like a library of unorganized books. When a computer program asks a question, the system searches for the most similar-looking words in the text and pulls out a few pages. This method works well for simple questions, but it struggles when the investigation requires connecting dots across different documents. A single hacker group might be called by three different names in three different reports, and the official records that link a specific software flaw to a known attack method are often buried deep inside long paragraphs of text. If the computer cannot see these hidden connections, it cannot solve the puzzle, no matter how smart the artificial intelligence program is.
A team of researchers has discovered that the problem is not the intelligence of the computer, but the way the information is stored. They built a new system called CTIFoundry, which acts as a specialized framework for organizing these security reports before the computer ever tries to read them. Instead of leaving the data as a jumble of text, the system first builds a clear map of the facts. It takes the official records from four major security databases and connects them with precise lines, showing exactly which software weakness leads to which attack pattern. It then reads the messy vendor reports and finds the specific sentences that mention these known groups or flaws, tagging them with their exact location in the original text. This process turns a flat pile of documents into a structured network where every piece of information knows its neighbors and its source.
The researchers tested this new system by giving the same artificial intelligence agent two different ways to access the same set of security data. In one scenario, the agent used the old, unorganized method. In the other, it used the new, structured map. The results were striking. When the agent used the new system, its ability to answer questions correctly jumped significantly, even when the researchers used a smaller, less powerful computer model. In fact, the smaller model using the new map performed better than the most powerful model using the old, unorganized pile of text. The improvement was not because the computer was searching harder or reading more pages. On the contrary, the agent using the new system found the right answers while making roughly half as many attempts to look up information. It simply had a better path to follow.
The study also revealed why the old method failed so often. Without the pre-built map, the computer would often get lost in the text, searching for similar words that led to dead ends or confusing results. It would miss the official links that experts had already written down because those links were hidden inside long sentences. The new system forces the computer to follow the official connections first, treating the text as a secondary source of detail rather than the primary guide. This approach proved so effective that the researchers found the structure of the data was more important than the raw power of the computer model itself. A small model with a clear map outperformed a giant model with a messy one.
To ensure the system was trustworthy, the researchers built it with strict rules that prevented it from inventing any facts. Every connection in the map had to come from an official record, and every piece of text it pulled from a report had to point back to the exact spot in the original document where it was found. This meant the computer could not make things up or guess at the answers. The system also included a set of simple instructions, or habits, for the computer to follow, such as checking the official map before searching for similar words. These habits, combined with the clear map, created a super-additive effect, meaning the two parts worked better together than they would have alone.
The cost of building this new system was surprisingly low, requiring only a small amount of computing power to process the initial data. Once built, the system allowed the computer to solve complex investigations for about two cents per question. The researchers concluded that for fields where information is highly structured and relies on official connections, the biggest bottleneck is not the intelligence of the artificial agent, but the quality of the data it is given. By organizing the data into a form that an agent can naturally understand and traverse, the system made the investigation process faster, cheaper, and far more accurate. The work suggests that in the future, the most effective way to improve artificial intelligence in specialized fields might not be to build smarter brains, but to build better libraries for them to read.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.