Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems
This paper introduces Borg, an agent-driven framework that utilizes a composable dynamic graph and a coding agent to automatically evolve a unified LLM serving simulator, thereby significantly reducing development time and improving throughput accuracy compared to existing tools.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of artificial intelligence, large language models have become powerful tools capable of writing code, solving complex problems, and holding conversations. However, running these models is not as simple as pressing a button; it requires a sophisticated system to manage the flow of information, decide which requests to process first, and allocate computing power efficiently. This field, known as serving, is the engine room that keeps these intelligent systems running smoothly for millions of users. Because the technology evolves at a breakneck pace, with new models and methods appearing constantly, engineers cannot always wait to build physical machines to test their ideas. Instead, they rely on computer simulations—virtual laboratories where they can predict how a new system design will perform before spending millions of dollars on hardware.
For years, these virtual laboratories have been built like rigid, pre-fabricated structures. Researchers would manually code every possible scenario into a single, fixed pipeline. This worked well when the technology changed slowly, but the landscape of artificial intelligence has shifted dramatically. New ways of using these models have emerged, such as "agentic" workflows where a single request triggers a chain of multiple decisions and tool calls, or "disaggregated" systems where different parts of the processing happen on separate machines. These modern behaviors do not fit into the old, rigid pipelines. Every time a new feature appeared, engineers had to tear apart the simulator and rebuild it from the ground up, a slow and error-prone process that left the virtual models falling behind the real systems they were meant to represent.
A team of researchers at KAIST in South Korea has introduced a new approach to solve this problem, called Simthesizer. Rather than building a new simulator for every new feature, they created a flexible framework that allows a computer program to write the simulator itself. At the heart of this system is a dynamic map of the workflow, where every step of the process is a node that can be added, removed, or rearranged on the fly. This map is not fixed; it is a living structure that can grow to accommodate new methods without collapsing the entire system. To make this work, the researchers employed a coding agent, a specialized artificial intelligence assistant, to act as the builder. When a researcher asks for a new feature, the agent reads the request, understands the rules of the simulation, and writes the necessary code to integrate it into the map.
The process is guided by strict guardrails to ensure the agent does not make mistakes. The system requires the agent to first write a clear specification of what the new feature should do, then map out exactly where it fits into the existing structure, and finally write the code. Before the new feature is accepted, the system checks its work against real-world data or established evidence to ensure it behaves correctly. This method allows the simulator to evolve as quickly as the technology it models. In their tests, the researchers asked the agent to add three complex features to the simulator: a method to compress memory usage, a technique to speed up text generation by guessing the next words, and support for a new type of model architecture. The agent successfully built these extensions, and the resulting simulations were remarkably accurate.
When the researchers compared the performance of these new simulations against real systems running on actual hardware, the results were striking. The extensions built using this new framework matched the real-world throughput with an average error of just 2.51 percent. In contrast, when the same coding agent tried to add the same features to existing simulators using their old, rigid methods, the error rate jumped to 6.03 percent. The new system was not only more accurate but also significantly faster. It ran simulations up to 284.96 times faster than one leading competitor and 23.19 times faster than another. This speed comes from the fact that the new system does not need to re-run the entire simulation engine for every small change; it simply updates the specific part of the map that needs to change.
The researchers demonstrated that this approach works with different types of coding agents, not just the one they initially used, suggesting that the method is robust and generalizable. By combining a flexible, composable infrastructure with an automated builder, Simthesizer bridges the gap between the rapid evolution of artificial intelligence and the tools used to study it. It offers a way to keep pace with a field that changes faster than human engineers can manually update their tools, ensuring that the virtual models used to design the future of computing remain accurate, reliable, and ready for whatever comes next.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.