BatchDAG: LLM-Planned Execution Graphs for Scalable Ad-Hoc Analysis Over Enterprise Data
BatchDAG is a scalable system that leverages LLMs to generate typed directed acyclic graphs for orchestrating parallel, entity-aware batched operations, enabling efficient and accurate ad-hoc analysis over large enterprise datasets while significantly reducing hallucinations and costs compared to sequential agent approaches.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a super-smart robot friend who can read a single book and tell you exactly what happened in it. This robot is great at summarizing one story, but what if you asked it to read 50,000 books and answer a question like, "Did every single character in these books ever talk about money?" If you just handed the robot all the books at once, its brain would explode from too much information. If you asked it to read them one by one, it would take years to finish. This is the problem facing modern "Large Language Models" (LLMs)—the AI brains behind many chatbots. They are brilliant at understanding individual documents, but they struggle when asked to analyze massive, messy collections of data (like thousands of business meetings) all at once. They get confused, lose track of which fact belongs to which meeting, and take forever to finish.
Enter BatchDAG, a new system designed to fix this by acting like a master conductor for a giant orchestra. Instead of letting the AI robot try to do everything itself, BatchDAG uses the AI only to write a detailed "recipe" or "map" of what needs to be done. Then, a super-fast, non-thinking computer engine follows that map, doing the boring, repetitive math and searching instantly, and only calling the "smart" AI when absolutely necessary. The result? The system can answer complex questions about tens of thousands of meetings in under a minute, costing pennies instead of dollars, and it doesn't make up facts.
The Problem: The "One-by-One" Trap
Think of trying to find a specific conversation in a library of 50,000 meeting transcripts. If you use a standard AI agent (like a ReAct bot), it acts like a person walking through the library, picking up one book, reading it, putting it down, picking up the next, and so on.
- The Brain Overload: If you try to feed the AI all the books at once, it hits a "context limit"—its memory is too full to hold everything.
- The Attribution Loss: If the AI searches for "money" across all books, it might tell you, "Yes, money was mentioned!" but it can't tell you which meeting said it. It loses the connection between the fact and the source.
- The Time Sink: Doing this one by one is painfully slow. If you have 3,000 deals to check, and each takes a few seconds, you're waiting hours.
The Solution: The "Master Planner" and the "Assembly Line"
BatchDAG changes the game by splitting the job into two distinct roles: the Planner and the Worker.
1. The Planner (The AI Architect)
When you ask a question like, "Did our sales team ask good questions in every meeting?", the AI doesn't start reading yet. Instead, it acts like an architect drawing a blueprint. It creates a Directed Acyclic Graph (DAG).
- Simple translation: Imagine a flowchart made of Lego blocks. Each block is a specific task: "Search the database," "Filter the results," "Group by meeting," or "Ask the AI to analyze."
- The AI writes this map in a strict, structured format (JSON), not in messy sentences. This ensures the computer knows exactly what to do next without getting confused.
2. The Worker (The Deterministic Engine)
Once the map is drawn, a super-fast computer engine takes over. It doesn't need to be "smart"; it just needs to be fast and obedient.
- Parallel Waves: Instead of doing tasks one by one, the engine looks at the map and sees which blocks can be done at the same time. It runs them in "waves." Imagine a stadium wave: everyone in section A stands up at once, then section B, then section C. This makes the process incredibly fast.
- Zero-LLM Steps: Most of the blocks on the map (like searching a database or sorting numbers) don't need the AI at all. They are done by regular code, which is free and instant. The AI is only called when the map says, "Now, use your brain to analyze this specific group."
The Secret Sauce: "Entity-Aware Batching"
This is the paper's biggest trick. Imagine you have 121 meetings, and each meeting has 48 pages of text.
- The Old Way (Row-Level): If you feed the AI 5 pages at a time, you might send it the first 5 pages of Meeting A, then the next 5 pages of Meeting A, and so on. You end up sending the AI the same meeting 10 times, but each time it only sees a tiny, confusing fragment. This is wasteful and leads to bad guesses.
- The BatchDAG Way (Entity-Aware): The system groups the pages by Meeting ID first. It says, "Here are 5 complete meetings, with all their pages included." It sends the AI 5 full stories instead of 100 tiny fragments.
- The Result: This simple grouping reduced the number of times the AI had to be called by 47 times. It's like sending a detective to investigate 5 whole houses instead of 235 different rooms scattered across the city.
What They Found
The researchers tested this system on real-world data from Brevian.ai, which includes over 50,000 meetings and 3,000 business deals.
- Speed: They analyzed 3,000 deals in under 60 seconds. A standard AI agent would have taken about 100 minutes to do the same job.
- Cost: Because the system avoids calling the expensive AI brain for every single step, the cost dropped dramatically. A simple database query cost just $0.02. Even a complex analysis of thousands of meetings cost only about $0.24 per query.
- Accuracy: The system was just as good as a human expert designing a custom workflow (scoring 3.74 out of 5 vs. 3.25 for a human-designed pipeline) and significantly better than a standard AI agent.
- Truthfulness: Most importantly, BatchDAG was much better at proving its answers. It could point to the exact transcript evidence 77% of the time, compared to only 46–60% for other methods. This is crucial for businesses that need to know why an AI made a claim.
Why This Matters
The paper suggests that for big data problems, we shouldn't just make the AI "smarter." Instead, we should stop letting the AI do the heavy lifting. The best approach is to let the AI plan the job, and then use a fast, boring, reliable machine to do the job.
By using structured data (like neat rows in a spreadsheet) instead of messy summaries between steps, the system also reduced "hallucinations" (made-up facts) by 27%. The authors found that when the AI summarizes things in plain English between steps, it tends to lose track of the details. But when it passes neat, structured data, the facts stay locked in place.
In short, BatchDAG isn't about making the AI a genius; it's about making the AI a great manager who knows exactly when to call in the experts and when to let the assembly line do the work. It turns a slow, expensive, and error-prone process into a fast, cheap, and reliable one.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.