🤖 AI

Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt Engineering Quality Assurance

This paper presents a case study of an iterative, agent-driven auditing process applied to the AEGIS multi-agent system, which utilized nine sequential rounds of LLM-based inspections to identify 51 prompt-specification defects, establish a new defect taxonomy, and demonstrate non-monotonic convergence in a production environment.

Original authors: Elias Calboreanu

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Elias Calboreanu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Orchestra" Problem

Imagine a massive orchestra called AEGIS. This isn't a normal orchestra with violins and drums; it's a team of seven AI "musicians" (agents) working together to manage a huge list of tasks (like a to-do list for a company).

Each musician has their own sheet music (a document called PROMPT.md) that tells them exactly what to play, when to play it, and how to talk to the other musicians. There is also one master rulebook (the Ticket Contract) that everyone agrees on.

The problem? These sheet music documents are written in natural language (like English), not computer code. They are long (about 7,150 lines total), they change often, and they depend on each other. If the first musician changes a note in their sheet, the second musician might get confused because their sheet still says the old note.

This paper is a story about how the team tried to fix these sheet music documents to make sure the orchestra doesn't play a disaster.

The Experiment: The "Self-Inspecting" Orchestra

Usually, when you write a long document, you might ask a friend to read it once to check for typos. But in this case, the authors realized that a single quick read wouldn't catch the tricky mistakes where one musician's instructions clash with another's.

So, they set up a repeated inspection process:

The Inspector: They used an AI (a "Claude" sub-agent) to act as the auditor.
The Checklist: The auditor had a specific checklist to look for things like: "Do the file names match?" "Are the rules for the 7th musician included?" "Is the contact info for the boss (Jira) up to date?"
The Loop: The auditor would find mistakes, the team would fix them, and then the auditor would come back and check again. They did this nine times in a row.

What They Found (The "Defects")

After nine rounds of checking and fixing, they found 51 specific mistakes in the sheet music. These weren't computer viruses or code crashes; they were "logic glitches" in the instructions.

Here are the types of mistakes they found, explained with analogies:

Stale References (The "Old Phone Book"): 23% of the errors were like having a phone number in the instructions that no longer worked because the person had moved. (e.g., pointing to a task ticket that had been deleted).
Version Drift (The "Outdated Map"): Some instructions said "We have 7 musicians," but the document was written when there were only 6.
Cross-Lane Mismatches (The "Wrong Handoff"): This was the most dangerous type. Musician #3 was told to hand a note to Musician #4 labeled "Priority Score," but Musician #4 was expecting a note labeled "Fix Priority." If they had played, Musician #4 would have ignored the note, and the task would have failed silently.
Missing Coverage (The "New Instrument"): When they added a new musician (Lane 7) to the orchestra, the old rulebooks didn't mention how to talk to them.

The Surprising Result: It Got "Worse" Before It Got Better

You might expect that after Round 1, the number of mistakes would go down steadily. But it didn't.

Round 1: Found 15 mistakes.
Round 2: Found 8 mistakes.
Round 3: Found 12 mistakes (more than Round 2!).

Why? The authors explain this with a "Peeling an Onion" analogy.
In the first few rounds, they fixed the obvious, surface-level errors (like typos or missing names). But by fixing those, they accidentally revealed deeper, hidden problems that were previously masked. It's like fixing a leak in a pipe only to realize the water pressure was actually causing a crack in the wall behind it. The "scope" of the audit got bigger and smarter as they went, finding harder-to-see problems.

The Key Takeaways

One Look Isn't Enough: If you just read one document at a time, you will miss the problems where two documents don't agree. You have to look at the whole system together.
Iterative Auditing Works: You can't just fix it once and be done. You have to check, fix, and check again. In this case, it took nine rounds to get to zero mistakes.
AI Auditing AI: The same family of AI models wrote the instructions and then audited them. The paper admits this is a bit risky (like asking a student to grade their own homework), but it worked well enough to find these 51 specific errors.
The "Silent Killer": The most dangerous errors were the ones where two parts of the system didn't match up. These wouldn't cause a loud crash; they would just cause the work to stop quietly, which is harder to detect.

What This Paper Does NOT Say

It does not say this method works for every AI system in the world. It only tested this one specific system (AEGIS).
It does not claim that AI auditors are perfect. They used the same AI family to write and check, which might have missed things a human or a different AI would have seen.
It does not promise that you can fix all AI problems in one go. The main lesson is that you need to keep checking and re-checking.

In a Nutshell

This paper is a case study showing that when you have a complex team of AI agents, their instruction manuals get messy and contradictory very quickly. To fix them, you can't just do a one-time check. You need a repeated, evolving inspection process where the auditor gets smarter with every round, peeling back layers of the onion until the instructions are perfectly aligned.