When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMs
This paper introduces Evidence-Carrying Termination (ECT), a novel framework for tool-using LLM agents that mandates typed certificates binding answer claims to valid, replayable trace evidence before completion, demonstrating in controlled experiments that it achieves zero unsafe completions and significantly reduces premature unsupported terminations compared to existing critic-based approaches.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of artificial intelligence, a new kind of software has emerged: the autonomous agent. Unlike a simple calculator that waits for a command, these agents are designed to think, plan, and act. They can browse the web, run code, or query databases to solve complex problems, much like a human researcher gathering information. However, this new capability brings a fundamental dilemma: how does the machine know when to stop? If an agent keeps working forever, it wastes resources and risks compounding errors. If it stops too soon, it might hand over an answer that is incomplete or based on a misunderstanding. For years, engineers have tried to solve this by asking the AI to simply declare itself finished or by having a second AI judge if the answer looks good. But these methods rely on the machine's own confidence, which can be misplaced. The real challenge is not just asking the agent to stop, but proving that the decision to stop is backed by solid, unchangeable proof that every part of the answer was actually found and calculated correctly.
A researcher at the University of California San Diego has tackled this problem with a system called Evidence-Carrying Termination. Instead of letting an agent decide to stop based on a feeling or a simple checklist, this system requires the agent to produce a formal certificate before it can finish a task. Think of this certificate as a receipt that the agent must generate for every single claim it makes in its final answer. The agent cannot just say, "I found the answer." It must show exactly which tool call provided the data, prove that the data was relevant to the specific question asked, and demonstrate that the final number or fact was derived using a strict, unchangeable set of rules. If any part of this proof is missing, or if the math doesn't add up when checked against the original data, the system forces the agent to keep working. It is a gatekeeper that refuses to let the agent leave the room until it has shown its homework in full detail.
To test if this rigorous approach actually works, the researcher created a controlled environment with forty-eight different tasks, ranging from looking up specific facts to performing complex data aggregations. They then introduced eight specific types of errors into these tasks, such as giving the agent the wrong piece of evidence, hiding a tool failure, or asking it to stop before it had found all the necessary information. They pitted their new system against a standard method where a critic AI simply judges the answer. The results were stark. In a test involving nearly three hundred faulty scenarios, the standard system failed to catch the errors and allowed the agent to stop with an incorrect or unsupported answer in two hundred and fifty-two cases. In contrast, the new Evidence-Carrying Termination system allowed zero unsafe completions. It successfully identified every single flaw and forced the agent to continue until the proof was solid.
The researcher did not stop at static tests; they moved on to a more dynamic, closed-loop experiment involving hundreds of simulated journeys where the agent had to recover from mistakes in real time. Here, the goal was to see if the strict proof requirement would cause the agent to give up too easily or fail to finish valid tasks. The new system again outperformed the standard approach. It prevented the agent from stopping prematurely in sixty-six critical instances where the older system failed forty times. Crucially, the new system did not become so cautious that it stopped useful work; it completed supported tasks at a rate that was statistically equal to the older system, proving that demanding proof does not mean sacrificing progress. When the agent got stuck, the system was able to guide it through a recovery process in eighteen out of sixty-six attempts, with seventeen of those eventually leading to a successful, verified conclusion.
This work does not claim to make the agent infallible or to guarantee that the answers are true in the real world. The system only verifies that the agent followed its own rules and that the final answer matches the evidence it collected. It is a check on the process, not a guarantee of external truth. However, the findings suggest a clear path forward for safer AI. By separating the act of proposing an answer from the act of proving it, and by requiring a deterministic replay of every step, the researcher has shown that it is possible to build agents that know exactly when they are done. The study concludes that an agent should not stop merely because it feels finished or because its answer sounds plausible. It should stop only when it can present a complete, unbroken chain of evidence that validates every claim it makes.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.