From Understanding to Action: A Survey on Closed-Loop Multimodal Agents
This survey proposes an evidence-centered framework for closed-loop multimodal agents, organizing 105 papers around five evidence operations and introducing an O-P-G-C evaluation lens to address the limitations of current outcome-only benchmarks by emphasizing process, grounding, and cost-risk analysis.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a computer that does not just look at a picture and describe it, but actually uses what it sees to do something in the real world. For years, artificial intelligence has been getting very good at understanding images, text, and sound, often acting like a brilliant librarian who can find the right book but cannot check it out. Now, these systems are evolving into agents that can open a web browser, click buttons, move a robot arm, or search for information, all while watching the results of their own actions. This shift turns a static observer into a participant that can change its environment. However, when these agents make a mistake, it is often hard to tell why. Did the computer look at the wrong part of the screen? Did it misunderstand a tool it was using? Or did it simply guess the right answer for the wrong reasons?
A new survey from researchers at the University of Auckland, the University of Cambridge, and Adelaide University takes a fresh look at how these multimodal agents work. Instead of just listing what these systems can do, the authors propose a new way to understand them: by following the trail of evidence. They argue that for an agent to be truly reliable, every step it takes must be backed by a clear, inspectable piece of proof. The researchers reviewed 105 recent studies to map out how these agents gather information, organize it, use outside tools, act on the world, and learn from their mistakes. They found that while current systems are getting better at finishing tasks, we still lack the tools to see clearly how they are doing it, often leaving us with a correct answer that was reached through a shaky or dangerous path.
The core idea of this work is that an agent's success should not be measured only by whether it finished the job. The authors suggest we must look at the "evidence" the agent uses at every stage. Think of evidence as a visible note or a highlighted section that connects what the agent sees to what it decides to do next. If an agent clicks a button to buy a ticket, the evidence is the specific part of the screen it looked at to find that button. If that evidence is missing, blurry, or misinterpreted, the action is risky, even if the ticket eventually gets bought. The researchers organized the entire field of study around five key ways agents handle this evidence. First, they must actively choose what to look at, rather than just staring at everything at once. Second, they need a mental workspace to hold and organize those observations while they think. Third, they must know when to reach out to outside sources, like a search engine or a calculator, to fill in gaps. Fourth, they have to take actions that change the world, like moving a cursor or a robot, and watch how the world changes back. Finally, they must use the results of those actions to improve their future behavior.
The survey reveals a significant gap in how we currently test these systems. Most existing benchmarks, which are like standardized tests for artificial intelligence, focus almost entirely on the final result: did the agent succeed or fail? The researchers found that these tests rarely check the process. An agent might complete a complex task, but it could have done so by guessing, by ignoring crucial details, or by taking unnecessary risks that could have caused damage. The authors introduce a new framework for evaluation that looks at four distinct things: the outcome, the process, the grounding, and the cost or risk. Outcome asks if the job is done. Process asks if the steps taken were logical and necessary. Grounding checks if the agent actually looked at the right thing before acting. Cost-risk measures how much time, money, or danger was involved in getting the result.
When the researchers applied this framework to the 105 papers they reviewed, a clear pattern emerged. The field is very good at measuring whether an agent finishes a task, but it is much weaker at checking if the agent used the right evidence to get there. For example, in tasks involving computer screens or robots, many systems can complete a goal, but the tests do not always show if the agent clicked the right button or if it moved the robot safely. The authors point out that without checking the evidence trail, we cannot know if an agent is truly intelligent or just lucky. They also found that when agents fail, it is often hard to tell where the error started. A mistake in looking at the screen can lead to a bad plan, which leads to a wrong action, which then changes the environment in a way that makes the next step impossible.
The paper concludes that for these agents to be trusted in real-world situations, we need better ways to record and inspect their decision-making. The researchers suggest that future tests should require agents to show their work, much like a student showing their math steps. This means recording exactly which parts of an image were looked at, which tools were used, and how the environment changed after each action. They also warn that as these systems become more powerful, the risks of getting it wrong increase. An agent that makes a mistake in a video game is harmless, but an agent that makes a mistake while managing a bank account or controlling a physical robot could cause real harm. Therefore, the ability to trace the evidence and understand the cost of every action is not just a technical detail; it is a safety requirement.
This survey does not claim to have solved the problem of making perfect agents. Instead, it offers a new map for the journey ahead. By shifting the focus from the final answer to the evidence that supports it, the researchers hope to guide the development of systems that are not only capable but also transparent and safe. They argue that the next generation of artificial intelligence should be judged not just on what it achieves, but on how it achieves it. If we can build systems that leave a clear, honest trail of evidence for every decision they make, we will be better equipped to understand their strengths, fix their weaknesses, and deploy them in the complex, high-stakes environments where they are needed most.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.