Monitoring Agentic Systems Before They're Reliable
This paper proposes a maturity-staged monitoring framework for agentic systems that prioritizes detecting structural defects over task-level errors by decomposing evaluation into quality, suitability, and efficiency dimensions across within-run, cross-run, and structural scopes, using variance and FMEA-based triage to guide human attention toward the most critical integration gaps.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you've just built a new, high-tech robot assistant to help your accounting team sort through thousands of receipts, invoices, and bank statements. You're excited, but you know the robot isn't perfect yet. It's a "work in progress."
This paper is about how to watch over this robot before it's smart enough to catch its own mistakes. The authors argue that if you try to check if the robot is doing the math right too early, you'll miss the real problem: the robot's brain isn't even connected to its hands correctly.
Here is the breakdown of their approach using simple analogies:
1. The Problem: The "Broken Assembly Line"
The authors say that when these AI systems are first launched, they aren't failing because they are bad at math or reading. They are failing because of structural defects.
- The Analogy: Imagine a car factory where the conveyor belt is moving, but the robot arm that puts the wheels on is actually holding a hammer instead of a wrench.
- The Reality: The robot might be trying to "read" a bank statement, but because of a wiring error, it's treating it like a restaurant receipt. It will produce a result, but it's the wrong kind of result.
- The Trap: If you just look at the final numbers (the "task-level errors"), you won't see the problem. The robot is so busy using the wrong tool that it can't even tell if the numbers are right or wrong. The "noise" of the broken assembly line drowns out the signal of the actual math errors.
2. The Solution: Three Different "Security Cameras"
Instead of just one camera watching the final product, the authors suggest using three different types of monitors, each looking at a different part of the process. They call this Triangulated Monitoring.
Monitor Type A: The "Within-Run" Camera (The Routine Check)
- What it watches: Does the robot do the same thing every time it processes a single batch of documents?
- What it found: This camera saw that the robot was consistently making the same mistake on every single document. It was like a factory worker who always puts the left shoe on the right foot. It wasn't a random mistake; it was a broken rule.
- The Signal: High volume of errors, but they are all the same (low variance).
Monitor Type B: The "Cross-Run" Camera (The Wild Card Check)
- What it watches: Does the robot behave differently from one batch to the next?
- What it found: Sometimes the robot works fine, and sometimes it completely crashes or skips steps. This is the "stochastic" (random) chaos caused by the broken connections.
- The Signal: Low volume of errors, but they are unpredictable and dangerous (high variance).
Monitor Type C: The "Structural" Camera (The Blueprint Check)
- What it watches: Is the robot even looking at the right documents?
- What it found: The robot was missing entire sections of the paperwork because the "manifest" (the list of what should be there) didn't match what the robot actually received.
- The Signal: A perfect, consistent failure. It happens every single time, exactly the same way.
3. The Filter: The "Traffic Cop" (Triage)
If you have 10,000 alerts from these cameras, a human manager will go crazy trying to read them all. The authors created a Traffic Cop system based on a safety method used in aerospace and medicine (called FMEA).
How it works: The system automatically sorts the alerts into four buckets based on how bad they are:
- L4 (Negligible): Tiny glitches. The system ignores them.
- L3 (Marginal): Annoying but not dangerous. The system tracks them automatically.
- L2 (Critical): The robot is missing big chunks of work or doing things randomly. Human must look at this.
- L1 (Catastrophic): The whole process is broken. Stop everything immediately.
The Result: The system automatically filtered out 97% of the noise (the routine, boring errors) and only sent the top 2% (the dangerous, structural problems) to the human team. This saved the humans from drowning in data and let them focus on fixing the actual broken wires.
4. The Big Lesson: "Fix the Foundation First"
The most important takeaway from the paper is a rule for maturity:
"Deploy monitoring early. The first thing it finds is the most important thing to fix."
- Stage 1 (Now): The system is a baby. It has broken bones (structural defects). Monitoring should just tell you where the bones are broken. Don't worry if it's doing the math right yet; it can't do that until the bones are fixed.
- Stage 2 (Later): Once the bones are fixed, monitoring shifts to checking if the robot is doing the math correctly.
- Stage 3 (Mature): Once the robot is smart, monitoring shifts to tracking how reliable it is over time.
Summary
The paper argues that for new AI systems, you shouldn't try to grade their homework (task errors) immediately. Instead, you should use a smart monitoring system to check if their desk, chair, and pen are set up correctly (structural defects). By using three different types of cameras and a smart traffic cop to filter the alerts, you can find the big, broken parts of the system quickly, fix them, and then start worrying about whether the robot is actually doing the math right.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.