ProcAgent: An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loop
The paper introduces ProcAgent, a fully on-device agentic framework running on NVIDIA Jetson hardware that combines continuous perception with on-demand visual reasoning and human-in-the-loop interaction to provide real-time, privacy-preserving guidance for complex procedural tasks like furniture assembly.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to build a complex piece of furniture, like a bookshelf, but the instructions are a confusing mess of tiny diagrams. You need someone to watch your hands, tell you if you're holding a screw the wrong way, and remind you that you can't put the top shelf on before the legs are attached. In the world of science, this is called "procedural guidance." For years, computers have been getting better at this, but they usually rely on a "brain" in the cloud—a giant server far away that processes video and sends answers back. This is like having a personal assistant who lives in another country; it works, but it's slow, and you have to send them a video of your messy room, which feels a bit like a privacy invasion.
The big question researchers are asking is: Can we build a smart assistant that lives entirely on a small device right next to you, like a smart speaker? This device needs to be fast enough to catch your mistakes in real-time, smart enough to understand what you are doing, and private enough to never send your video to the internet. The challenge is that these "smart" computers usually need massive power and memory, which makes them too hot and too slow for a small box on a table. This paper explores how to build a "local brain" that can guide you through a task without needing the cloud, balancing speed, smarts, and privacy.
Meet ProcAgent: The Local-Only Furniture Assembly Buddy
Meet ProcAgent, a new kind of digital assistant designed to help you build things like IKEA furniture. Imagine a robot butler that lives entirely inside a small computer box (specifically, an NVIDIA Jetson AGX Orin) sitting on your floor. Unlike other assistants that stream your video to a giant server in the sky, ProcAgent keeps everything right there in your living room. It watches you, thinks about what you're doing, and talks to you, all without ever connecting to the internet.
The authors built this system because they realized that trying to assemble furniture is a mental marathon. You have to read instructions, remember where you are, look at parts, and fix mistakes, all while holding heavy pieces of wood. Old-school manuals can't help if you get stuck, and video overlays can't tell you if you made a mistake. ProcAgent is designed to be a "just-in-time" coach that only speaks up when it's actually needed.
The "Propose-and-Verify" Dance
The secret sauce of ProcAgent is a clever two-step dance called "propose-and-verify." Imagine you are playing a video game where a fast, cheap character (the Proposer) runs around the screen guessing what you are doing. This character is quick and cheap to run, so it can watch your every move. It might say, "Hey, I think you just picked up a leg!"
But the Proposer isn't perfect; it might get confused by shadows or your hand blocking the view. So, it doesn't just shout its guess to the whole world. Instead, it whispers to a super-smart, but slower, character (the Visual Verifier). The Verifier is like a strict referee. It only wakes up when the Proposer says, "I think a big change happened!" The Verifier then takes a closer look to confirm: "Yes, you really did pick up the leg," or "No, you were just waving your hand."
This is a huge deal because it means the system doesn't have to run the expensive, slow "referee" on every single video frame. It saves battery, keeps the computer from getting hot, and makes the system fast enough to talk to you in real-time.
The "Rulebook" and the "Human Loop"
ProcAgent also carries a digital Rulebook (a Finite State Task Model). This isn't just a list of steps; it's a map of what is allowed to happen next. If you try to put the top shelf on before the legs are attached, the Rulebook says, "Whoa, that's illegal!"
But here is the most human part: when the computer gets confused or thinks you made a mistake, it doesn't just force you to stop. It asks you! This is called Human-in-the-Loop. If the system thinks you skipped a step, it might say, "Hey, did you remember to flip the table?" If you say, "No, I didn't," it helps you fix it. If you say, "Yes, I did, you're wrong," it trusts you and keeps going. This makes the system feel like a helpful partner rather than a bossy robot.
What They Found
The researchers tested ProcAgent by having 10 people build IKEA coffee tables while the system watched. Here is what happened:
- It's Fast Enough: Even though it's doing complex thinking on a small device, it answered simple text questions in about 2 seconds. When it needed to look at the table to answer a question (like "Is this leg aligned?"), it took about 8 seconds. That's fast enough to keep the conversation flowing.
- It's Private: Because it runs entirely on the device, it never sent a single frame of video to the cloud. In the user study, people felt much more comfortable and trusted the system because they knew their privacy was safe.
- It Works Better Than Just "Looking": When they compared ProcAgent to a system that just looked at the video and guessed (without the Rulebook or the "Propose-and-Verify" dance), ProcAgent was much better at giving correct and helpful advice. The "Rulebook" and the "Human Loop" were the keys to getting the sequence right.
- The Trade-off: The system does get a little warm (around 55.5°C) and uses a fair amount of power, but it stays within safe limits and doesn't shut down.
The Bottom Line
The paper suggests that you don't need a giant cloud server to have a smart, helpful assistant for physical tasks. By using a smart strategy where a "fast guesser" checks with a "slow referee" only when necessary, and by letting the human have the final say when things get confusing, you can build a system that is private, fast, and actually helpful.
The authors found that this approach works well for furniture assembly, but they admit it's still a work in progress. They aren't claiming it's perfect for every task in the world yet, but they have shown that it is possible to have a smart, agentic assistant that lives entirely on your edge device, keeping your data safe and your furniture standing straight.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.