Towards smart and adaptive agents for active sensing on edge devices

Imagine a camera that does not just record the world, but actively chooses what to look at, much like a human eye darting from one detail to another to build a complete picture. This is the realm of active sensing, a concept where machines do not passively wait for data but move their sensors to gather the most useful information. For years, the dream has been to put this kind of smart, adaptive behavior onto small, battery-powered devices that sit right where the action happens, such as on a security camera or a robot, without needing to send data to a distant cloud server. The challenge has been that the powerful artificial intelligence systems capable of this usually require massive amounts of computing power and memory, making them impossible to run on these small, local devices.

A team of researchers has now built a system that bridges this gap, creating a smart agent that can see, plan, and move a camera in real time while running on a device no larger than a small computer board. Published in a recent study, this work demonstrates that it is possible to combine two different approaches to artificial intelligence: one that is excellent at recognizing objects in images, and another that is excellent at making decisions under uncertainty. The result is a "saccade agent," named after the quick, jerky eye movements humans make to focus on details, which can control a camera to track people or explore a room autonomously. The system runs entirely on the edge device, meaning it processes everything locally, ensuring fast reactions and keeping data private.

The researchers faced a specific problem: standard deep learning models, which are very good at spotting things like people or cars in a video feed, are rigid. Once they are trained, they struggle to adapt if the environment changes or if they need to decide where to look next to find something new. They rely on huge amounts of data and computing power to learn, which is the opposite of what a small, low-power device needs. To solve this, the team did not try to make the deep learning model bigger or smarter. Instead, they added a second layer of intelligence on top of it, based on a principle called active inference. This approach treats the agent as a scientist that constantly updates its beliefs about the world. It asks, "What do I expect to see?" and "What would surprise me?" If the camera sees nothing new, the agent decides to move to a different spot to learn more. If it sees something interesting, like a person, it focuses on them. This decision-making process is incredibly lightweight, requiring very little memory, which allows it to run alongside the heavier object-detection software.

To test this idea, the team built a physical prototype using an NVIDIA Jetson device, a powerful computer designed for artificial intelligence tasks on the edge. They connected this device to a camera that could pan and tilt, similar to the security cameras found in many buildings. The system was given a simple but powerful job: to watch a scene and decide where to point the lens. The first part of the system, the perception module, used a well-known object detection tool called YOLO to scan the video feed and find people or objects. This tool was optimized to run quickly on the device's hardware. The second part, the planning module, took the list of things the camera saw and used active inference to decide the next move. It calculated which direction the camera should turn to either keep an eye on a person or to scan an empty area to see if anything new appeared.

The results showed that this combination worked remarkably well within the strict limits of the hardware. The entire system, including the software needed to run it, fit into a memory footprint of just 304 megabytes, which is tiny for an artificial intelligence system. In one configuration, the camera could process video and make a decision about where to look next 108 times every second, a speed fast enough to feel instantaneous to a human observer. In another setup, the system prioritized speed over memory, achieving 45 frames of video analysis per second while still making decisions 250 times a second. The researchers found that the time it took to make a decision was so short that the system could easily keep up with the video feed, allowing the camera to smoothly follow moving targets or sweep across a room without lag.

What makes this achievement significant is not just that the camera moved, but how it moved. The agent did not follow a pre-programmed path. Instead, it reacted to the environment in real time. If a person walked into the frame, the camera would lock onto them. If the person moved, the camera would follow. If the person left and the room was empty, the agent would decide to scan the rest of the room to see if anything else was happening. This behavior mimics the way humans naturally explore their surroundings, balancing the need to focus on what is important with the need to stay aware of the whole picture. The researchers demonstrated this with a camera mounted on a small robot, showing that the system could help a machine explore a new environment and gather information efficiently without human intervention.

The study also carefully tested different ways to run the software on the hardware to find the best balance between speed and memory use. They found that using specific tools designed for the device's graphics processor allowed the system to run much faster than using standard methods. However, they also discovered that for the decision-making part of the system, which is very small, trying to run it on the powerful graphics processor actually slowed it down because the overhead of managing the task was greater than the benefit of the extra power. This finding highlights the importance of matching the right software tools to the right hardware, a crucial step for making these systems practical for real-world use.

This work suggests that the future of smart devices does not necessarily require massive, cloud-based supercomputers. By combining a robust ability to see with a lightweight ability to plan, it is possible to create agents that are both smart and efficient. The researchers showed that these systems can operate independently, making split-second decisions to gather information in complex, changing environments. While the current system focuses on controlling a camera, the same principles could be applied to other tasks, such as helping a robot navigate a crowded room or a drone search for a missing person. The study concludes that active inference offers a promising path forward for creating adaptive, resource-efficient machines that can handle the unpredictability of the real world, bringing the power of intelligent sensing directly to the edge.