Budget-Constrained Embodied Perception: Four Resource Walls and a Pre-Registered Evaluation of Access-Structured Perception on Open Models at less than 31B

Imagine a robot assistant living in your home, tasked with helping you find a specific mug you used this morning. It has been walking around your house for hours, recording a continuous stream of video. To answer your question, it cannot simply look at the last few seconds of footage; it must search through hundreds of minutes of recorded history. However, the robot's brain has a strict limit: it can only process a tiny, fixed amount of information at the exact moment it needs to make a decision. It cannot store the entire video in its active memory, nor can it afford to re-watch the whole day every time you ask a question. This is the fundamental bottleneck of embodied intelligence: the gap between an endless stream of observations and a finite budget for thinking.

For years, researchers hoped that simply making artificial intelligence models larger would solve this problem. The prevailing idea was that if a model was big enough, it would naturally learn to remember everything it saw. But recent experiments have shown that even the most advanced models struggle when faced with long, continuous streams of video. They can recognize a cat in a single photo, but they often fail to track where that cat went an hour ago. The problem is not that the models lack the ability to see; it is that they lack a strategy for knowing what to look at when the clock is ticking and the memory is full.

A team of researchers set out to test a different approach. Instead of trying to build a bigger brain, they asked whether the way the robot accesses its memories matters more than the size of the brain itself. They proposed that a successful agent needs two distinct tools working together: a compressed summary of what has happened (like a running diary) and a precise, searchable archive of every moment (like a library of photos). Crucially, the agent must be able to choose, based on the specific question asked, how much of its budget to spend on reading the diary versus searching the library. They built a system to test this, wrapping existing robot brains in a new layer of logic that managed these resources without requiring any retraining.

The researchers tested their system on a synthetic environment designed to mimic a long walk through a building, where the robot had to answer questions about objects it had seen hours earlier. They pitted their new system against several standard methods, including models that simply looked at a random selection of frames, models that tried to summarize everything into a short text, and models that searched through a library without keeping a running summary. They ran these tests on seven different open-source models, ranging from small to large, all under the same strict limit of 4,096 "tokens" (units of information) per decision.

The results were striking, but nuanced. The new system, which intelligently routed the robot's attention, vastly outperformed every other method that did not condition its access on the specific question asked. When asked to find a specific object hidden in a stream of 300 frames (the episodic retrieval task), the new system succeeded 75–94% of the time, while the best alternative methods managed less than 20%. The researchers found that the ability to ask the right question and then fetch the right frame was the deciding factor. In fact, giving a standard model four times more memory to work with did not help it nearly as much as giving the new system a smart way to use its limited memory. The smart system with a small budget beat the dumb system with a huge budget.

However, the story did not end with a perfect victory. The researchers had designed their system with three parts: the summary, the archive, and the smart router that decided how to use them. When they tested each part individually, they discovered a surprising flaw. The smart router and the archive were the heroes; they did the heavy lifting. But the summary component, which was supposed to keep a running count of events, actually made the system worse. When they removed the summary, the system's performance on complex tasks improved significantly. In fact, the full system did not outperform the archive-only baseline on any of the seven models tested. The reason was that the robot's brain was not good enough at maintaining a perfect running tally over hundreds of steps; the small errors in the summary piled up and confused the system. A better-performing system that relied on an offline text index instead of a live summary actually beat the researchers' own design.

This failure was not a defeat for the main idea, but a crucial refinement. The study proved that under a strict time and memory limit, the structure of access is more important than the size of the model or the amount of memory available. The robot does not need to see everything; it needs to know exactly where to look. The researchers concluded that the future of embodied agents lies not in building bigger brains, but in building smarter ways to access information. The most effective strategy is to spend the budget on finding the right piece of evidence, rather than trying to hold the whole world in memory at once. While their specific design for the running summary failed, the core principle held firm: in a world of endless data and limited time, knowing what to ignore is just as important as knowing what to remember.