Where Cognition Lives: Dissecting Emergent from Computed Function in a Minimal Complete Cognitive Architecture
This paper demonstrates that while core cognitive competence can emerge from gradient descent in a minimal architecture, critical meta-cognitive functions like adaptive halting and value-based resource allocation do not emerge spontaneously and must be explicitly computed to achieve significant performance gains, with experimental results showing that explicit value allocation yields up to seven times the benefit of smooth estimates in high-stakes scenarios.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the quest to build machines that think, scientists have long debated where the spark of intelligence actually comes from. One school of thought believes that if you build a large enough system and let it learn from vast amounts of data, complex abilities like reasoning and decision-making will simply appear on their own, much like a child learning to walk without being taught the mechanics of balance. The opposing view suggests that thinking is a carefully engineered process, requiring distinct, dedicated parts for perception, memory, and judgment, much like a factory with specialized assembly lines. For years, this has been a philosophical tug-of-war, but a new study moves the conversation from theory to the laboratory floor. The researchers set out to build a small, complete thinking machine and then rigorously test every single part to see if it learned its job through experience or if it had to be explicitly programmed to do it. Their goal was not just to see if the machine could solve problems, but to understand exactly how it decided when to stop thinking, how it judged the importance of a task, and whether it could learn to manage its own mental energy.
The team constructed a minimal but complete cognitive system, a digital brain composed of a few key components working together. At its heart was a reasoner, a module that could work through a problem step-by-step. Surrounding this were tools to help it decide how long to think, a control system to manage its energy, and a value module to judge how important a task was. They trained this system on a series of logical puzzles, specifically tasks involving the rearrangement of symbols, which are difficult for standard computer models to solve without getting stuck. The researchers then subjected every part of this machine to a battery of tests, asking a simple question for each: did this ability emerge naturally from the training process, or did it have to be calculated by a specific, pre-designed mechanism? The answers they found were sharp and often surprising, revealing that while the ability to solve problems can emerge, the ability to decide when to stop and how to allocate effort cannot.
The most significant discovery concerned how the machine decided when to stop thinking. In many modern artificial intelligence systems, the model is allowed to run for as long as it needs, stopping only when it feels confident. The researchers found that their system did learn to stop, but this ability was an illusion of measurement rather than a true skill. When they forced the machine to stop at the exact same point the "smart" version chose, the performance was identical. The apparent advantage of the self-stopping machine came entirely from a quirk in how its final answer was read out, not from any deeper understanding of the task. Once the researchers corrected for this reading method, the value of the machine watching itself think dropped to almost nothing. Knowing exactly which specific puzzle needed more time than another turned out to be worth only a tiny fraction of a percent in performance. The machine's internal monitor was not actually making better decisions; it was just a byproduct of how the system was built.
This led to a clearer picture of where true decision-making lives. The researchers tested whether the machine could learn to prioritize important tasks over less important ones by simply training it to care about the outcome. They found that the training process actually made the machine worse at this. When the system was taught to weigh the consequences of its answers, it became confused and less accurate. The ability to decide how much effort to spend on a task did not emerge from the learning process at all. Instead, it had to be computed by an explicit, separate calculator that looked at the problem and decided the effort level before the machine even started working. This "computed decision" was far more effective than any attempt to let the machine learn the value of a task on its own. The machine's internal state, shaped by training, naturally shed the very information needed to make these value-based decisions, leaving the explicit calculator as the only way to get the job done.
The study also examined how these findings hold up when applied to the massive language models that are popular today. The researchers tested a large, frozen language model—one that cannot be retrained—using similar methods to see if it could govern its own thinking time. They found that the common strategy of generating many different answers and voting on the best one had very little room for improvement. The model tended to agree with itself even when it was wrong, meaning that asking it to think longer or vote more often rarely helped. Just as with the smaller machine, the ability to watch itself think and decide when to stop was nearly worthless. The researchers concluded that before we can try to govern how these large models think, we must first measure whether the tools we have to control them actually work. In many cases, the lever we try to pull simply does not move the machine in a meaningful way.
The final chapter of the research explored a specific type of difficult problem where the cost of thinking is hidden until the very end, like walking a path where you only know the distance when you arrive. In these "cliff" scenarios, the researchers predicted that the value of deciding in advance would skyrocket. Their prediction held true: in these situations, knowing the value of a task before starting became incredibly important, offering a massive boost in performance compared to just knowing the difficulty. However, this boost did not come because the machine suddenly understood the problem better; it came because the potential reward for getting it right was so much larger. The study showed that the size of the problem and the structure of the information are separate things. The machine did not need to learn a new way of thinking; it simply needed a clear, pre-calculated plan to handle the high stakes.
Ultimately, the map of this research shows where cognition does not live. It does not live in the machine's ability to watch itself think, nor does it live in the training process teaching it to value a task. Instead, the ability to manage effort and make second-order decisions about what matters must be built in as a separate, explicit calculation. The machine learns to solve the puzzle, but it must be told, by a separate mechanism, how hard to try. The researchers found that the most reusable part of their work was not a new type of intelligence, but a rigorous method for measuring what is real and what is an illusion. By proving that many assumed capabilities are actually artifacts of how we measure them, they have provided a clearer, more honest foundation for building machines that truly think.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.