It's a matter of timescale: non-linear utility in successor features and multi-objective planning and learning

In the world of artificial intelligence, machines are often taught to make decisions by chasing a single goal, like a robot vacuum cleaning a floor or a program winning a game of chess. But the real world is rarely that simple. Often, an agent must juggle several competing goals at once, such as a self-driving car trying to reach a destination quickly while also staying safe and saving energy. When these goals conflict, the machine needs a way to decide which one matters more. Furthermore, the consequences of a decision can play out over different stretches of time. A small mistake might cause an immediate problem, while a series of small mistakes might only add up to a disaster months later. For years, researchers have developed different mathematical tools to handle these situations, but they have generally treated each time frame as a separate problem, choosing one method for immediate risks and another for long-term averages.

A new study challenges this separation, arguing that real-world decisions often require an agent to worry about immediate, medium-term, and long-term consequences all at the same time. The researchers, working across institutions in Belgium, Brazil, and Australia, demonstrate that relying on just one of these traditional methods can lead to dangerous or inefficient outcomes when multiple time scales are involved. By simulating a scenario involving workers exposed to toxic substances, they show that a strategy designed to protect against long-term average exposure might accidentally sacrifice a worker to a short-term spike in danger, while a strategy focused on immediate safety might ignore a slow-building health crisis. Their work suggests that to build truly safe and effective artificial intelligence, we need new ways of thinking that can balance these different clocks simultaneously.

The core of the problem lies in how we measure "happiness" or "utility" for an agent. In many standard AI systems, the machine calculates the average reward it expects to get over a long period and tries to maximize that number. This works well for a factory that processes thousands of video files a day; if the average power usage and speed are good, the system is considered successful, even if one specific file took too long to process. However, this approach fails when a single bad outcome is catastrophic. Consider a patient undergoing a medical treatment. That patient will only experience the treatment once. They do not care about the average outcome of thousands of other patients; they care about their own specific result. If the treatment has a non-linear risk—meaning a small increase in dosage causes a massive jump in danger—the average calculation hides the true risk. The patient needs a system that evaluates the specific outcome of their single journey, not the average of many journeys.

There is a third way to look at time, which focuses on the very next step. Imagine a car hitting a wall. The danger of injury depends non-linearly on the force of the impact. A hard hit is far more dangerous than a soft one, and the risk does not simply add up over time; a hundred soft bumps do not equal one hard crash. In this view, the system must evaluate the risk of every single moment as it happens, applying a penalty immediately if a dangerous combination of factors occurs. For decades, researchers have treated these three perspectives—long-term averages, single-journey outcomes, and immediate step-by-step risks—as separate tools for separate jobs. The authors of this new paper argue that this separation is a flaw. They propose that many complex problems require an agent to care about all three time scales at once.

To prove this point, the researchers created a simplified simulation involving three employees and three different tasks, each with a different level of toxicity. The goal was to assign these workers to tasks over a month-long period while keeping them safe. The toxicity of the work was not just a simple number; it had different effects depending on when it was measured. There was an immediate risk of a high dose within a single day, a medium-term risk of total exposure over the month, and a long-term risk of cumulative damage over many months. The team ran the simulation using the three traditional methods separately, and then attempted to combine them.

When the system focused only on the long-term average, it found a strategy that kept two experienced workers safe by constantly rotating them, but it effectively sacrificed the third, less experienced worker, assigning them to the most dangerous tasks repeatedly. The logic was that the average risk for the group was low, so the system ignored the fact that one person was being overloaded. When the system focused only on the single-journey outcome, it learned to rotate the workers frequently to ensure no one ever faced a dangerously high total dose for that specific month. This kept everyone relatively safe but resulted in a lot of switching that might not be efficient. When the system focused only on the immediate step-by-step risk, it avoided switching workers entirely, preferring to keep them on tasks that matched their skill level, which was a safe but rigid approach.

The most revealing result came when the researchers tried to optimize for all three time scales simultaneously. They used a method that combined the immediate, monthly, and long-term risks into a single goal. The resulting strategy was a mix of the others, but it revealed a critical flaw in trying to balance these competing clocks without a new framework. The system again tended to sacrifice the least experienced worker, assigning them to the hardest tasks to protect the others from both immediate spikes and long-term accumulation. The simulation showed that simply trying to average out the different time scales does not work; the non-linear nature of the risks means that protecting one time scale can inadvertently endanger another. The researchers found that the policies that worked well for one time scale were often the worst choices when all scales were considered together.

This finding highlights a significant gap in current artificial intelligence research. While we have powerful tools to handle long-term averages or immediate risks, we lack a comprehensive method to handle them together. The authors suggest that future systems need to learn the distribution of possible outcomes, understanding not just the average result but the full range of what could happen across different time windows. They acknowledge that this is a difficult challenge, as changing a policy to improve safety on one time scale might worsen it on another, creating a complex landscape of trade-offs. However, they believe that developing these combined approaches is essential for real-world applications, from managing radiation therapy in medicine to optimizing customer service in business, where the safety and satisfaction of individuals must be balanced against the efficiency of the whole.

The study concludes that the way we currently teach machines to make decisions is too narrow. By treating time as a single dimension or by choosing one perspective over the others, we risk creating systems that are safe in theory but dangerous in practice. The path forward requires new algorithms that can hold multiple time scales in mind at once, ensuring that an agent does not just maximize an average or avoid a single bad day, but truly understands the full weight of its actions across the entire span of its existence. This is not just a technical adjustment but a necessary evolution to ensure that artificial intelligence can navigate the messy, multi-layered reality of human life.