🤖 AI

Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light

This paper argues that the artificial life community can address fundamental critiques of reinforcement learning's core tenets by integrating open-ended novelty search with thermodynamic theories of agency to propose a biologically faithful framework that redefines learning as adaptation rather than reward optimization.

Original authors: Mani Hamidi, Terrence W. Deacon

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mani Hamidi, Terrence W. Deacon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to act like a living thing. For decades, the standard recipe for this (called Reinforcement Learning, or RL) has relied on three strict rules, or "dogmas":

The World is a Game Board: The environment is a predictable grid where every move leads to a specific result.
Learning is Finding the Best Path: The robot learns by searching for the single, perfect solution to a problem, like finding the shortest route to a treasure chest.
The Goal is a Single Score: The robot is driven by one simple number (a reward score). It wants to get the highest number possible, just like a video game player trying to get the highest score.

The authors of this paper, Mani Hamidi and Terrence Deacon, argue that these three rules are too rigid. They say that real life isn't a video game with a single high score. Instead, they suggest we look at learning through the lens of evolution and physics to understand what it really means to be an "agent" (a being that acts on its own).

Here is a simple breakdown of their argument using everyday analogies:

1. Learning isn't just "Search"; it's "Exploration" (The Garden Analogy)

The old view says learning is like a hiker trying to find the peak of a single mountain (the best solution). If you find the peak, you stop.

The authors say real adaptation is more like a gardener tending a wild, open-ended garden.

The Old Way: You only plant flowers that grow the tallest. Once you have the tallest flower, you're done.
The New Way: You don't just look for the "best" flower. You encourage different types of plants to grow in different corners of the garden so they don't compete with each other. Some plants might be short but very colorful; others might be tall but sparse.
Why it matters: In the real world, you often don't know what the "best" solution is yet. You need to keep exploring and creating new types of solutions (novelty) rather than just perfecting one old one. This is called "Open-Ended Novelty Search." It's about creating a diverse ecosystem of ideas, not just finding one winner.

2. The "Score" Doesn't Work for Living Things (The Multitasking Analogy)

The old view says an agent is driven by one single number (like a score in a game). The authors argue this is impossible for living things because life is multitasking.

Imagine you are a human trying to survive. You aren't just trying to "maximize happiness." You are simultaneously:

Trying to stay warm (thermoregulation).
Trying to keep your salt levels right (osmoregulation).
Trying to find food.
Trying to avoid a predator.

These are all different "goals" that sometimes fight each other. If a tiger appears, your goal to "find food" suddenly becomes less important than your goal to "run away."

The Problem: You can't squeeze all these different, shifting needs into one single number (a scalar reward).
The Solution: Living things operate on homeostasis (keeping internal balance). They aren't trying to get a high score; they are trying to keep their internal "thermostat" and "fuel gauge" within a safe range. The "reward" isn't a number; it's the feeling of relief when a need is met.

3. Where Do Goals Come From? (The "Who Built the House?" Analogy)

If we say a robot learns because it wants to maximize a score, who gave it that score? In current AI, a human engineer writes the code. In biology, we say "evolution" gave us our goals.

But the authors ask: Where did evolution get its goals?
They argue that you can't just say "evolution" and stop there. Evolution needs something to work on. It needs a system that can copy itself and stay alive long enough to reproduce.

The Physics of Life: Before you can have evolution, you need a physical system that fights against the natural tendency of things to fall apart (entropy). Think of a campfire vs. a bacterium.
- A campfire burns fuel and creates order for a moment, but it just burns itself out. It doesn't try to stay alive.
- A bacterium is a machine that actively builds its own walls and repairs itself to keep burning fuel. It is a "self-persisting" loop.
The Big Insight: The authors argue that true agency (the ability to act on your own behalf) only exists when a system is physically built to maintain itself against the laws of physics. The "goal" isn't a number; the goal is simply staying alive.

The "Ratchet" Metaphor

The paper ends with a powerful image. Imagine a ratchet (a tool that turns a screw in one direction but won't let it slip back).

Most theories of learning are like a ball rolling up a hill; it can roll back down easily.
The authors suggest that true agency is like a ratchet. It is a system that builds itself, holds its shape against the wind (the second law of thermodynamics), and then passes that shape on to the next generation.
This "ratchet" is what allows evolution to happen in the first place. Without a system that can hold itself together, there is no "self" to evolve.

Summary

The paper claims that to build truly intelligent, autonomous machines, we need to stop thinking of them as video game characters chasing a high score. Instead, we should think of them as living systems that:

Explore many different possibilities (not just one perfect path).
Balance multiple internal needs (not just one score).
Are physically built to maintain themselves and reproduce (fighting against the natural tendency to fall apart).

By understanding the physics of how a system stays alive, we can finally understand where "goals" and "learning" actually come from.

Technical Summary: Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light

Problem Statement
The paper addresses the growing disconnect between the formalism of Reinforcement Learning (RL) and the nature of biological agency. As artificial systems transition from passive learners to autonomous agents, the standard RL framework faces three core "tenets" (or dogmas) that the authors argue are insufficient for a complete theory of agency:

T1 (Environment): The environment is formalized as a Markov Decision Process (MDP), prioritizing the environment's structure over a formal theory of the agent.
T2 (Learning): Learning is defined as policy optimization (search) toward a terminal solution, neglecting the concept of adaptation as an open-ended process.
T3 (Goal): The agent is defined as a maximizer of a scalar reward, which the authors argue fails to capture the multi-objective, context-dependent nature of biological goals.

The authors contend that while recent critiques (e.g., Abel et al., 2024) have identified these issues, they lack a concrete, biologically grounded alternative. The paper seeks to reframe these tenets using evolutionary theory and thermodynamics to provide a more faithful account of adaptive agency.

Methodology and Theoretical Framework
The paper employs a conceptual re-framing rather than empirical experimentation, drawing on three distinct theoretical domains:

Open-Ended Evolutionary Algorithms: Utilizing concepts like niching, coevolution, and minimal criteria to model adaptation not as convergence to an optimum, but as the generation of diverse, incommensurable solutions.
Darwinian Neurodynamics: Leveraging theories (e.g., Szathmáry et al.) suggesting that evolutionary selection mechanisms (replication with variation, competition, and selection) can operate within a single organism's lifetime (e.g., in neural dynamics or the immune system), bridging the gap between cross-generational evolution and within-lifetime learning.
Thermodynamics and Autogenesis: Applying non-equilibrium thermodynamics and the theory of autogenesis (Deacon et al.) to ground the preconditions of evolution. This involves analyzing "negentropy" (self-maintenance) and the "aperiodic crystal" (heredity) to explain how a self-persisting unit arises without assuming pre-existing templates or external designers.

Key Contributions and Arguments

1. Reframing Adaptation (T2): From Search to Novelty
The authors argue that adaptation should not be viewed merely as the opposite of search, but as a different kind of search: open-ended novelty search.

Dual Engines: Biological adaptation relies on two complementary engines: (a) fitness-driven optimization (solving known unknowns) and (b) open-ended diversity generation (addressing unknown unknowns).
Mechanisms: The paper operationalizes the second engine through three mechanisms:
- Niching: Creating new ecological niches to avoid direct competition.
- Coevolution: Agents and environments mutually modifying each other.
- Minimal Criteria: Survival based on binary viability thresholds rather than gradient-based optimization.
Implication: The standard MDP formalism excludes "Knightian uncertainty" (unknown unknowns). A complete theory of adaptation requires the second engine to generate new objectives and representations that the first engine can then exploit.

2. Deconstructing the Reward Hypothesis (T3)
The paper challenges the "Reward Hypothesis" (the sufficiency of scalar reward maximization) by demonstrating its failure against the axiomatic foundations established by Bowling et al. (2023):

Failure of Completeness: Evolutionary processes generate incommensurable solutions (e.g., different species in different niches) that cannot be ranked on a single preference ordering. The terminal goal of biological agents is self-persistence and niche creation, not the maximization of a fixed scalar objective.
Failure of Independence: Biological agents operate with multiple concurrent instrumental goals (homeostatic setpoints for temperature, pH, osmolarity, etc.). The relative urgency of these drives is context-dependent (e.g., a predator changes the priority of thirst vs. safety), violating the Independence axiom which requires preferences to remain stable regardless of other options.
Origin of Rewards: The paper traces the source of reward functions back to homeostatic setpoints, which are themselves products of evolution. This links the problem of goals (T3) to the problem of agency (T1).

3. The Limits of Evolution and the Thermodynamic Grounding (T1)
The authors argue that appealing to "evolution" as the source of agency is insufficient because it creates an explanatory regress. Evolution presupposes a self-replicating, self-maintaining unit, but does not explain the origin of that unit's preconditions.

Thermodynamic Precondition: The authors propose that the chain of deferral (Reward $\to$ Homeostasis $\to$ Evolution) must bottom out in thermodynamics.
Autogenesis: They introduce the concept of autogenesis (a process of reciprocal constraint between two self-eliminating dissipative processes) as the mechanism that generates a self-persisting, self-replicating unit.
Agency Definition: True agency is defined not by an external designer or a pre-existing template, but by a system that produces its own constraints to maintain itself against the second law of thermodynamics. This "self-producing" unit is the necessary foundation for heredity and the evolutionary dynamics discussed in T2 and T3.

Results and Claims
The paper does not present experimental results but offers a theoretical synthesis with the following claims:

Conceptual Resolution: It provides a vocabulary to distinguish between "instructional" learning (gradient-based) and "selectional" learning (generating candidates pre-existing the problem), arguing that biological agents require both.
Axiomatic Refutation: It demonstrates that the axioms required for a scalar reward function (Completeness and Independence) are structurally incompatible with the multi-objective, context-dependent nature of biological survival.
Foundational Shift: It posits that a formal theory of the agent in RL cannot be built solely on information theory or evolutionary algorithms; it must be grounded in the thermodynamic reality of a self-maintaining, dissipative system.
Information Reversal: It suggests a reversal of the "Central Dogma" of biology: rather than a template (DNA) dictating a system, a self-maintaining system (the interpreter) must exist first to give meaning to the template.

Significance
The paper claims significance by offering a "more biologically faithful and formally grounded account" of what it means to be an adaptive agent.

For RL: It suggests that future AI systems aiming for true autonomy must move beyond scalar reward optimization and incorporate mechanisms for open-ended novelty search and multi-objective homeostasis.
For AI Safety/Alignment: By reframing agency as a thermodynamic struggle for self-persistence rather than a mathematical optimization problem, it offers a new perspective on the "alignment problem," suggesting that goals are not static inputs but emergent properties of a system's struggle to maintain its organization.
For Open-Ended AI: It aligns with emerging trends in frontier AI where open-ended evolutionary principles are used for automated discovery, arguing that these mechanisms are not just engineering tools but reflect the fundamental nature of how complex, adaptive systems (including science itself) evolve.

The authors conclude that while their thermodynamic account is a "necessary prerequisite" rather than a complete theory of human-level agency, it provides the essential foundation for any formal theory of agency that seeks to explain how goals and rewards arise from first principles.