The Reward Function and the Least Cost Principle for Gravitation and other Laws of Physics

This paper proposes a "least cost principle" to frame physical laws as solutions to an inverse optimal control problem, inferring that gravitational and Coulomb forces optimize a reward function that favors high relative velocities and quasi-circular orbits.

The Gist

Imagine the universe as a massive, cosmic video game. In this game, every particle (like an electron or a star) is a character, and the "laws of physics" (like gravity) are the rules that dictate how they move.

Usually, we ask: "How do these characters move?" and the answer is, "They follow Newton's laws."

But this paper asks a much deeper, almost philosophical question: "Why do they move that way? What is the game trying to achieve?"

The author, Rubén Moreno-Bote, suggests that the universe isn't just randomly following rules. Instead, it's playing a game of optimization. It's trying to get the best possible score while using the least amount of effort.

Here is the breakdown of the paper's big ideas, translated into everyday language:

1. The "Least Cost" Principle: The Lazy Universe

Imagine you are driving a car. You want to get to your destination, but you also want to save gas and avoid hitting the brakes or the gas pedal too hard. You want a smooth ride.

The paper proposes that the universe works the same way. It follows a "Least Cost Principle."

• The Cost: This is the "effort" or "jerkiness" of the movement. If a particle has to accelerate suddenly (like slamming on the brakes), that costs a lot of "energy points." The universe hates wasting points on jerky movements.
• The Reward: This is the "score" the universe gets for doing something cool. The universe wants to maximize this score.

The laws of motion (like gravity) are simply the result of the universe trying to maximize the score while minimizing the effort.

2. Reverse Engineering the Game (Inverse Optimal Control)

Usually, scientists start with the rules (the laws of physics) and predict the movement.

• Standard Physics: "Here is gravity. Now, tell me where the planet will go."
• This Paper: "Here is where the planet actually goes. Now, tell me what the 'score' (reward) must have been to make it move that way?"

The author uses a mathematical trick called Inverse Optimal Control. Think of it like watching a master chef cook a perfect dish. You don't know the recipe, but by tasting the food and watching the movements, you can guess exactly what ingredients (rewards) the chef was trying to highlight.

3. What is the Universe "Rewarding"?

When the author reverse-engineered the laws of gravity and electricity (Coulomb forces), they discovered what the universe considers a "good move." The universe is secretly cheering for two specific things:

A. The "Dance Partner" Reward (Relative Motion)

The universe loves it when particles are moving relative to each other.

• The Analogy: Imagine a dance floor. If everyone stands still, it's boring. If everyone runs in the exact same direction at the exact same speed, it's also a bit dull.
• The Reward: The universe gets a high score when particles are zipping around relative to one another. It wants the particles to be active and dynamic, not frozen in place.

B. The "Circular Orbit" Reward (Orthogonal Motion)

This is the most fascinating part. The paper found that the universe gets a massive bonus when particles move sideways relative to each other, rather than straight toward or away from each other.

• The Analogy: Imagine two people holding hands.
  • If they run straight toward each other, they crash. (Bad score).
  • If they run straight away, they drift apart forever. (Bad score).
  • If they run in a circle around each other, holding hands, that's a perfect score.
• The Result: This explains why planets orbit stars in circles (or ellipses) instead of crashing into them or flying off into space. The universe is "rewarding" that circular, orbital dance because it's the most efficient, stable, and "interesting" way to move.

4. Why Does This Matter?

The paper suggests that complexity (like life, stars, and galaxies) needs this specific kind of motion to exist.

• If the universe just minimized effort without a reward, everything would just sit still or move in straight lines forever.
• If the universe just maximized motion without caring about effort, everything would be chaotic and explode.

By balancing effort (minimizing acceleration) with rewards (encouraging relative motion and circular orbits), the universe creates the structured, beautiful, and complex systems we see today.

Summary Metaphor

Think of the universe as a gymnast on a balance beam.

• The Cost: Falling off the beam (too much acceleration/force).
• The Reward: Performing a beautiful, complex routine (relative motion and orbits).

The laws of gravity aren't just random rules; they are the gymnast's strategy to perform the most spectacular routine possible without falling off. The universe is constantly trying to keep the dance going, spinning in circles, and staying active, all while using the least amount of energy possible to do it.

Technical Summary

1. Problem Statement

The paper addresses a fundamental question in theoretical physics and control theory: If the universe follows a specific design, what cost (or reward) function is being optimized by observed physical forces?

While classical mechanics describes how particles move (via Newton's laws), it does not explicitly explain why these specific laws exist in terms of optimization. The author frames this as an Inverse Optimal Control (IOC) or Inverse Reinforcement Learning (IRL) problem. The goal is to infer the underlying reward function $R(x, \dot{x})$ that, when optimized, yields the observed laws of motion for central forces like Newtonian gravitation and Coulomb forces.

2. Methodology

The author employs a rigorous mathematical framework combining optimal control theory with physical symmetries.

A. The Least Cost Principle

The paper establishes a new principle called the Least Cost Principle. Unlike the classical Principle of Least Action (which minimizes the integral of the Lagrangian $L = T - V$), this principle formulates motion as an optimal control problem minimizing a time-discounted cumulative cost ($C$):

$$C = \int_{t_0}^{\infty} dt \, e^{-\gamma(t-t_0)} \left[ \sum_{i} \frac{1}{2}m_i \|\ddot{x}_i(t)\|^2 - R(x(t), \dot{x}(t)) \right]$$

Key components of this formulation:

• Acceleration Cost ($C_a$): The term $\sum \frac{1}{2}m_i \|\ddot{x}_i\|^2$ represents a penalty for large accelerations (strong forces). The author derives this quadratic form from first principles (time/rotational invariance, additivity, and invariance under homogeneously accelerated reference frames) rather than assuming it for mathematical convenience.
• Reward Function ($R$): A state-dependent term depending on positions ($x$) and velocities ($\dot{x}$), but explicitly not on accelerations.
• Discount Factor ($\gamma$): A temporal discount exponent ($\gamma > 0$) is introduced to ensure mathematical tractability and to model a system with a finite expected lifetime, emphasizing transient dynamics over asymptotic stationary distributions.

B. Inverse Optimal Control Derivation

Using the Bellman equation and the Hamilton-Jacobi-Bellman (HJB) formalism, the author derives the relationship between the known forces $F_{ik}$ and the unknown reward function $R$.

1. The optimal cost-to-go $C^*$ is related to the forces via $C^* = -\sum F_{ik} \dot{x}_{ik} + B(x)$.
2. By substituting this into the HJB equation, the author derives a general expression for the reward function (Eq. 7):
   $$R(x, \dot{x}) = -\sum_{ik,jp} \dot{x}_{ik}\dot{x}_{jp} \frac{\partial F_{ik}}{\partial x_{jp}} - \frac{1}{2}\sum_{ik} \frac{F_{ik}^2}{m_i} + \dots$$
   This equation reveals that the reward function is composed of terms related to the gradient of the force and the magnitude of the force itself.

3. Key Contributions

1. Derivation of the Acceleration Cost: The paper proves that the quadratic acceleration cost ($\|\ddot{x}\|^2$) is the unique form required to satisfy fundamental physical symmetries (specifically invariance under homogeneously accelerated frames) in closed systems.
2. Explicit Reward Functions for Fundamental Forces: The author successfully infers the specific reward functions that make Newtonian gravitation and Coulomb forces the optimal solutions to the control problem.
3. Identification of Optimized Features: The analysis reveals that natural forces actively optimize for two specific dynamical features: high relative motion and quasi-circular orbits.
4. Linearity of Reward Functions: The paper demonstrates that if the reward functions for individual forces are known, the reward function for a sum of forces is simply the sum of the individual reward functions.

4. Results

A. The Inferred Reward Function for Gravitation

For a system of $N$ particles under Newtonian gravitation, the inferred reward function (Eq. 9) consists of three main terms:

• Term I (Positive): Proportional to $\|\dot{x}_i - \dot{x}_j\|^2$. This term rewards high relative velocities between particle pairs. It is modulated by distance ($1/r^3$), meaning high relative motion is most rewarded at short distances.
• Term II (Negative): Proportional to $[(x_i - x_j) \cdot (\dot{x}_i - \dot{x}_j)]^2$. This term penalizes motion that is not orthogonal to the distance vector. Since the dot product is zero when vectors are perpendicular, this term effectively rewards circular-like motion (where velocity is perpendicular to the radius vector).
• Term III: A higher-order interaction term arising from the quadratic cost structure.

Interpretation: The universe "prefers" states where particles move fast relative to each other and where that motion is tangential (circular) rather than radial.

B. Coulomb Forces

The reward function for Coulomb forces follows the same mathematical structure as gravitation, with mass ($m$) replaced by charge ($q$) and the gravitational constant ($G$) replaced by the Coulomb constant ($C_{Coulomb}$).

• Opposite Charges: Behave like gravity (promoting relative motion and circular orbits).
• Like Charges: The signs invert. The system minimizes relative motion and aligns motion parallel to the distance vector (repulsion), consistent with electrostatic repulsion.

C. Numerical Simulations

• Trajectory Analysis: Simulations of 5 and 10 particles under Newtonian gravity confirm that trajectories are curvilinear and tend to form circular-like orbits when particles are close.
• Cost Minimization: The "cost-to-go" is minimized when the force follows the exact $1/r^2$ law. When the force is perturbed (e.g., $1/r^{2+\epsilon}$), the cost-to-go increases, confirming that the standard inverse-square law is indeed the optimal solution for the derived reward function.
• Fluctuations: The reward contributions (Term I and II) fluctuate over time, but the system consistently maintains a low cumulative cost.

5. Significance and Implications

• Motion as an Optimized Quantity: The paper challenges the view that motion is merely a consequence of forces. Instead, it suggests that relative motion and structured orbits are the primary objectives optimized by nature.
• Origin of Complexity: By optimizing for rich, structured motion (high velocity + circularity), these forces provide the fundamental ingredients necessary for the emergence of complexity in the universe. This aligns with principles like "Maximum Occupancy" and "Empowerment" in cognitive science.
• Unification of Physics and Control: The work bridges the gap between classical mechanics and modern control theory (Reinforcement Learning), suggesting that the laws of physics can be viewed as the solution to an inverse reinforcement learning problem where the "agent" is the universe itself.
• Beyond Attraction/Repulsion: The paper moves beyond the qualitative description of forces as simply "attractive" or "repulsive," providing a quantitative characterization of how these forces shape the geometry of motion to maximize specific dynamical features.

In conclusion, Moreno-Bote demonstrates that the laws of gravitation and electromagnetism are not arbitrary but are the optimal control solutions that maximize a reward function favoring high relative velocity and orthogonal (circular) motion, thereby fostering the dynamic complexity observed in the universe.