🤖 machine learning

A Framework for Designing Reward Functions: From Objectives to Features to Human-Aligned Reward Functions

This paper presents a formal, three-step framework that enables non-experts to design human-aligned linear reward functions by systematically deriving measurable outcome variables, selecting a causally representative subset of reward terms via polynomial-time optimization, and fitting weights through a geometrically framed preference elicitation process that guarantees a conflict-free feasible region.

Original authors: Di Yang Shi, W. Bradley Knox

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Di Yang Shi, W. Bradley Knox

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car, but you can't just give it a manual. Instead, you have to give it a "scorecard" called a reward function. Every time the robot does something good, like staying in its lane, the score goes up. Every time it does something bad, like hitting a curb, the score goes down. The robot's goal is simple: get the highest score possible. But here's the tricky part: if you design the scorecard wrong, the robot might find a sneaky way to exploit the system. It might learn to drive perfectly fast but crash into everything because you forgot to tell it that crashing is bad. This is a huge problem in the field of Artificial Intelligence known as Reinforcement Learning. For a long time, only the world's top experts could build these scorecards, and even they often made mistakes that led to robots behaving in weird or dangerous ways. The big question is: How do we turn a vague human wish, like "drive safely and comfortably," into a precise mathematical formula that a robot can understand without getting confused or trying to game the system?

A team of researchers at the University of Texas at Austin has proposed a new, step-by-step framework to solve this puzzle, making it possible for non-experts to design these robot scorecards. They call their method a "formal process" that turns a natural language description of a task into a mathematically sound reward function. Think of it as a recipe that takes a messy, human idea and bakes it into a clean, conflict-free instruction set for a machine. Their approach is built on three main steps: first, breaking down the big goal into smaller, fundamental desires; second, picking the right tools to measure those desires without counting the same thing twice; and third, figuring out exactly how much each measurement matters by asking the human to compare different scenarios. The authors show that by following this strict process, you can avoid the common traps of "reward hacking" (where the robot exploits the system) and ensure the robot's behavior truly matches what the human actually wants.

The Three-Step Recipe for Robot Behavior

The paper presents a framework that acts like a translator, converting a human's natural language wish into a linear reward function—a mathematical equation where different factors are added together with specific weights. The authors argue that this process should be accessible to non-experts and consists of three distinct stages.

Step 1: From "I Want" to "What to Measure"
The first step is about clarity. Imagine you tell a robot, "I want a safe and fast trip." That's too vague for a computer. The framework suggests a guided workflow to "distill" this wish into fundamental objectives. You start by listing everything you want, then you keep asking "why?" until you hit the core values. For example, "avoid traffic" might boil down to "minimize time," and "drive safely" might become "minimize collisions."

Once you have these core goals, you need to turn them into outcome variables—things you can actually measure. You can't measure "safety" directly, but you can measure "peak acceleration" or "passenger satisfaction." The paper provides a checklist for this: if a goal can't be measured, break it down further. If the thing you want to measure isn't visible during training, find a cause that is visible. And if a measurement can be "gamed" (exploited), you need to diversify your measurements so that exploiting one doesn't mean you've actually achieved the goal. This step ensures you aren't rewarding the robot for faking the result.

Step 2: Picking the Right Tools (The Causal Filter)
Now you have a long list of things to measure. But measuring everything is expensive and confusing. If you measure both "speed" and "time to destination," you might be counting the same thing twice because speed causes the time to change. This is where the paper introduces a clever mathematical trick.

The authors treat the relationships between these measurements as a map (a graph) where arrows show what causes what. They propose selecting a subset of measurements that covers all the fundamental goals but costs the least amount of effort. They call this the Minimum-Cost Partial Cover problem. To solve this, they use a method from graph theory called max-flow, which is like finding the most efficient way to route water through a network of pipes. By turning the problem into a flow network, they can use a computer algorithm to find the perfect, non-redundant set of measurements in a mathematically guaranteed way. This replaces the old, messy way of just guessing which measurements to keep with a precise, optimal selection process.

Step 3: Tuning the Dials (Weight Fitting)
Finally, you have your chosen measurements (like time, cost, and comfort), but you need to know how much each one matters. Is saving 1 minute worth making the ride bumpy? To answer this, the framework uses preference elicitation. Instead of guessing the numbers, the system asks the human: "Would you rather have a trip that is 10 minutes longer but very smooth, or 5 minutes shorter but bumpy?"

The paper frames this as a geometric problem. Imagine a 3D space where every point represents a different set of weights (how much you care about time vs. comfort). Every time the human answers a question, the system draws a line (a hyperplane) that cuts away the half of the space where the answer would be wrong. The system then picks the best next question to ask—one that cuts the remaining space in half, just like a detective narrowing down suspects. The authors show that by using a method called the Analytic Center Cutting Plane Method, they can find the perfect weights with a specific number of questions (roughly proportional to the number of variables times the logarithm of the desired precision). This ensures the final scorecard is consistent and doesn't have conflicting rules.

Why This Matters

The paper suggests that this framework solves three major headaches in robot training. First, it stops redundancy by using the causal map to ensure we aren't measuring the same thing twice. Second, it prevents reward hacking by grounding the rewards in the fundamental objectives the human actually cares about, rather than intermediate behaviors the robot might exploit. Third, it fixes preference misalignment by mathematically guaranteeing that the final set of weights is consistent with all the human's answers, rather than just a best guess.

The authors admit that this is a new method and that the "preference elicitation" part still relies on a human (or an expert system) to answer the questions. However, they argue that by making the process a formal, step-by-step algorithm, they are taking the first major step toward a world where anyone can design a reward function that a robot will actually follow, without needing a PhD in mathematics or years of trial and error. They have turned a chaotic, expert-only art into a structured, solvable puzzle.

Technical Summary: A Framework for Designing Reward Functions

Problem Statement

The design of human-aligned reward functions in Reinforcement Learning (RL) remains a process largely restricted to experts, often relying on trial-and-error that leads to common pitfalls such as reward hacking, redundancy in reward terms, and mismatches between policy behavior and human intent. While Reinforcement Learning from Human Feedback (RLHF) has gained popularity, existing querying processes are often Bayesian or unspecified, lacking deterministic bounds. Furthermore, the selection of reward terms is frequently heuristic, leading to inefficiencies and misaligned incentives. This paper addresses the need for a formal, systematic process that enables non-experts to instantiate and iterate on human-aligned linear reward functions, ensuring adherence to a given preference ordering over trajectories.

Methodology

The proposed framework transforms a natural language task description into a linear reward function through three distinct, formalized steps:

1. Deriving Outcome Variables (Guided Workflow)

The first step converts a natural language task description into a set of measurable outcome variables.

Distillation: The task is iteratively analyzed to extract "fundamental objectives." This involves asking "why" an objective is important until a set of core values is reached (e.g., converting "avoid traffic" to "minimize time").
Proxy Derivation: Since fundamental objectives (e.g., "minimize passenger discomfort") are often not directly measurable, the framework derives measurable "outcome variables" (proxies) that capture these objectives.
Filtering: The framework identifies potential issues with proxies (e.g., unobservability or gameability) and suggests remedies, such as decomposing objectives or substituting with causally upstream variables.

2. Causally-Aware Reward Term Selection (Minimum-Cost Partial Cover)

The candidate set of outcome variables is often too large or redundant. This step selects a causally representative, low-cost subset to serve as the final reward terms.

Causal Modeling: The relationships between outcome variables are modeled as a Directed Acyclic Graph (DAG), where edges represent causal effects.
Formalization: The selection problem is formalized as a Minimum-Cost Partial Cover (MCPC) problem on the causal DAG. The goal is to find a subset of nodes $S$ that "covers" all demand nodes (representing fundamental objectives) while minimizing the total cost of selected nodes. Coverage is defined recursively: a node is covered if it is selected or if all its parents are covered.
Algorithmic Solution: The MCPC problem is reduced to a max-flow min-cut problem via node splitting. This allows for an exact, polynomial-time solution using standard max-flow algorithms (e.g., Edmonds-Karp or Dinic's algorithm).
Heuristic Alternative: In cases where global causal structure or costs are unknown, a locally greedy approach is proposed to refine a proposed set of terms against a cost budget.

3. Weight Fitting via Preference Elicitation (Convex Feasibility)

Once reward terms are selected, their weights must be determined to form a scalar reward function. The paper assumes a linear combination of features.

Geometric Framing: The weight fitting problem is reframed as a convex feasibility problem. The goal is to locate the true weight vector $w^*$ within a feasible region defined by preference constraints.
Separation Oracle: The feasible region is iteratively narrowed using a separation oracle. The system proposes pairs of synthetic trajectories $(\tau^+, \tau^-)$ to an oracle (e.g., a human expert). The oracle's preference decision defines a half-space constraint ( $s \cdot q^T w \geq 0$ ) that cuts the feasible region.
Synthetic Trajectories: Unlike methods sampling from fixed datasets, this framework constructs trajectory pairs from scratch to pose arbitrary "would you rather" comparisons. This ensures the query points are optimal for volume reduction.
Optimization Methods: The paper leverages existing separation oracle methods, specifically the Analytic Center Cutting Plane Method (ACCPM) and the Volumetric Center Method. These methods guarantee convergence with $O(n \log \kappa)$ oracle calls, where $n$ is the dimension and $\kappa$ relates to the initial region radius and error tolerance.

Key Contributions

Guided Workflow for Outcome Variables: A structured process for distilling fundamental objectives from natural language and deriving measurable proxies, addressing issues of unobservability and gameability.
Formal Reward Term Selection: The reduction of reward term selection to a Minimum-Cost Partial Cover problem on a causal DAG. This transforms a heuristic manual process into an optimal, algorithmic approach solved via max-flow, ensuring causally representative and non-redundant reward terms.
Deterministic Weight Fitting: A geometric framing of weight fitting as a convex feasibility problem. By using synthetic trajectory generation and separation oracle methods, the framework produces a deterministically conflict-free feasible weight region. It achieves asymptotically optimal preference querying efficiency ( $O(n \log \kappa)$ ) and likely-optimal runtime.

Results and Claims

The paper does not present empirical benchmarks on specific RL environments but rather provides a theoretical framework and algorithmic proofs.

Conflict-Free Feasibility: The method claims to be the first reward-design approach that maintains a deterministically conflict-free feasible weight region throughout the iteration process. This is achieved by constructing synthetic trajectories that guarantee consistency, avoiding the distributional assumptions and conflict-resolution procedures required by methods using fixed datasets.
Optimality: The selection of reward terms is claimed to be optimal by design (via max-flow reduction), and the weight fitting process inherits the asymptotically optimal query efficiency of volumetric center methods.
Addressing Failure Modes: The framework explicitly targets three recurring failure modes in reward design:
- Redundancy: Mitigated by the causal cover selection.
- Reward Hacking: Mitigated by grounding reward terms in elicited fundamental objectives rather than shaped intermediate behaviors.
- Preference Misalignment: Mitigated by the construction of a consistent feasible weight region.

Significance

The authors position this work as a foundational step toward enabling non-RL practitioners to design and iterate on reward functions. By formalizing the transition from natural language objectives to a linear reward function, the framework reduces the reliance on expert trial-and-error. The use of separation oracles and synthetic trajectory generation offers a rigorous, deterministic alternative to the often unspecified or Bayesian querying processes in current RLHF literature. The paper concludes that this principled foundation supports the development of interactive systems for automated reward design.