📊 statistics

Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions

This paper introduces a decision-centered state abstraction method for offline reinforcement learning that utilizes causal machine learning and orthogonal estimation to efficiently learn difference-of-Q functions, thereby isolating essential decision-making information from irrelevant state dynamics while ensuring consistent policy optimization.

Original authors: Defu Cao, Angela Zhou

Published 2026-09-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Defu Cao, Angela Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast world of data, machines are constantly learning to make decisions, from recommending a movie to managing a hospital's patient flow. This field, known as reinforcement learning, teaches computers by showing them the results of past actions. However, a major challenge arises when the data is too rich. Modern sensors capture everything: high-resolution images, text, and complex environmental details. While this information is valuable for predicting what will happen next, it often contains a heavy load of details that do not actually matter for making the best choice. A computer trying to learn the perfect move might waste its time studying irrelevant patterns, like the color of the sky, when the decision only depends on the price of a product. This inefficiency slows down learning and can lead to poor decisions when data is scarce.

Researchers Defu Cao and Angela Zhou from the University of Southern California have developed a new way to cut through this noise. They focus on a specific type of learning called offline reinforcement learning, where the computer must learn from a fixed history of past events without being able to try new things in the real world. Their work introduces a concept they call "decision-centered abstractions." Instead of trying to understand every single detail of a situation to predict the future, their method teaches the machine to ignore everything that does not change the difference between two possible actions. They found that the information needed to choose the best action is often much simpler than the information needed to predict the entire future. By stripping away the unnecessary complexity, they allow the computer to learn faster and more accurately, even when the data is messy or incomplete.

The core of their discovery lies in how they measure success. Traditional methods often try to estimate the total value of every possible action in a given situation. This is like trying to calculate the exact total cost of two different vacation packages, including every flight, hotel, and meal, just to decide which one is cheaper. Cao and Zhou realized that to make the choice, the computer does not need the total cost of each package; it only needs to know the difference in price between them. If one vacation is ten dollars more expensive than the other, the computer only needs to learn that ten-dollar gap. They call this the "difference-of-Q function." By focusing solely on this gap, the machine can ignore vast amounts of data that are identical for both options, such as the cost of a shared flight or a common hotel fee. This approach is similar to how a doctor might ignore a patient's general health history if they are only trying to decide between two specific treatments that have the same side effects, focusing only on the part of the history that makes one treatment better than the other.

To find these simpler patterns, the researchers created a new mathematical tool that acts like a filter. They use a technique called orthogonal estimation, which helps the computer separate the signal from the noise. Imagine trying to hear a specific conversation in a crowded room; this method allows the computer to tune out the background chatter of irrelevant state changes and focus only on the parts of the data that actually shift the balance between choices. They tested this idea using simulations where the data was generated with known rules, including scenarios with hundreds of different state variables. In these tests, their method successfully identified that only a tiny fraction of the available information was actually needed to make the right decision. For instance, in one experiment with 120 different state variables, their algorithm correctly determined that only three were truly important for the decision, while standard methods struggled to filter out the rest.

The researchers also showed that this method works even when the computer has to guess at other parts of the system, such as how likely a person is to take a certain action in the past. Their approach is robust, meaning it remains accurate even if those initial guesses are not perfect. They demonstrated that by using this focused approach, the computer could learn the optimal strategy much faster than traditional methods, which get bogged down trying to model the entire complex world. In a real-world-inspired simulation involving ridesharing, their method reduced the error in decision-making by a significant margin compared to existing techniques. The results suggest that in many complex systems, from managing hospital discharges to setting prices for products, the path to a better decision is not through knowing more, but through knowing what to ignore.

This work does not just offer a theoretical improvement; it provides a practical roadmap for building smarter decision-making systems. By proving that the information required for a good decision is often a small, sparse subset of the total available data, the researchers have shown that machines can be more efficient. They demonstrated that when the data is structured in a way where certain variables do not affect the choice between actions, their method can automatically discover and discard those variables. This leads to policies that are not only more accurate but also more reliable, as they are less likely to be confused by irrelevant details. The study confirms that in the era of big data, the key to better artificial intelligence may not be feeding it more information, but teaching it how to find the specific, narrow slice of information that truly matters.

Technical Summary: Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions

1. Problem Statement

Offline reinforcement learning (RL) aims to evaluate and optimize sequential decision policies using historical data when online experimentation is infeasible due to safety or cost constraints. While modern data sources provide rich, high-dimensional state information (e.g., text, images), standard offline RL algorithms often struggle because they attempt to learn full state dynamics and value functions ( $Q$ and $V$ ) that contain reward- and action-irrelevant information. This "curse of dimensionality" slows statistical learning and wastes scarce data on predicting future states that do not influence the optimal action choice.

The core problem addressed is the identification and learning of decision-centered abstractions: mappings of the state space to a simpler space that preserves only the information necessary to determine the optimal action. Specifically, the authors argue that preserving the difference between $Q$ -functions for different actions ( $\tau = Q(\cdot, a) - Q(\cdot, a')$ ) is sufficient for decision-making, whereas preserving the absolute $Q$ -values often requires modeling irrelevant dynamics.

2. Methodology

2.1 Decision-Centered Abstractions

The paper introduces a formal definition of decision-centered abstractions. Unlike traditional state abstractions that preserve the full $Q$ -function or model transitions, a decision-centered abstraction $\phi_t$ ensures that if two states map to the same abstract state, the difference in their $Q$ -values for any pair of actions remains constant.

Coordinate Support: The authors characterize the minimal set of state coordinates ( $K_{\Delta, t}$ ) required to preserve this difference. They refine existing graphical criteria (e.g., IFactor) by incorporating additive reward structures. They show that under additive rewards, coordinates that affect the reward but cancel out when differencing actions (e.g., uncontrollable exogenous variables) can be discarded, even if they are necessary for predicting absolute state values.
Limitation: The paper notes that while these abstractions preserve the regret (the cost of choosing a suboptimal action), they do not necessarily support standard dynamic programming (Bellman planning) on the abstracted space to recover the optimal policy directly. Instead, they are used to learn the contrast function $\tau$ for policy optimization.

2.2 Orthogonal Estimation of Difference-of-Q

To estimate the difference-of- $Q$ function ( $\tau$ ) efficiently, the authors propose a dynamic orthogonal estimator based on the R-learner framework (Nie et al., 2021; Lewis and Syrgkanis, 2021).

Pseudo-outcomes: The method constructs pseudo-outcomes by recursively unrolling the Bellman equation. The authors define a family of estimators parameterized by an unrolling depth $L \in \{0, \dots, T-t\}$ $L \in {0, \dots, T - t}$ :
- $L=0$ (One-step): Uses the immediate reward plus the estimated continuation value $V_{t+1}$ . This variant relies on estimating the future $Q$ -function but offers lower variance in the pseudo-outcome.
- $L=T-t$ (Full unrolling): Recursively substitutes future $Q$ -functions with rewards and future contrasts, eliminating the need for terminal value estimation but potentially increasing variance.
Orthogonality: The loss function is constructed to be Neyman-orthogonal with respect to nuisance parameters (the behavior policy $\pi_b$ and the conditional mean of the pseudo-outcome). This allows the estimator to achieve $\sqrt{n}$ -consistency even if the nuisance estimators (e.g., behavior policy or $Q$ -function approximators) converge at slower rates (specifically $o_p(n^{-1/4})$ in $L_4$ norm).
Cross-fitting: The method employs $K$ -fold cross-fitting to prevent overfitting when estimating nuisance functions and the target contrast $\tau$ .

2.3 Sparse Learning for Abstraction Recovery

To automatically discover the decision-centered abstractions, the method integrates sparse statistical learning (specifically thresholded LASSO) with the orthogonal estimator.

The algorithm minimizes a regularized loss (L1 penalty) to identify the support of the coefficient vector $\beta$ in a linear model for $\tau$ .
Theoretical guarantees (Proposition 6) show that under restricted eigenvalue conditions and a "beta-min" condition, the method recovers a superset of the true decision-relevant coordinates with high probability, with the cardinality of the selected set bounded by twice the true support size.
A horizon-union screening rule is proposed to aggregate supports across time steps for robustness.

3. Key Contributions

Definition and Characterization of Decision-Centered Abstractions: The paper formally defines abstractions that preserve only the difference-of- $Q$ functions. It provides graphical criteria (under additive rewards) to identify when these abstractions are strictly simpler than model-preserving or $Q$ -preserving abstractions, effectively filtering out action-irrelevant state dynamics.
Dynamic Orthogonal Estimator: The authors develop a generalization of the R-learner for sequential decision-making. They introduce two variants ( $L=0$ and $L=T-t$ ) that trade off between variance reduction and sensitivity to future-value estimation errors. They prove that this estimator achieves optimal convergence rates for policy evaluation under mild conditions on nuisance function estimation.
Structure-Adaptive Policy Learning: By combining orthogonal estimation with sparse learning (thresholded LASSO), the paper establishes guarantees for policy optimization that depend on the dimension of the decision-relevant state space rather than the ambient dimension. This allows for efficient learning in high-dimensional settings where the true decision signal is sparse.
Theoretical Guarantees: The paper provides non-asymptotic bounds showing that policy value convergence rates depend on the margin condition (low-noise assumption) and the complexity of the decision-centered support, rather than the full state dimension.

4. Experimental Results

The authors validate their approach on simulated data and simulator-augmented real-world data (ridesharing):

Adaptation to Structure: In experiments with varying data-generating processes (reward-filtered, exogenous-endogenous, and nonlinear main effects), the proposed thresholded-LASSO Difference-of-Q estimator consistently outperformed standard Fitted-Q Evaluation (FQE) and reward-filtered baselines. It successfully adapted to sparse decision structures even when the full state dynamics were dense or misspecified.
Sparse Support Recovery: In a linear-Gaussian MDP with 120 state coordinates (only 3 action-relevant), the proposed method achieved superior small-sample performance in recovering the true active coordinates and learning the policy compared to full-Q ridge regression.
Variance Reduction: Comparing the $L=0$ and $L=T-t$ variants, the $L=0$ (one-step) estimator demonstrated lower variance and better overall performance in policy improvement tasks, confirming the theoretical trade-off analysis.
Ridesharing Application: In a Markovian interference setting (ridesharing), the method reduced the Root Mean Squared Error (RMSE) of estimating the difference-of- $Q$ by 64% compared to a baseline without control variates and by 34% compared to Truncated Policy Gradient (TPG). It also enabled faster learning of state-dependent policy improvements.

5. Significance and Claims

The paper claims that decision-centered abstractions offer a more efficient path to offline RL optimization by focusing statistical learning resources on the specific information required to distinguish between actions, rather than modeling the entire environment dynamics.

Bridging Causal ML and RL: The work positions the difference-of- $Q$ function as the sequential analogue of the Conditional Average Treatment Effect (CATE) in causal inference. By leveraging orthogonal estimation techniques from causal ML, the authors demonstrate that sequential decision problems can benefit from the same variance reduction and robustness properties.
Practical Efficiency: The method allows for the use of black-box estimators for nuisance parameters (like behavior policies or $Q$ -functions) while still achieving fast convergence rates for the policy value, provided the decision-relevant structure is sparse.
Modest Scope: The authors acknowledge that decision-centered abstractions do not guarantee that standard dynamic programming on the abstracted space will recover the optimal policy (unlike model-based abstractions). Instead, their approach is designed to learn the contrast function $\tau$ directly to drive policy optimization (e.g., via greedy steps or local improvements).

In summary, the paper argues that for many operational decision problems, the "difference that matters" is significantly simpler than the full state representation, and orthogonal estimation provides the statistical machinery to learn this difference efficiently from limited offline data.