⚡ electrical engineering

Joint Scheduling of Deferrable and Nondeferrable Demand with Colocated Stochastic Supply

This paper addresses the optimal scheduling of deferrable and nondeferrable electric demands using colocated stochastic supply by proving that the optimal policy follows a "Principle of Procrastination" to reduce the problem's complexity and proposing a reinforcement learning algorithm that effectively approximates this optimal strategy under unknown probability distributions.

Original authors: Minjae Jeon, Lang Tong, Qing Zhao

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minjae Jeon, Lang Tong, Qing Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a household energy system. You have two types of "jobs" that need electricity to get done, and you have two ways to get that electricity.

The Two Jobs:

The "Urgent" Job (Non-deferrable): This is like keeping your house warm or cooling it down, or keeping the lights on. You can't turn this off, but you can decide how hard to work. If electricity is cheap, you might run the AC full blast. If it's expensive, you might just keep it on a low setting. You have to do this job right now.
The "Flexible" Job (Deferrable): This is like charging an electric car or running a heavy-duty server. You have a deadline (e.g., "the car must be full by 7 AM"), but you can choose when to do the work. You could charge it slowly tonight, or wait until tomorrow morning.

The Two Power Sources:

The "Free" Sun: You have solar panels on your roof. The sun is unpredictable (stochastic). Sometimes it's bright, sometimes cloudy. This power is free, but if you don't use it, you can sell it back to the grid for a small profit.
The "Expensive" Grid: You can buy electricity from the utility company. The price changes depending on the time of day (Time-of-Use pricing). It's cheap at night, but very expensive during the day.

The Problem:
The goal is to make money (or save money) by balancing these jobs and power sources.

If you use the free solar power for your urgent job, you save money.
If you use the free solar power for your flexible job, you save money.
But, if you use the free solar power for your flexible job now, you might not have enough left to sell back to the grid later when the price is better, or you might run out of power for your urgent job later.

This is a massive puzzle with infinite possibilities because the sun's behavior and the arrival of new charging requests are random.

The Paper's Big Idea: "The Art of Procrastination"

The authors discovered a clever rule to solve this puzzle, which they call the Principle of Procrastination.

Think of procrastination not as laziness, but as strategic waiting.

Imagine you have a deadline to finish a report. You know you have a lot of work to do.

The Old Way: You start working immediately, even if you have a lot of free time later.
The Paper's Way: You wait. You only start working when you absolutely have to to meet the deadline.

In the energy world, this means:

Don't buy expensive grid power yet. If you have a flexible job (like charging a car), wait. Maybe the sun will come out later, or maybe the price will drop. Only buy power when you are running out of time and must finish the job.
Don't use free solar power yet. If you have a flexible job, don't use the free solar power immediately if you can sell that solar power back to the grid for a profit right now. Wait to see if you need that solar power later to avoid buying expensive grid power.

The "Three Magic Numbers"

The paper proves that you don't need a super-computer to solve this infinite puzzle. You only need to track three simple numbers (thresholds) for every moment in time:

The "Wait" Threshold: If the remaining work is below this number, do nothing. Just sell your solar power to the grid. It's too early to worry.
The "Start" Threshold: If the work is between the "Wait" number and a higher "Start" number, start using your free solar power to do the work, but don't buy any expensive grid power yet.
The "Panic" Threshold: If the work is above this number, stop waiting. Use all your solar power AND buy whatever expensive grid power you need to finish the job on time.

By reducing a complex, infinite problem to just tracking these three numbers, the problem becomes much simpler.

The Solution: A Smart Learner (PTRL)

Since we don't always know exactly how the sun will behave or how much our neighbors will use, the authors created a learning algorithm called PTRL (Procrastination Threshold Reinforcement Learning).

Think of PTRL as a smart apprentice.

Instead of trying to learn every single possible scenario (which is impossible), the apprentice is taught to only learn the three magic numbers.
The apprentice watches real-world data (like actual solar output and electricity prices).
Over time, it learns exactly when to "Wait," when to "Start," and when to "Panic."

The Results

The authors tested this on real data from electric vehicle charging and solar panels.

It works: The "Procrastination" strategy performed almost as well as the perfect, theoretical solution (which we can't actually calculate in real life).
It's better than the competition: It beat standard methods like "Model Predictive Control" (which tries to predict the future but often gets it wrong) and other AI methods that try to learn everything at once.
It's fast: Because it only learns three numbers instead of millions of possibilities, it makes decisions very quickly, which is crucial for real-time energy management.

In short: The paper teaches us that when dealing with unpredictable energy and flexible tasks, the best strategy is often to wait until you absolutely have to act. By mathematically defining exactly when that "have to" moment is, we can save a lot of money and run our energy systems much more efficiently.

Technical Summary: Joint Scheduling of Deferrable and Nondeferrable Demand with Colocated Stochastic Supply

1. Problem Formulation

This paper addresses the optimal scheduling of two distinct types of electrical demands within an energy system featuring colocated stochastic supply (e.g., rooftop solar) and grid-imported electricity. The system is modeled as a finite-horizon, discrete-time, continuous-state constrained Markov Decision Process (MDP).

The two demand types are:

Deferrable Demand: Arrives randomly (e.g., Electric Vehicle charging) and must be completed by a specific deadline. The quantity served can be delayed, but unserved demand at the deadline incurs a strictly increasing, convex penalty.
Nondeferrable Demand: Always present (e.g., HVAC, lighting) and must be served immediately. However, the quantity served is price-responsive, determined by a known, differentiable, concave utility function.

The system operates under a regulated Net Energy Metering (NEM) tariff, which is deterministic, time-varying, and piecewise-linear. This tariff distinguishes between import rates ( $\pi^+_t$ ) and export rates ( $\pi^-_t$ ), where $\pi^+_t \ge \pi^-_t$ . The objective is to maximize the expected economic surplus (total utility of consumption minus electricity payments and completion penalties) over the scheduling horizon.

The core challenge lies in the infinite-dimensional policy space resulting from continuous state variables (remaining deferrable demand and stochastic generation) and continuous action variables (service rates for both demand types), compounded by the nonlinearity of the NEM pricing structure.

2. Methodology and Theoretical Analysis

The Principle of Procrastination

The authors establish that the optimal scheduling policy follows a "Principle of Procrastination." This principle dictates that purchasing electricity from the grid should be delayed until it is strictly necessary to meet deadlines, and conversely, local renewable generation should be prioritized for immediate consumption or export based on future price and generation risks.

Structural Characterization of the Optimal Policy

Through rigorous analysis of the Bellman equation, the paper proves that the optimal policy can be reduced from an infinite-dimensional space to a finite-dimensional Euclidean space defined by three procrastination parameters (thresholds) for each time step $t$ :

$\theta^-_t(g_t)$ : A lower threshold on remaining deferrable demand. Below this, the system prioritizes exporting local generation over serving deferrable demand, even if it means deferring the load further.
$\theta^+_t(g_t)$ : An upper threshold. Above this, the system must aggressively serve the deferrable demand using both local generation and grid purchases to avoid completion penalties.
$\theta^0_t(g_t)$ : A parameter governing the "net-zero" region where the household neither imports nor exports.

These thresholds partition the state space into three distinct operational modes:

Net-Producing: Local generation exceeds demand; surplus is exported.
Net-Consuming: Demand exceeds local generation; grid power is purchased.
Net-Zero: Local generation exactly matches demand; no grid interaction.

Within these regions, the policy defines a priority structure for allocating local generation among nondeferrable demand, deferrable demand, and export. For instance, in the "net-zero" region, the policy balances the marginal utility of nondeferrable devices against the marginal value of serving deferrable load, determined by the specific threshold values.

Procrastination Threshold Reinforcement Learning (PTRL)

To solve the MDP when underlying probability distributions are unknown, the authors propose PTRL, a structured Reinforcement Learning (RL) algorithm.

Dimensionality Reduction: Instead of learning a full action policy over the continuous $(K+1)$ -dimensional space (where $K$ is the number of nondeferrable devices), PTRL learns only the three procrastination thresholds ( $\theta_t$ ).
Algorithm Architecture: PTRL utilizes a Soft Actor-Critic (SAC) framework. The "Actor" network outputs the mean and covariance of the procrastination thresholds based on the current state. The "Critic" network estimates the value (Q-function) of a state-threshold pair.
Policy Execution: Once the thresholds are generated by the actor, the actual scheduling actions (service rates) are computed deterministically using the structural rules derived in Theorem 2.

3. Key Results

Theoretical Contributions

Optimality of Procrastination: The paper formally proves that for both time-invariant and time-varying NEM tariffs, the optimal policy is characterized by the three procrastination thresholds. This reduces the complexity of the scheduling problem significantly.
Linear Time Dependency: For time-independent generation models, the paper demonstrates that the procrastination thresholds satisfy a linear time dependency ( $\theta^+_t = \theta^+_{t+1} + \bar{v}$ ), further reducing the number of parameters to be learned.

Numerical Performance

Experiments were conducted using real-world data traces from the Adaptive Charging Network (ACN) for EV demand and the Pecan Street dataset for solar generation and household consumption.

Convergence: PTRL converged significantly faster (approx. 800 episodes for independent models, 2,000 for Markovian models) compared to standard benchmarks like SAC and DDPG, which required roughly 6,000 and 2,000 episodes respectively.
Optimality Gap: Under synthetic test data matching the training distribution, PTRL achieved a gap to the optimal policy of only 3.34%, compared to 36.76% for SAC and 25.91% for DDPG.
Field Data Robustness: When tested on out-of-sample field data, PTRL outperformed all benchmarks (SAC, DDPG, and Model Predictive Control) by substantial margins (e.g., ~107% improvement over SAC). This confirms the algorithm's robustness to model mismatch and real-world stochasticity.
Computational Efficiency: PTRL demonstrated a significant computational advantage over Model Predictive Control (MPC), being approximately 19 times faster in evaluation time, making it suitable for online implementation.
Energy Source Utilization: PTRL effectively learned to prioritize consuming renewable energy during off-peak hours (when export prices are lower) and saving high-value peak-hour renewables for export, a strategy that standard benchmarks failed to replicate as effectively.

4. Significance and Claims

The paper claims that its primary contribution is the structural characterization of the optimal policy for a complex, real-world energy scheduling problem. By identifying the "Principle of Procrastination," the authors transform an intractable, infinite-dimensional MDP into a low-dimensional parameter estimation problem.

The significance of this work lies in:

Bridging Theory and Practice: It provides a theoretical foundation (threshold policies) that enables the development of highly efficient, data-driven learning algorithms (PTRL) that outperform both standard RL and traditional control methods (MPC).
Handling Nonlinear Pricing: Unlike prior works that often assume linear wholesale pricing, this paper addresses the piecewise-linear, time-varying nature of regulated retail NEM tariffs, which is critical for practical deployment in residential and commercial settings.
Joint Optimization: It is the first work to jointly schedule deferrable and nondeferrable demands with colocated stochastic supply, capturing the inter-temporal dependencies and economic trade-offs inherent in modern prosumer systems.

The authors maintain modesty regarding future extensions, noting that the current framework assumes known utility functions for nondeferrable demand and a single deferrable task type. They suggest future work could involve integrating online inverse optimization to learn utilities and extending the framework to multiple simultaneous deferrable tasks.