Forecasting what Matters: Decision-Focused RL for Controlled EV Charging with Unknown Departure Times

Technical Summary: Forecasting what Matters: Decision-Focused RL for Controlled EV Charging with Unknown Departure Times

Problem Formulation
The rapid adoption of electric vehicles (EVs) introduces significant challenges to power systems, including peak demand spikes and grid instability. While Reinforcement Learning (RL) offers a data-driven approach to optimize EV charging schedules (e.g., minimizing costs or flattening loads), its effectiveness is often hindered by incomplete information. A critical uncertainty in real-world scenarios is the vehicle's departure time. Without knowing when a vehicle will leave, an RL agent cannot accurately determine the available time flexibility to shift charging to off-peak hours, potentially leading to incomplete charging sessions or suboptimal cost management.

Existing literature often assumes perfect foresight of departure times or relies on traditional forecasting models trained solely for predictive accuracy (e.g., minimizing Mean Squared Error). However, the paper argues that a forecaster optimized for accuracy does not necessarily produce predictions that lead to optimal control decisions. Errors in standard forecasts can propagate through the control loop, degrading the overall performance of the charging policy.

Methodology: Decision-Focused RL (DF-RL)
To address the disconnect between prediction accuracy and decision quality, the authors propose a Decision-Focused Reinforcement Learning (DF-RL) framework. The core innovation is the end-to-end joint training of a departure time forecaster and an RL control agent, where the forecaster is updated based on the quality of the downstream control decisions rather than just prediction error.

• MDP Formulation: The problem is modeled as a Markov Decision Process (MDP). The state space includes measurable variables (electricity price, remaining energy demand, time of day) and a predicted remaining session duration ($\hat{d}_t$). The action space is binary: charge at full power or remain idle.
• The Forecaster: A neural network ($f_\psi$) predicts the total session duration ($\hat{T}_t$) based on time features, energy charged so far, and remaining energy. The remaining duration is derived by subtracting elapsed time from this prediction.
• The RL Agent: The authors utilize Soft Actor-Critic (SAC), an off-policy actor-critic algorithm that maximizes expected return while maintaining policy entropy for exploration.
• Joint Training Objective: The forecaster is trained using a composite loss function (Eq. 9):
  $$L_{total} = \beta L_R + (1 - \beta) L_{DF}$$
  • $L_R$: A standard regression loss (MSE) measuring the deviation between the forecast and the actual session duration.
  • $L_{DF}$: The decision-focused component, which leverages the RL agent's advantage function ($A(s,a)$) to guide the forecaster's updates. This term encourages the forecaster to produce predictions that result in higher cumulative rewards for the RL agent.
  • $\beta$: A hyperparameter ($\in [0,1]$) that balances the weight between prediction accuracy and decision quality.

Key Contributions

1. DF-RL Framework for EV Charging: The paper introduces a novel application of decision-focused learning to EV charging scheduling, specifically targeting the uncertainty of departure times. Unlike previous works that integrate DF learning with Model Predictive Control (MPC) or optimization-based methods (which require complex implicit differentiation of KKT conditions), this approach integrates DF learning directly into an RL pipeline.
2. End-to-End Optimization: The framework allows the forecaster to be trained directly from the control objective. The forecaster and the RL policy are updated jointly, enabling the forecaster to learn "what matters" for the controller rather than just minimizing statistical error.
3. State Augmentation: The method enriches the RL agent's state representation with a neural network-based forecast of the session duration, allowing the agent to make informed decisions despite the lack of ground-truth departure data.

Experimental Results
The authors evaluated the DF-RL framework using a real-world dataset of EV charging sessions from a hospital parking facility. They compared their approach against:

• BAU: Immediate charging upon arrival.
• RL (No Forecast): RL without any departure time information.
• RL (Conventional Forecaster): RL with a forecaster trained only on regression loss ($\beta=1$).
• RL (Real Departure): An upper-bound baseline with perfect knowledge of departure times.

Key Findings:

• Performance Improvement: The proposed DF-RL method (specifically with $\beta=0.2$ or $0.4$) achieved a 14% improvement in total reward and a 55% reduction in unsupplied energy (charging that failed because the EV left) compared to the RL method without departure time forecasting.
• Comparison to Conventional Forecasting: Compared to the RL method using a conventional forecaster ($\beta=1$), the DF-RL approach reduced the unmet demand by approximately 14% (from 7.3% to 6.3% in the best case) and improved the total reward by 5%.
• Behavioral Insight: The paper notes that DF-RL does not necessarily produce more accurate numerical predictions of departure time. Instead, it produces "conservative" predictions that are more useful for decision-making. For instance, in a case study, the DF-RL forecaster predicted a slightly shorter session duration than reality, prompting the agent to charge earlier and successfully complete the session. In contrast, the conventional forecaster overestimated the duration, leading to delayed charging and an incomplete session.
• Trade-offs: Lower values of $\beta$ (prioritizing decision quality over raw accuracy) resulted in fewer unsatisfied EVs but slightly higher charging costs, reflecting a trade-off between meeting energy demands and minimizing electricity expenses.

Significance and Claims
The paper claims that the primary significance of this work lies in demonstrating that prediction accuracy is not the sole determinant of control performance. By training the forecaster with feedback from the control agent, the system learns to generate predictions that are robust to the specific constraints and objectives of the charging problem.

The authors modestly state that while their DF-RL approach yields superior charging decisions compared to baselines, it is a step toward handling uncertainty in EV charging without the complexity of integrating DF learning with traditional optimization solvers. They acknowledge that the current work focuses on a single vehicle scenario with a deterministic price profile and plan future work to extend the framework to multi-vehicle cooperative scenarios, dynamic price profiles, and more expressive forecasting models to capture temporal patterns like seasonality.