State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading

Technical Summary: State Representation Matters in Deep Reinforcement Learning for Energy Trading

Problem Formulation
Energy trading for pumped-storage hydroelectricity is a sequential decision-making problem under uncertainty. An agent must decide hourly whether to pump water (consume energy), release water (generate energy), or hold, based on current reservoir levels and market information. The objective is to maximize revenue by exploiting price differences between pumping and generation hours. While Reinforcement Learning (RL) is a natural fit for this setting, the quality of the learned policy is heavily dependent on the state representation provided to the agent. The paper investigates how different market feature families—specifically absolute, relative, and forecast features—affect the performance and generalization of model-free Deep RL agents when market conditions shift (e.g., changes in price levels, volatility, or geographic location).

Methodology
The authors utilize HydroDam, a controlled environment simulating a pumped-storage facility with fixed physical constraints (capacity, efficiency, flow rates) and a fixed action space (pump, hold, release at maximum rate). The reward function is defined as immediate trading revenue, accounting for positive and negative electricity prices.

To isolate the impact of state representation, the study employs a controlled ablation study using a fixed Double DQN (DDQN) architecture and training protocol. The only variable changed across experiments is the input state features. The feature families tested are:

1. Absolute Features: Normalized current prices, log-transformed prices, and calendar encodings (sine/cosine for hour/day, weekend indicators).
2. Relative Features: Features comparing current prices to recent history, including 24h/168h z-scores, rolling price scores (quantile-based), relative calendar features, and technical indicators (RSI, Bollinger bands, moving averages).
3. Forecast Features: A 24-hour price forecast generated by an ensemble of five LightGBM models, including future price z-scores and ensemble uncertainty estimates.

Experimental Setup

• Data: Training and validation data consist of Belgian day-ahead prices from 2007–2011.
• Evaluation:
  • Same-Market Test: Performance is evaluated on Belgian data from 2012–2025, a period containing significantly higher volatility, negative prices, and the 2021–2022 energy crisis.
  • Cross-Zone Test: Fixed policies are evaluated on 39 other ENTSO-E bidding zones to test zero-shot transfer across different market regimes.
• Baselines: A rolling price-score heuristic (buying when prices are low relative to recent percentiles, selling when high) serves as a strong, non-RL baseline. Optimal profit is calculated via dynamic programming with perfect foresight to normalize results as a percentage of the theoretical maximum.

Key Contributions

1. Controlled Testbed: The paper formulates HydroDam as a specific testbed to study state representation in model-free deep RL for energy trading, keeping the environment and algorithm fixed.
2. Feature Ablation: It systematically compares absolute, relative, and forecast state representations, isolating their individual and combined effects on policy performance.
3. Generalization Analysis: It evaluates policies on unseen time periods and unseen geographic markets, demonstrating that state representation is a critical factor in transferability, not merely a preprocessing detail.

Results
The study reveals that state representation drastically alters the perceived effectiveness of the RL agent:

• Absolute Features: Achieved the highest validation performance (78.9% of optimal profit) but suffered severe overfitting, dropping to 28.8% on the same-market test set and a median of 5.7% across cross-zones.
• Relative & Forecast Features (Single): Both performed poorly in isolation. Relative-only policies reached only 10.3% on the test set, and forecast-only policies reached 33.4%, both failing to consistently beat the simple rolling price-score heuristic in cross-zone medians.
• Combined Features: Combining feature families yielded significantly robust results.
  • Absolute + Relative: Reached 49.9% on the test set and a 39.8% cross-zone median.
  • Absolute + Relative + Forecast: Achieved the strongest performance, reaching 55.6% on the test set and a 47.5% median across 39 zones. This full model outperformed the heuristic in 38.2 of 39 zones on average.

Significance and Claims
The paper claims that in energy storage trading, state representation is a central part of policy design, not a minor preprocessing choice. The results suggest that:

• Relying on a single feature family leads to poor generalization when market regimes shift (e.g., from calm historical periods to crisis periods with negative prices).
• Robust transfer requires a hybrid state representation that combines:
  1. Price scale (absolute features) to understand the magnitude of opportunities.
  2. Recent relative context to adapt to changing price levels.
  3. Short-horizon forecast information to anticipate future conditions.
• While the combined approach significantly improves performance, it does not solve every market setting (e.g., Nordic zones with low intra-day spreads remain challenging), indicating that state design is necessary but not sufficient for universal success.

The authors conclude that for model-free RL in energy trading, agents require access to price scale, relative position, and forecast information simultaneously to maintain effectiveness under changing market conditions.