⚡ electrical engineering

A Reinforcement Learning-based Transmission Expansion Framework Considering Strategic Bidding in Electricity Markets

This paper proposes a multi-agent reinforcement learning framework that simultaneously optimizes transmission expansion planning and generator strategic bidding through a unified training process to capture their mutual influence in electricity markets.

Original authors: Tomonari Kanazawa, Hikaru Hoshino, Eiko Furutani

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tomonari Kanazawa, Hikaru Hoshino, Eiko Furutani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the electricity grid as a massive, bustling highway system, and the electricity market as a giant, high-stakes auction house where power is bought and sold every hour.

In this world, there are two main groups of players:

The Power Plants (Generators): They want to sell as much electricity as possible at the highest price to make a profit.
The Grid Managers (Transmission Operators): They are responsible for building and maintaining the "roads" (power lines) to make sure electricity can actually get from the plants to the homes.

The Problem: A Broken Feedback Loop

Traditionally, these two groups play a game of "guess and check" that often fails.

The Grid Managers try to plan new roads (transmission lines) based on what they think the power plants will do. They assume everyone plays fair and honest.
The Power Plants, however, are strategic. They know that if a road is congested (too much traffic), they can hold back their power and charge a premium price, just like a toll booth operator on a traffic jam.

The Analogy:
Imagine a city planner building a new bridge.

The Old Way: The planner assumes everyone will drive normally. They build a bridge, but once it's open, the drivers realize they can cause a traffic jam on the old roads to force everyone onto the new bridge, or they might take a different route to avoid tolls. The planner didn't account for the drivers' "tricks," so the bridge is either underused or causes new problems.
The Result: The city spends money on the wrong bridges, or the traffic jams get worse because the drivers are playing a game the planner didn't understand.

The Solution: The "Smart Coach" (Reinforcement Learning)

This paper introduces a new method using Reinforcement Learning (RL), which is like a super-smart coach training a team of video game characters.

Instead of the Grid Manager and the Power Plants playing separate games, the researchers put them all in the same training simulation.

The Training Camp (The Simulation):
- The computer creates a virtual world (the IEEE 30-bus system, which is like a small, test version of a real city grid).
- The Power Plants are "agents" (AI characters) that learn how to bid. They try different strategies: "If I bid high, do I make more money? If I bid low, do I sell more?"
- The Grid Manager is also an AI agent. It learns where to build new lines. It tries building a line here, then there, and sees what happens to the total cost.
The "Co-Optimization" (The Magic):
- In the past, the Grid Manager would build a road, then the Power Plants would react.
- In this new system, they learn simultaneously. The Grid Manager learns: "If I build a road here, the Power Plants will change their bidding strategy, which might actually lower the overall cost."
- The Power Plants learn: "If the Grid Manager builds a new road, my ability to charge high prices disappears, so I need a new strategy."

It's like a coach training a soccer team and a referee at the same time. The team learns how to play against a smart referee, and the referee learns how to officiate a team that is constantly trying to outsmart them. Eventually, they find a perfect balance where the game runs smoothly and efficiently.

What Did They Discover?

The researchers tested this on a model system and found some surprising things:

The "Two-Stage" Mistake: When they used the old method (Plan the road first, then see how the plants react), the Grid Manager often built too many roads. They assumed the plants would be aggressive and charge high prices, so they built extra capacity to handle it. But in reality, the plants adjusted their behavior, making that extra road a waste of money.
The "Smart Coach" Success: The new AI system found a "Goldilocks" solution. It built just the right amount of new lines. It realized that by adding a specific road, it could actually stop the power plants from being greedy, which saved money for everyone.

The Big Picture

Think of this framework as a simultaneous translator and negotiator.

Instead of the Grid Manager and the Power Plants speaking different languages and misunderstanding each other, this AI framework lets them "speak" to each other in real-time during the planning phase. It ensures that the roads we build are exactly what the market needs, preventing wasted money on unused bridges and avoiding the chaos of traffic jams caused by greedy drivers.

In short: This paper teaches us that to build a better electricity future, we can't just plan the roads in a vacuum. We have to simulate how the drivers (power companies) will actually behave, and let the road planners and the drivers learn the best strategy together.

1. Problem Statement

The paper addresses a critical gap in Transmission Expansion Planning (TEP) within liberalized electricity markets.

The Core Issue: Traditional TEP models often assume rational market equilibrium or fixed network configurations. They fail to account for the dynamic interplay between strategic bidding behaviors of Generation Companies (GENCOs) and transmission investment decisions.
The Coupling: In reality, network upgrades alter locational marginal prices and dispatch patterns, which in turn influence the economic value of those upgrades. Conversely, strategic bidding and congestion patterns determine the profitability of transmission expansions.
Limitations of Existing Methods:
- Multi-level optimization: Often assumes rationality and struggles with the complexity of strategic behavior.
- Operational RL: Simulates market dynamics but assumes fixed network topologies.
- Expansion RL: Optimizes long-term investment but often simplifies operational details into static cost functions, ignoring how strategic agents react to new infrastructure.

2. Methodology

The authors propose a Reinforcement Learning (RL)-based Co-optimization Framework that unifies market simulation and transmission design into a single learning process.

A. Market Simulation (Multi-Agent RL)

Environment: Modeled as a partially observable multi-agent system using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm.
Agents: Each GENCO acts as an autonomous agent aiming to maximize profit.
State & Action:
- Observation ( $o_i$ ): Includes total system demand, previous bid prices, and current transmission design parameters.
- Action ( $a_i$ ): A normalized action determining the bid price markup over marginal cost.
Market Clearing: A central Market Operator (MO) performs a DC Optimal Power Flow (DC-OPF) to clear the market based on submitted bids, subject to power balance and line flow constraints.
Reward: Agents receive rewards based on their realized profit (Clearing Price $\times$ Quantity - Marginal Cost $\times$ Quantity).

B. Co-optimization of Transmission Design

The novel contribution lies in treating transmission expansion as a learnable design policy that evolves alongside agent bidding strategies.

Design Variable ( $\omega$ ): Represented as a random variable sampled at the start of each episode.
- Continuous Case: Capacity increment ( $\Delta L_\ell$ ) sampled from a Gaussian distribution.
- Discrete Case: Binary siting decision ( $z_\ell \in \{0, 1\}$ ) sampled from a Bernoulli distribution.
Design Policy: A parameterized distribution $p_\mu(\omega)$ is learned to minimize the Total System Cost (Operational Cost + Expansion Cost).
Learning Loop:
1. Sample a network design $\omega$ from the current design policy.
2. Simulate the market episode where agents learn bidding strategies via MADDPG.
3. Calculate the total cost (negative return) for the episode.
4. Update the Design Policy ( $\mu$ ) using a policy gradient theorem to minimize total cost, while simultaneously updating Agent Policies to maximize profit.

3. Key Contributions

Unified Framework: First proposed RL-based framework for transmission expansion that simultaneously learns strategic bidding and investment decisions, bridging the gap between operational market simulation and long-term planning.
Design Policy Layer: Introduces a mechanism to treat network topology/capacity as a stochastic variable governed by a learnable policy, allowing the system to "co-evolve" infrastructure and market behavior.
Handling Strategic Behavior: Unlike traditional models that assume fixed or rational bids, this framework captures how strategic agents adapt their bidding to new network constraints, leading to more realistic investment outcomes.
Scalability: The approach is designed to handle complex, non-stationary environments and can scale to longer time horizons and multi-market settings better than traditional multi-level optimization.

4. Results (IEEE 30-Bus System Case Studies)

The framework was validated on the IEEE 30-bus system with two case studies:

Case 1: Continuous Expansion vs. Two-Stage Benchmark

Benchmark: A two-stage approach using PyPSA for expansion (assuming fixed "high" or "low" bidding) followed by RL market simulation.
Findings:
- The Two-Stage (High-Submission) benchmark over-invested (81.4 MW on Line 4-12) because it assumed uniformly high bids, leading to unnecessary capacity.
- The Proposed Co-optimization learned a more efficient expansion (76.1 MW on Line 4-12). By internalizing the strategic behavior, the RL agents adjusted their bids, revealing that slightly less capacity was needed to achieve optimal system costs.
- Outcome: The co-optimized solution achieved a lower total cost by avoiding over-investment driven by unrealistic static assumptions.

Case 2: Discrete Siting Decisions

Setup: Binary decisions to upgrade 6 candidate lines (fixed 50 MW increments).
Findings:
- The framework selected Lines 1-2 and 4-12 for expansion.
- Strategic Impact: The upgrade of Line 1-2 significantly reduced congestion in a specific corridor, which curtailed Gen 1's market power. Consequently, Gen 1 lowered its bid prices (from ~$90/MWh to ~$75.6/MWh).
- Outcome: Although the expansion cost increased due to the extra line, the reduction in operational costs (driven by reduced market power and congestion) resulted in a comparable or better total system cost. This demonstrates the framework's ability to identify investments that specifically target market inefficiencies caused by strategic behavior.

5. Significance

Paradigm Shift: Moves TEP from a sequential, static planning process to a dynamic, co-evolutionary learning process.
Policy Insight: Provides regulators and System Operators (SOs) with a tool to understand how infrastructure investments can mitigate market manipulation and strategic bidding, leading to more efficient market outcomes.
Future Applicability: The framework lays the groundwork for planning in increasingly complex, decentralized energy systems (e.g., high renewable penetration, multi-market coupling) where traditional optimization methods struggle with non-linearity and strategic uncertainty.