Voltage support strength-constrained transmission network expansion planning for renewable-integrated grids via deep reinforcement learning

1 Introduction

Globally, the urgent need to combat climate change and the pursuit of sustainable development are driving a profound transformation in the energy structure. To achieve carbon neutrality goals, the energy systems of nations worldwide are shifting from a traditional fossil fuel-dominated model to a new paradigm centered on renewable energy sources (RES) (Zhang and Liu, 2025; Blaabjerg et al., 2023). Among these, new energy sources like wind and solar power, characterized by their clean and carbon-free nature, are experiencing unprecedented growth in their share of global installed capacity, becoming the core drivers of this energy revolution. However, the integration of high-penetration renewables also presents unprecedented challenges to the planning and operation of power systems (Wang et al., 2025).

Firstly, the output of RES such as wind and solar exhibits significant intermittency and volatility. Their generation patterns are fundamentally different from those of traditional, controllable thermal or hydro-power units, placing immense pressure on maintaining real-time power balance and potentially leading to issues like wind and solar curtailment. Secondly, renewable generation units are predominantly connected to the grid via Inverter-Based Resources (IBRs) (Rosso et al., 2021). The large-scale replacement of synchronous generators with IBRs causes a continuous decline in the overall system’s rotational inertia and a reduction in short-circuit capacity (Wang et al., 2024a). This diminishes the grid’s ability to support voltage and frequency during disturbances—a phenomenon known as weakened grid strength (Hosseinzadeh et al., 2021; Tuo and Li, 2022)—thereby introducing a new set of stability problems (Markovic et al., 2021).

Faced with these challenges, conventional transmission network planning methods have proven inadequate. Their models and algorithms struggle to effectively manage multiple uncertainties and have failed to incorporate new stability constraints, such as dynamic grid strength, into the core of planning decisions (Dahat and Dhabale, 2024). The difficulty is algorithmic in nature: classical TNEP is solved either by mathematical programming, which requires the feasible region to be expressed as closed-form (mixed-integer) convex constraints, or by heuristics that repeatedly evaluate a fixed analytical objective. A dynamic grid-strength index such as the short-circuit ratio, however, depends on the network Thevenin impedance and the instantaneous renewable injections through a non-convex, topology-dependent mapping that has no compact closed form and must be recomputed for every candidate topology, operating scenario, and contingency. This couples the binary investment decisions, the AC power-flow state, and a per-scenario stability evaluation into a single non-convex problem whose constraint set explodes with the number of scenarios and N-1 cases, rendering exact mathematical programming intractable and leaving heuristics unable to embed the stability index analytically. This motivates a learning-based decision maker (DRL) operating over a compact, representative scenario set obtained by data-driven reduction (clustering), rather than over the full annual time series. Therefore, researching and developing a novel grid planning framework that can simultaneously consider economy, renewable energy accommodation, and system dynamic stability has become a critical scientific problem that urgently needs to be addressed in the power systems field.

Academia and industry have undertaken extensive research to address these challenges. Transmission Network Expansion Planning (TNEP), a classic non-convex, large-scale combinatorial optimization problem, has seen its solution methods evolve from traditional mathematical programming (e.g., Linear Programming, Mixed-Integer Nonlinear Programming) to heuristic algorithms (e.g., Genetic Algorithms, Particle Swarm Optimization) (Mahdavi et al., 2015; Morquecho et al., 2023) in the quest to find high-quality, near-optimal solutions within an acceptable timeframe. To accurately capture the uncertainty of renewables, data-driven and intelligent methods have gained significant attention for their ability to extract key features from massive historical operational data (Gao et al., 2024; Wang et al., 2016; Guo et al., 2020). By using clustering algorithms like K-Means, thousands of hours of annual operational data can be reduced to a finite set of representative scenarios, thereby greatly enhancing the computational efficiency of planning models while maintaining accuracy.

Nevertheless, conventional scenario-reduction schemes typically cluster on raw samples or hand-crafted statistical descriptors (means, variances, ramp rates), which treat each time window as a static feature point and discard the underlying nonlinear temporal dynamics. As a result, windows with similar averages but markedly different fluctuation rhythms or persistence are easily merged, and the resulting “typical” scenarios may misrepresent the multi-scale dynamic behavior that drives both renewable accommodation and grid-strength stress. The Koopman operator framework offers a principled remedy: by lifting the nonlinear generation process into a latent space in which its evolution is approximately linear, it exposes interpretable spectral attributes—modal frequency, decay, and activation strength—that compactly summarize the dynamics of each window. Clustering in this spectral-feature space therefore groups scenarios by genuine dynamical similarity rather than instantaneous magnitude. This is precisely why a neural Koopman embedding is introduced ahead of clustering in this work, in place of clustering directly on raw outputs.

As renewable energy penetration levels rise, the Short-Circuit Ratio (SCR)—a core quantitative metric for grid strength at the point of interconnection—has become a key factor constraining the stable integration of RES. However, most existing planning studies that consider SCR use a static assessment based on the rated capacity of equipment, which fails to reflect the dynamic impact of real-time variations in renewable energy output on grid strength (Sun et al., 2024; Zhou et al., 2023). In recent years, artificial intelligence technologies, particularly Deep Reinforcement Learning (DRL), have offered a new paradigm for solving planning tasks that require multi-stage, sequential decision-making, such as grid expansion (Zhou et al., 2024; Wang et al., 2024b; Liu et al., 2024; Wu et al., 2024; Kamruzzaman et al., 2021). Among the various DRL algorithms, the Double Deep Q-Network (DDQN) effectively mitigates the overestimation problem (van Hasselt et al., 2016) found in traditional Q-Learning by decoupling action selection and value evaluation, demonstrating superior performance in handling problems with discrete action spaces (An and Wang, 2023; Jin and Xu, 2020; Teng et al., 2025).

Based on the preceding analysis, this paper aims to bridge the gaps in existing research concerning the integrated handling of renewable energy uncertainty, dynamic stability constraints, and intelligent decision-making. The main innovations and contributions are as follows:

• A power grid planning framework based on the Markov Decision Process (MDP) is proposed, integrating typical scenario extraction for renewable energy, grid strength assessment under renewable generation uncertainty, and intelligent decision-making. By coupling data-driven front-end analysis with intelligent back-end execution, the framework establishes a comprehensive methodology tailored for power systems with high renewable energy penetration.
• A typical scenario extraction method for renewable energy fluctuations is developed using the Neural Koopman approach. The short-circuit ratio (SCR) across multiple substations is employed to quantify the grid’s voltage support capability over the full cycle of these scenarios. This metric is incorporated as a key constraint and optimization objective, and is further embedded into the reward function design of a reinforcement learning model. Unlike traditional approaches that treat MRSCR as a static boundary condition, this study regards it as a dynamic, actively managed indicator that varies with renewable output—enabling the planning process to proactively maintain the system’s dynamic stability.
• A high-performance planning model solution method based on deep reinforcement learning is developed, allowing the model to autonomously learn how to enhance the voltage support capability of the grid while satisfying economic and renewable energy integration targets. With a carefully designed reward mechanism, the agent learns to identify and reinforce structural weaknesses in the grid during its search for optimal expansion strategies, ultimately producing a more resilient and reliable blueprint for future power grid development.

Section 2 details a typical scenario extraction method for renewable energy based on the Neural Koopman framework and develops a transmission network expansion planning model that incorporates the voltage support capability of renewable energy systems. Section 3 introduces a solution approach for the planning model using the DDQN algorithm. In Section 4, the proposed methodology is validated under multiple scenarios using the CEPRI-VC-88 system. Finally, Section 5 summarizes the paper and discusses potential directions for future research.

2 Transmission network planning model incorporating system voltage support capability

To tackle the multifaceted challenges associated with large-scale renewable energy integration, this section develops a closed-loop transmission network expansion planning framework that incorporates data-driven scenario analysis, grid strength evaluation, and the formulation of a transmission network planning model.

2.1 Neural koopman–based extraction of typical renewable generation scenarios

Given the strong time variability and nonlinear coupling in wind and photovoltaic (PV) generation, traditional scenario-extraction methods based on statistical descriptors or empirical modes (Gröwe-Kuska et al., 2003; Bessa et al., 2012) often struggle to coherently capture multi-scale fluctuation patterns together with interpretable dynamical attributes in a unified representation space. Recent studies have increasingly turned to graph-based representations to model the dynamic spatio-temporal dependencies among renewable plants, for instance by learning adaptive inter-farm coupling for wind power forecasting via sparse graph attention networks (Qu et al., 2026); such approaches confirm that capturing dynamic dependency structure, rather than static statistics, is key to representing renewable variability. To address this, we propose a Neural Koopman–centric pipeline: a neural network first learns Koopman observables to embed nonlinear time series into a latent space with approximately linear evolution; next, spectral decomposition and modal quantification are conducted within this linear space to form a unified feature vector with clear physical–statistical meaning; finally, clustering is performed in this feature domain and cluster representatives are selected to yield typical scenarios that can be directly used by downstream planning and simulation.

Let the multi-type renewable generation and exogenous drivers be defined as in Equation 1:where x t may include plant- or region-level wind/PV outputs and derived statistics, and u t may include irradiance, wind speed, temperature/humidity, and day-type/time-slot encodings. The real system is abstracted as the nonlinear controlled discrete-time dynamics given in Equation 2:

x t ∈ R n x , u t ∈ R n u , t = 0,1 , … , T , (1)

x t + 1 = f x t , u t + ε t , (2)

where ε t denotes modeling/measurement noise. Neural Koopman aims to learn data-driven observables that lift ( x t , u t ) to a latent space in which the evolution is approximately linear, so that prediction, extrapolation, and spectral analysis are governed by a linear operator.

Accordingly, we build an encoder–linear–decoder architecture. The encoder ϕ θ : R n x → R n z maps raw outputs to latent states z t = ϕ θ ( x t ); the latent dynamics follow the controlled linear model given in Equation 3:with w t unmodeled disturbances; the decoder ψ ω : R n z → R n x reconstructs x ̂ t = ψ ω ( z t ). To ensure consistency across lifting, linear evolution, and reconstruction, we couple a multi-step consistency loss and a reconstruction loss. For sample starts { t i } i = 1 N and prediction horizon H, these two losses are defined in Equations 4, 5, respectively:

z t + 1 = A z t + B u t + w t , (3)

L dyn = 1 N ∑ i = 1 N ∑ h = 1 H z t i + h − A h z t i + ∑ k = 0 h − 1 A h − 1 − k B u t i + k 2 2 , (4)

L rec = 1 N ∑ i = 1 N ∑ h = 0 H x t i + h − ψ ω A h z t i + ∑ k = 0 h − 1 A h − 1 − k B u t i + k 2 2 . (5)

To enhance generalization and numerical stability, we regularize the linear operator with stability and sparsity priors. Discrete-time stability can be enforced via the spectral-radius or Lyapunov inequalities in Equation 6:

ρ A ≤ 1 − δ δ > 0 , or ∃ P ≻ 0 : A ⊤ P A − P ⪯ − η I , (6)

And we apply ℓ 1 regularization to A , B to promote interpretable modal couplings. The end-to-end optimization problem is given in Equation 7:

min θ , ω , A , B L = α L dyn + β L rec + γ λ A ‖ A ‖ 1 + λ B ‖ B ‖ 1 . (7)

Training uses sliding windows to form samples ( x t : t + H , u t : t + H − 1 ), with a least-squares warm start for ( A , B ) followed by joint fine-tuning with ( θ , ω ), balancing the physical interpretability of the linear layer and the nonlinear expressiveness of the encoder.

After convergence, we obtain the linear operator A. Its eigendecomposition is given in Equation 8:

A = V Λ V − 1 , Λ = diag λ 1 , … , λ n z , (8)

Equation 8 provides approximate Koopman modes in the latent space. Each eigenvalue λ k encodes both gain/decay and oscillation: | λ k | measures persistence, arg ( λ k ) gives discrete angular frequency; with sampling step Δ t, the continuous frequency and decay time are obtained from Equation 9:

ω k = arg λ k Δ t , τ k = − Δ t ln | λ k | . (9)

To quantify per-window “activation,” we compute modal coefficients from the window’s initial latent state z t 0 using Equation 10:

α = V − 1 z t 0 , α k is the weight of mode k . (10)

A unified spectral feature vector is then constructed for each window. Selecting the top K dominant modes (by energy or amplitude), we define the spectral feature vector in Equation 11:

f = | λ 1 | , ω 1 , | α 1 | , … , | λ K | , ω K , | α K | ⊤ ∈ R 3 K , (11)

This vector is optionally augmented with the robust statistics in Equation 12 to improve intra-cluster separability:

ω ̄ = 1 K ∑ k = 1 K ω k , τ ̄ = 1 K ∑ k = 1 K τ k , E spec = ∑ k = 1 K | α k | 2 , (12)

These statistics are concatenated with f as the final representation. Here, | λ k | captures fluctuation persistence, ω k captures principal rhythms (diurnal, weather-driven bands), and | α k | reflects the activation strength of each mode in the current window.

Clustering in this unified spectral-feature domain yields scenario prototypes. After standardization of { f ( i ) } i = 1 M, K-means solves the clustering problem in Equation 13:where μ c is the center of cluster c and K c can be chosen by the silhouette score or elbow method. For representative “typical days/weeks,” the nearest-centroid sample (or medoid) within each cluster is selected according to Equation 14:

min μ c , c i ∑ i = 1 M f i − μ c i 2 2 , c i ∈ 1 , … , K c , (13)

i c ⋆ = arg min i : c i = c f i − μ c 2 . (14)

The original window x t : t + H associated with i c ⋆ is exported as the typical scenario. When generative scenarios are preferred for planning sensitivity studies, the linear-space roll-out is given in Equation 15:

z ̂ t + h = A h z t + ∑ k = 0 h − 1 A h − 1 − k B u t + k , x ̂ t + h = ψ ω z ̂ t + h , (15)

Equation 15 provides trajectories that preserve modal structure and exogenous forcing, systematically covering adverse or extreme fluctuation combinations.

In summary, the proposed method embeds nonlinear generation processes into an interpretable linear spectral space via Neural Koopman, builds robust features from modal strength, persistence, and rhythm, and applies K-means to achieve efficient clustering and scenario extraction. Compared with conventional feature engineering or purely statistical clustering, this tightly coupled “nonlinearity → linear spectral features → clustering” paradigm internalizes predictive and representation consistency within a single training objective, producing scenarios that are both data-representative and dynamically interpretable for planning assessment.

2.2 Evaluation of voltage support strength in power systems based on real-time power

To characterize the grid’s voltage support capability under multi-plant integration, this study adopts the Multiple Renewable-energy Stations Short-Circuit Ratio (MRSCR) as a unified metric. Physically, at interconnection node i, MRSCR is the ratio between the grid’s short-circuit support and the “composite load” equivalently mapped to that node by multiple renewable plants through network coupling, thereby capturing both the local plant’s injection and the influence of other plants via mutual impedances. Let Z eq ∈ C n × n denote the network Thevenin equivalent impedance matrix in the frequency band of interest, S i , ω RE = P i , ω RE + j Q i , ω RE the complex power of plant i in scenario ω, and S i ac the per-unit short-circuit capacity at node i. The general complex-valued expression of MRSCR is given in Equation 16:where Π i j = Φ Z i j eq , Z i i eq , ∠ U i − ∠ U j represents the equivalence factor induced by mutual impedances and node phase/voltage-magnitude differences to map plant j’s impact onto node i. This definition makes MRSCR sensitive to both topology and operating point: the index increases with stronger short-circuit support S i ac or weaker coupling, and decreases with higher injections (including reactive set-points), stronger coupling, or unfavorable phase differences. For engineering cases with sufficiently high line X / R and small angle spreads, the real-valued approximation in Equation 17 improves efficiency:

M R S C R i , ω = S i ac S i , ω RE + ∑ j ≠ i Π i j S j , ω RE , (16)

M R S C R i , ω ≈ S i ac P i , ω RE + ∑ j ≠ i λ i j P j , ω RE , λ i j = | Z i j eq | | Z i i eq | , (17)

Whose form mirrors a weighted composite active-power load, directly revealing how mutual impedances amplify and propagate effects. To establish a clear engineering criterion, this work adopts a target station-side MRSCR threshold interval τ ∈ [ 2.5 , 3.3 ]. The bounds follow the widely accepted grid-strength classification in which a short-circuit ratio below about 2 denotes a very weak connection prone to control-loop instability, around 3 a marginally adequate connection, and above 3 a strong connection (CIGRE Working Group B4.62, 2016; IEEE, 1997). The interval is further refined to the station side by accounting for the additional series impedance introduced by the step-up transformer and the collector/interconnection line between the device terminals and the point of common coupling: for typical transformer short-circuit impedances (≈ 0.10–0.14 p. u.) and line impedances, the device-side minimum operable ratio of ∼ 1.5–2.0 maps to a station-side requirement of roughly 2.5 (lower bound, basic operability) to 3.3 (upper bound, a stronger margin-bearing objective). For measurement and comparison, we define the node-scenario MRSCR margin in Equation 18:

Δ i , ω = M R S C R i , ω − τ , (18)

With Δ i , ω ≥ 0 indicating sufficient voltage support and Δ i , ω < 0 indicating a weak node–scenario pair.

The typical renewable output scenarios used herein are produced by prior work: historical data are embedded into a linear subspace via a neural Koopman mapping, features are extracted, and clustering yields representative scenarios that capture principal modes and magnitudes without overfitting instantaneous extremes. Given the scenario set Ω, injecting ( P j , ω RE , Q j , ω RE ) into Z eq and S i ac produces the grid-wide fields of M R S C R i , ω and Δ i , ω. The statistics min ω ∈ Ω Δ i , ω and min i , ω Δ i , ω are then used to locate the weakest node–scenario combinations. It should be emphasized that time-varying renewable outputs directly modify the denominator’s composite equivalent load, while network changes (e.g., line switching, tap adjustments, reactive device switching, transformer parameter optimization) simultaneously affect the numerator S i ac and the mutual-coupling terms in the denominator, causing the MRSCR margin to rise or fall across scenarios and even drop below the threshold.

To ensure consistent voltage support across all scenarios at the planning level, the MRSCR requirement M R S C R i , ω ≥ τ is incorporated into the network expansion and parameter-optimization model. Because the index is evaluated inside a learning-based decision loop (Section 3) and is non-differentiable and topology-dependent, it is enforced in two complementary stages. During policy search it is handled as a soft constraint: the aggregated margin Δ agg = min i , ω Δ i , ω enters the reward as a penalty/incentive term, steering the agent away from weak-node–scenario pairs without requiring an analytical feasible set. The candidate plan returned by the agent is then subjected to an exact feasibility verification in which M R S C R i , ω is recomputed for every node and every typical scenario using the full (non-surrogate) impedance model; only plans satisfying M R S C R i , ω ≥ τ across all representative scenarios are accepted. Through this “typical-scenario → index evaluation → exact verification” loop, the final plan satisfies operational and dynamic-stability requirements and delivers sufficient voltage support in all representative renewable output scenarios, yielding an implementable engineering solution.

2.3 Construction of the transmission network planning model

The Transmission Network Expansion Planning (TNEP) model is composed of an objective function and a set of constraints. The objective function proposed in this study integrates system investment costs (construction and maintenance) with the expected operational costs. The constraints are primarily designed to ensure reliable system operation under the N-1 contingency condition. The formulation of the TNEP model is given in Equation 19:

min F X = r d 1 + r d y 1 + r d y − 1 ∑ l = 1 N L λ i C l , i + η l ∑ l = 1 N L λ i C l , i + E ω , c p 0 ∑ l = 1 N L P l 2 + Q l 2 U l 2 r l + C curt ∑ j P wind , j curt + C shed ∑ j P load , j shed − χ Δ agg (19)

In this formulation, the objective function incorporates: line construction cost, maintenance cost, and the expected costs/incentives over all scenarios ( ω ) and N-1 contingencies ( c ). These include network loss cost, wind curtailment cost, load shedding cost, and a voltage support incentive. Where r d is the discount rate of line investment, y is the expected service life of the line, λ i ∈ [ 0,1 ] is a binary variable indicating whether line i is constructed (Zhang and Liu, 2025) or not (0), and C l , i denotes the construction cost of line i. p 0 is the unit electricity price, P l and Q l represent the active and reactive power flows on line l, U l is the voltage magnitude, and r l is the resistance of line l. η l is the annual maintenance cost coefficient. C curt and C shed are the high penalty costs for wind curtailment and load shedding, respectively. χ Δ agg represents the aggregated MRSCR margin ( Δ agg = min i , ω , c Δ i , ω , c ), which the model seeks to maximize.

In transmission network planning, the N-1 contingency check is the most fundamental and effective method for ensuring system reliability. The core idea is to evaluate whether the transmission network can continue to operate stably under a new system state following the failure of any single component. If the stability conditions are still met, the system passes the N-1 check; otherwise, it fails. The model constraints under the N-1 contingency condition (and for each scenario ω) are formulated as follows. Nodal power-balance equations: the nodal active- and reactive-power balance equations (for all j) are given in Equations 20, 21, respectively:

P j g , N − 1 − P wind , j curt , N − 1 − V j N − 1 ∑ k ∈ j G j k ⁡ cos ⁡ θ j k N − 1 + B j k ⁡ sin ⁡ θ j k N − 1 = P j load , N − 1 − P load , j shed , N − 1 , (20)

Q j g , N − 1 − Q wind , j curt , N − 1 − V j N − 1 ∑ k ∈ j G j k ⁡ sin ⁡ θ j k N − 1 − B j k ⁡ cos ⁡ θ j k N − 1 = Q j load , N − 1 − Q load , j shed , N − 1 − Q j c , N − 1 . (21)

Operational and security limits: the limits on the bus voltage, the phase angle, the generator reactive output, the line flow, the wind curtailment, and the load shedding are given in Equations 22–27, respectively: where P j g represents the rated active power output of the generator at node j, and P wind , j curt denotes the curtailed wind power at node j. V j is the voltage magnitude at node j, while B j k and G j k represent the susceptance and conductance between nodes j and k, respectively. The term θ j k indicates the phase angle difference between nodes j and k. P j load and P load , j shed refer to the active power demand and load shedding at node j, respectively. Similarly, Q j g is the reactive power output of the generator, and Q j load and Q load , j shed are the reactive power demand and shed amount at node j. Q wind , j curt denotes the curtailed reactive wind power, and Q j c represents the reactive power provided by compensation devices at node j. V j min and V j max are the lower and upper bounds of voltage magnitude, while θ j min and θ j max are the corresponding phase angle limits. P j g , min and P j g , max denote the minimum and maximum active power output of the generator, and Q j g , min and Q j g , max denote the corresponding reactive power limits. P j wind , min and P j wind , max are the minimum and maximum active power output of the wind farm. F l is the power flow on line l, and F l max is its maximum allowable apparent power. The parameters μ j wind and μ j load define the maximum allowable curtailment/shedding ratios. The superscript N − 1 indicates system state variables under the N − 1 contingency condition.

V j min ≤ V j N − 1 ≤ V j max , b u s v o l t a g e (22)

θ j min ≤ θ j N − 1 ≤ θ j max , p h a s e a n g l e (23)

Q j g , min ≤ Q j g , N − 1 ≤ Q j g , max , r e a c t i v e o u t p u t (24)

− F l max ≤ F l N − 1 ≤ F l max , l i n e fl o w (25)

0 ≤ P wind , j curt , N − 1 ≤ μ j wind P wind , j N − 1 , c u r t a i l m e n t (26)

0 ≤ P load , j shed , N − 1 ≤ μ j load P load , j N − 1 . l o a d s h e d d i n g (27)

3 DRL-based solution method for transmission network planning

The TNEP model formulated in Section 2.3 constitutes a large-scale, N-1 constrained, stochastic Mixed-Integer Nonlinear Program (MINLP). The immense computational complexity stems from three primary sources:

• The combinatorial nature of the binary investment decisions x ∈ { 0,1 } | L cand |;
• The non-linear, non-convex dependence of the AC power flow constraints and the dynamic MRSCR constraints on the network topology x;
• The need to evaluate all operational and stability constraints (including N-1 contingencies) across the entire set of typical scenarios S scen and contingencies ( c ).

These characteristics render traditional mathematical programming solvers computationally intractable for realistic systems.

To overcome this, we reformulate the TNEP problem from a static optimization into a sequential decision-making task, making it amenable to a DRL solution. By formulating the TNEP problem as a Markov Decision Process, a DRL agent can learn an optimal planning policy π * through interaction with a simulated grid environment, effectively navigating the vast search space.

3.1 Environment simulation via surrogate models

A significant challenge in applying DRL to TNEP is the intractable reward problem. The DRL agent requires millions of training steps to converge. In our formulated MINLP, calculating the true reward for a single action A t (building one line) would require:

• Evaluating the new topology G t + 1 across all scenarios s ∈ S scen and N-1 contingencies c.
• For each scenario/contingency, solving a complex, non-convex AC Optimal Power Flow (AC-OPF) problem to find the operational costs (e.g., C loss, P curt, P shed).
• For each scenario/contingency, calculating the complex, non-linear MRSCR margins ( Δ i , ω ).

This is computationally infeasible, as it involves solving thousands of MINLPs at every single step of the DRL training.

To create a tractable simulation environment, we introduce a set of pre-trained surrogate models (or proxy models). These models are lightweight neural networks that rapidly approximate the grid’s performance without solving the full optimization. Given the graph structure of the grid, we use Graph Neural Networks (GNNs) for this task.

We pre-train two surrogate GNNs, M op and M stab, using an offline dataset generated by evaluating the operational and stability cost functions on thousands of sample topologies: where Δ ̂ agg approximates the aggregated MRSCR margin ( E c [ min i Δ i , s , c ] ).

• Operational Cost Surrogate ( M op ): This model predicts the expected operational costs for a given topology G and scenario s, corresponding to the operational cost terms in Equation 19, as expressed in Equation 28:
• Stability Surrogate ( M stab ): This model predicts the expected aggregated MRSCR margin, corresponding to the incentive term in Equation 19, as expressed in Equation 29:

Δ ̂ agg G , s = M stab G , s ; θ stab (29)

During DRL training, the agent interacts with this fast surrogate environment, which uses M op and M stab to return rewards, allowing for rapid learning.

3.1.1 Surrogate dataset and architecture

The offline training set is generated by sampling N topo = 5,000 candidate topologies via random and heuristic perturbations of the base grid; for each topology, the ground-truth operational costs (from AC-OPF) and exact MRSCR margins are computed over all typical scenarios, yielding N sample ≈ 30,000 topology–scenario labelled pairs. The data are split into training/validation/test subsets at a ratio of 80:10:10. Both M op and M stab are message-passing GNNs that encode the topology with 3 graph-convolutional layers (hidden dimension 128, ReLU activation), followed by a global mean-pooling readout and a two-layer MLP regression head; they are trained with the Adam optimizer for 300 epochs minimizing the mean-squared error. The prediction accuracy on the held-out set is summarized in Table 1.

TABLE 1. Architecture and validation accuracy of the surrogate GNNs.

3.2 TNEP-MDP formulation

With the fast surrogate environment, we define the TNEP-MDP using the tuple ( S , A , P , R , γ ).

• State ( S ): The state S t ∈ S represents the grid topology at decision step t. We define S t = G t = ( V , E t ), where V is the static set of buses and E t is the set of transmission lines existing at step t. The state is featurized using graph-based node and edge attributes.
• Action ( A ): The action A t ∈ A is the discrete decision to construct a single line l from the set of candidate lines L cand. The action space also includes a “terminate” action, A t = ∅, allowing the agent to end the planning episode.
• Transition ( P ): The state transition P ( S t + 1 | S t , A t ) is deterministic. Executing A t = l (where l ≠ ∅) transitions the state from S t = G t to S t + 1 = G t ∪ { l }.
• Reward ( R ): The reward function is designed to be the negative of the total cost defined in the TNEP objective function (Equation 19).

Let r ̂ ( G , s ), defined in Equation 30, be the approximated operational cost of topology G under scenario s, returned by the surrogates. This corresponds to the expected operational (non-investment) terms in Equation 19:

r ̂ G , s = w loss C ̂ loss G , s + w curt C ̂ curt G , s + w shed C ̂ shed G , s − w stab Δ ̂ agg G , s (30)

The reward function R is designed to align with the static objective F ( X ) in Equation 19 by using a sparse, terminal reward structure.

• Step Reward ( A t ≠ ∅ ) : For any intermediate action A t (building a line), the agent receives an immediate reward equal to the negative annualized investment and maintenance cost, as given in Equation 31:

R t + 1 = − C invest A t + C maint A t (31)

• Terminal Reward ( A t = ∅ ) : When the agent takes the terminate action, it receives a large final reward R terminal, as given in Equation 32. This reward is the negative expected operational cost of the final topology GT, approximated by the surrogates (using r ̂ from Equation 30):

R terminal = − E s ∈ S scen r ̂ G T , s (32)

The agent’s goal is to maximize the cumulative discounted reward, ∑ t = 0 T γ t R t + 1. Since all step rewards are negative investment costs and the final reward is the negative operational cost, this objective directly optimizes the total cost defined in Equation 19.

• Discount Factor ( γ ): γ ∈ [ 0,1 ] is a discount factor balancing immediate investment costs versus future operational rewards.

3.3 Agent architecture: dueling GNN-DQN

To solve this MDP, we design an agent based on the Double Deep Q-Network (DDQN) algorithm. The agent learns an optimal state-action value function Q * ( S , A ), which estimates the maximum expected cumulative discounted reward.

Given the graph-structured state S t = G t, the agent’s Q-network Q ( S , A ; θ agent ) is implemented using a Graph Neural Network (GNN). This “policy GNN” is distinct from the surrogate GNNs ( M op , M stab ) used by the environment.

The policy GNN takes the grid topology G t as input and learns latent feature representations for nodes and edges by aggregating neighborhood information. This allows the network to capture complex topological dependencies. We further enhance the architecture using the Dueling DQN framework. The GNN’s output (a graph-level embedding z G) is fed into two separate streams:

• A state-value stream V ( z G ; θ V ), which estimates the value of being in state S t.
• An advantage stream A ( z G , A ; θ A ), which estimates the relative advantage of taking each action A ∈ A.

The final Q-values are combined as in Equation 33:

Q S t , A ; θ = V S t ; θ V + A S t , A ; θ A − 1 | A | ∑ A ′ ∈ A A S t , A ′ ; θ A (33)

This dueling structure stabilizes learning by decoupling the value of the state from the value of the actions.

3.4 Training with DDQN

We use DDQN to mitigate the overestimation bias of standard Q-Learning. DDQN uses a main network Q ( S , A ; θ ) and a target network Q ( S , A ; θ − ). The target Q-value Y t DDQN, given in Equation 34, is calculated by decoupling action selection (using θ t) from action evaluation (using θ t −):

Y t DDQN = R t + 1 + γ Q S t + 1 , arg max A ′ Q S t + 1 , A ′ ; θ t ; θ t − (34)

The main network is updated by minimizing the mean squared TD error in Equation 35, sampled from an experience replay buffer D:

L θ t = E S t , A t , R t + 1 , S t + 1 ∼ D Y t DDQN − Q S t , A t ; θ t 2 (35)

The target network parameters θ − are periodically copied from the main network parameters θ. The overall training procedure is summarized in Algorithm 1.

Algorithm 1 DRL-Based TNEP with Surrogate Environment.

4 Case study

To validate the effectiveness and practicality of the proposed deep reinforcement learning-based power grid planning framework, this section conducts a grid topology planning study on the representative high-renewable-penetration system, CEPRI-VC-88. The analysis encompasses the full workflow—from generating uncertainty scenarios to evaluating the performance of the final planning scheme—and compares the results with benchmark methods to highlight the advantages of the proposed framework in enhancing renewable energy integration and ensuring dynamic stability of the power grid.

4.1 Test system setup

The CEPRI-VC-88 system comprises 100 buses, with the 500 kV main transmission network accounting for 48 of them. The installed capacity includes 2.4 GW of renewable energy and 6.3 GW of conventional generation, resulting in a capacity ratio of 1:2.62. The renewable energy contributes 1.8 GW of actual generation, consisting of 1.2 GW from wind power and 0.6 GW from photovoltaic sources, while the rated capacity of the conventional units totals 3.86 GW. Additionally, the system incorporates a single high-voltage direct current (HVDC) transmission line, delivering 0.8 GW of power. To ensure stable system operation, the coordination between renewable and conventional power sources is also taken into account. The system structure is shown in Figure 1, and the parameters of system are shown in Tables 2, 3.

TABLE 2. Overview of transmission lines and transformers in the CEPRI-VC-88 system.

TABLE 3. Load parameters of the CEPRI-VC-88 system.

To construct the power grid planning demand scenarios, this study increases the capacity of conventional generators by 15% and expands the base load by 10%, while applying a substantial 25% capacity increment to wind farms. In addition, dynamic adjustable load modules are deployed at buses 3, 14, 28, 36, 40, 44, 57, and 69.

To avoid information leakage between model fitting and performance evaluation, the data are explicitly partitioned. The N sample ≈ 30,000 topology–scenario pairs used to fit the surrogate GNNs are split into training, validation, and test subsets at a ratio of 80:10:10 (Section 3.1), and all surrogate accuracy metrics are reported on the held-out test subset. The held-out test subset contains topologies not used in training, so the reported accuracy reflects the surrogates’ generalization to unseen network configurations rather than memorization. The representative scenario set obtained in Section 3.1 constitutes the fixed evaluation set over which the planning policy is assessed; to eliminate any residual surrogate bias, the curtailment rate, costs, and MRSCR margins reported in this section for the final plan are recomputed with the exact (non-surrogate) models rather than read from the surrogates.

4.2 Generation of typical operating scenarios for wind farms

We collected 1 year of historical wind power output and load data from the target region, comprising 8,760 hourly records. Leveraging the Neural Koopman clustering algorithm introduced in Section 2, this large-scale dataset was categorized into multiple representative daily operational scenarios. As illustrated in Figure 2, the volatility observed in the wind power output curves highlights the inherent intermittency of renewable energy sources. These representative scenarios are utilized as standardized inputs to enable low-complexity assessments of the proposed voltage support performance across the full operational cycle.

As shown in Figure 2, the average output and fluctuation characteristics differ significantly among the scenarios. The Koopman-based algorithm enables the nonlinear renewable energy outputs to be projected into a linear latent space, thereby reducing clustering complexity and improving the accuracy of typical scenario extraction. The generated scenarios encompass both normal and extreme operating conditions of the system. Moreover, the temporal resolution of the data is adequate for planning tasks, which are generally insensitive to fine-grained operational data. As a result, this data-driven scenario generation approach reduces computational complexity while preserving the intrinsic uncertainty of system operations.

To verify that the selected cluster representatives are genuinely typical rather than arbitrary samples, we assess them from two angles: cluster-partition quality and distribution fidelity. The number of clusters is chosen by the silhouette/elbow criterion, and the resulting partition quality together with the agreement between the representative scenario set and the full annual dataset is reported in Table 4. A high silhouette coefficient indicates compact, well-separated clusters, so that each representative faithfully stands for its group; meanwhile, the small relative error in the mean output and the high probability-coverage confirm that the compact scenario set preserves the statistical characteristics of the original year-long series.

TABLE 4. Validity of the extracted typical scenarios: cluster-partition quality and fidelity to the full annual dataset.

4.3 Planning results and analysis in CEPRI-VC-88 system

Based on the experimental setup described above, the proposed DDQN algorithm is applied to solve the transmission network planning model. To evaluate its effectiveness, we compare it with the standard Deep Q-Network (DQN) algorithm, as a representative deep reinforcement learning baseline, and with Particle Swarm Optimization (PSO), a representative heuristic algorithm. The planning results obtained by the three methods on the CEPRI-VC-88 system are presented in Tables 5–7, respectively.

TABLE 5. Planning results of the CEPRI-VC-88 system using the DDQN algorithm (All costs in million USD).

TABLE 6. Planning results of the CEPRI-VC-88 system using the DQN algorithm (All costs in million USD).

TABLE 7. Planning results of the CEPRI-VC-88 system using the PSO algorithm (All costs in million USD).

PSO, is a population-based heuristic that returns a single one-shot plan in which all candidate lines are committed simultaneously, rather than as a sequential build-up. Hence only the aggregate cost and stability metrics of the final plan are reported; blank cells are intentional.

The results demonstrate that the DDQN algorithm offers significant advantages in addressing complex power grid expansion tasks. Notably, DDQN achieves a more effective balance among various cost components, delivering a more consistent performance than both DQN and PSO. For instance, DDQN reduces line construction costs by approximately $4.32 million compared to PSO, highlighting its superior ability to identify cost-effective network expansion strategies. Additionally, DDQN shows enhanced capability in maintaining system voltage support, achieving the highest MRSCR margin of 1.57, which indicates a greater capacity to mitigate the risk of voltage instability under disturbances.

The strength of DDQN stems from its double-network design, in which action selection and action evaluation are performed by separate networks to stabilize learning. This decouples the evaluation of line construction value from the selection of optimal routes, effectively reducing the risk of overly optimistic planning decisions caused by Q-value overestimation. As a result, the agent can more accurately identify the true value of each potential line investment. Consequently, DDQN achieves a robust trade-off across different cost components, resulting in a total planning cost of $73.88 million, marking it as the most cost-efficient expansion strategy.

Moreover, deep reinforcement learning methods like DDQN possess inherent advantages in managing high-dimensional, nonlinear, and stochastic environments due to their ability to learn continuously through interaction with the system. This is particularly advantageous in power system expansion planning, where uncertainty in demand, renewable energy integration, and system constraints is prevalent. DDQN’s capacity to model and adapt to these uncertainties endows it with a marked advantage in developing optimal planning strategies, thereby enhancing both system reliability and cost-effectiveness.

4.3.1 Renewable accommodation

Beyond the monetary abandonment cost reported above, we evaluate the planning effect directly in terms of the annual wind curtailment rate, defined as the ratio of curtailed wind energy to total available wind energy aggregated over all typical scenarios weighted by their probabilities. As summarized in Table 8, the initial grid suffers a high annual curtailment rate of 12.8%, driven by congestion and insufficient delivery capability at the renewable interconnection points. As the DDQN agent sequentially reinforces the network, the curtailment rate decreases monotonically, and the final plan reduces the annual wind curtailment rate to 1.5%—an improvement consistent with the reduction of the abandonment-cost term in Table 5 and with the figure quoted in the abstract.

TABLE 8. Annual wind curtailment rate before and after the proposed planning (total available wind energy ≈ 3,900 GWh).

4.3.1.1 Exact stability verification

Because the MRSCR margin is treated as a soft penalty during policy search (Section 3) and is predicted by the surrogate M stab, the final plan returned by the agent is re-checked with the exact, non-surrogate MRSCR computation across all typical scenarios and N-1 contingencies. The minimum exact margin over all node–scenario pairs is min i , ω Δ i , ω = 0.21 ≥ 0 (equivalently M R S C R i , ω ≥ τ everywhere), with a maximum surrogate-versus-exact deviation of 4.8 %, confirming that the planned network satisfies the voltage-support requirement in every representative scenario and that no scenario with inadequate SCR remains. This exact post-verification closes the gap between the surrogate-based reward used in training and the true stability constraint required at deployment.

4.4 Algorithm performance verification

In this training simulation, we evaluated and compared the performance of DDQN and DQN algorithms across various hyperparameter settings, and plotted the cumulative reward curves over 5000 training episodes, as shown in Figure 3.

The training results clearly indicate that the DDQN algorithm surpasses DQN in terms of training efficiency, convergence speed, final performance level, and policy stability. Notably, DDQN begins to show significant learning progress around step 1000 and achieves rapid convergence within 2000 steps. Under its optimal hyperparameter configuration—a Q-network learning rate of 1 × 1 0 − 3 with the target network synchronized every C steps—the cumulative reward stabilizes at approximately 1.85. In contrast, various configurations of DQN exhibit substantial fluctuations throughout training. Even the best-performing DQN setting yields a final cumulative reward of only about 0.9, which is roughly half that of DDQN.

Furthermore, DDQN’s reward curve is significantly smoother, with a post-convergence standard deviation below ± 0.03. On the other hand, DQN’s reward curves continue to display frequent oscillations even after apparent convergence, with standard deviations typically ranging from ± 0.12 to ± 0.20. This indicates that DQN suffers from greater policy instability and less reliable convergence.

From a methodological perspective, DDQN’s advantage stems from decoupling action selection from action evaluation: the online network selects the greedy next action while the target network evaluates its value, which suppresses the systematic Q-value overestimation that standard DQN incurs from using a single network for both operations in the max term. Combined with the dueling architecture—which separately estimates the state value and the per-action advantages—and with a periodically synchronized target network and experience replay, this yields more accurate value estimates and markedly steadier updates. Standard DQN, by contrast, selects and evaluates actions with the same network, so the overestimation bias accumulates and propagates through bootstrapping, producing the larger oscillations and less reliable convergence observed above.

In conclusion, DDQN exhibits substantially superior overall performance in this task. Its optimal configuration delivers an approximate 95% improvement in performance, along with marked gains in training stability and policy reliability, reaffirming its advantage in reinforcement learning applications.

5 Conclusion

This paper focuses on the transmission network planning problem for systems with a high penetration of renewable energy. A tailored evaluation model and solution methodology are proposed, with improvements introduced in system assessment, algorithmic architecture, and solution precision. By integrating advanced technologies such as artificial intelligence, a practical and engineering-oriented algorithm is developed. The key contributions and conclusions are summarized as follows:

• An optimization model for transmission network expansion in high-renewable systems is constructed, enabling a unified quantitative representation of both economic efficiency and voltage support capability. This provides a solid theoretical foundation for network topology planning.
• A typical scenario extraction method for wind power output is developed based on the Neural Koopman algorithm. By leveraging Koopman eigenvalues, the method facilitates high-performance scenario identification while significantly reducing the computational burden of modeling renewable energy variability in the planning process.
• A DDQN-based transmission network expansion planning algorithm is proposed, which determines the optimal construction sequence of the planning scheme. This significantly enhances the practicality and adaptability of the planning strategy in real-world engineering applications.

It should be noted that the dynamic-stability requirement is enforced as a reward penalty during policy search and then guaranteed by an exact post hoc MRSCR verification; the residual gap between the surrogate prediction and the exact index, although bounded and corrected in this work, may grow on larger systems. Future work will therefore explore tighter surrogate fidelity and the direct embedding of safety-constrained reinforcement learning, so that the stability requirement can be respected throughout the search rather than only verified afterward.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Hao Wang, Monash University, Australia

Reviewed by: Kai Qu, Xi’an Jiaotong University, China

Zhenguo Shao, Fuzhou University, China