Dynamic traffic signal scheduling system based on adaptive quad agent Double Deep Q -network algorithm

1 Introduction

Urban traffic congestion has become one of the most significant challenges faced by modern transportation systems. Rapid urbanization, population growth, and the continuous increases in vehicle ownership have significantly increased traffic demand in metropolitan areas. Signalized intersections are critical components of urban networks, where inefficient signal control often leads to extreme vehicle waiting times, increased fuel consumption, and higher levels of environmental pollution. Consequently, the development of efficient and adaptive traffic signal control strategies has become essential in intelligent transportation systems. Transportation traffic signal control strategies depend on predefined signal timing plans/limited detector based information. Although these approaches are simple to implement, repeatedly fail to respond effectively to dynamic traffic conditions. In particular, when vehicle queues extend beyond the detection range of traffic sensors, traditional models struggle to accurately estimate traffic states, resulting in inefficient signal timing decisions and increased congestion. Recent advances in deep reinforcement learning have provided promising solutions for adaptive traffic signal control. Mnih et al. (2015) and Van Hasselt et al. (2016) assist traffic controllers to learn optimal signal policies through interaction with the environment, reducing the rule based optimization. DQN approaches can approximate optimal Q-value functions using deep neural networks and dynamically adjust signal phases based on real-time traffic conditions. However, several challenges remain in practical implementation, including large state action spaces that increase computational complexity, sparse and delayed reward signals that delay effective learning, and Q-value overestimation in conventional DQN methods, which may lead to unstable training and suboptimal decision-making policies.

Many researchers have stated various results to address traffic congestion challenges and improve overall traffic signal performance. Review studies in Vandaele et al. (2000) analyze traffic flow models, and highlight the analytical strength of queuing theory in congestion analysis, performance evaluation, and environmental impact assessment. Similarly, the study in Akçelik and Rouphail (1993) reviews delay estimation at signalized intersections, observes that traditional models, including those based on the Highway Capacity Manual, often underestimate interruptions caused by fluctuating demand. The authors propose a peak adjusted framework that significantly improves capacity estimation accuracy. In large-scale traffic signal timing problems, the primary objective is to minimize overall travel time and vehicle waiting time while enhancing coordination among multiple intersections (Guo et al., 2019). The study presented in Jina and Ma (2017) proposes a genetic algorithm integrated with an agent based framework in which each agent represents an individual traffic signal, enabling dynamic improvement of signal timings. The traffic signal optimization problem using a memetic algorithm was investigated in Sabar et al. (2017) focusing on queue waiting time reduction. Dynamic flow networks and accumulation equilibrium with well-posedness conditions were studied in Como and Nilsson (2021). Researchers in Ahmed et al. (2019) developed a queueing model based on differential inclusions with constant routing matrices that allow outflow rates to exceed queue mass. To maintain smooth traffic flow during peak congestion periods, a continuous time mass flow conservation model was introduced in Uppaluru et al. (2022) to distribute green signal time effectively. A virtual arterial corridor model proposed in Li and Ban (2020) enables real-time vehicle communication, thereby reducing traffic signal stops and minimizing vehicle waiting time (Zhu et al., 2022). Traffic congestion modification strategies have also been explored through flow stabilization and experimental traffic analysis. The work presented in Xu et al. (2021) reduced traffic flow fluctuations, while Qu et al. (2021) conducted experimental investigations on complex traffic flow networks. A static equation for order traffic flow system was introduced in Sau et al. (2019) to optimize flow density equilibrium, and dynamically adaptive signal timing strategies (Szoke et al., 2023) proposed to minimize traffic distributions. A priority based algorithm aimed at reducing fuel consumption across traffic networks was introduced in Kim et al. (2023). Arterial intersection complexity and coordination optimization were addressed in Li et al. (2023), while the AM-BAND–PBAND coordination model proposed in Li et al. (2022) maximizes green wave bandwidth and improves travel time, delay, stops, emissions, and fuel consumption, validated through VISSIM simulations. The impact of different traffic signal patterns on delay and safety, considering both pedestrians and vehicles, was analyzed in Montazeri et al. (2022), which proposed a hybrid adaptive signal pattern based on real-time conditions. Decentralized and scalable coordination frameworks were introduced in Augusto Vieira et al. (2024), followed by further optimization methods for complex intersection configurations in Tomar et al. (2022). A comprehensive evaluation of traffic signal characteristics and their impacts on safety and operational efficiency was presented in Leitner et al. (2022). Fuzzy logic based traffic signal control was developed in Taskin and Gumustas (1997), demonstrating improved handling of uncertainty and variability compared with traditional control strategies. Subsequently, deep reinforcement learning integration achieved an 18.46% decrease in queue length, as reported in Meepokgit and Wisayataksin (2024). Queue length estimation using connected vehicles as mobile sensors was proposed in Gao et al. (2019), improving real-time sensing accuracy for adaptive signal control. Flexible routing and coordination for connected and automated vehicles at lane allocation free intersections were introduced in Hao et al. (2023), significantly improving intersection capacity and reducing delays. A dynamic network level signal and speed control framework leveraging connected vehicle data was proposed in Yuan and Zeng (2024), reducing delays, stops, fuel consumption, and emissions. A heterogeneous traffic flow model incorporating human driven vehicles, adaptive travel control was developed in Guan et al. (2023), demonstrating improved traffic stability with higher connected vehicle capacity. Adaptive decentralized traffic signal control using MATSim simulations was investigated in Thunig et al. (2019), while GPS based delay modeling for signalized intersections was presented in Alkaissi and Hussain (2020), showing performance comparable to actuated systems under varying traffic conditions. Field implementations of arterial control strategies were reviewed in Gomes (2010), identifying RHODES as one of the most effective approaches. An improved greedy algorithm based traffic signal scheduling system was proposed in Sun et al. (2024), achieving reduced delay and improved throughput. Automotive communication technologies and future research directions were reviewed in Abdur Rahima et al. (2020), while real-time optimization methods addressing temporal traffic constraints were discussed in Eom and Kim (2020). Adaptive signal control algorithms aimed at improving timing efficiency and reducing queue lengths were presented in Shelby (2004). Decentralized Nash dealing control strategies demonstrated superior congestion reduction compared with centralized methods in Maha et al. (2021).

Reinforcement learning based adaptive signal controllers integrated with vehicle speed guidance were proposed in Maadi et al. (2022), significantly reducing queue length and stop delays. Vehicle to infrastructure and vehicle to vehicle communication based adaptive signal control was investigated in Recker (2010). A convolutional neural network based multi agent deep Q learning framework for dynamic signal control was introduced in Ramesh Kumar et al. (2024), achieving significant reductions in queue length and waiting time. Signal distribution control algorithms dynamically adjusting signal timing based on vehicle arrivals were analyzed in Solaiappan et al. (2023). Conventional behavior analysis across multiple studies was reviewed in Feng et al. (2021), highlighting limitations related to data quality and ethical considerations.

Comprehensive surveys of traffic control system technique were presented in Majeed and Fahad (2022), comparing traditional, sensor based connected vehicle and AI-enhanced approaches. A deep reinforcement learning traffic signal control strategy integrating temporal traffic pattern learning within an actor critic framework was proposed in Ma et al. (2021), demonstrating superior delay reduction performance. Cooperative multi-agent deep actor-critic models were introduced in Song et al. (2024), while incentive based communication frameworks improving agent coordination were developed (Zhou et al., 2024). Finally, model based reinforcement learning approaches incorporating internal traffic models for optimized control planning were proposed in Shi et al. (2023).

In addition, many existing reinforcement learning frameworks give the intersection control problem as a single agent optimization assignment, where a single learning agent simultaneously determines all signal phase decisions. While this approach simplifies the model design, it often leads to large action spaces and reduced learning stability, particularly in complex multi-phase intersections. Furthermore, most current DRL based approaches rely primarily on data driven learning without incorporating analytical traffic flow models that explicitly describe queue dynamics, delay accumulation, and overflow conditions. As a result, the learned policies efficiency interpretability and fail to accurately represent real world traffic dynamics.

Therefore several research gaps remain in the development of efficient reinforcement learning based traffic signal control system. First, there is limited integration of analytical traffic state models with reinforcement learning frameworks, which restricts the ability of learning algorithms to capture detailed traffic queue and delay dynamics. Second existing approaches often lack structured phase level decomposition, leading to large action spaces and unstable learning performance for multi-phase intersections. Third, current multi-agent reinforcement learning strategies typically rely on conventional max-Q value action selection mechanisms, which may introduce overestimation preconception and reduce learning stability in stochastic traffic environments.

To address these limitations, this study proposes an Adaptive Quad Agent Double Deep Q Network framework for intelligent traffic signal control. The proposed approach decomposes the intersection control problem into four coordinated phase specific agents, allowing each agent to focus on limited traffic states while collectively optimizing global intersection performance. This structured decomposition significantly reduces the action space complexity and improves the stability of the learning process.

In addition, the proposed framework a Dynamic signal Distribution and traffic control (DSTCM) model logically evaluate the queue lengths, delay accumulation, and overflow dynamics. These logically derived traffic metrics are embedded directly into the reinforcement learning state representation and reward formulation, improving both learning interpretability and traffic state representation accuracy. Furthermore, a loss based cooperative phase selection is introduced to coordinate the decision making process among the multiple agents. Unlike conventional approaches that depends on maximizing Q-values, the proposed framework, determines signal phase decisions by minimizing aggregated temporal difference loss across the agents. This process reduce Q-value overestimation and enhances the robustness and stability of the leaning process.

To evaluate the effectiveness of the proposed approach, simulation experiments were conducted low, moderate and heavy traffic demand scenarios. The proposed AQDDQN frameworks was compared with existing methods. The experimental results demonstrate that the proposed framework significantly reduces vehicle waiting time, improves queue management and enhances overall traffic throughput under dynamic traffic conditions.

Though existing MARL based traffic signal control methods employ phase-level decomposition and decentralized learning strategies, most approaches mainly depend on reward maximization and communication based coordination process. In addition many existing frameworks are purely data-driven and do not explicitly incorporate analytical traffic state modeling into the reinforcement learning process. In contrast, the proposed AQDDQN framework integrates three key components: a quad agent DDQN architecture for scalable phase-wise learning a loss-based coordination process that minimizes temporal-difference loss instead of conventional max-Q selection, and the DSTCM analytical traffic model for explicit queue, delay, and overflow estimation. Therefore, the novelty of the proposed framework lies in the combined within a DDQN-based adaptive traffic signal control system.

The key contribution of this work as follows:

• A novel Adaptive Quad Agent Double Deep Q network framework is proposed, where each signal phase is modeled as an independent learning agent reducing state space complexity and improving scalability.
• A loss based coordination mechanism is introduced, where signal phase selection is determined by minimizing temporal difference loss, unlike conventional reward based or communication driven coordination.
• Integrating Dynamic Signal Distribution and Traffic Signal Control Model (DSTCM) with reinforcement learning is developed, enabling explicit modeling of queue length, delay and overflow.
• The proposed approach improves learning stability and reduce Q value overestimation through quad agent decomposition combined with DDQN architecture.

Compared with existing MARL based traffic signal control methods, the proposed AQDDQN framework differs not only through phase-wise decomposition, but primarily through the integration of analytical DSTCM based traffic modeling with a loss driven integrative DDQN coordination process. Distinct conventional MARL approaches that depend on reward-maximization/communication based coordination, the proposed framework performs phase selection by minimizing accumulated temporal-difference loss, thereby improving learning stability, interpretability, and robustness under stochastic traffic conditions.

The remainder of this paper is organized as follows: Section 2 reviews related work in reinforcement learning based traffic signal control. Section 3 presents the proposed AQDDQN framework and analytical traffic modeling approach Parameters in Table 1. Section 4 describes the experimental setup and simulation environment. Section 5 discusses the numerical results and performance analysis. Finally, Section 6 concludes the paper and outlines future research directions.

TABLE 1. Comparison of various traffic light systems.

List of symbols and representation as follows:

Table 2.

2 Studies on the time delay model

Delay is an essential metric at signalized intersections; it is necessary for measuring the traffic parameters. Conversely, the dynamic signal model coordinates the above concerns and analyzes the system performance. The following parameters are important for the signal delay analysis, such as Notations:

• d = mean vehicle delay
• c = signal cycle time
• g = available green duration
• x = demand-capacity ratio
• q = arrival frequency (veh/sec)
• Q o = queue backlog at the beginning of the signal cycle
• A = arrivals within the signal phase
• C = maximum outflow during green interval
• Δ C = provided the reserve capacity is zero in the cycle, else

Recent advancements in deep reinforcement learning (DRL) for traffic signal control have focused on improving coordination and scalability in multi-agent environments. Recent studies include spatio-temporal reflection based models, graph based reinforcement learning approaches and game theoretic matching strategies. Similarly, Double Deep Q network models enhances Q-value estimation, and Nash equilibrium-based methods improve cooperation among agents. However, most existing approaches depend on communication mechanisms, with limited focus on integrating analytical traffic flow models/loss based coordination strategies. The proposed AQDDQN framework within recent state of the art developments, additional MARL based traffic signal control approaches from 2023 to 2024, including heterogeneous MARL coordination models, graph attention reinforcement learning architectures and Nash-equilibrium based multi-agent methods have been comparatively analyzed in Table 2.

TABLE 2. Methodological differences between existing MARL approaches and the proposed framework.

The primary novelty of the proposed AQDDQN framework does not occur only from the use of phase-level multi-agent decomposition, since related decomposition strategies have been investigated in earlier MARL studies. Instead, the proposed contribution lies in the integrated design that combines DSTCM-based analytical traffic modeling, loss-based cooperative coordination, and DDQN-based overestimation reduction within a unified traffic signal control framework. This integrated formulation enables improved learning stability, reduced action-space complexity, and enhanced interpretability compared with existing purely data driven MARL approaches.

3 Traffic management system

The proposed traffic signal management framework aims to complete the balance between operational delays and road safety. Traditional traffic management schemes typically include: (i) parking restrictions, (ii) route restrictions, and (iii) right-of-way restrictions. Signal timing strategies are implemented through either traffic-responsive systems or interconnected control systems that require reinforced or wireless communication and are managed within centralized controller databases. According to the Federal Highway Administration (FHWA), the traffic management structure operates at three hierarchical levels. At the local level, intersection controllers collect detector data, adjust traffic parameters, and modify signal phases in real-time.

3.1 Reinforcement learning formulation

The traffic signal control problem is formulated as a reinforcement learning framework defined by a state space, action space and reward function.

State Space (S)

The state Space is defined as: s k = Q i k , V k , i , D i k / i = 1 , 2 , 3 , 4. This state representation captures both congestion level and temporal dynamics of signal control.

Action Space (A):

The action a i ∈ A corresponds to the adaptive adjustment of green time allocation for the four signal phases a i = Δ g 1 , Δ g 2 , Δ g 3 , Δ g 4.

Reward Function (R)

The immediate reward r k is designed to minimize congestion and delay while maintaining stable signal operations: r k = − ∑ i = 1 4 Q i k + D i k + γ ρ i t.This reward structure directly links queue evolution and delay reduction to the learning objective.

Learning Objective:

The AQDDQN agent learns an optimal policy π * that maximizes the expected cumulative discounted reward π * = arg max π E ∑ t = 0 T δ t r t

The Double Deep Q-learning framework with Quad Agent decomposition is employed to reduce overestimation and improve convergence stability. The above RL formulation is applied in Algorithm 1 and AQDDQN architecture shown in Figure 1.

3.2 Double Deep Q-Network (DDQN)

The Double Deep Q-Network (DDQN) extends the Deep Q-network (DQN) architecture by addressing the overestimation bias present in standard Q-learning. It employs two separate neural networks to decouple action selection and action evaluation during Q-value update process.

The agent network enables the DQN structure to estimate the accurate Q-value and reward value in an urban traffic environment. However, greedy approaches may quickly get optimization while increasing the possibility of overestimation preconception and unsteady convergence under stochastic traffic conditions (Mnih et al., 2015; Sun et al., 2024). The maximum target Q-value was selected by a ′ max Q s t + 1 , a ′ : θ −, and the result can be significant to flow deviation and predict the overestimation action values. Consider two distinct networks for the action selection process and evaluation. This model efficiently minimizes the overestimation error values and improves the stability level.

The evaluation network Q s , a : θ selects actions, while the target network Q s , a : θ − evaluates them. This separation improves stability and reduces overestimation errors.

The action selection process for state s t + 1 is defined as:

a ′ = arg a ∈ A sup Q s t + 1 , a : θ (3.1)

The target Q-value is computed as:

y t = r t + δ * Q s t + 1 . a ′ : θ − (3.2)

The decoupled mechanism effectively reduces Q-value overestimation and improves learning stability:

Q s t + 1 , a * : θ − ≤ max a ∈ A Q s t + 1 , a : θ − (3.3)

Using Equation 3.2, we obtain Equation 3.4 within the DQN framework.

y t = r t + δ * Q ( s t + 1 , ⁡ arg a ∈ A Inf Q s t + 1 , a : θ , θ − (3.4)

The loss function is defined as the mean squared error between predicted and target Q-values:

L θ = E s , a ( y t − Q s , a : θ 2 (3.5)

The network parameters are updated using gradient descent:

∇ L θ = E y t − Q s , a , θ ∇ Q s , a , θ (3.6)

Initially, parameters are updated θ ′ = θ + ∇ θ using fixed the step size σ, after which the target network parameters θ − = θ are periodically synchronized with the evaluation network. In Figure 2, the DDQN process integrates reinforcement learning and gradient descent to refine the action value function and learn an optimal control policy.

3.3 Quad Double Deep Q- Network (QDDQN)

The Quad-DQN architecture decomposes the Q-function into two components: the state value function e and the advantage function e ′, which are subsequently recombined to obtain the final Q-value estimate. This decomposition enables improved differential between state importance and action contribution.

To address unpredictability and overestimation issues observed in conventional DQN methods under stochastic traffic conditions, a Quad-Agent DQN framework is proposed. A preconditioned learning q e s ′ mechanism is incorporated and the temporal difference loss is computed using structured target values. Phase selection q e ′ s ′ is performed by minimizing the TD loss rather than directly maximizing Q-values, thereby improving convergence stability and estimation accuracy.

In the proposed AQDDQN framework, the temporal-difference (TD) loss is used as an indicator of learning reliability for each phase-specific agent. A lower TD-loss implies that the predicted Q-values are nearer to the target Q values, indicating more stable and accurate learning of traffic conditions. Therefore, instead of selecting actions only through maximum Q values, the proposed method selects the phase with minimum TD-loss. This strategy reduces the effect of Q-values overestimation under stochastic traffic conditions and improves learning stability, convergence robustness, and signal coordination performance.

The loss values of e and e ′ is evaluated relative to the learning rate to ensure stable updates, as in Figure 3. The action selection follows the Double Q-learning principle, where arg max operation q e s ′ is applied to the evaluation network and the resulting action is evaluated using the target network. This strategy further reduces overestimation preconception and enhances training robustness.

The formulation of Equations 3.7, 3.7b are obtained form, while Equation 3.7b from Van Hasselt et al. (2016), and are presented below:

The Classical Q learning update is:

Q * s , a = E r t + δ max a ′ Q * s ′ , a ′ | s , a (3.7)

The target Q-value becomes:

y t = r t + δ max a ′ Q s ′ , a ′ ; θ (3.7a)

The Expanded Double DQN target is:

y t = r t + δ Q s ′ , arg max a ′ Q s ′ , a ′ ; θ , θ − (3.7b)

Form the above equation the QDDQN update is:

Q s , a ; θ = r t + δ Q s ′ , arg max a ′ Q s ′ , a ′ ; θ , θ (3.8a)

But, proposed system controls four signal phases, thus

Q i s i , a i ; θ i = r t , i + δ Q i s ′ , arg max a i ′ Q i s ′ , a i ′ ; θ i , θ i (3.8b)

Define action selection form the evaluation network is e * = arg max e Q s , e ; θ, then the corresponding value is evaluated using target network:

D q e = Q s ′ , e * ; θ (3.9a)

So the complete Double Q learning target becomes:

y t = r t + δ Q s ′ , arg max a ′ Q s , e ; θ , θ (3.9b)

Define the loss function; it should represent the mean squared temporal difference error, The target value:

T e = r t + δ Q s ′ , e * ; θ (3.10)

Therefore, a Transformed Q-value, often representing a temporal difference error/loss adjusted Q-value is D q e = y e − Q s , e ; θ, which represents the TD error (i.e., difference between the target value and predicted Q value)

Thus, the loss function becomes:

L θ = r t + δ Q s ′ , ⁡ arg max e Q s , e ; θ ; θ − Q s , e ; θ 2 (3.11)

q e- Q value associated with phase (e’)

Performance Metrics:

Total reward per episode : R = ∑ t = 1 T r t (3.12)

Average Travel time : T a v g = 1 N ∑ i = 1 N T i (3.13)

Average return value : R a v g = 1 N ∑ i = 1 N R i (3.14)

Average number of steps : S a v g = 1 N ∑ i = 1 N S i (3.15)

Average reward per step : PR = 1 N ∑ i = 1 N R i S i (3.16)

The single-intersection formulation enables controlled evaluation of the proposed AQDDQN framework before extending it to large-scale network-level traffic systems.

4 Dynamic signal distribution under variable demand and time conditions

4.1 Definition of DSTCM and integration AQDDQN

Definition of DSTCM:

The Dynamic Signal Distribution and Traffic Control Model (DSTCM) is an analytical traffic state estimation model that computes phase-wise queue evolution, delay accumulation and overflow dynamics under stochastic vehicle arrivals. For each phase i ∈ 1 , 2 , 3 , 4 and cycle k

Vehicle arrivals follow a Poisson process: A i k ∼ P λ i , G i k

Queue update as follows: Q i k + 1 = max Q i k + A i k − μ i G i k , 0

Over Estimation:

Overflow occurs if Q i k > Q i max

Overflow delay term: ρ i k = max Q i k − Q i max , 0

Delay Accumulation:

Average delay: D i k = Q i k μ i

Total phase delay: I i k = Q i k * D i k

Total Intersection Delay: D t o t a l k = ∑ i = 1 4 I i k

4.2 Integration with AQDDQN

DSTCM does not act as an independent controller: Instead it provides structured analytical inputs to the reinforcement learning framework:

State Construction: For each agent ‘i’: s i k = Q i k , D i k , ρ i k , G i k , C k

Global state: s k = ∪ i = 1 4 s i k. Thus, DSTCM outputs directly define the RL state space.

Immediate reward: r k = − ∑ i = 1 4 α Q i k + β D i k + λ ρ i k

Where, α , β , γ are represents the weighting coefficients.

To analyze the contribution of the DSTCM segment, comparative experiments were conducted using reinforcement learning both with and without analytical traffic state integration, the results indicate that DSTCM improves queue estimation accuracy, overflow prediction capacity, and reward stability by providing structured traffic state representations to the AQDDQN framework. This analytical integration significantly improves learning interpretability and stabilizes policy updates under stochastic traffic demand conditions. In comparative simulations, the AQDDQN framework without DSTCM integration demonstrated higher queue instabilities, less stable reward evolution, and reduced overflow estimate accuracy under moderate and heavy traffic conditions. In contrast, the inclusion of DSTCM improved queue estimation consistency and produced more stable adaptive signal control behavior. These observation confirm the contribution of analytical traffic state modeling toward improving reinforcement learning stability and traffic performance.

4.3 Derivation of total vehicle delays for traffic signal cycle length in dynamic signal distribution

A dynamic traffic model with Poisson-distributed arrivals is analyzed. A traffic overflow formula and deterministic factors guide real-time adjustments. The use of the TRAF-NETSIM [2024] simulation enhances efficiency and reduces overflow.

The total waiting time of the interrupted queue length can be expressed as the sum of components obtained at each cycle phase is:

I k , i = ∑ i = 1 4 V k , i k < i (4.1)

The relationship between signals phases are currently active phase corresponds to the green signal and the remaining phases correspond to red signals.

E I t k , 1 = ρ q 0 ν 1 + q ν 1 ν 1 + 1 2 (4.2)

The exit flow rate in each direction can be estimated and defined as:

E S i = p * T i , T i = V k i V k , p = μ T c (4.3)

Similarly,

E ρ q ν 1 + ν 2 = ν 2 q s 1 + ν 1 q s 1 + ρ q 0 , (4.4)

E ρ q ν 1 + ν 2 + ν 3 = ν 3 q s 3 + ν 2 q s 2 + ν 1 q s 1 + ρ q 0 (4.5)

E ρ q ν = ν 4 q s 4 + ν 3 q s 3 + ν 2 q s 2 + ν 1 q s 1 + ρ q 0 (4.6)

Using these results in (4.3), we get

V k , 1 = ν 1 ρ q 0 + ( ν 1 ν 1 + 1 2 + ν 1 M f 1 2 P 1 (4.7)

V k , 2 = ν 2 ρ q 0 + P 1 ν 1 + 0.5 ρ s 2 + ν 2 ν 2 + 1 2 (4.8)

V k , 3 = ν 3 ρ q 0 + P 1 ν 1 + P 2 ν 2 + 0.5 ρ s 4 + ν 3 ν 3 + 1 2 (4.9)

V k , 4 = ν 4 ρ q 0 + P 1 ν 1 + P 2 ν 2 + P 3 ν 3 + 0.5 ρ s 4 + ν 4 ν 4 + 1 2 (4.10)

Therefore, the total waiting time of the interruption queue length is

I k , i = ∑ i = 1 4 V k , i (4.11)

The average change in queue length is.

γ τ = ∑ i = 1 4 P i ν i (4.12)

The total vehicle delays for the traffic signal cycle length arewhere ρ i s is defined as the total vehicle waiting time accumulated across all signal phases during the interrupted cycle.

I * k , i = I k , i + 0.5 ρ i s i = 1 , 2 , 3 , 4 (4.13)

Figure 4, illustrates the performance-based optimization framework in which traffic outflows are regulated through coordinated signal operations and iterative validation. Real-time traffic data collected from Radio Frequency Identification (RFID) sensors are used to identify the current traffic state and extract key variables, which are provided as inputs to the DSTCM. It predicts signal timings inputs and traffic conditions within a specified prediction horizon and functions as the traffic state modeling and delay estimation component of the adaptive framework. The phase-wise queue lengths, interruption delays, and overflow measures generated by DSTCM, used to construct the state space, reward function, and loss evaluation. Consequently, the proposed framework integrates DSTCM with reinforcement learning to achieve adaptive, data-driven and real-time traffic signal optimization.

5 Adaptive quad agent- dynamic signal distribution control optimization (AQDSD)

The proposed AQDSD framework aims to optimize urban traffic signal operations using a multi-agent reinforcement learning architecture. Each agent independently controls a signal phase and together adjusts with signal durations based on real-time traffic data collected through Radio Frequency Identification (RFID) sensors. The system continuously collects traffic information and evaluates traffic flow performance, signal cycle length and reward values for each phase is illustrated in Figure 1. Unlike conventional Deep Q-Network (DQN) approaches, the proposed framework first performs a precondition matching process, followed by computation of the prediction error (loss value) for the current traffic state. The computed error is integrated into the Double Deep Q Network (DDQN) learning process, phase selection is then performed by comparing the loss value is selected for activation. Finally, Quad-Agent Q network reduces prediction error by comparing the learned error values e and e ′, thereby improving the learning accuracy and stability of the system.

Figure 1, workflow of the Adaptive Quad Agent traffic signal control framework. Initially, the cycle length, signal phases and system processing start time are utilized to zero. Real-time traffic data are collected from the each agent and corresponding traffic state variables are updated, accordingly, based on updated state information, and the signal duration is adjusted. Evaluates, each agent performance using Quad agent network by comparing the corresponding loss values associated with updated phase durations. During this process, memory management is maintained by looping existing values when the memory capacity is exceeded. After that, update the agent information and signal timing are coordinated with neighboring agents to preserve intersection level synchronization. The vehicle waiting time, phase duration and cycle length are continuously recalculated using the updated traffic data. Subsequently, signal phase selection and timing adjustments are optimized through reinforcement learning techniques to reduce vehicle delays and improve traffic flow efficiency. Finally, the overall system performance is analyzed using the Quad Agent framework, where key traffic performance indicators, including queue length, cycle duration, and vehicle delay, are evaluated to measure improvements in traffic flow conditions.

In Table 3, the phase wise traffic states derived from the DSTCM presented for each decision step. In particular, the traffic variables, including queue length, delay and overflow are supplied to the corresponding agents. These state variables, together with the defined reward structure and learning parameters, are utilized in the Agent DQN update and phase selection process, as illustrated in Figure 1.

TABLE 3. Hyper parameter setup.

5.1 Adaptive quadruple agent system

5.1.1 Quad agent system architecture and coordination mechanism

The proposed framework adopts a centralized training with decentralized execution architecture for traffic signal control at a four phase intersection. The control problem is decomposed into four phase specific reinforcement learning agents i ∈ 1 , 2 , 3 , 4. Each agent is responsible for optimizing the green-time allocation of its corresponding signal phase.

Phase Level Agent Representation:

Each agent observes a localized traffic data form DSTCM model:

s i k = Q i k , V i k . D i k , ρ i k (5.1)

Using this state representation, each agent estimates its phase specific action value function using a Double DQN:

Q i s i , a i : θ i (5.2)

This decomposition reduces the dimensionality of the joint action space and enables phase wise learning aligned with physical traffic dynamics. Here, agents do not communicate directly each other. Instead of coordination is achieved through a centralized loss aggregation module located at the intersection controller level.

At each decision approach (one signal cycle)

Temporal Difference Loss Computation:where y i k represents the target Q-value

e i k = y i k − Q i s i , a i ; θ i 2 (5.3)

Global Phase Selection:

The computed loss values are transmitted to the centralized coordination layer. The optimal signal phase is selected by minimizing the aggregated loss:

a * k = arg min i ∈ 1 , 2 , 3 , 4 e i k (5.4)

Thus, the phase corresponding to the minimum temporal difference error is activated for the next signal interval.

5.1.2 Action space complexity reduction

In a conventional single agent formulation, the action space grows exponentially with the number of signal phases. If each phase ‘m’ discrete actions, then joint action space is

A j o int = m 4 (5.5)

In compare, the quad agent framework decomposes the problem into four independent subspaces:

A d e c o m p o s e d = 4 m (5.6)

This reduces the complexity from exponential growth O m 4 = O m

5.1.3 Q-learning update rule

The classical Q-learning update rule is expressed as

Q s , a ← Q s , a + α δ (5.7)

Where the temporal difference error is defined as

δ = r + γ max a ′ Q s ′ , a ′ − Q s , a (5.8)

5.1.4 Variance reduction in Q-estimation

Let the single agent Q-update rule be defined as

Q s , a ← Q s , a + α δ (5.9)

Where the temporal difference error is defined ashere, Q s , a represents the action value function.

δ = r + γ max a ′ Q s ′ , a ′ − Q s , a (5.10)

In high dimensional joint action spaces, the variance of TD-error increases with the size of action space.

Therefore, the variance of the TD error can be expressed aswhere, A i represents the action space corresponding to each phase specific agent.

V a r δ j o int ∝ A j o int (5.11)

Since

A i < A j o int (5.12)

It follows that

V a r δ d e c p m p o s i t i o n < V a r δ j o int (5.13)

Thus, quad agent decomposition significantly reduces the variance of the TD error. This improvement leads to enhanced learning stability, smoother gradient updates, and improved convergence reliability of the reinforcement learning algorithms.

5.1.5 Loss based coordination mechanism

Instead of selecting actions based on maximum Q-values

a * = arg max a s , a (5.14)

The proposed framework selects actions by minimizing prediction error:

a * = arg max i e i k (5.15)

The proposed coordination strategy differs from conventional max-Q action selection frequently used in reinforcement learning methods. In standard DDQN based traffic control, the action with the highest estimated Q values is selected, which may lead to unstable decisions due to temporary Q value overestimation under stochastic traffic conditions. In compare, the proposed AQDDQN framework selects the signal phase with minimum temporal difference (TD) loss. Since TD-loss measures the difference between predicted and target Q-values, lower loss indicates more reliable and stable learning. Consequently, the proposed loss based coordination process improves convergence stability, reduces overestimation preconception, and enables more consistent adaptive traffic signal control.

5.1.6 System learning objective

The overall objective of the proposed framework is to maximize the cumulative discounted reward:

π * = arg max π E ∑ t = 0 T γ t r t (5.16)

The reward function integrate consequences associated with queue length, delay and overflow derived from the DSTCM analytical model.

5.1.7 Comparative analysis of single agent DDQN, proposed quad agent AQDDQN and transformer based centralized control models

In single agent DDQN architecture, the control structure is fully centralized, where a single Q-network manages the joint action space of signal phases. This results in exponential growth of the action space O m 4, increased temporal difference variance, and higher sensitivity to overestimation preconception under heavy traffic demand. Although the implementation is relatively simple, convergence may be shower due to the high dimensional joint decision space.

In compare, the proposed model decomposes the intersection into four coordinated phase level agents, thereby reducing the effective action space complexity to linear order O m. Each agent operates on structured state variables derived from the DSTCM, which improves interpretability and stabilizes the learning dynamics. Instead of using conventional max-Q selection, global coordination is achieved through aggregated temporal difference loss minimization, which reduces overestimation preconception and enhances convergence robustness.

The proposed decomposition improves scalability, reduces variance, and achieves significant delay reductions under stochastic traffic conditions while maintaining moderate computational cost. Unlike transformer based centralized models, which depends on high dimensional embedding, large training datasets, and complex attention mechanisms, the proposed framework maintains stability and interpretability. Its analytically integrated structure avoids over parameterization, and provides a balanced trade-off between efficiency, robustness, and real-time deplorability for isolated intersection traffic signal control.

Figure 5 illustrates the operational workflow of the reinforcement learning framework based on the AQDDQN approach. Initially, the algorithm monitors the predefined limit. After initialization, the initial state value of the Q network is evaluated. The agents selects and executes actions within environment. As a result of action, the system observes a new state and corresponding reward. The transition tuple consisting of state, action, reward and next state is represented as s t , a t , r t , s t + 1. These observed tuples are stored in a memory replay buffer for future training. Periodically, the parameters of Q network are updated to maintain learning stability. This mechanism helps reduce the correlation between sequential updates and improves convergence stability. The training objective is to minimize the difference between the predicted Q-value and the target Q-value. The learning process updates the Q-network parameters iteratively until the loss function converges.

The proposed model evaluates traffic performance using a combined measure of queue length and vehicle waiting time to direct optimal traffic signal decisions. To predict traffic flow variations, two consecutive signal cycles denotes as λ i and λ i + 1 are analyzed. The change between these cycles determines the direction and scale of traffic flow.

Algorithm 1 Adaptive Quad Agent Double Deep Q Network.

5.2 Convergence and learning stability analysis

The learning stability of the proposed framework is analyzed from an empirical deep reinforcement learning perspective. Since the proposed implementation employs a fixed learning rate of α = 0.001, the classical Robbins-Monro stochastic approximation conditions required for strict asymptotic convergence guarantees are not satisfied. Therefore, the proposed framework does not claim formal theoretical convergence to a global optimum. Instead, the convergence behavior observed in this study should be interpreted as empirical learning stability established through simulation experiments under varying traffic demand scenarios. The use of a fixed learning rate is a common practical choice in deep reinforcement learning implementations to maintain stable and computationally efficient training behavior.

Instead, the stability of the learning process is attained through several established mechanisms commonly adopted in Deep Reinforcement Learning architectures.

For each phase specific agent ‘i’, the Q-network parameters are updated using gradient descent:where, L θ i denotes the temporal difference loss function with the DDQN update process.

θ i t + 1 = θ i t − α i ∇ θ i L θ i (5.17)

The loss function is defined as:

L θ i = E y i − Q s i , a i ; θ i 2 (5.18)

The proposed framework improves empirical learning stability through the following mechanisms:

• Experience replay memory reduces temporal correlation among training samples.
• Target network synchronization stabilize the Q-value update process and mitigates oscillator behavior during training.
• The DDQN structure reduces Q-value overestimation by decoupling action section and target evaluation.
• The quad agent decomposition reduces the effective action space dimensionality and lowers the variance of temporal difference updates.

Consequently, the proposed framework proves stable learning behavior and consistent policy improvement across simulation episodes below in low, moderate, and heavy traffic demand scenarios. The convergence characteristics reported in this study should therefore be interpreted as empirical convergence behavior observed during simulation experiments rather than as a formal stochastic approximation proof.

6 Numerical example and simulation results

To validate the effectiveness of proposed work, a comprehensive simulation study was conducted using a four way urban signalized intersection model. The simulation environment represents a signalized junction located near Sholinganallur, with stochastic vehicle arrivals modelled as a Poisson process under varying demand conditions. Although, the Sholinganallur station involves a multiple intersections junction area, the present study focuses on a single isolated four phase intersection as a main validation environment of the proposed work. This experimental setup permits the detailed evaluation of reinforcement learning stability, queue development features, overflow estimation and phase wise coordination performance of the proposed model without introducing additional network level suspicions. The isolated intersection configuration is commonly adopted in preliminary traffic signal control studies to validate algorithmic effectiveness before extending the framework to large scale coordinated urban traffic networks.

Three traffic demand levels were considered to evaluate the robustness of the proposed approach: low traffic (0.10 veh/s), moderate traffic (0.30 veh/s), and heavy traffic (0.60 veh/s). Each experiment was executed for 3,600 s, and a 30 independent simulation runs were performed to make sure statistical reliability. The described performance values correspond to the mean results obtained from the 30 independent simulations runs. The related standard deviations were additionally analyzed to evaluate the robustness and statistical consistency of the proposed framework under stochastic traffic conditions. The observed variations remained within acceptable limits across all traffic demand scenarios, indicating stable learning and reliable traffic control performance.

The stated performance metrics correspond to the mean values obtained through the 30 independent simulation runs, while the associated standard deviations were analyzed to evaluate the stability and strength of the proposed framework under stochastic traffic conditions.

The performance of the proposed framework was compared with three baseline methods: Fixed time control, Double Deep Q Network, and Improved DDQN. The evaluation metrics include vehicle waiting time, queue length evolution, reward convergence, and overflow delay estimation.

The hyper parameter settings used for training the proposed reinforcement learning model are summarized in Table 3.

6.1 Implementation details and training protocol

The experimental implementation details of the proposed framework with explicitly defined. Each phase specific agents employs a fully connected network consisting of three hidden layers with 128, 64 and 32 neurons respectively. Rectified Linear Unit (ReLU) activation functions are used in all hidden layers, while the output layer corresponds to discrete green time adjustment actions. The model is trained using in Adam optimizer with a learning rate of 0.001. A replay buffer of size 10,000 is maintained, and mini-batches of size 128 are sampled during training. The discount factor is set of 0.99, and the target network is updated every 200 iterations to make sure learning stability. A greedy exploration strategy is adopted, where ∈ decays from 1.0 to 0.01 over the training process. Each experiment is conducted over 500 episodes, with a maximum of 500 steps per episode. The simulation and learning framework are implemented using a custom MATLAB-based simulation depends on integrating the DSTCM analytical traffic model with the AQDDQN reinforcement learning architecture.

The complete hyper-parameter setting, training configuration, state representation reward formulation, and simulation environment specifications are clearly provided to improve experimental reproducibility and simplify future comparative studies.

6.1.1 Overflow delay analysis

Figure 6 shows the comparison of predicted overflow delay across different analytical models. The proposed DSTCM model captures stochastic traffic variations more effectively than traditional model such as HCM and CTMC. Though conventional approaches tend to minimize delay under high congestion, DSTCM identifies peak congestion intervals more precisely, particularly during high demand periods.

6.1.2 Reward and queue dynamics

To evaluate the empirical convergence characteristics of the proposed AQDDQN framework, training reward evolution and queue stabilization behaviors are analyzed through Figures 7–9.

Figure 7 shows the total reward obtained per signal cycle for each phase. The reward values exhibit stable convergence behavior after initial exploration, indicating effective learning of optimal signal policies. Phase wise analysis exposes that the proposed framework dynamically adjusts green time allocation to minimize delay and queue accumulation.

Figure 8 discusses the relation between queue length and reward values over the 80 signal control cycles. While the reward trajectory does not complete asymptotic convergence within the training period. The proposed modes validate progressive policy stabilization and consistent developments in traffic performance. The observed reward variations are estimated under dynamic multi-traffic environments, and results should be construed as empirical learning developments rather than full convergence strategy.

Figure 9 illustrates the relationship between queue length and reward value during the simulation period. The queue length increases with vehicle arrival rates, while the reward values fluctuate between − 40 and 40 reflecting the controller’s adaptive response to traffic conditions. The observed tendency indicates measured convergence toward an optimal signal control policy.

To evaluate the empirical convergence features of the proposed AQDDQN framework, training reward evolution and queue stabilization performances are analyzed through Figures 7–9. The figures priminiarly discuss the cycle wise traffic performance matrices and adaptive policy stabilization during simulation. While these results indicate progressive improvement in learning behavior and traffic control stability, they should be interpreted as empirical learning developments rather than strict asymptotic convergence guarantees. Further extensions of this work may incorporate episode level cumulative reward learning curves for more comprehensive convergence visualization.

6.1.3 Comparative performance analysis

Percentage calculations:

Formula : I m p r o v e m e n t % = B a s e l i n e − A Q D D Q N B a s e l i n e × 100

Figure 10 presents the comparative performance analysis of DDQN, Improved DDQN and the proposed AQDDQN framework under low, moderate, and heavy traffic conditions. The result express in Table 4, proposed AQDDQN framework consistently achieves lower vehicle waiting times compared with the existing approaches. Under low traffic conditions, the AQDDQN model reduces the waiting time from 121 units (Improved DDQN) to 91 units, corresponding to a reduction of approximately 36.36%. Similarly, compared with DDQN, the proposed framework achieves a significant reduction in waiting time. Under moderate traffic conditions, the AQDDQN framework continues to provide improved delay reduction and traffic flow stability. In heavy traffic conditions the proposed framework maintains superior performance by effectively minimizing queue accumulation and vehicle waiting time through its phase-wise decomposition and loss based coordination process. Overall, the obtained results confirm that the proposed AQDDQN framework improves traffic signal control efficiency and provides more stable performance under varying traffic demand conditions.

TABLE 4. Performance improvements (%).

The Improved performance observed in the proposed framework is strongly associated with the introduced loss-based coordination strategy. By selecting signal phases according to minimum temporal difference loss rather than maximum Q-values, the proposed controller achieves more stable phase’s transitions and reduces the impact of Q-values overestimation during training. This behavior is reflected in the smoother reward convergence and lower queue accumulation observed under moderate and heavy traffic demand scenarios. The results therefore provide empirical evidence that the proposed loss-base coordination process contributes to improved traffic signal timing decisions and enhanced learning stability.

6.1.4 Single agent vs. multi agent analysis

Figure 11 compares the performance of single agent reinforcement learning with the proposed quad agent multi agent framework. The results demonstrate that multi agent architecture significantly improves traffic control performance in terms of queue length reduction, convergence speed and signal coordination. The quad agent structure decomposes the intersection control problem into phase specific decision processes, supporting localized learning and reducing variance in temporal difference updates. This leads to improved stability and faster convergence compared with centralized single agent approaches.

7 Conclusion

This study presented an Adaptive Quad Agent Double Deep Q Network framework for intelligent traffic signal control at urban intersections. The proposed approach integrates a decentralized multi-agent reinforcement learning architecture with the DSTCM to effectively capture stochastic traffic dynamics and optimize signal timing decisions. By decomposing the intersection control problem into four coordinated agents, the proposed model enables each agent to focus on localized traffic states while contributing to global intersection optimization. The experimental results demonstrate that the proposed framework significantly improves traffic performance compared with conventional signal control methods, including fixed time control, standard DDQN and improved DDQN approaches. In particular, the model achieves substantial reductions in vehicle waiting time, queue length, and traffic congestion, especially under moderate and heavy traffic conditions. The decentralized quad agent architecture improves scalability and coordination among signal phases, allowing the system to adapt efficiently to dynamic traffic patterns.

Though existing MARL based traffic signal control approaches that primarily depend on reward maximization and communication driven coordination, the proposed framework integrates analytical traffic state estimation with loss based cooperative DDQN learning, thereby providing improved interpretability and stable phase wise optimization under stochastic traffic environments.

Furthermore, the incorporation of precondition based learning enhances the stability of policy updates by restricting phase transitions to contextually valid actions. This mechanism reduces the exploration of suboptimal control decisions and accelerates the convergence of the reinforcement learning process. Additionally, the integration of the Double Deep Q Network framework mitigates Q-value overestimation, resulting in more reliable policy learning and smoother signal control actions. These features collectively contribute to improved robustness and reduced performance variance across multiple simulation runs.

Simulation experiments conducted under different traffic demand scenarios confirm that the propose model consistently outperforms of existing approaches. The simulation results shows that the proposed AQDDQN framework attains significant reductions in average vehicle waiting time under low, moderate, and heavy traffic conditions compared with conventional Fixed-Time, DDQN, and improved DDQN approaches. The obtained results consistently express improved queue management, reduced congestion, and improved traffic flow stability across varying traffic demand scenarios. These finding highlight the ability of the proposed model to effectively manage traffic congestion and improve intersection throughput in high-density traffic environments.

The proposed loss-based cooperative coordination process further improves policy stability by prioritizing reliable Q-value estimation instead of conventional max-Q action selection.

Though the proposed AQDDQN framework demonstrates effective performance under isolated intersection conditions, the present study is limited to a single four-phase intersection and does not fully capture large-scale urban traffic complexities. In real traffic networks, neighboring intersections interact through spillback propagation, queue interference, and coordinated signal operations, which may influence traffic stability and queue evolution. Therefore, the reported performance improvements should be considered as preliminary validation under controlled stochastic conditions. Nevertheless, the proposed quad-agent architecture provides a scalable foundation for future multi-intersection traffic control using cooperative multi-agent reinforcement learning and inter-intersection coordination process.

The present study has certain limitations. The simulation framework considers only a single isolated signalized intersection, whereas real urban traffic environments such as Sholinganallur junction involve multiple interconnected intersections with complex traffic interactions and network level dependences. As a result, large-scale traffic dynamics, inter-intersection dependencies, and network-level flow propagation are not fully captured.

Moreover, the model validates stable empirical learning performance; a formal theoretical convergence make sure under stochastic approximation is not established due to the implementing of a fixed learning rate. Though the proposed AQDDQN framework demonstrates stable empirical convergence behavior across multiple simulation scenarios, the use of a fixed learning rate does not satisfy the classical Robbins-Monro conditions required for formal stochastic approximation convergence proofs. This implementation alternative follows common practice in deep reinforcement learning to ensure stable and efficient network training.

Future will address these limitations by extending the framework to multi-intersection and network level traffic control using cooperative multi agent coordination strategies and real world traffic datasets. Additionally, convergence analyses incorporating gradually reduce the learning rate schemes will be discovered to reinforce the theoretical base of the proposed approach.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Meng Xu, Beijing Jiaotong University, China

Reviewed by: Bin Zhou, The University of Hong Kong, Hong Kong SAR, China

Zhongnan Zhao, Harbin University of Science and Technology, China