Multi-Level Deep Deterministic Policy Gradient reinforcement learning for safety-constrained trajectory optimization of collaborative industrial robots

1 Introduction

Collaborative robots have become a key technology in modern intelligent manufacturing systems due to their ability to operate safely with human workers while providing high flexibility, precision, and automation efficiency. Unlike conventional industrial robots that execute predefined repetitive tasks in isolated environments, collaborative robots require adaptive motion planning capabilities to respond to dynamic workspaces, changing production requirements, and unexpected obstacles. Kim et al. (2024) highlighted that the integration of artificial intelligence into collaborative robotics has significantly improved autonomy, perception, and decision-making capabilities in industrial applications. However, generating optimal trajectories that simultaneously satisfy accuracy, safety, energy efficiency, and real-time constraints remains a major challenge for advanced robotic systems.

Traditional trajectory planning includes graph search, sampling-based planning, interpolation, and optimization-based control. The A* algorithm was introduced by Hart et al. (1968), while asymptotically optimal sampling-based planning was formalized for RRT* by Karaman and Frazzoli (2011). These approaches remain reliable in structured workspaces, but their performance depends on environmental models and online replanning capacity. Recent spline and model-predictive methods can produce smooth industrial manipulator trajectories, although additional mechanisms are still required for dynamic constraints and changing obstacle configurations (Momynkulov et al., 2025).

Deep reinforcement learning provides an alternative for continuous robotic control. Deep Deterministic Policy Gradient (DDPG) was introduced by Lillicrap et al. (2016) as an off-policy actor-critic method that combines deterministic policy gradients, replay memory, and target networks. TD3 later reduced function-approximation error through twin critics and delayed policy updates (Fujimoto et al., 2018), while SAC introduced entropy-regularized stochastic control (Haarnoja et al., 2018). These methods are relevant to collaborative robots because joint and Cartesian commands are continuous.

Practical deployment nevertheless remains constrained by unsafe exploration, sensitivity to initialization, imperfect simulator dynamics, and limited evidence across hardware platforms. Constrained policy optimization and action-layer correction offer complementary ways to restrict unsafe behavior (Achiam et al., 2017; Dalal et al., 2018). Recent industrial studies have also emphasized adaptive safety in human-robot workspaces (Feng et al., 2025), while a current survey of real-world robotic reinforcement learning identifies reproducibility and sim-to-real robustness as persistent challenges (Tang et al., 2025). Reinforcement-learning-based admittance control has also been investigated for physical human-robot interaction under output constraints (Gao et al., 2025).

To address these limitations, this study proposes a Multi-Level Deep Deterministic Policy Gradient reinforcement learning framework for safety-constrained trajectory optimization of collaborative industrial robots. The proposed approach integrates hierarchical decision-making, safety-aware reward optimization, adaptive exploration mechanisms, and simulation-to-real transfer strategies. The framework is designed to generate smooth, efficient, and collision-free trajectories while maintaining real-time feasibility for industrial operations. Experimental validation demonstrates the effectiveness of the proposed method in achieving autonomous trajectory planning and improving the adaptability of collaborative robots for intelligent manufacturing applications.

2 Related works

Classical motion planning provides transparent feasibility guarantees when the environment is sufficiently known. A* performs heuristic graph search (Hart et al., 1968), whereas RRT* converges asymptotically toward an optimal path under its sampling assumptions (Karaman and Frazzoli, 2011). Interpolation and spline methods can improve geometric continuity, but do not by themselves learn how to trade task progress against changing safety risk.

Recent industrial trajectory studies combine learning with optimization to improve responsiveness. Simulation-efficient deep reinforcement learning has been evaluated for manipulator trajectory planning (Zhao et al., 2025), and hierarchical reinforcement learning with explicit path optimization has been studied for adaptive robotic-arm obstacle avoidance (Ma et al., 2026). These studies motivate structured decision making, but they do not eliminate the need to define the hierarchy, information flow, and safety mechanism precisely.

Reinforcement learning is formalized as sequential return maximization under a Markov Decision Process (Sutton and Barto, 2018). Hierarchical reinforcement learning extends this formulation by introducing temporally extended actions or decomposed decision layers. The options framework defines initiation sets, internal policies, and termination conditions (Sutton et al., 1999), while broader hierarchical methods separate decisions across temporal or task abstractions (Barto and Mahadevan, 2003).

Deep Q-Networks demonstrated value-function approximation for discrete action spaces (Mnih et al., 2015), but manipulator control normally requires continuous actions. DDPG addresses this requirement through a deterministic actor and learned critic (Lillicrap et al., 2016). In the present framework, DDPG is the only learned control policy; the strategic and reactive levels are deterministic modules surrounding that policy rather than three separately trained agents.

The continuous-control baselines used in this study are attributed to their original sources. TD3 uses clipped double-Q targets, delayed actor updates, and target-policy smoothing (Fujimoto et al., 2018). SAC maximizes expected return and policy entropy (Haarnoja et al., 2018). PPO applies a clipped surrogate objective for stable on-policy updates (Schulman et al., 2017), but it is discussed only as related work because PPO was not included in the reported experiment table.

Phase-based and hierarchical robotic learning has previously decomposed long-horizon tasks into ordered stages. For example, reinforcement-learning motion planning for robotic bricklaying divides a collaborative-arm task into task-specific phases (Navazani et al., 2026). The present ML-DDPG differs in that its strategic level continuously supplies intermediate waypoints, the tactical level uses one shared continuous policy, and the reactive level projects every raw action into a feasible set; it does not train an independent policy for each phase. Reinforcement learning has also been applied to human-robot collaborative assembly (Zhang et al., 2022), and large language models have been used to generate reward functions for robotic manipulation (Turcato et al., 2025).

Safe reinforcement learning can enforce constraints through the objective, the policy update, or a separate action-correction layer. Constrained policy optimization targets expected-cost bounds (Achiam et al., 2017), whereas a safety layer can minimally modify an action before execution (Dalal et al., 2018). Recent human-robot research has combined adaptive perception and deep reinforcement learning for dynamic industrial safety (Feng et al., 2025), reinforcing the need to report both learned and non-learned safeguards.

Accordingly, the contribution claimed here is not hierarchical decomposition alone. It is the integration of continuous waypoint conditioning, a shared DDPG actor-critic controller, six-term task-and-safety reward shaping, and a per-step constrained projection within one trajectory-planning pipeline, followed by simulation, UR5e tracking, and a separate qualitative welding demonstration. This positioning narrows the novelty claim and distinguishes the architecture from option-based, multi-policy, and fixed phase-based formulations.

3 Materials and methods

The ML-DDPG framework was implemented in PyTorch within a custom three-dimensional kinematic and dynamic simulation environment configured around a six-degree-of-freedom UR5e collaborative manipulator. The method combines a deterministic strategic waypoint generator, one continuous DDPG actor-critic controller, and a non-learning safety projection. This definition clarifies which components are trained and how the hierarchy is executed.

3.1 Overview of the proposed Multi-Level DDPG framework

This study proposes a Multi-Level Deep Deterministic Policy Gradient (ML-DDPG) framework for safety-constrained trajectory optimization of collaborative industrial robots. The main objective of the proposed approach is to generate adaptive, smooth, and collision-free trajectories by integrating hierarchical decision-making, continuous control optimization, and safety-aware learning. The trajectory optimization problem is formulated as a reinforcement learning process, where the collaborative robot interacts with the industrial environment and progressively improves its control strategy based on accumulated experience.

Figure 1 represents the overall workflow of the proposed ML-DDPG framework, including industrial environment modeling, Markov Decision Process (MDP) formulation, hierarchical decision-making, DDPG-based policy optimization, trajectory generation, simulation validation, and real-world deployment. The framework starts by collecting environmental information, including robot states, target positions, obstacle locations, and workspace constraints. Based on this information, the reinforcement learning agent generates optimized actions while satisfying predefined safety and operational limitations.

The trajectory planning problem is defined as an MDP represented by:

M = ( S , A , P , R , γ )

where S denotes the state space, A represents the continuous action space, P defines the transition probability between states, R is the reward function, and γ represents the discount factor. At each time step t, the agent observes the current state s t, performs an action a t, receives reward r t, and moves to the next state s t + 1.

3.2 State and action space definition

The state representation is designed to include essential information required for adaptive robotic motion planning. The state vector consists of robot kinematic information, environmental conditions, target information, and previous control actions. The complete state representation is formulated as:

s t = [ p t , q , q ̇ t , o t , d t , a t − 1 ]

where p t represents the end-effector position, q t indicates joint positions, q ̇ t denotes joint velocities, o t represents obstacle information, d t defines the distance between the robot and target, and a t − 1 represents the previous action. This formulation enables the agent to understand both current robot conditions and environmental variations during trajectory execution.

The action space is defined as a continuous control vector that determines robot movement commands. The generated action is formulated as:

a t = [ Δ x t , Δ y t , Δ z t , Δ q t ]

where Δ x t, Δ y t, and Δ z t represent Cartesian displacement commands, while Δ q t corresponds to joint adjustment values. The continuous action representation enables smooth trajectory generation compared with discrete action-based reinforcement learning methods.

3.3 Multi-Level DDPG agent architecture

The proposed architecture contains three execution levels. Level 1 is a strategic waypoint generator that converts the goal and obstacle map into the next intermediate Cartesian target. Level 2 is the learned tactical controller: a single DDPG actor maps the augmented state and current waypoint to a raw continuous action. Level 3 is a reactive, non-learning safety layer that tests joint, velocity, workspace, and obstacle-clearance constraints and minimally corrects an infeasible action before it reaches the robot controller. Figure 2 depicts these roles and the shared replay-based update.

Information flows sequentially as s t → g t → [ s t , g t ] → a t → a t safe → ℰ. Where, s t—current state, g t—current waypoint (goal), [ s t , g t ]—augmented state, a t—raw action generated by the policy, a t safe—safety-projected action, ℰ—environment.

The transition stored in replay memory contains the executed projected action, not the rejected raw proposal. Only the actor, critic, and their target networks are optimized; the waypoint and projection modules are deterministic. Consequently, the term multi-level describes a coupled execution hierarchy rather than three independently trained policies.

Both actor and critic use three fully connected hidden layers of 256, 256, and 128 units with ReLU activations. The actor terminates in a tanh output scaled to the admissible command range, and the critic terminates in a linear scalar Q value. The architecture was selected as a capacity-latency compromise for the reported state vector and was held fixed across DDPG-family comparisons. No automated hyperparameter search was performed; the learning rates, replay size, batch size, discount factor, and soft-update coefficient in Table 1 were fixed before the reported comparisons.

Table 1. Experimental configuration and hyperparameters of the proposed ML-DDPG framework.

Bold values indicate the six fixed reward weights and the safety penalty coefficient used in the reported experiments.

For state dimension d s and action dimension d a, one actor inference has complexity O ( d s · 256 + 256 2 + 256 · 128 + 128 · d a ); d s is the state dimension, d a is the action dimension, 256 2 corresponds to the hidden-to-hidden fully connected layer, 256 · 128 corresponds to the second hidden layer, 128 · d a corresponds to the output layer mapping to the action space. The critic adds the corresponding state-action input and a scalar output. Replay sampling is O(B), and each gradient update is linear in batch size B and network parameter count. The measured planning latency reported in Table 2 is 17.1 ms for ML-DDPG on the stated RTX 4070 platform; this is a descriptive device-specific measurement rather than a hardware-independent real-time guarantee.

Table 2. Single-run quantitative comparison of trajectory optimization methods.

The DDPG agent consists of an actor network and a critic network. The actor network generates deterministic control actions based on the current state:

a t = μ ( s t ∣ θ μ )

where μrepresents the actor policy and θ μ indicates actor network parameters. The critic network estimates the expected return of the selected action using:

Q = Q ( s t , a t ∣ θ Q )

The learning procedure is summarized in Algorithm 1. At each step, the strategic level updates the local waypoint, the actor proposes a continuous command, the safety layer projects it to the feasible set, and the executed transition is used for critic and actor updates following DDPG (Lillicrap et al., 2016).

Algorithm 1 Proposed Multi-Level DDPG Algorithm for safety-constrained trajectory optimization. Algorithm flowchart-style text outlining steps for training a reinforcement learning policy for robot trajectory optimization with workspace, obstacle, and safety constraints. Includes initialization, episode loop, action projection, reward calculation, and network updates.

Algorithm 1 therefore trains one off-policy continuous controller while enforcing strategic guidance and reactive safety at every step. This corrected description is consistent with the reward definition in Section 3.4 and Figure 3.

3.4 Safety-aware reward function design

The reward comprises the same six components in the text, Figure 3, and Algorithm 1: goal progress, obstacle clearance, action smoothness, mechanical energy, execution time, and safety-constraint satisfaction. All component weights are non-negative, and normalization is required before aggregation so that differently scaled terms do not dominate the update.

The total reward function is mathematically represented as:

R ( s t , a t ) = w 1 R goal + w 2 R obs + w 3 R smooth + w 4 R energy + w 5 R time + w 6 R safe

where w i represents the weighting coefficient of each reward component. The goal-reaching reward minimizes the distance between the current robot position and target position:

R goal = − ‖ p t − p g ‖ 2

where p t is the current position and p grepresents the desired goal position. The smoothness reward reduces sudden motion changes and improves trajectory stability:

R smooth = − ‖ a t − a t − 1 ‖ 2

The safety component penalizes unsafe movements and prevents collision conditions:

R safe = − λ ∑ i = 1 N max ( 0 , g i ( s t , a t ) ) 2

where g i ( s t , a t ) represents the constraint violation function and λ denotes the penalty coefficient.

The overall reward function combines six normalized components corresponding to goal progress, obstacle avoidance, trajectory smoothness, energy efficiency, execution time, and safety constraint satisfaction:

R t = w goal R goal + w obs R obs + w smooth R smooth + w energy R energy + w time R time + w safe R safe .

All reward components are normalized to comparable numerical ranges before aggregation. The weighting coefficients were empirically selected during preliminary training to balance task completion efficiency and safe robot operation. The coefficients used throughout all experiments are

w goal = 1.0 , w obs = 0.8 , w smooth = 0.5 , w energy = 0.2 , w time = 0.2 , w safe = 1.2.

Safety constraint violations are penalized according to

R safe = − λ ϕ ( s t , a t ) ,

where ϕ ( s t , a t ) denotes the constraint violation function and the penalty coefficient is fixed to λ = 10.

This configuration prioritizes collision-free trajectory generation while simultaneously encouraging accurate goal reaching, smooth motion, energy efficiency, and reduced execution time.

3.5 Constraint optimization and policy learning

The reactive level realizes Π C as a minimum-deviation constrained projection. For raw action a t, it solves a t safe = arg min u ∥ u − a t ∥ 2 2 . subject to the one-step joint, velocity, workspace, and obstacle-clearance constraints. The projection is evaluated before every environment or hardware command.

a t safe = arg min u ∥ u − a t ∥ 2 2 subject to q min ≤ q t + Δt u ≤ q max , ∣ u ∣ ≤ v max , p ( q t + Δt u ) ∈ W , d ( p ( q t + Δt u ) , O t ) ≥ d safe .

Here p(.) is forward kinematics, W is the admissible workspace, O t is the current obstacle set, and

d safe is the configured clearance. The action is first clipped to joint and velocity limits, then the quadratic projection is solved for workspace and clearance constraints. If the problem is infeasible or misses the controller deadline, the command is replaced by a zero-velocity emergency action and the event is counted as a constraint violation.

During training, experiences are stored in replay memory:

D = { ( s t , a t , r t , s t + 1 ) }

The critic network is optimized by minimizing the temporal difference error:

L = 1 N ∑ i ( y i − Q ( s i , a i ∣ θ Q ) ) 2

where the target value is calculated as:

y i = r i + γ Q ′ ( s i + 1 , μ ′ ( s i + 1 ) ∣ θ Q ′ )

The actor is updated by the deterministic policy gradient, while target networks use the soft update shown above. Critic targets are computed from the projected next action so that value learning reflects commands that can actually be executed. All DDPG-family baselines use the same episode budget, maximum steps, batch size, replay capacity, and evaluation workspace.

3.6 Simulation-to-real deployment strategy

After simulation training, the policy was exported to the UR5e navigation platform through the robot-controller interface shown in Figure 4. State and action scaling, joint bounds, workspace coordinates, and controller timing were matched before execution. The present record does not contain a controlled ablation of domain randomization or system identification; therefore, the deployment is reported as an empirical transfer demonstration rather than proof of general sim-to-real robustness. Earlier studies used dynamics or domain randomization for simulation-to-real transfer of robotic control (Peng et al., 2018; Tobin et al., 2017).

The final optimized trajectory is represented as:

τ ∗ = { p 1 , p 2 , … , p n }

where each trajectory point satisfies:

C ( τ ∗ ) = { d obs > d safe , q min ≤ q t ≤ q max , v t ≤ v max }

The quantitative real-world trajectory experiment in Figure 5 uses a UR5e collaborative robot with a gripper. Figure 6 shows a different ABB six-axis manipulator in a welding cell. The ABB sequence is a separate feasibility demonstration and is not evidence that the same policy transferred unchanged between robot morphologies. The exact ABB model identifier was not retained in the available experimental record, which limits hardware reproducibility.

3.7 Experimental implementation, metrics, and statistical scope

Table 1 reports the available implementation record: a custom PyTorch-based 3D environment, 200 training episodes, 500 steps per episode, a 1 × 10 6 replay buffer, batch size 64, and the stated learning rates. The navigation and tracking configuration is based on UR5e kinematics. The welding image sequence uses a separate ABB platform and is treated qualitatively.

For a sampled end-effector path { p i } i = 1 N, the evaluation metrics are defined as follows:

L = ∑ i = 2 N ∥ p i − p i − 1 ∥ 2 ,

E traj = 1 N ∑ i = 1 N ∥ p i − p i ref ∥ 2 ,

J = 1 N − 3 ∑ i = 1 N − 3 ∥ Δ 3 p i ∥ 2 2 ,

SI = 1 1 + J ,

where identical temporal and spatial normalization is applied to all compared methods.

The mechanical energy consumption is calculated as

E mech = ∫ 0 T ∑ j = 1 n ∣ τ j ( t ) q ̇ j ( t ) ∣ dt .

The collision rate is defined as

R collision = N collision N episodes × 100 % ,

Where N collision the number of evaluated episodes containing at least one contact.

The success rate is defined as

R success = N success N episodes × 100 % ,

Where N success notes the number of episodes in which the goal is reached without collision or constraint-triggered termination.

The available archive contains one training trace per method and point estimates rather than independent-seed summaries. No standard deviation, confidence interval, or inferential significance test can therefore be reported without new runs. The smoothing rule used to export Figure 7 was also not retained. All comparisons in this manuscript are consequently descriptive; a reproducible resubmission should report at least five independent seeds, the smoothing window, and the number of held-out evaluation episodes per seed.

4 Results

The ML-DDPG framework was evaluated for convergence behavior, trajectory quality, safety, dynamic motion, component contribution, and hardware feasibility. Because the retained results are single-run point estimates, the following comparisons describe the supplied experimental record and do not establish statistical significance. Unless otherwise stated, all quantitative results reported in this study correspond to a representative single training run conducted under the experimental protocol described in Section 4. The reported values are presented as descriptive observations of the proposed framework under the specified experimental conditions and should not be interpreted as estimates of statistical variability across repeated independent training runs.

The reward weighting coefficients and safety penalty coefficient listed in Table 1 correspond to the configuration used in the reported experiments and are identical to those defined in Section 3.4. These values were empirically selected during preliminary training and remained fixed throughout all reported evaluations.

4.1 Training convergence analysis

Figure 7 presents the exported training traces for ML-DDPG, DDPG, TD3, and SAC. In the retained ML-DDPG run, reward rises more rapidly and becomes visually stable after approximately 140 episodes. This is descriptive stabilization within the shown budget, not a formal convergence proof. The single traces do not support a claim of statistically significant superiority.

The single-run success-rate trace reaches 98.7% at the end of the reported ML-DDPG training run, while the displayed actor and critic losses decrease over training. This value is an observation from one seed, not a mean across independent runs. Because the export record does not identify a rolling-average or other smoothing kernel, Figure 7 should be interpreted as the supplied rendered trace; variability and statistical significance cannot be estimated without new multi-seed experiments.

4.2 Trajectory optimization performance

Figure 8 demonstrates the evolution of trajectory generation during different training stages. During the exploration stage shown in Figure 8a, the agent generates highly irregular trajectories with significant deviations and inefficient obstacle avoidance behavior. The generated paths frequently approach obstacles and exhibit excessive motion variations, reflecting the absence of learned navigation knowledge.

Figure 8b represents the partial learning stage, where the agent begins to identify feasible routes toward the target while still exhibiting trajectory deviations and occasional close obstacle interactions. Figure 8c demonstrates substantial performance improvement, with the learned policy producing smoother trajectories and more effective obstacle avoidance. Figure 8d depicts the final optimized trajectory generated after convergence. The trajectory follows a smooth and efficient path, maintains safe distances from all obstacles, and reaches the target without unnecessary deviations. These observations indicate that the proposed ML-DDPG framework progressively learns optimal navigation behavior through continuous interaction with the environment.

4.3 Quantitative comparison of trajectory optimization methods

Table 2 reports single-run point estimates for the evaluated planners. In that retained run, ML-DDPG has the shortest reported path length (1.27 m), lowest trajectory error (1.08 mm), highest smoothness index (0.98), no recorded collision, and a 98.7% success rate. These observations are not presented as means over independent seeds and do not support inferential claims.

Figure 9 visualizes the recorded trajectories. Linear interpolation provides limited obstacle adaptation, while A* and RRT* contain sharper directional changes in this workspace. The DDPG-family paths are smoother, and the ML-DDPG trace has the most direct route among the displayed trajectories. The previous external citation attached to this manuscript-generated comparison has been removed.

4.4 Safety performance evaluation

Safety is a critical requirement for collaborative robotic systems operating in industrial environments. Figure 10 presents the safety performance analysis of the proposed reward optimization strategy. Figure 10a demonstrates that the collision rate continuously decreases throughout training and approaches zero for the proposed ML-DDPG framework. The reduction in collision events indicates that the safety-aware reward function successfully encourages obstacle avoidance behavior.

Figure 10b shows the minimum distance maintained between the robot and surrounding obstacles. The proposed framework consistently achieves larger obstacle clearance compared with other reinforcement learning approaches, indicating more conservative and safer navigation decisions. Figure 10c further illustrates a substantial reduction in constraint violation rates as training progresses. The proposed approach achieves the lowest violation frequency, confirming the effectiveness of the constraint optimization mechanism.

The bar-chart analysis presented in Figures 10d–f further supports these observations. The proposed ML-DDPG achieves the highest safe navigation success rate, the largest average safety reward, and the lowest emergency stop activation frequency. Collectively, these results demonstrate that the proposed reward optimization strategy effectively balances task completion and safety requirements during trajectory generation.

4.5 Dynamic motion profile analysis

Figure 11 compares the dynamic motion characteristics generated by different planning approaches. Figure 11a demonstrates the linear velocity profiles during trajectory execution. The proposed ML-DDPG maintains a stable velocity profile with smooth acceleration and deceleration phases, reducing sudden motion changes. Figures 11b,c illustrate angular velocity and acceleration behaviors, respectively. The proposed framework produces significantly fewer oscillations and lower acceleration peaks than competing approaches.

Figure 11d presents the jerk profiles, which represent the derivative of acceleration and provide an important indicator of motion smoothness. The proposed ML-DDPG exhibits the smallest jerk variations, indicating improved dynamic feasibility and reduced mechanical stress on robotic components. Figure 11e shows cumulative energy consumption during task execution. The optimized trajectory generated by the proposed framework requires the lowest energy expenditure, demonstrating improved efficiency. Figure 11f further reveals the lowest trajectory tracking error among all evaluated methods. These results confirm that the proposed framework generates dynamically feasible trajectories suitable for practical industrial deployment.

4.6 Ablation analysis

Table 3 presents the ablation study conducted to evaluate the contribution of individual components within the proposed framework. The baseline DDPG model achieves a success rate of 93.1% and a trajectory error of 3.42 mm. The introduction of the multi-level control strategy improves success rate and reduces trajectory error, indicating that hierarchical decision-making contributes to more effective navigation behavior.

Table 3. Single-run ablation analysis of the proposed ML-DDPG framework.

Within the retained single-run ablation record, adding the safety-aware reward is associated with fewer observed collisions and a higher reward, while adding the constraint-projection module is associated with fewer safety violations and a higher smoothness index. The complete ML-DDPG configuration produced observed values of 98.7% success, 1.08 mm trajectory error, a 0% collision rate, and the highest reward in that run. These measurements are descriptive single-run observations; they do not establish the statistical contribution of each component without independent-seed repetition.

4.7 Real-world deployment evaluation

Figure 5 demonstrates the deployment of the proposed ML-DDPG framework on a collaborative robot operating in a real industrial environment. Figure 5a presents the experimental setup, including obstacle placement, start location, and target destination. Figure 5b depicts sequential trajectory execution snapshots, illustrating successful navigation around obstacles while maintaining smooth motion behavior.

Figure 5 reports the UR5e navigation and tracking demonstration. The planned and executed trajectories remain close in the displayed trial; the embedded summary reports a 1.28 m path, 10.21 s execution time, 0.018 m average tracking error, 0.042 m maximum tracking error, 0.128 m minimum obstacle distance, no collision, and 17.3 ms computation time per planning step. These values quantify the shown demonstration but do not replace repeated hardware trials.

Figure 6 shows a separate ABB welding-cell sequence. In this demonstration, ML-DDPG supplies the Cartesian path reference and waypoint progression, while the robot’s low-level controller executes joint tracking and the welding equipment controls the process. The images document completion of the illustrated path, but synchronized position, velocity, force, current, weld-quality, and repeatability logs were not supplied. The figure is therefore qualitative feasibility evidence only; it does not establish quantitative welding accuracy, safety certification, or policy transfer across the UR5e and ABB platforms.

5 Discussion

The descriptive results suggest that coupling strategic waypoints, one continuous DDPG policy, and a reactive safety projection can reduce decision burden relative to an unstructured single-level controller. This interpretation differs from option-based hierarchical reinforcement learning, which learns temporally extended subpolicies (Sutton et al., 1999), and from fixed phase-based manipulation, which switches among task stages (Navazani et al., 2026). The present architecture uses continuous waypoint conditioning and a shared policy, so its contribution lies in the coupling mechanism rather than hierarchy alone.

The trajectory results also indicate a useful trade-off between path directness, tracking accuracy, smoothness, and planning latency in the evaluated workspace. Classical planners remain important because they offer interpretable geometric behavior, whereas the learned policy adapts its continuous commands from interaction. The reported advantage should nevertheless be understood as environment-specific until identical evaluation protocols are repeated across seeds, obstacle distributions, payloads, and robot platforms.

Safety improvements in the supplied run are consistent with combining reward penalties and explicit action projection. The projection can prevent a raw command from violating modeled bounds even when the policy is imperfect, which complements objective-level constraints (Achiam et al., 2017; Dalal et al., 2018). However, no recorded collision in one evaluation is not equivalent to a probabilistic safety guarantee or certification for human-robot collaboration.

Several limitations constrain generalization. The retained archive includes one training seed per method and does not preserve the smoothing rule or evaluation-trial count. The numerical reward-weight vector and safety penalty coefficient used in the experiments are fully reported in Section 3.4 and Table 1. The quantitative hardware evidence is limited to the UR5e demonstration, whereas the ABB welding sequence is qualitative and uses a different robot morphology. The study does not test additional collaborative robots, multiple human motion patterns, sensor occlusion, payload variation, network delay, or multi-robot operation. These limitations preclude broad statistical or cross-platform claims.

Potential failure cases include projection infeasibility in narrow passages, repeated action clipping near joint limits, delayed obstacle observations, out-of-distribution obstacle motion, calibration drift, controller latency, and reward trade-offs that favor task progress over clearance. Future work should report five or more independent seeds, confidence intervals and effect sizes, repeated hardware trials on multiple cobot models, ablation of each sim-to-real mechanism, dynamic human-in-the-loop scenarios, and formally verified fallback control.

6 Conclusion

This study presents ML-DDPG as a coupled three-level trajectory-planning framework comprising strategic waypoint generation, one learned DDPG controller, and reactive constrained action projection. In the retained single training run, the framework produced observed values of 98.7% success, 1.08 mm trajectory error, a 0.98 smoothness index, and no recorded collision. These values are single-run observations without estimates of variability and therefore do not establish statistical superiority. The UR5e experiment provides quantitative evidence from one demonstration, while the separate ABB welding sequence provides qualitative feasibility evidence only. The results support further investigation but do not establish certified safety or robot-independent transfer. The principal limitations are the single-seed training record, unavailable variability estimates, an unpreserved smoothing rule and evaluation-trial count, limited quantitative hardware trials, and the absence of diverse cobot and human-interaction scenarios. Future research will prioritize multi-seed statistical evaluation, complete reporting of evaluation procedures, repeated tests on additional collaborative robot platforms, dynamic human-aware constraints, and verified emergency fallback mechanisms.

Generative AI statement

The author(s) declared that Generative AI was used in the creation of this manuscript. The authors used a large language model to assist with proofreading and stylistic editing. The model was not used to generate scientific content or analyze data. All authors reviewed and approved the final text.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Mari Prasath, KSRM College of Engineering, India

Reviewed by: Isak Karabegović, University of Bihać, Bosnia and Herzegovina

Mohsen Navazani, University of Texas at San Antonio, United States