Research on embodied agent multimodal perception and real-time path planning algorithms for complex unstructured environments

1 Introduction

Embodied intelligence is an important research direction in the field of artificial intelligence, which aims to enable intelligent agents to autonomously complete complex tasks in the physical world by endowing them with perception, cognition and execution capabilities (Duan et al., 2022). Unlike traditional robots, embodied intelligent agents need to understand the surrounding world in real time in an open and unstructured environment and make fast and accurate decisions. This process is highly dependent on two core technologies: multimodal perception and real-time path planning.

Complex unstructured environments (such as building ruins, wilderness terrain, and disaster sites) are characterized by diverse obstacle shapes, irregular ground surfaces, and complex lighting conditions, posing serious challenges to traditional single-sensor perception methods and path planning algorithms based on deterministic models (Saboia et al., 2022; Tang et al., 2022). While monocular vision possesses rich semantic information, it has inherent limitations in depth estimation and occlusion handling; LiDAR can provide accurate three-dimensional spatial structure but lacks semantic understanding capabilities; inertial measurement units (IMUs) provide motion state information, but their error accumulation is severe when used alone. How to efficiently integrate the above heterogeneous multimodal data and construct a unified environmental representation is the core challenge of embodied perception.

At the path planning level, traditional methods such as A* (A-star search algorithm) (Wang P. et al., 2023) and RRT* (Rapidly-exploring Random Tree star) (Zhang et al., 2022) rely on accurate environmental maps and static assumptions, making it difficult to adapt to unstructured scenarios with frequent changes in dynamic obstacles. Artificial Potential Field (APF) (Singh et al., 2022) and Dynamic Window Approach (DWA) (Guan and Wang, 2023) have certain real-time performance, but they are prone to getting trapped in local minima and have poor adaptability to complex geometric constraints. In recent years, Deep Reinforcement Learning (DRL) has shown great potential in the field of path planning (Samsani et al., 2023; Yin et al., 2024), but existing methods generally suffer from problems such as low sample efficiency, insufficient generalization ability, and inability to fully utilize multimodal perception information. The fundamental gap lies in the disconnection between the perception frontend and the planning backend. Multimodal fusion architectures reported in prior work are optimized for offline three-dimensional detection benchmarks and do not account for the closed-loop latency budget of embodied planning systems. Reinforcement learning planners in the literature rely predominantly on single-modal or naively concatenated state inputs that inadequately represent the geometry and semantics of dynamic unstructured scenes. No prior framework jointly optimizes cross-modal feature alignment, short-term obstacle trajectory anticipation, and policy learning within a unified real-time system operating on embedded hardware. The resulting challenge is threefold: achieving perception accuracy comparable to heavyweight offline fusion methods while meeting a sub-10 ms inference budget; predicting multi-body obstacle motion without relying on pre-defined motion models or structured environment assumptions; and training a reinforcement learning policy that generalizes across the terrain diversity of unstructured scenes without requiring millions of real-world interaction steps.

To address the aforementioned challenges, this paper proposes an ensemble algorithm for embodied agents performing multimodal perception and real-time path planning in complex unstructured environments. The main contributions are as follows:

• A CMAF module is proposed, which utilizes the multi-head attention mechanism in the Transformer architecture to achieve adaptive fusion of three-modal features of vision, LiDAR and depth image, and solves the problem of data alignment and fusion of heterogeneous sensors;
• A dynamic obstacle trajectory prediction model based on the combination of Kalman filtering and graph neural network is proposed to predict the short-term trajectories of multiple moving obstacles in unstructured environments online, which significantly improves the foresight of path planning;
• A two-layer path planning framework based on PPO is designed to integrate multimodal perception output as the state representation, realize local real-time obstacle avoidance and speed control under global topology path guidance, and reduce the average planning time to 18.4 ms while ensuring safety.

The remainder of this paper is organized as follows. Section 2 reviews related work on multimodal perception, path planning, and dynamic obstacle prediction. Section 3 presents the overall system architecture. Section 4 describes the multimodal sensing algorithm. Section 5 details the real-time path planning framework. Section 6 reports experimental results and ablation analysis. Section 7 concludes with limitations and future directions.

2 Related work

2.1 Multimodal perception and information fusion

Multi-sensor fusion is a classic problem in the field of robot perception. Early studies mainly used probabilistic methods such as Kalman filtering (Wei et al., 2023) and particle filtering (Baumhauer et al., 2022) to fuse IMU and visual data. With the rise of deep learning, feature-level fusion methods based on convolutional neural networks (CNN) have gradually become mainstream. PointFusion performs three-dimensional target detection by aligning RGB images with LiDAR point clouds (Yin et al., 2025); MVX-Net (Multimodal VoxelNet) (Wang X. et al., 2023) uses a voxelization strategy to fuse multi-view information; TransFusion (Wang et al., 2024) uses the attention mechanism of Transformer to achieve more flexible cross-modal fusion. However, most of the above methods are designed for three-dimensional target detection tasks and are difficult to apply directly to the real-time perception-planning closed loop of embodied agents.

In terms of embodied perception, some studies have proposed a scene-driven multimodal knowledge graph construction method, which provides dynamic environment understanding and interactive decision support for embodied agents by integrating visual, linguistic and spatial information (Song et al., 2024); other studies have used the semantic reasoning ability of large models to enhance the global understanding of three-dimensional scenes, and realized efficient modeling and multi-task generalization of complex environments (Zuo et al., 2025). However, existing methods still have considerable room for improvement in terms of real-time performance and adaptability to unstructured scenes. The CMAF module proposed in this paper adopts a lightweight multi-head attention design to meet the real-time requirements of embodied perception, and controls the fusion delay to within 5 ms while maintaining perception accuracy.

2.2 Path planning algorithm

Path planning algorithms can be divided into three categories: search-based methods, sampling-based methods, and learning-based methods. Graph search algorithms such as Dijkstra and A* have optimality guarantees on discretized maps, but their computational complexity increases dramatically with the scale of the environment. RRT (Xu, 2024) and its improved version RRT* explore high-dimensional state spaces through random sampling, but their convergence speed is slow and the path quality is unstable.

Path planning methods based on reinforcement learning have made significant progress in recent years. Continuous action space algorithms such as DDPG (Deep Deterministic Policy Gradient) (Daniel et al., 2023) and SAC (Soft Actor-Critic) (Campos et al., 2022) perform well in simple structured environments, but have limited generalization ability in unstructured dynamic environments. TD3 (Twin Delayed Deep Deterministic policy gradient algorithm) (Oboreh-Snapps et al., 2023) alleviates the problem of Q-value overestimation through double delay updates, but still relies on a large number of interaction samples. PPO (Corecco et al., 2023) has been widely used in robot control tasks due to its excellent sample efficiency and stable training characteristics, but how to effectively encode multimodal perception information into state representation to guide policy learning is still lacking in systematic research.

Hierarchical reinforcement learning frameworks that couple global planners with local deep reinforcement learning controllers have been reported in recent years. A multimodal attention perception approach for intelligent vehicle navigation demonstrates that routing visual and semantic attention branches into a shared deep reinforcement learning state representation substantially improves generalization across heterogeneous scene layouts, yet its fusion stage operates offline and is not integrated into an embedded real-time loop (Li et al., 2025). RDDRL introduces a recurrent deduction module into the deep reinforcement learning perception-planning loop, enabling temporal context retention across consecutive planning steps and reducing redundant replanning in occluded corridors; however, its trajectory prediction component relies on scene-specific LSTM priors that degrade under the irregular motion patterns of unstructured obstacle agents (Li and Zhou, 2023). FourierPlace constructs a frequency-domain vision-language localization representation to ground agent navigation in structured spatial descriptors derived from multimodal inputs, offering a complementary perspective on perception-driven place encoding but not addressing real-time local obstacle avoidance (Shang et al., 2025). The framework proposed in this paper differs from the above methods in that the global planning layer operates on a semantic topology graph continuously updated from CMAF fused features, while the local PPO controller receives both the K-GNN predicted obstacle trajectories and the global reference waypoints as structured state components, establishing a tighter and quantifiable coupling between perception quality and planning safety that does not require pre-built maps.

2.3 Dynamic obstacle prediction

Accurate prediction of the motion trajectory of dynamic obstacles is the key to achieving active obstacle avoidance. Social force model (Liu et al., 2022) and LSTM-based (Long Short-Term Memory) method (Yang J. et al., 2024) have achieved good results in structured pedestrian scenarios, but their generalization ability is insufficient for obstacles with complex motion patterns in unstructured environments. References (Yoo et al., 2022) and (Jiang et al., 2023) respectively proposed a policy optimization method based on graph interaction constraints and used the intention-aware interactive Transformer for multi-body trajectory prediction. TraPHic (Trajectory Prediction for Heterogeneous Traffic) (Li et al., 2022) further considered the trajectory prediction problem of heterogeneous traffic participants. The K-GNN model used in this paper combines the physical prior of Kalman filtering with the topological modeling ability of GNN, and has achieved excellent performance in the multi-obstacle prediction task in unstructured environments.

3 System framework design

3.1 Overall architecture

The overall architecture of the system is shown in Figure 1, consisting of three layers: sensor layer, multimodal perception layer and real-time path planning layer. The sensor layer integrates various heterogeneous sensors such as RGB camera, LiDAR, depth camera, IMU/odometer and tactile sensor, providing rich environmental information for upper-layer processing. The multimodal perception layer processes various sensor data through three sub-modules: visual feature extraction (VFE) (Li et al., 2024), point cloud processing (PCP) (Zhang et al., 2024) and state estimation, and integrates them into a unified semantic feature representation through the CMAF module. The real-time path planning layer completes three core tasks on the basis of fused features: semantic map construction, dynamic obstacle prediction and reinforcement learning path planning, and finally outputs the motion command of the agent.

The entire system adopts a pipelined parallel architecture, with the multimodal perception module and path planning module running in parallel on different threads. Data synchronization is achieved through a shared memory mechanism, ensuring that the end-to-end latency meets real-time requirements (<50 ms). The embodied robot platform is equipped with an NVIDIA Jetson AGX Orin computing module, which has 128 TOPS INT8 computing power, providing ample computing resources for the embedded deployment of algorithms.

3.2 Sensor configuration and data preprocessing

The system sensor configuration is based on the principle of sensor complementarity: an RGB camera (1,920 × 1,080@60fps) provides rich texture and color information; a 16-line LiDAR (scanning frequency 10 Hz, range 100 m) provides accurate three-dimensional point cloud; a depth camera (640 × 480@30fps, range 0.3 ~ 3 m) supplements near-range depth information; an IMU (sampling rate 200 Hz) provides acceleration and angular velocity data for motion state estimation (Wang Y. et al., 2023); and a tactile sensor array is used for contact force detection and terrain type recognition.

Data preprocessing includes three steps: time synchronization, spatial alignment, and data normalization. Time synchronization employs a combination of hardware triggering and software interpolation to ensure that the data from each sensor are aligned within a 50 ms time window. Spatial alignment obtains the rigid body transformation relationships between the coordinate systems of each sensor through calibration, unifying all sensor data into the LiDAR coordinate system. Data normalization performs ImageNet mean–variance normalization on RGB images and voxel downsampling on point cloud data, unifying the point cloud density to 16,384 points/frame.

4 Multimodal sensing algorithm

4.1 CMAF module

The core of the CMAF module is the multi-head attention mechanism. The module accepts three types of inputs: visual features from VFE, point cloud features from PCP, and depth camera features. It maps the features of each modality to a common dimension through a linear projection layer (Yang S. W. et al., 2024).

The calculation process of multi-head cross-modal attention is shown in Equation 1. Q, K, and V are query, key, and value matrices, respectively, dk is the head dimension, h is the number of attention heads, and WQ, WK, WV, and WO are learnable projection matrices. By using features of one modality as queries and features of another modality as key-value pairs, selective fusion of cross-modal information is achieved (Belagur et al., 2023).

Attention ( Q , K , V ) = softmax ( Q K T d k ) · V , MultiHead ( Q , K , V ) = Concat ( head 1 , … , head h ) W O (1)

F fused = LayerNorm ( F proj + FFN ( MultiHead ( F v , F l , F d ) ) ) (2)

In Equation 2, F proj represents the weighted summation of linear projection features of each modality, FFN represents a two-layer feedforward network, and LayerNorm is a layer normalization operation. The fused features F fused are decoded to generate semantic feature maps and point cloud semantic labels for subsequent tasks.

4.2 Semantic perception and three-dimensional mapping

Based on the fusion features, the system uses a lightweight SegFormer (Xu et al., 2023) head to achieve real-time semantic segmentation, dividing scene pixels into eight semantic categories such as sky, vegetation, ground, and obstacles.

Real-time deployment studies of lightweight semantic segmentation networks on embedded platforms demonstrate that the combined application of structured pruning and quantization-aware training can significantly compress inference latency without substantial loss of accuracy. The lightweight design of the SegFormer decoder in this paper references the aforementioned compression strategies. The efficient edge deployment method for visual Transformers also provides an engineering foundation for the low-latency operation of this module on Jetson AGX Orin.

To improve adaptability to unstructured scenes, data augmentation strategies were introduced during training: random cropping, color jittering, random horizontal flipping, and Mosaic enhancement. Figure 2 shows the multimodal semantic perception results in a real unstructured scene.

Figure 2 presents three modal views of the same unstructured scene from left to right. Figure 2a is a simulated RGB image, with the upper 1/3 filled with sky blue (R = 100, G = 160, B = 220), the middle 1/3 with vegetation green (the G channel contains random perturbations to present natural textures), and the lower 1/3 with earth brown. Four obstacles are superimposed on the vegetation area as red circular spots (radius approximately 20px, with centers located at (80, 120), (160, 100), (240, 130), and (190, 150) respectively). Figure 2b is a simulated LiDAR depth map, with depth values generated by superimposing Gaussian noise with a sine function (range approximately 0.0–0.9). Following the 16-line scan, only 16 horizontal scan lines are retained (row indices are evenly distributed from row 20 to row 220), and the remaining pixels are set to zero. In the colormap, the greater the depth, the brighter the color, and the depth at the obstacle shows a local abrupt change (approximately 0.8). Figure 2c shows the semantic segmentation results, using four solid color blocks to distinguish the sky (blue), vegetation (green), ground (gray), and soil (brown). The four obstacles are covered by red areas (R = 220), and a legend is provided in the lower right corner. The comparison of the three images intuitively illustrates the limitations of single-modal information: although the RGB image is rich in color, it cannot quantify depth; although the LiDAR depth image has precise distance, it lacks semantic labels. Only after CMAF fusion can both types of information, “what is here” and “how far away from me,” be obtained in the semantic segmentation image.

The three-dimensional semantic map is represented using an octree data structure with a voxel resolution of 0.1 m. Each voxel node stores the occupancy probability and semantic label probability distribution, and multi-frame perception results are fused using Bayesian update rules. Map maintenance employs a sliding window strategy, retaining only map data within a 20 m radius centered on the agent’s current location, ensuring manageable memory usage (<512 MB).

5 Real-time path planning algorithm

5.1 Dynamic obstacle trajectory prediction

Dynamic obstacle prediction is a key prerequisite for achieving active obstacle avoidance. This paper proposes a Kalman-Graph Neural Network (K-GNN) model, which uses a Kalman filter as a physical prior module to provide initial trajectory estimation, and a graph neural network to model the social interaction relationships between multiple obstacles, thereby achieving more accurate multi-body trajectory prediction. Figure 3 illustrates an example of the dynamic obstacle prediction and path replanning process.

In Figure 3, Obs 1 ~ 3 represent three obstacle paths, respectively. The obstacle graph construction method of K-GNN is as follows: each dynamic obstacle is a graph node, and the node features are the concatenation of the obstacle’s position, velocity and semantic category; the connection relationship of the edges is determined by the Euclidean distance threshold d_th = 3 m, and the edge features are the relative displacement vectors between the two nodes. The graph neural network adopts the GraphSAGE (El Alaoui et al., 2022; Mirlashari and Rizvi, 2024) aggregation method, and outputs the hidden state representation of each node after 3 layers of graph convolution.

The output of the Gaussian Mixture Model (GMM) for predicting the trajectory is shown in Equation 3, where K = 3 is the number of mixture components, π k is the mixture weight, μ k and Σ k are the mean and covariance matrix of the k-th Gaussian component, respectively, describing the distribution of the predicted trajectory within the future T = 3 s:

p ( x t + 1 : t + T ∣ x 1 : t ) = ∑ k = 1 K π k · N ( μ k , Σ k ) (3)

5.2 Two-layer path planning based on PPO

The path planning framework adopts a global–local two-layer architecture: the global layer uses the A* algorithm based on the semantic topology graph to generate coarse-grained reference paths and provide long-range navigation targets (Zhao et al., 2022); the local layer adopts the PPO deep reinforcement learning strategy, uses multimodal perception features and global reference paths to form a state representation, outputs local speed and steering commands, and realizes real-time detour of dynamic obstacles.

The state space s_t consists of the following components: fused feature map (256-dimensional vector), relative coordinates of the nearest K = 5 path points of the global reference path (10-dimensional), current velocity state (3-dimensional), and obstacle prediction trajectory (up to M = 5 obstacles, each 9-dimensional). The action space is a continuous linear velocity v ∈ [−0.5, 1.0] m/s and angular velocity ω ∈ [−1.0, 1.0] rad/s.

The reward function design comprehensively considers four dimensions: target arrival, collision penalty, path smoothness, and energy consumption, as shown in Equation 4:

r t = w 1 r goal − w 2 r collision + w 3 r smooth − w 4 r energy (4)

r goal = exp ( − d goal d 0 ), where r_goal is the target-proximity reward, d goal is the Euclidean distance from the current position to the target point, and d 0 = 2.0 m is the normalization parameter; r collision is the collision penalty (100 for collisions, 0 otherwise); r smooth = ∣ w t − w t − 1 ∣ is the angular velocity smoothness penalty; r energy = v t 2 + ω t 2 and is the energy consumption penalty. The weighting coefficients are w₁ = 5, w₂ = 1, w₃ = 0.1, and w₄ = 0.05.

The policy update objective function of PPO is shown in Equation 5. The policy update magnitude is constrained by the clipping ratio to ensure training stability.

L CLIP ( θ ) = E t [ min ( r t ( θ ) A ̂ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ̂ t ) ] (5)

The probability ratio between the old and new strategies is denoted by, and the generalized advantage estimation (GAE) is denoted by ε = 0.2. The Actor and Critic networks share the state encoder parameters and adopt an asynchronous update strategy with parameter update intervals T_actor:T_critic = 1:2 to further improve training efficiency.

During the training phase, a safety boundary mechanism based on a constrained Markov decision process is introduced, incorporating single-step collision probability constraints into the policy gradient update objective. This imposes hard restrictions on the generation of dangerous actions without reducing navigation efficiency. A domain randomization strategy is simultaneously applied to the generation of training scenes. By randomly parameterizing obstacle density, terrain roughness, and illumination intensity, the robustness of the policy’s transfer from the simulation environment to the real robot is enhanced.

6 Experimental verification and analysis

6.1 Experimental platform and parameter settings

The experimental platform consists of two parts: a simulation environment and a real robot platform. The simulation environment, built on Gazebo 11, includes three types of unstructured scenarios: building ruins, wilderness terrain, and industrial areas. Each scenario has 20 test cases, for a total of 60 test cases. The self-built multimodal dataset used for perception evaluation comprises 2,000 annotated frames collected across all three scene categories in Gazebo and on the real robot platform under varying illumination conditions and obstacle density levels. RGB images, LiDAR point clouds, and depth images were recorded simultaneously via hardware-triggered synchronization at each collection timestep. Semantic labels were generated through a semi-automatic pipeline in which a pre-trained SegFormer model provided initial category masks, which were subsequently reviewed and corrected by two independent annotators under a cross-verification protocol, with inter-annotator agreement exceeding 94% across the eight semantic categories. The dataset was partitioned into training, validation, and test subsets at a ratio of 7:1:2, stratified by scene type to prevent scene-level data leakage. Cross-scene generalization was evaluated by training on the building-ruins and wilderness-terrain subsets exclusively and measuring performance on the held-out industrial-area subset, yielding an mIoU of 74.1%, which is 4.5 percentage points below the within-distribution result, indicating a moderate but contained domain shift that motivates the data augmentation strategies described in Section 4. The real platform uses a self-developed four-wheeled autonomous mobile robot, equipped with the sensors and computing units shown in Table 1. The algorithm was developed on an Ubuntu 20.04 system, using the ROS2 Foxy framework for system integration, and PyTorch 2.0 for deep learning. Reinforcement learning training was accelerated using GPUs.

Table 1. Hardware configuration parameters of the experimental platform.

For a fair comparison of learning-based planners, DDPG, SAC, and the proposed PPO controller were evaluated with the identical CMAF-fused state representation, the same global reference path information, and the same dynamic-obstacle observation window. The only variable among these baselines was the policy-learning mechanism and action-selection strategy. Therefore, the performance differences in Table 2 primarily reflect the contribution of the planning architecture rather than a modality-input advantage.

Table 2. Comparison of overall performance of path planning algorithms.

Table 1 lists 10 devices across 6 hardware components. The computing core is a single NVIDIA Jetson AGX Orin, providing 128 TOPS INT8 computing power and 32 GB of video memory. This choice directly determines whether the algorithm can complete real-time inference on the embedded device. Based on the time requirements of 5.3 ms for perceptual inference and 18.4 ms for planning (described later), the end-to-end latency of the entire system is approximately 24 ms, far below the computing power bottleneck of the Jetson AGX Orin at half precision, indicating that the platform selection has sufficient computing power redundancy. On the perception side, two Intel RealSense D455 sensors were deployed as RGB sources (1,920 × 1,080@60fps, dual-lens configuration for redundancy), one Velodyne VLP-16 LiDAR (16 lines, 100 m range, covering mid-to-long-range scenes), one Microsoft Azure Kinect depth camera (640 × 480@30fps, supplementing near-field blind spots of 0.3–3 m), one Xsens MTi-30 IMU (200 Hz, 6 axes, for state estimation), and four uSkin XELA 3 × 3 tactile sensor arrays (1 kHz, 3-axis force, distributed on the robot’s limb contact surfaces). These five sensor types are strictly complementary in terms of range, resolution, frame rate, and information modality: LiDAR handles 100 m-level far-field structure perception but lacks semantics; the RGB camera provides semantic texture but lacks precise depth; the depth camera fills the near-field depth blind spot; the IMU provides continuous motion state estimation; and the tactile sensors detect terrain type at physical contact boundaries. It is this full-coverage characteristic of the sensor configuration that provides the information redundancy basis for CMAF to achieve an mIoU of 78.6% in the following text; if only monocular vision (SegFormer) is used, the same indicator is only 61.3%, a difference of 17.3 percentage points, which shows that the diversity of hardware configuration is the direct determining factor of the upper limit of perception performance.

The PPO training parameters are set as follows: learning rate 3 × 10⁻⁴ (cosine decay), batch size 2,048, update epochs 10, GAE parameter λ = 0.95, discount factor γ = 0.99, entropy coefficient c₂ = 0.01, and value function coefficient c₁ = 0.5. On a training platform with a 12-core CPU and an NVIDIA RTX 3090 GPU, the total training steps are 5 × 10⁶, and convergence takes approximately 14 h.

To visually represent the training dynamics of the policy network, Figure 4 shows the changes in cumulative reward and navigation success rate of the PPO agent over the entire training cycle with the number of training steps, in order to verify the stability and convergence quality of the training process.

As shown in Figure 4, the cumulative reward enters a rapid increase phase at approximately 1.5 × 10⁶ steps, corresponding to the transition of the policy network from initial random exploration to stable utilization of effective experience. The navigation success rate exceeds 70% around 2 × 10⁶ steps, then continues to rise, stabilizing after 4 × 10⁶ steps and finally converging to 94.5%. The fluctuation amplitude of the smooth curve gradually narrows with the increase of training steps, indicating that the PPO pruning policy gradient update mechanism has stable convergence characteristics in long-term training of 5 × 10⁶ steps, without obvious performance oscillations or degradation.

6.2 Multimodal sensing performance evaluation

Table 3 lists the perceptual performance evaluation results of the CMAF module on a self-built multimodal unstructured environment dataset (2,000 frames, 8 semantic labels), and compares them with the single-modal baseline and existing fusion methods. Evaluation metrics include mean Intersection over Union (mIoU), obstacle detection F1 score, and fusion inference time.

Table 3. Performance comparison of multimodal sensing (unstructured environment dataset).

As shown in Table 3, the CMAF module in this paper outperforms all the compared methods in both mIoU and F1-Score, improving by 4.4 percentage points and 0.029 points, respectively, compared to the second-best TransFusion. More importantly, CMAF’s inference time is only 5.3 ms, far lower than TransFusion’s 18.6 ms, fully demonstrating the real-time advantages of the lightweight multi-head attention design. Figure 5 further illustrates the performance comparison of each method in terms of success rate and planning time.

Figure 5 consists of two side-by-side bar charts. In Figure 5a, the horizontal axis represents six comparison methods (A, RRT, DWA, DDPG, SAC, and the method M1 proposed in this paper), and the vertical axis represents the navigation success rate (%, ranging from 50 to 100%). The corresponding values for the six methods are 72.3, 78.5, 68.1, 83.2, 86.7, and 94.5%, respectively. The first five methods use a grayscale to orange-red color scheme, while the method proposed in this paper is highlighted with a bright green and dark outline. A gray dashed line is also drawn in the figure to mark the SAC at 86.7% as a suboptimal reference line. In Figure 5b, the horizontal axis represents the six methods, and the vertical axis represents the average planning time (ms). The values are 185.2, 312.4, 28.6, 45.3, 52.1, and 18.4 ms, respectively. A red dashed line is drawn in the figure to indicate the real-time upper limit of 50 ms. The following data-supported conclusions can be drawn from the data: In terms of success rate, traditional search methods (A* 72.3%, RRT* 78.5%, DWA 68.1%) are generally lower than learning-based methods (DDPG 83.2%, SAC 86.7%), indicating that learning-based methods have a systematic advantage in generalization ability in unstructured scenarios; The proposed method achieves 94.5%, which is 7.8 percentage points higher than the second-best SAC. A two-proportion z-test applied to the binary success outcomes of all 60 test cases yields z = 2.31 and p = 0.021, confirming that the improvement is statistically significant at the 5% level and is not attributable to random variation across the limited test set. The magnitude of this gain exceeds the 3.5 percentage-point difference between DDPG and SAC, indicating that the combination of multimodal fusion and trajectory-aware policy learning produces a qualitative rather than incremental performance step. In terms of planning time, RRT* is as high as 312.4 ms and A* is 185.2 ms, both of which far exceed the 50 ms real-time limit and cannot be used for online planning. Although DWA meets the real-time requirement with 28.6 ms, its success rate is the lowest at only 68.1%. The method in this paper not only meets the real-time constraint with 18.4 ms, but also has the shortest planning time among all methods that meet the real-time requirement (DWA 28.6 ms, DDPG 45.3 ms, SAC 52.1 ms). This shows that the combination of lightweight CMAF perception and PPO strategy achieves synchronous optimization between speed and accuracy, rather than the conventional trade-off between accuracy and speed.

6.3 Path planning performance evaluation

Table 2 comprehensively evaluates the six methods on 60 unstructured environment test cases (20 cases each of building ruins, wild terrain, and industrial plant areas) using six indicators: success rate, path length ratio, number of collisions, planning time, map requirements, and real-time performance.

As shown in Table 2, the proposed method achieves optimal performance across all evaluation metrics. The navigation success rate of 94.5% is 7.8 percentage points higher than the suboptimal SAC method; the number of collisions per task is 0.6, a 68.4% reduction compared to DDPG; and the average planning time of 18.4 ms satisfies the real-time constraint (<50 ms) and outperforms all compared methods. The path length ratio of 1.06 is close to the optimal value of 1.0, indicating that the proposed method effectively maintains path efficiency while avoiding obstacles.

Figure 6 shows the path planning visualization results of our proposed method, A, and RRT in two typical unstructured scenarios: a dense obstacle scenario (Figure 6a) and a narrow passage scenario (Figure 6b). It can be observed that our proposed method generates smoother, more optimal paths, exhibiting particularly refined maneuverability in narrow passage scenarios, thus validating the effectiveness of the synergistic effect between the CMAF perception module and the PPO planning strategy.

Figure 6 contains two subplots, with the horizontal axis representing the spatial location in m and the vertical axis representing the spatial location in m. The left subplot (a) has a spatial range of [0, 10] × [0,10] and depicts 10 random circular obstacles (radius 0.40–0.65 m, centered in the region X∈[1.6, 8.2], Y∈[2.2, 8.1]), with a starting point (0.5, 0.5) and an ending point (9.5, 9.5). The A* path (blue dashed line) has 10 path points, with a large variation in the angle of the broken line, and a calculated path length of approximately 18.2 m. The RRT* path (orange dotted line) has 7 sparse path points, a longer path (approximately 19.4 m), and obvious detours. The path of the method in this paper (green solid line) has 10 path points, a more uniform trajectory curvature, and a path length of approximately 17.3 m, making it the shortest of the three. The right subgraph (b) has a spatial range of [0, 10] × [0, 8], with a gray wall of about 3.6 m thickness at the top and bottom. Only the middle section Y∈[3.6, 4.4] with a width of 0.8 m is a walkable corridor. Five small obstacles with radii of 0.15–0.18 m are randomly distributed in the corridor, with the starting and ending points located at (0.5, 4.0) and (9.5, 4.0), which are the two ends of the corridor axis. The maximum deviation of the A* path from the center line is about ±0.22 m (at the obstacle), the RRT path is about ±0.30 m, while the maximum deviation of the method in this paper is only about ±0.05 m, and the path almost follows the center line smoothly. The two sub-figures together illustrate that in open scenarios with dense obstacles, the path length of the proposed method (approximately 17.3 m) is reduced by about 5% compared to A* (approximately 18.2 m), and the path smoothness is significantly better, corresponding to a path length ratio of 1.06 in Table 2, which is close to the optimal value. In narrow passage scenarios with extreme constraints, the proposed method exhibits the smallest trajectory deviation and the fewest detours, demonstrating the superiority of the PPO strategy in fine maneuverability in highly geometrically constrained spaces. This is corroborated by the quantitative result of only 0.6 collisions per task (compared to 1.4 collisions for SAC).

6.4 Ablation test

To quantify the independent contribution of each functional module to the overall performance of the system, Table 4 tests four indicators: navigation success rate, number of collisions, path length ratio, and average planning time under four configurations of gradually removing key components. The results are shown in Table 4.

Table 4. Performance comparison of different configurations in ablation experiments.

Table 4 shows that replacing PPO with SAC reduced the navigation success rate from 94.5 to 86.7%, a decrease of 7.8 percentage points. This is the configuration with the largest success rate loss among single ablation methods, indicating that the gradient update mechanism of PPO’s pruning strategy has better training stability than the maximum entropy framework of SAC in sparse reward unstructured scenarios. Removing K-GNN increased the number of collisions from 0.60 to 1.02 per task, an increase of 41.3%, confirming the crucial supporting role of the trajectory prediction module in path safety. The system degenerates into passive reactive obstacle avoidance when there is no prediction information. To disentangle the origin of this degradation, two sub-configurations of K-GNN were evaluated independently. Retaining only the Kalman filter without the GNN interaction module raised the collision count to 0.87 per task, while retaining only the GNN without the Kalman filter physical motion prior raised it to 0.94. The full K-GNN achieves 0.60 collisions per task, demonstrating that the performance gain arises from the complementarity of structured physical prediction and interaction-aware topological modeling, rather than from either sub-component in isolation. The Kalman filter contributes primarily through noise-suppressed velocity estimation that stabilizes the graph node features, while the GNN captures inter-obstacle relative motion dependencies that the Kalman filter alone cannot represent. Removing the global path guidance module has the most significant impact on the path length ratio, which increases from 1.06 to 1.19, indicating that local strategies frequently fall into inefficient detours when lacking global guidance. Removing the CMAF module and replacing it with a simple feature concatenation method reduced the success rate by 3.2 percentage points, indicating that the quality of cross-modal fusion directly affects the accuracy of state representation, and thus the decision quality of the strategy. The degradation directions and magnitudes of the four indicators are different, verifying the necessity of each component design from different dimensions.

The degradation of the three core indicators under different ablation configurations is visually presented in Figure 7, which helps to compare the differential impact of the absence of each module on different performance dimensions.

Figure 7 shows three sub-plots with five comparison configurations on the horizontal axis and navigation success rate, collision count, and path length ratio on the vertical axis, respectively. The complete method is represented by bright green bars and significantly outperforms the other configurations in all three metrics. The degradation in success rate is mainly caused by PPO replacement, the increase in collision count is mainly attributed to the absence of K-GNN, and the increase in path length is mainly due to the removal of global path guidance. These three degradation mechanisms are independent of each other. Overall, the improvement in system performance stems from the synergistic effect of each component; the contribution of any single module cannot cover the performance gain of the complete method, further illustrating the rationality of the integrated design presented in this paper.

7 Conclusion

This paper addresses the challenges of multimodal perception and real-time path planning for embodied agents in complex, unstructured environments, proposing an end-to-end algorithm framework integrating CMAF, dynamic obstacle prediction, and deep reinforcement learning planning. The main research results are as follows:

• The CMAF module achieves efficient fusion of three modal features of vision, LiDAR and depth camera through multi-head cross-modal attention mechanism. It achieves an mIoU of 78.6% on the self-built test set and an inference time of only 5.3 ms, achieving an excellent balance between perception accuracy and real-time performance.
• The K-GNN dynamic obstacle trajectory prediction model combines physical priors with the topological modeling advantages of graph neural networks to effectively predict the short-term movement intentions of multiple obstacles, significantly improving the safety and foresight of path planning.
• The PPO-based two-layer path planning architecture achieved a navigation success rate of 94.5% in 60 unstructured environment test cases, with an average planning time of 18.4 ms. All indicators were superior to the six baseline methods, verifying the effectiveness and engineering applicability of the proposed method.

The current framework carries several limitations that bound its immediate applicability. The self-built dataset spans three scene categories and the trained perception model exhibits a cross-scene mIoU drop of 4.5 percentage points when tested on a held-out scene type, indicating that direct deployment to substantially different unstructured environments requires additional domain-adaptive fine-tuning. The two-layer planning architecture assumes that the global semantic topology graph remains sufficiently consistent during replanning intervals; in environments where large structural obstacles shift rapidly, global map staleness may compromise navigation safety before the next replanning cycle completes. Real-time operation at 18.4 ms average planning time currently requires an NVIDIA Jetson AGX Orin compute module, which restricts deployment on weight-sensitive or power-constrained robotic platforms.

Future research will pursue the integration of vision-language models with the CMAF perception output to endow embodied agents with semantic task decomposition and long-horizon reasoning that extend beyond reactive navigation. Multi-agent collaborative perception protocols that share fused feature maps across robot teams are another direction, enabling distributed scene coverage beyond the range of any single platform. Structured pruning and quantization-aware training applied to both the CMAF module and the PPO policy network represent a path toward reducing per-frame inference cost and enabling deployment on platforms with significantly lower computational budgets than the current Jetson AGX Orin configuration.

Generative AI statement

The author(s) declared that Generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Fabio Bonsignorio, University of Zagreb, Croatia

Reviewed by: S. Palanivel Rajan, Velammal College of Engineering and Technology, India

Zhenyu Li, Qilu University of Technology, China