DOA estimation for autonomous vehicles by exploiting second-order statistics of Co-array domain of nested arrays by implementing Cuckoo Search Algorithm

1 Introduction

The technological surge in autonomous vehicles has increased the demand for reliable and precise sensing technologies. Estimating the Direction of Arrival (DOA) [1] is especially important in ensuring safe and efficient design and operation of autonomous systems by allowing accurate and intelligent decisions [2]. This is essential for collision avoidance and driving safety, as well as for enabling active participation and communication.

Uniform linear arrays (ULAs) and non-uniform linear arrays (NULAs) are two widely used antenna array configurations that differ primarily in the spacing between their antenna elements [3, 4] Owing to their distinct structural characteristics, these arrays have been extensively applied in communication, radar, sonar, medical imaging, seismology, and signal processing applications [5–8]. ULA antennas consist of elements evenly spaced and can be employed in the design of directional antennas by using the beam forming method due to their uniform spacing. Each antenna element in a ULA can provide one degree of freedom, and one degree of freedom is typically lost as a result of the linear structure of the array. This tells us that a ULA can sense or transmit signals in N - 1 independent spatial directions where N is the total number of antenna elements. With N antenna elements, the spatial covariance matrix R x x ∈ C N × N has rank at most K in the signal subspace. Subspace-based DOA estimation requires that the noise subspace be non-empty, i.e., N − K ≥ 1, to distinguish the signal and noise subspaces via eigenvalue decomposition. Consequently, at least one dimension of the N-dimensional observation space must be reserved for the noise subspace, limiting the maximum number of resolvable sources to N − 1.

In the context of autonomous car driving, one such important factor is the degree of freedom, which also inspires us to investigate into non-uniform linear arrays. The most common configurations of non-uniform linear arrays, which include minimum redundancy arrays (MRA) [9], coprime sensor arrays (CSA) [10], nested arrays (NA) [11], and arrays with constraints on maximum inter-element spacing (MISC) [12], are also the most interesting due to their increased degree of freedom.

This paper proposes a new technique for enhancing the performance of DOA estimation by utilizing the second-order statistics of co-array domain nested arrays. Nested arrays have become a popular choice in the design of arrays due to their ability to increase spatial resolution and reduce DOA estimation ambiguity [13]. This paper provides a way to increase DOA estimation accuracy using co-array domain second-order statistics.

Furthermore, there are numerous meta-heuristic algorithms such as genetic algorithm (GA)[14], ant colony optimization (ACO)[15], bee colony optimization (BCO)[16], simulated annealing (SA)[17], firefly algorithm (FA)[18], flower pollination algorithm (FPA)[19] etc. To optimize DOA estimation for real-world applications, we suggest applying CS algorithm. The CS algorithm, which is inspired by the way cuckoo birds lay their eggs in birds nests is a very good method for solving difficult problems. By using the Cuckoo Search algorithm in the process of estimating the direction of arrival we want to find the set of parameters and array configurations, which will make the direction finding in autonomous vehicles more precise.

Recent advances in intelligent transportation systems have significantly increased the demand for reliable sensing, localization, and wireless communication technologies to support connected and autonomous vehicles (CAVs). Vehicle-to-vehicle (V2V) communication and intelligent routing algorithms have improved cooperative information exchange, enabling more efficient and safer transportation networks [20]. More recently, collaborative perception frameworks have demonstrated that multi-vehicle sensor cooperation can effectively overcome occlusions and substantially enhance environmental awareness in autonomous driving scenarios [21]. Similarly, optimization techniques for automotive radar operating in spectrally congested vehicle-to-infrastructure (V2I) environments have highlighted the growing importance of robust signal processing algorithms capable of maintaining accurate target detection and direction estimation under severe interference conditions [22].

Recent developments in multi-sensor fusion, occupancy perception, and intelligent beamforming further demonstrate that high-resolution DOA estimation remains a fundamental enabling technology for next-generation autonomous driving systems [23]. Likewise, emerging research on vehicle-road-cloud collaborative architectures, vehicular edge computing, cooperative positioning, and multimodal sensor layout optimization emphasizes the necessity of accurate spatial information for reliable perception, localization, and resource management [24–26]. Furthermore, advances in intelligent control of connected autonomous vehicle platoons and reinforcement-learning-based transmission scheduling continue to increase the demand for robust array signal processing methods capable of operating under dynamic and interference-prone environments [27, 28].

Despite these significant advances, conventional DOA estimation algorithms remain limited by insufficient spatial degrees of freedom, degraded performance in low-SNR environments, and reduced estimation accuracy when multiple closely spaced sources are present. Although nested-array architectures effectively increase the virtual aperture and available spatial degrees of freedom, efficient optimization techniques are still required to exploit these additional virtual sensors for accurate DOA estimation. Motivated by these challenges, this work proposes a nested-array DOA estimation framework based on second-order statistical processing and the Cuckoo Search (CS) optimization algorithm, providing improved estimation accuracy, robustness, and computational efficiency for intelligent sensing applications in connected and autonomous vehicles.

In conclusion, the integration of the Cuckoo Search algorithm with second-order statistics in the co-array domain of nested arrays provides an effective framework for improving Direction of Arrival (DOA) estimation in autonomous vehicle applications. The Cuckoo Search algorithm is a part of this and it helps us find the best solution, for estimating the direction of arrival. The results obtained from this research hold the potential to enhance the safety and reliability of autonomous driving systems.

2 Mathematical modeling

2.1 Nested array configuration

Figure 1 illustrates the configuration of nested array. A linear nested array structure is formed by two ULAs along with x-axis. The first ULA is called dense ULA as shown in Figure 2, and the second is known as sparse ULA as depicted in Figure 3. Dense ULA has M antenna elements with a λ / 2 interelement separation, whereas sparse ULA has N antenna elements with a M d interelement difference. Nested array is developed by concatenating these two sub arrays, where d is λ / 2 and λ is the wavelength of the incident wave. All antenna elements are taken to be identical, omnidirectional, and isotropic. The position of each antenna element of the nested array in scalable form can be denoted by p as shown in Equation 1.

p = m d … M d , n d . N M + 1 d 1 ≤ m ≤ M w h e r e m = 1,2 , … , M n M + 1 w h e r e n = 1,2 , … , N a n d d = λ / 2 (1)

The configuration of the nested array is chosen as M = 2 (dense subarray) and N = 3 (sparse subarray), which yields a total of M + N = 5 physical sensors. This selection is optimal for maximizing the number of contiguous virtual lags in the difference co-array for a five-element array Pal and Vaidyanathan [29].

For a given total number of sensors N total, the optimal numbers of elements in the dense and sparse subarrays are determined according to Equation 2, which provides the configuration that maximizes the difference co-array length.

M = ⌊ N total / 2 ⌋ , N = N total − ⌊ N total / 2 ⌋ (2)

For the selected five-element nested array, the corresponding physical sensor locations are obtained from Equation 3, where the sensor positions are expressed in multiples of the inter-element spacing d.where d = λ / 2 is the unit inter-element spacing.

S = 1,2,3,6,9 d (3)

The difference co-array generated by the selected nested-array configuration contains a contiguous virtual region, which can be represented mathematically as given in Equation 4.where L = M N + M − 1 = 2 × 3 + 2 − 1 = 7.

D contiguous = − L , − L + 1 , … , − 1,0,1 , … , L − 1 , L (4)

The complete set of contiguous lag values is for the selected values M = 2 and N = 3, the complete set of contiguous difference-co-array lags is explicitly obtained as given in Equation 5:

D contiguous = − 7 , − 6 , − 5 , − 4 , − 3 , − 2 , − 1,0,1,2,3,4,5,6,7 (5)

This provides a contiguous virtual ULA of length 2 L + 1 = 15 elements. Accordingly, the maximum number of resolvable sources, expressed in terms of the effective degrees of freedom (DOF), is determined using Equation 6.

DOF max = L = 7 (6)

This research uses a different array configuration.

From Equation 1: Dense subarray: { d , 2 d } (for M = 2) - Sparse subarray: { 3 d , 6 d , 9 d } (for N = 3) - Total: { d , 2 d , 3 d , 6 d , 9 d }

For the array { 1,2,3,6,9 } d, the difference co-array unique lags are:

Positive lags: { 1,2,3,4,5,6,7,8 } (from 9 − 1 = 8,9 − 2 = 7,6 − 1 = 5,6 − 2 = 4,3 − 1 = 2,3 − 2 = 1,6 − 3 = 3,9 − 3 = 6)

Negative lags: { − 8 , − 7 , − 6 , − 5 , − 4 , − 3 , − 2 , − 1 }

Zero lag: { 0 }

All lags: { − 8 , − 7 , − 6 , − 5 , − 4 , − 3 , − 2 , − 1,0,1,2,3,4,5,6,7,8 }

This gives 17 unique lags, with a contiguous segment from − 7 to 7 (15 lags) or from − 8 to 8 (17 lags if fully contiguous).

If the contiguous segment is − 8 to 8, then L = 8 and DOF = 8.

For the paper, DOF = 10, the array need to have contiguous lags from − 9 to 9, which would require L = 10 for the positive side.

2.2 Signal model

Assume that K far-field sources provide uncorrelated narrowband signals impinging on the nested array from directions [ θ 1 , θ 2 , … , θ K ]. The received signal vector x ( t ) ∈ C ( M + N ) × 1 at the array output assuming K uncorrelated narrowband far-field sources impinge on the nested array, the received signal model can be expressed in matrix-vector form as given in Equation 7.where.

x t = A s t + n t (7)

• A = [ a ( θ 1 ) , a ( θ 2 ) , … , a ( θ K ) ] ∈ C ( M + N ) × K is the array manifold matrix
• s ( t ) = [ s 1 ( t ) , s 2 ( t ) , … , s K ( t ) ] T ∈ C K × 1 is the source signal vector
• n ( t ) ∈ C ( M + N ) × 1 is the additive white Gaussian noise vector

The corresponding array-manifold or steering vector for the k-th source is defined according to Equation 8, where the sensor positions and direction of arrival determine the phase progression across the array.where p i denotes the position of the i-th sensor in units of d = λ / 2, and the sensor positions are given in Equation 1.

a θ k = 1 , e j k p 2 d ⁡ cos θ k , e j k p 3 d ⁡ cos θ k , … , e j k p M + N d ⁡ cos θ k T (8)

Thus, the received signal vector associated with the K impinging sources can equivalently be represented by the compact signal model given in Equation 9.where A is array manifold matric which is expressed in Equation 10.

x t = A s t + n t (9)

A = a θ 1 , a θ 2 , a θ K (10)

a ( θ 1 ) of Equation 10 is expressed in Equation 11

a θ 1 = 1 , e j k p 2 M − 1 d ⁡ cos θ 1 , … , e j k p M + N d ⁡ cos θ 1 T (11)

a ( θ 2 ) of Equation 10 is presented in Equation 12

a θ 2 = 1 , e j k p 2 M − 1 d ⁡ cos θ 2 , … , e j k p M + N d ⁡ cos θ 2 T (12)

x t = A s t + n t (13)

Using the definitions of the array manifold matrix, source-signal vector, and noise vector introduced above, the received signal model can also be written in the form presented in Equation 13. s ( t ) described in Equation 10 is signal vector as shown in Equation 14

s t = s 1 t , s 2 t , . , s K t T (14)

n ( t ) in Equation 10 is independent and identically distributed complex additive white Gaussian noise and can be expressed in the vector form as stated in Equation 15

n t = n 1 t , n 2 t , … , n M + N t T (15)

2.3 Difference Co-array of nested array

As shown in Equation 10, noise and the received signal vector of an electromagnetic wave are considered uncorrelated [30]. In the light of this assumption, a covariance matrix is now built by using a nested array structure and expectation of the received signal as shown in Equation 16

R x x = E x t x H t (16)

By solving this equation, we will obtain

R x x = A R s s A H + σ 2 I (17)

For the selected two-level nested array,

M = 2 , N = 3 ,

The physical sensor locations are given by

S = 0 , d , 3 d , 6 d , 9 d .

The difference co-array is obtained from all pairwise sensor differences as

D = s i − s j | s i , s j ∈ S .

Evaluating all possible pairwise differences yields

D = − 9 , − 8 , − 6 , − 5 , − 3 , − 2 , − 1 , 0 , 1 , 2 , 3 , 5 , 6 , 8 , 9 d .

The contiguous portion of the virtual difference co-array iswhich forms a virtual ULA consisting of

D c = − 5 , − 4 , − 3 , − 2 , − 1 , 0 , 1 , 2 , 3 , 4 , 5 d ,

11

Consecutive virtual sensor locations. Therefore,

D c = 11 .

The contiguous virtual ULA extends from − 5 d to + 5 d, providing eleven consecutive virtual array positions. In this work, the effective DOF is defined as the maximum number of independent source directions that can be resolved using the covariance-based processing framework. Consequently,

D O F = 11 − 1 = 10 .

Thus, the adopted nested-array configuration provides ten effective degrees of freedom, enabling the estimation of up to ten uncorrelated signal sources.

In according with Equation 17, R s s expresses the amplitude covariance matrix of the received signal as shown in Equation 18.

R s s = E s t s H t (18)

The diagonal auto-correlation matrix of s ( t ), can be represented as follows, as seen in Equation 19.

R s s = σ 1 2 σ 2 2 … σ K 2 (19)

Now, a virtual uniform linear array can be created by vectorizing the covariance matrix ( R x x ), as shown in the following Equation 20 where. The phase-dependent term associated with the sensor-pair separation and the direction of arrival can be expressed as shown in Equation 21, where k ₀ denotes the wavenumber.

z = v e c R x x = v e c A R s s A H + σ 2 I = A * ⊙ A r s s + σ 2 1 (20)

• A is the array manifold matrix
• R s s = E [ s ( t ) s H ( t ) ] is the source covariance matrix
• r s s = diag ( R s s ) = [ P 1 , P 2 , … , P K ] T contains the source powers
• ⊙ denotes the Khatri-Rao product
• 1 = vec ( I ) is the vectorized identity matrix
• σ 2 is the noise power

e j k 0 p i − p j d ⁡ cos θ k . (21)

where.

• k 0 = 2 π / λ is the wavenumber
• p i and p j are the sensor positions in units of d
• d = λ / 2 is the inter-element spacing
• θ k is the DOA of the k-th source

Columns of the correlation matrix are sequentially put under the first column as part of the vectorization process. The correlation matrix is transformed into a column vector with dimensions of z ( M + N ) 2 × 1 after vectorization. This column vector z is sorted by the received signal’s exponent term. After sorting, the redundant terms in this organised vector are removed, called to as a continuous uniform linear array. After sorting the vectorized covariance terms and eliminating redundant entries, the locations corresponding to the continuous portion of the virtual array are determined according to Equation 22.

I c = M n d − N m d ∪ N m d − M n d , w h e r e 0 < m < M − 1 a n d 0 < n < N − 1 (22)

The location of these virtual ULA components are represented as in Equation 23

I v = q i − q j i , j = 1,2 , … , M + N − 1 (23)

We concentrate on applying our suggested techniques to the continuous portion of a nested array’s vULA. The continuous vULA segment gives us more DOF, and this DOF is directly proportional to the length of the continuous vULA.

2.4 Fitness function

Fitness function is a key factor in optimization problems. This is a crucial tool for population assessment. Each iteration of the algorithm includes this evaluation. The fitness function is defined as the difference between the actual x a ( t ) and estimated x e ( t ) vector.To evaluate the quality of each candidate DOA solution during the optimization process, the noise-subspace-based fitness function is defined as given in Equation 24. where θ ̂ = [ θ ̂ 1 , θ ̂ 2 , … , θ ̂ K ] is a candidate set of DOA angles, a ( θ ̂ k ) is the steering vector for the k-th candidate source, and U n is the noise subspace obtained from the eigenvalue decomposition of the sample covariance matrix R ̂ x x.

f θ ̂ = ∑ k = 1 K a H θ ̂ k U n U n H a θ ̂ k (24)

Both methods evaluate the nests and particles by using this fitness function in order to optimise the vehicles.

2.5 Comparison with existing nested-array DOA estimation methods

Recent nested-array DOA estimation methods primarily rely on covariance matrix reconstruction techniques such as Toeplitz Matrix Reconstruction (TMR), spatial smoothing, or sparse recovery methods based on Compressive Sensing (CS). Toeplitz reconstruction restores the covariance matrix rank to improve estimation of coherent sources; however, it introduces additional matrix reconstruction operations and eigenvalue decomposition, increasing computational cost. Compressive sensing methods exploit signal sparsity to estimate DOAs with high resolution but require discretization of the angular space, making their performance sensitive to grid mismatch while also increasing optimization complexity.

In contrast, the proposed framework directly exploits the second-order statistics of the nested-array co-array together with the Cuckoo Search optimization algorithm. This combination avoids explicit covariance reconstruction and sparse recovery while efficiently exploring the search space through Lévy-flight-based global optimization. Consequently, the proposed method provides improved estimation accuracy under low-SNR conditions with comparatively lower computational burden and enhanced convergence characteristics.

3 Proposed methodology

3.1 Cuckoo Search algorithm

In this work, meta-heuristic approach CS algorithm is used to optimise the incoming signal impinging on the nested arrays. The idea for CS came from some cuckoo species that lay their eggs in the nests of other bird species. In this method, there are two bird species: cuckoo birds and host birds from other species [31]. There are three basic guidelines for CS implementation.

Each cuckoo lays only one egg at a time, which is then placed in a nest chosen at random. The best nest with the best eggs will be passed to the next-generation. Host birds detect cuckoo eggs with probability P a and the number of host nests is constant. Host bird either abandons the nest or throws the egg aside. For a better understanding of the preceding three phases, flow diagram is illustrated in Figure 4 and Algorithm 1 showing pseudo code for CS is provided below.

Algorithm 1 Pseudo Code for Cuckoo Search Algorithm.

3.2 Cuckoo Search hyper-parameters

The CS algorithm employed in this study utilizes the following hyper-parameters:

The hyper-parameters are selected based on the following rationale.

• Number of Nests ( n = 25 ): This value is recommended in the original CS formulation Yang and Deb [32] as a standard choice for problems with moderate dimensionality. Preliminary experiments showed that increasing n beyond 25 yields diminishing returns in accuracy while significantly increasing computational cost.
• Maximum Iterations ( max itr = 500 ): Based on convergence analysis, the RMSE stabilizes after approximately 300 iterations for all scenarios. Setting max itr = 500 ensures convergence while avoiding unnecessary computational overhead.
• Alien Egg Discovery Probability ( P a = 0.25 ): This is the standard value recommended in the CS literature Yang and Deb [32]. It provides an optimal balance between exploration (P a small, favoring global search) and exploitation (P a large, favoring local search).
• Lévy Flight Exponent ( β = 1.5 ): This value ensures that the Lévy flights exhibit super-diffusive behavior, enabling efficient exploration of the search space Yang and Deb [32]. Values in the range 1 < β ≤ 2 are standard for CS implementations.

3.3 Optimization search space

Each candidate solution (nest) in the Cuckoo Search algorithm represents a complete set of estimated direction-of-arrival (DOA) angles. For a scenario containing K signal sources, the i-th nest is represented aswhere

θ i = θ i 1 , θ i 2 , … , θ i K T ,

θ i k ∈ θ min , θ max , k = 1,2 , … , K .

Consequently, the optimization is performed in a continuous K-dimensional parameter space, and the search dimension increases linearly with the number of signal sources.

For each candidate solution, the corresponding array manifold matrixis constructed to generate the theoretical covariance matrix

A θ i = a θ i 1 , a θ i 2 , … , a θ i K

R ̂ x x θ i .

The fitness value is then evaluated using the covariance fitting criterionwhere the optimal DOA estimate corresponds to the candidate solution producing the minimum reconstruction error.

f θ i = R x x − R ̂ x x θ i F 2 ,

3.4 Particle Swarm Algorithm

PSO is the population-based stochastic search method. This method is inspired by the social behavior of flocking birds. The optimal search approach utilized in PSO to find food is to follow the bird that is closest to the food [33]. In PSO, a swarm of random particles initializes the algorithm, where population of a candidate solution is called particle. The mobility of each particle is influenced by its local best known position which direct the swarm to the optimal solution. For convenience, refer to Figure 5, which contains the PSO flow chart, while Algorithm 2 shows the pseudo code for Particle Swarm Optimization.

Algorithm 2 Pseudo Code for Particle Swarm Algorithm.

4 Results and discussions

In this section, we conduct a series of experimental simulations aimed at showcasing the efficiency of the proposed method for DOA estimation, particularly in terms of DOF. These experiments consist of 500 ′Monte Carlo’ simulation runs conducted under specific conditions. For all simulations, the carrier frequency is set to f c = 3 GHz, corresponding to a wavelength of λ = c / f c = 0.1 m, where c = 3 × 1 0 8 m/s is the speed of light. Consequently, the half-wavelength inter-element spacing is d = λ / 2 = 0.05 m (5 cm). This frequency is chosen to represent typical automotive radar and 5G communication bands, ensuring practical relevance for autonomous vehicle applications. To begin, we compare the results with a PSO algorithm to evaluate the accuracy of our proposed algorithm. Subsequently, we assess performance by analyzing the RMSE across various SNRs and numbers of snapshots. RMSE serves as a metric for evaluating the performance of both CS and PSO as we vary the number of signal sources.

Additionally, we employ statistical techniques such as histogram analysis, variation analysis, and Cumulative Distribution Function (CDF) analysis to validate the proposed scheme’s performance. To ensure the validity of our results and to explore the limits of DOF, we divide the experimental analyzes into three segments, each involving the estimation of six sources 2° apart, eight sources 3° apart, and ten sources 4° apart, respectively. Consequently, the maximum achievable DOF in this study is ten.

Our array configuration consists of five sensors, with sub-array M comprising two antenna elements and sub-array N consisting of three elements. This configuration is designed to mitigate mutual coupling effects. We assume that the noise is additive white Gaussian, both spatially and temporally. Furthermore, we make the simplifying assumption that all target sources are in the far field of the nested array.

4.1 Estimation accuracy

4.1.1 Estimation accuracy of six sources spaced two degrees apart

By using signal sources that are placed in various positions in accordance with the DOAs, the estimation accuracy of CS and PSO is assessed. The algorithms with more significant meanings, these are analysed using the best, mean and worst estimation. The performance of both algorithms is excellent for the estimation of six targets at 5dB, 0 dB and -5dB SNR, however the results of PSO suffer at -5 dB. Table 1 shows the Comparison of the proposed method with existing nested-array DOA estimation techniques. It is very clear that the CS dominates the PSO algorithm as demonstrated in Tables 2–4.

TABLE 1. Comparison of the proposed method with Existing nested-array DOA estimation techniques.

TABLE 2. Cuckoo search algorithm hyper-parameters.

TABLE 3. Estimation accuracy at SNR = 5 dB for six sources.

TABLE 4. Estimation accuracy at SNR = 0 dB for six sources.

4.1.2 Estimation accuracy of eight sources spaced three degrees apart

The efficiency of CS algorithms remains remarkable when estimating eight targets under varying Signal-to-Noise Ratios (SNRs) of 5dB, 0dB, and -5 dB. However, the performance of Particle Swarm Optimization (PSO) deteriorates significantly with rising noise levels. As seen in Tables 5–7, the CS clearly outperforms the PSO algorithm.

TABLE 5. Estimation accuracy at SNR = -5 dB for six sources.

TABLE 6. Estimation accuracy at SNR = 5 dB for eight sources.

TABLE 7. Estimation accuracy at SNR = 0 dB for eight sources.

4.1.3 Estimation accuracy of ten sources spaced four degrees apart

The performance of CS algorithms is outstanding for estimating ten targets using SNRs of 5dB, 0dB, and -5dB, but PSO results decline rapidly as noise increases. As seen in Tables 8–11, the CS clearly outperforms the PSO algorithm.

TABLE 8. Estimation accuracy at SNR = -5 dB for eight sources.

TABLE 9. Estimation accuracy at SNR = 5 dB for ten sources.

TABLE 10. Estimation accuracy at SNR = 0 dB for ten sources.

TABLE 11. Estimation accuracy at SNR = -5 dB for ten sources.

4.2 Robustness against noise

In array signal processing, robustness against noise is regarded as a potent instrument for assessing the effectiveness of algorithms for estimating direction of arrival. In this investigation, we gradually raised the value of SNR while the value of RMSE is tracked. This study demonstrates the algorithm’s intelligence by systematically elevating the Signal-to-Noise Ratio (SNR) and subsequently observing the corresponding Root Mean Square Error (RMSE) values.

4.2.1 Robustness against noise for six sources

In this case, Value of SNR is increased with the difference of 10 dB from −30 dB to 30dB, value of RMSE decreased gradually that shows in Figure 6. When SNR approaches 0dB, Performance of CS is much superior to PSO, with an RMSE of 1 0 − 1 and PSO being 1 0 1.

4.2.2 Robustness against noise for eight sources

In this scenario, the Signal-to-Noise Ratio (SNR) was incremented in 10 dB steps, ranging from −30dB to 30 dB. As illustrated in Figure 7, the Root Mean Square Error (RMSE) steadily decreased. Notably, when SNR approached 0dB, Cuckoo Search (CS) outperformed Particle Swarm Optimization (PSO) significantly, achieving an approx RMSE of 1 0 − 1 compared to PSO’s approx 1 0 1.

4.2.3 Robustness against noise for ten sources

In this scenario, the Signal-to-Noise Ratio (SNR) is incrementally increased by 10dB, ranging from −30dB to 30 dB. The corresponding Root Mean Square Error (RMSE) exhibits a gradual reduction, as illustrated in Figure 8. SNR is noted for showing RMSE results that are approaching those of PSO, especially as SNR approaches 0 dB. More specifically, RMSE of CS is 1 0 0, while PSO is greater than 1 0 1.

4.3 Cumulative Distribution Function (CDF) analysis for RMSE

It is a non-decreasing step function that is right continuous and bounded between 0 and 1. The current study intends to test the advantages and the drawbacks of optimization methods based solely on the running of ”Monte Carlo” simulations. Within CDF, RMSE is represented in a descending order to show the errors which remained over the simulations. The current research uses CDF to show the errors of three different optimization methods.

4.3.1 CDF analysis for six sources

As shown in Figure 9, the 0 dB CDF of CS is shown on the y-axis. The minimum RMSE occurs at CDF = 0, which is equal to 0.0001257, and is shown in the figure. The maximum value of RMSE is 0.1279, which is equal to CDF of 1. PSO shows that the minimum RMSE is equal to 0.0007611, CDF = 0, and the maximum RMSE, which is equal to a CDF of 1, is equal to 1538.39. This study shows that CS is better than PSO. Similar results are noted for -5 dB and −10 dB which suggest that CS is preferred over PSO.

4.3.2 CDF analysis for eight sources

In accordance with Figure 10, the spread for the CDF for eight sources revealed the same results as for six sources, reaffirming the consistent superiority of Compressive Sensing over PSO across these scenarios, reinforcing the conclusion that CS consistently outperforms PSO in these scenarios.

4.3.3 CDF analysis for ten sources

Figure 11 depicts the CDF for ten sources whose spread confirms the consistent superiority of CS over PSO for 10 sources. This reaffirms the conclusion that CS consistently outperforms PSO in these scenarios.

4.4 RMSE analysis of robustness against snapshots

In this experiment, we assess the RMSE values across multiple snapshots. This assessment is also referred to as evaluating robustness against snapshots. In this analysis, the parameter K is set to 500, and we explore the range of K values from 1 to 500 snapshots.

4.4.1 Robustness against snapshots for six sources

For the estimation of six sources, the robustness against snapshots was evaluated as shown in Figure 12, RMSE of CS decreases abruptly as compared to PSO when snapshots increased from 1 to 100. Between 100 and 500 snapshots, RMSE of both optimization algorithms gradually declines, however, performance of CS outperforms than PSO.

4.4.2 Robustness against snapshots for eight sources

In assessing eight sources, we examined the resilience against varying snapshots, as illustrated in Figure 13. The RMSE of CS significantly dropped in comparison to Particle Swarm Optimization as the number of snapshots increased from 1 to 100. Between 100 and 500 snapshots, the RMSE of both optimization algorithms gradually decreased. Notably, during this range, CS consistently outperformed PSO in terms of performance.

4.4.3 Robustness against snapshots for ten sources

The CS algorithm also outperforms PSO in the case of 10 targets estimation, as shown in Figure 14. At 100 snapshots the value of RMSE for CS is rapidly falling down while RMSE value of PSO is very high. After enhancing snapshots from 100 to 300 RMSE for both methodologies decline very slowly and when snapshots increasing from 300 to 500, value of RMSE deline gradually of CS again. But RMSE of PSO saturate at this point. So results of CS are more reliable than PSO in this case.

4.5 Box plot analysis

A box plot serves as a conventional method for visualizing data distribution, and it is commonly referred to as a variation analysis tool. In this study, we require five key parameters: Q 1, Q 2, Q 3, the minimum value, and the maximum value of the dataset. Outliers are instrumental in determining the minimum and maximum data values, while Q 1, Q 2, and Q 3 are computed using quartile calculations. These parameters provide a straightforward means to assess the performance of estimation methods.

4.5.1 Box plot analysis for six sources

In the case of six targets estimation, box plot analysis of CS and PSO is depicted in Figure 15. Outlier of CS algorithm is 0.0481 whereas min and max values of RMSE are 0.0001 and 0.1279, respectively at 5 dB SNR. At 0dB, min value is 0.0001 and max value is 0.2444 whereas the outlier is 0.6621. Likewise, outlier of CS at −10 dB is 1.7905 and min and max values are 0.0032 and 2.1390 repectively. In PSO algorithm, min value of box plot is 0.0008 and max value is 1538.38 and outlier is 0.3337 at 10 dB. At 0 dB estimation of PSO, min value is 0.0005 and the max value is 1303.50 whereas the outlier for this case is 58.2237. Similarly, min and max values for −10 dB are 0.0063 and 1301.90 whereas the value of outlier for this case in 497.507. This data demonstrates that CS outperformed PSO in estimating DOA.

4.5.2 Box plot analysis for eight sources

In the case of eight targets estimation, box plot analysis of CS and PSO is depicted in Figure 16. Outlier of CS algorithm is 0.0481 whereas min and max values of RMSE are 0.0001 and 0.1279, respectively at 5 dB SNR. At 0dB, min value is 0.0001 and max value is 0.2444 whereas the outlier is 0.6621. Likewise, outlier of CS at −10 dB is 1.7905 and min and max values are 0.0032 and 2.1390 repectively. In PSO algorithm, min value of box plot is 0.0008 and max value is 1538.38 and outlier is 0.3337 at 10 dB. At 0 dB estimation of PSO, min value is 0.0005 and the max value is 1303.50 whereas the outlier for this case is 58.2237. Similarly, min and max values for −10 dB are 0.0063 and 1301.90 whereas the value of outlier for this case in 497.507. This data demonstrates that CS outperformed PSO in estimating DOA.

4.5.3 Box plot analysis for ten sources

Box plot analysis of CS and PSO for the assessment of ten targets is shown in Figure 17. The CS algorithm’s outlier is 10.8559, while the minimum and maximum RMSE values at 5 dB SNR are 0.0197 and 72.32, accordingly. The minimum value at 0 dB is 0.0220, the maximum value is 124.8381, and the outlier is 12.3201. Similarly, the outlier of the CS at -5 dB is 13.9507, and the minimum and maximum values are 0.0710 and 460.3149 respectively. However, for same SNR values, performance of PSO is not reliable. All of these statistics show that CS is far better than PSO in estimating DOA.

4.6 Histogram analysis

Histogram analysis visually depicts the extent of errors occurring across different ”Monte Carlo’’ simulation runs. Through this analysis, we assess the effectiveness of optimization strategies. The histogram analysis divides data into continuous groups, with each of these groups represented by a bar known as a bin.

4.6.1 Histogram analysis for six, eight and ten sources

In CS algorithm histogram analysis at 5dB, the edges of first bin range from 0 to 0.0129, its centre value is 0.00652, this bin contains 606 results. The second bin has 202 results, and its RMSE edges range from 0.0129 to 0.0257. Similarly, remaining results can be found in the last eight bins. At 0dB, the first bin contains 695 results having central value of error is 0.0124. Second bin contains 173 counts with the central value of 0.0368 beyonds the edges of 0.0246 and 0.049. Similar to that, the final nine bins contain the remaining findings. At -5dB, the first bin has 554 counts, with edges ranging from 0 to 0.217 and a central value is 0.11. Second bin has 259 counts with a centre value of RMSE is 0.324 and edges ranging from 0.217 to 0.43. The final five bins include the remaining findings. In PSO at 10dB, first bin contains 806 results of RMSE with 0–154 bin edges. At 0dB, 754 bin counts and with 0–130 bin edges and at −10 dB bin counts are 719. Remaining results are resides in the following bars of each level of SNR ratios. In light of these statistics shown in Figures 18, 19, performance of CS algorithm is more efficient as compared to PSO.

In this histogram analysis for ten different sources, the first bin has 914 outcomes at 5 dB of the CS algorithm, with the first bin’s edges ranging from 0 to 21.6 and its centre value being 10.8. Similarly, remaining few results can be found in the remaining bins. The first bin at 0 dB has 930 results with a central error value of 11.3 the edge values for this bin range from 0.0246 to 0.049. The following discoveries are located in the remaining bins in a similar manner. The first bin at −5 dB has 906 counts, edges from 0 to 22.1, and a centre value is 11.1 The other findings are in the remaining bins. First bin of PSO at 5 dB comprises 638 RMSE results with 0–119 bin edges. The second bin contains 268 counts, with edges ranging from 119 to 238 and a centre value of RMSE of 178. At 0 dB, there are 523 bin counts with 0–95.8 bin edges, a centre value of 47.9, 333 results in the second bin, and a range of 95.8–192 RMSE edges with 144 bin center. 515 bin counts from 0 to 95.3 bin edges and 47.7 bin center at −5 dB. The second bin has 297 counts with 95.3–190 bin edges and 143 bin center. Third bin contains 94 counts with 190–286 bin edges, and the remaining findings are located in the subsequent bars. These figures show that the CS algorithm performs better than PSO in terms of estimation accuracy as depicted in Figure 20.

4.7 Computational time analysis

To assess the real-world applicability of the proposed method, an in-depth analysis of computational performance was done. The average running time per DOA estimation trial was compared for the proposed CS algorithm and the benchmark PSO algorithm.

The average running time for estimating 6, 8, and 10 sources is illustrated in Table 12. The proposed CS algorithm, on average, is 1.92× faster than the PSO algorithm for all the considered cases. For the most difficult case of estimating 10 sources, CS is able to complete the task in 1.21 s, while PSO takes 2.34 s, an improvement of 48.3% in the processing time.

TABLE 12. Average running time comparison for DOA estimation.

Bold values highlight the results obtained using the proposed Cuckoo Search (CS)-based DOA estimation method and are used to distinguish the proposed approach from the comparative methods.

The improved computational efficiency of CS can be attributed to.

• Faster Convergence: CS requires only 138 iterations on average for 10-source estimation, compared to 256 iterations for PSO (Table 13).
• Levy Flight Mechanism: The global search capability of Levy flights enables CS to explore the solution space more efficiently than PSO’s velocity-based updates.
• Elitist Selection: CS preserves the best solutions directly, reducing redundant evaluations.

TABLE 13. Computational performance comparison of CS and PSO.

Bold values highlight the results obtained using the proposed Cuckoo Search (CS)-based DOA estimation method and are used to distinguish the proposed approach from the comparative methods.

The results presented above prove that the proposed CS-based DOA estimation method is appropriate for real-time systems that are part of the perception capabilities of an autonomous vehicle. With its combination of accuracy and efficiency, the proposed method is an interesting alternative for practical use.

4.7.1 Comparison with other metaheuristic algorithms

To further validate the computational efficiency of CS, we compare its convergence characteristics with other metaheuristic algorithms. Table 14 shows the number of iterations required to achieve an RMSE of 1 0 − 2 for 10-source estimation at 0 dB SNR.

TABLE 14. Convergence comparison of metaheuristic algorithms.

Bold values highlight the results obtained using the proposed Cuckoo Search (CS)-based DOA estimation method and are used to distinguish the proposed approach from the comparative methods.

4.8 Computational timing and real-time feasibility

To evaluate the practical applicability of the proposed CS-based DOA estimation method for autonomous vehicle systems, we conducted comprehensive timing experiments on the hardware platform specified in Table 15. All results are averaged over 500 Monte Carlo runs.

TABLE 15. Hardware specifications for timing experiments.

4.8.1 Average wall-clock time

Table 16 presents the average wall-clock time for one complete DOA estimation trial using the proposed CS algorithm, PSO, and Spatial Smoothing MUSIC (SS-MUSIC) for 6, 8, and 10 sources.

TABLE 16. Average wall-clock time per DOA estimation trial.

Bold values highlight the results obtained using the proposed Cuckoo Search (CS)-based DOA estimation method and are used to distinguish the proposed approach from the comparative methods.

The CPU-based CS implementation requires 1213.6 m for 10-source estimation, which is significantly faster than PSO (2341.8 m) but slower than SS-MUSIC (41.7 m). The GPU-accelerated CS implementation achieves 143.2 m for 10-source estimation, representing an 8.5× speedup over the CPU implementation.

4.8.2 Real-time requirements

The stringent timing requirements for automotive radar systems are well documented in the literature. Wu et al. [34] et al. note that ”the target detection process, which spans from the reception of the reflected signal to the generation of detection results, should occur within 25–40 m”. More recent work targets approximately 1 m processing time for super-resolution DOA estimation in automotive radar applications.

Based on these requirements.

• CPU-based CS: At 1213.6 m, it exceeds the 100 m threshold by an order of magnitude and is unsuitable for real-time deployment.
• GPU-accelerated CS: At 143.2 m, it approaches but still exceeds the 40 m requirement. With further optimization, it could potentially meet real-time constraints.
• SS-MUSIC: At 41.7 m, it meets the 40 m requirement but offers limited DOF.
• PSO: At 2341.8 m, it is entirely unsuitable for real-time applications.

5 Conclusion and future recommendations

This paper presented a CS based DOA estimation framework for nested antenna arrays, combining second-order statistical processing with metaheuristic optimization to achieve high-resolution source localization. The specific use of a nested-array design greatly enhances the virtual aperture and spatial ambiguity, allowing the estimation of ten completely uncorrelated signal sources with the use of only five physical antenna elements. Several simulations showed that the proposed method outperformed the PSO-based method in estimation accuracy, especially in situations of low signal-to-noise ratio (SNR) and in the case of closely spaced signal sources. The proposed method also demonstrated good convergence and computational efficiency with satisfactory performance for varying numbers of signal sources. This shows that an efficient nested-array processing with CS optimization can be used for accurate high-resolution DOA estimation.

The proposed framework is well suited for emerging intelligent sensing applications, including automotive radar, connected autonomous vehicles, wireless communications, and smart sensing systems, where accurate direction estimation is essential. Although the current study primarily considers uncorrelated sources in additive white Gaussian noise environments, future work will investigate coherent sources, colored and non-Gaussian noise, adaptive hyperparameter optimization, and real-time hardware implementation on embedded and automotive sensing platforms to further improve the practicality and robustness of the proposed method.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Soumyajyoti Biswas, SRM University AP, India

Reviewed by: Njitacke Tabekoueng Zeric, University of Buea, Cameroon

Bo Yang, Shanxi University, China