Enhancement of multi-objective Darwinian particle swarm optimization for neural-network-based multimodal medical image fusion

1 Introduction

Multimodal medical image fusion has greatly assisted physicians in obtaining an accurate and dependable patient diagnosis, in image-guided surgery, and in radiotherapy (Vigelis et al., 2020). The main goal of most multimodal image fusion methods is to increase the clinical preciseness of medical imaging. Various imaging techniques are applied to obtain interpretable details of health problems, for example, computed tomography (CT) is used to obtain images of hard structures and bones, while magnetic resonance imaging (MRI) captures soft tissue (Liu et al., 2019). The fusion of MRI and CT when joined with surgical navigation assists surgeons in putting together, more accurately, a preoperative plan and possibly gives good clinical benefits in surgery (Ghorai et al., 2021). However, obtaining fused images of high-quality and accurately preserved image information is still a problem being addressed by ongoing research (Zhu et al., 2021). Preserving the essential features of images as well as reducing uncertainty and redundancy are important to ensure the fused images obtained will be useful to physicians for effective monitoring and all kinds of diagnoses (Li et al., 2021).

Tsallis cross-entropy, which was introduced on the foundation of Shannon entropy and cross-entropy, not only overcomes the limitations of Shannon entropy but also takes into consideration the information of source and fused images (Vigelis et al., 2020). Therefore, the implementation of Tsallis cross-entropy for pulse-coupled neural networks (PCNNs) will increase the quality of the fused images (Vigelis et al., 2020). Tsallis cross-entropy stimulated PCNN-based multimodal image fusion is a multi-objective optimization problem because it is expedient to optimize the entropies to obtain the optimal threshold which will then serve as external stimuli to the PCNN, and also to carry out the entire fusion in the shortest possible time. Of the multi-objective optimization algorithms currently used by researchers in research works [e.g., the Pareto archive evolutionary strategy (PASE; Liu et al., 2019), the strength Pareto evolutionary algorithm (SPEA 2; Ghorai et al., 2021), the non-dominated sorting genetic algorithm II (NSGA-II; Zhu et al., 2021), non-dominated sorting particle swarm optimization (Li et al., 2021), and multi-objective particle swarm optimization (MOPSO; Han et al., 2021)], MOPSO has the highest optimization capacity and rate of convergence (Lin et al., 2019). However, because image fusion processes are commonly known to have the shortcoming of premature convergence (Pires et al., 2010), multi-objective Darwinian particle swarm optimization (MODPSO) will be applied in this work because of the ability of the Darwinian particle swarm optimization (DPSO) algorithm to escape from local optima (Ahilan et al., 2019). The main concept of MODPSO is to process multiple concurrent particle swarm optimization (PSO) algorithms, each of which indicates a swarm. The effectiveness of the MODPSO algorithm is improved when compared with PSO in ensuring particles are not stuck in local optima. This helps to optimize the Tsallis cross-entropy effectively so that the optimal parameters of the neural network are selected for the most favorable output of the neural network for the fusion of the coefficients of the decomposed source images.

The main contributions of this paper can be summarized as follows:

• A novel multimodal medical image fusion algorithm is developed that incorporates a new multi-objective optimization algorithm (FOMODPSO) for the optimization of Tsallis cross-entropy to stimulate a PCNN externally, leading to a zero-learning/pretrained neural network.
• A higher firing count to driving the selection of the correct pixel to be sent to the fused image is generated, thereby ensuring complete extraction of essential details of fused images.
• The proposed method includes consideration of low computational needs and high speed of processing.
• Extensive experiments on 14 groups of multimodal images demonstrate the high-performance potential of the proposed multimodal medical image fusion algorithm in the recognition of the optimal neural network parameters for feature extraction, as regards medical image fusion.

The core motivation for this work is the long-standing challenge of achieving computational efficiency in high-quality PCNN-based multimodal medical image fusion (MMIF), which is essential for clinical practice such as image-guided surgery. Previous optimization algorithms used for PCNN parameter selection suffer from low convergence rates, directly resulting in slow processing times. Our work addresses this by proposing the novel fractional-order multi-objective Darwinian particle swarm optimization (FOMODPSO) to dramatically increase algorithmic convergence speed and exploration capability, thereby achieving the necessary ultra-low processing time (0.545 s; see Section 5.4, Table 1) required for real-time clinical application.

Table 1. Various approaches and their fusion processing time (in s).

The figure in bold indicates the average time (the shortest amongst all) consumption using our proposed method.

The rest of this paper is organized as follows. The preliminaries of the Tsallis cross-entropy, non-subsampled contourlet transform (NSST), FOMODPSO, and PCNN are introduced in Section 2. Section 3 describes the proposed FOMODPSO algorithm and its application in the optimization of the Tsallis cross-entropy for input to the PCNN for application to MMIF. Section 4 compares the proposed image fusion algorithm and other multimodal image fusion algorithms experimentally through a series of test instances. Finally, the conclusions are discussed in Section 5 along with ideas for further improvements.

2 Related work

Recently, the application of AI-based tools for improving diagnostic efficacy and healthcare assistance has become an active area of research. Over the years, different medical image fusion methods have been proposed (Raha et al., 2020; Tang et al., 2019; Kumar et al., 2021). The application of deep learning to image processing has made great progress. Deep learning was introduced to medical image fusion while attempting to tackle the design challenge of activity level measurement and fusion rule design found in traditional methods (Zhang et al., 2021). In this section, we review some recent works on deep learning and neural network-based medical imaging.

In Challa et al. (2019), a deep complete-CNN architecture for the exposure of diabetic retinopathy and its five levels was developed; however, the speed of the medical image processing was not considered in the work. Medical image translation using a fully conditioned bounded deep network was carried out in Rai et al. (2023). Bayesian deep learning was applied (Deshpande and Bhatt, 2019) for the registration of noisy medical images by non-linear geometric irregularities. Medical image denoising was carried out in Rai et al. (2021) using a multilayer deep residue network joined with sectionalized dictionaries. In Rai et al. (2022) a deep cascade restructuring of low-grade reduced resolution CT images of the chest to resuscitated and well-resolved CT chest images was developed as an accessible, less expensive, and safe recourse for monitoring lung wellness in COVID-19. The speed of these AI-based diagnostic processes was, however, not taken into consideration, or improved on.

Many works on PCNN-based MMIF have been presented in the recent past. Tang et al. (2019) applied the multi-swarm fruit fly optimization algorithm (MFOA) in the optimization of PCNN for MMIF. Enhanced fusion effect results were obtained; however, the convergence rate of the MFOA is quite low. Kumar et al. (2021) presented a hybrid MMIF method that applies PSO in the optimization of PCNN for the fusion of CT and MRI images. The results obtained gave fused images of reduced noise; however, many iterations in the fusion algorithm caused results to be low-performing in some evaluation metrics. Raha et al. (2020) proposed a multimodal image fusion method that applies the whale optimization algorithm (WOA) in the optimization of PCNN to give fused image results of image details; however, the convergence rate of the optimization algorithm is low. Das et al. (2022) applied multi-objective differential evolution (DE) in the optimization of dual-channel PCNN for multimodal image sensor fusion. Fused image results of preserved structural and textural details were obtained; however, many iterations caused the fused image results to be low-performing in some evaluation metrics. Nie et al. (2021) used PSO and a total vibration-based derivative optimization to optimize PCNN for MMIF based on multisource information exchange encoding. The results obtained showed the good visual quality of fused images, but the convergence rate of optimization algorithms needs to be improved. Wang et al. (2016) optimized PCNN with MOPSO for multimodal image fusion. The results obtained showed good-quality fused images, but the speed of the fusion process is low because of several iterations in optimization algorithms. Liu R. et al. (2022) presented a multimodal image fusion method that applies PSO and standard DE in the optimization of PCNN parameters to give fused image results of improved fusion effect. The many iterations of the optimization algorithms caused some evaluation metrics of the fused image results to not perform so well. Xu et al. (2016) used the quantum-behaved PSO (QPSO) to optimize the adaptive PCNN for multimodal image fusion. The results obtained showed higher-quality fused images; however, the convergence rate of the optimization algorithm needs to be improved for better fusion results. In all these works, proper optimization of the parameters of the PCNN with an efficient optimization algorithm would go a long way toward improving the performance of PCNN-based multimodal image fusion methods.

MMIF can be noted as a branch of incomplete multi-view clustering (IMC). IMC is the fusion of varying views of specific data of which none of the views is complete. MMIF however, focuses on medical data that are assumed to have complete multiple views. The multimodal data being used in this case may or may not have consistent and complementary information. External factors such as patients' absence and the low quality of the medical dataset (Wen et al., 2022) could affect the consistency and completeness of the source data. The most significant impact on the fusion effect for fusion tasks can be found in the aspects of the extent of completeness and balance of the source data (Liu et al., 2023; Liu C. et al., 2022), and also the effectiveness of the fusion rules with which the source data are being fused. If the multimodal data are incomplete with only one available modality, we can generate a fusion result from only one modality, provided an effective information recovery method is applied to recover the necessary missing information completely and uniformly (Liu et al., 2023).

3 Preliminaries as foundation of the proposed method

3.1 Notation

To aid with understanding of the notation, Table 2 contains a list of notation and the respective definitions.

Table 2. Table listing notation and its meaning.

3.2 Tsallis cross-entropy

Entropy is generally associated with the ratio of instability found within a system. According to Shannon Entropy (Lin et al., 2020), entropy is the ratio of chaos and relates to the information weight of the system. Shannon stated that when a physical system is taken apart as two free subsystems A and B, then the entropy value can be represented as

S(A+B)=S(A)+S(B).(1)

With consideration to Shannon's theory, a limited entropy concept was introduced by Vigelis et al. (2020) and expressed as

Sq=1-∑i=1T(pi)qq-1,(2)

where T represents the system potentials and q represents the entropic index (Vigelis et al., 2020). Equation 2 will be equivalent to Shannon entropy when q → 1.

Tsallis cross-entropy (Vigelis et al., 2020), however, will be applied in this work. If the set of all probability that is on I_*n* = {1, …, n} by n={(p1,…,pn): ∑i=1npi=1 and pi≥0 for all i} for the two probability distributions p=(pi) and q= (qi) in Δn0. Also, if the subsystems A and B are two probability distributions, A = {a₁, a₂, …, a_*N*} and B = {b₁, b₂, …, b_*N*}, then Tsallis cross-entropy can be expressed as

Sq(A:B)=∑i=1Npi(piqi)1-q-1q-1+ ∑i=1Nqi(qipi)1-q-1q-1.(3)

Tsallis cross-entropy is selected as the objective function due to its non-extensive nature, which allows for a more flexible and sensitive measurement of information discrepancy than standard Shannon entropy. This sensitivity is crucial for capturing subtle diagnostic details and non-linear correlations in complex multimodal medical images.

3.3 Non-subsampled shearlet transform

Non-subsampled shearlet transform (NSST) is popular alongside some other tools used for multi-resolution processing, such as wavelet curvelet and non-subsampled contourlet transform, because of NSST's applications in the decomposition of signals or images into single low-frequency subbands and many high-frequency subbands (Mishra and Dhabal, 2020). However, NSST indicates some peculiarity over other analytical tools by involving a shear matrix which enables direction filtering.

3.4 Low-frequency subband fusion

Two source images A and B, as represented in this proposed approach, are fused to obtain Q, the final fused image. A specific integration of fusion is represented as Q = (A, B). The fusion process in low-frequency subbands can be described as follows.

The proposed model integrates the low-frequency sub-images using the averaging method (Hu et al., 2018) to reach the fused low-frequency coefficients. A fused low-frequency sub-image result is obtained as

LFIGR(i,j)=12(LFIGA+LFIGB),(4)

where LFIGA and LFIGB are the two sub-images representing the two low-frequency sub-images from two modalities, and g = (1, …, G) denotes the scale of representation found in the transformed space at direction h(i, j) in alignment with a spatial location (i, j).

3.5 Multi-objective optimization

The multi-objective optimization minimization problem can be described mathematically as shown in the following equation (Zhu and Han, 2021):

minF(x)=(f1(x), f2(x), …, fm(x) ),such that gi(x)≤0, i=1, 2, …, p,hj(x)=0, j= 1, 2, …, l,(5)

where x = (x₁, x₂, …, x_*D*) is a D-dimensional decision variable, m is the number of objective functions in the multi-objective optimization problem, f_*m*(x) is the mth objective optimization problem, g_*i*(x) is the ith inequality constraint, p is the number of inequality constraints, h_*j*(x) is the jth equation constraint, and l is the number of equation constraints.

MMIF is treated as a complex multi-objective optimization problem (MOP) requiring simultaneous trade-off between conflicting goals. We will explicitly define the MOP objective function vector F(x) to maximize fusion quality while minimizing processing time: Optimize F(x) = [max[f₁(x)), min[f₂(x))]. Here, f₁(x) is the fusion quality maximized using Tsallis cross-entropy (as described in Section 4.1.1), and f₂(x) is the overall fusion processing time (T_*run*), the minimization of which validates the method's applicability for real-time surgical procedures.

Definitions 1–4 encompass the concept of multi-objective optimization (Han et al., 2021).

Definition 1 (Pareto dominance). The two solutions in the feasible region X_Ω are represented as x₁, x₂; x₁ can be seen as the solution that dominates x₂, and this relationship is mathematically denoted by x₁ > x₂ if and only if

∀i {1, 2, …, m}, fi(x1)≤ fi(x2 ),∃ j {1, 2, …, m}, fj(x1)< fj(x2).(6)

In the context of multi-objective optimization, Pareto dominance is a rule used to compare two different solutions to determine which is superior when multiple goals are being balanced (such as image contrast vs. detail preservation).

A solution is said to dominate another if it meets two specific criteria:

• It is at least as good as the other solution in all objectives.
• It is strictly better than the other solution in at least one objective.

Essentially, if a solution is “dominant,” it means you cannot improve it further in one area without making it worse in another. The goal of the FOMODPSO algorithm is to find a set of these “non-dominated” solutions, which together form what is known as the Pareto front.

Definition 2 (Pareto optimal). The solution x^* ∈ Ω is applied as the Pareto optimal of Equation 1 if and only if

¬∃x∈ XΩ, x>x*.(7)

Definition 3 (Pareto optimal set). The Pareto optimal set is put together by solutions that are finally obtained in the MOP, where each meets the conditions of Definition 2. It is defined as

P= {x∈ XΩ ∣¬∃x∈XΩ: x>x}.(8)

Definition 4 (Pareto front). The region formed by the objective function values of all Pareto optimal solutions is called the Pareto front surface, which is defined as

PF= {F(x)∣x∈P}.(9)

The proposed FOMODPSO algorithm optimizes a multi-objective function vector, F(x), which is designed to balance image quality and efficiency. The two objectives are (1) maximize f₁: the Tsallis cross-entropy, which serves as the primary metric for fusion quality and information transfer, and (2) minimize f₂: the overall fusion processing time (T_*run*), ensuring the method's suitability for real-time clinical applications.

3.6 Darwinian particle swarm optimization

The PSO algorithm is generally given as Wu et al. (2021):

vt+1n=wVtn+ρ1r1(ḡtn-xtn)+ρ2r2(ẋtn-xtn)+ρ3r3(ñtn-xtn),(10)

xt+1n=xtn+vt+1n,(11)

where n is the moving particle, xtn is the position of the particle, vtn is the velocity of the particle, xtn is the local best. ntn is the neighborhood best, gtn is the global best, ρ1, ρ2, and ρ3 assign weights to the global best, local best, and neighborhood best, respectively, when determining the new velocity, and γ1, γ2, and γ3 are random vectors with each component generally a uniform random number between 0 and 1.

DPSO is different from PSO only in the fact that some simple rules are followed to delete a swarm, delete particles, and spawn a new swarm and a new particle.

After the deletion of the particle, instead of being zero, the counter is reset to a value approaching the threshold number, according to Ahilan et al. (2019):

SCc(Nkill)=SCcmax[1-1Nkill+1 ],

where N_kill is the number of particles deleted from the swarm over a period in which there was no improvement in fitness.

3.7 Fractional-order multi-objective Darwinian particle swarm optimization (FOMODPSO)

Fractional calculus (FC) is an essential mathematical tool in the applied sciences (Sabatier et al., 2007). FC has been influential in increasing the performance of several algorithms applied in modeling, curve-fitting, filtering, pattern recognition, edge detection, identification, stability, controllability, observability, and robustness (Couceiro et al., 2012). FC has been applied to increase the convergence rate of PSO in some works, such as Pires et al. (2010) and Couceiro et al. (2012), where the convergence rate was improved for the velocity of PSO and DPSO, respectively. Fractional-order Darwinian particle swarm optimization (FODPSO) was applied in Ahilan et al. (2019) for multilevel thresholding of medical images, and the results obtained showed that the fractional coefficient gives the advantage of a greater level of exploration, thereby giving rise to the global solution of the algorithm. FODPSO was also applied in Waseem et al. (2020), where FODPSO was hybridized with a neural network to obtain better solutions for ordinary differential equations, and high-performance results were obtained.

The FC derived from the Grünwald–Letnikof definition based on the concept of fractional definition with fractional coefficient α ε c of a general signal x(t) is given by:

D∝[x(t)]limh→o[1hα∑k=0+∞(−1)kγ(α+1)x(t−kh)γ(k+1)γ(α−k+1)],(12)

where γ is the gamma function.

The discrete-time implementation of Equation 12 is

D∝[x(t)][1Tα∑k=0r(−1)kγ(α+1)x(t−kh)γ(k+1)γ(α−k+1)],(13)

where T represents the sampling period and r represents the truncation order.

The equation of fractional-order particle swarm optimization (FOPSO) is expressed as

D∝[Vt+1n]=ρ1r1(ḡtn-xtn)+ρ2r2(ẋtn-xtn)+ρ3r3(ñtn-xtn).(14)

FODPSO offers an O(r) memory need. For r = 4 the differential derivative is given as

vt+1n=αvtn+12αvt-1n+16α(1-α)vt-2n+124α(1-α)(2-α)vt-3n ρ1r1(ḡtn-xtn)+ρ2r2(ẋtn-xtn)+ρ3r3(ñtn-xtn),(15)

where α is the FC and DPSO is an exceptional case of FODPSO with α = 1.

The FC operator is introduced into the DPSO velocity update formulas (Equation 15) to provide the particle velocity with a crucial “memory effect,” which fundamentally alters the algorithm's search behavior. This enhanced capability serves two functions: it significantly increases exploration, allowing the algorithm to escape common local optima and avoid premature convergence prevalent in image fusion tasks, and mathematically, it is proven to increase the fundamental convergence rate of the optimization process. This increased speed is the direct mechanism that ensures the high computational efficiency required for real-time application.

To clarify the methodological novelty, the proposed FOMODPSO represents a substantive algorithmic evolution from standard PSO variants. While MOPSO handles multiple objectives, it lacks the multi-swarm diversity of Darwinian principles, often leading to premature convergence. Conversely, DPSO excels at escaping local optima through natural selection but is traditionally limited to single-objective tasks. MODPSO bridges these by applying Pareto dominance to the Darwinian swarm-deletion logic. The final enhancement, FOMODPSO, introduces fractional-order velocity updates to add non-local memory, significantly accelerating convergence speed beyond integer-order methods. Table 3 summarizes these distinctions.

Table 3. Substantive comparison of evolutionary optimization variants.

3.8 Pretrained neural network

The multichannel PCNN (m-PCNN) is well known to be able to effectively manage different types of medical images (Zhao et al., 2014). The simplified m-PCNN is usually referred to as a three-dimensional PCNN because it has only three parameters (Raha et al., 2020). In this paper, we actualize the efficient optimization of parameters in the m-PCNN model. This is because it is essential to adjust the parameters to align with the input images for more accurate image fusion results. The simplified m-PCNN model can be seen in Figure 1.

The system utilizes a simplified m-PCNN, which is a zero-learning architecture that eliminates the need for large training datasets or traditional loss functions. Instead of manual tuning or backpropagation, the network's performance depends on the selection of optimal parameters (β, Θ, α_θ, V_θ) through the proposed FOMODPSO algorithm. This optimization process treats the Tsallis cross-entropy between source and fused images as the fitness function, ensuring that the network's firing activity, which drives the fusion of high-frequency subbands is maximized for information content. The architecture is applied to 14 groups of CT/MRI images (256 × 256 pixels), which are co-registered and normalized as a preprocessing step to ensure diagnostic accuracy.

The integration of these specific techniques is motivated by their complementary strengths: FOMODPSO is utilized to provide global optimization with fast convergence via fractional-order updates; Tsallis cross-entropy is selected as a robust objective function to measure multisource information transfer more effectively than standard entropy; and the PCNN is employed as the fusion rule for its biological pulse-synchronization property, which is superior at preserving fine spatial details and edges.

4 The proposed method

This section mainly describes the process of FOMODPSO-optimized Tsallis cross-entropy for application as external stimuli for the PCNN for the fusion of high-frequency subband. Then the step-by-step fusion framework of our proposed method is described.

4.1 High-frequency subband fusion involving PCNN

4.1.1 Generation of optimized Tsallis cross-entropy

The Tsallis cross-entropy between A and B in the image analysis is defined as the sum of the Tsallis cross-entropy between pixels found within the target area of the image and the Tsallis cross-entropy between pixels found within the background area of the image. Then, with consideration of the limited feature of the Tsallis entropy, the total Tsallis cross-entropy of the image can be expressed as

Sq(A:B;t)=SqA(t)+SqB(t)+(1-q)SqA(t)+SqB(t).(16)

When S_*q*(A:B; t) accepts an optimum value, the corresponding gray value is the optimal image potential value, that is,

Topt=argmax0≤t≤n[Sq(A;B;t)](17)

To account for the varying intensity ranges of high-frequency coefficients from different source modalities, the NSST coefficients are first normalized to a uniform range (e.g., [0, 1]). Subsequently, their probability distributions are estimated using histogram-based binning, which allows for the consistent calculation of the Tsallis cross-entropy regardless of the original coefficient magnitudes.

The fusion criterion is required for choosing the optimal fusion points and iteration number in PCNN. Tsallis cross-entropy is, therefore, applied as the criterion function for optimal fusion points. Tsallis cross-entropy can also be further explained to be used as the source of stimulation to the PCNN model. It is applied as the objective function to validate the optimal fusion points for considered images. FOMODPSO maximizes the objective function and is also used to find the optimal solution of the Tsallis cross-entropy function. This implies that the function f(T_*opt*) is maximized by the FOMODPSO, and the optimal values obtained indicate the region of the source images that should be fused, making the fusion results more reasonable. After the decomposition of the source images in the NSST domain, Tsallis cross-entropy is computed for all the coefficients. Then the Tsallis cross-entropies are sent to the PCNN as external stimuli.

Tsallis cross-entropy is strategically utilized as the objective function because, unlike standard Shannon entropy, it is a generalized form that inherently models the information transfer from both source images to the fused image simultaneously. This consideration of information from all inputs ensures the preservation of essential image features, thereby enhancing the overall fusion quality. Furthermore, maximizing the Tsallis cross-entropy function yields the optimal threshold, which is critical as it serves as the optimal external stimulus to the PCNN (Equation 18), making the fusion results more reasonable and accurate by selecting the most favorable coefficients.

4.1.2. The structure of PCNN

PCNN is applied to confirm the information sensitivity of the image pixels. The neural network accepts the Tsallis cross-entropy S_*q*(A:B) = S as its input and outputs a “firing count” (Mishra and Dhabal, 2020). The PCNN model equations are expressed in Equations 16–20. The greater the aligning “count” is, the more applicable the optimal fusion points T_*opt*:

Fij(n)=Sij,(18)

Lij(n)=∑klWijklYnl(n-1),(19)

Uij(n)=Fij(n)(1+ βLij(n)),(20)

{1, if Uij(n)≥θij(n), 0, otherwise, (21)

θij(n)=e-αθLij(n)+VθYij(n-1),(22)

Rij(n)=Rij(n)+Yij(n-1),(23)

where F_*ij* and L_*ij* are the feedings and linking synapses that align to the original ith and jth position of the pixel in the original image, S_*ij* is the external stimulus and controls the time of ignition, W_*ijkl* indicates the local connection matrices made up of weights that can quantify the influence of each neuron that is near the central neuron, β is the linking coefficient, Y_*ij* is the binary output of the model, α_θ is the attenuation time constant, V_θ are the corresponding inherent voltage potentials, n represents the iteration time, and R_*ij* denotes the PCNN firing times.

Equations 18–22 are the control equations of the model, ensuring the inevitability of higher computational accuracy of the proposed model.

4.2 Steps of the fusion process

To denote the point of the input images by similar corresponding pixels, the source images are required to be co-registered for the proposed method to give efficient results. The implementation of the multimodal image fusion scheme is described in the following steps and Algorithm 1 shows the framework of FOMODPSO.

• Step 1. Decompose the previous co-registered medical images through NSST to generate low- and high-frequency sub-images (LFIs and HFIs, respectively).
• Step 2. Carry out the fusion on LFI coefficients using Equation 4.
• Step 3. Obtain the Tsallis cross-entropies that align with all HFIs as stated in Section 3.2.1. The generated entropies are then applied as external stimuli to PCNN using Equations 18–22.
• Step 4. Compute the firing times R_*ij* by Equation 23.
• Step 5. Steps 1–4 are repeated until a pre-selected number of iterations are achieved.
• Step 6. Carry out the fusion of HFIs (HFIGR) using the rule
• Step 7. Finally, the fused image Q is generated by using inverse NSST on the fused coefficients of LFIs and HFIs.

Figure 2 shows the schematic diagram of the steps of the proposed image fusion framework.

5 Experimental results and discussion

5.1 Experimental setup

To investigate the accuracy and robustness of the proposed FOMODPSO–PCNN method, the experimental setup utilized 14 groups of public benchmark medical image pairs (28 images total), consisting of CT and magnetic resonance imaging (MRI) modalities with a resolution of 256 × 256 pixels. These images (Johnson and Becker, 1999) were sourced from the Harvard Medical Image Database (https://dataverse.harvard.edu/). For statistical validity, the multi-objective search quality (inverted generational distance (IGD) and HV) was evaluated based on 31 independent experimental runs, reporting the mean and standard deviation for each test instance. Additionally, a three-fold cross-validation protocol was implemented for the diagnostic performance evaluation (Table 4), ensuring that the reported accuracy (90.7%) and sensitivity results are consistent across multiple subsets of the data.

Table 4. Performance evaluation using three-fold cross-validation.

The figures in bold indicate the highest values obtained by our proposed method for the evaluation parameters (Average and Standard deviation).

As seen in Figure 3, images A are the source CT images and images B are the source MRI images, which are to be fused. In total, we used 28 source images to conduct our experiments; 14 from each modality. The images have 256 × 256 pixel resolutions. All the images in the dataset are correctly preregistered. The simulation software is adopted by Matlab R2018a version 9.4.0.813654 64-bit for the search of the optimal values of the test functions, and the hardware used is 3.20 GHz, 4 GB RAM, and Windows 7.

The common parameters of the FOMODPSO, the population size and maximum iteration, are kept randomly and conveniently at 20 and 200, respectively; the stagnancy threshold is set randomly and conveniently at 15; we set the α parameter to 0.6 as it has been proven to give better results when applied (Chakraborty and Verma, 2021).

To evaluate the performance of the proposed method denoted as the FOMODPSO + PCNN MMIF method, some comparative experiments are conducted with eight image fusion algorithms on five multi-objective problems (test instances) f₁ − f₅ (Gao et al., 2014; Couceiro et al., 2012). The eight image fusion algorithms are QPSO+PCNN (Tian et al., 2016), MOPSO+PCNN (Wang et al., 2016), PSO+PCNN (Kumar et al., 2021), MFOA+PCNN (Tang et al., 2019), WOA + PCNN (Raha et al., 2020), PSO-TV+PCNN (Nie et al., 2021), PSO-DE+PCNN (Liu R. et al., 2022), and DE+PCNN (Das et al., 2022). QPSO+PCNN is an MMIF method that uses QPSO for the optimization of the PCNN to obtain good-performance fused images. MOPSO+PCNN is an MMIF method that applies MOPSO in the optimization of the PCNN to obtain fused images of very good visual quality. PSO+PCNN is an MMIF method that applies hybrid PSO for the optimization of the PCNN to obtain high-performance fused image results. MFOA+PCNN is an MMIF method that applies an MFOA for the optimization of the PCNN to obtain fused image results of high quality. WOA+PCNN is an MMIF method that optimizes PCNN using the WOA for fused image results of well-preserved image details. PSO-TV+PCNN is an MMIF approach that applies PSO and total vibration (TV) for the optimization of the PCNN to obtain fused image results of high visual quality. PSO-DE+PCNN is an MMIF method that optimizes PCNN using PSO and standard DE algorithm to obtain fused images of very good fusion effect. DE+PCNN is an MMIF method that applies the DE for the optimization of the PCNN to obtain fused images of both good visual quality and superior quantitative evaluation results.

Multi-objective optimization vs. single-objective optimization indicates how much more the complexity of multi-objective optimization is. Another difference is in the non-uniqueness of the solution of multi-objective optimization, which is a solution set. Table 5 shows the optimal solution set. The most suitable visual effect corresponding to the image fusion results is selected. They are the final optimal PCNN parameters: the connection weight W = [0.5, 1.0, 0.5;1.0, 0, 1.0;0.5, 1.0, 0.5], N = 100; β = 0.9950; θ = 0.1233; V_θ = 1.8119; α_θ = 0.1482.

Table 5. Set of optimal solutions.

The optimization parameters can be summarized as

• FOMODPSO: population size = 20; max. iterations = 200; α (fractional order) = 0.6.
• Final optimized PCNN parameters: W = [0.5, 1.0, 0.5; 1.0, 0, 1.0; 0.5, 1.0, 0.5]; β = 0.9950; θ = 0.1233$; V_θ = 1.8119; α_θ = 0.1482.

5.2 Performance metrics

5.2.1 Quantitative metrics

We apply two metrics, IGD and hypervolume (HV; Li et al., 2019), in this work as the characteristic factors to increase the search performance of our FOMODPSO algorithm. The Pareto set is the set of all Pareto optimal solutions, while the Pareto front is the set of objective values of Pareto optimal solutions. The approximation set is the set of the non-dominated solutions that are obtained from searching, while the approximation front (AF) is the set of objective values of the non-dominated solutions. When tackling multi-objective optimization problems, it is beneficial when AF converges to PF as closely as possible (convergence) to ensure its accuracy and also to distribute along the entire PF as uniformly as possible (diversity) to generate numerous representative solutions. IGD can verify the convergence and diversity of multi-objective evolutionary algorithms when multi-objective problems are being solved. HV can also verify convergence and diversity simultaneously by computing the size of the space existing between reference points and individuals in AF. This implies that low IGD and high HV indicate that AF is near to PF and distributes evenly as closely as possible along the entire PF. When computing HV in the following experiments, the reference points are set as z=(2, 2)T, z=(2, 2, 2)T, z=(2, 2, 2)T, z=(2, 2)T, z=(2, 3)T for f1-f5, respectively.

5.2.2 Objective quality metrics

The performance of the proposed method is measured by taking into consideration some popular evaluation metrics. The chosen evaluation measures are entropy (ENT), mutual information (MI), edge information similarity (Q_*abf*), and structural similarity index [SSIM; for mathematical information see (Li et al., 2020; Singh and Anand, 2020; Hermessi et al., 2018)]. The selected evaluation measures, entropy (ENT), MI, edge information similarity (Q_*abf*), and structural similarity index (SSIM) are chosen to provide a multidimensional assessment of fusion quality. ENT and MI are utilized to measure the richness and volume of information transferred from the source images, while Q_*abf*and SSIM validate the preservation of edges and structural integrity, ensuring the fused images are diagnostically reliable. To guarantee statistical robustness, all multi-objective search quality results are reported as the mean and standard deviation across 31 independent runs. Diagnostic performance is further validated using three-fold cross-validation, ensuring that the superior performance of the FOMODPSO + PCNN method is consistent across the 14 clinical image groups and various test instances.

5.3 Experimental results

5.3.1 Quantitative evaluation results and analysis

According to the experimental results obtained 31 times running, IGD and HV (mean and standard deviation) values of fused images on test instances f₁ − f₅ are computed; these are tabulated in Table 6 to demonstrate the basis of comparison between FOMODPSO + PCNN and other image fusion methods. The first quality metric value noted in Table 6 is the mean, and on the right side of the mean is the standard deviation for each of the IGD and HV experimental results. The rankings are indicated in square brackets on the right side of the quality metric values. To show that the performance of FOMODPSO + PCNN is better than, worse than, or similar to that of the compared method, we use “⊢,” “≬,” and “~,” respectively. Finally, mean ranking (MR) is obtained by each method in all test instances and the number of “⊢,” “≬,” and “~” indicated for each compared method are computed to evaluate the average performance of the image fusion methods.

Obtained results (mean (std. dev.)[rank]) of IGD and HV metric values of the proposed FOMODPSO + PCNN and eight state-of-the-art image fusion algorithms.

Table 7.

Table 8.

The figure in bold indicate that amongst all the image fusion methods, our proposed method, FOMODPSO + PCNN has the best performance on test instances, concerning, convergence and diversity (search quality).

From Table 6, it can be observed that FOMODPSO + PCNN has 77 better mean metric values as compared to the other methods, and FOMODPSO + PCNN has nine best mean metric values and is the only method with MR less than two quality metrics. MFOA + PCNN performed better than FOMODPSO + PCNN on f₄ concerning IGD values, while DE + PCNN has a similar performance to FOMODPSO + PCNN on f₅ concerning IGD and HV values. All the image fusion methods are ranked according to MR: FOMODPSO + PCNN, DE + PCNN, PSO-DE + PCNN, PSO-TV + PCNN, WOA + PCNN, MFOA + PCNN, PSO + PCNN, MOPSO + PCNN, and QPSO + PCNN. Of the image fusion methods, FOMODPSO + PCNN has the best performance on test instances concerning convergence and diversity (search quality).

5.3.2 Visual quality evaluation results and analysis

From Figure 4, which shows various medical image fusion results, it is clear that our method richly preserves the subtle essential image details, enhances spatial information, and has higher visual quality than the other comparative methods. The QPSO + PCNN, MOPSO + PCNN, and PSO + PCNN gave fairly good image details retention ability, but the general image contrast is not good. MFOA + PCNN, WOA + PCNN, and PSO-TV + PCNN, in comparison, gave results of good fused image quality, but the edge preservation ability was low. The PSO-DE + PCNN shows a good fusion effect but may contain some noise in the fused image results.

From this observation, it can be easily summarized that our proposed method gave fused images of better visual quality, had more edge-details preservation capability, and had less chance of showing artifacts.

5.3.3 Quantitative evaluation results and analysis

To evaluate the effectiveness of the proposed method, the accuracy, sensitivity, specificity, and F₁ score are chosen as evaluating parameters. Our proposed method has proved to have an average accuracy of 90.7%, sensitivity of 0.796, specificity of 0.876, and F₁ score of 0.789.

The superior diagnostic performance of the proposed method, characterized by a 90.7% average accuracy and 0.789 F₁ score, is fundamentally driven by the non-local memory features introduced by the FC operator (α = 0.6). Standard integer-order optimization methods often exhibit “myopic” behavior, accounting only for the current state and increasing the risk of premature convergence to local minima. In contrast, the fractional-order “memory effect” allows FOMODPSO to leverage historical search trajectories, ensuring the algorithm identifies the global optimum within the complex PCNN parameter space. This robust global search identifies the precise settings for β, V_θ, and α_θ required to maximize the Tsallis cross-entropy, thereby ensuring that the most salient pathological features from both CT and MRI modalities are preserved. Because MMIF serves as a critical preprocessing step for computer-aided diagnosis systems, the provision of these high-fidelity, information-rich composite images directly enhances the reliability of subsequent diagnostic classifications, accounting for the significant performance gains observed in this work.

The quantitative evaluation task focuses on assessing the quality of information transfer from source images to the fused output. This is achieved by measuring the fused images against four objective metrics: peak signal-to-noise ratio, MI, structural similarity index (SSIM), and spatial frequency.

We extracted a small dataset from that described in Section 4.1 in order to carry out a three-fold cross-validation. This is for clarity of assessment and ease of operation. In the presented evaluation, we have undertaken an analysis of each comparative method and our proposed method, so that the method with the best values can be selected based on the performance measures. An in-depth analysis reveals that our proposed method gives the best fusion results. Experimental results for each of the cross-validation stages are shown in Table 4, while detailed experimental results of all the other stages can be found at:

• https://github.com/GoddessChysomme/FOMODPSO/files/13933461/G1-G14.pdf
• https://github.com/GoddessChysomme/FOMODPSO/issues/1

The codes used in this work are open-sourced and can be found at:

• https://github.com/GoddessChysomme/FOMODPSO/blob/main/fusion
• https://github.com/GoddessChysomme/FOMODPSO/blob/main/xfusmaxmin
• https://github.com/GoddessChysomme/FOMODPSO/blob/main/xfusmean

The inclusion of diagnostic classification metrics (accuracy, sensitivity, specificity, and F₁ score) goes beyond conventional image processing metrics (such as entropy and SSIM) to provide crucial clinical and diagnostic validation for the fused images. The purpose of MMIF is to aid in accurate patient diagnosis, and these metrics evaluate the practical utility of the fused images by measuring their ability to reliably retain or enhance diagnostically relevant features, treating the fusion result as input to a simulated diagnostic process. The highest achieved average accuracy (90.7%) confirms that the FOMODPSO-optimized PCNN parameters yield fused images that significantly improve potential downstream diagnostic classification reliability compared to all comparative methods (Table 4).

From Table 4, we can see that the performance of our proposed method is the highest for all evaluation parameters.

While the quantitative and qualitative metrics demonstrate the technical superiority of the FOMODPSO method in terms of fusion quality, these results should be interpreted as indicative of algorithmic performance rather than a comprehensive clinical validation. Given the limited dataset of 14 two-dimensional images, the current study serves as a robust proof-of-concept. Future clinical studies involving larger, multi-institutional datasets are required to validate the method's diagnostic efficacy in real-world medical practice.

5.4 Computational efficiency

To evaluate the computational efficiency, we compute and list the average time consumption on all image groups for different methods, as shown in Table 1. Our method takes the shortest time, while MFOA + PCNN has a high time consumption. The reason our method has low time consumption is because of an increased rate of PCNN parameter optimization. Increasing the convergence rate of MODPSO using the FC operator, and applying the improved optimization algorithm to Tsallis cross-entropy maximization, and the optimized entropy is used as input to the PCNN network causing the reduction of the average time consumption of our method in comparison to other comparative methods.

This computational efficiency confirms that our method can be used to carry out MMIF within a very short time, and this is the main criterion for effective real-time image fusion applications (Fang et al., 2023).

6 Conclusions

In this paper, a novel method of MMIF is presented. The method combines improved MODPSO and Tsallis cross-entropy to obtain the best fusion potential for the input of PCNN, for effective MMIF. The MODPSO is improved by applying a fractional calculus operator, a mathematical tool capable of increasing the performance of MODPSO with a higher convergence speed. The main goal of the proposed algorithm is to develop an MMIF algorithm that provides fused images of high visual quality without being time-consuming. To demonstrate the performance of the FOMODPSO after using it to optimize the Tsallis cross-entropy for the input of the m-PCNN for MMIF, we compare the proposed MMIF algorithm with some other image fusion algorithms on several test instances. The experimental results demonstrate that the proposed FOMODPSO + PCNN image fusion method excels in merging CT and MRI data. However, it is important to frame these findings as evidence of enhanced fusion performance. Further validation with larger and more diverse datasets is necessary to establish its clinical utility in various diagnostic environments. In the future, we would like to improve our FOMODPSO algorithm to guarantee the finding of an optimal solution by incorporating a wave function as the factor that indicates the state of the particle.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Moulay Akhloufi, Université de Moncton, Canada

Reviewed by: Jie Wu, Shaanxi Normal University, China

Rakesh Choudhary, SUNY Upstate Medical University, United States