Explainable artificial intelligence approaches in cardiovascular imaging: methodological advances and clinical implications

1 Introduction

Cardiovascular disease (CVD) remains one of the most pressing global public health challenges, closely associated with adverse clinical outcomes and high mortality. Epidemiological data indicates that deaths caused by CVD account for approximately 32% of total global deaths, claiming nearly 17.8 million lives annually (Martin et al., 2025). This severely compromises the quality of life for millions of patients and imposes substantial economic and resource burdens on global healthcare systems.

Advanced diagnostic imaging techniques, such as Coronary Computed Tomography Angiography (CCTA), Cardiac Magnetic Resonance (CMR), Echocardiography (Echo), and Chest X-Ray (CXR), constitute the core pillars of the modern cardiac diagnosis and treatment system (Dey et al., 2019). Through non-invasive, high-resolution, and multi-dimensional assessments, these modalities enable detailed characterization of cardiac structure and function, playing a pivotal role in early screening, disease subtyping, risk assessment, and treatment management (Barison et al., 2025). However, with the exponential surge in cardiac imaging data, traditional analysis models struggle to meet the dual clinical demands for efficiency and accuracy (Moor et al., 2023). In recent years, Artificial Intelligence (AI) technologies, represented by Machine Learning (ML), have demonstrated significant advantages in cardiovascular medical imaging analysis due to their robust data processing capabilities (Kagiyama et al., 2022).

Deep Learning (DL), as a core branch of ML, is built on multi-layer artificial neural networks and offers strong end-to-end feature learning capabilities. It can automatically extract highly discriminative deep abstract features from raw imaging data (LeCun et al., 2015). Consequently, DL effectively overcomes performance bottlenecks when processing high-resolution, high-noise, and structurally complex cardiovascular images, accurately capturing subtle pathological alterations (Kosaraju, 2024). Currently, DL networks—represented by Residual Neural Networks (ResNet) (He et al., 2015), U-Net (Ronneberger et al., 2015), Transformers (Vaswani et al., 2017), and Med-SAM (Ma et al., 2024)—have been widely applied to tasks such as cardiovascular image acquisition and reconstruction (Apostolopoulos et al., 2023), anatomical structure segmentation (Chen et al., 2020), as well as disease diagnosis and risk prediction (Zhou et al., 2024). These applications have substantially improved the precision, speed, and automation of image analysis, thereby facilitating clinical use.

Despite the remarkable performance of AI in cardiovascular imaging, its complex non-linear architecture often results in opaque decision-making, creating a persistent “black box” phenomenon (Petch et al., 2022). This lack of interpretability prevents clinicians from comprehending the underlying logic of automated diagnoses or risk assessments, thereby undermining trust in model outputs (Szabo et al., 2022). Furthermore, this opacity hinders the fulfillment of established standards for evidence-based medicine and medical accountability, as the basis for algorithmic judgments remains largely inaccessible (Marey et al., 2024). Driven by the dual mandates of demystifying the “black box” and strengthening clinical trust, eXplainable Artificial Intelligence (XAI) has emerged as a critical field. By offering human-understandable insights through visualizations and semantic representations, XAI elucidates model decision-making mechanisms, rendering AI logic transparent, comprehensible, and accountable (Samek et al., 2017).

Building on this rationale, the need for explainability is particularly urgent in high-stakes cardiovascular care, where patient safety is paramount; clarifying the basis of model decisions to mitigate bias is a fundamental prerequisite for adopting AI-driven assistance (Marey et al., 2024). Consistent with this trend, advances in other diagnostic domains have increasingly embedded explainability into fusion-oriented, clinically interpretable AI pipelines. Evidence from these fields supports this direction: interpretable deep learning for lung cancer detection (Wani et al., 2024a) and fusion-based explainable frameworks for breast cancer classification (Wani et al., 2024b) both demonstrate the practical value of integrating predictive performance with clinically meaningful interpretation. Collectively, these developments indicate a broader shift from isolated, opaque predictions toward clinically grounded reasoning, which is directly relevant to the design of next-generation cardiovascular XAI systems. Figure 1 provides a conceptual overview of this integration, illustrating how deep learning models, explainability methods, and evaluation strategies converge to support more reliable clinical decision-making in cardiovascular imaging.

1.1 Motivation

Recent reviews have summarized the major techniques, opportunities, and challenges of XAI in broader healthcare and Internet of Medical Things (IoMT) settings, providing an important foundation for trustworthy medical AI (Wani et al., 2024c). Building on this foundation, a dedicated review of cardiovascular imaging is needed because this field presents distinctive interpretability challenges, including the dynamic temporal characteristics of Echo, the complex three-dimensional structures of CMR, and the heterogeneous information across modalities. These domain-specific features require XAI strategies beyond general-purpose methods, while a persistent gap remains between technical model development and their clinical evaluation, interpretation, and adoption in cardiovascular practice. In particular, improving how XAI integrates with specialized imaging modalities across cardiovascular diagnosis and treatment is critically important for real-world impact. Motivated by these issues, this review aims to systematically organize current research at the intersection of XAI and cardiovascular imaging and to support methodological progress, clinically meaningful interpretation, and translational adoption.

1.2 Contributions

In view of the aforementioned motivations, this paper comprehensively reviews XAI technologies in the field of cardiovascular imaging. The main contributions of this review are summarized as follows:

• Systematic Taxonomy of XAI Methods: We provide a detailed classification and analysis of current XAI techniques applied specifically to cardiovascular imaging, connecting algorithm mechanisms with imaging characteristics.
• Modality-Specific Clinical Applications: We systematically review the application of XAI across diverse clinical tasks, including anatomical segmentation, disease diagnosis, and risk prediction, as well as across core cardiovascular imaging modalities such as CCTA, CMR, Echo, and CXR.
• Comprehensive Review of Evaluation Metrics: We summarize the quantitative and qualitative evaluation metrics currently used to assess the reliability, robustness, and clinical utility of XAI models in cardiology.
• Challenges and Future Directions: We identify the current technical and clinical bottlenecks in the field and propose a roadmap for future research, providing theoretical references to accelerate the clinical translation of transparent AI systems.

2 XAI approaches

In the field of cardiovascular medical image analysis, XAI is predominantly realized through two complementary paradigms, namely, visual explanation and feature attribution. The overarching objective of XAI is to generate attribution maps or saliency maps that provide an intuitive yet quantitative characterization of the contribution of individual pixel regions in the input image to the model’s final prediction. In practice, visual explanations are most commonly instantiated as feature attribution maps, which makes the two paradigms closely aligned in imaging applications. On this basis, existing mainstream XAI approaches can be broadly classified into two principal categories: backpropagation-based methods and perturbation-based methods. Figure 2 schematically illustrates the fundamental principles underlying these two categories. As shown in Figure 2A, backpropagation-based methods exploit gradient-based or relevance-propagation information within the neural network to compute the sensitivity of the target output class with respect to the input image, thereby producing saliency maps that highlight regions most influential to the model’s decision. In contrast, Figure 2B depicts perturbation-based methods, which estimate feature importance by introducing predefined or systematic perturbations to the input image and analyzing the resulting variations in the model output, thus enabling the construction of attribution maps in a largely model-agnostic manner. Beyond the methods detailed here, other widely used approaches such as Integrated Gradients, Layer-wise Relevance Propagation, and RISE have also been reported in the literature and can be viewed as complementary techniques within these two categories.

2.1 Perturbation-based methods

2.1.1 Occlusion sensitivity

To interpret the decision-making process of deep neural networks (Zeiler and Fergus, 2014), pioneered the Occlusion Sensitivity method. This approach is founded on the intuitive hypothesis that perturbing critical regions responsible for a model’s prediction will induce a significant alteration in its output. By systematically occluding local regions of the input and monitoring the resulting fluctuations in prediction scores, this method generates feature maps that quantify the contribution of each image segment. Mathematically, given an input image x and a model’s prediction function f c ( ⋅ ) for a target class c, the process generates a perturbed image x ( i , j ) by masking a local spatial patch centered at location ( i , j ). The attribution score, S c ( i , j ), for that specific region is then formulated as the difference in the network’s output:where f c ( x ) is the original baseline confidence, x ( i , j ) represents the occluded image where the pixels within the designated patch are replaced by a reference value, such as zero or the dataset mean, and S c ( i , j ) explicitly quantifies the positive or negative contribution of that occluded segment toward the final classification. As indicated by Equation 1, this perturbation-based paradigm is inherently model-agnostic, as it requires neither access to internal network architecture nor gradient information; however, because each sliding-window perturbation necessitates an additional forward inference, the computational burden scales substantially with the number of evaluated patches, thereby constraining practical efficiency in high-resolution medical imaging contexts (Kakogeorgiou and Karantzalos, 2021).

S c i , j = f c x − f c x i , j , (1)

2.1.2 Local interpretable model-agnostic explanations

Local Interpretable Model-agnostic Explanations (LIME) (Ribeiro et al., 2016) is a representative model-agnostic framework that is conceptually grounded in perturbation-based local sensitivity analysis. In contrast to basic occlusion strategies, LIME constructs an interpretable surrogate model to approximate, within a local neighborhood, the decision behavior of a complex black-box predictor. Formally, for a target instance x, explanation generation is formulated as the following optimization problem:In Equation 2 ξ ( x ) denotes the resulting explanation, obtained by selecting a surrogate model g from an interpretable hypothesis class G. The objective balances two competing criteria: local fidelity, quantified by L ( f , g , π x ), and interpretability, controlled by the complexity regularizer Ω ( g ). Here, the proximity kernel π x assigns larger weights to perturbed samples that are closer to x, thereby constraining approximation quality to the local decision manifold.

ξ x = arg min g ∈ G L f , g , π x + Ω g , (2)

For image-based applications, this local neighborhood is typically instantiated by segmenting x into perceptually coherent superpixels, followed by stochastic perturbation through selective masking or activation of superpixel subsets. Model outputs on these perturbed samples are then used to fit the surrogate model, from which superpixel-level importance coefficients are estimated. Consequently, LIME provides spatially localized attribution maps that translate opaque model behavior into anatomically and morphologically interpretable visual evidence.

2.1.3 Meaningful perturbation

To overcome the limitations of rigid, discrete occlusion grids (Fong and Vedaldi, 2017), proposed Meaningful Perturbation, which formulates attribution as an optimization problem over continuous spatial masks. Instead of exhaustively scanning fixed patches, the method seeks a minimally sufficient perturbation that maximally attenuates the model response while preserving semantic plausibility of the perturbed image. Typical perturbation operators include constant-value replacement, additive noise, and Gaussian blurring, all intended to approximate more realistic local degradations. Formally, the optimal soft mask m * is obtained by minimizing:where f c denotes the target-class confidence produced by the black-box model, and Φ ( x , m ) denotes a mask-conditioned perturbation operator applied to input x. The L 1 penalty term ‖ m ‖ 1 promotes sparsity, thereby discouraging trivial full-image masking and encouraging localization of the most influential evidence. The total variation regularizer R TV ( m ) enforces spatial smoothness and suppresses high-frequency artifacts that may otherwise yield non-physiological explanations. The hyperparameters λ 1 and λ 2 control the trade-off among predictive suppression, mask compactness, and spatial coherence. As reflected in Equation 3, this formulation generally yields semantically coherent and morphologically informative attribution regions, thereby improving the clinical interpretability of model-specific decision rationales.

m * = arg min m f c Φ x , m + λ 1 ‖ m ‖ 1 + λ 2 R TV m , (3)

2.1.4 SHapley additive exPlanations

SHapley Additive exPlanations (SHAP), introduced by Lundberg and Lee (2017), provides a unified theoretical framework for interpreting predictions from a broad class of ML and DL models. Rooted in cooperative game theory, SHAP operationalizes feature attribution through Shapley values, thereby quantifying each feature’s marginal contribution to an individual prediction under principled axiomatic constraints. Formally, for feature i, the Shapley value ϕ i is defined as:where N denotes the full feature set, and S is any subset that excludes feature i. The term f x ( S ∪ { i } ) − f x ( S ) represents the marginal contribution of feature i when added to coalition S, whereas the combinatorial coefficient assigns a fairness-aware weight across all possible coalition orderings. The resulting attribution therefore reflects the expected marginal effect of feature i over the entire coalition space.

ϕ i = ∑ S ⊆ N \ i | S | ! | N | − | S | − 1 ! | N | ! f x S ∪ i − f x S , (4)

By aggregating feature-wise Shapley values into an additive explanation model, as in Equation 4, SHAP yields both locally faithful and globally comparable explanations, which is particularly advantageous in settings with non-linear dependencies and complex feature interactions. Despite potentially high computational cost for exact estimation, SHAP remains one of the most widely adopted perturbation-based XAI approaches because of its strong theoretical grounding, model-agnostic applicability, and empirical interpretive utility in high-dimensional biomedical imaging tasks.

2.2 Backpropagation-based methods

2.2.1 Deconvolutional networks

To improve the interpretability of convolutional representations (Zeiler and Fergus, 2014), introduced Deconvolutional Networks (DeconvNet), which project high-level activations back into input space through an approximate inverse mapping of forward CNN operations. Let R l denote the reconstructed response at layer l. Reconstruction toward layer l − 1 proceeds iteratively via unpooling U ( ⋅ ) with recorded max-pooling switches z l, followed by element-wise rectification σ ( ⋅ ) and transposed convolution with spatially flipped filters F l T. The backward reconstruction step is expressed as:In Equation 5, σ ( x ) = max ( 0 , x ) enforces non-negativity by preserving only positive backward responses. This constrained reconstruction highlights input patterns that most strongly excite specific neurons, thereby providing a spatially explicit interpretation of feature selectivity in deep convolutional layers.

R l − 1 = F l T ∗ σ U R l , z l , (5)

2.2.2 Saliency map

Within the Saliency Map paradigm, interpretability is formulated as pixel-level sensitivity analysis of the target prediction with respect to the input. Backpropagation-based interpretability techniques were pioneered by Simonyan et al. (2013), which use input gradients to quantify the influence of individual features on model predictions. Formally, given an input image I and the unnormalized class score S c for target class c, the saliency map M is obtained by computing the absolute magnitude of the partial derivatives in Equation 6.

M I = ∂ S c ∂ I . (6)

This definition directly operationalizes the core objective of Saliency Map methods: quantifying how strongly infinitesimal perturbations at each pixel influence the class-specific score. Accordingly, higher values in M ( I ) indicate regions with greater first-order influence on the decision boundary. From a computational perspective, saliency estimation is efficient because it requires only one backward pass after standard forward inference.

As an important extension within the Saliency Map family (Springenberg et al., 2014), introduced Guided Backpropagation, which combines conventional backpropagation with DeconvNet-style gradient gating to improve spatial sharpness and suppress irrelevant responses. Specifically, gradients are propagated through ReLU units only when both the forward activation f l and the incoming backward signal R l + 1 are positive:In Equation 7, I ( ⋅ ) denotes the indicator function. This dual-constraint mechanism generally yields cleaner and higher-resolution saliency maps, thereby improving localization of discriminative image structures.

R l = R l + 1 ⋅ I f l > 0 ⋅ I R l + 1 > 0 , (7)

Nevertheless, the Saliency Map framework remains subject to intrinsic gradient-based limitations, particularly gradient saturation and local linearity assumptions. As discussed by Shrikumar et al. (2017), for piecewise-linear activations such as ReLU, gradients vanish in the negative regime, as shown in Equation 8:

∂ f x ∂ x = 0 for x < 0 . (8)

Consequently, features located in saturated or negatively activated regions may receive negligible attribution despite potential predictive utility, which can reduce explanatory completeness. Therefore, Saliency Map methods should be interpreted as first-order sensitivity approximations of model behavior rather than exhaustive causal accounts of decision making.

2.2.3 Layer-wise relevance propagation

Layer-wise Relevance Propagation (LRP), introduced by Bach et al. (2015), is a decomposition-based interpretability framework founded on the relevance-conservation principle. The central premise is that the prediction score can be redistributed from the output layer to the input space while preserving total relevance across adjacent layers. In this way, LRP provides a structured attribution of the model decision to lower-level neurons and, ultimately, to input features. Formally, relevance conservation between layer l + 1 and layer l is expressed as:

∑ p R p l = ∑ q R q l + 1 . (9)

As formalized in Equation 9, this conservation condition enforces layer-wise consistency of total relevance. To operationalize this constraint, the relevance score R k ( l + 1 ) of neuron k in layer l + 1 is redistributed to neuron j in layer l in proportion to its forward contribution to neuron k. A widely used redistribution strategy is the ϵ-rule:where a j denotes the forward activation of neuron j, w j k is the synaptic weight connecting neurons j and k, and ϵ is a stabilization constant introduced to avoid numerical singularities when the denominator approaches zero. Equation 10 therefore implements a normalized relevance redistribution that is both contribution-aware and numerically robust.

R j l = ∑ k a j w j k ∑ h a h w h k + ϵ R k l + 1 , (10)

By recursively applying this rule from output to input, LRP yields a pixel-/feature-level relevance map that decomposes the model output into additive explanatory components. This property makes LRP particularly valuable in high-stakes settings, where transparent traceability of model evidence and alignment with domain reasoning are critical for trustworthy interpretation.

2.2.4 Deep SHAP

To mitigate gradient saturation and non-linear attribution instability in deep architectures (Shrikumar et al., 2017), introduced DeepLIFT, which explains predictions relative to a reference input x ̃ rather than relying exclusively on local gradients. Specifically, DeepLIFT attributes importance by propagating activation differences between the observed input and the reference state, thereby capturing both positive and negative feature effects under a difference-from-reference formulation. Formally, let C i denote the attribution assigned to input feature i, let f ( x ) denote the model output for the observed input, and let f ( x ̃ ) denote the corresponding output under the reference baseline. Under this notation, DeepLIFT enforces the Summation-to-Delta constraint:

∑ i C i = f x − f x ̃ . (11)

Equation 11 states that the sum of feature-level attributions exactly reconstructs the output deviation between the observed and reference inputs, thereby ensuring global attribution completeness under the reference-based decomposition framework. Building on this framework and cooperative game-theoretic attribution (Lundberg and Lee, 2017), proposed Deep SHAP, which combines the computational efficiency of DeepLIFT-style backpropagation with the axiomatic structure of Shapley-value explanations. For feature i, the exact Shapley value is given by:where N denotes the full feature set and S denotes any coalition that excludes feature i. The term f ( S ∪ { i } ) − f ( S ) quantifies the marginal contribution of feature i when added to coalition S, while the combinatorial coefficient | S | ! ( | N | − | S | − 1 ) ! | N | ! assigns the appropriate averaging weight over all possible feature orderings. Equation 12 therefore defines ϕ i as the expected marginal contribution across the coalition space. However, exact computation over all coalitions is generally intractable for high-dimensional inputs. Deep SHAP addresses this limitation by approximating Shapley values through layer-wise multiplier propagation and local linearization of non-linear operators, yielding an additive surrogate of the model response, as shown in Equation 13:where ϕ i denotes the contribution of feature i, and ϕ 0 denotes the expected model output under a background distribution. Consequently, Deep SHAP provides a practically scalable yet theoretically grounded compromise between attribution fidelity and computational tractability, making it well suited to large-scale medical imaging settings.

ϕ i = ∑ S ⊆ N \ i | S | ! | N | − | S | − 1 ! | N | ! f S ∪ i − f S , (12)

f x ≈ ϕ 0 + ∑ i = 1 M ϕ i , (13)

2.2.5 Class activation mapping

Class Activation Mapping (CAM), introduced by Zhou et al. (2016), is a landmark post hoc visualization method for interpreting class-specific evidence in convolutional neural networks (CNNs). CAM assumes a modified classifier head in which terminal fully connected layers are replaced by a Global Average Pooling (GAP) operation, thereby preserving a direct correspondence between high-level feature maps and class logits. Under this architecture, class-discriminative localization can be obtained through a linear projection of convolutional activations.

Formally, let f k ( x , y ) denote the activation of channel k at spatial position ( x , y ) in the final convolutional layer, and let w k c denote the learned weight linking channel k to class c. The class activation map M c ( x , y ) is defined as:

M c x , y = ∑ k w k c f k x , y . (14)

Equation 14 shows that CAM performs a weighted superposition of spatial feature responses, where larger values indicate regions contributing more strongly to class c. After upsampling to the input resolution, this map provides a spatially explicit estimate of model attention, enabling localization of discriminative image patterns and improving interpretability of CNN decision processes.

2.2.6 Gradient-weighted CAM

To remove the architectural constraint of original CAM on GAP (Selvaraju et al., 2020), proposed Gradient-Weighted Class Activation Mapping (Grad-CAM), which generalizes class-discriminative localization to a broad range of CNN-based models. The key idea is to derive class-specific channel importance from gradients and then project these importance weights back onto convolutional feature maps.

Let A k denote the k-th feature map at a selected convolutional layer, y c denote the score for target class c, and Z denote the number of spatial locations in A k. Grad-CAM computes the channel weight α k c as:

α k c = 1 Z ∑ i , j ∂ y c ∂ A i , j k . (15)

Equation 15 estimates the average sensitivity of class score y c to channel k by spatially averaging partial derivatives, such that channels with stronger positive influence receive larger weights. The class activation map is then obtained by a weighted linear combination of feature maps followed by rectification:

L Grad - CAM c = R e L U ∑ k α k c A k . (16)

As expressed in Equation 16, the ReLU operation suppresses negative evidence and retains only positively contributing regions for class c, yielding a coarse but class-discriminative localization map. Although Grad-CAM is widely adopted in medical image interpretation, its spatial precision is limited by the low resolution of deep convolutional feature maps.

To improve fine-grained visualization, Guided Grad-CAM combines the semantic localization of Grad-CAM with the high-frequency detail preserved by Guided Backpropagation via element-wise fusion:

L Guided c = L Grad - CAM c ⊙ R GuidedBackprop . (17)

Equation 17 integrates coarse class-relevant localization ( L Grad - CAM c ) with fine-grained gradient structure ( R GuidedBackprop ), producing a refined map with improved anatomical detail while preserving class specificity. Building on this paradigm, subsequent variants further improve explanatory quality: LayerCAM (Jiang et al., 2021) enhances spatial granularity via pixel-level weighting, whereas Grad-CAM++ (Chattopadhay et al., 2018) leverages higher-order derivatives to better handle multiple target instances. Collectively, these extensions have strengthened the robustness and clinical interpretability of gradient-based visual explanations in complex cardiovascular imaging tasks.

2.2.7 SmoothGrad

To reduce high-frequency noise and improve the visual stability of gradient-based attributions, Smilkov et al. (2017) proposed SmoothGrad, a stochastic smoothing strategy that estimates saliency under input perturbations. Rather than interpreting a single gradient map from the original input, SmoothGrad samples multiple noisy realizations and averages their attribution responses, thereby suppressing variance-driven artifacts. Formally, given an input image x and a base attribution operator M ( x ) (e.g., vanilla input gradients), the smoothed attribution map M ̂ ( x ) is defined as:where n denotes the number of noisy samples and N ( 0 , σ 2 ) denotes additive Gaussian noise with zero mean and variance σ 2. Equation 18 can be interpreted as a Monte Carlo approximation to the expected attribution under a local noise distribution around x. Consequently, SmoothGrad attenuates spurious pixel-level fluctuations while preserving structurally consistent evidence regions, yielding saliency maps with improved coherence and signal-to-noise characteristics.

M ̂ x = 1 n ∑ i = 1 n M x + N 0 , σ 2 , (18)

Notably, SmoothGrad is an attribution enhancement framework rather than an independent explanation model; its primary role is to improve robustness and interpretive clarity when coupled with existing gradient-based methods in high-resolution medical imaging analysis.

3 Taxonomy of XAI methods

Given the heterogeneity of evaluation dimensions and perspectives adopted in existing studies on XAI, this paper categorizes current XAI methods into four groups according to different classification criteria, as illustrated in Figure 3.

• Intrinsic vs. post hoc interpretability: Based on when explanations are generated relative to model inference, XAI methods can be classified as intrinsically interpretable models (Colaner, 2022) or post hoc interpretability methods (Wei et al., 2024). Intrinsically interpretable models possess inherent transparency due to their architectural design, enabling explanations to be produced concurrently with predictions. Classical models such as decision trees (Wan et al., 2020) and linear regression (Kuchibhotla et al., 2019) fall into this category, as their decision logic is directly understandable and incurs no additional computational cost. In contrast, post hoc interpretability methods analyze trained “black box” models—most notably deep neural networks—using external techniques after model training. By decoupling interpretability from model structure, post hoc approaches are applicable to complex DL architectures and therefore represent the predominant direction in contemporary XAI research.
• Model-agnostic vs. model-specific explanations: According to their dependence on model architecture, XAI methods can be divided into model-agnostic (Lundberg and Lee, 2017) and model-specific explanations (Adadi and Berrada, 2018). Model-agnostic approaches rely solely on input–output behavior and are thus applicable to a wide range of “black box” models. Their sensitivity to input perturbations facilitates the identification of features or regions that influence predictions. However, the lack of access to internal model representations limits the depth of insight they can provide. In contrast, model-specific methods are tailored to particular architectures and exploit internal parameters such as gradients or weights to generate explanations. While these approaches often yield more faithful interpretations of model behavior, their strong architectural dependence reduces their generalizability; in practice, some methods exhibit hybrid properties depending on implementation choices.
• Global vs. local explanations: Based on the scope of explanation, XAI methods can be categorized into global and local explanations (Van Der Velden et al., 2022). Global explanations aim to characterize a model’s overall decision-making behavior by analyzing feature representations learned across the entire dataset, for example, through dataset-level feature importance or pattern visualization. Such analyses are valuable for model auditing, bias detection, and generalization assessment. Local explanations, by contrast, focus on individual predictions and quantify the contribution of specific features to a single instance. In medical applications, local explanations are particularly important, as they support case-level interpretation and patient-specific decision analysis and more directly align with clinician reasoning.
• Textual, visual, and concept-based explanations: Based on the form of explanation output, XAI methods can be grouped into textual explanations (Vinyals et al., 2015), visual explanations, and concept-based explanations (Hoffer and Ailon, 2015). Textual explanations map deep feature representations to natural language descriptions, explicitly articulating model reasoning and attribution. Visual explanations emphasize spatial interpretability, typically using saliency or attention maps to highlight regions that most strongly influence predictions. Concept-based explanations extend beyond low-level features by identifying high-level semantic concepts with explicit physical or pathological meaning, thereby aligning model explanations more closely with human cognition. This output-oriented taxonomy provides practical guidance for selecting appropriate explanation strategies in specific clinical and application contexts.

Based on this framework, Table 1 classifies the XAI methods reviewed in Section 2 and summarizes their clinical applications across different imaging modalities.

TABLE 1. Classification of different XAI methods and their applications in cardiovascular imaging.

4 Clinical applications of XAI in cardiovascular imaging

To enhance the transparency, reliability, and clinical acceptance of DL models in cardiovascular imaging, XAI techniques have been increasingly integrated across multiple imaging modalities. Recent studies have demonstrated that XAI can provide clinically meaningful insights by revealing model attention, feature attribution, and decision rationales, thereby facilitating model validation and trust building. In this section, we systematically review representative applications of XAI in four major cardiovascular imaging modalities: CCTA, CMR, Echo, and CXR, highlighting how explainability methods are employed to support disease diagnosis, functional assessment, risk stratification, and clinical decision-making in diverse cardiovascular scenarios.

4.1 CCTA

CCTA is primarily used to evaluate coronary artery anatomy, quantify the degree of luminal stenosis, and characterize atherosclerotic plaque composition, making it a key modality for the diagnosis and risk stratification of coronary artery disease (Gulati et al., 2021). To improve the transparency and clinical reliability of DL models in CCTA analysis, numerous studies have incorporated explainability-driven visualization techniques.

In coronary lesion assessment, Penso et al. (2023) proposed a ConvMixer-based framework integrated with Grad-CAM to enable CAD-RADS compliant stenosis classification while maintaining precise localization of pathologic regions. In parallel, Huo et al. (2019) introduced a three-dimensional Grad-CAM strategy to interrogate the decision process of AID-Net within volumetric CCTA, yielding class-discriminative heatmaps that delineate coronary calcifications. Extending this paradigm, De Vos et al. (2019) integrated saliency mapping with deconvolutional networks for coronary artery calcium burden prediction in CCTA, enabling concurrent calcium quantification and localization of plaque territories with maximal decision contribution, thereby supporting the clinical plausibility and anatomical validity of model attention patterns. In automated scoring workflows, Gerbasi et al. (2024) proposed a multi-axis Vision Transformer–based framework for CAD-RADS scoring on CCTA, in which visualization-driven explanations substantially improved transparency and interpretability across the scoring pipeline. Likewise, Wang et al. (2021) demonstrated that SOSPCNN can identify congenital heart diseases, including Tetralogy of Fallot, with Grad-CAM derived heatmaps exhibiting high concordance with radiologist annotations. Moreover, Lo Iacono et al. (2023) introduced a SHAP based radiomics framework to differentiate cardiac amyloidosis from aortic stenosis on CCTA, with attribution analysis identifying texture-derived descriptors as the most influential discriminative features.

4.2 CMR

Owing to its multiparametric and multisequence imaging capabilities, CMR is widely regarded as the gold standard for assessing left and right ventricular structure and function, and it plays a pivotal role in myocardial tissue characterization and the differential diagnosis of complex cardiovascular diseases (McDonagh et al., 2021).

For qualitative validation of pathological tissue characterization, Paciorek et al. (2024) employed Grad-CAM to demonstrate that their proposed model exhibited strong attention to myocardial regions with edema or fibrosis on T1 mapping and phase-sensitive inversion recovery sequences. This finding provides diagnostic-level evidence that the model captures specific histopathological alterations rather than merely fitting data patterns. Chauhan et al. (2022) compared the performance of ML and DL approaches in classifying anatomical structures on CMR and, through CAM-based analysis, showed that DL models exhibit superior capability in recognizing complex anatomical configurations. This advantage renders DL-based decision-making more anatomically consistent when diagnosing diseases such as cardiomyopathies that involve global morphological remodeling.

In the evaluation of atrial and ventricular structures, Pérez-Pelegrí et al. (2021) estimated left ventricular end-diastolic volume under a weakly supervised learning framework and generated segmentation masks using CAM. This approach allows clinicians not only to access quantitative volume measurements but also to visually verify whether the endocardial boundaries were accurately delineated, thereby reducing diagnostic bias caused by segmentation errors. Tong et al. (2019) further demonstrated that attention modules enhance the network’s focus on the ventricles and myocardium, effectively mimicking the “visual attention” mechanism employed by clinicians during image interpretation—namely, selectively concentrating on myocardial wall motion and chamber morphology that are most critical for functional assessment.

For the differentiation of complex diseases such as myocarditis, DL models are capable of uncovering subtle latent features that are difficult to discern visually. Cau et al. (2023) used SHAP analysis to identify left atrial conduction strain and strain rate as key discriminative features for distinguishing Takotsubo cardiomyopathy from acute myocarditis. Wang et al. (2024) combined Guided Grad-CAM with Shapley value analysis in CMR-based studies to intuitively visualize the distribution of decision weights across different cardiac anatomical regions and multimodal imaging sequences. Zhang et al. (2025) integrated CMR and CCTA data to predict microvascular obstruction in patients with ST-elevation myocardial infarction (STEMI) following percutaneous coronary intervention (PCI), with SHAP-based analysis revealing the importance of angiography-derived microvascular resistance and other clinical parameters, thereby enhancing both model interpretability and clinical insight. In addition, Sufian et al. (2024) incorporated fairness constraints into interpretable DL models using SHAP and LIME for cardiovascular risk prediction based on 3D/4D CMR and structured data, effectively mitigating algorithmic bias across different population groups while maintaining high segmentation and classification performance and substantially improving diagnostic fairness and clinical trustworthiness.

4.3 Echo

Echo is a fundamental imaging modality for assessing cardiac anatomy, valvular abnormalities, and hemodynamic status (Szabo et al., 2022). With the increasing adoption of DL techniques, XAI methods have been widely applied to validate the clinical plausibility of model decisions, spanning applications from adult disease diagnosis to the localization of subtle anatomical structures in pediatric imaging. Chao et al. (2024) developed an apical four-chamber view–based interpretation model that demonstrated strong performance in differentiating constrictive pericarditis from cardiac amyloidosis, a clinically challenging task. Grad-CAM–derived heatmaps clearly indicated that distinctive interventricular septal motion patterns constituted the key discriminative feature, which is highly consistent with known pathophysiological mechanisms and provides strong validation of the model’s clinical reasoning.

In the analysis of valvular morphology and anatomical segmentation, Vafaeezadeh et al. (2022) focused on automating Carpentier functional classification of the mitral valve and employed Grad-CAM to reveal the anatomical basis underlying the model’s complex morphological classifications. Wang et al. (2022) proposed a domain knowledge–integrated U-Net architecture for two-dimensional echocardiographic segmentation, where heatmap-based explanations confirmed the model’s ability to focus on specific anatomical boundaries, such as the atria and ventricles, thereby validating the effectiveness of attention mechanisms. Lagopoulos (2022) conducted an in-depth SHAP analysis of the EchoTree-Geom model and found that predictions were primarily driven by left ventricular volume during the mid-cardiac cycle and the ShiftAVG feature, which reflects ventricular contraction and relaxation rates. These dynamic indices were identified as key determinants for estimating ejection fraction. Ghorbani et al. (2020) introduced the EchoNet framework, which accurately identifies cardiac structures such as pacemaker leads, left atrial enlargement, and left ventricular hypertrophy, while also predicting ventricular volumes, ejection fraction, age, sex, and other systemic phenotypes. By combining SmoothGrad with gradient-based attribution methods, the interpretability analysis demonstrated that the model consistently attends to clinically relevant cardiac structures and highlights regions with potential biological significance when predicting complex phenotypes.

Given the small size and high anatomical variability of pediatric and fetal cardiac structures, XAI is primarily leveraged to assist in abnormality localization and to enhance clinician confidence in screening results. In prenatal screening, Nurmaini et al. (2022) combined DenseNet with Guided Grad-CAM to not only detect congenital heart disease effectively but also precisely localize high-impact regions responsible for abnormal classifications, providing intuitive support for clinical decision-making. Day et al. (2024) applied Grad-CAM visualization to evaluate how AI assists in diagnosing fetal atrioventricular septal defects, with the goal of improving interactive trust between clinicians and AI systems. Ragnarsdottir et al. (2024) used multiview video data to predict neonatal pulmonary hypertension and employed Grad-CAM to confirm the model’s reliance on key anatomical landmarks. Lee et al. (2022) extended CAM techniques to pediatric imaging and successfully highlighted pathological regions associated with Kawasaki disease and pneumonia in short-axis views, further validating the effectiveness of model feature extraction.

4.4 CXR

CXR imaging is primarily used to evaluate cardiac size, the morphology of great vessels, and the positioning of implanted devices (McDonagh et al., 2021). DL techniques have demonstrated the ability to extract latent cardiovascular disease–related information from chest radiographs that is often imperceptible to the human eye. For global cardiac function assessment (Ueda et al., 2023), leveraged large multicenter datasets of posteroanterior chest radiographs to predict atrial fibrillation. By employing CAM and Guided Backpropagation, they visualized anatomical regions associated with arrhythmia-related features, thereby validating the model’s effectiveness in quantifying left ventricular ejection fraction and identifying valvular heart disease. Furthermore (Kusunose et al., 2022), extended this approach to the hemodynamic domain by predicting exercise-induced pulmonary hypertension, providing strong evidence for the potential of explainable DL in early, noninvasive cardiovascular screening and interpretable diagnosis.

5 XAI evaluation paradigms in cardiovascular imaging

Current evaluation frameworks for XAI largely follow the three-level evaluation pyramid proposed by Doshi-Velez and Kim (2017), which stratifies assessment into functionally grounded evaluation, human-grounded evaluation, and application-grounded expert evaluation. As the evaluation level ascends, the degree of realism and clinical relevance increases, but so do the associated implementation costs and experimental complexity. This framework has become the conceptual foundation for assessing XAI methods across high-stakes domains, including medical imaging.

Within this paradigm, existing studies have gradually converged toward two complementary evaluation pathways: computation-centered evaluation and human-centered evaluation. These two perspectives address different but interdependent aspects of explainability and are often jointly adopted in recent cardiovascular imaging research.

5.1 Computation-centered evaluation

Computation-centered evaluation quantitatively assesses the intrinsic reliability of explanation methods without direct human involvement. A fundamental criterion in this category is faithfulness (Lamprou et al., 2024), which measures the extent to which an explanation accurately reflects the true decision-making behavior of the underlying model. Faithfulness is commonly evaluated using perturbation-based strategies, where features identified as important are systematically removed or altered and the resulting changes in model predictions are measured (Lago et al., 2026). These perturbation-based evaluations provide an empirical basis for determining whether highlighted regions genuinely drive model decisions in imaging tasks.

Another critical dimension is stability or robustness, which examines whether explanations remain consistent under small, semantically irrelevant perturbations of the input or variations in model parameters (Skliarov et al., 2025). For instance, robustness assessments using similarity metrics such as structural similarity index, convolution-density measure (Repetto et al., 2026), or pixel-wise distances (Stodt et al., 2023) have been proposed to quantify how explanation maps change under noise or data corruption. Such robustness analyses are particularly important in medical imaging, where real-world perturbations can affect both model outputs and explanation consistency. Studies have shown that widely used saliency-based methods, including Grad-CAM variants, can exhibit substantial sensitivity to input perturbations, raising concerns about their reliability for clinical interpretation (Venkatesh et al., 2024).

5.2 Human-centered evaluation

In contrast, human-centered evaluation foregrounds the subjective utility of explanations for end-users, with a particular emphasis on clinicians (Kim et al., 2024). This evaluative pathway typically necessitates structured user studies or controlled experiments in which medical experts appraise explanations regarding their trustworthiness, clinical relevance, and efficacy in decision support (Kim et al., 2024). Such assessments are indispensable within cardiovascular imaging, necessitating that explanations align precisely with established anatomical frameworks and diagnostic workflows.

Concurrently, acknowledging the idiosyncratic constraints of the medical field, a recent guideline-based evaluation paradigm has emerged that integrates proxy metrics with domain expertise. For instance, Jin et al. (2023) proposed a comprehensive framework of clinical XAI guidelines encompassing five optimization criteria: interpretability, clinical relevance, truthfulness, information plausibility, and computational efficiency. By systematically interrogating common heatmap-based explanation techniques across multimodal medical imaging tasks, they identified notable shortcomings in truthfulness and information plausibility, thereby validating the efficacy of these guidelines in supporting the development of clinically viable XAI systems.

6 Discussion

In recent years, the application of XAI in cardiovascular imaging has advanced steadily. This review systematically summarizes the recent progress and current applications of XAI methods in cardiovascular image analysis, with an emphasis on two major categories—perturbation-based and backpropagation-based approaches—across multiple imaging modalities, including CCTA, CMR, and Echocardiography. These methods have been applied to clinically relevant tasks such as coronary artery disease assessment, myocardial pathology differentiation, congenital heart disease diagnosis, and cardiac function analysis. By providing feature importance rankings, saliency maps, and concept-level explanations, XAI can improve clinicians’ understanding of model decision mechanisms, thereby supporting transparency, trustworthiness, and diagnostic reliability.

Despite these developments, XAI remains at an early stage of adoption in the medical imaging domain. Methodological limitations and the lack of standardized evaluation frameworks continue to hinder broader clinical translation. The following sections discuss key challenges from both methodological and evaluation perspectives.

6.1 Methodological limitations of current XAI approaches

From a methodological standpoint, existing XAI techniques exhibit trade-offs between generality, efficiency, and clinical relevance. Perturbation-based methods are model-agnostic and intuitive, enabling explanations without access to internal model structures. However, they are often computationally expensive and inefficient for high-resolution medical imaging tasks. Widely used approaches such as SHAP suffer from slow inference speed, limiting scalability to large imaging datasets. Moreover, SHAP primarily emphasizes global feature attribution, whereas clinical decision-making often requires localized, case-specific explanations to support personalized diagnosis.

Backpropagation-based methods generate explanations efficiently by directly leveraging gradients and intermediate feature representations. Nevertheless, these methods are typically dependent on specific network architectures and are sensitive to gradient noise and local perturbations. The most commonly used Grad-CAM technique is designed mainly for convolutional neural networks and shows limited adaptability to emerging architectures such as Transformers. In addition, Grad-CAM explanations are generally coarse-grained, highlighting broad regions rather than the fine-grained, feature-level cues required for detailed clinical interpretation.

Overall, no single XAI method achieves optimal performance across generalizability, stability, and clinical relevance. In practice, combining complementary explanation strategies is often necessary to improve robustness and interpretability.

6.2 Challenges in evaluating XAI for cardiovascular imaging

Beyond methodological limitations, the absence of robust and standardized evaluation frameworks remains a major barrier to clinical translation of XAI in cardiovascular imaging. In many studies, explanation quality is assessed primarily through qualitative expert inspection, which, although clinically meaningful, is inherently subjective and sensitive to inter-reader variability, prior experience, and task context. Such dependence on manual judgment increases clinical workload and limits scalability in large-cohort validation settings.

Generalizability across modalities and centers represents a major challenge for XAI evaluation in cardiovascular imaging. Explanations developed under one setting often degrade when transferred to other settings because acquisition protocols, scanner vendors, reconstruction pipelines, contrast timing, and preprocessing steps can alter image statistics and feature representations. Heterogeneity in patient case-mix, disease prevalence, and annotation conventions further amplifies this transfer gap, making explanation patterns less stable across institutions. In addition, weak supervision can distort saliency alignment metrics by rewarding coarse correlations rather than true lesion-level attribution. As a result, apparent interpretability performance may be overestimated under distribution shift, limiting confidence in real-world deployment.

A second challenge concerns metric heterogeneity. Existing studies employ diverse evaluation criteria, including overlap-based localization scores, deletion/insertion analyses, sensitivity tests, and expert reader ratings, without a harmonized benchmark framework. Consequently, findings are difficult to reproduce and compare across studies. In addition, prospective evidence demonstrating that XAI improves downstream clinical outcomes remains limited.

7 Future directions

To enable real-world clinical translation, future XAI research must move beyond post hoc visualization toward explanation frameworks structurally aligned with clinical reasoning. While heatmap-based approaches effectively capture spatial correlations, cardiovascular diagnosis is an integrative, hypothesis-driven process requiring the synthesis of anatomical, functional, and contextual data. Recent advances in explainable fusion modeling further support this shift, suggesting that interpretable AI should move beyond isolated feature attribution toward the integration of complementary information within a unified reasoning framework (Dwivedia et al., 2026). Consequently, next-generation XAI systems should not only explain predictions but also model the reasoning pathways that generate them.

Multimodal XAI constitutes a central direction in this evolution. Recent efforts have explored fusion-oriented explainable frameworks for cardiovascular prediction. For instance, Wani et al. (2025) proposed a hybrid CNN-LightGBM framework that integrates predictive modeling with SHAP-based local and global explanations, demonstrating how fused learning pipelines enhance both predictive performance and interpretability. Although this approach was designed for risk assessment rather than medical imaging, it marks a vital conceptual shift: explainable cardiovascular AI is transitioning toward integrated frameworks that account for heterogeneous evidence rather than isolated model outputs.

Cardiovascular decision-making routinely synthesizes coronary morphology from CCTA, myocardial tissue characterization from CMR, and functional assessment from echocardiography, alongside non-imaging evidence such as laboratory biomarkers, medication trajectories, and longitudinal clinical narratives. Most existing frameworks process these streams in isolation, producing parallel outputs rather than an integrated interpretative account. Robust multimodal explainability should instead address three foundational dimensions: quantifying modality contribution to clarify individual data source influence; modeling cross-modal interactions to characterize feature synergy; and delineating hierarchical reasoning transparency to transform low-level signals into diagnostic conclusions.

Looking ahead, multimodal XAI must explicitly characterize how heterogeneous data sources jointly influence a decision. Rather than merely highlighting image regions, future methods should delineate the influence of CCTA findings, functional measurements, and clinical factors. This necessitates models that disentangle modality-specific contributions and formalize how these elements combine to yield a diagnosis, thereby aligning explanations with established clinical reasoning. Furthermore, such systems should account for modality-level uncertainty; when a data source is degraded, explanations should reflect a strategic shift in reliance toward remaining evidence. In high-risk applications, this stability under data variation is as critical as raw predictive performance.

Beyond integration, clinically meaningful XAI requires embedding medical knowledge into model design. Incorporating clinical guidelines, physiological principles, or validated biomarkers into the learning process fosters consistency between model outputs and expert reasoning. Emerging neurosymbolic frameworks suggest that future explainability may benefit from combining multimodal signal fusion with causal graph discovery, symbolic rule construction, and patient-specific adaptation. For example, Adnan et al. (2025) introduced a neurosymbolic digital twin framework that illustrates how explainability evolves from simple saliency display toward structured, clinically actionable reasoning. By organizing relationships among heterogeneous variables into transparent causal structures, such approaches move XAI toward mechanism-oriented interpretation and guideline-compatible decision support. Their emphasis on privacy-preserving federated learning and longitudinal state updating highlights a key translational goal: future cardiovascular XAI must be personalized, interoperable across institutions, and seamlessly aligned with real-world clinical workflows.

In parallel, attention mechanisms should be systematically integrated to model the dynamic weighting of features across spatial, modal, and temporal contexts. When constrained by clinical priors, these distributions provide mechanistic cues regarding how a model prioritizes diagnostically relevant evidence, bridging the gap between computational inference and expert interpretation. In this manner, interpretability becomes an intrinsic model property rather than an appended visualization. Finally, evaluation must prioritize demonstrable clinical impact. Explanations must be mathematically faithful while simultaneously reducing inter-reader variability and enhancing decision confidence. Through shared benchmarks, cross-institutional validation, and alignment with regulatory standards, multimodal XAI can mature from a theoretical concept into a reliable, collaborative tool in cardiovascular care.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Ibrahim Karabayir, Wake Forest University, United States

Reviewed by: Niyaz Ahmad Wani, Manipal University Jaipur, India