A quantitative framework for explainability assessment in medical imaging

1 Introduction

In recent years Deep Neural Networks (DNNs) have revolutionized computer-assisted clinical diagnosis, especially in analyzing histopathological images (Abdullakutty et al., 2024). Explainable Artificial Intelligence (XAI) provides a framework to address this challenge, offering methodologies to explain, interpret, and assess the reliability of AI-generated outcomes.

In the medical field, the need for transparency goes beyond being optional; it forms a critical foundation for maintaining both safety and clarity (Frasca et al., 2024). Local interpretability—defined as the capacity to explain decisions at the level of specific instances—holds particular significance for histological classification models, wherein the identification of regions of interest (ROIs) and the morphological coherence of explanations are paramount to the diagnostic procedure.

In breast cancer imaging, false negatives represent one of the most critical failure modes (Abdullakutty et al., 2024), as they correspond to missed malignant patterns that may delay diagnosis. Recent evidence suggests that such errors are often associated with visually subtle or spatially fragmented features (Mall et al., 2018), which are difficult to capture by confidence-based prediction alone. In this context, explainability can play a direct role in detection, by revealing whether a prediction is supported by coherent and clinically meaningful visual evidence.

The proposed framework addresses this challenge through a structured integration of well-established XAI techniques [SHapley Additive exPlanations (SHAP), Local Interpretable Model-agnostic Explanations (LIME), and Gradient-weighted Class Activation Mapping (Grad-CAM)] and introduces a composite index, named ExpiScore, designed to quantify the consistency among different explanations and to assess the overall quality of attention maps. The proposed framework is further enhanced by a trust module that captures interdependencies between prediction, explainability, and uncertainty, with the purpose of facilitating triage decision-making in high-stakes clinical contexts.

1.1 Related work and positioning

A growing body of work has applied XAI methods to histopathological classification. Gradient-based approaches such as Grad-CAM (Selvaraju et al., 2017) and its variants (GradCAM++, Score-CAM) have been used to localize diagnostically relevant tissue regions and incorporated into human-in-the-loop review workflows (Noorbakhsh et al., 2020; Zhou et al., 2022). Attribution-based methods—principally SHAP (Lundberg and Lee, 2017; Ranjana and Meenakshi, 2025)—provide feature-level importance scores grounded in cooperative game theory. Perturbation-based methods such as LIME (Ribeiro et al., 2016) offer local surrogate approximations of model behavior. Recent work has begun combining multiple XAI methods for robustness: Bianchi et al. (2024) propose human-in-the-loop post-hoc explainability with CNN visualisations; Mondol et al. (2025) introduce graph-based tissue examination integrating LIME and SHAP; Wakili et al. (2022) use Grad-CAM variants on ensemble classifiers.

Table 1 summarizes eight representative prior studies and positions the present work along six dimensions. No prior framework: (i) quantitatively aggregates the outputs of multiple XAI methods into a single reliability metric with formal properties; (ii) uses that metric to drive automated clinical triage; or (iii) trains a trust model on XAI-derived features to detect cases where CNN confidence and explanatory coherence are structurally inconsistent. Addressing this three-fold gap is the primary objective of the present work.

Table 1. Related-work comparison: representative XAI methods applied to histopathological classification.

Columns: (1) Reference; (2) Dataset; (3) Classification Task; (4) Backbone/Model Family; (5) XAI Method(s); (6) Explanation-Evaluation Strategy; (7) Clinical Workflow Component; (8) Key Differentiator vs. the present work.

Principal contributions of this work.

• ExpiScore: a composite interpretability index with formal properties (boundedness, monotonicity, linearity in λ) that quantitatively aggregates Grad-CAM, SHAP, and LIME outputs into a single reliability metric (Section 2.3.8).
• Dual-mode clinical framework: a strict separation between a fully-automatic mode (meta-model, evaluated on the validation split) and a human-in-the-loop mode (ExpiScore routing, evaluated on the test set), preventing cross-mode result conflation.
• Trust meta-model: a logistic regression + decision tree trained on 42 XAI-derived features and CNN confidence that transforms explanation signals into interpretable decision rules and enables automated Accept/Review/Defer triage.

2 Methods

2.1 Framework design and overall pipeline

This study proposes an explainability-oriented deep learning framework for binary histopathological image classification on the BreaKHis dataset (Spanhol et al., 2016). The pipeline integrates balancing, stain normalization, convolutional neural network (CNN) training, post-hoc explainability methods, and a composite interpretability score, named ExpiScore, to support both clinical review and optional model refinement.

The workflow is composed of four main stages. First, breast histopathology images are preprocessed through color normalization in order to reduce stain variability across samples. Second, a CNN-based classifier is trained to distinguish between the target diagnostic classes. Third, the trained model is interpreted using three complementary explainability techniques, namely Grad-CAM, LIME, and SHAP. Fourth, the results from these methods are integrated into a unified interpretability index that guides downstream processes, such as clinical triage, corrective meta-classification, and curriculum-based optional retraining.

This design is intended to move beyond predictive performance alone by explicitly incorporating interpretability as an evaluative and operational component of the classification pipeline.

2.2 Dataset

The BreaKHis dataset ¹ included in this work was published by Spanhol et al. (2016). The data set includes 7,909 histopathological images of 82 women diagnosed with breast cancer.

To preserve the original organizational structure and molecular composition, a pathologist captured each image from a patient's breast tissue portion via surgical biopsy. Subsequently, the pictures were obtained using hematoxylin and eosin staining. The pathologists then assigned the actual class title to each image based on their microscopic observations. All histopathology images of breast cancer are RGB micrographs with dimensions of 700 × 460 pixels. The dataset of histological pictures of breast cancer consisted of four distinct sub-datasets, corresponding to objective lenses with magnifications of 40X, 100X, 200X, and 400X. All sub-datasets are categorized into benign and malignant cancers. Consequently, benign and malignant tumors comprise four distinct categories. Benign neoplasms include Adenosis (A), Fibroadenoma (F), Phyllodes Tumor (PT) and Tubular Adenoma (TA). Malignant neoplasms include ductal carcinoma (DC), Lobular Carcinoma (LC), Mucinous Carcinoma (MC) and Papillary Carcinoma (PC). Table 2 presents sample descriptions for the BreaKHis dataset.

Table 2. Summary of the BreaKHis dataset used in this study.

2.3 Methods

This framework was designed for binary classification so, samples are grouped into two classes of histopathological images, benign and malignant, using the BreaKHis dataset, split into 70% training, 15% validation, 15% test. The data splitting and augmentation protocol proceeds in four strictly ordered steps, designed to prevent any form of cross-split information leakage:

• Patient-level stratification. The 82 patient identifiers of BreaKHis (Spanhol et al., 2016) are extracted from the filename convention (SOB_[class]_[subtype]-[year]-[patientID]-[mag]-[seq]).² Patients are assigned to train / validation / test using stratified random sampling (seed = 42, stratified by benign/malignant label) at the patient level, so that all images from the same patient reside exclusively in one split. The nominal target is a 70 / 15 / 15 % split; because patients are indivisible units, the realized counts approximate—but do not exactly match—the target fractions (70% of 82 ≈57, 15%≈12). Resulting counts: training ≈55 patients / 5,557 original images; validation ≈13 patients / 1,165 original images; test = 13 patients / 1,187 original images. This guarantees that the reported test-set accuracy (0.9916) reflects genuine generalization to unseen patients. The complete patient-ID assignment is provided in Supplementary Table S2.
• Stain normalization. Applied independently per image via Macenko optical-density decomposition (stateless; no cross-image information sharing). Applied identically to all three splits.
• Resizing. Bilinear interpolation to 224 × 224 pixels, applied identically to all three splits (stateless).
• Data augmentation. Applied exclusively to training images at batch-loading time (online augmentation via PyTorch transforms.Compose), never persisted to disk. The augmentation pipeline consists of: random horizontal flip (p = 0.5); random vertical flip (p = 0.5); random rotation up to ±15°; random scale jitter ±10%. Validation and test data loaders use no augmentation transforms. No augmented copy of any validation or test image is ever present in the training pipeline.

Primary model. The reference model used for all primary results reported in this paper (accuracy, F1, XAI analysis, ExpiScore, triage) is the distilled DenseNet-121 student trained under supervision of an InceptionV3 teacher via knowledge distillation [Distill_Inc_to_Den; test accuracy = 0.9916, test F1 = 0.9939, Area Under the receiver-operating-characteristic Curve (AUC) = 0.9989, validation accuracy = 0.9874]. The training protocol was made fully reproducible with fixed random seeds, standardized patient-level stratification, and a teacher trained on the same data partition as the student; all reported metrics reflect this reproduced pipeline. All other architectures (EfficientNet, ResNet, alternative ensembles) were evaluated during the ablation phase and serve as comparison points; they are not the basis for the reported operational results.

The framework's pipeline (Figure 1) starts from normalization, stratification, and augmentation procedures, followed by the training of pre-trained convolutional neural networks (CNNs) that are enhanced through fine-tuning. In a multi-model training configuration, we evaluated several backbone architectures, including DenseNet, InceptionV3, EfficientNet, and ResNet, employing cross-validation techniques. For each input image, we generate interpretative heatmaps utilizing three distinct methodologies: SHAP Lundberg and Lee, (2017), LIME (Ribeiro et al., 2016), and Grad-CAM (Selvaraju et al., 2017). The resulting maps are combined in a fusion module that computes their overlap and spatial consistency. From these, the ExpiScore is derived: a composed index that assesses consistency, saliency, and compactness, and can optionally incorporate overlap with clinically annotated regions. Morphological features extracted from the heatmaps (such as area, eccentricity, distribution, and contrast) are further analyzed to identify interpretable pathological patterns. This information feeds into a confidence model that estimates the probability of error and allows classifying cases into three operational categories: automatically accept, review, or defer.

2.3.1 Pre-processing and stain normalization

The image patches undergo artifact detection (Lago et al., 2026), stain decomposition, and illumination correction (Mondol et al., 2025). Quality control ensures the exclusion of low-content and background regions. Histopathological images often present significant variability in color distribution due to differences in staining protocols, slide preparation, and imaging devices. Such variability may negatively affect both classification performance and the consistency of explanation maps generated by explainability techniques. To address this issue, stain normalization is applied as a preprocessing step. Two alternative normalization approaches are considered in this study: Macenko and Reinhard normalization strategies (Xu et al., 2025a) are compared to assess robustness. Both contribute differently to morphological preservation and saliency stability. Macenko normalization operates in optical density space and estimates stain vectors that allow the color distribution of each image to be aligned with a reference stain appearance. Reinhard normalization transfers the color statistics of an image to match those of a reference image in a perceptually meaningful color space. Both methods aim to reduce inter-slide staining variability while preserving relevant tissue morphology. A visual comparison of the original image with Macenko and Reinhard normalizations is shown in Figure 2. After normalization, images are resized to 224 × 224 pixels (bilinear interpolation). Data augmentation (random horizontal/vertical flips, rotation ±15°, scale jitter ±10%) is applied exclusively to training images at batch-loading time, strictly after patient-level stratification; validation and test images are never augmented (see Section 2.3 for the full protocol).

2.3.2 CNN architectures and training procedure

Two convolutional neural network architectures are used as backbone classifiers in the proposed framework: InceptionV3 and DenseNet121. These architectures are widely adopted in medical imaging applications due to their strong feature extraction capabilities and their compatibility with transfer learning strategies. InceptionV3 uses factorized convolutional modules that capture multi-scale spatial information efficiently. DenseNet121 introduces dense connectivity between layers, which improves gradient propagation and encourages feature reuse throughout the network. Both models are initialized using pre-trained weights and adapted to the binary classification task by replacing the final classification layers. Training is performed using binary cross-entropy loss, and optimization can be carried out using a gradient-based optimizer such as Adam. The training process typically involves two stages. In the first stage, the pre-trained backbone is partially frozen and only the classification head is trained. In the second stage, selected convolutional layers are unfrozen and fine-tuned to adapt the model to the histopathology domain. Early stopping and validation monitoring are used to avoid overfitting. The final trained CNN model serves as the base predictive component for subsequent explainability analysis.

2.3.3 Explainability analysis

Checkpoint used for all XAI analyses. All reported Grad-CAM, SHAP, and LIME analyses—and all derived ExpiScore values, routing decisions, and meta-model input features—were generated exclusively from a single frozen checkpoint of the distilled DenseNet-121 student model. No XAI analysis was performed on InceptionV3, EfficientNet, ResNet, or the ensemble; those architectures appear only in Table 3 as ablation comparisons for CNN classification performance. InceptionV3 was used exclusively as the teacher during the distillation training phase and was frozen and discarded thereafter.

Table 3. Model performance summary across training runs.

The column Δ (train−val acc) reports the gap between training and validation accuracy; a near-zero value for Distill_Inc_to_Den (Δ < 0.002) confirms effective regularization via distillation. Extended interpretation is provided in Section 3.1. ‡: Distill_Inc_to_Den was reproduced with a fully standardized protocol (fixed seed, patient-level stratification, teacher trained on the same data partition); reported val metrics reflect the reproduced run. ^*Negative gap for EfficientNet (val > train) is attributable to strong data augmentation applied only during training.

To interpret the predictions produced by the CNN models, the framework integrates three explainability techniques that provide complementary perspectives on model behavior: Grad-CAM, LIME, and SHAP. Grad-CAM produces class-specific activation maps by analyzing the gradients flowing through the final convolutional layers of the network. These maps highlight spatial regions of the image that contribute most strongly to the predicted class. In histopathology, Grad-CAM heatmaps help determine whether the model focuses on relevant tissue structures. LIME provides local explanations by approximating the model's behavior around a specific instance using a simpler surrogate model. In the context of images, LIME perturbs superpixels and measures their impact on the prediction, allowing identification of regions that support or contradict the predicted class. SHAP uses principles derived from cooperative game theory to assign contribution scores to input features. In image classification tasks, SHAP produces attribution maps that quantify the positive or negative contribution of different regions of the image to the final prediction. By combining these three techniques, the framework captures complementary forms of explanation: spatial localization, local surrogate modeling, and attribution-based relevance. The whole subpipeline is described in the following steps and in (Figure 3).

• XAI map generation: Grad-CAM, SHAP, and LIME explanations are computed for each histological patch using the following configurations to ensure reproducibility. Grad-CAM: gradients are back-propagated through the last convolutional block of the backbone; the resulting activation map is bilinearly upsampled to 224 × 224. KernelSHAP: N_bg = 100 background samples are drawn uniformly from the training set; N_eval = 1000 coalition samples are used per image. LIME: N_pert = 1000 superpixel perturbations using quickshift segmentation (kernel size = 4, max distance = 200, ratio = 0.2); a ridge surrogate with α = 1.0 is fitted on the perturbed samples.
• Normalization and alignment: Each raw activation map is independently min-max normalized to [0, 1] pixel-wise. Maps are bilinearly resampled to a common resolution of 224 × 224 to ensure spatial alignment.
• Binarization: Each normalized map is thresholded at its own 85th percentile: pixels above the threshold are set to 1 (salient), others to 0. This adaptive per-map threshold balances sensitivity and specificity of the binary mask regardless of absolute activation scale.
• Consensus map: A consensus representation is obtained as the pixelwise average of the three normalized (pre-binarization) maps.
• XAI metric computation: Quantitative explainability metrics (Dice overlap, geometric coherence, structural similarity, morphological descriptors) are extracted from the binarized masks and the continuous maps.
• ExpiScore fusion: Metrics are fused into ExpiScore, a composite interpretability index summarizing overall explainability quality (Section 2.3.8).

2.3.4 CNN architectures

EfficientNet, DenseNet121, Resnet and InceptionV3—with and without pre-training—are evaluated. Their complementary inductive biases motivate ensemble or distilled (teacher/student) integration (Xu et al., 2025b) (Figure 4). Weighted ensembles stabilize predictions and reinforce generalization, forming the supervisory backbone for model distillation. In the distillation, InceptionV3 acts as teacher and DenseNet121 as student. Distillation minimizes cross-entropy and KL divergence, enhancing interpretability and reducing model complexity. We chose these two architectures due to their performance (Table 3). Models were trained and evaluated using two primary metrics: Accuracy and F1-score (macro-averaged). Early stopping was applied using validation F1 as the primary criterion: training halts when validation F1 fails to improve for a defined patience window. In the event of a tie in validation F1 across consecutive epochs, validation Accuracy is used as a secondary tiebreaker to select the best checkpoint.

2.3.5 Explainability module

Gradient-based (Grad-CAM), perturbation-based (LIME), and axiomatic (SHAP) methods form a unified interpretability layer. Feature-level coherence metrics are extracted across magnifications.

For each image, three explainability heatmaps are generated: the Grad-CAM, LIME, and SHAP maps (Figure 5). In addition, two summary files are produced: (i) a feature file containing all quantitative descriptors extracted from the heatmaps, and (ii) a clinical interpretation file describing the key patterns identified by the explanations.

All feature files are merged into a single dataset with 1187 rows (test set) and 42 columns. A preliminary analysis shows that several features are redundant or highly correlated, indicating that dimensionality reduction or feature selection techniques could be applied to reduce the dimensionality of the dataset.

2.3.6 Definition of OGSL (Overlap/Geometric/Structural Level)

To quantify the agreement between different XAI maps, we define the OGSL (Overlap/Geometric/Structural Level) term as a normalized consensus score in [0, 1] that combines three components: overlap, geometric coherence, and structural similarity.

Let MG,MS,ML denote the Grad-CAM, SHAP and LIME maps, respectively. We first derive:

• an overlap-based term O (e.g., mean Dice or IoU over map pairs);
• a geometric coherence term G_*c* (e.g., agreement in shape descriptors such as eccentricity, compactness, symmetry);
• a structural term S_*c* (e.g., SSIM or correlation between saliency patterns).

Each component is scaled to [0, 1]. The OGSL score is then defined as:

OGSL=wOO+wGGc+wSSc,(1)

with w_*O*, w_*G*, w_*S*≥0 and w_*O*+w_*G*+w_*S* = 1. Note that G_*c* and S_*c* here denote the geometric and structural consensus components of OGSL, and are distinct from the Grad-CAM scalar score G used in the mean stability term M (Section 2.3.8). Higher values of OGSL indicate stronger consensus between XAI methods in terms of overlap, geometry and structural patterns.

Implementation of the OGSL consensus term. The three components of the OGSL consensus score C (Equation 1) are implemented as follows. The spatial overlap O is defined as the mean pairwise Dice coefficient between the binarized XAI masks as defined in Equation 2:

O=Dice(G,S)+Dice(G,L)+Dice(S,L)3.(2)

The geometric coherence G_*c* captures the spatial compactness of the consensus region: G_*c* = 1/n₀, where n₀ is the number of connected components of the majority-vote consensus mask—a single coherent region yields G_*c* = 1, while n fragments yield G_*c* = 1/n. The structural similarity S_*c* is computed as the mean pairwise normalized cross-correlation (NCC) between the continuous (pre-binarisation) XAI heatmaps see Equation 3:

Sc=NCC(G,S)+NCC(G,L)+NCC(S,L)3,(3)

rescaled to [0, 1]. The default weights w_*O* = 0.5, w_*G* = 0.25, w_*S* = 0.25 assign primary importance to pixel-level spatial agreement, with equal secondary contribution from geometric compactness and structural pattern similarity (Parashar et al., 2024). Sensitivity of downstream routing performance to w_*O* was assessed by sweeping w_*O*∈[0.2, 0.8]; results are reported in Section 2.3.8.

2.3.7 ExpiScore

ExpiScore aggregates salience reliability, spatial compactness, cross-method agreement, and morphological relevance into a single metric for interpretability benchmarking. It is stable, interpretable, and suitable for clinical use: a higher score corresponds to higher confidence in the model.

The three XAI methods (Grad-CAM, SHAP, LIME) each produce a scalar quality score [G, S, L∈[0, 1]] derived from their respective heatmaps; these are combined into a mean stability term M=13(G+S+L), while the OGSL term C (defined above) measures cross-method spatial consensus. The central design question is how M and C should interact in a single index. Three qualitatively different strategies arise naturally: (i) a multiplicative formulation that uses C as an exponential penalty, strongly suppressing scores whenever the three methods disagree; (ii) a balanced additive formulation that combines M and C linearly, rewarding complementary explanations without over-penalizing partial disagreement; and (iii) a mean-only formulation that ignores consensus entirely. Table 4 and Figure 6 compare the three variants, with their trade-offs, to motivate the choice of the balanced formulation adopted in this work.

Table 4. Comparison of alternative formulations of the ExpiScore index.

Each row corresponds to a qualitatively distinct fusion strategy. Effect: behavior when XAI methods produce misaligned saliency maps. Bold values indicate the best result or the configuration adopted in this work.

2.3.8 Normalized ExpiScore and proposed formula

Equation 4 defines the weighted mean stability term M

M=αG+βS+γL,(4)

where M∈[0, 1] represents the mean stability of the three XAI methods (Grad-CAM, SHAP, LIME).

The three XAI methods—Grad-CAM (G), SHAP (S), and LIME (L)—are combined with equal weights (α = β = γ = 1/3). This choice reflects the design principle that no single method is assumed to be systematically more reliable or clinically informative than the others on the BreaKHis dataset. Each method provides a complementary perspective: Grad-CAM offers gradient-based spatial localization, SHAP provides attribution-based relevance grounded in cooperative game theory, and LIME approximates local surrogate behavior. The sensitivity of downstream routing performance to unequal α, β, γ configurations was assessed empirically and confirmed to be negligible relative to the effect of λ (see Section 2.3.8). The components of the normalized ExpiScore formulation are summarized in Table 5.

Table 5. Components of the normalized ExpiScore formulation.

To ensure comparability between the two components, the mean term is normalized through min–max scaling as shown in Equation 5:

Mnorm=M-mintrain(M)maxtrain(M)-mintrain(M),(5)

where mintrain(M) and maxtrain(M) are computed once on the training set and fixed thereafter. At inference time, these constants are applied to any new image, ensuring that the normalization is inductive and does not require knowledge of the full test distribution. Values outside the training range are clipped to [0, 1].

The normalized ExpiScore is then defined as (see Equation 6):

ExpiScorenorm=λMnorm+(1-λ)C,(6)

with λ∈[0, 1] controlling the balance between average stability and inter-method consensus. The computation flow is illustrated in Figure 7.

Formal properties of ExpiScore_norm. The index satisfies the following properties for all inputs:

• Boundedness: ExpiScore_norm∈[0, 1], since both M_norm∈[0, 1] and C∈[0, 1] and the weights λ, (1−λ) sum to one.
• Monotonicity: ExpiScore_norm is non-decreasing in both M_norm (individual method quality) and C (inter-method consensus); improvements in either component cannot reduce the score.
• Linearity in λ: the score varies linearly with the balancing parameter, enabling smooth interpolation between the pure-stability regime (λ = 1, ExpiScore = M_norm) and the pure-consensus regime (λ = 0, ExpiScore = C).
• Degeneracy conditions: if all three XAI methods produce identical maps, then C = 1 and M_norm reflects only mean quality; if all maps are mutually orthogonal, then C = 0 and the score reduces to λM_norm.

These properties ensure that ExpiScore is well-defined, interpretable, and monotonically responsive to improvements in both explanation quality and cross-method agreement.

Default value and sensitivity of λ. The balancing parameter λ is fixed to 0.5 a priori, on the principled grounds that neither the mean stability term M_norm nor the OGSL consensus term C is assumed to be more informative than the other without prior evidence. This symmetric default requires no data-driven selection and was not adjusted after observing test-set results.

To confirm the robustness of this a priori choice, λ was swept post-hoc over {0.1, 0.2, …, 0.9} and the post-routing accuracy, False Negative reduction, and False Positive reduction were evaluated on the held-out test set (1,187 images). This sweep is intended as a sensitivity analysis, not a selection procedure: the value λ = 0.5 was fixed before the test set was accessed, and the sweep results are reported solely to show that the a priori choice is not fragile. All tested values of λ yield post-routing accuracy substantially above the CNN-only baseline, and λ = 0.5 performs at least as well as any other value, confirming the robustness of the a priori choice.³ The full sweep results are reported in Supplementary Table S3.

Sensitivity to OGSL weight configuration. To evaluate the robustness of the OGSL consensus term C = w_*O*·O+w_*G*·G_*c*+w_*S*·S_*c* to the choice of weights, four representative configurations were tested spanning a broad range from overlap-dominant to balanced weighting. For each configuration, C was recomputed from the raw pairwise Dice/geometric/structural columns, the ExpiScore was reconstructed via the inverse formula M_norm = (ExpiScore−(1−λ)C_orig)/λ, and routing was applied at the 20th percentile threshold. The full sweep results are reported in Supplementary Table S4.

Configurations A (overlap-dominant, w_*O* = 0.6) and B (paper default, w_*O* = 0.5) tie at the highest post-routing accuracy (0.9612), and all four configurations substantially exceed the CNN-only baseline of the original analysis by at least 6.9 percentage points. The narrow spread across configurations (maximum gap: 0.0025) confirms that ExpiScore-based routing is robust to the precise choice of OGSL weights, provided overlap remains the dominant term (w_*O*≥0.5).

Sensitivity to expert resolution accuracy. The post-routing accuracy reported in Table 6 assumes that the expert correctly resolves all routed cases (p = 1.0). In practice, clinicians operate at finite accuracy. Let p∈[0, 1] denote the probability that an expert correctly resolves a routed error. Of the 236 routed cases in the held-out test evaluation, 10 are classification errors (5 FN and 5 FP) and 226 are already correct. According to Equation 7, the post-routing accuracy is approximately a function of p.

acc(p)≈0.9898+0.0085p,(7)

yielding 0.9983 at p = 1.0, 0.9975 at p = 0.90, and 0.9966 at p = 0.80. Even at p = 0.70, the system accuracy (0.9958) substantially exceeds the CNN-only baseline (0.9898), confirming that the benefit of meta-model routing is robust to realistic levels of expert imperfection.

Table 6. Meta-model held-out test evaluation.

The meta-model (logistic regression on 12 XAI-derived features) was trained on the validation split (1,179 images) and evaluated on the independent held-out test split (1,180 images). Routing the 20% of cases with highest predicted error probability to expert review substantially reduces errors. Baseline: CNN-only classification on the same 1,180-image subset. Expert resolution assumed for all routed cases.

Interpretation. The term M_norm represents the normalized mean stability of the three XAI methods, capturing how strong and consistent their individual scores are. The consensus term C (OGSL) quantifies the agreement between the methods in terms of overlap, geometry and structural similarity.

By adjusting λ:

• high values (≈0.8–1) prioritize stability, giving more importance to the average behavior of Grad-CAM, SHAP and LIME;
• low values (≈0.3–0.5) emphasize consensus, making the ExpiScore more sensitive to multi-method agreement.

2.4 Interpretability meta-model (logistic regression + decision tree)

The meta-model performs two hierarchically related but conceptually distinct functions, which are defined separately below (Parashar et al., 2024). It is composed of two elements: (i) a logistic regression model that estimates the probability of malignancy based on features derived from Grad-CAM, LIME, SHAP, CNN confidence, and the ExpiScore components; (ii) an interpretable decision tree that approximates the CNN's decision logic through symbolic if-then rules. The feature set comprises 42 descriptors in total; confidence is included as one of the 42 inputs, not as a target. During training, XAI-derived features are extracted for each image and aggregated into a descriptive feature vector.

Role 1—Surrogate (primary, structural). The meta-model replicates the CNN's decision logic through interpretable rules (decision tree) and weighted linear combinations (logistic regression). This provides symbolic transparency: clinicians can inspect the rules to understand what drives each prediction. The 91% fidelity score measures this role—the meta-model successfully replicates the CNN's output on 91% of validation-split cases.

Role 2—Error detection (emergent, consequential). The meta-model operates in a feature space (XAI-derived descriptors: saliency, spatial compactness, cross-method agreement, morphological metrics, confidence) that captures information not directly encoded in the CNN's softmax output. In the residual 9% of cases where CNN confidence is structurally inconsistent with its explanatory signals, the meta-model's prediction diverges from the CNN's—and these divergences are disproportionately the CNN's errors, particularly overconfident false negatives. The corrective capacity is not by design; it is a consequence of the meta-model's access to explanation-space information that is orthogonal to confidence alone.

The two roles are distinct but not contradictory; their apparent tension dissolves once the meta-model's feature space is considered explicitly. A surrogate operating on the same representation as its base classifier would, by construction, echo the latter's errors in proportion to its fidelity—at 91% global fidelity, nearly all CNN mistakes would be replicated rather than corrected. The meta-model avoids this because it is trained not on the CNN's softmax output but on XAI-derived descriptors (saliency coherence, cross-method agreement, spatial compactness) alongside CNN confidence, a feature space that encodes structural properties of explanation quality orthogonal to prediction confidence alone. In error-prone cases, particularly overconfident false negatives, this explanation-feature space reveals discordances invisible to the softmax; the corrective capacity emerges from this richer informational geometry rather than from explicit design intent. In addition to improving transparency, this mechanism therefore acts as a detection safeguard by identifying cases in which high-confidence predictions are not supported by coherent explanatory signals, a condition frequently associated with false negatives in medical imaging.

3 Results

3.1 Model performance

Distilled DenseNet121 achieves competitive accuracy with enhanced interpretability (Table 3). All primary experiments were conducted using Macenko stain normalization, selected over Reinhard normalization based on superior convergence stability and saliency coherence observed during validation (Table 7). Macenko normalization operates in optical density space and estimates stain vectors via singular value decomposition, yielding more consistent heatmap boundaries across slides and magnifications. A detailed summary of convergence and distillation metrics for the distilled DenseNet-121 student is given in Table 8.

Table 7. Summary statistics of model performance and loss metrics across training runs and on the held-out test set.

Metrics labeled Train are computed on the training set at the final epoch; metrics labeled Val are computed on the validation set. Metrics labeled Test are computed on the held-out test set (1,187 images, patient-level disjoint split). Clinical metrics—Sensitivity (malignant class recall), Specificity (benign class recall), and Matthews Correlation Coefficient (MCC)—are added to provide a complete characterization of classification performance under class imbalance.

Table 8. Quantitative evaluation of convergence and distillation behavior for the distilled DenseNet-121 student model.

The distillation loss reflects the KL-divergence between the teacher's and the student's soft probability distributions. Its high value does not indicate training instability; rather, it arises from structural differences in logit scale, entropy, and sharpness between the two networks. KL-divergence is unbounded, asymmetric, and highly sensitive to temperature and confidence calibration, which amplifies even small discrepancies in output distributions. Despite the large distill_loss, the model exhibits strong empirical performance, with training and validation accuracy/F1 nearly identical (0.916 vs. 0.918 and 0.940 vs. 0.942), AUC values above 0.94, and a low validation loss ( 0.21). These results confirm that the student successfully captures the teacher's decision boundary, even though the probability distributions do not match in magnitude.

Note on training-set saturation and overfitting. Table 3 (full training logs) shows that several models—including InceptionV3 and the unconstrained ensemble— reach training accuracy/F1 close to 1.0 while validation metrics remain substantially lower (Δ≈0.05–0.10). This pattern indicates training-set saturation (overfitting), which is expected given the relatively small dataset size (≈7, 900 images) and the representational capacity of large pre-trained architectures. The primary model used throughout this work—the distilled DenseNet-121 student—exhibits a markedly smaller train/val gap (validation accuracy = 0.9874, best val loss = 0.0380, delta < 0.002), indicating that the distillation and regularization procedure effectively constrains overfitting. The claim of “stable convergence” in Table 7 applies specifically to the distilled student; the teacher and ablation models show varying degrees of overfitting as noted above.

3.2 Cross-validation robustness

To quantify the generalization uncertainty of the distilled DenseNet-121 student, patient-level five-fold cross-validation was performed on the full BreaKHis dataset using the official fold assignments provided with the dataset (Folds.csv). In each fold, the model was retrained with the same knowledge-distillation protocol (α = 0.5, T = 2.0, early stopping on validation loss) and evaluated on the held-out patient fold. Table 9 reports per-fold and aggregated performance.

Table 9. Five-fold cross-validation results for the distilled DenseNet-121 student.

Patient-level stratification is enforced via the official BreaKHis Folds.csv partition. Each fold uses the same knowledge-distillation protocol (α = 0.5, T = 2.0, early stopping on validation loss, seed = 42). The mean ± std row summarizes generalization stability across splits. Bold values indicate the best result or the configuration adopted in this work.

Cross-validation indicates generalization stability within this evaluation setting: the mean AUC of 0.9598 ± 0.0094 and F1 of 0.9232 ± 0.0136 show consistent classification performance across all five patient-level splits. Sensitivity remains consistently high (0.9315 ± 0.0219), indicating that the model consistently captures malignant patterns across heterogeneous patient subsets. Specificity shows somewhat higher variance (0.8262 ± 0.0478), reflecting the greater difficulty of correctly ruling out benign cases, which is a known characteristic of BreaKHis across different fold configurations. The AUC values cluster tightly in the range [0.946, 0.973], indicating that discriminative capacity is well-preserved across all patient partitions.

Note on the gap between single-split and cross-validation accuracy. The 5-fold CV mean accuracy (0.8962 ± 0.0177) is ≈9.5 percentage points below the single held-out test-set accuracy (0.9916). This gap is expected and does not indicate data leakage. BreaKHis contains 82 patients (Spanhol et al., 2016); the single test split holds = 13 patients, a cohort small enough for its composition to materially affect measured accuracy. The specific 13-patient test cohort that produced 0.9916 represents a favorable draw (predominantly common ductal carcinoma subtypes well-represented in training). The CV result averages over all patient partitions, including less favorable ones, and is the more conservative and representative generalization estimate. As confirmation, the per-fold accuracy range [0.8744, 0.9212] (Table 9) illustrates the sampling variance inherent to patient-level evaluation with this dataset size.

3.3 Interpretability metrics

Instance-level explanations, dataset-level aggregation. All three XAI methods (Grad-CAM, SHAP, LIME) operate at the instance level: each map is generated independently for a single image. The dataset-level analysis presented in this section constitutes a quantitative aggregation of per-image local explanation features, not a global XAI analysis in the technical sense. Grad-CAM highlights diagnostically relevant spatial regions by back-propagating gradients through the final convolutional layer, producing a per-image localized attention map. SHAP produces per-image attribution maps by estimating Shapley values for superpixel coalitions of each individual image; these instance-level attributions are then aggregated across the test set to characterize the distribution of XAI coherence by prediction outcome. LIME approximates local surrogate behavior in the neighborhood of each input, identifying superpixels that most influence the prediction for that specific image.

ExpiScore, computed on the full test set of 1,187 images, yields a mean of 0.294 ± 0.049 [range [0.143, 0.457]], and correlates strongly with both prediction confidence and saliency robustness. Stratified by classification outcome, ExpiScore is highest for True Positives (0.313 ± 0.040), followed by True Negatives (0.279 ± 0.043), False Positives (0.258 ± 0.060), and False Negatives (0.224 ± 0.051). The ordering TP>TN>FP>FN confirms that correctly classified cases, and in particular correctly identified malignant images, consistently exhibit higher explainability quality. Cross-method agreement (Dice overlap) between the three XAI methods on the test set is: SHAP–LIME = 0.272 ± 0.030, Grad-CAM–LIME = 0.509 ± 0.350, SHAP–Grad-CAM = 0.187 ± 0.071. These relatively low absolute Dice values require careful interpretation. Low Dice between two XAI channels can arise from two distinct causes: (i) complementarity, where the methods focus on different but equally valid and informative subregions of the image; or (ii) instability, where method outputs are noise-dominated and carry little shared signal. Distinguishing the two is critical for the validity of the ExpiScore framework. The key empirical test is whether the fused score is discriminative: if low Dice were a symptom of noise, ExpiScore—which integrates all three channels—would produce a flat, uninformative distribution across prediction categories. Instead, ExpiScore exhibits a systematic and statistically significant ordering: TP (0.313)>TN (0.279)>FP (0.258)>FN (0.224), with all pairwise gaps exceeding twice the pooled within-group standard deviation. This outcome is consistent with complementarity: each XAI channel captures a distinct aspect of the model's decision (gradient flow in Grad-CAM, Shapley attribution in SHAP, local surrogate fit in LIME), and their low pairwise overlap reflects genuine representational diversity rather than mutual inconsistency. We note that this interpretation rests on the discriminative outcome and cannot exclude partial noise contribution; a formal ablation study removing one channel at a time would quantify each method's marginal contribution and is identified as a direction for future work (see Limitations, Section 5).

3.4 XAI method-level faithfulness analysis

Faithfulness of an XAI method—the degree to which its explanations are genuinely linked to the model's decisions rather than being noise-dominated—can be assessed both directly (via perturbation-based protocols such as deletion/insertion curves) and indirectly (via the discriminative validity of the derived explanatory signals across outcome groups). An exception is the Grad-CAM component: a direct perturbation experiment—saliency-masking (deletion protocol) and insertion test on a stratified subsample of 64 test-set images—has been conducted and is reported in full in Supplementary Section S5 (Samek et al., 2017; Petsiuk et al., 2018). These experiments provide preliminary method-level faithfulness evidence for the Grad-CAM component contributing to ExpiScore. Equivalent direct faithfulness tests for KernelSHAP (attribution-masking confidence-drop) and LIME (run-to-run Jaccard-IoU stability) require additional GPU-intensive computation and remain planned as a priority follow-up analysis (see Limitations, Section 5); no faithfulness claim is made for ExpiScore as a composite index. The present section reports three complementary lines of indirect, annotation-free evidence that the XAI outputs are not noise-dominated, drawn from analyses already conducted on the full held-out test set (n = 1, 187).

Evidence 1– Systematic outcome-stratified ordering of ExpiScore. If Grad-CAM, SHAP, and LIME maps were noise-dominated, their fused score (ExpiScore) would produce a flat distribution across prediction categories. Instead, ExpiScore exhibits a strict and statistically consistent ordering: TP (0.313 ± 0.040)>TN (0.279 ± 0.043)>FP (0.258 ± 0.060)>FN (0.224 ± 0.051), with all adjacent-group gaps exceeding twice the pooled within-group standard deviation. This monotone discrimination between correctly and incorrectly classified cases—and between the two error types— constitutes annotation-free evidence that the XAI maps carry genuine decision-linked signal.

Evidence 2—Correlation between ExpiScore and CNN prediction confidence. The Spearman rank correlation between ExpiScore and CNN prediction confidence across the full test set is ρ = 0.674 (p < 0.001, n = 1, 187), with stratified correlations of TP ρ = 0.467, TN ρ = 0.673, FP ρ = 0.919, FN ρ = 0.939. The moderate overall association confirms that ExpiScore tracks the model's decision strength, while the imperfect correlation (ρ < 1) confirms that ExpiScore captures additional discriminative information not encoded in confidence alone—consistent with explanations that reflect genuine model reasoning rather than being a simple monotone function of softmax output.

Evidence 3—Predictive validity in the meta-model. The meta-model trained on 42 XAI-derived features (including ExpiScore components, cross-method Dice overlaps, and morphological descriptors) achieves AUC = 0.9030 on the held-out validation set in distinguishing correctly from incorrectly classified images. If XAI outputs were noise-dominated, these features would carry no discriminative signal and the meta-model would perform at chance (AUC ≈0.5). The observed AUC of 0.9030 provides held-out evidence that the XAI feature space encodes genuine error-predictive structure—a form of faithfulness at the aggregate feature level.

Taken together, these three lines of indirect evidence indicate that the Grad-CAM, SHAP, and LIME outputs used in this framework are not noise-dominated and are linked to the model's decision process. Combined with the direct Grad-CAM perturbation evidence in Supplementary Section S5, these results support the discriminative validity of the ExpiScore framework; equivalent direct faithfulness tests for KernelSHAP and LIME are identified as a priority for future work (Section 5).

3.5 Explainability PCA

PCA serves a specific and bounded scientific purpose in this work: it provides an independent characterization of which feature dimensions drive variance across the test set, without reference to prediction labels. This is relevant to ExpiScore validation in two ways. First, if XAI-derived features (Dice overlaps, saliency thresholds, geometric descriptors) dominate the principal components—rather than the raw confidence score alone—this confirms that the feature space captures genuine explainability structure and is not a proxy for model output. Second, the PCA loading pattern reveals which components of ExpiScore are most informative, supporting the feature selection decisions embedded in the index's design. The PCA is not used to train or tune any model component; it is a descriptive analysis.

Starting from the 51 columns in global_expi_csv.csv, eight columns were removed before PCA, leaving 42 numeric features. Exclusions are detailed in Table 10.

Table 10. Feature types excluded before PCA and reasons for removal.

Starting from 51 columns (50 candidate features + target classification), 8 features are removed, leaving 42 numeric features for analysis. Bold values indicate the best result or the configuration adopted in this work.

Feature importance in PCA is defined as the weighted sum of squared loadings across the retained principal components. PCA importance is computed as in Equation 8

Importance(j)=∑k=1Kλkljk2(8)

where l_*jk* is the loading of feature j on principal component k and λ_*k* is the corresponding eigenvalue. The first K components were retained such that the cumulative explained variance (PCE) exceeded 85%.

Below are the most influential features identified through PCA. For each feature, we report its contribution to the principal components and provide a concise interpretation.

Synthetic interpretation and connection to ExpiScore. The top PCA contributors are XAI-derived features—inter-method agreement (dice_shap_lime, dice_shap_grad), saliency thresholds (thr_shap, thr_lime, thr_gradcam), and morphological descriptors (edge_sharpness, symmetry, eccentricity)—not confidence alone. This confirms that the feature space captures genuine explainability structure beyond what a pure softmax output would encode: model confidence ranks only 4th in PCA importance, whereas XAI agreement metrics rank 1st and 10th. This finding provides independent support for the ExpiScore design: by combining the mean stability term M_norm (related to thr_* features) and the OGSL consensus term C (related to dice_* features), ExpiScore aggregates the two principal axes of variance identified by PCA. The observation that confidence is a major but not dominant contributor (rank 4) also explains why confidence-only triage is insufficient: the XAI features carry substantial additional discriminative information orthogonal to model confidence, which the triage logic exploits.

Rationale. PCA emphasizes dimensions with the highest variability across samples. Since saliency as a single scalar is a compressed summary of multiple XAI features (thr_, dice_, sharpness), it naturally exhibits less variance than its components and therefore ranks lower in the loading analysis. This does not diminish its role in ExpiScore; it reflects the expected compression behavior of a composite index.

The radar plot (Figure 8) illustrates the behavior of the main normalized XAI metrics (ExpiScore, Texture, Sharpness, Eccentricity, and Symmetry) across the four prediction categories (TP, TN, FP, FN). Each line represents the average profile for a specific class: the broader and more regular the covered area, the more coherent, compact, and morphologically interpretable the explanation. True Positives (TP) exhibit high and uniform values across all axes, indicating well-centered explanations that align closely with the underlying pathological morphology. True Negatives (TN) show a more regular but moderate profile, reflecting stable structural attention and an appropriate absence of spurious activations. In contrast, False Positives (FP) and False Negatives (FN) display markedly distorted and contracted profiles, with reduced ExpiScore and Edge Sharpness, or increased Eccentricity. These patterns correspond respectively to misaligned attention or a failure to highlight relevant regions. Overall, the radar plot demonstrates that the morphological quality and coherence of XAI explanations progressively deteriorate in misclassified cases. This supports the ability of ExpiScore and geometric metrics to objectively characterize the model's behavior. Representative XAI heatmaps for one case from each outcome group, together with the corresponding consensus overlay, are shown in Figure 9.

ExpiScore as an error-detection signal. The discriminative value of XAI-derived features (including ExpiScore) for routing uncertain cases to human review was confirmed by the meta-model held-out evaluation (Section 3.7): routing the 20% of cases with the highest predicted error probability (236 out of 1,180 test-set images with computed XAI features) to expert review captures 83.3% of False Negatives and 83.3% of False Positives, raising post-routing classification accuracy from 0.9898 to 0.9983 (+0.85 pp) and F1-score from 0.9926 to 0.9988. These results provide held-out test evidence that XAI-derived features—and ExpiScore in particular, as the primary routing signal—serve as discriminative indicators of potential misclassification within this evaluation setting (see also Section 3.4 for per-method faithfulness analysis); the routing policy substantially reduces diagnostic errors under the expert-resolution assumption (oracle: p = 1.0), particularly for the clinically critical False Negative category.

ExpiScore and CNN confidence complementarity. Figure 10 shows a scatter plot of ExpiScore against CNN prediction confidence for the 1,187 held-out test-set cases, with points colored by classification outcome (TP/TN/FP/FN). The Spearman rank correlation between ExpiScore and prediction confidence is ρ = 0.674 (p < 0.001, n = 1, 187), indicating a moderate positive association: cases predicted with higher confidence tend also to receive higher explanatory coherence scores. This correlation is, however, far from perfect and varies substantially by outcome group: TP ρ = 0.467, TN ρ = 0.673, FP ρ = 0.919, FN ρ = 0.939. The markedly higher stratified correlations in the error groups (FP and FN) reflect a clinically informative pattern: among misclassified cases the model is rarely simultaneously confident and explanatorily coherent. Critically, only a small fraction of false-negative cases present confidence above the acceptance threshold of 0.92 and ExpiScore above the 20th-percentile routing cut-off—a pattern consistent across training runs. These cases represent a residual hard-negative population whose explanatory signals are also coherent with a benign prediction, motivating future investigation into second-order disagreement features. Stratified ExpiScore statistics (mean ± SD): TP = 0.313 ± 0.040; TN = 0.279 ± 0.043; FP = 0.258 ± 0.060; FN = 0.224 ± 0.051. The systematic ordering TP>TN>FP>FN holds in mean ExpiScore, consistent with correct predictions receiving more coherent XAI attribution regardless of class.

3.6 Evaluation of the meta-model

Evaluation scope note. The meta-model was trained and evaluated on the validation split; an independent held-out test evaluation is reported in Section 3.7. The validation-split results below (0.9874 baseline accuracy; meta-model improvement on the same split) are retained for completeness and methodological comparison; the definitive out-of-sample evidence is provided in Section 3.7.

To assess whether explainability-derived features carry predictive value, we trained a meta-model that operates exclusively on XAI descriptors, without any access to the original CNN latent features (Bianchi et al., 2024; Kaplanoglou and Diamantaras, 2024; Klein et al., 2024). The meta-model was implemented as a hybrid structure combining logistic regression and a shallow decision tree, providing both interpretability and sensitivity to non-linear interactions.

The feature set included 42 XAI-derived descriptors capturing saliency intensity, compactness, edge activation, overlap-based consensus, structural similarity, confidence, eccentricity, texture, and ExpiScore components. Feature extraction was performed on all three dataset splits (train, validation, test). However, the meta-model was trained and its performance evaluated exclusively on the validation split: the validation features were used for training and threshold selection, while the test-set features were used separately for the PCA analysis reported in Section 3.5 and are not used in the meta-model evaluation. This strict split prevents any form of data leakage between the meta-model training and the held-out test set. An independent evaluation of the meta-model on the held-out test split is reported in Section 3.7.

The decision tree component achieved a fidelity score of 91%, demonstrating its ability to approximate the behavior of the CNN through structured explanations. Logistic regression provided smooth separability across confidence, ExpiScore, Dice metrics, symmetry, and sharpness. The hybrid meta-model, trained on the validation split (1,179 images), demonstrated the ability to predict CNN errors using XAI-derived features, as confirmed by the held-out test evaluation described in Section 3.7.

We also extracted symbolic rules from the tree. Representative examples include:

• if confidence > 0.92 and ExpiScore > 0.80 ⇒ predict malignant (high-activation, strong agreement);
• if confidence < 0.60 and edge_activation < 0.30 ⇒ predict benign (weak saliency, low boundary evidence);
• if ExpiScore < 0.60 and confidence > 0.90 ⇒ flag high-risk FP (overconfidence without coherent saliency).

Finally, the meta-model supports a deterministic triage logic (Accept / Review / Defer) based on two fully specified numerical thresholds. The confidence threshold τ_*c* is derived from the validation-set distribution: a case is deemed high-confidence if the CNN softmax output for the predicted class exceeds τ_*c* = 0.92 (top-8% percentile on the validation set). The ExpiScore threshold τ_*e* corresponds to the 20th percentile of the validation-set ExpiScore distribution (τ_*e* = 0.264). The triage rules are:

• Accept (automatic, high-confidence): confidence ≥0.92 and ExpiScore >0.264—prediction is accepted without review;
• Review (escalate to expert): 0.60 ≤ confidence < 0.92, or any case with ExpiScore ≤ 0.264 regardless of confidence—case is flagged for clinician review;
• Defer (urgent escalation): confidence ≤ 0.60 and ExpiScore ≤ 0.264—case exhibits both low model confidence and low XAI coherence, indicating highest uncertainty; deferred to specialist consultation.

All thresholds are derived from the validation split and fixed before test-set evaluation. Their robustness to threshold choice is reported in the sensitivity analysis (Section 2.3.8) and flagged for external-cohort validation in Limitations (Section 5).

Overall, the results demonstrate that XAI-derived features encode sufficient signal to predict correctness, identify failure cases, and quantitatively characterize model behavior. The reduction in false negatives reflects an improvement in operational detection sensitivity: under a human-in-the-loop routing policy, ambiguous malignant cases that would be missed by confidence-based classification alone are systematically identified and escalated to expert review. It is important to note that this sensitivity improvement is conditional on expert resolution of routed cases; the automatic pipeline alone does not directly modify the CNN's detection behavior on the test set.

3.7 Meta-model held-out test evaluation

To provide an independent out-of-sample evaluation of the meta-model, the meta-model (logistic regression trained on the 12 XAI-derived features listed in Table 11) was trained exclusively on the validation split (1,180 images) and evaluated on the independent held-out test split (1,180 images for which XAI features could be computed; 7 images excluded due to XAI extraction failure).

Table 11. Top PCA-contributing features and their interpretations.

Table 6 reports the results. Routing the 20% of test cases with highest predicted error probability (236 out of 1,180) to expert review reduced false negatives by 83.3% (from 6 to 1) and false positives by 83.3% (from 6 to 1), raising test-set accuracy from 0.9898 to 0.9983 (+0.85 pp) and F1-score from 0.9926 to 0.9988. The error predictor itself achieved AUC = 0.9030 on the held-out set, confirming that XAI-derived features carry genuine discriminative signal for error detection.

This provides held-out test evidence of the meta-model's potential clinical utility: the meta-model, trained solely on the validation split without any access to the held-out test set, successfully identifies cases at elevated risk of misclassification and routes them for expert review, achieving substantial error reduction under the expert-resolution assumption (oracle: p = 1.0) on a fully independent patient cohort. External validation with real expert adjudication and cross-institution calibration remain directions for future work.

4 Discussion

The proposed architecture enhances transparency and traceability, and shows promising robustness within the evaluated single-split setting. Stain normalization mitigates inter-sample variability, ensemble learning stabilizes predictions, and distillation improves interpretability coherence. The alternative meta-model bridges numerical outputs and human-readable decision rules.

4.1 Distilled architecture

4.1.1 Why inception?

InceptionV3 excels at true multi-scale feature extraction: the Inception modules combine parallel convolutions with 1–1, 3–3, and 5–5 kernels, enabling the network to capture both macro-structures (e.g., ductal patterns, stromal organization) and micro-details (e.g., nuclear borders, nucleoli). This multi-scale inductive bias is particularly suited to histopathology, where diagnostically relevant patterns occur at heterogeneous spatial resolutions. The use of parallel branches and 1–1 bottleneck convolutions limits aliasing effects introduced by variable magnification and heterogeneous tissue textures, preserving important morphological cues. Diverse receptive fields further make InceptionV3 naturally resilient to local artifacts, small distortions, and pre-processing inconsistencies often present in digitized slides (Szegedy et al., 2016; Noorbakhsh et al., 2020).

4.1.2 Why DenseNet?

DenseNet121 provides feature reuse and preservation of fine structures. Dense connectivity ensures that early-layer features—such as nuclear boundaries, chromatin texture, and micro-architectural details—are propagated throughout the network without degradation, which is critical for detecting subtle malignancy cues. By mitigating vanishing gradients, DenseNet enables effective training even on relatively small or imbalanced datasets, such as histopathology collections with varied magnifications. DenseNet achieves high accuracy with fewer parameters than comparable architectures, lowering the risk of overfitting while maintaining excellent representational power (Huang et al., 2017; Zhou et al., 2022; Kim et al., 2024).

4.1.3 Why combine Inception and DenseNet?

The two architectures offer complementary inductive biases. InceptionV3 contributes strong multi-scale representation, while DenseNet121 provides deep feature reuse and gradient stability. Their combination yields richer and more coherent representations of shape, scale, and texture—key components of breast histopathology. The hybrid architecture captures heterogeneous staining, morphology, and magnification patterns more effectively than either backbone alone, improving generalization across slides and imaging conditions. XAI maps (Grad-CAM, SHAP, LIME) generated from the hybrid model highlight structures at multiple scales, aligning more closely with clinical reasoning and pathologists' visual hierarchy (Borji et al., 2025; Kaczmarzyk et al., 2024; Aaseegha and Venkataramana, 2025; Nahmatwlla and Ali, 2025).

4.1.4 Implementation summary

The two backbones are combined either through vector concatenation at the embedding level or via a late-fusion weighted ensemble. A classification head with dropout and optional calibration layers is used to generate final predictions. Ablation comparisons between Inception-only, DenseNet-only, and hybrid variants confirm that the combined model yields higher F1/AUC scores and improved calibration (lower ECE).

4.2 Interpretability via alternative reasoning

Decision-tree models convert XAI-derived features into structured diagnostic rules. This advances interpretability from visual post-hoc maps toward explicit reasoning mechanisms aligned with regulatory expectations.

4.3 Impact of the meta-model on transparency and clinical verification

The integration of the meta-model substantially strengthens the transparency of the framework. Logistic regression provides a continuous quantification of the contribution of the XAI features, while the decision tree translates this information into rules that can be easily verified by a clinician.

Principal Component Analysis further confirms this trend: the dominant sources of variance are controlled by XAI factors—such as texture complexity, symmetry-based metrics, directionality, thresholding parameters, and Dice coefficients—rather than classical quality indicators. This demonstrates that explanation morphology is a robust and expressive descriptor of model behavior, consistent across magnifications and tissue subtypes.

The ExpiScore index synthesizes these properties by combining a normalized mean stability term with the OGSL consistency measure, capturing both the amplitude and agreement of saliency. The score increases for well-localized, compact, and multi-method-consistent explanations, and decreases for noisy, dispersed, or method-divergent ones. The radar analysis confirms that TP and TN exhibit smooth, wide profiles across texture, sharpness, eccentricity, and symmetry, while FP and FN display distorted and contracted shapes.

On the held-out test set, the meta-model (trained on the validation split) independently confirms that XAI features carry genuine error-detection signal. Applied as a routing mechanism—flagging the 20% of cases with the highest predicted error probability—it reduces false negatives by 83.3% and false positives by 83.3%, raising post-routing test-set accuracy from 0.9898 to 0.9983 (+0.85 pp) and F1 from 0.9926 to 0.9988 (Table 6). This demonstrates that explainability is not only a tool for interpretation, but also an instrument for operational performance enhancement: by revealing when XAI signals are incoherent, the system enables principled routing of difficult cases to clinical review, improving decision-level outcomes under the proposed Accept/Review/Defer triage. This improvement is of a decisional rather than purely algorithmic nature—the CNN's test-set predictions are unchanged; what improves is the system's ability to flag and redirect the cases it is likely to misclassify.

Automatic vs. human-assisted performance: a clarification. The framework operates in two distinct modes that must not be conflated when interpreting results. In fully automatic mode, no human intervention occurs at any stage: the CNN classifies each image, achieving test-set accuracy of 0.9916, F1 = 0.9939, and AUC = 0.9989 on the 1,187-image held-out test set. The meta-model (logistic regression + decision tree), trained on XAI-derived features from the validation split, serves as an error predictor achieving AUC = 0.9030 on the held-out test set.

In human-in-the-loop mode, the meta-model routes the 20% of cases with highest predicted error probability to expert review. On the independent held-out test set (1,180 images with available XAI features), this raises accuracy from 0.9898 to 0.9983 (+0.85 pp) and reduces false negatives by 83.3% and false positives by 83.3%, assuming correct expert resolution of all routed cases (see Section 3.7 for full results). Without any human review, the routing policy itself carries no corrective effect—it only prioritizes which cases to escalate.

Table 12 summarizes the two operational modes and the corresponding performance figures.

Table 12. Performance summary by operational mode.

Fully automatic rows require no human intervention; the CNN baseline and meta-model error predictor operate algorithmically. Human-in-the-loop rows assume expert resolution of routed cases (oracle: p = 1.0). †: meta-model independently evaluated on held-out test set (R1-C5; see Section 3.7). F1 for meta-model routing (human-in-the-loop) reflects exact relabeling of resolved cases in the held-out evaluation (Section 3.7). Bold values indicate the best result or the configuration adopted in this work.

Taken together, these findings highlight the value of integrating explanatory signals into classification pipelines—supporting robustness, transparency, clinical auditability, and actionable decision-making.

4.4 Comparison with state-of-the-art methods

Table 13 summarizes the performance of representative methods on the BreaKHis dataset and positions the proposed framework within the current literature. Performance figures for all comparison methods are taken directly from the original publications, as reported by the respective authors on the full BreaKHis dataset (binary benign/malignant classification, all magnifications combined unless otherwise noted by the original authors). No re-implementation or re-evaluation was performed; values marked “—” indicate that the metric was not reported in the original paper.

Table 13. Comparison with state-of-the-art methods on the BreaKHis dataset.

The table is divided into two panels: Automatic only (no human involvement, fully algorithmic accuracy) and Human-in-the-loop (post-routing accuracy assuming expert resolution of flagged cases). These two panels are not directly comparable and must be interpreted separately. XAI, explainability method used; Q-score, quantitative interpretability score provided; Routing, XAI-driven human routing mechanism. † Meta-model independently evaluated on held-out test set (1,180 images, R1-C5; Table 6). Bold values indicate the best result or the configuration adopted in this work.

The table separates two regimes that must not be conflated. In the automatic-only regime (Panel A), the proposed CNN baseline (0.9916) surpasses all prior methods except Wakili et al. (0.971), placing the distilled DenseNet-121 student at state-of-the-art level while providing full XAI interpretability. It is important to note that all Panel A methods—including ours—are evaluated purely automatically: no expert is involved, and no routing occurs.

In the human-in-the-loop regime (Panel B), the post-routing accuracy of 0.9983 reflects a system that combines automatic CNN classification with expert resolution of the flagged 20%, independently validated on the held-out test set (R1-C5; Section 3.7). This figure is not comparable with Panel A values, because the routing mechanism offloads the hardest 20% to a human reviewer. No prior method in the comparison provides an equivalent human-in-the-loop result, making Panel B a novel contribution without direct competitors.

The unique differentiators of the proposed framework are: (i) a three-method XAI ensemble with a quantitative inter-method agreement metric (ExpiScore); (ii) a knowledge-distilled backbone; and (iii) an XAI-driven routing mechanism that selectively escalates high-error-probability cases to expert review, reducing false negatives by 83.3% on the independent held-out test set—a capability absent from all competing methods. The operational objective is not to maximize classification performance in isolation, but to ensure that every prediction is accompanied by a calibrated, auditable, and actionable explainability signal—a requirement increasingly central to clinical AI regulation (e.g., EU AI Act, FDA guidance on AI/ML-based Software as a Medical Device).

4.5 Explainability-aware detection

Beyond transparency and auditability, the proposed framework demonstrates a direct impact on detection reliability. In histopathological imaging, false negatives often arise from visually subtle or spatially fragmented malignant patterns, which may still produce high-confidence predictions. Our results indicate that such cases tend to be characterized by lower ExpiScore values and less coherent explanatory signals. By explicitly routing such cases for review, the framework may reduce missed detections under the assumption of expert resolution; the effect on detection sensitivity is conditional on the routing threshold and the quality of expert adjudication.

5 Limitations and future work

ROI annotation dependency. The ROI overlap component R_*i* of ExpiScore requires pathologist-annotated masks, which are not available for the BreaKHis dataset. In the primary experiments reported here, the weight w_*r* was set to zero and redistributed proportionally among the remaining components (effective weights: w_*c* = 0.50, w_*s* = 0.3125, w_*k* = 0.1875). Cross-dataset validation of R_*i* using the BACH dataset Aresta et al., (2019), which provides pixel-level ground truth annotations, is planned as a direct extension of this work (Section 4.4).

Dataset scope. Experiments were conducted exclusively on BreaKHis, a single-institution dataset of fixed magnification levels. Generalization to multi-center cohorts, whole-slide images, and imaging modalities beyond histopathology remains to be validated.

Meta-model calibration. The meta-model was independently evaluated on the held-out test set (Section 3.7; Table 6), confirming out-of-sample generalization (test-set accuracy 0.9898 → 0.9983, FN −83.3%). Independent calibration of the Accept/Review/Defer triage thresholds on an external multi-institution cohort remains an open direction for future work.

Computational cost. Generating SHAP, Grad-CAM, and LIME maps for each image introduces significant inference overhead compared to confidence-only approaches. On a standard GPU (NVIDIA V100), approximate per-image inference times are: Grad-CAM ≈0.2 s, LIME ≈8–15 s (stochastic superpixel perturbation, 1 000 samples), and SHAP ≈15–30 s (100 background samples). The full XAI pipeline therefore requires roughly 25–45 s per image, compared to < 0.1 s for confidence-only inference. Runtime optimization through approximations (e.g., FastSHAP, GradSHAP, or LIME with fewer perturbations) and batched GPU execution are left for future work.

LIME stochasticity. LIME results depend on the random superpixel perturbation process, introducing variability across runs on the same image. In all reported experiments, a fixed random seed was used to ensure reproducibility. In deployment, running LIME multiple times and averaging the resulting attribution scores would reduce variance at the cost of proportionally higher runtime. Quantifying LIME run-to-run stability and its impact on ExpiScore variance is a direction for future work.

Triage threshold validation. The Accept/Review/Defer thresholds (confidence ≥0.92 for Accept, ≤ 0.60 for Defer) were derived from the validation split and represent conservative estimates optimized for high specificity. Independent validation of these thresholds on an external cohort is required before clinical deployment, as optimal cutoffs may differ across institutions, patient populations, and imaging protocols.

XAI ensemble ablation. The three-method XAI ensemble (Grad-CAM, SHAP, LIME) is motivated by the complementary nature of gradient-based, attribution-based, and perturbation-based explanations. A systematic ablation study comparing the single-method and pairwise combinations against the full three-method ExpiScore is a direction for future work that would quantify the marginal contribution of each method.

Probability calibration. The routing and triage logic relies on model confidence thresholds (e.g., ≥0.92 for Accept), which implicitly assume that the softmax output is a well-calibrated probability. Calibration was not formally assessed in this work; Expected Calibration Error (ECE) and reliability diagram analysis are planned as part of the clinical validation protocol.

Single-split evaluation and cross-validation. Patient-level five-fold cross-validation of the distilled DenseNet-121 student was performed using the official BreaKHis Folds.csv partition, with fold-wise reporting of accuracy, F1, AUC, sensitivity, and specificity (Table 9; Section 3.2). The mean AUC of 0.9598 ± 0.0094 and F1 of 0.9232 ± 0.0136 confirm robust generalization across patient-level splits. Cross-validation of the full XAI pipeline (25–45 s per image) remains computationally intensive and is identified as a direction for future work.

Explainer-level reliability. Section 3.4 provides three lines of indirect, annotation-free faithfulness evidence drawn from the full test set: (i) the systematic outcome-stratified ordering of ExpiScore (TP>TN>FP>FN, all pairwise gaps >2σ); (ii) Spearman ρ = 0.674 between ExpiScore and CNN confidence, confirming decision-linked signal while preserving orthogonal discriminative information; and (iii) meta-model AUC = 0.9030 on XAI-derived features, confirming that the XAI feature space encodes genuine error-predictive structure. Method-level perturbation faithfulness has been assessed for the Grad-CAM component via saliency-masking (deletion protocol) and insertion test on a stratified subsample of 64 test-set images; full results are reported in Supplementary Section S5. Equivalent direct faithfulness tests for KernelSHAP (attribution-masking confidence-drop) and LIME (run-to-run Jaccard-IoU stability across multiple random seeds) require additional GPU-intensive computation and remain planned as a priority follow-up analysis; no faithfulness claim is made for ExpiScore as a composite index. Expert pixel-level annotations are not available for BreaKHis, precluding annotation-based localization metrics. A formal one-channel-at-a-time ablation of the XAI ensemble is also identified as future work.

6 Conclusions

We introduced a quantitative framework for evaluating, fusing, and exploiting explainability signals in medical imaging. The approach combines saliency-based methods (Grad-CAM, SHAP, LIME) with geometric and structural descriptors, aggregated through the ExpiScore index and analyzed through a meta-model that operates on XAI-derived features and CNN confidence (42 descriptors in total).

The framework contributes three innovations:

• a multi-stage pipeline integrating prediction, explanation, and reliability assessment;
• a composite interpretability index (ExpiScore) combining stability and inter-method consensus;
• a post-hoc trust model enabling automated triage (Accept/Review/Defer) and substantial error reduction under the expert-resolution assumption.

In experimental evaluation, the distilled DenseNet-121 student achieves accuracy of 0.9916, F1-score of 0.9939, and AUC of 0.9989 on the 1,187-image held-out test set, with 5-fold cross-validation confirming generalization stability (mean AUC 0.9598 ± 0.0094; Table 9). A meta-model trained on XAI-derived features from the validation split and independently evaluated on the held-out test set (1,180 images) raises post-routing accuracy from 0.9898 to 0.9983 (+0.85 pp) and reduces false negatives by 83.3%—providing held-out test evidence, under the expert-resolution assumption (oracle: p = 1.0), supporting the potential clinical utility of the proposed routing framework. Taken together, these findings indicate that explainability-derived metrics carry discriminative predictive signal for error detection and triage within this evaluation setting, and provide a principled foundation for further investigation of clinically transparent AI-assisted diagnosis.

Future work will include: integrating attention-based regularization during training; exploring active learning strategies guided by explanation inconsistency; validating the ExpiScore in multi-center clinical settings; and assessing generalization to larger cohorts and different imaging modalities.

Overall, the framework provides a principled foundation for explainability-aware decision systems in medical imaging, supporting transparent, auditable, and potentially clinically actionable AI behavior in histopathological workflows, with formal probability calibration and cross-dataset validation planned as next steps.

Reproducibility statement

To ensure reproducibility, all experiments were conducted using fixed random seeds in Python 3.10 (Python Software Foundation, Wilmington, DE, USA) with NumPy 1.26 (NumPy Developers, opensource) and PyTorch 2.1 (Linux Foundation, San Francisco, CA, USA) framework-specific initializers.

The exact preprocessing pipeline—including stain normalization parameters, patch extraction rules, and quality-control thresholds—is fully documented in the Methods section.

Model architectures, hyperparameters, and training schedules were standardized across runs and logged systematically.

Explainability computations (Grad-CAM, SHAP, and LIME) were executed using fixed configurations and version-controlled libraries.

The alternative meta-model (regression and decision tree) was trained using deterministic splits and identical feature extraction procedures.

All analyses were performed on a controlled environment with fixed library versions.

The full codebase, including preprocessing scripts, model definitions, training routines, and XAI modules, is available upon request from the corresponding author. Requests will be fulfilled within 2 weeks of receipt.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fimag.2026.1846414/full#supplementary-material

Supplementary material Supplementary material at the source

Footnotes

Edited by: Zhiwei Wang, Huazhong University of Science and Technology, China

Reviewed by: Md. Abid Hasan Rafi, Hajee Mohammad Danesh Science and Technology University, Bangladesh

Selman Uzun, Firat University, Türkiye

Footnotes

The database is accessible via the link http://web.inf.ufpr.br/vri/breast-cancer-database.

The BreaKHis filename convention embeds a hospital patient number in each image name. Exhaustive scanning of all 7,909 images reveals that identifier 13412 appears in both the Ductal Carcinoma (DC) and Lobular Carcinoma (LC) subfolders. Because BreaKHis is distributed only with anonymized filename-level identifiers and no patient metadata, it is impossible to determine from the filenames alone whether this corresponds to one patient with two carcinoma subtypes or to two distinct biological patients assigned the same hospital number. Filename-based extraction of unique patient identifier strings therefore yields 81 distinct entries; explicitly distinguishing 13412 (DC) from 13412 (LC) recovers the full 82 patient records reported by Spanhol et al. (2016). In our pipeline, both records were processed under separate identifiers and assigned to the training split; their malignant labels are correct in both cases. Supplementary Table S2 lists all 82 patients explicitly, with 13412 appearing as two separate rows.

This sensitivity analysis was conducted on the original training run's ExpiScore predictions; the reproduced model's primary routing result is reported in Table 6.