An explainable digital twin framework for structural health monitoring: decision-level fusion of vibration and image diagnostics using SHAP and Grad-CAM

1 Introduction

Structural health monitoring (SHM) has emerged as a fundamental technology for ensuring the safety, reliability, and long-term serviceability of civil infrastructure by enabling the continuous or periodic assessment of structural condition through sensor-based measurements and data-driven analysis (Farrar and Worden, 2006). Among various infrastructure systems, bridges represent one of the most critical applications of SHM because their structural performance is continuously influenced by aging, increasing traffic demands, environmental exposure, material degradation, and extreme loading events. Early identification of structural deterioration is therefore essential for minimizing maintenance costs, extending service life, and preventing catastrophic failures that may threaten public safety (Sun et al., 2020). Traditionally, bridge condition assessment has relied predominantly on periodic visual inspections conducted by experienced engineers. Although widely adopted, these inspections are labor-intensive, subjective, and often incapable of detecting hidden or incipient damage before significant deterioration occurs. Beyond visual inspection, modern SHM draws on a range of complementary sensing strategies, including strain-based monitoring (López-Cuervo et al., 2026), acoustic emission detection (Wang et al., 2026), fiber-optic sensing (Boros et al., 2026), ultrasonic and impact-echo testing (Sagar et al., 2025), and vision-based inspection using cameras and unmanned aerial vehicles (Chou et al., 2025), each suited to different damage mechanisms and structural components (Bao et al., 2019). Vibration-based monitoring and image-based inspection, the two modalities examined in this study, are among the most widely investigated of these approaches because they can be deployed non-intrusively at scale and are readily compatible with automated, learning-based analysis. Recent advances in sensing technologies, wireless communication, cloud computing, and artificial intelligence have accelerated the transition from periodic inspection practices toward continuous, data-driven monitoring systems capable of supporting predictive maintenance and intelligent infrastructure management.

In parallel with improvements in predictive performance, the increasing adoption of artificial intelligence has highlighted another critical challenge for practical SHM applications: the lack of model interpretability. Modern machine learning algorithms, including gradient-boosted decision trees and deep neural networks, often operate as black-box models whose internal decision-making processes are difficult to understand. Although these models frequently achieve excellent classification accuracy, they rarely provide sufficient evidence explaining why a particular structural state has been predicted. Such lack of transparency limits their acceptance in safety-critical engineering applications, where maintenance decisions must be supported by technically justifiable evidence rather than predictive performance alone. Recent review articles have consistently identified interpretability as one of the principal barriers to the wider adoption of artificial intelligence in structural health monitoring, emphasizing that trustworthy SHM systems should combine high predictive capability with transparent and physically meaningful explanations of model behavior (Spencer et al., 2025; Ta et al., 2025; Azanaw, 2025).

Addressing this challenge requires combining accurate predictive models with physically meaningful interpretation of the evidence used to reach diagnostic decisions, across both the vibration-based and image-based modalities used in practice. Section 2 reviews related developments in vibration-based SHM, explainable artificial intelligence, image-based inspection and multi-modal fusion, and digital twin and augmented reality technology.

2 Literature review

2.1 Vibration-based structural health monitoring

Among the various SHM approaches, vibration-based damage detection has received considerable attention because changes in structural stiffness caused by damage are reflected in measurable variations of modal properties, including natural frequencies, damping ratios, and mode shapes. By analyzing these dynamic characteristics, vibration-based methods can detect structural changes without requiring direct visual access to the damaged component, making them particularly suitable for monitoring large-scale bridges and other inaccessible infrastructure (Avci et al., 2021). The integration of machine learning with vibration-based SHM has further improved the capability of identifying complex structural behavior from large monitoring datasets, with algorithms such as ensemble learning models, support vector machines, and deep neural networks demonstrating promising predictive performance for structural damage classification.

Despite these advances, the practical deployment of intelligent SHM systems remains constrained by the influence of environmental and operational variability on structural response measurements. Modal characteristics are affected not only by structural deterioration but also by changes in temperature, humidity, traffic loading, wind conditions, and other environmental factors encountered during long-term operation. In many cases, these environmental effects produce variations comparable to those associated with actual damage, making it difficult to distinguish genuine structural deterioration from normal operational fluctuations. Consequently, machine learning models trained without adequately accounting for environmental variability are susceptible to false alarms, missed detections, and reduced reliability when deployed under realistic field conditions (Worden and Cross, 2018).

2.2 Explainable artificial intelligence in structural health monitoring

The challenges mentioned above have stimulated growing interest in explainable artificial intelligence (XAI), which aims to improve the transparency of complex machine learning models by identifying the features or input regions responsible for individual predictions. Rather than treating predictions as isolated outputs, XAI enables engineers to examine the evidence supporting each diagnostic decision and evaluate whether the model is relying on physically meaningful structural information. Early model-agnostic approaches such as Local Interpretable Model-Agnostic Explanations (LIME) demonstrated that complex predictive models could be locally approximated using interpretable surrogate models, thereby enabling explanations without requiring knowledge of the internal architecture of the original classifier (Ribeiro et al., 2016). Building upon this concept, Shapley Additive Explanations (SHAP) introduced a unified game-theoretic framework that attributes predictions to individual input features while satisfying desirable theoretical properties such as local accuracy, consistency, and additivity, leading to its widespread adoption across engineering applications (Lundberg and Lee, 2017). Nevertheless, explainability remains an active area of research, with ongoing efforts focused on improving the reliability, consistency, and practical usefulness of explanations, particularly within safety-critical domains where engineers must understand not only the predicted structural condition but also the evidence supporting that prediction (Doshi-Velez and Kim, 2017; Adadi and Berrada, 2018). Recent studies in aerospace structural health monitoring have demonstrated that integrating explainability techniques with machine learning models can improve user confidence by providing physically meaningful interpretations of damage localization, while contemporary reviews of intelligent sensing technologies increasingly recognize explainability as a fundamental requirement for the development of trustworthy SHM systems alongside prediction accuracy, robustness, and computational efficiency (Pedraza et al., 2025; Mardanshahi et al., 2025).

2.3 Image-based inspection and multi-modal fusion

While vibration-based SHM provides valuable information regarding the global dynamic behavior of a structure, it offers limited capability for identifying the location and visual characteristics of surface deterioration. Consequently, image-based inspection has become an important complementary approach for structural condition assessment, driven by advances in high-resolution imaging, unmanned aerial vehicles, and deep learning-based computer vision. Convolutional neural networks (CNNs) have demonstrated excellent performance in automatically detecting and classifying defects such as cracks, spalling, corrosion, and delamination, thereby improving the efficiency and consistency of bridge inspections compared with conventional manual assessment. Beyond predictive accuracy, explainability methods such as Gradient-weighted Class Activation Mapping (Grad-CAM) further enhance the transparency of image-based models by highlighting the regions that contribute most strongly to each prediction, enabling engineers to verify whether the classifier is focusing on physically meaningful damage rather than irrelevant background features (Altaf et al., 2025; Selvaraju et al., 2020; Ruggieri et al., 2025; Xu et al., 2025).

The complementary nature of vibration and image data has motivated increasing interest in multi-modal SHM, where heterogeneous sensing information is combined to achieve a more comprehensive assessment of structural condition. Vibration measurements are sensitive to global changes in structural stiffness and dynamic behavior but provide limited spatial information regarding the location of visible damage. Conversely, image-based inspection accurately localizes surface defects but cannot readily identify hidden or internal deterioration. Integrating these sensing modalities therefore has the potential to improve both the reliability and completeness of structural diagnosis. Previous studies have demonstrated improved damage classification by combining vibration-derived features with optical images collected simultaneously from the same laboratory specimen under controlled conditions (Al-Qudah et al., 2025). However, such datasets are rarely available in practice. For most operational bridges, vibration measurements and inspection images are acquired independently, often by different inspection teams and at different stages of structural deterioration. Consequently, developing practical multi-modal SHM frameworks capable of operating under realistic field conditions remains an important research challenge.

2.4 Digital twin and augmented reality for infrastructure management

In parallel with advances in sensing and artificial intelligence, digital twin technology has emerged as a promising paradigm for intelligent infrastructure management. A digital twin is a continuously evolving virtual representation of a physical asset that integrates sensor data to reflect its current structural condition and support maintenance decision-making throughout its service life. In bridge engineering, digital twins have primarily been developed using finite element model updating, Building Information Modeling (BIM), and sensor-driven condition assessment to improve infrastructure visualization, maintenance planning, and asset management (Boje et al., 2020). Although these frameworks significantly enhance situational awareness, they generally communicate the estimated structural state without explicitly revealing the measurements or observations responsible for the underlying diagnostic decisions. Consequently, conventional digital twins improve information management but do not inherently address the need for transparent and explainable artificial intelligence in safety-critical infrastructure applications.

To further enhance field-based decision-making, augmented reality (AR) has increasingly been adopted as an intuitive interface for visualizing digital twin information directly on physical infrastructure. Recent studies integrating scan-to-BIM workflows, digital twins, and AR have demonstrated considerable potential for improving bridge inspection and maintenance by allowing engineers to visualize structural information within the physical environment rather than through conventional desktop interfaces (Fawad et al., 2025). Similar developments have also been reported for railway infrastructure, where immersive visualization technologies have been employed to support maintenance planning and inspection activities (Ricci et al., 2025). Nevertheless, existing AR-enabled digital twin platforms primarily focus on displaying geometric models, finite element results, or sensor measurements, while comparatively little attention has been devoted to presenting explainable artificial intelligence outputs, such as feature attribution from SHAP or localized defect evidence from Grad-CAM. Despite these advancements, existing studies have largely investigated explainable AI, multi-modal sensing, digital twins, and augmented reality as separate research directions. A unified framework that seamlessly integrates these technologies for transparent and trustworthy structural health monitoring remains largely unexplored. This gap motivates the development of the explainable multi-modal digital twin framework proposed in this study.

2.5 Research gaps

Despite significant progress in intelligent structural health monitoring, four gaps in the current literature motivate this study.

• Vibration-based damage detection studies have primarily emphasized predictive accuracy while providing limited insight into how modal and environmental features influence diagnostic decisions.
• Vibration- and vision-based SHM have largely evolved as independent research directions despite providing complementary information regarding structural condition.
• Although SHAP and Grad-CAM have individually demonstrated their effectiveness for model interpretation, they are rarely integrated within a unified explainable framework for structural health monitoring specifically.
• Current digital twin and augmented reality platforms primarily function as visualization tools rather than decision-support systems capable of communicating explainable artificial intelligence outputs.

2.6 Novelty and objectives

Digital twin and augmented reality platforms have already been demonstrated for bridge condition visualization; for example, Fawad et al. (Fawad et al., 2025) developed an immersive bridge digital twin platform that integrates scan-to-BIM workflows, a digital twin, and augmented reality visualization for damage assessment and asset model updates. That platform, and comparable augmented-reality-enabled digital twin systems, deliver geometric models, finite element results, or raw sensor measurements to the inspector, but do not deliver the outputs of an explainable artificial intelligence model, such as SHAP feature attributions or Grad-CAM localization evidence. Likewise, the pairing of a tabular attribution method with a spatial attribution method for model interpretation is established practice outside structural engineering, but is rarely applied within a single structural health monitoring pipeline. Building on these two observations, this study does not claim to be the first to combine explainable AI, multi-modal sensing, and digital twin concepts in general; rather, it contributes a specific, evaluated combination that has not been demonstrated in the SHM literature reviewed for this study: a digital twin whose Structural Health Index is directly and continuously traceable to SHAP feature attributions and Grad-CAM localization evidence, evaluated on a real, openly available bridge monitoring dataset spanning a genuine damage and repair cycle.

To address the gaps identified in Section 2.5, this study proposes an explainable digital twin framework that integrates vibration-based damage diagnosis, image-based defect classification, explainable artificial intelligence, and digital twin technology within a unified structural health monitoring workflow. SHAP is employed to interpret vibration-based predictions, while Grad-CAM provides visual explanations for image-based defect classification. The complementary outputs from both sensing modalities are combined through decision-level fusion to compute a Structural Health Index (SHI), which continuously updates a data-driven digital twin for structural condition assessment.

Accordingly, the objectives of this study are as follows.

• To develop and evaluate a vibration-based damage classification model for distinguishing healthy, damaged, and repaired structural states of the KW51 railway bridge using physics-informed modal features.
• To interpret vibration-based predictions using Shapley Additive Explanations (SHAP) by quantifying the contribution of modal and environmental variables at both the global and local levels.
• To develop an EfficientNet-B0-based image classification model for identifying six categories of concrete surface defects and validate its predictions using Gradient-weighted Class Activation Mapping (Grad-CAM).
• To integrate vibration- and image-based diagnostic outputs into a Structural Health Index and implement the proposed framework within a data-driven digital twin for structural condition assessment.
• To evaluate the proposed framework in terms of predictive performance, explainability, environmental robustness, and the practical challenges associated with integrating unpaired multi-modal datasets.

3 Materials and datasets

3.1 KW51 railway bridge vibration dataset

The vibration branch of this study utilizes the publicly available KW51 railway bridge monitoring dataset (Maes and Lombaert, 2020), collected from a steel bowstring railway bridge located in Leuven, Belgium. The bridge was instrumented with accelerometers, strain gauges, and environmental sensors and continuously monitored between October 2018 and January 2020. During this monitoring period, a genuine defect developed in the connections between the diagonal members and the bridge deck, was subsequently repaired, and monitoring continued after retrofitting (Maes and Lombaert, 2021). Consequently, the dataset contains three real structural conditions rather than artificially simulated damage scenarios: healthy (October 2018–March 2019), damaged (April 2019–September 2019), and repaired (October 2019–January 2020). The dataset authors employed operational modal analysis to extract modal characteristics, including natural frequencies, damping ratios, and Modal Assurance Criterion (MAC) values, together with environmental variables such as steel temperature, relative humidity, wind speed, and wind direction (Maes et al., 2022).

Following data alignment and quality filtering, 11,328 time-stamped observations were retained, comprising 4,368 healthy, 4,392 damaged, and 2,568 repaired samples. To reduce the influence of environmental variability on the dynamic response, environmental detrending was applied to the modal features prior to feature engineering, following established practices for long-term bridge monitoring (Han et al., 2021). Subsequently, 237 candidate features describing modal behavior and environmental effects were generated, including frequency shift ratios, pairwise frequency ratios, damping-based indicators, and residual environmental features. Feature selection was then performed to retain the 119 most informative features for vibration-based damage classification. Figure 1 illustrates representative trends in natural frequency, damping ratio, environmental conditions, and the resulting class distribution used in this study.

3.2 CODEBRIM concrete defect image dataset

The image branch of this study employs the Concrete Defect Bridge Image (CODEBRIM) dataset (Mundt et al., 2019), a publicly available benchmark dataset comprising image patches extracted from photographs of real bridge structures across Europe. The dataset is annotated for six defect categories: background (no visible defect), crack, spallation, efflorescence, exposed reinforcement, and corrosion stain. Since CODEBRIM is a multi-label dataset, a single image patch may contain multiple defect types simultaneously, reflecting the complex deterioration patterns commonly observed in reinforced concrete bridge structures. To ensure fair performance evaluation and prevent data leakage, the original training, validation, and test partitions provided with the dataset were preserved throughout this study. CODEBRIM was selected over alternative concrete defect datasets, such as SDNET2018 or single-label crack-only datasets, for three reasons. First, it is drawn from photographs of real, in-service bridge structures across Europe rather than laboratory specimens or pavement surfaces, making it directly relevant to bridge inspection. Second, its multi-label annotation captures the co-occurrence of multiple defect types within a single image patch, which better reflects the deterioration patterns observed in practice than single-label alternatives. Third, it is a peer-reviewed, openly available benchmark introduced at a major computer vision venue (Mundt et al., 2019), which supports reproducibility and comparison with prior work. No dataset pairing CODEBRIM-like imagery with co-located vibration measurements from the same structure is currently available, which is the reason the present study could not construct a genuinely paired multi-modal dataset (see Sections 4.5,6.2). Table 1 summarizes the class distribution of the datasets used in the present work.

TABLE 1. Summary of the KW51 vibration dataset and CODEBRIM image dataset used in this study.

KW51 class counts refer to the temporally labeled vibration measurements; CODEBRIM, class counts refer to the number of training image patches carrying each label, and sum to more than the total number of patches because the dataset is multi-label.

3.3 Data preprocessing

Prior to model development, independent preprocessing pipelines were applied to the vibration and image datasets to improve data quality and ensure compatibility with the respective machine learning models. The preprocessing procedures were performed separately because the two datasets originate from different sensing modalities and were analyzed using distinct classification frameworks.

For the KW51 vibration dataset (Han et al., 2021), the raw tracked-mode record released by the original dataset authors already comprised 11,328 time-stamped observations, corresponding to approximately four modal parameter estimates per day over the fifteen-month monitoring campaign from October 2018 to January 2020. No additional temporal downsampling or date-range filtering was applied. Preprocessing consisted of imputing occasional gaps in the tracked natural frequency, damping ratio, modal assurance criterion, and environmental channels, arising from intermittent dropouts of the automated mode-tracking algorithm, using forward-fill followed by backward-fill within each channel; any residual row that could not be filled by this procedure would have been dropped, though in practice the full set of 11,328 observations was retained after imputation. Subsequently, a set of physics-informed features describing frequency variations, damping behavior, modal relationships, and residual environmental effects was extracted from the detrended measurements. Feature selection was performed using an importance-based ranking derived from the XGBoost model, retaining the most informative variables while reducing feature redundancy and computational complexity. The selected features were standardized using z-score normalization prior to model training.

For the CODEBRIM image dataset, all images were resized to 224 × 224 pixels to match the input requirements of the EfficientNet-B0 architecture. Pixel values were normalized using the ImageNet mean and standard deviation, consistent with the pretrained network initialization. To improve model generalization and reduce overfitting, online data augmentation was applied to the training images, including random horizontal flipping, random rotation, random resized cropping, and color jittering. The original validation and test sets were preserved without augmentation to ensure an unbiased evaluation of model performance.

The original class labels and predefined dataset partitions were retained for both datasets throughout the study. Since the vibration and image datasets originate from different bridge structures and do not possess sample-level correspondence, no attempt was made to artificially pair individual observations during preprocessing. Instead, each sensing modality was processed independently, and their outputs were subsequently integrated through the proposed decision-level fusion framework described in the following section.

4 Methodology

4.1 Framework overview

The proposed framework integrates multi-modal structural health monitoring, explainable artificial intelligence (XAI), and digital twin technology to provide transparent and interpretable damage assessment. In this study, the term multi-modal denotes the combination of two structurally distinct sensing modalities, vibration and image data, each processed independently and combined through decision-level fusion, rather than sample-paired measurements collected jointly from the same structure and damage event. This distinction is stated explicitly here because it affects how the fusion results in Section 5.4 should be interpreted. As illustrated in Figure 2, the framework comprises four primary components. The vibration branch analyzes physics-informed modal features extracted from the KW51 railway bridge dataset using an XGBoost classifier, with model predictions interpreted through Shapley Additive Explanations (SHAP). The image branch identifies surface defects from the CODEBRIM dataset using a fine-tuned EfficientNet-B0 model, and the corresponding predictions are explained using Gradient-weighted Class Activation Mapping (Grad-CAM). The outputs from both branches are subsequently integrated through a decision-level fusion strategy to compute a Structural Health Index (SHI), which provides a continuous measure of structural condition. The SHI is then incorporated into a data-driven digital twin for continuous structural condition assessment and threshold-based health monitoring. In addition, an augmented reality (AR) interface is proposed as a future extension to visualize the SHI and AI-generated explanations during field inspections; however, this component is outside the scope of the present study. The overall workflow of the proposed framework is illustrated in Figure 2.

4.2 Vibration branch: feature engineering and classification

The vibration branch performs structural state classification using the preprocessed modal features extracted from the KW51 dataset. Following environmental detrending, 237 physics-informed candidate features were generated, including frequency shift ratios, pairwise frequency ratios, damping-based indicators, and residual environmental features. Feature importance analysis was subsequently performed using XGBoost, and the 119 most informative features were retained for model development. Structural state classification was carried out using the Extreme Gradient Boosting (XGBoost) algorithm, an ensemble learning method that combines multiple decision trees through gradient boosting to improve predictive accuracy.

XGBoost was selected for both feature ranking and classification because it natively provides feature importance scores and is directly compatible with SHAP TreeExplainer for consistent downstream interpretability. It also handles the mixed frequency, damping, and environmental features used in this study without extensive preprocessing. Furthermore, XGBoost has demonstrated strong performance on tabular structural health monitoring data compared with deep learning approaches, which typically require substantially larger datasets than the 11,328 measurements available here (Kashavar et al., 2026; Qiu et al., 2024). Using the same model for feature ranking and final classification also ensures consistency between the identified feature importance, classifier evaluation, and subsequent interpretation presented in Sections 5 and 6.

The model prediction can be expressed aswhere x i represents the input feature vector, f k denotes the k th decision tree, and K is the total number of trees in the ensemble. The model was trained to classify the bridge into three structural states: healthy, damaged, and repaired. The optimized classifier consisted of 112 decision trees, and the predicted class probabilities were used in the subsequent explainability and Structural Health Index (SHI) analyses.

y ^ i = ∑ k = 1 K f k x i ,

4.3 Image branch: efficientNet-B0 for multi-label defect classification

The image branch employs EfficientNet-B0, a convolutional neural network pretrained on the ImageNet dataset, for multi-label classification of concrete surface defects. Transfer learning was adopted by fine-tuning the pretrained network using the CODEBRIM dataset, enabling the model to learn defect-specific features while leveraging the rich visual representations acquired during large-scale pretraining. The network predicts six defect categories, namely, background, crack, spallation, efflorescence, exposed reinforcement, and corrosion stain. Since multiple defects may coexist within a single image, the classification task was formulated as a multi-label problem using sigmoid activation for each output node and the binary cross-entropy (BCE) loss function:where N is the number of output labels, y i is the ground-truth label, and p i is the predicted probability for the i th defect class. The trained model produces class-wise defect probabilities, which are subsequently interpreted using Grad-CAM and incorporated into the proposed Structural Health Index (SHI). The network was trained for 15 epochs on 6,438 training images, validated on 611 images, and evaluated on a held-out set of 628 test images.

L BCE = − 1 N ∑ i = 1 N y i ⁡ log ⁡ p i + 1 − y i log ⁡ 1 − p i ,

4.4 Explainability: SHAP and Grad-CAM

To enhance the transparency and interpretability of the proposed framework, Shapley Additive Explanations (SHAP) and Gradient-weighted Class Activation Mapping (Grad-CAM) were employed to explain the predictions of the vibration and image branches, respectively. For the vibration branch, SHAP was applied using the TreeSHAP algorithm, which is specifically designed for tree-based ensemble models such as XGBoost. SHAP explains an individual prediction by quantifying the contribution of each input feature to the model output:where f x is the model prediction, ϕ 0 represents the baseline prediction, M is the total number of input features, and ϕ i denotes the contribution of the i th feature. Both global feature importance and local sample-level explanations were generated to interpret the vibration-based damage predictions.

f x = ϕ 0 + ∑ i = 1 M ϕ i ,

For the image branch, Grad-CAM was applied to the final convolutional layer of the fine-tuned EfficientNet-B0 model to identify the image regions that contributed most strongly to each predicted defect class. The Grad-CAM heatmap is computed aswhere A k denotes the feature map of the k th convolutional channel and α k c represents the corresponding importance weight for class c. The resulting heatmaps were overlaid on the original images to provide visual explanations of the model predictions.

L Grad ‐ CAM c = ReLU ∑ k α k c A k ,

4.5 Ensemble fusion and the structural health index

To integrate information from the vibration and image branches, a Structural Health Index (SHI) was formulated using a decision-level fusion strategy. The SHI provides a continuous measure of structural condition, ranging from 0 (healthy) to 1 (critical), and is computed as a weighted combination of the vibration-based damage probability and the image-based defect probability:where P_v and P_i denote the normalized probability outputs of the vibration and image branches, respectively, and w_v and w_i are their corresponding weights, satisfying w_v + w_i = 1. The weights were set to w_v = 0.65 and w_i = 0.35, reflecting that the vibration branch is derived from continuous, structure-specific monitoring data, while the image branch is derived from a general-purpose defect classifier not trained on images of the KW51 bridge itself. The resulting SHI was used to update the digital twin described in Section 4.6.

SHI = w v P v + w i P i ,

4.6 Digital twin implementation

A data-driven digital twin was developed by continuously updating the structural condition using the proposed Structural Health Index (SHI) throughout the fifteen-month monitoring period of the KW51 bridge. At each monitoring instant, the SHI obtained from the decision-level fusion framework was used to represent the current health state of the structure and track its temporal evolution. To facilitate structural condition assessment, the continuous SHI was mapped into three operational states using predefined thresholds:

State = Healthy , SHI < 0.40 , Alert , 0.40 ≤ SHI < 0.70 , Critical , SHI ≥ 0.70 .

The updated SHI and the corresponding structural state were stored within the digital twin, enabling continuous condition visualization, damage tracking, and timely identification of abnormal structural behavior. The digital twin therefore serves as a dynamic representation of the monitored bridge by integrating AI-based damage assessment with temporal health monitoring.

The implementation reported here satisfies the core digital twin requirement of maintaining a continuously updated structural state tied directly to sensor-derived measurements, and the SHI computation is causal and per-observation, meaning it depends only on the current and past measurements and would update identically if deployed on a live data stream. However, the present study applies this computation retrospectively to the archived fifteen-month KW51 record rather than to a live sensor feed, and does not implement bidirectional interaction with the physical asset or a geometric or finite-element representation of the structure. The framework is therefore best characterized as a data-driven digital twin core, ready for streaming deployment but not yet operated as one.

4.7 Evaluation protocol

To ensure a realistic assessment of model performance, a temporal holdout strategy was adopted for the KW51 vibration dataset. Since the structural states occur sequentially over time, randomly splitting the data would introduce temporal dependence between the training and test sets. Therefore, for each structural state, the final 20% of the observations were reserved for testing, while the remaining 80% were used for model training. This approach yielded a temporally independent test set comprising 873 healthy, 878 damaged, and 513 repaired samples.

During model development, five-fold cross-validation was employed for hyperparameter optimization, whereas the temporal holdout dataset was used exclusively for final performance evaluation. Model performance was assessed using accuracy, precision, recall, F1-score, and confusion matrices for the vibration classification task. For the multi-label image classification task, performance was evaluated using binary accuracy, precision, recall, F1-score, and receiver operating characteristic (ROC) analysis. The effectiveness of the proposed digital twin framework was further assessed by analyzing the temporal evolution of the Structural Health Index (SHI) and its ability to identify changes in structural condition throughout the monitoring period.

5 Results

5.1 Vibration-based damage classification

The proposed XGBoost classifier was evaluated using the temporal holdout strategy described in Section 4.7. On the test set, the model achieved an overall accuracy of 68.2%, a macro F1-score of 0.682, a weighted F1-score of 0.675, and a macro AUC-ROC of 0.880. The overall performance metrics are summarized in Table 2, together with the cross-validation results obtained during model development.

TABLE 2. Overall performance of the proposed XGBoost vibration classifier on the temporal holdout test set.

The class-wise performance reveals (Table 3) a clear variation in classification accuracy across the three structural states, as illustrated in Figures 3, 4. The classifier achieved the highest performance for the repaired class, which was identified without any false negatives, indicating that the learned feature representation effectively captured the modal characteristics associated with the repaired structural condition. Similarly, the healthy class exhibited satisfactory discrimination, with most samples correctly identified and only a moderate proportion being classified as damaged.

TABLE 3. Class-wise performance of the proposed XGBoost vibration classifier on the temporal holdout test set.

In contrast, the damaged class proved substantially more challenging to identify. A considerable number of damaged observations were predicted as belonging to the repaired state, resulting in a comparatively lower recall and the smallest class-wise AUC among the three structural conditions. This finding suggests that the modal characteristics of the damaged and repaired states exhibit partial overlap, making their separation more difficult for the classifier. Since the damaged class represents the most safety-critical structural condition, improving its discrimination remains an important objective for future work.

The ROC curves shown in Figure 4 further support these observations. While the healthy and repaired classes achieved excellent discriminative performance with AUC values exceeding 0.95, the damaged class obtained a substantially lower AUC, confirming that this class represents the primary source of classification uncertainty.

5.2 Image-based defect classification

The proposed EfficientNet-B0 model demonstrated strong performance on the CODEBRIM test set, achieving a macro F1-score of 0.886, precision of 0.890, recall of 0.889, and a macro AUC-ROC of 0.977 across the six defect categories. The learning curves presented in Figure 5 illustrate the training process, while Figure 6; Table 4 summarize the class-wise classification performance on the held-out test set.

TABLE 4. Class-wise performance of the proposed EfficientNet-B0 image classifier on the CODEBRIM test set.

The learning curves in Figure 5 indicate stable convergence of the proposed model. Training loss decreased consistently throughout the optimization process, while validation loss stabilized after the initial epochs with only a slight increase during later training, suggesting mild overfitting. Similarly, both training and validation F1-scores improved rapidly during the early epochs before reaching stable values, indicating satisfactory convergence of the network.

The class-wise results indicate that background and exposed reinforcement were classified with the highest accuracy, achieving F1-scores above 0.94, demonstrating that these categories possess highly discriminative visual characteristics. In contrast, efflorescence exhibited the lowest recall among all defect classes despite achieving very high precision. This indicates that the model generated few false-positive predictions for efflorescence but failed to detect a proportion of true instances, likely due to the subtle appearance and low visual contrast of efflorescence deposits. The remaining defect categories, including crack, spallation, and corrosion stain, achieved balanced precision and recall, indicating robust performance across diverse surface deterioration patterns.

5.3 Explainability results

5.3.1 SHAP results for the vibration branch

To interpret the predictions of the XGBoost vibration classifier, SHAP was employed to quantify the contribution of individual features to the predicted structural state. Figure 7 presents the global feature importance ranking based on the mean absolute SHAP values averaged across the three structural classes, while Table 5 summarizes the fifteen most influential features together with their corresponding feature categories.

TABLE 5. Fifteen most influential features ranked by mean absolute SHAP value together with their corresponding feature categories.

The global SHAP ranking demonstrates that frequency shift features dominate the model predictions, indicating that changes in modal frequencies provide the strongest evidence for distinguishing structural states. Among these, freq_mode_14_shift and freq_mode_9_shift exhibit the highest importance, highlighting the sensitivity of modal frequency variations to structural damage. Environmental variables, particularly temperature and relative humidity, also appear among the most influential features, suggesting that environmental effects continue to contribute to the model predictions despite the application of environmental detrending.

The class-specific SHAP summary plots presented in Figures 8, 9 reveal that the importance of individual features varies across structural states. For the healthy class, frequency-related features dominate the prediction, whereas the damaged class is influenced by a combination of modal frequency shifts and environmental variables. The wider SHAP value distributions observed for several environmental features indicate that their contribution depends on the operating conditions of individual monitoring instances rather than remaining constant across the dataset.

Representative SHAP force plots for the healthy, damaged, and repaired classes are presented in Figures 10–12. These local explanations illustrate how individual features contribute either positively or negatively toward a specific prediction, thereby providing an interpretable explanation of the classifier’s decision for a single monitoring instance. Such local explanations complement the global SHAP analysis by enabling engineers to understand the feature-level evidence supporting individual damage assessments.

Overall, the SHAP analysis demonstrates that the proposed framework provides both global model interpretability through feature importance ranking and local interpretability through instance-level explanations, thereby improving the transparency of AI-assisted structural health monitoring.

5.3.2 Grad-CAM results for the image branch

To interpret the predictions of the EfficientNet-B0 model, Gradient-weighted Class Activation Mapping (Grad-CAM) was employed to visualize the image regions contributing most strongly to each predicted defect class. Representative Grad-CAM visualizations for crack, spallation, efflorescence, corrosion stain, and background samples are presented in Figures 13–17.

For the defect classes, the Grad-CAM heatmaps consistently localize on the visually deteriorated regions of the concrete surface. In particular, the network focuses on the crack path, spalled region, mineral deposits associated with efflorescence, and rust-stained areas corresponding to corrosion, demonstrating that the model primarily relies on physically meaningful defect characteristics rather than unrelated background textures. This spatial correspondence between the highlighted regions and the visible defects provides qualitative evidence that the proposed image classifier has learned discriminative visual features relevant to structural condition assessment.

In contrast, the background sample exhibits a diffuse activation pattern without strong localization to any specific surface feature, which is consistent with the absence of visible structural defects. Overall, the Grad-CAM analysis confirms that the proposed framework provides visual interpretability by highlighting the regions responsible for individual defect predictions, thereby improving the transparency and trustworthiness of AI-assisted image-based structural health monitoring.

5.4 Fusion and digital twin results

The proposed decision-level fusion framework was evaluated by combining the outputs of the vibration and image branches using the Structural Health Index (SHI) formulation described in Section 4.5. The performance of the fused model was compared with that of the vibration-only classifier, and the results are summarized in Table 6; Figure 18.

TABLE 6. Comparison of the vibration branch alone against the fixed-weight fusion of the vibration and image branches, evaluated on a matched comparison set.

w_xgb and w_cnn denote the fusion weights assigned to the vibration and image branch outputs respectively.

The decision-level fusion strategy resulted in lower classification performance than the vibration-only model, with reductions observed in both accuracy and macro F1-score. These findings indicate that the inclusion of the image-based predictions did not improve diagnostic performance under the experimental conditions considered in this study.

The temporal evolution of the Structural Health Index throughout the fifteen-month monitoring period is presented in Figure 19. The SHI clearly differentiates the healthy, damaged, and repaired structural states, exhibiting low values during the healthy period, consistently high values during the damaged period, and substantially reduced values following repair. The corresponding threshold-based performance is summarized in Table 7.

TABLE 7. Digital twin threshold-crossing behavior by true structural state, computed across all 11,328 time steps of the fifteen-month KW51 monitoring period.

The threshold analysis demonstrates that the proposed digital twin correctly identified the repaired period without generating alert-level predictions. During the healthy period, only a small proportion of observations exceeded the alert threshold, whereas the damaged period was predominantly classified as critical. A limited number of false alerts and missed detections were observed, indicating that the proposed SHI provides reliable temporal tracking of structural condition while maintaining a low false-alarm rate.

5.5 Summary of experimental results

The experimental evaluation demonstrates that the proposed framework achieved strong performance in vibration-based structural state classification, image-based defect recognition, and explainable artificial intelligence through SHAP and Grad-CAM (Table 8). The digital twin successfully tracked the temporal evolution of structural condition using the proposed Structural Health Index, providing meaningful differentiation between healthy, damaged, and repaired states. Although the decision-level fusion strategy did not improve classification performance under the present experimental configuration, the results highlight the importance of using paired multi-modal datasets for effective fusion-based structural health monitoring. Overall, the proposed framework demonstrates the feasibility of integrating explainable machine learning and digital twin technology for transparent structural condition assessment.

TABLE 8. Summary of primary results against predefined targets.

6 Discussion

6.1 Interpretation of findings

The experimental results provide several important insights into the strengths and limitations of the proposed explainable multi-modal structural health monitoring framework. First, the gap between the 0.967 cross-validation macro F1 and the 0.682 temporal holdout macro F1 for the vibration classifier is large enough, and consistent enough with the known autocorrelation structure of hourly modal features, that it should be read as evidence of information leakage in the randomly shuffled cross-validation folds rather than as evidence that the model generalizes at the higher figure (Kohavi, 1995; Bergmeir and Benítez, 2012). Reporting the temporal holdout result as the primary metric provides a more defensible estimate of how the classifier would perform on genuinely new, unseen time periods, and future work on this dataset should adopt temporally blocked cross-validation throughout model development rather than only at final evaluation.

Second, the presence of a raw temperature variable among the three most important SHAP features, and the fact that environmental features collectively account for around one-fifth of total feature importance, is consistent with the seasonal overlap between the damaged period and the warmer months of the KW51 monitoring campaign shown in Figure 1. Although frequency-shift features remain the dominant predictors of structural condition, the relatively high importance of environmental variables indicates that residual seasonal effects continue to influence the model predictions despite environmental detrending. The two most important individual features are frequency shift features with a clear physical connection to stiffness loss, and the per-class SHAP summaries in Figures 8, 9 show meaningful contributions from multiple frequency-domain features beyond the environmental terms. It does mean that some fraction of the damaged-class signal is not separable, using the analysis performed in this study, from a seasonal proxy, and this should be disclosed to any engineer relying on the model’s explanations rather than left implicit.

Third, the confusion between the damaged and repaired classes, in which 44.5 percent of true damaged samples were classified as repaired, is a more consequential failure mode than the model’s overall accuracy figure suggests. A false alarm, in which a healthy structure is flagged as damaged, costs an unnecessary inspection. A missed detection, in which a damaged structure is flagged as repaired, risks a structure being treated as safe when it is not, which is the error a structural health monitoring system exists specifically to prevent. The current model does not explicitly account for this asymmetry in its decision thresholds, which were optimized primarily for overall class separation rather than the specific cost of damaged-to-repaired misclassification. For safety-critical deployment, the classifier should therefore not be used to autonomously certify a structure as repaired or safe in its present form. Any Alert- or Critical-level output, or transition toward the Repaired state, should trigger human-in-the-loop verification by a qualified inspector before any clearance or repair decision is made.

The reduced performance of the decision-level fusion strategy represents an important finding of the present study. This outcome is consistent with the absence of sample-level correspondence between the vibration and image datasets, which limits the effectiveness of decision-level fusion under the present experimental configuration. The comparison with a recent study that fused vibration and image data collected from the same physical specimen and reported a substantial accuracy gain (Al-Qudah et al., 2025) is instructive: that result appears to depend on the two modalities being genuinely co-located, which is a precondition the present study does not satisfy. This comparison motivates the recommendation in Section 7 that future fusion work on this framework prioritize collecting or simulating paired multi-modal data from a single structure before any claim of fusion benefit is made. The image branch in this study used EfficientNet-B0 as its backbone; residual network architectures have also been used extensively as backbones in prior Grad-CAM based crack and defect detection studies (He et al., 2016), and a direct comparison between backbone families was outside the scope of this paper but would be a reasonable robustness check in future work.

6.2 Limitations

While the proposed framework demonstrates promising performance, several limitations should be considered when interpreting the present findings.

• The vibration and image branches were developed using independent benchmark datasets collected from different structures. Consequently, the proposed decision-level fusion represents a proof-of-concept rather than a fully validated multi-modal structural health monitoring framework. Future validation should use paired vibration and image measurements acquired from the same structure.
• The vibration analysis was performed using a single bridge monitoring dataset containing one documented damage–repair event. Although the framework demonstrated promising performance, its generalizability to other bridge types, damage mechanisms, and environmental conditions requires further validation using additional benchmark datasets.
• The augmented reality component was proposed as a future extension of the digital twin framework and was not implemented or experimentally evaluated. Accordingly, its effectiveness for real-time visualization and field decision support remains to be investigated.
• The digital twin component was evaluated retrospectively on archived monitoring data rather than deployed on a live sensor stream, and does not include a geometric or finite-element representation of the structure or bidirectional interaction with the physical asset. While the underlying SHI computation is causal and applicable unchanged to streaming data, real-time deployment and integration with a geometric digital twin remain to be demonstrated.

7 Conclusion and future work

This study presented an explainable multi-modal structural health monitoring framework that integrates vibration-based damage diagnosis, image-based defect recognition, explainable artificial intelligence, and digital twin technology for transparent structural condition assessment. The proposed framework combines an XGBoost-based vibration classifier and an EfficientNet-B0 image classifier with SHAP and Grad-CAM explainability techniques, respectively, and integrates their outputs through a Structural Health Index (SHI) for continuous digital twin updating.

The experimental results demonstrate that the proposed framework provides reliable vibration-based structural state classification under temporally independent evaluation while achieving high performance in concrete defect recognition. The explainability analyses confirmed that the vibration classifier relied primarily on physically meaningful modal features, whereas the image classifier consistently focused on relevant defect regions, thereby improving the transparency and interpretability of model predictions. Furthermore, the proposed Structural Health Index successfully distinguished healthy, damaged, and repaired structural conditions within the digital twin environment.

The study also provides several important practical insights. First, temporally independent evaluation offers a more realistic estimate of classifier performance than randomly shuffled cross-validation for long-term structural health monitoring datasets. Second, the results demonstrate that decision-level fusion of independently collected vibration and image datasets does not necessarily improve structural condition assessment, highlighting the importance of physically corresponding multi-modal measurements for effective information fusion. These findings contribute practical guidance for the future development of explainable multi-modal structural health monitoring systems.

Future research should focus on validating the proposed framework using paired vibration and image datasets collected from the same monitored structure, thereby enabling more effective multi-modal fusion. Additional studies should investigate advanced environmental compensation techniques, cost-sensitive learning strategies for reducing safety-critical misclassifications, and adaptive fusion approaches capable of dynamically weighting multiple sensing modalities. The proposed augmented reality visualization layer should also be implemented and experimentally evaluated to investigate its potential for supporting explainable digital twins during field inspection and maintenance activities.

Overall, this work demonstrates the feasibility of integrating explainable artificial intelligence, multi-modal sensing, and digital twin technology within a unified structural health monitoring framework. By combining transparent machine learning models with interpretable engineering decision support, the proposed framework provides a reproducible foundation for the development of trustworthy AI-assisted infrastructure monitoring systems. Although further validation across diverse structures and paired multi-modal datasets is required, the proposed methodology represents a meaningful step toward the practical deployment of explainable digital twins for intelligent infrastructure management.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Pavithra Rathnasiri, Northumbria University, United Kingdom

Reviewed by: Prathap Soma, Northumbria University, United Kingdom

Ezekiel Olatunji, University of Wolverhampton, United Kingdom