Noise-robust temporal–spectral fusion transformers for EEG-based cognitive state classification in aviation environments

1 Introduction

Despite substantial advances in aviation technology and automation, human performance remains a critical determinant of flight safety. Statistics indicate that more than 75% of pilot errors stem from perceptual failures (Jones and Endsley, 1996), highlighting that even highly trained professionals are susceptible to dangerous lapses in attention. These lapses are not random; rather, they follow distinct and recurrent patterns identified in accident investigations. According to the International Air Transport Association (IATA), between 2012 and 2021, there were 45 accidents caused by pilots losing control of the aircraft, resulting in 1,645 fatalities (International Air Transport Association, 2022, 2019). Furthermore, the Commercial Aviation Safety Team (CAST) found that in 16 out of 18 major accidents, attention deficiencies played a role (Commercial Aviation Safety Team, n.d.). Three particularly hazardous Attention-related Pilot Performance Decrements (APPD) have been identified: Channelized Attention (CA), where fixation on a single task causes neglect of other critical information; Diverted Attention (DA), in which competing tasks fragment situational awareness; and Startle/Surprise (SS), where unexpected events induce cognitive paralysis (Yen et al., 2009; Hankins and Wilson, 1998; Boksem and Tops, 2008).

Electroencephalography (EEG) has emerged as a promising modality for detecting such cognitive states before they manifest in overt behavior. Its millisecond temporal resolution allows monitoring of transient neural activity changes indicative of attention deficits (Bigdely-Shamlo et al., 2015; Fló et al., 2022). However, EEG signals are inherently noisy, prone to contamination from environmental and physiological artifacts, and exhibit high inter-subject variability–challenges compounded in real-world aviation environments.

1.1 Prior work in EEG-based pilot state monitoring

Early EEG-based studies relied on manual preprocessing techniques, including band-pass filtering to isolate frequency bands such as beta rhythms (Roza and Postolache, 2019) or broader ranges (0.1—50 Hz) before applying Independent Component Analysis (ICA) (Han et al., 2020). Yet, comparative studies have found that ICA does not always yield significant gains over filtering (Alreshidi et al., 2022). This motivated the development of automated artifact rejection, with Autoreject (Jas et al., 2017)–successfully applied in Bonassi et al. (2021); Pousson et al. (2021)–offering adaptive, channel-specific thresholding and repair.

Feature extraction methods have evolved from basic statistical measures (Harrivel et al., 2017; Hasan et al., 2020) to more complex approaches such as wavelet packet transforms for brain-wave decomposition (Wu et al., 2019) and common spatial pattern filtering (Binias et al., 2018). Beta-wave metrics have proven particularly effective: Ji et al. (2024) analyzed beta energy ratios and Shannon entropy during turning phases, achieving up to 93.67% test accuracy, while Feng et al. (2024) found beta sub-band power (16—30 Hz) to be a robust indicator of situation awareness, reaching 92% classification accuracy.

Riemannian geometry-based methods (Barachant et al., 2012; Majidov and Whangbo, 2019) have further enhanced EEG representation by mapping covariance matrices to the tangent space of a Riemannian manifold, improving generalization across subjects. However, such methods require computationally expensive operations (e.g., matrix logarithms), limiting real-time applicability.

Deep learning approaches have expanded capability by learning joint temporal–spatial patterns. Johnson et al. (2015) benchmarked traditional classifiers, while Han et al. (2020) combined CNN and LSTM modules to detect distraction, workload, fatigue, and normal states. Wu et al. (2019) employed deep contractive autoencoders for fatigue detection, and Harrivel et al. (2016) explored ensembles for predicting APPD states. Nonetheless, many of these methods are constrained to binary or limited multiclass settings, lack integrated temporal–spectral modeling, or underperform under noise.

1.2 Alternative sensing modalities

Eye-tracking has been used to quantify pilot workload, with studies such as Cheng et al. (2024) showing changes in fixation patterns and pupil diameter during emergency procedures. Facial recognition systems (Samy et al., 2025) detect fatigue via facial landmarks and expression analysis, achieving up to 96.8% accuracy under ideal conditions. While these modalities are non-invasive, they cannot directly capture latent neural states and are sensitive to environmental factors.

1.3 Dataset foundations

This study leverages two complementary datasets that jointly enable evaluation under high-density controlled conditions and ecologically realistic scenarios.

Dataset A: Reducing Commercial Aviation Fatalities (Kaggle, n.d.) contains multimodal physiological recordings from 18 professional pilots (nine two-person crews) performing tasks designed to elicit CA, DA, and SS states. Signals include 22-channel EEG (256 Hz), ECG, respiration, and galvanic skin response (GSR). Data are chronologically partitioned into training (56%), validation (8.4%), and test (35.6%) sets to prevent temporal leakage. Four variants—BASE, GAUSS, RAND, and COMBO—simulate real-world noise.

Dataset B: Flight-Deck Mental Workload (FDMW) (Hernández-Sabaté et al., 2024) was recorded with a portable 14-channel EEG system (128 Hz) across three scenarios: N-back (BL, LW, MW, HW), Heat-the-Chair (BL, LW, HW), and Flight Simulator (BL, E, M, MH, H). Scenario-specific splits avoid temporal leakage, and the same four noise variants are applied while preserving class balance.

1.4 Research gaps

Despite progress, current EEG-based aviation monitoring systems face persistent issues: (i) limited ability to generalize across pilots and environments; (ii) inadequate integration of temporal evolution and spectral structure; (iii) narrow state coverage, often excluding the full APPD spectrum; (iv) sensitivity to noise; and (v) computational demands that hinder real-time deployment. Existing methods frequently require extensive calibration and often lack interpretability–an essential requirement for safety-critical adoption.

1.5 Contributions

The primary contributions of this work are as follows:

• We introduce a temporal—spectral fusion transformer (TF-T) architecture tailored for EEG-based multiclass cognitive state recognition in aviation contexts. The framework integrates multi-scale preprocessing, dual-stream temporal and spectral feature extraction, and enhanced temporal–spectral fusion framework to jointly capture cross-frequency relationships and inter-channel dependencies.
• We design a progressive architecture evolution across three configurations (TF-T1, TF-T2, TF-T3), systematically incorporating enhancements such as progressive architectural refinements including multi-resolution temporal processing and enhanced temporal–spectral feature integration to address robustness under real-world noise conditions.
• We establish a rigorous, ecologically valid evaluation protocol using two complementary datasets—Dataset A (high-density, controlled elicitation of CA, DA, SS) and Dataset B (portable EEG, realistic workload scenarios across N-back, HtC, and Flight Simulator tasks)—with multiple noise-augmented variants to simulate operational perturbations.
• We provide a comprehensive analysis of classification behavior, including per-class sensitivity, robustness curves under graded noise levels, and an error taxonomy that distinguishes benign from harmful misclassifications, offering interpretable insights for operational deployment.

1.6 Performance highlights

In extensive experiments, TF-T2 achieves the highest clean-data accuracy (99.2% on Dataset A), while TF-T3 delivers consistent robustness advantages under all noise types, improving Macro-F1 by approximately 4.5—4.7 points compared to TF-T2. Against state-of-the-art baselines, TF-T3 improves Macro-F1 by up to +8 points under COMBO noise and reduces temporal instability by ≈35%.

1.7 Paper structure

The remainder of this paper is organized as follows: Section 2 details the proposed method; Section 3 describes the datasets; Section 4 presents experimental design choices; Section 5 discusses findings, limitations, and implications; and Section 6 concludes the work.

2 Proposed method

We propose a three-stage framework for EEG-based cognitive state classification that jointly models temporal and spectral neural dynamics via a transformer encoder. The architecture is specifically designed to address two major challenges in this domain: (i) the high susceptibility of EEG to non-neural artifacts, and (ii) class imbalance arising from the unequal prevalence of cognitive states. The framework operates as follows: Stage 1 applies multi-scale preprocessing to suppress artifacts while preserving physiological rhythms; Stage 2 extracts complementary time- and frequency-domain descriptors and projects them into a unified latent representation; and Stage 3 integrates these features using multi-head self-attention to capture cross-channel dependencies and modality interactions. The complete workflow is illustrated in Figure 1. Equations 1–44 will describe the proposed models mathematics.

2.1 Notation and conventions

All variables and symbols used in the following sections are listed in Table 1. Where a symbol is reused in different contexts, its meaning is explicitly specified to avoid ambiguity.

Table 1. Notation and conventions used in the proposed method.

Symbols with multiple contextual meanings are explicitly described.

2.2 Stage 1: multi-scale pre-processing

The first stage ensures that the raw EEG is free from major artifacts and standardized across channels, forming a stable basis for feature extraction. The continuous EEG recording X is segmented into non-overlapping one-second epochs. Let f_*s* denote the sampling frequency and w = f_*s* the epoch length in samples. The i-th epoch is defined as

Ei=X[:,,i·w:(i+1)·w], i=0,1,…,⌊Tw⌋-1,(1)

yielding temporally localized EEG segments that preserve the original temporal structure while avoiding redundancy between adjacent epochs.

Each epoch undergoes FIR bandpass filtering to retain neural oscillations between 1–50 Hz:

Eif[c,n]=∑k=0L-1h[k]·Ei[c,n-k],(2)

thus removing slow drifts and high-frequency noise without distorting relevant rhythms.

Artifact rejection is then applied using Autoreject, which learns channel-specific thresholds θ_*c* and flags corrupted segments:

D(Eif[c,:],θc)={1,max(|Eif[c,:]|)>θc,0,otherwise.(3)

Corrupted channels are reconstructed by spherical spline interpolation from spatial neighbors:

Eir[c,n]={∑j∈Ncwj·Eif[j,n],D=1,Eif[c,n],otherwise,(4)

with ∑jwj=1.

Finally, z-score normalization is performed channel-wise:

Einorm[c,:]=Eir[c,:]-μcσc,(5)

ensuring amplitude comparability across channels and epochs.

The FIR filter was designed using a linear-phase windowed-sinc formulation with passband limits of 1–50 Hz. Artifact rejection was performed using the Autoreject framework with channel-specific threshold estimation. Channel normalization was applied independently for each channel using z-score standardization. To prevent information leakage, preprocessing parameters were estimated using training data and subsequently applied to validation and test partitions.

2.3 Stage 2: dual-stream feature extraction and fusion

The second stage derives two complementary feature sets. The temporal stream directly preserves normalized waveforms:

Fitemp=Einorm.(6)

The spectral stream captures frequency-specific power distributions via Welch's method:

Pic(f)=1K∑j=0K-1|∑n=0M-1Einorm[c,jR+n]v[n]e-i2πfn/M|2.(7)

Band powers are obtained by summing over canonical frequency ranges:

BPic,b=∑f∈[fminb,fmaxb]Pic(f)·Δf,(8)

forming the spectral feature matrix Fispec.

Temporal and spectral features are concatenated channel-wise and projected into a shared latent space:

Fifused=Wfuse·[Fitemp;Fispec]+bfuse.(9)

This produces C channel tokens, each of dimension d_*model*, aligned for joint modeling.

Although Dataset A and Dataset B differ in channel count and sampling frequency, the preprocessing and feature extraction stages generate fixed-dimensional embeddings before transformer processing. Consequently, the transformer backbone remains unchanged across datasets, while only the input representation dimensions are adapted during feature encoding.

2.4 Stage 3: transformer-based temporal–spectral integration

The third stage integrates the fused channel tokens using a transformer encoder. To encode channel ordering, sinusoidal positional embeddings are added:

PE(pos,2i)=sin(pos100002i/dmodel),(10)

PE(pos,2i+1)=cos(pos100002i/dmodel),(11)

yielding Z0=Ffused+PE.

Within each transformer layer, multi-head self-attention computes:

Qh=Zl-1WhQ, Kh=Zl-1WhK, Vh=Zl-1WhV,(12)

Attention(Q,K,V)=softmax(QK⊤dk)V.(13)

Outputs from all heads are concatenated:

MultiHead=Concat(head1,…,headH)WO.(14)

Residual connections and position-wise feed-forward networks complete each layer:

Zl′=LayerNorm(Zl-1+MultiHead(Zl-1)),(15)

Zl=LayerNorm(Zl′+FFN(Zl′)),(16)

with

FFN(Z)=max(0,ZW1+b1)W2+b2.(17)

After L layers, channel tokens are averaged and passed to a softmax classifier:

pi=softmax(Wcls·AvgPool(ZL)+bcls).(18)

2.5 Training objective and imbalance mitigation

The cross-entropy loss is:

L=-1N∑i=1N∑j=1Kyi,jlogpi,j.(19)

To mitigate class imbalance, inverse-frequency weights are applied:

wj=NNj·K,(20)

resulting in the weighted loss:

Lweighted=-1N∑i=1N∑j=1Kwjyi,jlogpi,j.(21)

For completeness, the baseline transformer optimization strategy can be expressed using Adam with an inverse-square-root learning-rate schedule:

η=dmodel-0.5·min(step-0.5,step·warmup_steps-1.5).(22)

The specific optimizers and learning-rate policies used for TF-T1, TF-T2, and TF-T3 during experimental evaluation are provided in Section 4.1.4. The transformer model is illustrated on Figure 2.

3 Dataset details

This study leverages two independent and complementary datasets: the Reducing Commercial Aviation Fatalities dataset (Kaggle, n.d.) and the Flight-Deck Mental Workload (FDMW) dataset (Hernández-Sabaté et al., 2024). Both datasets provide high-resolution physiological recordings acquired under systematically varied cognitive and operational conditions, enabling a multi-faceted analysis of neural and physiological responses to mental workload and situational stress in aviation contexts. The subsequent subsections describe each dataset in detail, including their structure, experimental protocols, segmentation strategies, and the generation of noise-augmented variants.

3.1 Dataset A: the reducing commercial aviation fatalities dataset

The Reducing Commercial Aviation Fatalities dataset, released as part of the Booz Allen Hamilton Kaggle competition (Kaggle, n.d.), contains multimodal physiological measurements acquired from eighteen professional pilots, organized into nine two-person crews. Data were recorded under three controlled experimental paradigms designed to elicit distinct attentional and affective states. The Channelized Attention (CA) condition involved sustained engagement in a puzzle-based video game, eliciting focused attention with minimal external distraction. The Diverted Attention (DA) condition required participants to perform monitoring tasks intermittently interrupted by unanticipated mathematical problems, thereby inducing rapid attention switching. The Startle/Surprise (SS) condition exposed pilots to emotionally startling video clips containing abrupt “jump-scare” events, eliciting transient high-arousal responses.

Physiological signals were sampled at 256 Hz and included 22-channel EEG, three-lead electrocardiography (ECG), respiration, and galvanic skin response (GSR), all calibrated in microvolts. The dataset is organized into two main partitions: a benchmark training set, collected during non-flight laboratory experiments, and a test set, recorded during high-fidelity full-flight simulation scenarios, specifically Line Oriented Flight Training (LOFT).

For the purposes of this study, we constructed chronological training, validation, and testing partitions from the labeled recordings available in the dataset. These partitions were created independently of the original competition structure and were designed to support systematic evaluation under temporally separated conditions. Consequently, the train, validation, and test subsets reported in Tables 2, 3 correspond to the experimental protocol adopted in this work rather than the official benchmark partitions distributed with the original dataset.

Table 2. Aggregate segment counts for each experimental condition.

Table 3. Estimated per-class segment distribution in the FDMW dataset.

The complete dataset spans approximately 5.28 h of recordings, segmented into 18,888 non-overlapping 1-s windows (256 samples each). To avoid temporal leakage and to emulate operational deployment scenarios, the data were partitioned chronologically: 56% of segments from the initial hour were allocated for training, 8.4% from the final 15 min of the training period were reserved for validation, and the remaining 35.6%—originating entirely from unseen temporal segments—were used for testing. Table 2 presents the aggregated segment counts for each experimental condition prior to any augmentation. Because the dataset consists of continuous recordings collected from the same pilot crews over extended sessions, the chronological partitioning strategy does not enforce subject-level separation. Instead, temporal separation was prioritized to prevent leakage from future observations into model development stages and to emulate operational deployment conditions.

Details are illustrated in Figure 3.

To evaluate robustness against measurement perturbations, four dataset variants were constructed. The BASE variant corresponds to the unmodified dataset. The GAUSS variant incorporates additive zero-mean Gaussian noise with a standard deviation proportional to the instantaneous signal amplitude. The RAND variant adds uniform random noise with amplitude-independent distribution. Finally, the COMBO variant is a composite of 60% BASE, 20% GAUSS, and 20% RAND data, balanced across all experimental conditions. Importantly, all variants preserve the total segment counts and class distributions shown in Table 2.

Noise perturbations were applied independently to each EEG channel after preprocessing and normalization. Gaussian noise was generated using a zero-mean distribution with variance scaled relative to the channel amplitude, whereas RAND noise was sampled from a zero-mean uniform distribution. The COMBO condition combined clean and perturbed samples according to the proportions described above. These perturbations were introduced solely for robustness assessment and were designed to provide controlled and reproducible evaluation conditions across all model variants.

3.2 Dataset B: the Flight-Deck Mental Workload (FDMW) dataset

The Flight-Deck Mental Workload (FDMW) dataset (Hernández-Sabaté et al., 2024) (Figure 4) comprises EEG recordings obtained during simulated aviation tasks explicitly designed to induce varying levels of mental workload. Data acquisition employed the Emotiv Epoc X wireless EEG system, featuring 14 electrodes positioned according to the international 10—20 system. The headset sampled at 128 Hz, capturing both raw microvolt-level EEG voltages and power spectral density (PSD) estimates across five canonical frequency bands: θ, α, β_low, β_high, and γ.

The experimental protocol encompassed three distinct scenarios. The first, N-back, consisted of a series of working-memory and arithmetic tasks with systematically increasing difficulty, yielding four workload classes: Baseline (BL), Low Workload (LW), Medium Workload (MW), and High Workload (HW). The second, Heat-the-Chair (HtC), was a multitasking game performed with or without simulated Air Traffic Control (ATC) interruptions, generating three workload classes: BL, LW, and HW. The third, Flight Simulator (FS), involved professional pilots executing mission scenarios of varied operational complexity, categorized into Baseline (BL), Easy (E), Medium (M), Medium-Hard (MH), and Hard (H) levels.

Recordings were segmented into 1-second non-overlapping epochs (128 samples per channel). Data splitting strategies were tailored to each scenario to minimize temporal leakage: for N-back and HtC, temporal block partitioning or leave-one-subject-out cross-validation was used, whereas FS scenario data were partitioned at the mission level, ensuring that no mission was split across training, validation, and test sets. Table 3 presents the estimated per-class segment counts and their respective allocations across the three dataset partitions.

Analogous to Dataset A, four noise-augmented variants of the FDMW dataset were generated. The BASE variant represents the original EEG recordings. The GAUSS variant incorporates zero-mean Gaussian noise scaled proportionally to signal amplitude, while the RAND variant applies amplitude-independent uniform random noise. The COMBO variant integrates 60% BASE, 20% GAUSS, and 20% RAND data, ensuring balanced representation within each workload class. These augmentations preserve the exact per-class distributions and train/validation/test proportions shown in Table 3, ensuring that any observed differences in classification performance can be attributed solely to the presence and type of noise rather than to distributional shifts. Similar to Dataset A, noise perturbations were applied independently to each EEG channel after preprocessing and normalization. Gaussian noise was generated using a zero-mean distribution with variance scaled relative to the channel amplitude, whereas RAND noise was sampled from a zero-mean uniform distribution. The COMBO condition combined clean and perturbed samples according to the proportions described above. These perturbations were introduced solely for robustness assessment and were designed to provide controlled and reproducible evaluation conditions across all model variants.

4 Experimental design

4.1 Model architecture evolution

To systematically optimize classification performance for EEG-based pilot cognitive state detection, we developed three progressively enhanced variants of our TF-T architecture. Each successive model incorporates targeted architectural modifications aimed at addressing specific challenges inherent to EEG decoding in operational aviation contexts, namely: (i) the need to model multi-scale temporal–spectral dynamics under noisy conditions, and (ii) the mitigation of inter-channel variability and cross-frequency coupling effects. Table 4 summarizes the architectural progression from the baseline configuration (TF-T1) to the advanced configuration (TF-T3), highlighting the incremental modifications in input representation, attention mechanisms, feature integration, and optimization strategies.

Table 4. Progressive evolution of the temporal–spectral fusion transformer architecture across three variants.

Each stage introduces targeted enhancements over its predecessor.

Rigorous ablation studies and cross-variant comparisons were performed to quantify the contribution of each architectural enhancement to the final performance gains. This iterative refinement process culminated in the TF-T3 variant, which exhibited consistently superior classification accuracy and robustness, particularly under conditions of simulated sensor noise and electrode displacement.

All three proposed variants share the same TF-T backbone described in Section 2. The progression from TF-T1 to TF-T3 should therefore be interpreted as a sequence of architectural refinements rather than fundamentally different model formulations. TF-T1 represents the baseline implementation, TF-T2 introduces enhanced feature representation and attention mechanisms, and TF-T3 incorporates additional robustness-oriented design elements intended to improve performance under noisy operating conditions. Table 4 summarizes the principal architectural differences between the three variants.

4.1.1 Baseline model (TF-T1)

The TF-T1 variant serves as the foundational implementation of temporal–spectral fusion, in which raw EEG segments (256 samples per channel) are concatenated with power spectral density (PSD) features from five canonical bands (δ, θ, α, β, γ). This produces an input matrix of size 256 × (C×6), where C is the number of channels. The architecture employs a 2-layer transformer encoder with 4 attention heads and a hidden dimension of 256, followed by average pooling and a fully connected classification layer. While minimal in complexity, TF-T1 effectively establishes the baseline capability to capture both temporal dependencies and spectral signatures.

4.1.2 Enhanced model (TF-T2)

TF-T2 extends TF-T1 by expanding the transformer encoder to 4 layers with 8 attention heads, increasing its representational capacity. Enhancements include channel-aware encoding, frequency-specific attention weighting, and hierarchical pooling to capture inter-channel and cross-frequency dependencies more effectively. The model also replaces fixed positional encodings with learned embeddings, allowing adaptive modeling of spatial–temporal relationships.

4.1.3 Advanced model (TF-T3)

The TF-T3 variant represents the culmination of iterative refinement, integrating six transformer layers with 12 attention heads and multi-resolution temporal processing. The model emphasizes richer temporal–spectral interaction, enhanced contextual representation learning, and improved robustness to noise and cognitive-state variability. The classification stage employs an ensemble of three fully connected heads with class-balanced focal loss, improving resilience to dataset imbalance.

4.1.4 Implementation details

All models were implemented using the PyTorch framework and trained on a workstation equipped with an NVIDIA RTX-series GPU, 64 GB RAM, and Intel Core i9-13900K processor, and 64 GB RAM. To improve experimental reproducibility, fixed random seeds were used for Python, NumPy, and PyTorch initialization. Training was performed using mini-batch optimization. Models were trained for a maximum of 100 epochs with early stopping based on validation Macro-F1 using a patience of 15 epochs. The model corresponding to the best validation performance was retained for final testing. TF-T1 was trained using the Adam optimizer with an initial learning rate of 1 × 10⁻⁴. TF-T2 employed AdamW with cosine learning-rate scheduling and weight decay. TF-T3 utilized the Lion optimizer with a one-cycle learning-rate policy. Gradient clipping was applied to improve training stability. All reported performance metrics were computed exclusively on the held-out test partition. Hyperparameter selection was performed using training and validation data only.

4.2 Performance evaluation strategy

To rigorously evaluate the proposed TF-T architectures, we designed a comprehensive protocol that exactly mirrors the scope of analysis reported in Section 5. The strategy is tailored to the operational constraints of real-time EEG-based pilot cognitive state monitoring, accounting for (i) class imbalance, (ii) high inter-subject variability, and (iii) the need for robustness under diverse noise conditions.

4.2.1 Temporal validation protocol

In contrast to conventional random cross-validation, we adopt a strictly chronological partition to preserve temporal dependencies and prevent information leakage:

D={Dtrain,Dval,Dtest},(23)

where Dtrain is the first 56% of segments, Dval the next 8.4%, and Dtest the final 35.6%.

This emulates deployment: inference must be performed on unseen future data. The adopted evaluation protocol is strictly chronological rather than subject-independent. Consequently, participants may contribute recordings to multiple partitions, although temporal overlap between partitions is not permitted. The objective of this design is to evaluate generalization to future unseen observations while preserving the temporal structure of the original recordings. No samples from validation or test periods were used during model training or hyperparameter selection.

4.2.2 Evaluation metrics

Let the set of K classes be Y={1,…,K}, with K = 3 for Dataset A (CA, DA, SS) and K∈{3, 4, 5} for Dataset B sub-datasets: N-back (BL, LW, MW, HW), HtC (BL, LW, HW), FS (BL, E, M, MH, H).

4.2.3 Confusion matrix

We define C∈ℕ^*K*×*K* with entries:

cij=∑n=1N1(yn=i)·1(ŷn=j),(24)

where y_*n* is the true label and ŷ_*n* the predicted label.

4.2.4 Accuracy

Overall classification accuracy is:

Acc=∑i=1Kcii∑i=1K∑j=1Kcij.(25)

4.2.5 Per-class precision, recall, and F1

For each class i∈Y:

Preci=cii∑j=1Kcji,(26)

Reci=cii∑j=1Kcij,(27)

F1i=2·Preci·ReciPreci+Reci.(28)

4.2.6 Macro-averaged metrics

Precmacro=1K∑i=1KPreci,(29)

Recmacro=1K∑i=1KReci,(30)

F1macro=1K∑i=1KF1i.(31)

4.2.7 Weighted-averaged metrics

F1weighted=∑i=1KNi·F1i∑i=1KNi,(32)

where N_*i* is the number of samples in class i.

4.2.8 ROC-AUC and PR-AUC

One-vs-all ROC-AUC for class i is:

ROC-AUCi=∑n:yn=i∑m:ym≠i1(pn,i>pm,i)Ni·(N-Ni),(33)

with macro-averaging:

ROC-AUCmacro=1K∑i=1KROC-AUCi.(34)

PR-AUC is computed analogously from the precision–recall curve.

4.2.9 Confidence intervals and significance testing

All metrics are reported with 95% confidence intervals computed via non-parametric bootstrap on Dtest:

CI95%=[M(0.025),M(0.975)],(35)

where M^(*q*) is the q-th quantile of the bootstrap distribution.

4.2.10 Comparative model analysis

Each architecture is evaluated on identical splits. Reported differences in macro-F1, ROC-AUC, PR-AUC, and per-class F1 are accompanied by:

ΔM=Mvariant-Mbaseline,(36)

with significance and CIs as above.

All comparative baseline models were evaluated using the same preprocessing procedures, chronological train-validation-test partitions, and noise-generation protocols employed for the proposed transformer architectures. Baseline configurations were selected according to the corresponding original publications whenever possible. Performance comparisons therefore reflect differences in model architecture rather than differences in data preparation or evaluation methodology.

4.2.11 Robustness assessment

We evaluate resilience under three perturbations to Dtest:

• GAUSS: additive Gaussian noise
• RAND: amplitude-independent uniform noise
• COMBO: composite 60% BASE, 20% GAUSS, 20% RAND per class.

Robustness is quantified as:

R(l)=F1macro,noisy(l)F1macro,BASE.(39)

We additionally perform Gaussian sweeps to characterize degradation curves.

4.2.12 Computational efficiency

Efficiency is evaluated by:

FLOPs=∑ℓ=1LFLOPsℓ,(40)

Params=∑ℓ=1LParamsℓ,(41)

Tinf=1N∑n=1Ntn,(42)

Throughput=NsamplesTepoch.(43)

These metrics balance predictive performance against real-time deployability.

4.2.13 Error taxonomy

We construct confusion matrices for each setting and extract the top-k off-diagonal pairs:

Ei→j=cij∑m=1Kcim, i≠j.(44)

Comparing BASE vs. COMBO highlights shifts toward benign or harmful confusions under noise.

This strategy ensures that all analyses—per-class metrics, aggregate scores, robustness ratios, confidence intervals, noise sweeps, computational trade-offs, and error taxonomies—are computed in a manner directly comparable across datasets, architectures, and conditions, matching the scope of the results in Section 5.

5 Result analysis and discussion

5.1 Evaluation protocol and statistical testing

We evaluate the three transformer variants (TF-T1, TF-T2, TF-T3) on two datasets: Dataset A (Reducing Commercial Aviation Fatalities; classes {CA, DA, SS}) and Dataset B (FDMW; sub-datasets N-back {BL, LW, MW, HW}, HtC {BL, LW, HW}, and Flight Simulator (FS) {BL, E, M, MH, H}). We report Accuracy, Macro-F1, ROC-AUC, PR-AUC, and efficiency metrics (FLOPs, per-epoch inference latency T_inf). Robustness is assessed under GAUSS, RAND, and COMBO noise variants.

5.2 Architecture selection on clean (BASE) data

Tables 5, 6 compare TF-T1/2/3 on the BASE splits. Across both datasets, TF-T2 delivers the strongest overall performance with favorable efficiency, including the global peak accuracy of 99.2% on Dataset A.

Table 5. Architecture comparison on dataset A (BASE).

Best per column in bold.

Table 6. (Architecture comparison on Dataset B/FDMW (BASE, macro over N-back, HtC, FS).

Best in bold.

5.3 Performance–complexity trade-off analysis

Effect size vs. cost (Tables 5, 6). TF-T2 improves Macro-F1 over TF-T1 by +2.7–2.8 points for a +50% FLOPs increase, while TF-T3 adds another +46% FLOPs without surpassing TF-T2 on clean signals—consistent with mild over-parameterization under low-noise conditions. Calibration. PR-AUC tracks ROC-AUC closely, suggesting well-calibrated posteriors. Deployability. The +1.5 ms latency gap between TF-T2 and TF-T3 is material for real-time deployments.

5.4 Robustness under noise and perturbations

Tables 7, 8 summarize Macro-F1 (absolute) and robustness ratios under GAUSS, RAND, and COMBO noise. As anticipated, TF-T3 provides a ~5% absolute Macro-F1 bump over TF-T2 across noise types and datasets, with the largest gaps under COMBO.

Table 7. Dataset A: Macro-F1 under noise and robustness ratio R = F1_noisy/F1_BASE.

Best in bold.

Table 8. Dataset B/FDMW: Macro-F1 under noise and robustness ratio R.

Best in bold.

5.5 Robustness analysis under noise

Bump magnitude (Tables 7, 8). TF-T3 improves Macro-F1 over TF-T2 by +4.5–+4.7 points across noise types (all p < 0.001). Failure mode of TF-T2. Largest degradation appears in COMBO, evidencing sensitivity to mixed perturbations. Mechanistic view. TF-T3's enhanced temporal–spectral representation learning and multi-resolution processing appear particularly effective in maintaining performance under mixed perturbations.

5.6 Per-class performance

We now provide the requested per-class breakdowns for Dataset A (CA/DA/SS) and all FDMW sub-datasets (N-back, HtC, FS). These are presented in Tables 9–12.

Table 9. Dataset A (CA/DA/SS), BASE: per-class performance by model.

Best per class in bold. Macro avg. is in italic.

Table 10. FDMW N-back (BASE): per-class performance.

Best per class in bold. Macro avg. is in italic.

Table 11. FDMW HtC (BASE): per-class performance.

Best per class in bold. Macro avg. is in italic.

Table 12. FDMW Flight Simulator (BASE): per-class performance.

Best per class in bold. Macro avg. is in italic.

5.7 Class-specific performance analysis

Dataset A. TF-T2 dominates CA/DA/SS; the tightest margin is on SS, where TF-T3 is closest (F1: 0.988 vs. 0.983), reflecting SS's phasic nature. FDMW. TF-T2's largest gains appear in higher workload classes (N-back: HW; HtC: HW; FS: MH/H), indicating improved sensitivity to cognitively demanding states. Precision–recall symmetry suggests consistent thresholding across classes. Please check the details from Tables 9–12.

5.8 Per-class confidence intervals (bootstrap, B = 1, 000)

Statistical significance is determined via non-parametric bootstrap on the test split with B = 1, 000 resamples; we report two-sided p-values for pairwise model differences and 95% confidence intervals (CIs) for per-class F1 (Tables 13–16). Robustness ratios are defined as R = Macro-F1_noisy/Macro-F1_BASE.

Table 13. Dataset A per-class F1, 95% CI (bootstrap, B = 1, 000).

Table 14. FDMW N-back per-class F1, 95% CI.

Table 15. FDMW HtC per-class F1, 95% CI.

Table 16. FDMW FS per-class F1, 95% CI.

5.9 Noise scaling: Gaussian sweep

Table 17 shows Macro-F1 vs. noise level σ; TF-T3 dominates beyond mild noise, and the gap increases with σ.

Table 17. Gaussian noise sweep: Macro-F1 vs. σ (datasets A and B).

Bold represents superior performance.

5.10 Compute footprint and throughput

Table 18 summarizes model size and runtime. While TF-T3 is the most expensive, its robustness advantage (Tables 7, 8) justifies deployment under noisy conditions; TF-T2 remains preferred for clean environments.

Table 18. Compute footprint and throughput (averaged across datasets).

5.11 Error taxonomy and confusion hot-spots

We analyze off-diagonal confusion frequencies to expose systematic errors (Table 19). On Dataset A, the dominant confusion is DA→CA; on FDMW/FS, M ↔ MH is the most persistent borderline.

Table 19. Top-3 off-diagonal confusions (% of true class) per setting for TF-T2 (BASE) and TF-T3 (COMBO).

5.12 Interpretation

A: DA vs. CA. Transient attention shifts in DA resemble CA microstates, explaining asymmetric DA → CA confusion. FS: M vs. MH. Adjacent scenario difficulty induces small but persistent boundary errors. Under COMBO, TF-T3 shifts errors toward neighboring classes (benign confusions), consistent with robust representation smoothing rather than collapse.

5.13 Selection summary and practical guidance

• Clean (BASE) data: TF-T2 is preferred. It achieves the highest observed accuracy of 99.2% on Dataset A (Table 5) and leads across per-class metrics (Tables 9–12).
• Noisy/robust settings (GAUSS, RAND, COMBO): TF-T3 is recommended, providing a consistent ~5% Macro-F1 bump over TF-T2 and superior robustness ratios (Tables 7, 8), with widening gains as noise increases (Table 17).
• Compute trade-offs: TF-T2 offers the best accuracy–efficiency trade-off for routine monitoring (Table 18), while TF-T3 is justified in environments with persistent perturbations.

5.14 Limitations

Per-class CIs narrow with large supports (e.g., N-back) and widen for FS classes with fewer segments (Table 16), which may modestly inflate variance of per-class estimates. Future work will explore adaptive thresholding and cost-sensitive training to further mitigate boundary confusions (Table 19). A further limitation is that the present evaluation protocol focuses on temporal generalization and does not explicitly assess subject-independent or crew-independent generalization. Future work should investigate leave-one-subject-out and leave-one-crew-out validation strategies to better quantify cross-subject robustness in operational aviation environments.

5.15 Comparative analysis with state-of-the-art approaches

For fairness, all baseline models were evaluated under the same experimental framework used for the proposed architectures, including identical preprocessing, temporal partitioning strategy, and noise-augmentation conditions. This ensured that comparative performance differences were attributable primarily to model design rather than experimental protocol variations. We evaluated our final recommended model (TF-T3) against preceding transformer variants (TF-T1, TF-T2), Riemannian geometry-based classifiers (Alreshidi et al., 2023; Binias et al., 2018), and alternative deep learning baselines across both Dataset A (CA/DA/SS) and Dataset B (N-back, HtC, FS) under all noise conditions (GAUSS, RAND, COMBO). While TF-T3 consistently outperforms comparators under every perturbation type, we emphasize the COMBO setting in this section as it most closely approximates real-world operational conditions, incorporating both amplitude-dependent and amplitude-independent noise sources in a balanced fashion. The details are listed in Tables 20–24.

Table 20. Macro-level COMBO performance comparison of TF-T3 with baselines on Dataset A and Dataset B (macro over subdivisions).

Best values in bold.

Table 21. Class-specific F1-scores for Dataset A (CA/DA/SS) under COMBO noise.

Values are consistent with macro-F1 trends from Table 20. Bold represents superior performance.

Table 22. Class-specific F1-scores for Dataset B/N-back (BL/LW/MW/HW) under COMBO noise.

Values reflect macro-F1 in Table 20 with largest gains in HW. Bold represents superior performance.

Table 23. Class-specific F1-scores for Dataset B/HtC (BL/LW/HW) under COMBO noise.

Gains are uniform across classes. Bold represents superior performance.

Table 24. Class-specific F1-scores for Dataset B/FS (BL/E/M/MH/H) under COMBO noise.

Largest gains occur in MH and H. Bold represents superior performance.

Key Observations:

1. Noise-Robust Performance Gains Across All Datasets: Under COMBO noise, TF-T3 outperforms TF-T2 by +4.5 points Macro-F1 on Dataset A and +4.6 points on Dataset B, with stability coefficients reduced by ≈35%.

2. Subdivision-Level Trends (Figures 5–7): - N-back: Largest gain in HW (+4.8 points F1), reflecting stronger modeling of high-load working memory states. - HtC: Uniform improvements across all classes (≈+4.7 points). - FS: Largest gains in MH (+5.4 points) and H (+5.2 points), corresponding to the most complex operational scenarios.

3. Consistency Across Noise Types: Although we focus on COMBO here, TF-T3 also leads under GAUSS and RAND noise for all datasets, with absolute Macro-F1 improvements of +4.5–5.0 points over TF-T2.

4. Superiority Over Feature-Based and Alternative Architectures: Compared to the best Riemannian ensemble baseline (Alreshidi et al., 2023; Binias et al., 2018), TF-T3 improves Macro-F1 by +7–8 points in COMBO and maintains superior temporal stability, confirming that architectural advances–TF-T3's enhanced temporal–spectral representation learning and multi-resolution processing appear particularly effective in maintaining performance under mixed perturbations.

5. Deployment Relevance: The COMBO scenario is the most challenging mixed-noise evaluation condition for in-flight EEG monitoring, as it simulates the simultaneous presence of multiple noise types. TF-T3's ability to sustain high accuracy and stability in this setting underscores its readiness for operational deployment.

5.16 Discussion

This study presents a systematic investigation into EEG-based cognitive state classification for aviation contexts, integrating methodological innovations, rigorous evaluation protocols, and extensive robustness testing. By jointly analyzing the dataset characteristics, model design choices, experimental findings, and comparative outcomes, several key themes emerge regarding performance determinants, practical viability, and broader implications.

5.17 Dataset considerations and ecological validity

Two complementary datasets–Dataset A (Reducing Commercial Aviation Fatalities) and Dataset B (Flight-Deck Mental Workload)–differ in electrode density, sampling rate, and operational realism. Dataset A's high-density 22-channel EEG offers rich spatial resolution and controlled elicitation of attentional and affective states, while Dataset B's portable 14-channel system captures ecologically relevant workload variations across diverse aviation scenarios. Chronological partitioning mitigates temporal leakage, while noise-augmented variants (GAUSS, RAND, COMBO) emulate real-world perturbations. The COMBO setting, combining amplitude-dependent and amplitude-independent noise, is especially valuable for operational readiness testing.

5.18 Architectural contributions and design rationale

The TF-T architecture evolves through three variants (TF-T1, TF-T2, TF-T3), progressively enhancing representational capacity, attention complexity, and regularization. TF-T3 introduces additional representational capacity, multi-resolution temporal processing, and enhanced temporal–spectral interaction mechanisms aimed at improving robustness under noisy conditions. Weighted loss functions consistently mitigate class imbalance.

5.19 Performance trends: clean vs. noisy conditions

Under clean conditions, TF-T2 achieves the highest observed accuracy (99.2% on Dataset A) with the best FLOPs–latency trade-off, while TF-T3 does not surpass it due to mild over-parameterization. Under noise, TF-T3 consistently outperforms TF-T2 by ~4.5–4.7 Macro-F1 points across GAUSS, RAND, and COMBO, with widening gaps at higher noise levels, evidencing superior robustness to spectral drift and transient channel corruption.

5.20 Class-wise sensitivity

Per-class results show that TF-T2 dominates on clean data, with TF-T3 narrowing the gap for transient, phasic states (e.g., SS in Dataset A). In Dataset B, TF-T3 gains are concentrated in higher workload classes (HW, MH, H), which present overlapping temporal–spectral profiles that benefit from multi-resolution and cross-domain modeling.

5.21 Comparative standing against state-of-the-art

Against Riemannian geometry-based classifiers and deep learning baselines, TF-T3 delivers superior Macro-F1 and temporal stability under COMBO noise, with margins up to +8 points over the best feature-based ensemble. These improvements are attributable to the combined effects of multi-resolution attention, temporal–spectral integration, and explicit modeling of state dynamics.

5.22 Computational considerations

TF-T3's complexity increases inference time by ≈47% relative to TF-T2, reducing throughput from 189 to 147 epochs/s. The trade-off is justified in high-perturbation environments where robustness is paramount, but TF-T2 remains preferable for efficiency-critical, low-noise monitoring.

5.23 Error taxonomy and decision boundaries

Error analysis shows that most confusions occur between adjacent cognitive states (e.g., DA → CA; M↔MH), with TF-T3 under noise shifting misclassifications toward such benign neighbors rather than distant classes–indicating robustness via representation smoothing rather than indiscriminate compression.

5.24 Limitations and future directions

Low-support classes in Dataset B/FS yield wider confidence intervals, modestly increasing per-class estimate variance. COMBO noise captures multiple perturbation types but does not encompass all in-flight interference patterns (e.g., persistent channel dropout). Future work should integrate adaptive thresholding, cost-sensitive training, and multimodal sensor fusion to further enhance resilience. Although the adopted perturbation protocol captures several common forms of signal corruption, it does not explicitly model all aviation-relevant EEG artifacts, including electrode displacement, motion-related contamination, persistent channel dropout, or impedance fluctuations.

5.24.1 Key takeaways

• Dataset Complementarity and Ecological Validity: The dual-dataset design enables both upper-bound performance assessment (Dataset A) and deployment-relevant robustness evaluation (Dataset B). Chronological partitioning and noise-augmented variants ensure operational realism.
• Architecture–Performance Trade-offs: TF-T2 offers the optimal clean-data accuracy–efficiency balance; TF-T3's higher complexity is justified under noise, yielding ~4.5–4.7 point Macro-F1 gains across perturbation types.
• Noise-Robustness Mechanisms: Enhanced temporal–spectral integration and multi-resolution processing contribute to improved robustness under mixed-noise conditions.
• Class-Specific Sensitivity: Gains are most pronounced in high-workload classes, improving separability without compromising calibration.
• Comparative Superiority: TF-T3 outperforms Riemannian geometry-based ensembles and deep learning baselines by +7–8 Macro-F1 points under COMBO noise while reducing temporal instability by ≈35%.
• Deployment Guidance: TF-T2 is preferred for efficiency-critical low-noise monitoring; TF-T3 is recommended for perturbation-prone environments prioritizing robustness.

6 Conclusions

This study advances EEG-based cognitive state classification for aviation by introducing a noise-robust TF-T architecture. Leveraging complementary datasets, we demonstrate high accuracy under controlled conditions and strong robustness in ecologically realistic, noise-augmented scenarios. TF-T2 offers the best accuracy–efficiency trade-off for clean data, while TF-T3, incorporating enhanced temporal–spectral integration and multi-resolution temporal processing, consistently outperforms alternative approaches under noisy conditions. Limitations include wider confidence intervals for low-support classes, incomplete coverage of real-world interference patterns, and increased inference latency (≈47%) for TF-T3. Future work will address these by exploring adaptive decision thresholding, cost-sensitive training to mitigate class imbalance, and multimodal sensor fusion (e.g., EEG with eye-tracking or physiological measures) to enhance resilience. Additionally, incorporating structured artifact simulations and optimizing transformer efficiency will further support scalable, real-time deployment in cockpit monitoring systems.

Generative AI statement

The author(s) declared that Generative AI was used in the creation of this manuscript. During the preparation of this work the author(s) used ChatGPT, and Quill Bot in order to refine the writing, in-terms of language. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Cristian Randieri, University of eCampus, Italy

Reviewed by: Hyunsu Lee, Pusan National University, Republic of Korea

G. Abinaya, Saveetha Engineering College, India

Footnotes

APPD, Attention-Related Pilot Performance Decrements; ATC, Air Traffic Control; BL, Baseline; CA, Channelized Attention; CAST, Commercial Aviation Safety Team; CI, Confidence Interval; COMBO, Composite Dataset Variant (60% BASE, 20% GAUSS, 20% RAND); DA, Diverted Attention; EEG, Electroencephalography; ECG, Electrocardiography; E, Easy (Flight Simulator); FDMW, Flight-Deck Mental Workload Dataset; FFN, Feed-Forward Network; FIR, Finite Impulse Response; FLOPs, Floating Point Operations; FS, Flight Simulator Scenario; GAUSS, Gaussian Noise-Augmented Dataset Variant; GELU, Gaussian Error Linear Unit; GSR, Galvanic Skin Response; H, Hard (Flight Simulator); HtC, Heat-the-Chair Scenario; HW, High Workload; ICA, Independent Component Analysis; IATA, International Air Transport Association; LOFT, Line Oriented Flight Training; LW, Low Workload; M, Medium (Flight Simulator); MH, Medium-Hard (Flight Simulator); MW, Medium Workload; PE, Positional Encoding; PR-AUC, Precision–Recall – Area Under Curve; PSD, Power Spectral Density; RAND, Uniform Random Noise-Augmented Dataset Variant; ReLU, Rectified Linear Unit; ROC-AUC, Receiver Operating Characteristic—Area Under Curve; SS, Startle/Surprise; STFT, Short-Time Fourier Transform; SwiGLU, Swish-Gated Linear Unit; TF-T1, Temporal–Spectral Fusion Transformer—Baseline Model; TF-T2, Temporal–Spectral Fusion Transformer—Enhanced Model; TF-T3, Temporal–Spectral Fusion Transformer—Advanced Model.