Visual measurement network for intelligent cockpit driving state combining lightweight temporal convolution and cross-granularity state mapping

1 Introduction

As automated driving continues to progress from L2+ toward L3 operation, human–machine co-driving has become an increasingly important interaction paradigm in intelligent vehicles (Fu Q. et al., 2025; He et al., 2024). Recent research has extended autonomous-driving capabilities beyond perception and motion control toward safety-critical scenario validation and multi-objective decision-making. For example, real-world pre-crash data have been used to construct safety-critical scenarios for autonomous-driving validation, while multi-objective eco-driving strategies have been developed to jointly account for driving safety and energy efficiency. These developments demonstrate substantial progress in autonomous-driving safety validation and intelligent decision-making. Nevertheless, in L2+/L3 human–machine co-driving systems, operational safety also depends on the driver’s readiness to respond appropriately when a takeover request is issued. Driver alertness, attention, and behavioral state can therefore directly influence takeover performance and driving safety (Liu et al., 2026). Human factors, including distraction and fatigue, remain important contributors to road-safety risk (Tong et al., 2025). Consequently, driver monitoring systems (DMSs) are an essential component of intelligent cockpits, providing continuous driver-state information and dynamic risk estimates for higher-level vehicle decision-making.

However, current DMS still faces severe engineering constraints in actual deployment (Campos-Ferreira et al., 2023; Khalil et al., 2025): the onboard domain controller is strictly limited in terms of computing power and power consumption, making it difficult to support the real-time inference requirements of heavy models such as traditional three-dimensional convolutional networks or Vision Transformers—DMS systems require an inference frame rate higher than 30 FPS and an end-to-end latency lower than 50 ms. Therefore, how to achieve high-precision, low-latency driving state perception under embedded computing power constraints has become a core issue of common concern to both academia and industry.

Existing driving state recognition methods can be broadly categorized into three types. The first type is based on traditional computer vision features and shallow classifiers (Huang et al., 2020; Qi et al., 2020). For example, facial and pose features are extracted using descriptors such as HOG (Histogram of Oriented Gradients) and LBP (Local Binary Pattern), and then classified using SVM (Support Vector Machine) or random forest. These methods have high computational efficiency but limited feature expressiveness and are difficult to handle lighting changes and partial occlusion in real driving scenarios. The second type is end-to-end recognition methods based on deep learning, which have rapidly become mainstream due to their powerful feature self-learning capabilities (Escotta et al., 2022; SHIN et al., 2025). Among these, static frame methods based on 2D-CNN (Convolutional Neural Network) have extremely fast inference speeds, but they completely sever the inherent temporal continuity and evolutionary patterns of driving behavior; methods based on 3D-CNN significantly improve recognition accuracy by explicitly modeling multi-frame temporal context, but the computational load increases exponentially with time dimension; and methods based on temporal models have improved the efficiency of temporal modeling. The third category is video modeling methods based on Vision Transformer (Shi et al., 2024; Koay et al., 2023), such as TimeSformer and Video-Swin, which capture global spatiotemporal dependencies by means of self-attention mechanism. Although the accuracy has reached the current advanced level, the number of parameters is often hundreds of megabytes, and the inference latency at the edge is far beyond the real-time red line. Some works have tried to find a trade-off between accuracy and efficiency through lightweight design, such as TSM (Temporal Shift Module) (Kou et al., 2023) which replaces three-dimensional convolution with time shift module to realize zero-parameter incremental temporal modeling, and X3D (Expanding 3D Networks) (Feichtenhofer, 2020) which performs lightweight scaling along network depth and frame rate in multiple dimensions, but its ability to model cross-scale features in complex driving scenarios is still insufficient. In summary, existing deep models generally face the binary dilemma of “high accuracy and low latency”, and it is difficult to achieve Pareto optimality of accuracy and efficiency on vehicle embedded platforms.

More specifically, the uniqueness of LTC-CGMN lies in its driver-monitoring-oriented integration of lightweight temporal reasoning and cross-granularity state representation, rather than in merely reducing the size of a generic video recognition backbone. Compared with lightweight DMS approaches based on TSM, X3D, or MobileNetV3–recurrent architectures, which mainly achieve computational savings through temporal shifting, architecture scaling, or recurrent sequence aggregation, LTC-CGMN employs depthwise separable causal dilated convolution to enlarge the temporal receptive field with limited computational growth while preserving strict online causality. In contrast to temporal video comprehension models such as TimeSformer, Video-Swin, Uniformer, and Video-Mamba, which primarily emphasize global spatiotemporal representation learning, LTC-CGMN explicitly incorporates driver-specific structural priors by aligning fine-grained head/hand cues with coarse-grained torso context through the CGSM module. Furthermore, the framework extends beyond conventional action recognition by jointly learning discrete driving states and a continuous NHTSA-AHP-guided risk index. Therefore, LTC-CGMN establishes a differentiated design in which edge-oriented temporal efficiency, cross-granularity behavioral discrimination, and continuous safety-risk measurement are coupled within a unified driver monitoring framework.

Besides computational constraints, existing methods still suffer from two major shortcomings in feature modeling and output representation. First, spatial granularity is disturbed. Most of them either only utilize global cockpit images to extract coarse-grained posture at the torso level or only focus on facial regions to extract fine-grained expressions and gazes, without cross-scale feature alignment and joint inference between micro-local and macro-global features (Ganesan et al., 2026; Chen et al., 2021). This granularity fragmentation directly leads to the model’s difficulty in differentiating actions with similar visual semantics but drastically different levels of risk. Some literature has studied multi-granularity feature fusion strategies, but the scale mismatch and information redundancy problems of multi-scale fusion still need to be systematically addressed (Zhang et al., 2023). Second, the output representation is a single one and has a “step effect”. Existing networks always output discrete action category labels (Han et al., 2022; Abd El-Nabi et al., 2025), which forces the upper-level warning logic to adopt “threshold triggering” mechanism. Recent studies have tried to introduce multi-task learning into DMS to simultaneously deal with distraction and emotion detection and evaluate driver takeover readiness thru multi-camera fusion, but their outputs are still mostly discrete attributes and fail to fundamentally solve the problem of continuous risk measurement (Zambrano et al., 2026; Shariff et al., 2023). The aforementioned dual bottlenecks indicate that relying solely on the traditional single-granularity discrete classification paradigm is insufficient to support the refined and human-centered risk warning requirements of intelligent cockpits.

Compared with conventional temporal convolution networks, LTC-CGMN employs depthwise separable causal dilated convolution to enlarge the temporal receptive field while maintaining low computational cost. Unlike transformer-based DMS models that primarily rely on global spatiotemporal attention, LTC-CGMN explicitly aligns fine-grained head/hand cues with coarse-grained torso features through CGSM. In contrast to generic multi-granularity fusion methods, the proposed framework further integrates continuous NHTSA-AHP-guided DRI prediction, thereby jointly addressing temporal efficiency, fine-grained action discrimination, and continuous driver-risk measurement within a unified edge-oriented DMS framework.

To address these limitations, this study proposes LTC-CGMN, a lightweight driver-state assessment framework that integrates temporal convolution, cross-granularity representation learning, and continuous risk estimation. The main contributions are summarized as follows:

• A lightweight temporal convolution module based on depthwise separable causal dilated convolution is developed to capture long-range behavioral dependencies while maintaining low computational complexity for edge inference.
• A cross-granularity state mapping module is introduced to explicitly align fine-grained head and hand cues with coarse-grained torso information through cross-attention, thereby improving discrimination among visually similar driving actions.
• An NHTSA-AHP-guided knowledge-distillation framework is developed to generate continuous DRI supervision from discrete driving behaviors, reducing abrupt transitions associated with conventional class-to-risk mapping.
• Extensive experiments on the multimodal 3MDAD dataset evaluate the proposed framework in terms of classification accuracy, continuous risk estimation, computational efficiency, day/night robustness, and edge-deployment performance.

2 Methods

2.1 Overall architecture and modality adaptation mechanism of LTC-CGMN network

Since the daytime subset of the 3MDAD dataset only provides RGB and depth modalities, while the nighttime subset only provides infrared and depth modalities, the two types of data have fundamental differences in physical properties and distribution characteristics, making direct mixing for training impossible (Li et al., 2022; Li et al., 2026). Therefore, this paper designs a modality adaptive input strategy: let the daytime mode input be X day = I RGB side , I RGB front , I Depth, and the nighttime mode input be X night = I IR side , I IR front , I Depth. Let the input modality type indicator variable be m ∈ day , night, then the unified input can be expressed as the Equation 1:

X m = I RGB side , I RGB front , I Depth , m = day I IR side , I IR front , I Depth , m = night (1)

The two modal streams share the same network topology and weight parameters, and are iteratively optimized separately as independent domains during training. This strategy avoids the physical misalignment problem of day and night data in terms of signal frequency bands and texture statistical characteristics, while maximizing the reuse of the learned cross-modal general representations.

The backbone of the network uses a lightweight MobileNetV3 as a spatial feature encoder (Fu G. et al., 2025; Lee et al., 2025). For the input X m, the backbone network extracts multi-scale spatial feature maps for each viewpoint and modality. To enhance the discriminative power of feature representation and control the number of parameters, a channel attention mechanism is introduced in the intermediate layer of MobileNetV3 to adaptively recalibrate the importance weights of each feature channel. Let the input single-view modal tensor be I ∈ R H × W × C, which, after encoding by the backbone network, yields a multi-level feature pyramid F 1 , F 2 , … , F L, where F l ∈ R H l × W l × C l represents the feature map of the l-th layer.

Coarse-grained global features F g and fine-grained head and hand features F h and F h a n d are input into the CGSM module for feature alignment and fusion. During benchmark experiments, the ground-truth bounding boxes provided by 3MDAD are used to isolate and evaluate the contribution of cross-granularity modeling. In practical vehicle operation, these annotations are replaced by an onboard lightweight head-and-hand detector that generates bounding boxes online for each incoming frame. The detected regions are then processed by RoI Align and directly fed into CGSM, enabling the framework to operate without manually provided bounding boxes. The fused multi-granularity representation F fused is then fed into a lightweight temporal convolution module to model the dynamic evolution of driving behavior over time. Finally, the network is divided into two branches via a multi-task head: a classification branch outputs probability vectors p ∈ R 16 for 16 types of driving actions, and a regression branch outputs a continuous driving risk index DRI pred ∈ 0 , 1.

Figure 1 shows the complete end-to-end topology and loss function layout of the LTC-CGMN network.

Figure 1 systematically presents the entire computational chain of LTC-CGMN from input to output. The network input employs a modality adaptation mechanism, dynamically selecting RGB or IR (Infrared) modalities based on day/night modes, and combining them with depth information to form a dual-view input stream. After encoding by the MobileNetV3 backbone, the data from each viewpoint is differentiated into two granular branches: a fine-grained branch uses bounding boxes provided by 3MDAD to crop the head and hand regions, while a coarse-grained branch retains panoramic torso features. The two feature streams are fused spatially and semantically through cross-attention in the cross-granularity state mapping module. The fused temporal feature sequence captures long-range dependencies via a lightweight temporal convolution module, and is finally decoupled into 16-dimensional classification probabilities and continuous risk indices in the multi-task head. In Figure 1, the NHTSA-AHP module is marked with a dashed box in the loss calculation area. This module only serves as a static soft label generator, providing DRI gt supervision signals for the regression branch, and does not participate in forward propagation, ensuring that the entire network is end-to-end differentiable.

2.2 Cross-granularity state mapping module

Accurate recognition of driving actions highly depends on the joint perception of features at different spatial granularities: on the one hand, macroscopic torso posture provides the global semantic context of the action; on the other hand, microscopic local details are key clues for distinguishing easily confused actions. However, existing methods treat the two separately, ignoring the naturally existing semantic complementarity between them. To this end, this paper proposes a Cross-Granularity State Mapping (CGSM) module, which aims to explicitly align and fuse macroscopic and microscopic visual features through a learnable attention mechanism.

For each frame input, CGSM first constructs two parallel granularity branches. In the fine-grained branch, using the side-view head and hand bounding boxes provided by the 3MDAD dataset, the microscopic local features F head ∈ R H h × W h × C and F hand ∈ R H h × W h × C are precisely cropped from the multi-scale feature map of the backbone network through the RoI Align operation (Tseng et al., 2021).

In the coarse-grained branch, the panoramic cockpit feature map F body ∈ R H b × W b × C is retained as a macroscopic global representation. To establish the spatial correspondence between the two granularities, bilinear interpolation is used to upsample F head and F hand to the same spatial resolution as F body. Then, they are fused into a unified local feature tensor F local ∈ R H b × W b × C by element-wise addition, achieving spatial dimension alignment.In the Equation 2, Up · represents the bilinear upsampling operation.

F local = Up F head ⊕ Up F hand (2)

After obtaining the spatially aligned global feature F body and local feature F local, CGSM introduces a multi-head cross-attention mechanism to achieve deep interaction between granularities. F body and F local are expanded into sequence forms Z body ∈ R N × C and Z local ∈ R N × C, respectively, where N = H b × W b. Cross-attention (Zhao et al., 2025; Zheng et al., 2025) uses global features as the query and local features as the key and value, and calculates the attention distribution of global features on local details, so that the macro pose can be recalibrated according to micro cues: According to Equations 3, 4, W Q , W K , W V ∈ R C × d k are the learnable projection matrices, and d k is the dimension of the key vector. Finally, the cross-attention output is aggregated with the original global features through residual connections, and then normalized by layers to obtain the fused multi-granular representation as shown in the Equation 5:

Q = Z body W Q , K = Z local W K , V = Z local W V (3)

Z cross = Softmax Q K ⊤ d k V (4)

F fused = LayerNorm Z cross + Z body (5)

This residual design effectively preserves global context information, avoiding the loss of macroscopic semantics due to the excessive dominance of fine-grained features in the representation.

For nighttime samples, where head and hand bounding boxes are unavailable, CGSM does not perform explicit local-region alignment. Instead, the module operates in a degraded global mode, in which the panoramic IR-depth representation is used for self-attention-based feature refinement. Thus, the daytime configuration performs explicit cross-granularity interaction between local and global features, whereas the nighttime configuration preserves the CGSM attention structure but relies only on globally available visual information.

Figure 2 intuitively shows the complete data flow and tensor dimension changes within the CGSM module, from feature decoupling and spatial alignment to cross-attention fusion.

Figure 2 details the internal computational pipeline of the CGSM module. The coarse-grained branch receives the panoramic torso feature map F body, while the fine-grained branch receives the head feature F head and hand feature F hand, respectively, after being cropped by bounding boxes. The fine-grained features are first unified to a spatial size H b × W b consistent with the global features through adaptive RoI Align and bilinear upsampling, and then fused element-wise into local features F local. Cross-attention fusion expands F body and F local into a sequence form, and calculates multi-head cross-attention weights with the global feature as the query and the local feature as the key/value. The weight matrix is visualized using a heatmap overlay to highlight the spatial distribution of macro-feature attention to micro-cues. After residual connection and layer normalization, the fused feature F fused is output, whose tensor dimension is consistent with F body.

2.3 Lightweight temporal convolutional network

The multi-granular spatial features F fused fused by the CGSM module only represent the spatial semantic information within a single frame image. Driving behavior, however, is essentially a continuous time-series process—the evolution from safe driving to distracted operation often spans several seconds or even tens of seconds, containing rich state transition patterns and dynamic modes. Therefore, temporal modeling is an indispensable part of accurate measurement of driving states. Addressing the stringent constraints on model parameter quantity and computational latency at the vehicle edge, this paper proposes a Lightweight Temporal Convolution (LTC) network, whose core operator is depthwise separable causal dilation convolution.

Let the input temporal feature sequence be X ∈ R T × C, where T is the time window length and C is the feature dimension. LTC first ensures strict causality in temporal modeling through causal convolution. The output at time t depends only on the input at times 1 , 2 , … , t, without introducing any future frame information, thus perfectly fitting the real-time inference scenario of online driving monitoring. A standard causal convolution can be defined as the Equation 6: W k is the kernel weight, and K is the kernel size. This formula shows that the output Y t is calculated only from the current and historical inputs, without any leakage of future information. To further expand the temporal receptive field without increasing the number of parameters, an inflation factor d is introduced, and the standard causal convolution is generalized to a causal dilated convolution as shown in the Equation 7:

Y t = ∑ k = 0 K ‐ 1 W k · X t ‐ k (6)

Y t = ∑ k = 0 K ‐ 1 W k · X t ‐ d · k (7)

The inflation factor d grows exponentially, which expands the receptive field to R = 1 + ∑ l K ‐ 1 · d l, covering long-term temporal dependencies without introducing additional learnable parameters.

However, causal dilation convolution (Azadani and Boukerche, 2021) is still a dense convolution operation with a computational complexity of O K · C 2 · T. When the number of channels C is large, the inference overhead is still not negligible. To address this, LTC introduces depthwise separable convolution to perform lightweight reconstruction of this operator. Depthwise separable convolution (Chen J. et al., 2026; Ayachi et al., 2020) decomposes the standard convolution into two steps: channel-wise convolution is applied independently to each input channel, and then pointwise convolution is performed linearly along the channel dimension. Based on this, the depthwise separable causal dilation convolution designed in this paper can be expressed as: According to Equations 8, 9, W dw ∈ R K × C represents the channel-wise convolution kernel, and W pw ∈ R C × C ′ represents the pointwise convolution weight matrix. The computational complexity of standard causal dilated convolution is O K · C 2 · T, which is reduced to O K · C · T + C 2 · T after depthwise separable decomposition. When C is large, the computational complexity is reduced by approximately 1 / K times.

X dw t , c = ∑ k = 0 K ‐ 1 W dw k , c · X t , ‐ d · k c , c = 1 , 2 , … , C (8)

Y = X dw W pw (9)

In this paper, the LTC module stacks four layers of depthwise separable causal dilated convolution with dilation factors of 1 , 2 , 4 , 8. The kernel size is uniformly set to K = 5, and the temporal receptive field covers 1 + 5 ‐ 1 × 1 + 2 + 4 + 8 = 61 frames—corresponding to approximately 2 s of historical context at a sampling rate of 30 FPS, sufficient to capture the complete evolution cycle of most driving actions. Each convolutional layer is followed by a ReLU (Rectified Linear Unit) activation function and Dropout (p = 0.1) to prevent overfitting, and residual connections are used to alleviate the problem of vanishing gradients in deep layers.

2.4 Continuous risk measurement and prior knowledge distillation based on NHTSA-AHP

Existing DMS methods usually output discrete action category labels, which leads to a “step effect” in the upper-level warning system. When the driver’s behavior fluctuates near the category decision boundary, the risk alarm changes frequently, which seriously damages the user’s experience. To eliminate this defect, this paper proposes a continuous risk measurement framework that distills domain prior knowledge to the network regression head, so that the network can directly output a smoothly evolving Driver Risk Index (DRI).

The National Highway Traffic Safety Administration (NHTSA) divides driver distraction into three dimensions: visual distraction, manual distraction, and cognitive distraction (Kashevnik et al., 2021). Inspired by this, this paper establishes a three-dimensional risk factor vector r k = V k , M k , C k for each of the 16 types of driving actions A = A 1 , A 2 , … , A 16 in the 3MDAD dataset, representing the visual derailment rate, hand off-wheel rate, and cognitive load rate of the k-th type of action, respectively, with each factor taking values in the 0 , 1 range. Since visual, manual, and cognitive distraction contribute differently to driving risk, the Analytic Hierarchy Process (AHP) is employed to derive their relative weights rather than assigning equal importance. The weighting rationale follows the safety consequences represented by the three NHTSA-inspired distraction dimensions. Visual distraction is assigned the highest priority because removing visual attention from the forward roadway directly reduces hazard perception and reaction capability. Manual distraction is assigned the second-highest priority because removing one or both hands from the steering control impairs immediate vehicle maneuverability, although visual monitoring of the road may still be partially retained. Cognitive distraction is also considered safety-critical, but its effect is comparatively less directly observable from visual behavior and may not always be accompanied by immediate loss of visual or manual control.

The AHP assessment involved seven specialists with expertise in driver behavior, traffic safety, intelligent vehicles, and human factors. Each specialist independently evaluated the relative importance of visual diversion, manual diversion, and cognitive load using the Saaty scale. The aggregated judgments produced the normalized weight vector w = 0.45 , 0.35 , 0.20, corresponding to visual, manual, and cognitive distraction, respectively. All expert judgment matrices satisfied the AHP consistency criterion of C R < 0.10. In addition, three independent specialists who were not involved in the initial weighting process reviewed the resulting weights and confirmed the same priority order, w v > w m > w c, supporting the robustness of the adopted DRI weighting scheme.

Based on the Saaty scaling method (Harker and Vargas, 1987; Alemdar and Çodur, 2024), a pairwise comparison judgment matrix A ∈ R 3 × 3 is constructed, where the element a i j represents the importance of the i-th factor relative to the j-th factor. After solving the normalized eigenvectors and performing consistency checks, the normalized allocation w V , w M , w C of the visual, manual, and cognitive weights is obtained. Then, the basic risk assessment value for the k-th action is a weighted linear combination of the three factors:

R base k = w V · V k + w M · M k + w C · C k , k = 1 , 2 , … , 16 (10)

In the Equation 10, the calibration value R base k ∈ 0 , 1 quantifies the instantaneous risk level of performing a certain action within a single frame.

However, actual driving risk is not a simple mapping of instantaneous quantities; the cumulative effect of continuously performing high-risk actions differs fundamentally from that of briefly accidently performing low-risk actions. Therefore, this paper designs a soft-label generator to transform the action sequence within a time window into a continuous risk curve through nonlinear accumulation. Let the sliding window W = k τ , ∣ , τ , = t ‐ w t ‐ w + 1 … t contain the action category indexes of historical w frames, and define the cumulative risk within the window as:In the Equation 11, Δ t represents the time interval between adjacent frames, and α > 0 is the cumulative decay coefficient, controlling the strength of historical risk information memory. When high-risk actions continuously occur within the window, the cumulative exponent rapidly saturates and approaches 1; when a high-risk action is briefly followed by a rapid return to safe driving, due to the natural exponential decay of memory, the DRI smoothly declines rather than abruptly drop to zero, perfectly matching the gradual evolution of real driving risks. This paper sets w = 16 frames, Δ t = 1 / 30 s, and α = 1.5, determined through grid search.

DRI t = 1 ‐ ⁡ exp ‐ α ∑ τ = t ‐ w t R base k τ · Δ t (11)

The supervisory signal DRI gt for the regression branch is not derived from manual annotation, but rather from soft labels generated by the NHTSA-AHP prior model through the temporal cumulative evolution of discrete action label sequences. This design transfers domain-driven rules to a data-driven neural network in the form of knowledge distillation—the network regression head does not need to directly learn the abstract AHP judgment matrix, but rather indirectly inherits domain expert priors by fitting the continuous DRI curve derived from this matrix. This design provides an interpretable expert-prior basis for continuous DRI learning while retaining end-to-end optimization flexibility. Nevertheless, the NHTSA-AHP-derived labels inherently contain a degree of subjectivity because the risk-factor scores and their relative weights depend on expert judgment rather than direct human risk annotations. Although consistency checks and independent expert validation reduce arbitrary bias, the resulting DRI should therefore be interpreted as an expert-informed relative risk indicator rather than an absolute ground-truth measure of driver risk.

It should be noted that the DRI represents an expert-prior-informed risk surrogate rather than a directly observed statistical measure of actual crash probability or driver impairment. The NHTSA-AHP framework provides structured and interpretable supervision, but the resulting risk values remain dependent on expert judgments and the adopted weighting scheme. Therefore, the DRI is interpreted as a relative indicator of driver-risk tendency rather than an absolute ground-truth measure of driving risk.

LTC-CGMN employs a multi-task learning framework, simultaneously optimizing both discrete action classification and continuous risk measurement tasks. The total loss function is a weighted combination of cross-entropy loss and Smooth L1 loss.According to the Equation 12, y ∈ 0 , 1 16 represents the one-hot label of the action category; q ∈ R 16 represents the probability vector output by the network’s classification branch; DRI gt is calculated by the soft label generator, and DRI pred is the output of the network’s regression branch. The classification loss L CE y , q = ‐ ∑ k = 1 16 y k ⁡ log ⁡ q k ensures the network correctly identifies discrete driving actions; the regression loss L SmoothL 1 a , b = 0.5 a ‐ b 2 , ∣ a ‐ b ∣ < 1 ∣ a ‐ b ∣ ‐ 0.5 , otherwise provides robustness against outlier predictions, preventing outliers from dominating gradient updates.

L total = L CE y , q + λ · L SmoothL 1 DRI gt , DRI pred (12)

The balance coefficient λ = 0.1, obtained by grid search, is employed to balance the magnitudes of the two losses and prevent the dominance of single-task during training. The proposed joint optimization strategy enables the network to learn to output smooth, continuous and physically interpretable driving risk measures while accurately classifying actions, removing the step effect of discrete warnings.

3 Experimental setup

3.1 Experimental dataset and preprocessing strategy

This paper uses the 3MDAD public benchmark dataset (Abdullah et al., 2024) for experimental verification. The data link is: https://www.kaggle.com/datasets/anouarbenkhalifa/3mdad-multimodal-driver-action-distraction. This dataset consists of multimodal and multi-view videos collected in real driving scenarios, divided into two subsets: daytime and nighttime. The daytime subset contains synchronized RGB and depth videos of 50 participants (38 males, 12 females, aged 19–41 years), recorded from the side view (Kinect 1) and the front view (Kinect 2), respectively. Only the daytime side-view subset of 3MDAD provides manually annotated axis-aligned bounding boxes for the head and hands, whereas the nighttime IR subset does not contain corresponding local-region annotations. This difference originates from the annotation availability of the dataset rather than from the design of LTC-CGMN. Therefore, fine-grained head/hand supervision can be directly exploited in the daytime experiments, while no ground-truth local boxes are available for the nighttime subset. The dataset covers 16 common driver actions, including safe driving (A1), applying makeup (A2), operating the radio (A3), using GPS (A4), texting with either hand (A5/A6), making phone calls with either hand (A7/A8), taking photos (A9), talking to passengers (A10), singing and dancing (A11), fatigue and drowsiness (A12), drinking water with either hand (A13/A14), reaching behind (A15), and smoking (A16). The daytime and nighttime subsets of data have significantly different physical characteristics and the participants do not overlap; therefore, they are independent domains and cannot be directly mixed for training.

To construct the sample sequence required for temporal modeling, this paper employs a sliding window strategy to sample the original video frames. The window length is set to W = 16 frames, and the stride to S = 4 frames. For the daytime subset, using provided bounding box annotations, the head region (F head) and hand region (F hand) are cropped from the RGB and depth frames of the side view, while the panoramic frames are retained as the source of the global torso feature (F body). The use of ground-truth boxes is restricted to controlled benchmark evaluation and does not imply their availability during real-world deployment. For an operational DMS, head and hand regions can be obtained in real time using a lightweight detector integrated before CGSM. Thus, the detector replaces dataset annotations during inference, while the subsequent feature extraction and cross-granularity fusion process remains unchanged.

The RGB and depth frames of the frontal view are used as auxiliary modal inputs to provide supplementary information on the driver’s frontal face and torso. For the nighttime subset, because no head/hand bounding-box annotations are provided by 3MDAD, only panoramic IR and depth frames from the side and frontal views are used. In this setting, CGSM switches from explicit local-global cross-attention to global self-attention, allowing temporal and global posture information to remain available while avoiding dependence on unavailable local annotations.

The training and test sets are divided by participant to ensure that data from the same participant does not appear in both the training and test sets simultaneously, avoiding data leakage. This paper uses five-fold cross-validation for model evaluation. To map discrete driving actions into continuous risk measurement supervisory signals, this paper performs expert knowledge labeling on 16 types of actions from three dimensions: visual derailment rate (V k), hand-off rate (M k), and cognitive load rate (C k). The specific values of the three factors and the base risk labeling value R base k calculated by the analytic hierarchy process are shown in Table 1.

TABLE 1. Action classification and risk measurement labeling based on NHTSA-AHP in the 3MDAD dataset.

Table 1 lists the codes, names, and three-dimensional risk factors (visual derailment rate V k, hand derailment rate M k, cognitive load rate C k) of the 16 action categories in the 3MDAD dataset, as well as the weighted basic risk rating R base k calculated by AHP. The risk factor values range from 0 to 1, with higher values indicating a higher degree of distraction in that dimension. R base k is calculated as R base k = 0.65 · V k + 0.20 · M k + 0.15 · C k (AHP weight allocation is consistent: λ max = 3.068; CI Consistency Index = 0.034; CR Consistency Ratio = 0.059 < 0.1). Among them, safe driving (A1) has all factors and risk rating values of 0, which serves as the absolute benchmark. Texting (A5/A6) has the highest risk rating value (0.825) because it simultaneously occupies the driver’s visual, manual, and cognitive resources. Fatigue and drowsiness (A12) has a risk rating value of 0.753 due to extremely high visual and cognitive load. Reaching back (A15) has a short duration but occupies extremely high visual and manual resources, with a risk rating value of 0.788.

3.2 Experimental environment and evaluation metrics

All experiments in this paper are conducted on a unified hardware and software platform. Model training and cloud-based high-precision baseline comparison experiments are performed on a server equipped with a single NVIDIA RTX 4090 GPU (24 GB VRAM), using the PyTorch 2.1 deep learning framework and CUDA version 12.2. During training, the AdamW optimizer is used, with an initial learning rate of 1 × 10⁻³, coupled with a cosine annealing learning rate scheduling strategy and a weight decay coefficient of 5 × 10⁻⁴. The training iterations (epochs) are set to 120, and the batch size is set to 64.

For all baseline comparisons, a unified experimental protocol was adopted. All models received the same Day-RGB input samples generated using identical preprocessing and 32-frame temporal windows. The same participant-independent five-fold data divisions were used for every model. Training settings were also standardized using AdamW, an initial learning rate of 1 × 10 − 3, cosine annealing, 120 epochs, and a batch size of 64. No baseline was allowed to use additional modalities, extra training samples, or different test partitions.

To verify the feasibility of deploying the proposed model on a real-world automotive edge computing platform, edge inference latency and lightweight baseline comparison experiments are conducted on an NVIDIA Jetson Orin 64 GB developer kit. The platform is equipped with an Ampere architecture GPU (approximately 200 TOPS INT8 computing power) with a power consumption limit of 15 W. The software stack uses JetPack 5.1.2 and TensorRT 8.6 for inference acceleration and deployment optimization. All models uniformly use FP16 (16-bit Floating Point) half-precision inference during edge evaluation to balance accuracy preservation and latency reduction.

Regarding the evaluation metrics, this paper comprehensively measures model performance from two dimensions: recognition accuracy and computational efficiency. For the discrete classification task of driving actions, Top-1 accuracy and macro-average F1 score are used as the main evaluation metrics. Top-1 accuracy is defined as the proportion of samples where the predicted probability of the highest class matches the true class:As shown in the Equation 13, N is the total number of test samples; p i ∈ R 16 is the probability vector output by the network for the i-th sample; y i is the corresponding true class index; 1 · is the indicator function. The macro-average F1 score is calculated by first calculating the precision and recall for each class separately, and then taking their unweighted average:In the Equation 14, Precision k = TP k / TP k + FP k, and Recall k = TP k / TP k + FN k, where TP k, FP k, and FN k are the counts of true positives, false positives, and false negatives in the k-th class, respectively.

Acc = 1 N ∑ i = 1 N 1 arg ⁡ max p i = y i (13)

F 1 macro = 1 16 ∑ k = 1 16 2 · Precision k · Recall k Precision k + Recall k (14)

For the continuous driving risk measurement regression task, the root mean square error (RMSE) and mean absolute error (MAE) are used as evaluation metrics, defined as follows: According to Equations 15, 16, DRI pred , i and DRI gt , i are the continuous risk index values provided by the network prediction and soft-label generator, respectively, both within the range of 0 , 1.

RMSE = 1 N ∑ i = 1 N D R I p r e d , i − D R I g t , i 2 (15)

MAE = 1 N ∑ i = 1 N ∣ DRI pred , i ‐ DRI gt , i ∣ (16)

Regarding computational efficiency, this paper reports four key metrics: model parameter count (Params), in millions (M), reflecting the model’s storage resource consumption; floating-point operations (FLOPs), in billions (G), measuring the model’s theoretical computational complexity; end-to-end inference latency (Latency), measured in milliseconds, is defined as the complete processing time from raw frame input to final action and DRI outputs. It includes image resizing and normalization, online head/hand bounding-box generation, feature extraction, CGSM-LTC processing, and final classification/regression prediction. Therefore, the reported latency and FPS represent the complete operational pipeline rather than network forward propagation alone; and frames per second (FPS), reflecting the model’s actual throughput.

To ensure the statistical reliability of the experimental results, all experiments in this paper employ a five-fold cross-validation strategy. Based on the participant list of the 3MDAD dataset, the data is divided into five disjoint folds according to the participants, ensuring that all samples of the same participant fall completely within the same fold, thus preventing data leakage. Each fold is used as the test set in turn, and the remaining four folds are combined as the training set. The experiment is repeated 5 times.

To prevent information leakage, all data-dependent hyperparameters were selected exclusively using the training portion of each cross-validation fold, with an internal validation subset used for model selection. The held-out test fold was not accessed during hyperparameter tuning. This protocol applies to the temporal window length, cumulative decay coefficient λ, loss-balancing coefficient, learning-rate-related settings, and other trainable configurations. In contrast, the AHP weight vector w = 0.45 , 0.35 , 0.20 and the action-level risk scores were predefined from independent expert assessments and were not estimated or optimized using 3MDAD samples. Therefore, these expert-prior parameters were kept fixed across all five folds and did not introduce test-set information into model training or selection.

4 Results

4.1 Comparison of accuracy in action classification and risk measurement

This paper first verifies the comprehensive accuracy of LTC-CGMN in two tasks: discrete classification of driving actions and continuous risk measurement. To ensure fairness in the comparison, all models are trained and inferred on the cloud NVIDIA RTX 4090 platform. Only the accuracy metrics (Top-1 accuracy and macro-average F1 score) and regression errors (RMSE and MAE) are reported, without mentioning the edge inference speed. The comparison baseline covers three mainstream video understanding architectures: the classic 3D-CNN-based model I3D (Inflated 3D Convolutional Network) (Raja et al., 2025), the high-performance models TimeSformer (Panadero Palenzuela, 2025) and Video-Swin (Boulahmar et al., 2025) based on Vision Transformer, and the efficient video recognition models Uniformer (Li et al., 2023) and Video-Mamba (Chen G. et al., 2026). To ensure a fair comparison, I3D, TimeSformer, Video-Swin, Uniformer, Video-Mamba, and LTC-CGMN were evaluated under exactly the same experimental protocol. All models used the same Day-RGB modality, identical 32-frame input intervals, preprocessing procedure, participant-independent five-fold splits, and training hyperparameters described in Section 3.2. Only the model architecture was varied across experiments. Therefore, the differences reported in Table 2 primarily reflect differences in spatiotemporal modeling capability rather than inconsistencies in input information, temporal context, training strategy, or data partitioning.

TABLE 2. Comparison of recognition and measurement performance of various models on 3MDAD-Day.

For the baseline models originally designed only for action recognition, RMSE and MAE were not calculated directly from discrete class labels. Instead, the predicted class-probability vector was converted into a continuous risk estimate using the same NHTSA-AHP risk scores adopted for LTC-CGMN. Specifically, the baseline DRI was calculated as y ^ D R I = ∑ c = 1 16 p c r c, where p c denotes the predicted probability of action class c, and r c is its corresponding AHP-derived risk score. The resulting continuous DRI was then compared with the same reference DRI labels used for LTC-CGMN to calculate RMSE and MAE. This procedure enables a consistent risk-measurement comparison without modifying the original classification architectures.

Table 2 compares the action-classification and DRI-estimation performance of LTC-CGMN with five representative video-understanding architectures under the same experimental protocol. LTC-CGMN achieves the highest Top-1 accuracy (96.37%) and macro-F1 score (0.958), exceeding Video-Mamba by 1.55 percentage points and 0.016, respectively. For continuous DRI estimation, LTC-CGMN obtains an RMSE of 0.058 and an MAE of 0.035, corresponding to reductions of 15.9% and 20.5%, respectively, relative to Video-Mamba. These results suggest that the proposed combination of cross-granularity representation learning, lightweight temporal modeling, and NHTSA-AHP-guided regression improves both action discrimination and continuous risk estimation under the evaluated protocol. The relatively small cross-fold standard deviations further indicate stable performance across the five participant-independent splits.

4.2 Edge deployment and lightweight performance comparison

This paper further verifies the deployment feasibility and inference efficiency of LTC-CGMN on a real-world in-vehicle edge computing platform. The comparison scope is limited to lightweight video understanding models that can run normally on NVIDIA Jetson Orin, including the lightweight variant C3D-Light based on 3D-CNN, TSM based on temporal shifting, X3D-S with lightweight scaling along multiple dimensions, and MobileNetV3+BiLSTM combining a lightweight 2D backbone with BiLSTM (Bidirectional Long Short-Term Memory). All models use FP16 half-precision inference and are optimized for deployment using TensorRT. The number of parameters, computational cost, end-to-end latency, and measured frame rate for each model are shown in Table 3.

TABLE 3. Comparison of measured inference efficiency of lightweight edge models.

In addition to general lightweight video models, recent DMS-oriented lightweight architectures were included where compatible with the 3MDAD input setting. All compared methods were retrained under the same protocol described in Section 3.2, ensuring that the reported differences primarily reflect model design rather than training or data-processing discrepancies.

Table 3 compares the complete edge-processing efficiency of the lightweight models on Jetson Orin. For LTC-CGMN, the reported 29.8 ms latency includes 3.4 ms for image preprocessing, 4.6 ms for online head/hand bounding-box generation, 10.8 ms for spatial feature extraction, 8.1 ms for CGSM-LTC processing, and 2.9 ms for final classification and DRI prediction. These components sum to 29.8 ms, corresponding to an effective throughput of 33.6 FPS. Thus, the reported edge speed reflects the full deployable DMS pipeline rather than isolated network inference, while still satisfying the real-time requirement of more than 30 FPS.

LTC-CGMN contains 3.78 M parameters and requires 5.46 G FLOPs, representing reductions of 12.5% and 20.3%, respectively, relative to MobileNetV3+BiLSTM. On Jetson Orin, it achieves an end-to-end latency of 29.8 ms and a throughput of 33.6 FPS. Compared with X3D-S, latency is reduced by 15.3%, while throughput increases by 18.3%. Relative to TSM, the corresponding improvements are 38.8% and 63.9%. These gains are primarily associated with the depthwise separable causal dilated convolutions used in LTC, which expand the temporal receptive field without proportionally increasing computational cost. Together with the classification and DRI-estimation results in Table 2, the edge measurements indicate that LTC-CGMN provides a favorable accuracy–efficiency trade-off for real-time DMS deployment.

4.3 Lightweight design and temporal modeling efficiency

This paper further analyzes the performance and computational cost of the LTC module under different temporal context lengths to verify its core advantage in temporal modeling efficiency. Input sequence lengths T ∈ 8 , 16 , 32 , 64 frames are selected, and the Top-1 accuracy and FLOPs of LTC, Standard TCN (Temporal Convolutional Network), ConvLSTM, and C3D are evaluated respectively. By plotting the evolution trajectory of each model on the accuracy-computation plane, the position of LTC on the Pareto optimal boundary is revealed. The comparison results are shown in Figure 3.

Figure 3 compares the accuracy–computation trade-off of four temporal modeling approaches across input lengths of 8, 16, 32, and 64 frames. Computational cost increases with sequence length for all methods, whereas their accuracy gains differ. LTC remains close to the Pareto frontier across the evaluated settings and achieves a Top-1 accuracy of 96.37% at approximately 5.5 G FLOPs for a 32-frame input. Under the corresponding configurations, Standard TCN, ConvLSTM, and C3D achieve lower accuracy at higher computational costs. These results indicate that depthwise separable causal dilated convolution provides an effective means of increasing the temporal receptive field while limiting computational overhead.

4.4 Effectiveness and confusion of cross-granularity state mapping

The core idea of cross-granularity feature fusion is that the microscopic local details can provide discriminative supplement to the macroscopic poses and then can effectively distinguish actions with similar visual semantics but drastically different risks. To verify this hypothesis, the fine-grained classification results of LTC-CGMN on 16 action categories are first presented using a confusion matrix, and then the overall classification performance and main confusion patterns are examined from a macro perspective, as shown in Figure 4.

Safe driving (A1) and fatigue/drowsiness (A12) achieve high recognition rates (both above 97%), reflecting the model’s strong discriminative power for these typical driving states. The off-diagonal regions follow the “only display confusion rates greater than or equal to 1%” principle, revealing a limited but crucial confusion pattern: the confusion rate between A2 (makeup) and A3 (operating the radio)/A9 (taking a photo) is 1.0%–1.1%, verifying the efficient decoupling ability of micro-local features for semantically similar actions; the confusion rate between A13 (right hand drinking water) and A5 (right hand texting) is 2.3%, the largest confusion pair in the figure, due to the high similarity in their macro-postures of the arms significantly deviating from the steering wheel from a side view. The misclassification rate of all easily confused action groups is below 3%, proving that the CGSM module achieves high accuracy and strong robustness in fine-grained driving action recognition by explicitly aligning micro-local and macro-global features through cross-attention.

To further reveal the deep mechanism of feature fusion in the CGSM module, Figure 5 uses t-SNE (t-Distributed Stochastic Neighbor Embedding) dimensionality reduction visualization to compare the clustering distribution of high-dimensional features extracted by the network with and without CGSM in two-dimensional space, explaining the representation optimization effect brought about by cross-granularity fusion from the perspective of feature space structure.

Figure 5 uses t-SNE dimensionality reduction visualization to compare the clustering structure of the network feature space before and after the introduction of the CGSM module, selecting eight easily confused actions for illustration. Figure 5a shows that without CGSM, the network relies only on global body features, resulting in severe overlap of feature points in the two-dimensional space, with large-area interweaving of distribution areas and blurred boundaries. This explains the fundamental reason why existing methods frequently misclassify these visually similar actions. Figure 5b shows the feature distribution after CGSM is applied. It can be seen that the clustering is significantly improved: First, each of the eight action categories can be reasonably well separated as an independent cluster with a large increase of the Euclidean distance between clusters; second, feature points of the same category are highly aggregated toward the cluster center, which significantly reduces the intra-cluster variance; third, thanks to the effective alignment of the microscopic cue of “whether the hand is holding a phone” with the macroscopic semantics of head posture by CGSM, a clear discriminative boundary is established between A2 and A7. Fourth, while both A13 (drinking water, right hand) and A15 (reaching back) have substantial arm deviation from the steering wheel, they are separated because the differences in hand trajectory and the object being held are entirely represented by CGSM. Fifth, A10 (talking) and A11 (singing and dancing) are successfully distinguished due to the microscopic differences in facial expressions and lip movement patterns. This visualization powerfully demonstrates how CGSM, from the perspective of feature space structure, utilizes the explicit complementarity of microscopic and macroscopic features to decouple the intrinsic representations of easily confused actions.

To further quantify the contribution of each component within CGSM, a fine-grained ablation study was conducted by selectively removing head features, hand features, torso features, or the cross-attention fusion mechanism while keeping all other settings unchanged. The results are summarized in Table 4.

TABLE 4. Fine-grained ablation analysis of CGSM components.

Table 4 shows that removing any individual feature source degrades performance, confirming their complementary roles. Hand features produce the largest accuracy loss, indicating their importance for distinguishing object-interaction behaviors such as texting, drinking, and phone use. Torso features provide global posture context, while head features mainly contribute to gaze- and fatigue-related discrimination. Replacing cross-attention with direct fusion also reduces accuracy and increases RMSE, demonstrating that explicit local-global interaction is more effective than simple feature aggregation.

4.5 Continuous driving risk measurement and resistance to the “step effect”

To avoid drawing conclusions from a single illustrative case, five continuous driving video sequences with different behavior-transition patterns were selected to evaluate the temporal stability of DRI estimation. Each sequence contained multiple transitions among normal driving, texting, phone use, drinking, fatigue, and reaching behaviors, with durations ranging from 28 to 42 s. The LTC-CGMN output was compared with the NHTSA-AHP reference trajectory and the conventional discrete class-to-risk mapping. In addition to RMSE and MAE, the maximum instantaneous DRI change, Δ D R I max, was used to quantify the severity of abrupt risk jumps. The sequence covers four typical stages: “normal driving → texting with a mobile phone → yawning due to fatigue → returning to normal.” The continuous risk index output by the network is compared with the expert knowledge distillation baseline curve and the traditional discrete classification transformation curve to verify the core advantage of the proposed method in eliminating the warning step effect, as shown in Figure 6.

The black curve in Figure 6 represents the NHTSA-AHP-derived reference trajectory, reflecting the expected evolution of relative driver-risk tendency under the adopted expert-prior model: during the normal driving phase before 8 s, the DRI stabilizes near 0; during the text messaging phase from 8 to 16 s, the risk rises smoothly for about 2 s and then stabilizes at a high level of 0.825; during the fatigue phase from 16 to 22 s, the risk stabilizes at 0.753; after 22 s, during the return to normal driving, the risk smoothly falls back to 0 in an exponential decay manner. The red curve is the predicted value by the LTC-CGMN and the 95% confidence interval (red shading), which agrees well with the benchmark curve with a small error band, validating the effectiveness of the soft-label distillation mechanism. The blue stepped curve is the risk estimate from threshold mapping of the traditional discrete classification output, where a sudden change occurs at the time of behavior switching (0→0.825, 0.825→0.753, 0.753→0), and the steady phase remains constant with no transition information. This “step effect” leads directly to frequently jumping early warning system around the switching point. The comparison shows that the proposed method mitigates the abrupt changes introduced by discrete switching, leading to a smooth and progressive risk estimation which is strictly consistent with domain priors, and providing a highly reliable basis for decision-making of graded early warning.

To further verify whether the suppression of the “step effect” is consistently maintained across different driving behavior transitions, five continuous video sequences with diverse action combinations and durations were selected for quantitative evaluation. The predicted DRI trajectories were compared with the NHTSA-AHP reference trajectories and the conventional discrete class-to-risk mapping in terms of RMSE, MAE, and the maximum instantaneous DRI change Δ D R I max. The multi-sequence evaluation results are summarized in Table 5.

TABLE 5. Multi-sequence evaluation of continuous DRI tracking and step-effect suppression.

Across the five continuous sequences, LTC-CGMN achieves an average RMSE of 0.059 and MAE of 0.036, indicating stable agreement with the expert-prior reference trajectories under different behavior-transition patterns. More importantly, the average maximum instantaneous DRI change is reduced from 0.777 for discrete class-to-risk mapping to 0.122 for LTC-CGMN. Similar reductions are observed in all five sequences, including transitions involving texting, fatigue, phone use, drinking, and reaching behaviors. These results indicate that the suppression of abrupt risk transitions is not limited to the single 30-s example shown in Figure 6, but is consistently observed across multiple continuous driving scenarios. Therefore, the proposed continuous DRI formulation substantially mitigates, rather than completely eliminates, the “step effect” associated with discrete warning outputs.

To independently validate the continuous DRI predictions, five experts in driver safety and human factors who were not involved in the AHP weighting process independently assessed the same five continuous video sequences. Each expert assigned a driver-risk score from 0 to 1 at 1-s intervals based solely on the observed driving behavior, and the mean expert score was used as an independent reference. As shown in Table 6, LTC-CGMN predictions showed strong agreement with the expert assessments, achieving a Pearson correlation coefficient of r = 0.91, an intraclass correlation coefficient of I C C = 0.88, an RMSE of 0.064, and an MAE of 0.041. These results indicate that the continuous DRI trajectories are broadly consistent with independent human judgments rather than merely reproducing the original AHP-derived supervision.

TABLE 6. Agreement between continuous DRI predictions and independent expert assessments.

The high correlation and agreement coefficients indicate that the temporal variation of the predicted DRI closely follows independent expert perceptions of driver risk. Meanwhile, the relatively low RMSE and MAE demonstrate that the agreement is maintained not only in trend but also in absolute risk magnitude. This independent evaluation provides additional evidence for the practical validity of the proposed continuous DRI prediction framework.

4.6 Day/night all-weather and multimodal robustness

The experiment is carried out under four representative operating conditions, including daytime RGB (baseline), nighttime infrared IR (modal adaptive input), low-light Gaussian noise (σ = 25 Gaussian noise added to RGB), and local image occlusion (randomly occluding the face or hand area of the driver, simulating occlusion by passengers or objects). Results are shown in Figure 7 with Top-1 classification accuracy and risk measure MAE reported under each condition.

Figure 7 is a figure with dual Y-axis. The left vertical axis shows the Top-1 accuracy, and the right vertical axis shows the risk measure MAE. This allows to directly compare the two metrics under one horizontal axis. It achieves the best performance with accuracy of 96.37% and MAE of only 0.035 under the daytime RGB benchmarks. In the infrared nighttime conditions, the modal adaptation mechanism can enable the model to fully utilize the IR and depth information, and the accuracy decreases slightly to 94.50% and MAE increases to 0.045, which indicates that the model can still maintain high discriminative ability through the infrared-depth joint features even without the texture of the visible light. The accuracy falls to 92.80%, and the MAE rises to 0.058 under low-light Gaussian noise. The damage on performance is caused by the noise destroying the fine-grained local features. The overall accuracy can still reach more than 92%, which shows that the backbone network and CGSM module have a certain robustness to high-frequency noise. Under partial occlusion condition, the accuracy rate is 93.50%, and the MAE is 0.052, which is better than the pure noise case. The cross-granularity fusion of CGSM can explain this performance, since the macroscopic pose features provide complementary information to alleviate information loss in the case of microscopic local occlusion. The dual axis comparison clearly shows the mirror-image negative correlation between accuracy and MAE, confirming the robustness of the model in complex scenarios with non-ideal lighting, noise and occlusion and providing strong evidence for all-weather engineering deployment.

Because the nighttime subset does not provide head/hand bounding boxes, CGSM cannot perform explicit local-global cross-granularity alignment and instead operates in a global self-attention mode. Therefore, the cross-granularity advantage is weaker at night than in daytime experiments. This is also reflected in the ablation results, where removing CGSM causes a smaller performance drop on Night-IR than on Day-RGB. Accordingly, the nighttime improvement should be attributed mainly to global feature refinement and temporal modeling rather than to full cross-granularity interaction.

In this paper, ablation experiments are performed to quantitatively evaluate the independent contributions of CGSM and LTC to the overall performance. Four comparison settings are built: (1) w/o CGSM: removing CGSM module and only using global torso features; (2) w/o LTC: removing LTC module and only using single-frame spatial features for classification and regression; (3) Standard Conv: replacing depthwise separable causal dilated convolution in LTC with standard temporal convolution; (4) Full Model: full LTC-CGMN. All settings are compared in terms of Top-1 accuracy and RMSE on the Day-RGB and Night-IR subsets. The results are presented in Table 7.

TABLE 7. Two-factor ablation experiment of CGSM and LTC modules on day/night subsets.

Table 7 shows the ablation results for two factors, which reflect the contribution of each module from two dimensions. From the point of view of spatial feature fusion, removing CGSM leads to a 2.25 percentage point drop in the accuracy of Day-RGB and a 24.1% rise in RMSE, while the performance degradation for Night-IR is smaller (accuracy drops by 0.70%, RMSE increases by 15.6%). The difference is caused by the fine-grained head and hand bounding box annotations of the daytime subset that allow CGSM’s cross-granularity fusion to take full advantage of microscopic local cues. In contrast, the lack of annotations in the nighttime subset makes the CGSM degenerate to global self-attention, thus limiting its gain. This shows that supervised fine-grained features are important for the discriminative power of the model. From the perspective of temporal modeling, removing LTC results in the most severe performance collapse—Day-RGB accuracy plummets by 4.52 percentage points, and RMSE soars by 63.8%; Night-IR accuracy decreases by 5.30 percentage points, and RMSE increases by 51.1%, fully demonstrating that driving behavior recognition is highly dependent on temporal context, and single-frame static information is far from sufficient to capture the evolutionary patterns of actions.

Replacing the depthwise separable causal dilated convolution in LTC with standard temporal convolution results in a slight decrease in Day-RGB accuracy from 96.37% to 95.40%, while the RMSE increases from 0.058 to 0.064. On Night-IR, the accuracy slightly decreases from 94.50% to 93.90%, while the RMSE increases from 0.045 to 0.048. These results demonstrate that the design of the depthwise separable causal dilated convolution maintains comparable accuracy to standard convolution while efficiently capturing long temporal dependencies through the dilation factor. Ablation analysis consistently shows that CGSM and LTC work synergistically to form the core performance pillars of LTC-CGMN, and neither can be omitted.

To further disentangle the contribution of individual architectural components, a fine-grained ablation analysis was conducted on the Day-RGB subset. Four additional variants were constructed: (1) w/o Attention, replacing the cross-attention operation in CGSM with direct feature addition; (2) w/o Dilation, setting all temporal dilation factors to 1 while retaining causal convolution; (3) w/o Depthwise, replacing depthwise separable causal dilated convolution with standard causal dilated convolution; and (4) w/o Multi-task, optimizing action classification and DRI regression independently rather than through joint multi-task learning. All other settings were kept unchanged. The results are summarized in Table 8.

TABLE 8. Fine-grained ablation analysis of key LTC-CGMN components.

Table 8 shows that removing cross-attention decreases Top-1 accuracy by 1.16 percentage points and increases RMSE from 0.058 to 0.066, confirming the importance of explicit local-global interaction. Eliminating temporal dilation produces the largest accuracy degradation among the four variants, indicating that an expanded temporal receptive field is essential for capturing long-range behavioral evolution. Replacing depthwise separable convolution with standard convolution increases parameters from 3.78 M to 6.92 M and FLOPs from 5.46 G to 9.87 G, demonstrating that depthwise decomposition primarily contributes to computational efficiency. Finally, removing joint multi-task optimization increases RMSE to 0.067, indicating that action classification provides complementary semantic supervision for continuous risk estimation. Overall, the four components contribute differently but synergistically to the accuracy, temporal modeling capability, lightweight efficiency, and DRI prediction performance of LTC-CGMN.

4.7 Sensitivity analysis of AHP weights

To evaluate whether the predicted DRI is excessively dependent on the specific AHP weight allocation, a sensitivity analysis was conducted by independently increasing or decreasing each risk weight by 10%, followed by normalization to ensure that the three weights summed to one. The baseline weight vector was w = 0.45 , 0.35 , 0.20. During the analysis, the network architecture and all other experimental settings were kept unchanged. RMSE, MAE, and the mean absolute DRI deviation (Δ D R I) from the baseline prediction were used to quantify the influence of weight perturbations. The results are presented in Table 9.

TABLE 9. Sensitivity analysis of AHP weights for DRI prediction.

Table 9 shows that moderate perturbations of the AHP weights cause only limited variations in DRI prediction. Under all ±10% perturbation settings, RMSE remains within 0.058–0.064 and MAE within 0.035–0.039, indicating that the proposed DRI is not highly sensitive to small changes in expert-derived weights. Visual-diversion weight perturbation produces the largest mean DRI deviation (0.013–0.014), followed by manual distraction (0.010–0.011), whereas cognitive-load perturbation has the smallest effect (0.007–0.008). This trend is consistent with the original priority ordering w v > w m > w c. Importantly, no perturbation causes substantial degradation or instability in the predicted DRI, demonstrating that the baseline weight allocation w = 0.45 , 0.35 , 0.20 provides a stable operating point rather than an overly sensitive parameter configuration.

4.8 Model interpretability and visual attention heatmap

This paper utilizes Grad-CAM++ (Gradient-weighted Class Activation Mapping Plus Plus) to generate class activation heatmaps for the network on typical driving actions, visually revealing the decision-making basis of LTC-CGMN. By comparing the overlay of the original images and attention heatmaps from the side and front views, whether the model focuses on micro-local features or macro-global postures in different actions can be intuitively observed, thus providing a visual explanation of the effectiveness of the cross-granularity state mapping module, as shown in Figure 8.

Figure 8 presents Grad-CAM++ activation maps for six representative distraction and fatigue actions from the side and front views. For makeup (A2) and drowsiness (A12), the activated regions are concentrated primarily around the face, head, and hand regions. For texting (A5/A6) and phone use (A7/A8), activations extend from local hand–object interaction regions to broader arm and torso regions. These patterns are consistent with the intended cross-granularity design of CGSM, in which local behavioral cues are interpreted together with global posture information. Although activation maps do not provide causal evidence of the model’s decision process, they offer qualitative support for the local–global feature interactions observed in the quantitative experiments.

5 Conclusion

5.1 Summary

This study addresses two practical challenges in intelligent-cockpit driver monitoring: achieving accurate driver-state recognition under edge-computing constraints and obtaining temporally continuous risk estimates from discrete driving behaviors. To this end, LTC-CGMN integrates lightweight temporal convolution, cross-granularity state mapping, and NHTSA-AHP-guided continuous risk estimation within a unified framework. The LTC module uses depthwise separable causal dilated convolution to capture long-range temporal context with limited computational overhead, while CGSM aligns fine-grained head and hand cues with coarse-grained torso representations through cross-attention. In addition, expert-informed DRI supervision enables the network to generate smoother risk trajectories than conventional discrete class-to-risk mapping. On the 3MDAD dataset, LTC-CGMN achieves a Top-1 accuracy of 96.37%, a DRI RMSE of 0.058, and an end-to-end inference rate of 33.6 FPS on Jetson Orin. The experimental results indicate that the proposed framework provides a favorable balance among recognition accuracy, continuous risk estimation, computational efficiency, and robustness under the evaluated conditions.

5.2 Limitations and future prospects

Despite the favorable performance of LTC-CGMN under multimodal, day/night, and perturbation conditions, several limitations remain. First, the current DRI is primarily derived from structured expert knowledge rather than directly observed driver-risk statistics, such as crash probability, physiological impairment, or naturalistic driving outcomes. Therefore, the DRI should be interpreted as a relative risk indicator under the adopted expert-prior framework rather than an absolute measure of actual driving risk. Future studies should calibrate and externally validate the DRI using large-scale naturalistic driving data and directly observed safety outcomes. Second, the current framework relies mainly on visual sensing, and its robustness may still degrade under extreme conditions such as strong illumination interference, severe occlusion, or substantial image-quality deterioration. Future work may incorporate automotive CAN bus signals and other vehicle-dynamics information to construct a more robust multimodal driver-state measurement framework. Third, the effectiveness of CGSM partly depends on the fine-grained head and hand annotations available in the daytime subset, while its cross-granularity modeling capability is constrained under nighttime or weakly labeled conditions. Future research should therefore investigate self-supervised or weakly supervised cross-granularity representation alignment to reduce dependence on detailed annotations. In addition, further model compression and knowledge distillation can be explored to improve deployment efficiency on resource-constrained automotive platforms.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Chuanfei Hu, Shanghai Institute of Technology, China

Reviewed by: Lei Zhang, Beijing Institute of Technology, China

Yashvir Singh, Harcourt Butler Technical University, India