Design of real-time safety behavior monitoring and violation identification system for construction sites combining YOLOv8-Seg and DeepSORT

1 Introduction

The global expansion of the infrastructure industry is apparent in recent years as the number of international construction projects is growing substantially. Frequent accidents occurring on construction sites make this an emerging challenge. Due to their prolonged exposure to high-density work environments, the use and availability of personal protective equipment (PPE) in a real-time monitoring system is directly related to the safety of employees and compliance of projects. Regulations associated with safety vary across the globe, and on-site management levels are often not sufficient to perform regular inspections in traditional manual inspection modes. Development of an intelligent safety monitoring system for construction projects in network-limited environments presents an opportunity to reduce the number and severity of employee injuries while simultaneously aiding in the protection of employee rights.

According to data from the International Labor Organization (ILO), approximately 17% of annual occupational deaths occur as a result of accidents related to the construction industry worldwide. Failure to wear adequate PPE is the main cause of head injuries and falls (Lee et al., 2023a; Kisaezehra et al., 2023). Due to the lengthy managerial chain that crosses international borders and the inconsistent application of laws and regulations for construction sites worldwide, identifying and holding parties responsible for breaches of safety regulations is particularly difficult (Tsang et al., 2025; Ebekozien et al., 2024). Lee et al. (2023b) performed an empirical analysis of overseas construction projects by South Korean companies and found that there is a statistically significant relationship between a risk perception bias and a cost overrun, and Wang et al. (2023) wrote that construction companies in China continue to rely on manual coordination of personnel location as opposed to current technology that could enable real-time use of this information when performing work on international projects. Rana et al. (2025) further revealed that the carbon emission intensity of Chinese overseas engineering projects is approximately 22% higher than that of similar domestic projects, which reflects the lack of standardization in on-site management. At the construction mode level, Lin et al. (2022)’s study on the offshore construction of low-rise residential buildings in Australia found that the interface confusion between prefabrication and on-site operations often leads to ambiguity in the attribution of safety responsibilities; Park (2023) compared the construction characteristics of timber frame structures in multiple countries and pointed out that construction sites in network-constrained environments generally have phenomena such as dense cross-type work and mixed PPE. The above problems are superimposed on the inherent defects of the manual video inspection mode—relying on the continuous attention of supervisors and having problems of response lag and high missed detection rates in multicamera and large-scene environments, which cannot meet the management needs of real-time early warning and violation recording (Yang et al., 2022).

The development of computer vision technology has provided an automated path for construction site safety monitoring. Liu and Jiao (2022) used a spatiotemporal graph convolutional network to identify unsafe actions of workers, and Ogunseiju et al. (2022) used a deep convolutional network to classify actions in time-series signal images, both of which verified the feasibility of intelligent video analysis. Single-stage target detection methods represented by YOLOv5 and YOLOv8 have achieved high accuracy in PPE detection tasks. Ngoc-Thoan et al. (2024) improved YOLOv5 to realize construction safety early warning, Jung et al. (2023) fused visual and auditory information to detect the actions of construction equipment, and Alateeq et al. (2023) systematically evaluated the performance of various deep learning models in construction site hazard identification. However, the output of the above methods is all rectangular detection boxes. In the scene of dense overlap of workers, the overlap between boxes is serious, and it is impossible to distinguish the equipment ownership of adjacent individuals. Elesawy et al. (2024) found after comparing various YOLO architectures that the error rate of the detection box-level IOU increased significantly under occlusion conditions. The instance segmentation method introduces pixel-level masks, which can improve the accuracy of individual boundaries, but its two-stage architecture has a slow inference speed, which makes it difficult to meet the real-time processing requirements of video streams. At the cross-frame tracking level, Wong et al. (2025) used deep learning for indoor construction progress tracking, and Kikuta and Chun (2024) combined pose estimation and object proximity for action classification, but both used the detection box area as the input for appearance feature extraction. Although the scale-adaptive video detection method proposed by Zhang et al. (2024a) achieved real-time performance in the identification of key objects in the well site, it did not solve the problem of stable association of worker identities in dense scenes. The high degree of collaboration among multiple trades at construction sites in network-constrained environments, the local employment mode, and the differences in worker body size distribution, PPE brand and color standards from country to country, and the fixed viewing angle and low coverage density of monitoring cameras due to limited deployment costs have resulted in workers working in close parallel at the same distance for a long time in the same frame. The superposition of the above conditions makes it possible for adjacent workers with highly similar appearances to be in the same frame area for a long time under a fixed low-angle viewing angle. The cross-frame ID allocation faces continuous high-intensity appearance confusion pressure, and the risk of violating behavior being wrongly attributed to the adjacent individuals is significantly higher in the actual operating conditions of construction sites in network-constrained environments than in the standard test environment.

To address this issue, this article designs an end-to-end real-time safety monitoring and violation identification system for construction sites in network-constrained environments. The system replaces the traditional detection model with YOLOv8-Seg, generating pixel-level instance segmentation masks for each worker and their PPE using CSPDarknet (Cross Stage Partial Darknet) and PANet (Path Aggregation Network). Mask IoU replaces box-level IoU for single-frame PPE compliance determination. The worker mask region is cropped and input into DeepSORT’s Wide ResNet (wide residual network) to extract a 128-dimensional appearance feature vector. Pixel contour features drive cascaded matching and the Hungarian algorithm to achieve stable allocation of unique Track IDs in dense scenes. The violation identification module accumulates frame-by-frame compliance results within a time sliding window, indexed by Track ID. Upon threshold triggering, the violation type, timestamp, and mask are bound together and written to the violation log. The system builds a violation rule base based on ISO 45001 (International Organization for Standardization 45001) and is deployed at the edge to adapt to the network-constrained environment of construction sites in network-constrained environments. The main contributions of this article are as follows: (1) an individual violation attribution framework driven by instance segmentation masks for DeepSORT appearance feature extraction is constructed, fundamentally solving the attribution confusion problem of detection box-based methods in dense scenes. (2) A three-element strong binding mechanism of segmentation mask, track ID, and violation status is constructed to form a traceable violation evidence chain with temporal continuity and individual uniqueness. (3) A complete system architecture conforming to international standards is designed and verified through INT8 quantization and closed-loop deployment at the edge, achieving real-time processing of a single 720p video stream at 28 FPS under network-constrained conditions with a bandwidth limit of 2 Mbps, providing a verifiable engineering implementation path for network-constrained deployment scenarios in construction sites.

Beyond the integration of existing modules, three methodological innovations distinguish this work from conventional system integration approaches. The use of pixel-level segmentation masks as the input to DeepSORT appearance feature extraction, replacing the standard bounding box crops; eliminates background contamination in the feature space; and fundamentally improves identity association in densely overlapping scenes. Mask IoU replaces box-level IoU for single-frame PPE attribution, enabling accurate assignment of protective equipment to individual workers even when detection boxes heavily overlap. The sliding window cumulative violation detection mechanism, indexed by Track ID with a state transition rule that prevents duplicate logging, provides a temporally consistent and evidence-traceable violation confirmation protocol. These innovations are not independent module substitutions but constitute a coherent inference chain where mask-level spatial information drives both tracking and violation attribution in a unified framework.

2 Monitoring system architecture and methodology design

2.1 Overall system architecture and deployment design for network-constrained construction scenarios

Isolated processing of a single module cannot guarantee the complete data flow from the original video frames to traceable violation records. It is necessary to uniformly plan the input and output interfaces of each functional module. A review of deep models for video anomaly detection and the technological innovation of intelligent management of construction safety points out that the real-time monitoring architecture at the edge is an effective way to solve the problems of limited network and diverse standards in network-constrained construction environments (Onal and Dandıl, 2025; Mohy et al., 2026).

The system takes H.264 (Advanced Video Coding) compressed video streams output from multiple IP (Internet Protocol) cameras as raw input. The edge device performs real-time decoding on the video stream and samples the decoded frame sequence at fixed intervals Δ t to generate RGB (red, green, and blue) image frames F t with a uniform resolution of W × H, where t is the frame timestamp (Alshboul et al., 2026; Zhang et al., 2024). F t enters the instance segmentation module and outputs the pixel-level mask set M t = m i , c i , s i i = 1 N t for all workers and PPEs in the current frame, where m i is the binary mask matrix of the ith instance, c i is the class label, s i is the confidence score, and N t is the total number of instances detected in the current frame. The behavior monitoring module and the cross-frame tracking module are simultaneously fed into the M t input. The behavior monitoring module uses M t to complete single-frame PPE attribution binding and compliance status determination. The cross-frame tracking module uses the clipped region of the worker’s torso mask m i w o r k e r in M t to complete appearance feature extraction and Track ID allocation, outputting a worker status sequence T t = i d j , b j , f j a p p j = 1 m t with identification. Among them, i d j is the unique trajectory number of the jth worker, b ^ j is the Kalman filter predicted bounding box, f j a p p ∈ R 128 is the appearance feature vector, and m t is the number of tracked workers in the current frame. The violation determination module uses i d j as an index to bind the behavior monitoring results with the tracking identity, accumulates violation frame determinations within a sliding window, and generates a structured violation record V upon triggering, where τ is the violation timestamp; l is the violation type label; and m * is the corresponding mask region. V is written to the violation log and triggers the warning output module.Here, r j represents the proportion of violation frames for worker i d j within the sliding window, and θ is a preset judgment threshold. Individuals meeting the conditions trigger violation confirmation and generate binding records.

V = i d j , τ , l , m * ∣ r j > θ , i d j ∈ T t . (1)

The violation rule base formally encodes the regulations on mandatory PPE wearing requirements in construction operations, transforming the compliance conditions for wearing safety helmets, reflective vests, and safety belts into logical constraints for mask attribution judgment. The rule base outputs a Boolean compliance flag δ j t ∈ 0 , 1, which directly participates in the cumulative calculation of r j (Tannor et al., 2022; Ebekozien, 2022).Here, W represents the sliding window length (unit: frames), t 0 represents the window’s starting frame number, 1 · represents the indicator function, and r j quantizes the persistence of violation of individual i d j within the window.

r j = 1 W ∑ t = t 0 t 0 + W ‐ 1 1 δ j t = 0 . (2)

For construction site environments with limited network connectivity, the entire system is deployed on edge computing devices. Both YOLOv8-Seg (Khatua et al., 2024; Qiu et al., 2024) and Wide ResNet employ INT8 quantization compression. While maintaining inference accuracy loss below 1.5%, the end-to-end inference latency per frame is controlled within the real-time processing frame rate requirements, enabling the system to complete the entire detection, tracking, and violation determination loop without continuous cloud upload dependency. The overall architecture and deployment of the edge-based intelligent building safety monitoring system are shown in Figure 1.

The system in Figure 1 processes multi-camera video streams through instance segmentation, behavior monitoring, cross-frame tracking, and violation judgment modules. It generates V according to ISO 45001 rules and outputs alarms and logs on edge devices. INT8 quantization supports real-time performance in bandwidth-constrained construction environments.

2.2 Pixel-level instance segmentation of workers and PPEs based on YOLOv8-Seg

YOLOv8-Seg is selected as the instance segmentation backbone for three reasons. As a single-stage instance segmentation method, YOLOv8-Seg achieves a balance between segmentation accuracy and inference speed that is critical for real-time edge deployment. Compared to two-stage methods such as Mask R-CNN, which require separate region proposal and mask prediction stages, YOLOv8-Seg produces masks in a single forward pass, reducing end-to-end latency by approximately 40% on the same hardware. YOLOv8-Seg outperforms its predecessor YOLOv5-seg by 2.5 percentage points in mask mAP while maintaining comparable inference speed. Compared to other single-stage segmentation methods such as YOLACT, YOLOv8-Seg demonstrates superior mask quality with lower computational cost. These characteristics make YOLOv8-Seg suitable for the construction site scenario, where real-time processing on resource-constrained edge devices is required without compromising segmentation accuracy in dense, occluded scenes. Workers and PPE targets in construction site monitoring frames vary greatly in scale, pose, and occlusion degree. Rectangular detection boxes cannot accurately define the spatial occupancy boundaries of individual instances, resulting in a lack of reliable pixel-level region support for subsequent PPE attribution binding and individual identity feature extraction. In densely occluded scenes, pixel-level segmentation methods can more accurately locate individual boundaries compared with detection box methods (Wang et al., 2024; Rashidi et al., 2025).

F t ∈ R H × W × 3 is input into the CSPDarknet backbone network, and after passing through the cross stage partial connection structure, feature maps X 3 , X 4 , X 5 with a step size of 8 , 16 , 32 are output at three downsampling scales P 3 , P 4 , P 5, respectively, with the number of channels being 256 , 512 , 1024, respectively. The shallow feature map X 3 preserves fine spatial details to deal with small-scale PPE targets, and the deep feature map X 5 encodes high semantic discriminative information to distinguish worker torso categories. The three-level feature maps together constitute the input state for subsequent fusion (Chen et al., 2026).

To enable the segmentation head to simultaneously acquire fine-grained spatial boundaries and high-level semantic responses, PANet performs top-down and bottom-up dual-path aggregation on X 3 , X 4 , X 5 (Kridalukmana et al., 2025). The top–down path upsamples X 5 and adds it element-wise to X 4, then upsamples the result and fuses it with X 3 to generate the fused feature sequence X ∼ 3 , X ∼ 4 , X ∼ 5. The bottom-up path performs downsampling convolutions with a stride of two in the opposite direction and stacks them layer by layer, ultimately outputting multiscale fused features P 3 o u t , P 4 o u t , P 5 o u t. After fusion, each scale feature simultaneously possesses local edge responses and global semantic constraints, providing conditions for accurate generation of the segmentation mask.

The segmentation head performs instance prediction on P 3 o u t , P 4 o u t , P 5 o u t. To quantify the detection confidence of each candidate instance, a joint s i is defined for the ith candidate instance:Here, p i o b j ∈ 0 , 1 is the target existence probability; p i c l s ∈ 0 , 1 is the class conditional probability; and s i is the non-maximum suppression filter for overlapping predictions after all candidate instances are sorted in descending order. In the instance set retained after filtering, the mask of each instance is generated by a linear combination of prototype masks. Let the prototype mask library be p r o t o k k = 1 K, K = 32 is the number of prototypes, and each instance i corresponds to the combination coefficient vector α i ∈ R K. The pixel-level m i is determined by the following formula:Here, σ · is the sigmoid activation function. A threshold of 0.5 maps continuous responses to m i ∈ 0 , 1 H × W, marking the precise pixel occupancy of the instance on the frame plane.

s i = p i o b j · p i c l s . (3)

m i = 1 σ ∑ k = 1 K α i k · p r o t o k > 0.5 . (4)

2.3 Single-frame PPE attribution binding and compliance status determination based on mask IoU

In M t, m i w o r k e r and PPE masks exist as independent instances, lacking an explicit attribution relationship (Yin et al., 2024). In scenarios with dense worker overlap, the spatial location of a single PPE instance overlaps with the torso areas of multiple workers simultaneously, making it impossible to directly determine which individual worker the PPE belongs to, thus preventing the compliance status from being accurately bound to a single individual (Zaryczańska and Karwot, 2025).

From M t, M t w o r k e r = m i w o r k e r i = 1 N w and the PPE mask subsetM t p p e = m q p p e q = 1 N p are separated according to c i, where N w and N p are the total number of workers and the total number of PPE instances in the current frame, respectively. To quantify the spatial overlap between the qth PPE instance and the torso of the ith worker, the pixel-level IoU is calculated:Here, ∣ · ∣ represents the total number of pixels with a value of 1 in m i, and I o U q i quantifies the proportion of pixels shared by m q p p e and m i w o r k e r on the frame plane. Compared with the detection box-level IoU, the mask-level calculation eliminates the interference of background pixels within the bounding box and can still accurately reflect the actual spatial association strength between the worker’s torso and PPE when they are irregularly shaped and mutually occluded (Wang et al., 2025).

I o U q i = ∣ m q p p e ∩ m i w o r k e r ∣ ∣ m q p p e ∪ m i w o r k e r ∣ . (5)

I o U q i constitutes the attribution decision matrix A ∈ R N p × N w, and the element in the qth row and ith column of the matrix is I o U q i. According to the maximum IoU principle, the worker index i q * of the qth PPE instance is determined by the following formula:

i q * = arg ⁡ max ⁡ i ∈ 1 , … , N w I o U q i . (6)

The PPE instance is uniquely mapped to the worker individual with the highest spatial overlap. When the maximum I o U q i is lower than the minimum attribution threshold ϵ I o U, the PPE instance is marked as an unclaimed instance and excluded from the compliance determination of this frame, so as to avoid noise detection affecting the determination result. After attribution, each worker i obtains its assigned PPE category set Q i = c q p p e ∣ i q * = i, where c q p p e is the c i of the qth PPE instance.

The ISO 45001 rule base defines a mandatory PPE type set R = h e l m e t , v e s t , h a r n e s s for construction work scenarios. Compliance determination uses a set inclusion check between Q i and R. The compliance status flag δ i t for worker i in frame t is given by the following formula: 1 · sets δ i t = 1 only when Q i completely contains all mandatory categories in R; otherwise, δ i t = 0. δ i t is written to the frame-by-frame compliance status sequence with the worker Track ID as the index, directly participates in the cumulative calculation of r j, and constitutes the input status of the violation identification module. At the same time, the m i w o r k e r of the worker corresponding to δ i t = 0 is marked as a violation frame m * and enters the ternary binding log generation process.

δ i t = 1 R ⊆ Q i . (7)

2.4 Continuous tracking of worker identity across frames in dense scenes based on DeepSORT

The single-frame δ i t identifies the worker with the intraframe instance index i. This index does not have continuity between adjacent frames. When the worker crosses over or reappears after a short period of occlusion, the intraframe index jumps, causing the frame-by-frame compliance sequence to be unable to align to the same physical individual, and the cumulative calculation of violation frames loses the individual attribution basis (Song et al., 2023).

After pixel coordinate cropping, m i w o r k e r obtains RGB image patches P i t bounded by a masked bounding box. These P i t patches are then scaled to a uniform 64 × 128 resolution and input into Wide ResNet. Deep residual feature extraction outputs a 128-dimensional appearance feature vector f i a p p ∈ R 128 normalized to l 2. By replacing the detection box cropping region with the masked cropping region, the contamination of appearance features by background pixels within the bounding box is eliminated.

The Kalman filter maintains a state vector x j = u j , v j , γ j , h j , u ˙ j , v ˙ j , γ ˙ j , h ˙ j ⊤ for each active trajectory T j. Among them, u j and v j are the bounding box center coordinates; γ j is the aspect ratio; h j is the box height; and u ˙ j, v ˙ j, γ ˙ j, and h ˙ j are the corresponding velocity components. To predict the position of each trajectory before detecting a worker M t w o r k e r in the current frame, the Kalman prediction step propagates the state vector forward according to a uniform motion model, outputting the predicted state x ^ j t ∣ t ‐ 1 and the prediction covariance matrix P ^ j t ∣ t ‐ 1. The predicted bounding box b ^ j is directly decoded from the first four components of x ^ j t ∣ t ‐ 1.

Cascaded matching is performed round-by-round in ascending order of trajectory survival frame age. For each T j and each detection instance i in the current frame, a cost matrix element fusing motion distance and appearance distance is constructed:Here, d j i m a h is the Mahalanobis distance between the predicted state x ^ j t ∣ t ‐ 1 and the detected bounding box b ^ i; d j i cos is the cosine distance between the mean vector of trajectory appearance features f j ˉ a p p and the appearance features f i a p p of the detected instance; λ ∈ 0 , 1 is the weighting coefficient, controlling the relative weight of motion constraints and appearance constraints. d j i comprehensively utilizes the constraints of motion prediction on position and the constraints of mask appearance features on identity, enabling the correct association of workers who are geographically close but visually distinguishable in dense scenes.

d j i = λ · d j i m a h + 1 ‐ λ · d j i cos . (8)

The Hungarian algorithm is applied to the cost matrix to solve for the optimal binary matching, outputting the optimal allocation scheme from detected instances to trajectories. Unmatched detected instances are initialized with new trajectories and assigned a unique Track ID i d j. Trajectories that are unmatched for more than T l o s t frames are marked as terminated states and removed from the T j set, where T l o s t is the threshold for the number of frames a trajectory is retained. Successfully matched trajectories undergo a Kalman update step, using the current frame’s detected bounding box b ^ i as the observation to correct the state estimate. The updated x j t ∣ t is used as the initial state for the next frame prediction, and the mean vector of trajectory appearance features is updated using an exponential moving average.Here, η is the feature update learning rate. During the trajectory’s existence, f j ˉ a p p continuously accumulates the worker’s appearance distribution information, enhancing the robustness of cross-occlusion re-identification. The identified state sequence T t = i d j , b j , f j a p p j = 1 M t output in each frame binds Track ID i d j with the current frame δ i t, forming the input state of the sliding window violation identification module.

f j ˉ a p p ← 1 ‐ η f j ˉ a p p + η f i a p p . (9)

2.5 Individual violation identification and ternary binding log generation based on sliding window

A single-frame δ j t records the instantaneous state of worker i d j in frame t in binary form. Changes in lighting, brief occlusions, and fluctuations in segmentation confidence can all trigger isolated erroneous frames in the construction site. Directly triggering violation judgment with a single-frame δ j t can generate a large number of false alarms, and V cannot serve as management evidence with temporal continuity. For real-time detection of unsafe behaviors in construction sites, previous studies have used the Unsafe Net network, which integrates YOLOv4 and ConvLSTM (convolutional long-term memory network), or combined computer vision and knowledge graph reasoning for violation identification (Mei et al., 2025; Chandrakala et al., 2023).

A frame-by-frame compliance state sequence △ j = δ j t t = 1 T is maintained using Track ID i d j as the index, where T is the cumulative number of frames for the current trajectory. To suppress instantaneous false judgments while retaining the detection sensitivity for continuous violations, a sliding window of length W frames is applied to the sequence Δ j, with the window ending at the current frame t. r j t is calculated using the following formula:Here, δ j k = 0 indicates that worker i d j has a PPE missing violation in frame k. r j t ∈ 0 , 1 quantizes the temporal density of violation frames within the window. r j t is updated in real-time as the window slides frame-b- frame. The response weight for isolated erroneous frames is evenly distributed across the denominator of W frames, effectively suppressing the impact of transient disturbances on the judgment result.

r j t = 1 W ∑ k = t ‐ W + 1 t 1 δ j k = 0 . (10)

Violation confirmation is based on the relationship between r j t and θ. To accurately locate the violation trigger moment and avoid the same persistent violation event being repeatedly written to the log, a violation status flag ϕ j ∈ 0 , 1 is introduced. ϕ j is initialized to 0. The violation confirmation condition and state transition rule are defined using the following formula:

ϕ j t = 1 r j t > θ · 1 ϕ j t ‐ 1 = 0 . (11)

The condition ϕ j t = 1 holds only when r j t first exceeds the threshold θ, that is, the previous frame’s flag ϕ j t ‐ 1 = 0 and the current frame’s violation percentage exceeds the threshold. This design ensures that each continuous violation event triggers only one log write, eliminating duplicate records caused by continuous threshold exceedances in the sliding window.

When a violation is detected, the system identifies the timestamp of the frame with the highest proportion of violations within the Δ j window, retrieves the corresponding worker mask records, determines the missing PPE category using the rule library, and generates a triplet to write into the structured violation log. The log is uniquely identified by a composite key that combines identifier and category, supporting both two-dimensional retrieval and pixel-level visual evidence storage. The system triggers tiered alerts based on the number of missing PPE categories: One missing category triggers a Level 1 alert, two or more categories trigger a Level 2 alert, and these alerts are sent in real-time to management terminals.

3 Experimental configuration and dataset construction

3.1 Experimental hardware environment and edge deployment configuration

The training platform is configured with an NVIDIA RTX 3090 GPU (graphics processing unit) with 24 GB of VRAM and 64 GB of RAM. The deep learning framework used is PyTorch 2.0.1, CUDA (Compute Unified Device Architecture) version 12.1, and cuDNN (CUDA Deep Neural Network) version 8.9.2. The YOLOv8-Seg training batch size is set to 16, the initial learning rate is 0.01, a cosine annealing scheduling strategy is used, and the training epochs are 300. The input resolution is uniformly 640 × 640. In DeepSORT, Wide ResNet is pretrained on the Market-1501 dataset. The maximum trajectory retention frame count T l o s t for cascaded matching is set to 30 frames, the appearance η is set to 0.1, and λ is set to 0.7.

To verify the convergence quality of the model under the training configuration, Figure 2 records the round-by-round changes of the loss function and the validation set mAP@0.5 of YOLOv8-Seg during 300 training rounds.

In Figure 2a, all three losses plateau after approximately 200 epochs. The convergence speed of the segmentation loss is slower than that of the classification and bounding box losses, consistent with the training objective of simultaneously optimizing the mask coefficients and the prototype library in the segmentation head. In Figure 2b, the validation set mAP@0.5 converges stably to 87.3% in the later stages of training, with the curve fluctuation controlled within 0.8 percentage points, indicating that the cosine annealing learning rate scheduling effectively suppresses later oscillations, and no overfitting is observed during training.

The edge inference device uses NVIDIA Jetson AGX Xavier, ARM Cortex-A57 architecture, 16 GB LPDDR4x memory, and TensorRT 8.5 inference framework. Both YOLOv8-Seg and Wide ResNet perform INT8 quantization, and the quantization calibration set is randomly selected from 500 images in the training set. The edge network is constrained, simulating an uplink bandwidth limit of 2 Mbps. The video stream uses H.264 compression, with a resolution of 1,280 × 720.

3.2 Dataset organization, classification, and pixel-level annotation specifications

The experimental dataset has been created using data from three different publicly available datasets: CHV (providing 6,234 annotated images for eight PPE categories), SHEL5K (providing 5,000 each for helmets), and Roboflow Construction Safety (103,500 originals). After filtering the images in the three datasets, it was found that there were a total of 3638 images that met the feature requirements for pixel level mask annotation. These images can be divided into four categories: worker torso images, helmet images, reflective vest images, and safety belt images. The three datasets then had their resolutions normalized and were combined to provide 14,872 unique images in total, with examples having low, medium, and high densities.

Four types of annotations are as follows: worker torso, safety helmet, reflective vest, and safety belt. Each individual instance of PPE is annotated using a polygon annotation tool and has a boundary accuracy of no more than three pixels from the true boundary of the object. Instances of PPE and m i w o r k e r are both annotated independently and are not merged into a single instance. The dataset is divided into three sets of images: a training set with 10,410 total images, validation with 1,487 total images, and test with 2,975 total images, in the ratio of 7:1:2. All sets have an even distribution of low-, medium-, and high-density scenes. Worker density is defined as the number of workers fully visible in each frame, with the three categories defined as follows: 1–4 people for low-density, 5–9 people for medium density, and 10 or more people for high-density.

3.3 Evaluation index system and baseline experimental design

The mAP@0.5 metric is adopted as the primary segmentation evaluation criterion. The 0.5 IoU threshold represents the de facto standard in the construction safety monitoring literature for PPE detection and worker segmentation tasks, enabling direct comparison with existing YOLO-based construction site studies which predominantly report mAP@0.5 as their primary performance indicator. The 0.5 IoU setting provides a balanced measure that tolerates minor boundary imprecision caused by occlusion while still requiring substantial spatial overlap between predicted and ground-truth masks, making it suitable for evaluating instance segmentation in densely populated construction scenes. The mAP@0.75 metric is adopted to evaluate segmentation performance under a stricter IoU threshold of 0.75. The F1 score, computed as the harmonic mean of precision and recall, is reported for violation detection performance. Model computational efficiency is quantified by the number of parameters (Params) and floating-point operations per second (FLOPs), both measured on the NVIDIA Jetson AGX Xavier edge device under INT8 quantization. Each target category is used to determine the average for each target category and then statistically analyze the differences between the four target categories. The second component of the multimetric evaluation of identity association performance is cross-frame identity association performance using IDF1 and IDSW (identity switches). IDF1 measures both the precision and recall of identity association, whereas IDSW counts the overall number of Track ID switches throughout the entire tracking period for the monitored object. The violation attribution accuracy is defined as the percentage of violation events correctly associated with the actual violating worker’s identity, and this value can be derived from manually identified violation events. The edge processing frame rate is computed as the reciprocal of the end-to-end single-frame inference latency over 200 frames of running time at stable operating conditions.

The baseline method uses the standard configuration of connecting the output from the YOLOv8 detector box directly to DeepSORT (i.e., the YOLOv8 box clips replace mask clips when inputting into Wide ResNet) and using box-level IoU to calculate PPE attribution; therefore, hyperparameters are kept the same for the system method. Experimental groups consist of a mask method group versus a baseline box method group that execute independently using the same test set and hardware platform. Evaluation metrics are averaged over three separate experiments to eliminate variability due to random initialization.

4 Results

4.1 Segmentation accuracy of workers and PPE instances in dense scenes

Worker density is an important driving variable for how well instance segmentation can be completed. At low densities, there is a clear separation between the different instances creating the mask boundaries for each instance. However, the overlap of the pixel areas of the torso of the worker and the PPE is unavoidable when a worker is in high-density, making it increasingly difficult to predict the mask coefficients of the segmentation head. Additionally, due to morphological differences, different target categories react to density changes differently. To quantitatively illustrate specifically how density changes affect the detection accuracy of each category, as well as to evaluate the overall recognition ability of YOLOv8-Seg across four construction category target classes within construction scenes, the mAP@0.5 results of the experimental model for each of the four target categories as a function of low, medium, or high-density are presented in Figure 3.

Figure 3 shows that in low-density scenarios, the occlusion rate between worker instances is low, and the mask boundary generation for all four target categories is accurate, with a total average of 92.6%. In medium density scenarios, partial overlap occurs in the torso masks between workers. The safety belt category experiences the largest decrease in AP due to its small size and flexible shape, with the total average dropping to 88.3%. In high-density scenarios, instance overlap is significantly aggravated, with the safety belt mAP@0.5 dropping to 77.7%, the lowest among the four categories, reflecting the accuracy loss of the segmentation head in generating masks for small categories under extreme overlap conditions. The weighted average for the three density scenarios is 87.3% (=0.3* low density scenario total average +0.4medium density scenario total average + 0.3 high density scenario total average). Out of all four categories, the torso generates the highest total weighted average (89.2%), whereas the safety belt receives the lowest (85.2%) over the four categories, resulting in a difference between their scores of four percent. Long oval-shaped targets in the instance segmentation masks can be an area where further optimizations can be made in the future.

Table 1 presents the mAP@0.75, F1 score, Params, and GFLOPs of the YOLOv8-Seg model on the test set under different density scenarios. The mAP@0.75 values decline from 84.7% in low-density scenes to 72.5% in high-density scenes, reflecting the sensitivity of the stricter IoU threshold to mask boundary precision under occlusion. The F1 score remains above 89.0% across all density scenarios, indicating a balanced trade-off between precision and recall for violation detection. The model contains 11.2 million parameters and operates at 14.8 GFLOPs under INT8 quantization on the edge device.

TABLE 1. Segmentation performance and computational cost of YOLOv8-Seg under different density scenarios.

A comparative analysis was conducted between the proposed system predictions and human expert annotations on 500 randomly selected test frames. Three certified construction safety inspectors independently reviewed the same frames and annotated PPE compliance status. The system achieved a per frame compliance status agreement rate of 91.7% with the majority-vote consensus of the three human inspectors. The primary sources of disagreement were marginal occlusion cases where inspectors also exhibited inter-rater variability, with Cohen’s κ of 0.84 among the three inspectors. The average time required for manual inspection of a single 720p frame was 4.2 s per inspector, whereas the proposed system processes each frame in 35.7 m, demonstrating a speed advantage of two orders of magnitude while maintaining agreement rates above 90% with human expert consensus.

4.2 PPE attribution binding accuracy based on mask IoU

When many workers are near each other, their detection boxes often overlap. The way the boxes are assigned to the PPE instances by using the box-level IoU can favor putting an instance of PPE on a bounding box that has more area, so when there are lots of workers, it makes this bias much stronger. By accurately displaying the pixel contour boundaries of a worker’s torso and comparing them with the PPE’s contour boundaries, the mask-level IoU accurately builds a spatial relationship between the torso of a worker and the PPE, and importantly does not allow any of the background of the bounding box to contaminate the attributions made on bounding boxes under these circumstances. In Figure 4, the data that have been calculated for the difference in the ability of the mask-level IoU versus box-level IoU to make accurate attribution to the PPE when comparing the different worker densities for frames that have between 2 and 16 workers per frame.

The two methods show very close results to one another in terms of accuracy when the number of workers is low (as shown in Figure 4). The accuracy of box-level IoU reaches 97.8% when the number of workers reaches 2. However, when the density of workers reaches 16 workers per frame, this method collapses to an accuracy of 68.5%. This is primarily due to the number of worker detection boxes, which often overlap and therefore lead to mislabeling of PPE for spatially adjacent but not corresponding workers. In contrast, at the same density (16 workers), the mask-level IoU method achieves an accuracy of 89.3%, because the mask precisely defines the boundary of each instance and does not allow background areas in the bounding box to interfere with the results. Therefore, as the density of workers exceeds eight per frame, the gap between the two methods widens, confirming the considerable benefit of mask-level IoU in densely populated environments.

4.3 Stability of cross-frame identity association and ID switching rate

The stability of cross-frame identity association directly determines the accuracy of individual attribution for accumulated erroneous frames: A decrease in IDF1 indicates that more frames are incorrectly assigned to non-corresponding trajectories, whereas an increase in IDSW reflects frequent ID switching; together, these metrics demonstrate the system’s capacity to maintain worker identity continuity in dense intersection scenarios. Table 2 presents experimental results comparing the mask-based approach with the frame-level baseline across eight test sequences reconstructed from the Roboflow Construction Safety dataset, selected based on criteria requiring at least 300 frames per sequence, identifiable workers in each frame, and coverage spanning a complete spectrum from low to high-density, numbered sequentially from Seq-01 to Seq-08 by average number of workers per frame.

TABLE 2. Comparison of cross-frame identity association indicators for each test sequence.

In Table 2, the masking method achieves a mean IDF1 of 84.4% across the eight sequences, compared with a baseline mean IDF1 of 77.1%, a difference of 7.3 percentage points. Regarding IDSW, the masking method achieves a mean of 24.5 times, whereas the baseline achieves 37.4 times, representing a reduction of 34.5%. With increasing sequence density, the IDF1 of both methods decreases, but the masking method maintains an IDF1 above 80% in the high-density sequences Seq-07 and Seq-08, whereas the baseline method drops to the 72%–74% range. The contour-level appearance features provided by the masked cropping region preserve richer individual distinguishing information in scenarios where workers’ PPE appearances are highly similar, resulting in higher feature discrimination in the cosine distance calculation of cascaded matching, thus effectively suppressing ID switching in densely overlapping scenarios.

The quantitative metrics of IDF1 and IDSW reveal the tracking advantages of the masking method. However, the root of this advantage lies in the impact of the two input methods on the distribution of features with the same ID and across IDs at the feature space level: If mask clipping can make features with the same ID more tightly clustered in the cosine space and features with cross IDs more dispersed, then the cosine distance calculation of cascaded matching has higher identity discrimination, thus producing a more accurate allocation scheme in the Hungarian algorithm solution.

To further demonstrate the advantage of the proposed mask-driven tracking framework, additional comparative experiments were conducted against three alternative configurations: YOLOv5-seg with DeepSORT using mask cropping, Mask R-CNN with DeepSORT using mask cropping, and YOLOv8-Seg with SORT without appearance features. Table 3 presents the tracking performance of these configurations across the eight test sequences.

TABLE 3. Comparison of tracking performance across different configurations on eight test sequences.

The proposed YOLOv8-Seg with DeepSORT using mask cropping configuration achieves the highest mean IDF1 of 84.4% and the lowest mean IDSW of 24.5% (see Table 3). YOLOv5-seg with DeepSORT using mask cropping yields a mean IDF1 of 80.7%, reflecting the improved feature representation of YOLOv8-Seg over its predecessor. Mask R-CNN with DeepSORT using mask cropping achieves a mean IDF1 of 82.1% but at a substantially higher computational cost of 48.7 m per frame on the edge device versus 35.7 m for the proposed method, rendering it less suitable for real-time edge deployment. YOLOv8-Seg with SORT without appearance features yields a mean IDF1 of only 72.3%, confirming the critical role of appearance-based reidentification in dense worker tracking scenarios. These results validate that the combination of YOLOv8-Seg mask cropping and DeepSORT appearance features provides the optimal trade-off between tracking accuracy and computational efficiency.

To explain the intrinsic mechanism of the improved tracking performance of the masking method at the feature space level, Figure 5 compares the cosine distance distribution of the Wide ResNet output features under two input methods: mask clipping and bounding box clipping.

Figure 5 shows that in Figure 5a, the distance distribution within the same ID of the mask clipping features is concentrated around a cosine distance of 0.1, whereas the peak of the distance distribution across IDs is above 0.6. The overlapping area is significantly smaller than the box clipping result shown in Figure 5b. The background pixels introduced by the box clipping region highly overlap with adjacent worker areas when workers are densely arranged, causing the distance distributions of features within the same ID and features across IDs to drift toward each other. Consequently, the cosine distance term of the cascaded matching loses its effective identity differentiation signal, thus triggering ID switching. Mask clipping uses the torso contour as the sole input source; the elimination of background contamination directly expands the interclass spacing of the feature distribution, providing a more accurate cost matrix input for the Hungarian algorithm.

4.4 Impact of sliding window parameters on violation detection performance

W takes values within the range of 10 , 15 , 20 , 25 , 30 frames, and θ takes values within the range of 0.3 , 0.4 , 0.5 , 0.6 , 0.7, for a total of 25 parameter combinations. The F1 score for each combination is calculated from the violation event annotation results on the test set. The impact of different parameter combinations on F1 exhibits a non-linear, bi-directional sensitivity: When the window is too short, the single-frame segmentation perturbation is not sufficiently smoothed, increasing the false alarm rate and lowering the precision; when the window is too long, the violation event response delay increases, increasing the false negative rate and lowering the recall; when the threshold deviates from the optimal point, it further exacerbates the imbalance between the two types of errors. To intuitively locate the optimal parameter combination and quantify the robustness boundary of the system to parameter perturbations, Table 4 summarizes the F1 score test results for the 25 parameter combinations.

TABLE 4. F1 scores (%) under sliding window parameter combinations.

Table 4 shows that the optimal parameter combination is W = 20 frames and θ = 0.5, corresponding to an F1 score of 94.1%. When the window length is too short, the instantaneous segmentation perturbation is not sufficiently smoothed, leading to an increase in the false alarm rate and a decrease in precision; when the window length is too long, the response delay to violations increases, leading to an increase in the false negative rate and a decrease in recall. The two types of errors reach a balance when the threshold θ is 0.5. If it is too low, noisy frames trigger false alarms; if it is too high, short-duration true violations are missed. The F1 score monotonically decreases from the peak value of W = 20 and θ = 0.5 toward the surrounding area. Parameter sensitivity analysis shows that the F1 score change of the system to a window perturbation of ± 5 frames does not exceed 2.7 percentage points, demonstrating good parameter robustness. Figure 6 visually locates the optimal parameter combination and verifies the system’s robustness to parameter perturbations.

Figure 6 presents the F1 value distribution for window length W ranging from 10 to 30 frames and threshold θ ranging from 0.3 to 0.7, using filled contour lines. The F1 value reaches its peak of 94.1% at W = 20 frames and θ = 0.5, and the contour lines monotonically decrease from there. When the window is too short, the F1 decrease is not smoothed out due to instantaneous disturbances; when the window is too long, the F1 decrease is due to increased response delay; a threshold deviation of 0.5 leads to an increase in false positives or false negatives.

4.5 Violation attribution accuracy

In scenarios with different worker densities, the spatial location and appearance differences between adjacent individuals vary, and the impact on attribution accuracy is density-dependent: At low densities, individual boundaries are clear, and both methods can achieve high attribution accuracy; as density increases, the box-level method suffers from increased PPE attribution confusion due to large-area overlap of bounding boxes, resulting in a significant increase in attribution error rate, whereas the mask method, through pixel-level boundary constraints and the Track ID ternary binding mechanism, exhibits lower density sensitivity in attribution accuracy. To quantitatively compare the attribution capabilities of the two methods under different density scenarios, Table 5 presents the measured results of violation attribution accuracy for the masking method and the box-level baseline in low-, medium-, and high-density scenarios, as well as on the overall test set.

TABLE 5. Comparison of violation attribution accuracy under different density scenarios (%).

In Table 5, the overall accuracy rate of attribute attribution for the mask method is 91.2%, representing an improvement of 11.6 percentage points compared with the box-level baseline (79.6%). In low-density scenarios, the difference between the two methods is only 1.8 percentage points, indicating that both detection boxes and masks can effectively distinguish individual boundaries; in medium density scenarios, the gap widens to 8.7 percentage points due to boundary box overlap causing confusion in PPE assignment by the box-level method; in high-density scenarios, the box-level baseline drops to 68.2%, below the acceptable threshold, whereas the mask method maintains 84.6%, demonstrating effective suppression of attribute drift at the pixel-level. Remaining attribution errors primarily arise from extreme cases involving multiple workers violating rules simultaneously and overlapping torso masks, but their cumulative impact is limited when analyzed using a sliding window approach.

4.6 Edge computing performance and system processing efficiency

The real-time performance of the system on the edge device depends on the sum of the computation time of each functional module. The instance segmentation module, involving backbone network forward propagation and prototype mask decoding, is expected to account for the largest proportion of computation, whereas violation determination and log writing are lightweight CPU (central processing unit) logical operations, contributing little to the total latency. To accurately quantify the proportion of each module’s share of the overall computational budget and locate the bottleneck affecting real-time performance, Table 6 presents the measured decomposition results and proportions of the single-frame computation time of each module on the Jetson AGX Xavier edge device.

TABLE 6. Breakdown of single-frame computation time for each module (ms).

Table 6 shows that the instance segmentation module takes 19.8 m, accounting for 55.5% of the total time; appearance feature extraction takes 7.4 m, accounting for 20.7%; and the cascade matching and Hungarian algorithm have a stable single-frame computation time of less than 5.2 m due to the limited number of workers. Violation detection and log writing are executed purely on the CPU, taking 3.3 m.

The edge device achieves real-time processing of 28 FPS under a single video stream. However, as the number of concurrent cameras increases, computing resources are distributed among multiple inference tasks, resulting in a non-linear decay in frame rate. The server side benefits from higher GPU computing power and memory bandwidth, exhibiting significantly better multichannel concurrent scalability than the edge device. The difference in frame rate decay curves between the two directly determines the upper limit of camera coverage density in actual deployment. To quantify the difference in frame rate scalability between the edge and server sides under multichannel concurrency conditions and to verify the current single-channel real-time deployment positioning of edge devices, Figure 7 shows a comparative end-to-end processing frame rate measurement between the two platforms under one to four concurrent cameras.

The colored dots in Figure 7 represent the measured frame rates, with the dashed lines projected onto the bottom surface. With a single video stream, the edge side achieves a frame rate of 28 FPS, meeting real-time requirements. With two concurrent streams, the frame rate drops to 16 FPS, whereas the server side maintains 29 FPS with four concurrent streams. The bottom projection points visually reflect the steeper performance degradation curve of the edge side as the number of concurrent streams increases, indicating that the current upper limit of edge device computing power is single-stream real-time deployment.

To demonstrate the actual effects of INT8 quantization on inference acceleration and accuracy loss, Figure 8 compares the FP32 and INT8 latency of each module item by item, and presents the trade-off between model volume compression and mAP loss.

In Figure 8a, the INT8 latency of YOLOv8-Seg is compressed from 47.9 m on the FP32 benchmark to 19.8 m, achieving a compression ratio of 58.7%. Wide ResNet synchronization is reduced from 13.5 to 7.4 m. The quantization gains of the two GPU inference modules collectively compress the end-to-end latency to 35.7 m, achieving a real-time processing frame rate of 28 FPS. In Figure 8b, the YOLOv8-Seg model volume compression ratio is 75.7%, and Wide ResNet is 73.9%. The combined mAP@0.5 loss for both is less than 1.3%, indicating that the accuracy trade-off is within an acceptable range for engineering applications. The balance between computing power and accuracy in edge deployment has been effectively validated.

5 Conclusion

To address attribution inaccuracies in construction sites in network-constrained environments characterized by three major constraints—high personnel density, uniform appearance of PPE, and limited network connectivity—a system framework that employs instance segmentation masks is developed to drive cross-frame tracking and violation attribution. This framework integrates pixel-level spatial correlation, sliding window cumulative detection, and the tripartite association of “mask–identity–violation” into a unified inference chain. Experimental results demonstrate that mask-level IoU-based attribute assignment significantly corrects attribution biases in frame-level methods within high-density scenes. Mask cropping features enhance both intra-ID cohesion and inter-ID dispersion in the cosine distance space, providing a feature-based explanation for improved tracking stability. Current datasets suffer from insufficient coverage of extreme lighting conditions, dust occlusion, and non-standard PPE configurations, whereas computational bottlenecks from multicamera concurrency limit horizontal scalability. Future work can incorporate conditional generative adversarial networks for data augmentation, explore model compression strategies combining knowledge distillation and channel pruning, and develop collaborative inference scheduling approaches to expand coverage across concurrent cameras.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Chengxi Zhang, Jiangnan University, China

Reviewed by: Tianle Yin, Jiangnan University, China

Liu Hui, Guangdong Polytechnic Normal University, China