Optimization method for distributed storage of power system video based on improved CRUSH algorithm and dynamic metadata scheduling

1 Introduction

With the accelerated construction of new power systems and the in-depth advancement of the “dual carbon” strategy, China’s smart grid market has entered a stage of rapid development. In this wave of digital transformation, video surveillance, as the core support for “visual operation and maintenance” of the power grid, has achieved full coverage in key scenarios such as substations, transmission lines, and distribution rooms (Ku et al., 2014). From real-time monitoring of substation equipment status to remote inspection of transmission lines in mountainous areas, video data has become an indispensable basis for equipment fault diagnosis, accident traceability, and safety supervision in the power industry (Chang et al., 2018). However, the explosive growth of video data has also brought unprecedented challenges to storage systems. A single substation can generate an average of 500 GB of data per day, and important sites require video storage for up to several years, resulting in an exponential increase in storage pressure. How to achieve efficient, reliable, and low-cost storage of massive power video data has become a key bottleneck restricting the intelligent development of the power grid (Sadeghi et al., 2019; Li et al., 2014).

With the rapid development of new power systems, distributed energy resources, digital substations, transmission-line monitoring terminals, unmanned inspection devices, and dispatching-center video systems are being deployed at larger scale (Ran et al., 2018). Power-system video is no longer used only for passive monitoring; it increasingly supports intelligent inspection, fault diagnosis, emergency response, operation safety, and visualized dispatching (Tan and Cao, 2020). Therefore, the storage system must satisfy not only large-capacity retention, but also low-latency access, differentiated reliability, regional data locality, and event-driven burst handling.

Distributed storage technology has gradually become the mainstream choice for power system video storage due to its advantages of high scalability, high availability, and low operating costs (Wang et al., 2018). Unlike traditional centralized storage, distributed storage disperses data across multiple nodes, achieving linear expansion of storage capacity and performance (Ren et al., 2019). In the power industry, where data volume grows rapidly and access requirements are diverse, distributed storage systems can effectively meet the needs of concurrent access to multi-scenario video data (Xu et al., 2020).

Recent studies further show that distributed storage and edge-video systems are moving toward device heterogeneity, recovery-cost optimization, dynamic replication, online video caching, and cloud-edge surveillance scheduling (Salehe et al., 2019). Ceph-oriented studies, such as ZNS-friendly Ceph storage design, heterogeneous coded-storage recovery, and cost-effective Ceph architecture, emphasize storage-media heterogeneity and metadata update overhead (Hung et al., 2018). Edge caching and surveillance-scheduling studies focus on latency-sensitive video access and request bursts. However, these studies do not fully address the combined requirements of power-system video, including geographic grid correlation, differentiated reliability of fault/event videos, tidal access patterns, event-driven traffic bursts, and low-latency metadata scheduling (Zhang et al., 2017; Jiang et al., 2018; Sun and Yin, 2016).

The first challenge is the imbalance between storage efficiency and data integrity. With the popularization of 4K/6 MP high-definition cameras, the data volume of power video has increased sharply, and the storage cycle requirements are strict—conventional substations require 90 days of continuous storage, and important places even need years of retention (Jiang et al., 2016). Traditional compression technologies often cause loss of key details such as line connector status and instrument readings while reducing storage capacity, which directly affects the accuracy of AI fault analysis and accident traceability (Huang and Johari, 2014). Although chip-level lossless compression technologies have emerged to improve storage efficiency, how to integrate them with distributed storage architectures to achieve global optimization of “compression-storage-access” still lacks in-depth research. The second challenge is the mismatch between data distribution strategies and power grid application scenarios (Spiteri et al., 2020). More than 60% of substations and transmission lines in China are located in remote areas such as mountains and deserts, where bandwidth resources are limited and unstable. The traditional data distribution algorithm cannot dynamically adjust the storage location according to the network status and data access frequency, leading to problems such as “unable to transmit back” of key video frames in remote areas and “hot spots” in data access (Lu et al., 2019). The third challenge is the inefficiency of metadata management under high concurrency. Metadata, as the “index” of distributed storage systems, directly affects the efficiency of data access (Harrabi et al., 2024). In the power system, scenarios such as simultaneous calls of multi-channel video and real-time retrieval of fault videos lead to high concurrency of metadata access (Govea et al., 2025). However, the existing static metadata scheduling mechanism cannot adapt to the dynamic changes of workloads, resulting in increased access latency and even system performance bottlenecks.

The CRUSH (Controlled Replication Under Scalable Hashing) algorithm, as a core component of distributed storage systems such as Ceph, has been widely used in power grid distributed storage due to its advantages in data distribution and fault recovery (Lai et al., 2024). The algorithm realizes the mapping of data objects to storage nodes through a scalable hash mechanism, which can minimize data migration when storage devices are added or removed, thereby reducing the impact of hot disks on cluster performance (Wang et al., 2025). However, in the face of the temporal and spatial characteristics of power video data, the original CRUSH algorithm has obvious limitations. On the one hand, it lacks consideration of node load status (Xu et al., 2025). When a large number of high-definition video data are concentratedly written to certain nodes, it is easy to cause load imbalance and reduce the overall throughput of the system (Yuan et al., 2025). On the other hand, it does not combine the access frequency of video data (Yin et al., 2025). For high-priority data such as fault videos that need to be accessed in real time, the algorithm cannot prioritize storage in high-performance nodes, resulting in increased access latency (Nezami et al., 2025; Yuzhu et al., 2026). Therefore, improving the CRUSH algorithm to adapt to the load characteristics and access requirements of power video data has become an important breakthrough to optimize distributed storage performance.

In response to the above problems, this study proposes an optimization method for distributed storage of power system video based on an improved CRUSH algorithm and dynamic metadata scheduling. The core ideas of this method include three aspects: First, improve the CRUSH algorithm by integrating node load perception and data priority weighting. The improved algorithm will monitor the real-time load status of storage nodes (including disk I/O, network bandwidth, and CPU utilization) and assign different weights to video data according to their importance. When mapping data objects to nodes, the algorithm comprehensively considers the hash value, node load, and data priority to achieve balanced distribution of storage resources and optimization of high-priority data access performance. Second, design a dynamic metadata scheduling mechanism based on power business characteristics. Combined with edge computing technology, metadata of frequently accessed video data is pre-stored in edge nodes in remote areas to reduce the dependence on long-distance network transmission. At the same time, a metadata cache update strategy based on LRU (Least Recently Used) is adopted to ensure the timeliness and hit rate of metadata access. Third, construct an integrated architecture of “improved CRUSH + dynamic metadata scheduling”. In this architecture, the improved CRUSH algorithm is responsible for the distribution and storage of video data blocks, while the dynamic metadata scheduling mechanism manages the index information of data blocks. The two modules interact in real time—metadata scheduling feeds back access hotspots to the CRUSH algorithm to guide data redistribution, and the CRUSH algorithm provides node status information to metadata scheduling to optimize prefetching strategies.

2 Model for distributed storage of power system video

2.1 Overall architecture

Power system video data (e.g., substation monitoring video, transmission line patrol video, power load monitoring video) has the characteristics of massive volume, multi-source heterogeneous, real-time transmission requirement, and high data reliability demand. Traditional centralized storage models face bottlenecks such as single point of failure, low storage expansion efficiency, and high bandwidth pressure for cross-regional video data access. A distributed storage model is constructed for power system video, which takes data reliability, storage resource utilization, and real-time access performance as core objectives, and adapts to the industrial characteristics of power systems (e.g., hierarchical management of power grids, regional differentiation of video data generation).

The distributed storage model for power system video is a hierarchical peer-to-peer (P2P) architecture divided into three layers: terminal perception layer, edge storage layer, and cloud storage layer, with the following functional positioning:

• Terminal perception layer: Composed of power system video acquisition terminals (camera, patrol robot), responsible for video data collection, preprocessing (compression, framing), and initial uploading to the nearest edge storage node;
• Edge storage layer: Deployed in substation, power supply bureau and other edge nodes, responsible for storing high-real-time video data (e.g., real-time monitoring video within 72 h) and providing local data access services;
• Cloud storage layer: Deployed in the provincial/state-level power dispatching center, responsible for storing historical video data (archived for more than 72 h) and global data scheduling, and realizing unified management of edge storage resources.

The model adopts a data sharding + multi-copy storage strategy: the original video file is divided into fixed-size data shards S = {s ₁, s ₂, …, s _*n*} (the shard size is set to 4MB/8 MB according to the video resolution, adapted to power system video compression standards such as H.265), and each shard generates k copies, which are stored in different nodes to avoid data loss caused by single node failure.

2.2 Mathematical model

Define the set of storage nodes in the model as N = {n ₁,n ₂,…,n _*m*}, where each node n _*i* has attribute parameters: storage capacity C _*i*, remaining storage space R _*i* (R _*i* ≤ C _*i*), node load L _*i* (the ratio of current data processing amount to the maximum processing amount, 0<L _*i* < 1), and node geographic attribute G _*i* (corresponding to the power grid partition, e.g., urban area, suburban area, rural area).

Define the set of power system video data shards as S = {s ₁,s ₂,…,s _*n*}, each shard s _*j* has attribute parameters: data volume V _*j*, real-time level τ _*j* (τ _*j*∈[0,1], the larger the value, the higher the real-time access requirement), reliability level ρ _*j* (ρ _*j*∈[0,1], the larger the value, the higher the data storage reliability requirement), and associated power grid area A _*j* (matching the node geographic attribute G _*i*).

For the storage of data shard s _*j*, the selected node n _*i* must satisfy the following hard constraints to ensure the basic operation of the power system video storage (Equation 1):where the remaining storage space of the node is not less than the data volume of the shard, to avoid storage overflow; L _max is the maximum load threshold of the node, and the node load does not exceed the threshold to prevent node downtime due to overload; the geographic attribute of the node is consistent with the associated power grid area of the shard, reducing the network delay of cross-regional video data transmission (adapting to the hierarchical management characteristics of the power grid).

R i ≥ V j L i ≤ L max G i = A j (1)

As shown in Figure 1, the reliability of data shard storage is measured by the survival probability of the data copy set. Assuming that the failure probability of a single storage node is p (the node failure rate of the power system is set to p ≤ 0.001 due to the high stability of industrial equipment), and the k copies of the shard s _*j* are stored in k mutually independent nodes, the survival probability P _*surv* (s _*j*) of the shard s _*j* is Equation 2:

P s u r v s j = 1 − p k (2)

Combined with the reliability level ρ _*j* of the data shard, the number of copies k must satisfy:

1 − p k ≥ ρ j (3)

By transforming Equation 3, the minimum number of copies required for the shard s _*j* is obtained Equation 4:

k ≥ log p 1 − ρ j (4)

Since the number of copies k is a positive integer, the actual minimum number of copies is k min = ⌈ ⁡ log p 1 − ρ j ⌉ (the ceiling function represents rounding up). For example, when ρ _*j* = 0.999 and p = 0.001, k _min = 2, that is, at least 2 copies need to be stored to meet the reliability requirement.

The access latency of power system video data is the key index to measure the real-time performance of the model, which is composed of network transmission latency and node response latency, and the total access latency T (s _*j*,n _*i*) of the shard s _*j* from the node n _*i* is Equations 5, 6:where,where T _*trans* (s _*j*,n _*i*) is the network transmission latency, B _*i*-*j* is the network bandwidth between the data access terminal and the node n _*i*, D _*i*-*j* is the fixed network delay (related to geographic distance); T _*resp* (n _*i*) is the node response latency, α is the latency coefficient (related to node hardware performance), and the node response latency is positively correlated with the node load.

T s j , n i = T t r a n s s j , n i + T r e s p n i (5)

T t r a n s s j , n i = V j B i − j + D i − j T r e s p n i = α · L i (6)

For high-real-time video data (e.g., substation real-time monitoring, τ _*j* > 0.8), the model requires that the total access latency T (s _*j*,n _*i*)≤T _max (the maximum allowable latency of the power system is set to 200m).

2.3 Model solving and evaluation index

The objective function of the model is to minimize the total storage resource consumption and minimize the average access latency under the constraints of node selection, copy number and real-time performance, which is a multi-objective optimization problem (Equation 7):where x _*ij* is the 0–1 decision variable, x _*ij* = 1 means the shard s _*j* is stored in the node n _*i*, and x _*ij* = 0 otherwise.

min ⁡ F 1 = ∑ j = 1 n ∑ i = 1 m x i j · V j min ⁡ F 2 = 1 n ∑ j = 1 n T s j , n i (7)

The model uses the weighted sum method to convert the multi-objective problem into a single-objective problem for solving, and the comprehensive objective function is Equation 8:where ω ₁ and ω ₂ are the weight coefficients (ω ₁ + ω ₂ = 1), which are adjusted according to the power system′s demand for storage resource utilization and real-time performance; F _1*max* and F _2*max* are the maximum values of the two objective functions for normalization.

min ⁡ F = ω 1 · F 1 F 1 ⁡ max + ω 2 · F 2 F 2 ⁡ max (8)

The core evaluation indexes of the model include: data storage reliability (survival probability), storage resource utilization η, average data access latency, and node load balance degree σ (Equation 9),where L ¯ is the average load of all nodes.

η = ∑ j = 1 n ∑ i = 1 m x i j · V j ∑ i = 1 m C i σ = 1 m ∑ i = 1 m L i − L ¯ 2 (9)

3 Optimization with improved CRUSH and dynamic metadata scheduling

3.1 Research motivation of the optimization method

The traditional CRUSH (Controlled Replication Under Scalable Hashing) algorithm is a classic hashing-based distributed storage scheduling algorithm, which has the advantages of high scalability and no single point of failure, but there are two key problems when applied to power system video distributed storage:

• The original CRUSH algorithm uses static node weight allocation, which cannot adapt to the dynamic change of remaining storage space and node load of power system storage nodes, leading to unbalanced node load and low storage resource utilization;
• The original algorithm does not consider the real-time and reliability differentiation of power system video data, and the randomness of data shard mapping leads to the failure to meet the access latency and reliability requirements of high-priority video data;
• The traditional metadata scheduling adopts a centralized mode, which has the problems of high scheduling delay and single point of failure, and cannot match the real-time access demand of power system video.

Based on the above problems, this chapter proposes an improved CRUSH algorithm combined with dynamic metadata scheduling strategy, which optimizes the node weight calculation, data shard mapping rules and metadata management mode of the original CRUSH algorithm, and adapts to the technical characteristics and industrial requirements of power system video distributed storage.

3.2 Improvement of CRUSH algorithm for power system video storage

The original CRUSH algorithm realizes the mapping from data objects to storage nodes through hash function and weighted random sampling: 1) Use the hash function hash (o,r) to generate a random sequence for the data object o and the replica number r; 2) Perform weighted random sampling on the storage node set according to the node weight w _*i*, and select the node corresponding to the sampling result as the storage node of the replica; 3) The node weight w _*i* is determined by the hardware performance of the node, and remains unchanged in the scheduling process.

The core mapping formula of the original CRUSH algorithm is Equation 10:where W = {w ₁,w ₂,…,w _*m*} is the static weight set of nodes, and CRUSH-MAP (·) is the mapping function of the algorithm.

n = CRUSH − MAP h a s h o , r , W (10)

3.2.1 Dynamic node weight calculation model

A dynamic weight calculation model considering the multi-attribute of power system storage nodes is proposed, and the node weight w _*i*(t) is a time-varying parameter, which is jointly determined by the remaining storage space utilization rate, node load and hardware performance baseline weight of the node, to realize the dynamic adjustment of node weight with the change of node state.

First, define the basic parameters for weight calculation:

• Hardware performance baseline weight w _*i*0: Determined by the node′s storage capacity, computing power and network bandwidth, the larger the hardware performance, the higher the baseline weight (normalized to [0,100]);
• Remaining storage space utilization rate η _*Ri* = R _*i* /C _*i*: The ratio of remaining storage space to total capacity, the larger the value, the more idle the node storage resource (normalized to [0,1]);
• Node load normalization factor η _*Li* = 1-L _*i* /L _max: The inverse of the node load relative to the maximum load threshold, the smaller the node load, the larger the factor (normalized to [0,1]).

The dynamic weight calculation formula of the node is Equation 11:where β ₁ and β ₂ are the weight coefficients of the remaining storage space and node load (β ₁ + β ₂ = 1), and for power system video storage, β ₁ = 0.6, β ₂ = 0.4 (prioritize the utilization of idle storage space, and then consider the node load).

w i t = w i 0 · β 1 · η R i + β 2 · η L i (11)

The storage-node status is monitored continuously, but the CRUSH placement weight is not updated for every request. Under normal conditions, the placement weight is updated every 5 s. When remaining capacity, node load, or available bandwidth changes sharply, an event-triggered update is activated. To avoid system oscillation and large-scale data migration caused by frequent weight changes, the proposed method introduces weight smoothing, dead-zone triggering, and migration-rate limitation.

Before giving the formula, we added two explanatory sentences in the manuscript. The purpose of dynamic weight adjustment is to reflect node heterogeneity and load variation, but the raw weight should not be directly injected into the placement rule. Therefore, a smoothed weight is used to avoid abrupt placement changes and excessive data rebalancing (Equation 12).where w i 0 is the baseline hardware weight, η R i n is the remaining-storage factor, η L i n is the load factor, β 1 + β 2 = 1, and α ∈ 0 , 1 is the smoothing coefficient. In the revised experiment, α = 0.2. To further avoid unnecessary rebalancing, the weight is applied only when (Equation 13),where ε w is the dead-zone threshold. Meanwhile, the amount of migrated data in one update interval is constrained by Equation 14,

w ∼ i n = w i 0 β 1 η R i n + β 2 η L i n w i n = 1 − α w i n − 1 + α w ∼ i n (12)

w i n − w i n − 1 w i n − 1 > ε w (13)

D mig n ≤ D max (14)

This prevents large-scale migration caused by small short-term load fluctuations. With this revision, the improved CRUSH algorithm becomes a quasi-dynamic placement method rather than an unstable real-time remapping method.

3.2.2 Data shard mapping with priority constraints

A priority-based data shard mapping rule is added to the improved CRUSH algorithm, which classifies the power system video data shards according to the real-time level τ _*j* and reliability level ρ _*j*, and sets different mapping constraints for different priority shards to ensure the access performance and storage reliability of high-priority data.

First, divide the data shards into three priority levels: 1) High priority (P1): τ _*j* > 0.8 and ρ _*j* > 0.99, e.g., substation fault monitoring video, transmission line emergency patrol video; 2) Medium priority (P2): 0.5≤τ _*j* ≤ 0.8 or 0.9≤ρ _*j* ≤ 0.99, e.g., normal monitoring video of urban substation; 3) Low priority (P3): τ _*j* < 0.5 and ρ _*j* < 0.9, e.g., historical patrol video of rural transmission lines (archived data).

For the mapping of shards with different priorities, the improved CRUSH algorithm adds the following constraints on the basis of Equation 8:

• P1 shard: Only map to nodes with L _*i* < 0.5 (low load) and B _*i*-*j*>100 Mbps (high bandwidth), and the number of copies k = k _min+1 (increase one copy to improve reliability); 2) P2 shard: Map to nodes with L _*i* < 0.8 and B _*i*-*j*>50 Mbps, and the number of copies k = k _min; 3) P3 shard: No additional bandwidth constraints, map to nodes with L _*i* < L _max, and the number of copies k = max (1,k _*min*-1) (reduce one copy to save storage resources, the minimum is 1).

The core mapping formula of the improved CRUSH algorithm is Equation 15:where W(t) = {w ₁(t), w ₂(t), …, w _*m*(t)} is the dynamic node weight set at time t; P _*j* is the priority level of the shard s _*j*; Θ = L t h , B t h , k Δ is the constraint parameter set of the power system, L _*th* is the node load threshold of different priorities, B _*th* is the bandwidth threshold, k _*Δ* is the copy number adjustment value.

n = Improve − CRUSH − MAP h a s h s j , r , W t , P j , Θ (15)

3.2.3 Hash function optimization for power grid partition

A geographic hash function adapted to the hierarchical management of the power grid is designed to replace the original random hash function, which ensures that the data shards of the same power grid partition are mapped to the storage nodes of the corresponding partition, reducing the cross-regional network delay.

The geographic hash function for power system video data shards is Equation 16:Where A _*j* is the associated power grid area code of the shard s _*j*; ID (s _*j*) is the unique identification code of the data shard; r is the replica number; H (·) is the SHA-256 hash function (ensures the randomness of the hash result within the partition); M is the total number of hash buckets; ∥ represents the string concatenation operation.

h a s h s j , r = H A j ∥ I D s j ∥ r mod M (16)

In this study, the geographical location is not limited to physical longitude and latitude. It is represented by a power-grid area code A _*j* generated according to the hierarchical management structure and topology relationship of the power system. For substation video, A _*j* is mainly determined by the physical substation location. For transmission-line inspection video, A _*j* is determined by the line section and the nearest operation-maintenance region. For dispatching-center video, A _*j* is determined by the corresponding control domain. Thus, the proposed geographic hash is a grid-topology-aware hash rather than a simple latitude-longitude hash.

As shown in Figure 2, this hash function first uses the power grid area code to lock the mapping range of the data shard in the corresponding regional node set, and then uses the shard ID and replica number to generate a random hash result, which combines the geographic relevance of power system video data and the randomness of distributed storage mapping.

Geographic constraints improve locality, but they may also reduce distribution uniformity if one partition generates much more video than others. To reduce data skew, virtual buckets are introduced inside each power-grid partition. The mapping is performed in two stages: the first stage selects the power-grid partition, and the second stage distributes shards uniformly among virtual buckets and storage nodes inside that partition (Equation 17):where M _*Aj* is the number of virtual buckets assigned to partition A _*j*, W _*Aj*(t) is the dynamic weight set of candidate nodes in the partition, and C _*j* is the priority constraint set. To reduce skew, M _*Aj* can be adjusted according to the observed video generation rate (Equation 18):where λ _*Aj* is the video-arrival rate of partition A _*j*. This design preserves locality while maintaining distribution uniformity.

b j , r = H A j I D s j r mod M A j n o d e j , r = C R U S H _ M A P b j , r , W A j t , C j (17)

M A j = ⌈ M · λ A j ∑ g λ g ⌉ (18)

3.3 Dynamic metadata scheduling strategy

Metadata is the core information of power system video distributed storage, including data shard attribute (V _*j*, τ _*j*, ρ _*j*), storage node address (IP, port), copy distribution, data access record, etc. The dynamic metadata scheduling strategy adopts a hierarchical distributed metadata management mode (edge + cloud), which is matched with the improved CRUSH algorithm to realize low-latency and highly reliable metadata scheduling.

Divide the metadata into edge metadata and cloud metadata according to the data real-time level, and store them in the edge storage layer and cloud storage layer respectively:

• Edge metadata: Store the metadata of high/medium priority video data (P1, P2) with high real-time access requirement, and deploy a metadata node in each power grid partition (substation/power supply bureau), responsible for the metadata management of the local partition;
• Cloud metadata: Store the metadata of low priority video data (P3) and the global metadata index (recording the storage location of edge metadata), deploy multiple metadata nodes in the cloud storage layer, and realize metadata backup through multi-copy storage.

The metadata synchronization between edge and cloud adopts the incremental synchronization mode, and only the changed metadata (e.g., new data shard storage, node state update) is synchronized in real time, while the unchanged metadata is synchronized periodically (update cycle is 1 h), which reduces the metadata synchronization bandwidth overhead.

The core of the dynamic metadata scheduling algorithm is metadata node selection and metadata cache update, which aims to minimize the metadata access latency and improve the hit rate of metadata cache.

3.3.1 Metadata node selection model

Define the metadata access terminal set as U = {u ₁,u ₂,…,u _*k*} (e.g., power dispatching terminal, patrol terminal), and the metadata node set as M = {m ₁,m ₂,…,m _*l*} (including edge metadata node M _*e* and cloud metadata node M _*c*). For the access request of the terminal u _*p* for the metadata of the shard s _*j*, the selected metadata node m _*q* must satisfy the matching principle of data priority and metadata node level: 1) If s _*j*∈P1/P2, select the edge metadata node m _*q*∈M _*e* of the same power grid partition as s _*j*; 2) If s _*j*∈P3, select the cloud metadata node m _*q*∈M _*c* with the smallest network delay to u _*p* . The metadata access latency T _*m* (u _*p*, m _*q*) is calculated as Equation 19:where D _*p*-*q* is the network delay between the terminal u _*p* and the metadata node m _*q*, Q _*q* is the number of pending requests of the metadata node m _*q*, and γ is the request processing latency coefficient (set to 5m/request for power system metadata nodes).

T m u p , m q = D p − q + γ · Q q (19)

The objective function of metadata node selection is to minimize the access latency (Equation 20):where M _*match* is the set of metadata nodes matching the data priority.

min ⁡ T m = min m q ∈ M m a t c h D p − q + γ · Q q (20)

3.3.2 Dynamic metadata cache update strategy

A LRU-K (Least Recently Used-K) cache update strategy adapted to power system video data is proposed to replace the traditional LRU strategy, which improves the metadata cache hit rate by considering the access frequency and access time of the metadata. Define the access record of the metadata md _*j* as T (md _*j*) = {t ₁, t ₂, …, t _*k*} (the time of the last k accesses, k = 3 for power system), the access frequency f (md _*j*) = k/Δt (Δt is the time interval from the first access to the latest access), and the cache priority P _*m* (md _*j*) is:

Power-system video metadata access has both tidal and event-driven characteristics. Normal inspection video produces periodic access, whereas fault events produce sudden bursts. Therefore, the cache score includes recent access, repeated access, and event priority (Equation 21):where t _*j*,*K* is the Kth most recent access time, I _*j* ^*event* is an event-priority indicator, and δ avoids division by zero. The last term is added to improve response under fault-triggered traffic bursts.

P m m d j = λ 1 1 T now − t j , K + δ + λ 2 K t j , 1 − t j , K + δ + λ 3 I j e v e n t (21)

The parameter K determines how many historical accesses are used to judge whether metadata are truly hot. When K = 1, LRU-K degenerates into conventional LRU and is sensitive to short-term bursts; metadata accessed only once during a temporary event may be cached. When K is too large, such as K = 4 or K = 5, the strategy can identify stable hot metadata more accurately but responds more slowly to sudden fault-video access. Therefore, K = 3 is used as a compromise between burst responsiveness and access-frequency stability (Equation 22).where T _*now* is the current time, λ ₁ and λ ₂ are the weight coefficients (λ ₁ = 0.7, λ ₂ = 0.3, prioritize the recent access metadata), and the larger the P _*m* (md _*j*), the higher the cache priority.

P m m d j = λ 1 · 1 max T n o w − t k , 1 + λ 2 · f m d j (22)

When the metadata cache space is full, the metadata with the smallest cache priority is eliminated first, and the newly accessed metadata is added to the cache. For the high-priority data metadata (P1), the cache exclusive mechanism is adopted (not eliminated by the cache update strategy) to ensure the zero latency of high-priority metadata access.

3.3.3 Metadata consistency guarantee mechanism

A multi-version timestamp-based metadata consistency guarantee mechanism is proposed to solve the problem of metadata inconsistency caused by node failure and data copy update in the distributed storage process, which is adapted to the high reliability requirement of power system data.

• Add a version number V _*md*(t) and timestamp T _*md*(t) to each metadata record, which are updated in real time with the change of metadata (e.g., data copy addition/deletion, node state update);
• When the metadata node fails, the backup node uses the latest version number to synchronize the metadata, and discards the metadata with the old version number to ensure the consistency of the metadata content;
• For the data copy update operation, the metadata is updated first and then the data copy is modified (write ahead log, WAL), and the log record is added to ensure that the metadata can be recovered in case of update failure.

The metadata consistency verification formula is Equation 23:where m _*q* and m _*q*’ are two metadata nodes storing the same metadata, ΔT is the maximum allowable time difference (set to 1s for power system), and C _*on*(md _*j*) = 1 means the metadata is consistent.

C o n m d j = 1 , V m d m q = V m d m q ′ ∩ T m d m q − T m d m q ′ < Δ T 0 , otherwise (23)

In practical deployment, the system can be organized as an edge-cloud collaborative architecture. Edge storage nodes are deployed near substations, power supply bureaus, inspection centers, or regional operation centers, while cloud storage nodes are deployed in provincial or centralized data centers. Real-time or event-related video streams, such as substation fault monitoring, transmission-line emergency inspection, and dispatching-center abnormal-event video, are classified as high-priority data. Their metadata are cached in edge metadata nodes, and their replicas are preferentially placed in low-load, high-bandwidth nodes within the corresponding power-grid partition. Historical inspection video and low-frequency archived video are stored in cloud or low-priority storage pools.

The improved CRUSH module can be implemented as a placement-decision layer before object writing. It periodically receives node status, including storage utilization, CPU load, network bandwidth, fault-domain status, and partition code. The dynamic metadata scheduler can be deployed as an independent service connected to edge metadata nodes and cloud metadata nodes. In realistic systems, node weights can be updated periodically and triggered by abnormal load changes, while data migration is constrained by a migration-rate threshold to avoid frequent rebalancing.

4 Case analysis

4.1 Test environment

The hardware environment adopts an Intel Core i7-12700H processor (2.7 GHz main frequency, 14 cores and 20 threads), 32 GB DDR5 4,800 MHz memory, a 2 TB NVMe solid-state drive, and a 10 Gbps Ethernet network environment (latency ≤1m). The software platform builds a simulation system based on MATLAB R2023a, implements distributed storage node scheduling logic with Python 3.9, processes power system video data (H.265 format, 1920 × 1080 resolution, 25fps frame rate) using FFmpeg tools, tests network transmission performance through iPerf3, and monitors storage node load status with Prometheus.

Three types of typical power system video data are selected as the test set, covering different application scenarios and data characteristics:

Scenario 1: Substation equipment inspection video (single frame size 2.3 MB, duration 30 min, total capacity 10.35 GB, including static scenarios such as equipment start-stop and instrument reading); Scenario 2: Transmission line fault monitoring video (single frame size 3.1 MB, duration 20 min, total capacity 9.3 GB, including dynamic scenarios such as conductor galloping and lightning discharge); Scenario 3: Power dispatching center monitoring video (single frame size 1.8 MB, duration 60 min, total capacity 16.2 GB, including composite scenarios such as multi-screen segmentation and data overlay).

The core parameter configurations of the improved CRUSH algorithm (proposed method) and comparison algorithms (traditional CRUSH algorithm (method 1), consistent hashing algorithm (method 2)) are kept consistent to ensure fairness in comparison: 1) Storage cluster configuration: 15 storage nodes (divided into 3 fault domains, 5 nodes per fault domain), node storage capacity 8TB/unit, maximum read/write rate of a single node 500 MB/s; 2) Algorithm parameters: The initial value of the dynamic weight adjustment coefficient of the improved CRUSH algorithm is 0.85, the attenuation factor is 0.003, the number of replicas is 3, and the data shard size is 64MB; the traditional CRUSH algorithm uses a fixed weight (value 0.7), and the consistent hashing algorithm has 100 virtual nodes per physical node; 3) Simulation parameters: Simulation duration 120 min, data writing rate 100 MB/s, metadata query request frequency 50 times/s, node failure simulation probability 0.02 times/min (fault duration 3min).

To strengthen the comparison, two additional baselines are included. The first is a load-aware CRUSH variant, which adjusts node weights according to storage utilization and node load but does not consider power-grid geographical partitions, differentiated video priority, or metadata scheduling. The second is an edge-cache metadata scheduling method, which deploys edge metadata caches to improve metadata access but does not optimize data-shard placement through improved CRUSH mapping. The revised comparison therefore includes five methods: consistent hashing, traditional CRUSH, load-aware CRUSH, edge-cache metadata scheduling, and the proposed improved CRUSH with dynamic metadata scheduling.

Two types of core evaluation indicators are defined to comprehensively assess the optimization effect of distributed storage.

• Storage performance indicators: a) Data read-write latency (T): Total time from request initiation to completion of data transmission (unit: ms); b) Storage load balance degree (σ): Standard deviation of storage utilization of each node (smaller σ indicates better balance); c) Metadata query response time (t_m): Time consumed by the metadata server to process query requests (unit: ms);
• Reliability and efficiency indicators: a) Data recovery success rate (R): Proportion of complete data recovery after node failure (unit: %); b) Network bandwidth occupancy rate (B): Percentage of network bandwidth occupied during data transmission (unit: %); c) Storage system throughput (S): Total data read-write capacity completed per unit time (unit: MB/s).

4.2 Results analysis

Figure 3 shows the comparison of average read-write latency of the three algorithms in different video scenarios. It can be seen that the improved CRUSH algorithm performs optimally in all scenarios: the read-write latency in Scenario 1 is 28.3 ms, which is 32.1% lower than that of the traditional CRUSH algorithm (41.7 ms) and 46.8% lower than that of the consistent hashing algorithm (53.2 ms); the read-write latency in Scenario 2 is 35.6 ms, which is 32.1% lower than that of the traditional CRUSH algorithm (52.4 ms) and 47.6% lower than that of the consistent hashing algorithm (67.9 ms); the read-write latency in Scenario 3 is 24.7 ms, which is 33.1% lower than that of the traditional CRUSH algorithm (36.9 ms) and 49.1% lower than that of the consistent hashing algorithm (48.5 ms). This is due to the dynamic weight adjustment strategy and metadata pre-scheduling mechanism of the improved CRUSH algorithm, which effectively reduce the node selection overhead and data transmission path length.

Figure 4 shows the storage utilization distribution of each storage node after the three algorithms run for 60min. The standard deviation of node utilization of the traditional CRUSH algorithm is 12.8%, that of the consistent hashing algorithm is 18.3%, while that of the improved CRUSH algorithm is only 5.7%, showing a significant improvement in load balance. By real-time monitoring the storage status and computing resource occupancy of nodes, the improved algorithm dynamically adjusts the distribution weight of data shards, avoiding overload or idleness of a single node and ensuring uniform utilization of cluster resources.

To test the metadata query response time of three algorithms under varying load intensities (0.3–1.0 pu, where 1.0 pu represents full load). Figure 5 shows the average response time and fluctuation range (standard deviation) of each algorithm. It can be observed that the improved CRUSH algorithm maintains stable response time (12.3–15.8 ms) with small fluctuations (σ ≤ 0.8 ms) across all load levels. In contrast, the traditional CRUSH and consistent hashing algorithms show significant performance degradation under high loads (0.8–1.0 pu), with response times exceeding 27 ms and 32 ms respectively, and fluctuations increasing to 2.1 ms and 2.8 ms. This verifies that the dynamic metadata scheduling mechanism of the improved CRUSH algorithm effectively resists load shocks, ensuring reliable metadata service performance in complex power system video storage scenarios.

Table 1 lists the comparison results of comprehensive performance indicators of the three algorithms. On the premise that the data recovery success rate (99.8%) is the same as that of the traditional algorithm, the improved CRUSH algorithm reduces the network bandwidth occupancy rate to 28.3% and increases the storage system throughput to 426.5 MB/s, which are optimized by 23.5% and 31.2% respectively compared with the traditional CRUSH algorithm, fully verifying its collaborative advantages in reliability and efficiency.

TABLE 1. Comparison with additional improved baselines.

Compared with the two added baselines, the proposed method achieves lower read-write latency, smaller storage-utilization standard deviation, and more stable metadata response time. The load-aware CRUSH baseline improves node balance but does not distinguish power-grid partitions or fault-video priority. The edge-cache metadata scheduling baseline reduces metadata access latency but cannot correct uneven data-shard placement. The proposed method integrates remaining capacity, node load, hardware baseline weight, power-grid partition, video priority, and cache scheduling, so its improvement is more stable under mixed workloads.

Table 2 is used to analyze the influence of the LRU-K parameter on metadata-cache behavior. In power-system video storage, metadata access is affected by both regular inspection tasks and sudden fault-triggered requests, so the value of K directly changes the balance between short-term responsiveness and stable hotspot recognition. A small K reacts quickly to recent accesses but may cache temporary metadata generated by one-time burst events. A larger K can better identify repeatedly accessed metadata, but it may delay the response to newly emerging fault-video requests. For this reason, Table 2 compares K values from 1 to 4 using cache hit rate, metadata response time, latency fluctuation, and qualitative observation. The table supports the selection of K = 3 by showing that it achieves the best compromise between hit-rate improvement and response stability in the simulated power-system metadata workload.

TABLE 2. Sensitivity analysis of the LRU-K parameter.

The sensitivity results show that K = 3 achieves the best balance between cache hit rate and stability. K = 1 responds quickly but is easily affected by temporary event requests, while K = 4 improves long-term hot-metadata identification at the cost of slower response to sudden fault traffic.

Table 3 reports the stability of the proposed method under extreme event-driven traffic. This test is important because power-system video workloads are not always smooth: during faults, storms, emergency inspections, or dispatching operations, multiple terminals may upload high-priority video streams at the same time. The table therefore increases the write request rate to 1.5, 2.0, and 3.0 times the normal load and observes read-write latency, metadata response time, storage-utilization fluctuation, data recovery success rate, and migration bandwidth. These indicators show whether the dynamic weight adjustment and metadata scheduler remain stable when the system faces a short but intense burst. By referring to Table 3, the experiment demonstrates that the proposed method can maintain controllable latency, balanced storage utilization, and high recovery reliability while limiting migration overhead during simulated fault events.

TABLE 3. Extreme event-driven traffic test of the proposed method.

The extreme-condition test includes tidal traffic during inspection periods and event-driven burst traffic after a simulated fault event. During the burst, multiple substations and line-monitoring terminals upload high-priority video simultaneously for 10 min, and the write request rate is increased to 1.5, 2.0, and 3.0 times the normal load. The proposed method maintains stable performance because high-priority event video is mapped to low-load and high-bandwidth nodes in the same or nearby power-grid partition, while metadata are served preferentially by edge metadata nodes. The migration-rate constraint prevents dynamic weight adjustment from causing excessive rebalancing during the burst period.

The computational overhead of dynamic metadata scheduling mainly comes from periodic node-state collection, LRU-K access-record maintenance, and incremental synchronization between edge and cloud metadata nodes. The objective of overhead analysis is not to assume zero cost, but to verify that the additional cost is smaller than the latency reduction and reliability improvement obtained from scheduling optimization.

Table 4 evaluates the computational overhead introduced by dynamic metadata scheduling under different request intensities. Although edge-cloud metadata cooperation and LRU-K cache updating reduce query latency, they also require node-state collection, access-record maintenance, cache-score calculation, and incremental metadata synchronization. Therefore, it is necessary to verify whether the additional CPU and memory cost remains acceptable as the request intensity increases from 0.3 pu to full load. Table 4 presents CPU utilization, memory usage, and metadata response time in the same experimental environment, making it possible to judge the relationship between overhead and performance improvement. The results indicate that CPU consumption grows gradually and memory usage increases approximately linearly with cache size, while metadata response time remains within a stable range. Thus, Table 4 supports the conclusion that the metadata optimization is practical rather than computationally excessive.

TABLE 4. Metadata-server overhead under different request intensities.

Under full-load metadata access, the CPU utilization of the metadata server remains within a controllable range, and memory consumption increases approximately linearly with cache size. This indicates that the proposed method improves access latency without causing excessive metadata-server overhead.

Table 5 provides a final comprehensive comparison of the three main algorithms under the unified evaluation framework. Unlike the preceding tables, which focus on a single mechanism or stress scenario, this table summarizes the overall storage performance, reliability, network cost, and throughput of Method 1, Method 2, and the proposed method. The selected indicators include read-write latency, load balance degree, metadata query response time, data recovery success rate, network bandwidth occupancy rate, and storage throughput. These metrics cover both user-facing access efficiency and backend resource utilization, which are critical for power-system video platforms that must support long-term storage and rapid retrieval. By referring to Table 5, the paper shows that the proposed method reduces latency and bandwidth occupancy while improving load balance and throughput, without sacrificing recovery reliability compared with the traditional baseline.

TABLE 5. Comparison of comprehensive performance indicators.

To test robustness under variable fault intensities (1–4 failed nodes, corresponding to 6.7%–26.7% cluster fault rate). Figure 6 shows data recovery time, fluctuation range, and degradation trend. Results indicate that the improved CRUSH algorithm’s recovery time increases linearly (42.8s–98.5s) with stable fluctuations (σ ≤ 3.2m), while traditional CRUSH and consistent hashing show exponential growth (exceeding 180s and 220s at 4 failed nodes) and severe fluctuations (σ ≥ 7.5ms). This proves the improved algorithm’s strong fault tolerance, attributed to dynamic fault domain sensing and pre-allocated redundancy, ensuring reliable operation under high fault intensities.

5 Conclusion

This study addresses the technical bottlenecks of power-system video distributed storage and proposes an integrated optimization method based on the improved CRUSH algorithm and dynamic metadata scheduling. The improved CRUSH algorithm uses smoothed dynamic weight adjustment, priority-constrained shard mapping, and grid-topology-aware geographic hashing to mitigate load imbalance and improve the storage reliability of high-priority event video. The dynamic metadata scheduling strategy combines edge-cloud hierarchical storage, LRU-K cache updating, and incremental metadata synchronization to reduce access latency and enhance system stability. Comparative and extreme-condition experiments verify that the method outperforms traditional CRUSH, consistent hashing, load-aware CRUSH, and edge-cache metadata scheduling in read-write latency, load balance, metadata response, and throughput. The method is suitable for edge-cloud collaborative deployment in power-grid video platforms and can support intelligent inspection, emergency response, and visualized dispatching in new power systems. Future research can integrate pre-trained models to further improve adaptation to complex power scenarios.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Tuan-Vinh Le, Fu Jen Catholic University, Taiwan

Reviewed by: Lexi Xu, China United Network Communications Group, China

Guangxing Wang, Jiujiang University, China