Energy-aware bitrate ladder for VVC streaming with client-side super-resolution

1 Introduction

Video streaming has become one of the largest contributors to global Internet traffic and energy consumption, accounting for a substantial share of data center load, network transport, and end-user device power usage (Bianco et al., 2016). At scale, even small inefficiencies in video delivery translate into significant economic cost and environmental impact. Reducing the energy footprint of video streaming is therefore no longer only a device-level concern, but a system-level challenge that spans encoding infrastructures, content delivery networks, and heterogeneous client devices.

HTTP Adaptive Streaming (HAS) is the dominant mechanism for large-scale video delivery, enabling continuous adaptation of video quality to varying network conditions and device capabilities (Bentaleb et al., 2019; Seufert et al., 2014). In practice, HAS relies on bitrate ladders that define a discrete set of resolution–bitrate representations. Most deployed ladders follow fixed or guideline-based designs (e.g., Apple HLS recommendations), which prioritize interoperability and robustness but largely ignore content diversity, device heterogeneity, and energy efficiency (Apple Inc, 2024; Sodagar, 2011; Zwantschko, 2020). As streaming demand continues to grow, such static designs increasingly limit both efficiency and sustainability (Kalan and Dulger, 2024).

Two converging trends further amplify this challenge. First, modern video codecs such as Versatile Video Coding (VVC) achieve substantial bitrate reductions compared to previous standards, enabling higher visual quality at lower transmission rates (Bross et al., 2021). However, these gains come at the cost of significantly increased computational complexity, particularly during encoding, and non-negligible decoding energy at high resolutions (Uhrina et al., 2024; Siqueira et al., 2020; Pakdaman et al., 2020). This raises concerns for both large-scale encoding infrastructures and energy-constrained client devices. Second, contemporary client hardware increasingly integrates Graphics Processing Units (GPUs), Neural Processing Units (NPUs), and specialized accelerators capable of executing deep-learning workloads efficiently (Silvano et al., 2025; Alam et al., 2024). This has enabled client-side enhancement techniques, most notably Video Super-Resolution (VSR), which can reconstruct high-resolution visual quality from lower-resolution decoded content.

The coexistence of complex next-generation codecs and client-side video enhancement techniques fundamentally challenges traditional bitrate ladder design. When VSR is available at the client, the decoded resolution is no longer determines the displayed resolution, and lower-resolution streams may provide comparable perceptual quality after VSR. At the same time, such enhancement introduces additional computation and energy consumption on the client. Consequently, bitrate ladder design must move beyond codec-centric rate–distortion optimization and explicitly consider system-level energy trade-offs across encoding, delivery, decoding, and enhancement.

Despite growing interest in content-adaptive streaming, existing approaches typically optimize bitrate ladders for perceptual quality or bitrate efficiency alone. Energy consumption is often considered only implicitly, approximated through bitrate or complexity proxies, or limited to isolated pipeline components. In particular, the interaction between VVC encoding complexity, client-side VSR, and device-specific energy consumption remains insufficiently understood. This gap hinders principled decisions about when downscaled-plus-VSR pipelines are beneficial and when they may be counterproductive from an energy perspective. In this work, we focus on offline bitrate-ladder construction and content provisioning, and do not evaluate end-to-end streaming behavior such as adaptive bitrate (ABR) logic, rebuffering events, or network-induced dynamics. This scope allows us to isolate and analyze the fundamental rate–quality–energy trade-offs introduced by client-side super-resolution under controlled and reproducible conditions.

In this work, we propose Energy-Empowered Streaming (E2-Stream), a VSR-aware streaming framework that integrates client-side energy consumption directly into bitrate ladder construction. E2-Stream is designed as a measurement-driven offline optimization framework for representation selection, rather than a runtime streaming system. Unlike heuristic or prediction-based methods, E2-Stream exhaustively evaluates all combinations of encoding resolution, bitrate, and client-side VSR model on real hardware. To capture both spatial-only and temporally-aware enhancement behaviour, E2-Stream evaluates lightweight single-image SR (SISR) models (FSRCNN, ESPCN) alongside a proper temporal VSR model (EDVR). This enables direct comparison between frame-wise enhancement deployed on low-power devices and temporally consistent VSR suitable for GPU-enabled clients. We further evaluate E2-Stream across two heterogeneous playback devices: (i) a CPU-only Linux laptop performing software-based VVdeC decoding with CPU-bound VSR, and (ii) a GPU-equipped Windows laptop offering hardware-accelerated VVC decoding and GPU-accelerated VSR inference. This dual-device evaluation highlights how hardware acceleration fundamentally reshapes the rate–quality–energy trade-off. While FSRCNN and ESPCN operate frame-by-frame without temporal alignment, the included temporal VSR model (EDVR) leverages multi-frame propagation to improve temporal consistency. This distinction is critical for understanding when SISR is sufficient for energy-constrained clients and when temporal VSR provides measurable perceptual benefits on GPU-capable hardware.

By jointly measuring perceptual quality and energy required for decoding and VSR inference, E2-Stream constructs bitrate ladders that explicitly balance visual quality against client-side energy cost, while remaining compatible with adaptive streaming workflows, rather than directly optimizing or simulating ABR behavior.

The main contributions of this paper are:

• A measurement-driven streaming framework that integrates VVC encoding, both SISR and temporal VSR models, and device-specific energy profiling;
• An energy-aware bitrate-ladder construction method that jointly optimizes perceptual quality and device-side decoding and enhancement energy across heterogeneous clients; and
• A cross-device UHD evaluation demonstrating when downscaled-plus-VSR pipelines are beneficial on GPU-accelerated clients and when temporal or spatial SR becomes inefficient on CPU-only devices, without claiming end-to-end streaming or QoE optimization under dynamic network conditions.

Overall, E2-Stream shifts bitrate ladder design from a purely codec-centric perspective toward a device-aware, energy- and enhancement-aware paradigm that reflects the realities of modern streaming systems. Accordingly, the experimental evaluation concentrates on offline encoding, client-side decoding and enhancement, and measured energy and latency characteristics, leaving network adaptation and ABR policy evaluation to future work. The remainder of this paper is organized as follows: Section 2 reviews related work; Section 3 presents the proposed framework; Section 4 describes the experimental setup; Section 5 discusses the results; and Section 6 concludes the paper.

2 Related work

Video streaming research has evolved along four tightly coupled axes: (i) bitrate ladder design and HAS adaptation, (ii) learned enhancement, notably VSR, executed on clients, (iii) modern codec evolution with increased computational complexity (e.g., VVC), and (iv) energy-aware processing across server, network, and device as shown in Figure 1. Earlier work predominantly optimized rate–distortion (RD) behavior and Quality of Experience (QoE), but rising traffic volumes, hardware heterogeneity, and sustainability targets now require decisions that balance quality with latency, throughput, and energy across the full pipeline. E2-Stream positions itself at the intersection of these axes. In this paper, we consider this intersection from the perspective of offline representation and bitrate-ladder construction, rather than runtime adaptive bitrate (ABR) control or network-level adaptation.

2.1 Bitrate ladder and HAS adaptation

HAS enables per-segment bitrate switching under variable network conditions and device capabilities (Bentaleb et al., 2019; Sodagar, 2011). Early industry ladders, e.g., Apple HLS, followed static resolution–bitrate pairs (Apple Inc., 2024), ensuring interoperability but ignoring content variations in spatial detail or motion. This mismatch results in wasted bitrate for “easy” sequences and insufficient quality for “hard” ones, degrading QoE (Seufert et al., 2014; De Cock et al., 2016).

To overcome this rigidity, content-adaptive ladder design emerged. Classical approaches rely on spatial–temporal complexity features to select optimal operating points (Šuljug and Rimac-Drlje, 2024; De Cock et al., 2016). Learning-based approaches—regressors, random forests, or shallow neural networks—predict RD behavior and generate per-title or per-shot ladders that better approximate the theoretical convex hull (Katsenou et al., 2021; Menon et al., 2023b; Telili et al., 2022). These studies consistently demonstrate quality improvements and rate savings relative to static ladders (De Cock et al., 2016; Katsenou et al., 2021).

Ladder design also needs to incorporate latency constraints in live and low-delay scenarios, where encoding time per segment is limited (Bentaleb et al., 2019). Latency-aware ladder generation restricts presets, resolutions, or both, enabling stable real-time encoding (Menon et al., 2023b; Telili et al., 2022). However, most prior work assumes that decoded output is consumed directly, overlooking the growing use of client-side enhancement such as VSR. Since VSR can significantly alter perceptual quality at display resolution, ignoring its effect yields ladders suboptimal for modern playback systems (Amirpour et al., 2021; Katsenou et al., 2021). This motivates enhancement-aware and energy-aware ladder design, central to E2-Stream. In contrast to ABR-centric and network-adaptive studies, our work focuses on pre-streaming decisions at the content preparation stage, where representations are selected based on measured rate–quality–energy characteristics before deployment.

2.2 VSR techniques and neural upscaling

Deep learning has reshaped super-resolution for both images and videos, with large gains over classical interpolators (e.g., bicubic). Lightweight CNNs such as Fast Super-Resolution Convolutional Neural Network (FSRCNN) and Efficient Sub-Pixel Convolutional Neural Network (ESPCN) achieve real-time upscaling through efficient sub-pixel convolution and compact backbones (Dong et al., 2016; Shi et al., 2016). More advanced residual or attention-based architectures—Enhanced Deep Residual Networks for Single Image Super-Resolution (EDSR), Deep Residual Channel Attention Network (RCAN), Convolutional Anchored Regression Network (CARN), restore high-frequency detail more effectively but at increased computational cost (Lim et al., 2017; Lin et al., 2022; Ledig et al., 2017). Residual learning (He et al., 2016) has been key to enabling deeper, higher-performing VSR networks.

Modern VSR leverages temporal redundancy via propagation, alignment, or recurrent mechanisms. Prominent examples include Recurrent Back-Projection Network (RBPN) (Haris et al., 2019), BasicVSR++ (Chan et al., 2022), and efficiency-focused models like Efficient Video Super-Resolution with Neural Architecture Search (EVSRNet) (Liu et al., 2021). Surveys such as (Liu et al., 2022) highlight two major trends: increasing model expressiveness and improving hardware-efficient VSR inference. Recent directions include diffusion-based streaming VSR (Zhuang et al., 2025) and lookup table-based ultra-low-power VSR (Yin et al., 2024).

Unlike prior streaming-focused studies that evaluate only SISR models, the present work includes both lightweight SISR (FSRCNN, ESPCN) and a temporally-aware VSR model (BasicVSR-lite). This enables controlled comparison between frame-wise enhancement suited for CPU-only devices and temporal propagation approaches feasible on GPU-accelerated clients. By incorporating a temporal VSR model into the streaming pipeline, we explicitly capture temporal stability improvements—and their associated energy and latency costs—which have been largely overlooked in prior bitrate ladder design research.

For streaming, VSR breaks the traditional assumption that decoded resolution fully determines perceptual quality: a downscaled encode enhanced with VSR may rival or outperform native high-resolution encodes at comparable bitrates (Amirpour et al., 2021). Figure 2 illustrates this effect using rate–VMAF curves. In both examples, applying a lightweight model such as FSRCNN substantially lifts the quality of 540p and 1080p encodes, narrowing the gap to native 2160p, especially at low and mid bitrates. These cross-overs shift the optimal operating points, enabling bitrate ladders that favor lower base resolutions while still achieving high display-resolution quality, provided VSR inference remains real-time.

Prior work such as ViSOR (Menon et al., 2024) explicitly models post-VSR quality and encoding latency to optimize resolution per bitrate. However, most existing VSR-aware streaming approaches remain agnostic to device heterogeneity and energy consumption, typically assuming a single enhancement model and similar hardware characteristics. The absence of temporally-aware VSR models in existing bitrate-ladder design research is particularly notable, despite the substantial temporal consistency improvements they offer and their increasing feasibility on GPU-enabled consumer devices. As a result, hardware-aware, multi-model VSR selection, particularly for heterogeneous or energy-constrained clients, remains an open research challenge.

2.3 VVC and modern codec complexity

VVC delivers up to 40% bitrate reduction over HEVC (Sullivan et al., 2012; Bross et al., 2021) via a significantly expanded toolset: examples are multi-type tree partitioning (Park and Kang, 2021), advanced intra/inter prediction (Venugopal et al., 2019), improved transforms (Zhao et al., 2021), and stronger in-loop filters (Lakshmi Amruthavalli and Nalluri, 2022). This dramatically increases encoder complexity and, to a lesser degree, decoder load (Siqueira et al., 2020; Pakdaman et al., 2020; Mercat et al., 2021). Detailed VTM profiling identifies motion estimation, transform/quantization, and entropy coding as primary contributors on the encoder side (Pakdaman et al., 2020), while loop filtering and motion compensation dominate decoder compute (Islam and Ahmed, 2021).

The open-source implementations VVenC and VVdeC (Wieckowski et al., 2021; Wieckowski et al., 2020) enable practical deployment, supporting presets, threading, and coding configuration tuning. Recent studies explore Pareto-optimal configurations (Brandenburg et al., 2021) and complexity-reduction techniques via tool pruning or machine learning (Zupancic et al., 2021; Chou and Chen, 2023). Decoder-focused efforts include VVC “green metadata” to facilitate power-efficient playback (Herglotz et al., 2024). Comparative evaluations across codecs show that while AV1 can approach VVC’s efficiency, VVC remains the most computationally intensive for UHD/8K content (Uhrina et al., 2024). This compounds the cost of multi-representation encoding used in HAS, making brute-force ladder construction increasingly expensive.

Decoder energy scales with resolution, coding tool usage, and memory bandwidth (Wieckowski et al., 2020). Complexity-aware ladder prediction for VVC has therefore emerged, combining device-specific decoding-time prediction with resolution selection (Azimi et al., 2024). Yet, most works optimize only RD or decoding complexity in isolation, ignoring the impact of client-side enhancement. VSR shifts RD envelopes by allowing lower base resolutions to achieve comparable perceptual quality, thereby altering encoder cost, decoder load, and client-side energy. Integrating VVC complexity with enhancement-aware and energy-aware decisions remains underexplored.

2.4 Energy-aware video processing

Energy efficiency is increasingly central to streaming due to datacenter costs and the environmental impact of large-scale content delivery. Prior work addresses energy-conscious encoding (e.g., preset selection (Menon et al., 2022), multi-encode budgeting (Menon et al., 2023a)), decoder power measurement, and system-level energy estimation tools such as CodeCarbon (Courty et al., 2024). These works show that encoding energy scales with runtime and resolution, that decoder energy scales with pixel count and activated coding tools, and that client-side enhancement—such as VSR—introduces significant inference energy to the playback device. Most of these works either rely on analytical models or profile isolated pipeline components, whereas our approach is based on direct, per-representation measurements of decoding and enhancement energy on real devices.

However, most research examines encoding, transport, and client processing separately, or considers pre-VVC codecs like AVC/HEVC. Some VSR-aware studies apply JND-based pruning to reduce redundant ladder entries (Zhu et al., 2022; 2023; Amirpour et al., 2021; Menon et al., 2023b), but they generally exclude decoding and VSR energy from optimization. This oversight creates potential energy shifts, where bitrate or server savings from downscaling are negated by increased client-side inference energy, especially on mobile devices. Addressing this requires client-side decoding and enhancement energy accounting and device-aware adaptation policies (Seufert et al., 2014; Courty et al., 2024). These observations motivate E2-Stream’s measurement-driven and energy-aware design.

2.5 Summary and gap analysis

Existing literature provides solid foundations for (i) content-adaptive and latency-aware ladder design, (ii) VSR models that lift perceptual quality at low bitrate, (iii) VVC tools that advance compression at higher complexity, and (iv) initial techniques and tools for measuring energy at different stages of the pipeline (Bentaleb et al., 2019; Katsenou et al., 2021; Bross et al., 2021; Courty et al., 2024). Yet, no prior system holistically unifies these axes into a single selection loop that (a) explicitly incorporates client decoding energy, (b) accounts for VSR-enhanced quality at display resolution, (c) uses JND-based pruning to control ladder size and saturation, and (d) is demonstrated on VVC with open implementations (Wieckowski et al., 2021; Wieckowski et al., 2020). Enhancement-aware studies typically optimize perceptual quality or bitrate without device/energy awareness (Amirpour et al., 2021), while VVC-focused works emphasize RD and preset tuning without client enhancement in the loop (Siqueira et al., 2020). Likewise, energy-focused efforts often profile isolated stages rather than optimizing end-to-end trade-offs (Courty et al., 2024; Seufert et al., 2014).

E2-Stream addresses these gaps by measuring the actual rate–quality–decode-energy surface over a grid of VVC encodes and multiple VSR models, selecting per-bitrate representations using a decoding-energy–aware objective (quality minus a scaled decoding energy term), and pruning the ladder via JND thresholds and saturation to minimize redundancy. This approach yields compact, enhancement-aware ladders that respect device constraints and provide a principled path to energy-efficient streaming in the VVC era (Wieckowski et al., 2021; Wieckowski et al., 2020; Katsenou et al., 2021).

3 E2-stream: an energy-empowered VVC + VSR streaming architecture

E2-Stream is a measurement-driven framework for constructing enhancement-aware and energy-conscious bitrate ladders for VVC-based adaptive streaming. Each representation defined by an encoding resolution r, target bitrate b, and VSR model m is fully encoded, decoded, upscaled using VSR, and energy-profiled on real hardware. This produces an empirical rate–quality–energy landscape that directly reflects the behavior of VVC coding toolchain and practical VSR models, avoiding reliance on heuristics or predicted quality. By integrating client-side decoding and VSR inference energy into its objective, and by applying perceptual redundancy pruning, E2-Stream synthesizes compact ladders that preserve visual quality while respecting device-level energy constraints. This section presents the core components of the architecture, its offline workflow, and the decoding-energy–aware selection and pruning rules. All stages of E2-Stream operate offline during content preparation and representation selection; runtime network adaptation, ABR logic, and playback scheduling are intentionally outside the scope of this framework.

Let r ∈ R, b ∈ B, and m ∈ M denote the encoding resolution, bitrate, and VSR model, respectively. For each representation ( r , b , m ), E2-Stream evaluates the decoding-energy–aware utilitywhere Q ( r , b , m ) is the VMAF-measured perceptual quality at display resolution, and E dec and E vsr represent the measured decoding and VSR inference energies on the target device. While decoding and VSR inference times are measured alongside energy, latency is not explicitly optimized in the objective function; instead, real-time feasibility is validated empirically against segment playback deadlines in the experimental evaluation. The tunable parameter α D ≥ 0 controls the quality–energy balance: α D = 0 reduces to a quality-only selection, while larger values increasingly favor energy-efficient representations. This scalarization allows reproducible, measurement-based exploration of quality–energy trade-offs, as captured by the utility formulation in Equation 1, and provides a reference point for future hardware-adaptive extensions. We intentionally adopt linear scalarization to preserve interpretability and reproducibility across devices. More complex multi-objective solvers obscure trade-offs and hinder deployment. Exposing α D = 0 allows system designers to explicitly express energy budgets rather than implicitly embedding them in heuristics. The resulting measured design space can be interpreted as an empirical Pareto surface across bitrate, perceptual quality, and client-side energy, as illustrated in Figure 3.

J α D r , b , m = Q r , b , m − α D E dec r , b + E vsr r , b , m , (1)

3.1 Offline pipeline

Figure 4 outlines the offline workflow. For every segment, the server evaluates all combinations of ( r , b ), measures their rate–quality–energy characteristics after decoding and VSR enhancement, and applies bitrate-wise selection followed by JND-based pruning to produce a compact, perceptually meaningful ladder.

3.1.1 Grid of encodes

Given an input segment x, we encode all ( r , b ) ∈ R × B using a VVC encoder. Each elementary stream E S ( x ; r , b ) provides the achieved bitrate, file size, and encoding runtime. The corresponding encoding energy E enc ( r , b ) is directly measured on the encoding platform.

3.1.2 Decode and enhance

Each stream E S ( x ; r , b ) is decoded on the target client device, yielding y r , b. The decoding energy E dec ( r , b ) and decoding time are recorded. To account for client-side enhancement, each VSR model m ∈ M is applied to upscale y r , b to the display resolution R out, producing y ̃ r , b ( m ). We evaluate lightweight and moderate-strength VSR models suited for mobile, laptop, and GPU-equipped clients. For each model, inference time and VSR energy E vsr ( r , b , m ) are measured on the same device. The feasibility of executing VSR within segment playback deadlines is inherently device-dependent: lightweight SISR models are suitable for CPU-only clients, whereas temporally-aware VSR models are feasible only on GPU-accelerated platforms.

3.1.3 Measurement and logging

For every representation ( r , b , m ), E2-Stream records:

• Decoding energy and time E dec ( r , b ),
• VSR inference energy and time E vsr ( r , b , m ),
• Perceptual quality Q ( r , b , m ) computed as VMAF between y ̃ r , b ( m ) and the reference segment.

Measurements use platform-level instrumentation (e.g., Intel RAPL, GPU SMI) and are cross-validated with CodeCarbon. The aggregated dataset serves as the basis for computing the utility in Equation 1 and for synthesizing an energy-aware bitrate ladder.

3.2 Decoding-energy-aware objective

E2-Stream jointly evaluates quality and energy using the utility J α D, enabling direct comparison of representations that differ in resolution, bitrate, and enhancement strategy.

3.2.1 Quality metrics

Perceptual quality is quantified using VMAF at display resolution using Equation 2, ensuring that VSR-enhanced outputs are evaluated consistently.

Q r , b , m = V M A F y ̃ r , b m , reference . (2)

Signal-based metrics such as PSNR and SSIM are optionally reported for completeness (Katsenou et al., 2021; Seufert et al., 2014).

3.2.2 Energy definitions

Client-side energy is defined as the combined decoding and enhancement cost, expressed in Equation 3, asrepresenting the cost of decoding and enhancement for a single representation. Server-side encoding energy is measured for completeness and analysis but is not included in the ladder selection objective, which focuses on client-side decoding and enhancement costs.

E tot r , b , m = E dec r , b + E vsr r , b , m , (3)

3.2.3 Scalarized utility

The utility in Equation 1 integrates both quality and client energy into a single scalar score, enabling consistent multi-objective comparison across discrete representations. α D ≥ 0 is a tunable regularization parameter that controls the relative importance of energy consumption. Setting α D = 0 yields a quality-driven selection equivalent to classical RD optimization, while larger values of α D increasingly favor energy-efficient representations with lower client-side cost.

3.2.4 Rate-conditioned selection

Bitrate ladders are constructed for a set of target bitrates b ̂ ∈ B. For each b ̂, E2-Stream identifies the representations whose achieved bitrate is closest to b ̂ as shown in Equation 4.Among these candidates, the final selection is performed by maximizing the utility defined in Equation 5.

C b ̂ = arg min r , b ∈ R × B | bits r , b − b ̂ | , (4)

r b ̂ , b b ̂ , m b ̂ ∈ arg max r , b , m ∈ C b ̂ × M J α D r , b , m . (5)

Repeating this process for all b ̂ yields a preliminary ladder L α D.

3.3 Ladder synthesis with JND-Aware pruning

The preliminary ladder contains one representation per target bitrate, but adjacent entries may be perceptually redundant. E2-Stream therefore applies JND-aware pruning to ensure perceptual distinctiveness and suppress unnecessary storage and decoding cost. Unlike prior JND-based pruning applied to codec-centric RD ladders, E2-Stream applies perceptual pruning after decoding-energy–aware selection and on VSR-enhanced display quality, directly coupling perception with device-side cost.

3.3.1 Bitrate-wise selection

Ordering the bitrate levels as b ̂ 1 < … < b ̂ N, each entry i has an associated perceptual quality, defined in Equation 6, as

Q i = Q r b ̂ i , b b ̂ i , m b ̂ i . (6)

3.3.2 Perceptual pruning

Adjacent entries ( b ̂ i , Q i ) and ( b ̂ i + 1 , Q i + 1 ) are deemed perceptually redundant if their quality difference falls below the JND threshold, as expressed in Equation 7:where v J is the VMAF-based JND threshold. In such cases, the higher-bitrate entry b ̂ i + 1 is removed. Furthermore, if an entry reaches the perceptual saturation threshold defined in Equation 8,all higher-bitrate entries are discarded. Together, the conditions in Equations 7, 8 yield a compact ladder with perceptually meaningful steps and reduced storage and delivery overheads.

Q i + 1 − Q i < v J , (7)

Q i ≥ v T , (8)

3.3.3 Algorithm summary

The complete offline ladder construction consists of:

• Encoding all ( r , b ) pairs and measuring E enc.
• Decoding each stream, applying each VSR model, and measuring E dec, E vsr, and quality.
• Computing J α D for all ( r , b , m ).
• Selecting the best representation for each b ̂ via Equation 5.
• Applying JND-aware pruning using thresholds v J and v T.

This process produces energy-aware, enhancement-aware bitrate ladders that adapt effectively to device capabilities while maintaining perceptually meaningful quality increments.

4 Experimental setup

This section describes the dataset, hardware, metrics, and configuration parameters used to evaluate E2-Stream across VVC encoding, VSR enhancement, and end-to-end energy measurement. The objective is to ensure fair, reproducible comparisons that reflect realistic adaptive streaming scenarios.

4.1 Dataset

We use sequences from the Inter-4K dataset (Stergiou and Poppe, 2023), a diverse 4K corpus designed for large-scale video compression and streaming research. Its variation in motion, texture, and spatial detail makes it suitable for evaluating how codecs and VSR models behave under heterogeneous content conditions. All experiments are conducted at 60 fps, a common operating point for premium streaming and compatible with real-time decoder/VSR pipelines.

4.2 Hardware configuration

Server-side encoding is performed on a dual-processor Intel Xeon Gold 5218R workstation (80 cores @ 2.10 GHz). Each VVenC process is allocated four threads to preserve repeatability while reflecting multi-instance production environments. The client-side decoding and VSR inference are executed on two devices (cf. Table 1), a low-end and a high-end laptop, representing typical end-user hardware. This combination allows profiling of energy consumption across server (encoding), and client (decoding + VSR) components.

TABLE 1. Client devices used for evaluation.

4.3 Evaluation metrics

4.3.1 BDR_P and BDR_V

Following the Bjøntegaard formulation (HSTP-VID-WPOM, 2020), BDR_P and BDR_V quantify the average bitrate difference needed to achieve equivalent PSNR or VMAF relative to a reference ladder. Negative BDR values indicate improved coding efficiency.

4.3.2 Bjøntegaard delta total energy (BDCE)

To capture end-user device efficiency, we compute BDCE, defined as the Bjøntegaard delta over total client-side energy,representing the decoding plus VSR-inference energy. BDCE quantifies the percent change in total energy consumption required to maintain equivalent perceptual quality relative to a reference ladder (Herglotz et al., 2019). Negative BDCE values indicate energy savings at equal quality, while positive values reflect higher energy cost. In addition to energy, per-segment decoding time and VSR inference time are recorded to assess real-time feasibility. These measurements are compared against segment playback duration to verify whether decoding and enhancement can complete within playback deadlines on each device.

E tot r , b , m = E dec r , b + E vsr r , b ,

4.3.3 Storage metrics

The relative storage difference between an optimized ladder b opt and a reference ladder b ref is:

Δ S = ∑ b opt ∑ b ref − 1 . (9)

All energy measurements are conducted following a controlled protocol. Each experiment is repeated at least five times, with reported values corresponding to the mean and standard deviation. Devices are thermally stabilized prior to measurement, CPU frequency governors are fixed to performance mode, and background processes are minimized. Energy is sampled at a fixed interval and aggregated over complete decode and VSR inference runs. RAPL-based CPU measurements and GPU vendor tooling are validated under sustained load to ensure consistency.

4.4 Experimental parameters

Table 2 summarizes the configuration space. We evaluate three encode resolutions (540p, 1080p, 2160p) and twelve bitrates, forming a dense grid for Pareto-front estimation.

TABLE 2. Experimental parameters used to evaluate E2-Stream.

4.4.1 JND-based pruning and saturation constraint

We employ JND-based pruning using thresholds v J ∈ { 2,4,6 } (Kah et al., 2021; Ozer, 2017). These values are chosen to span a range of practical operating points, from conservative pruning ( v J = 2 ), which retains finer quality steps, to more aggressive pruning ( v J = 6 ), which yields compact ladders with fewer representations.

In addition to pairwise pruning, we define a saturation thresholdwhere 100 corresponds to the maximum attainable VMAF. This definition ensures that the perceptually near-lossless region, i.e., quality levels within a JND margin of the maximum score—is not over-represented in the ladder. Specifically, quality values in the range [ v T , 100 ] are considered perceptually equivalent within the chosen JND tolerance, according to the saturation condition defined in Equation 10.

v T = 100 − v J , (10)

4.5 Decoding-energy regularization parameter

The regularization parameter α D governs the balance between perceptual quality and client-side energy consumption in the selection objective used by E2-Stream. In practice, α D is varied over a predefined range (cf. Table 2) to explore different operating points, from quality-driven ( α D = 0 ) to energy-aware configurations ( α D > 0 ).

To ensure a meaningful trade-off between the two terms in the objective function, quality and energy are first normalized to comparable numerical ranges. This prevents either component from dominating due to differences in scale and ensures that α D consistently reflects the intended relative weighting between perceptual quality and energy consumption across all representations.

4.5.1 VVC encoding and decoding configuration

All video representations are encoded using the open-source VVC encoder, VVenC (Wieckowski et al., 2021), configured in the Random Access (RA) mode according to JVET common test conditions (CTC) (Bossen, 2019). The RA configuration is chosen to reflect typical streaming scenarios, where periodic intra frames enable efficient random access and bitrate adaptation while maintaining high compression efficiency. We employ the medium_lowDecEnergy preset (Kränzler et al., 2022), which is specifically designed to reduce decoder complexity without significantly compromising rate–distortion (RD) performance. This preset selectively restricts computationally intensive coding tools, thereby lowering the decoding workload and associated energy consumption on client devices. Such configurations are particularly relevant in the context of energy-aware streaming, where decoder efficiency becomes a critical system parameter. All encodings are performed with a fixed intra period of one second, aligning with typical segment durations in HAS. Furthermore, the encoder operates under constrained computational settings (e.g., fixed number of threads per instance) to ensure reproducibility and to approximate practical multi-instance encoding environments.

Decoding is performed using the corresponding open-source VVC decoder, VVdeC (Wieckowski et al., 2020). For the CPU-only client (Client A), decoding is executed entirely in software, representing a worst-case scenario in terms of computational load and energy consumption. For the GPU-enabled client (Client B), decoding leverages available hardware acceleration where supported by the operating system and driver stack, resulting in reduced decoding latency and improved energy efficiency. Both decoding configurations are instrumented to measure runtime and energy consumption, enabling accurate characterization of the client-side decoding cost across different resolutions and bitrates.

4.5.2 VSR methods

We evaluate a set of spatial upscaling and lightweight single-image super-resolution (SISR) methods, including traditional interpolation techniques (Bicubic, Lanczos) and learned models, FSRCNN (Dong et al., 2016) and ESPCN (Shi et al., 2016). These methods operate on a frame-by-frame basis without exploiting temporal information and are designed to provide efficient upscaling suitable for real-time deployment on resource-constrained devices.

In addition, we consider a temporally-aware VSR model, Enhanced Deformable Video Restoration (EDVR) (Wang et al., 2022), which leverages multi-frame alignment and aggregation to improve temporal consistency and reconstruction quality. Due to its higher computational and memory requirements, EDVR is evaluated only on the GPU-enabled client (Client B).

4.6 Benchmark methods

To contextualize the performance of E2-Stream, we compare against two representative baselines:

• Default: A fixed bitrate ladder following Apple HLS guidelines (Apple Inc., 2024). This reflects common industry practice and provides a consistent static reference across all content.
• DynResVMAF (De Cock et al., 2016): A content-adaptive approach that predicts the optimal resolution for each target bitrate using complexity features and VMAF-based modeling. Unlike E2-Stream, it does not incorporate decoding or VSR energy and assumes the decoded signal is displayed without VSR. In our discussion, this scheme is equivalent to the E2-Stream formulation with α D = 0. This baseline represents state-of-the-art enhancement-agnostic ladder design and serves as a direct quality-only reference for evaluating the benefits of energy-aware and VSR-aware optimization.

5 Experimental results and analysis

This section presents the empirical evaluation of E2-Stream, focusing on: (i) rate–distortion (RD) behaviour with and without enhancement and under the decoding-energy–aware objective J α D, (ii) storage and transmission energy implications; (iii) ablations on the JND threshold v J, the regularization parameter α D, and VSR model selection.

5.1 Rate–distortion analysis

5.1.1 Measured RD behavior and cross-overs

For each representation ( r , b , m ), we obtain a measured RD sample ( b i t s ( r , b ) , Q ( r , b , m ) ) as described in Section 3.2. Figure 5 illustrates these RD curves for three representative sequences. Consistent with observations in enhancement-aware encoding (Katsenou et al., 2021; Amirpour et al., 2021), applying post-decode VSR significantly elevates the quality of low-resolution encodes. Across all sequences in Figure 5, cross-over effects are clearly visible in the low-to-mid bitrate region: (i) 1080p + VSR frequently matches or surpasses the VMAF of native 2160p, and (ii) 540p + VSR approaches native 1080p at constrained bitrates. These cross-overs reshape the upper-envelope RD frontier, effectively shifting high-quality operating points toward lower bitrates and altering which ( r , b ) pairs remain Pareto-optimal.

5.1.2 Quality-decoding-energy trade-off

While DynResVMAF select representations by maximizing VMAF, the E2-Stream formulation instead optimizes the scalar utility function (Equation 1), where higher α D places more emphasis on reducing client-side decode and VSR energy. The effect of this trade-off is clearly reflected in the rate–client-energy curves in Figure 5, complementing the RD plots above them.

Two consistent behaviors emerge across both client platforms:

5.1.2.1 Quality-dominant regime ( α D = 0 )

When α D = 0, the optimization reduces to a quality-only formulation equivalent to DynResVMAF. As shown in Tables 3, 4, this regime yields the strongest bitrate savings across all models (e.g., FSRCNN achieves BDR_P of − 28.24 % on Client A and − 29.38 % on Client B at v J = 0). However, this comes at a significant energy cost on the CPU-only device, where BDCE increases dramatically (up to 2835.91 % for FSRCNN), reflecting the high computational burden of VSR inference. In contrast, on the GPU-enabled device, the same configurations lead to net energy reductions (e.g., FSRCNN BDCE − 20.71 %), highlighting the importance of hardware acceleration.

TABLE 3. Average results of E2-Stream compared to Default encoding (Apple Inc., 2024) for Client A.

TABLE 4. Average results of E2-Stream compared to Default encoding (Apple Inc., 2024) for Client B.

5.1.2.2 Balanced regime ( α D = 0.25 )

At α D = 0.25, E2-Stream achieves a balanced trade-off between quality and energy. Bitrate savings remain substantial (e.g., FSRCNN BDR_P between − 26.18 % and − 24.09 % on Client A), while energy consumption is significantly reduced compared to α D = 0. On Client A, BDCE decreases from extreme values to a more moderate range (e.g., ≈ 85 % for FSRCNN), indicating partial mitigation of inference cost. On Client B, all VSR models consistently achieve negative BDCE (energy savings), with FSRCNN reaching up to − 57.92 % for v J = 4. This demonstrates that moderate regularization effectively suppresses energy-expensive configurations while preserving most of the RD gains.

5.1.2.3 Energy-aware regime ( α D = 0.5 )

For α D = 0.5, the optimization prioritizes energy efficiency more strongly. As shown in Table 3, bitrate savings slightly decrease (e.g., Bicubic BDR_P changes from − 14.28 % to − 12.36 % as v J increases), but energy reductions become more pronounced. Importantly, Bicubic consistently achieves negative BDCE (down to − 26.81 %), demonstrating that purely interpolation-based enhancement can yield both bitrate and energy gains on CPU-only devices. For learned models, FSRCNN and ESPCN still provide substantial bitrate savings but incur higher energy cost on Client A (BDCE up to 89.62 %), while on Client B they deliver significant energy reductions (BDCE down to − 57.49 %). This confirms that higher α D shifts the ladder toward lower-resolution, energy-efficient representations.

5.1.2.4 Device-dependent behavior

The results reveal a clear divergence between the two client platforms. On the CPU-only device (Client A), VSR inference dominates the energy budget, leading to substantial increases in total client-side energy, particularly for learned models such as FSRCNN and ESPCN. In contrast, on the GPU-enabled device (Client B), hardware-accelerated decoding and parallel VSR execution significantly reduce the energy cost per frame, resulting in negative BDCE values across most configurations. This fundamental difference demonstrates that the effectiveness of enhancement-aware bitrate ladders is strongly dependent on the underlying hardware capabilities.

5.1.3 Aggregate indicators

To summarize the joint behaviour of bitrate, quality, and energy, we compute BDR_V and BDCE relative to Default.

On the CPU-only client (Table 3), E2-Stream achieves consistent bitrate reductions across all models, with BDR_V values ranging from approximately − 26.17 % (Bicubic) to − 36.94 % (FSRCNN) at α D = 0.5. Energy behaviour, however, varies significantly depending on the enhancement method: while Bicubic yields net energy savings (BDCE down to − 26.81 %), learned VSR models incur higher energy cost (BDCE > 80 %). These results indicate that lightweight interpolation can provide a favorable trade-off on energy-constrained devices.

On the GPU-enabled client (Table 4), both bitrate and energy improvements are achieved simultaneously. FSRCNN and EDVR attain BDR_V gains up to − 40.83 % and − 41.44 %, respectively, while also reducing total energy consumption (BDCE as low as − 57.49 %). This confirms that hardware acceleration fundamentally reshapes the rate–quality–energy landscape, enabling configurations that are both bandwidth-efficient and energy-efficient.

Overall, these results highlight that the decoding-energy–aware objective effectively selects representations that adapt to device capabilities, preserving perceptual quality while minimizing client-side energy consumption.

5.2 Storage consumption analysis

We additionally evaluate storage implications by measuring the cumulative encoded size of all ladder representations. Across sequences, E2-Stream achieves substantial storage reductions, reaching up to 58.06 % on Client A and similar levels on Client B (cf. Tables 3, 4). These reductions are primarily driven by the elimination of high-bitrate representations that provide minimal perceptual gain.

5.2.1 Effect of JND pruning and saturation

JND pruning removes representations with sub-threshold quality gains, while the saturation constraint truncates the ladder once perceptual quality approaches the near-lossless region. Together, these mechanisms effectively reduce ladder density, particularly in the high-bitrate regime.

5.2.2 Effect of α D

Increasing α D further amplifies storage savings by favoring lower-resolution encodes and energy-efficient configurations. This leads to systematic removal of high-resolution representations that are both energy-intensive and perceptually redundant. The combined effect of energy-aware selection and perceptual pruning results in compact ladders that reduce storage and delivery costs without compromising visual quality.

5.3 Latency and real-time feasibility

In addition to energy efficiency, real-time feasibility is a critical requirement for practical deployment. For each representation, we measure decoding and VSR inference times and evaluate the condition:where T seg denotes the playback duration of a video segment.

T dec r , b + T vsr r , b , m ≤ T seg , (11)

The results reveal a strong correlation between decoding energy consumption and execution time. In particular, configurations with high BDCE values on the CPU-only device (Client A) correspond to significantly increased processing times, indicating that SISR/VSR inference dominates both energy and latency. As a result, several VSR-based configurations on Client A do not satisfy real-time constraints, especially at higher resolutions and bitrates.

In contrast, on the GPU-enabled client (Client B), hardware-accelerated decoding and parallel VSR inference significantly reduce both energy consumption and processing time. Most configurations, including those using FSRCNN and EDVR, satisfy real-time constraints across the evaluated bitrate range. This demonstrates that real-time feasibility and energy efficiency are closely aligned on GPU-capable devices, whereas they diverge on CPU-only systems.

These findings further reinforce that enhancement-aware bitrate ladders must be adapted to device capabilities: while CPU-only clients require lightweight or interpolation-based enhancement to maintain real-time playback, GPU-enabled devices can support more complex VSR models without violating latency constraints.

5.4 Ablation study

We analyze how v J, α D, and VSR model selection contribute to the behaviour of E2-Stream.

5.4.1 JND threshold sensitivity

Figure 6 shows that increasing v J progressively reduces ladder density while maintaining the original RD envelope. Only entries with perceptually meaningful quality increments are retained, and pruned curves remain closely aligned with the full candidate set. Storage savings scale with v J: for Sequence 0251, Δ S improves from 45.02 % ( v J = 2 ) to 72.20 % ( v J = 6 ); Sequence 0345 shows a similar trend (43.64 % to 64.80 %). Smoother content prunes more aggressively, while high-motion sequences retain more operating points to preserve quality. Increasing v J also lowers the saturation threshold v T, further reducing high-bitrate tiers. BDR_V remain stable, indicating that pruning maintains the fundamental RD behaviour, with the main trade-off being reduced bitrate granularity at high JND levels.

5.4.2 Regulation parameter sensitivity

Varying α D reveals a clear quality–energy trade-off. At α D = 0, the optimization prioritizes perceptual quality, achieving maximum bitrate savings but incurring high energy cost on CPU-only devices. Increasing α D to 0.25 reduces energy consumption substantially while retaining most bitrate gains. At α D = 0.5, energy reductions become dominant, with minimal additional quality degradation. For example, FSRCNN BDR_P changes from approximately − 28.24 % ( α D = 0 ) to − 24.68 % ( α D = 0.5 ) on Client A, while BDCE decreases significantly in the same range. This demonstrates that α D provides a controllable mechanism to navigate the rate–quality–energy trade-off.

5.4.3 VSR model sensitivity

The choice of VSR model has a pronounced and highly device-dependent impact on the rate–quality–energy trade-off, as evidenced by Tables 3, 4. On the CPU-only client (Client A), lightweight SISR models such as FSRCNN and ESPCN achieve strong bitrate reductions (e.g., BDR_P down to −24.68% and −23.04% at α D = 0.5, v J = 0), but incur substantial energy overhead due to CPU-bound inference, with BDCE values exceeding 85% for FSRCNN and 45% for ESPCN. In contrast, Bicubic interpolation consistently achieves negative BDCE values (down to −26.81%), demonstrating that its negligible computational cost enables simultaneous bitrate and energy savings.

On the GPU-enabled client (Client B), the situation reverses: hardware acceleration allows learned models to outperform interpolation both in efficiency and quality. FSRCNN achieves BDCE reductions of up to −57.49% and maintains strong bitrate savings (BDR_P ≈ − 25 %), while EDVR further improves rate–distortion performance (BDR_V up to −41.44%) with moderate but still negative energy impact (BDCE ≈ −31.90%). This highlights that temporally-aware VSR becomes viable only when supported by dedicated hardware, whereas on CPU-only systems its computational cost would be prohibitive.

Lightweight VSR models (FSRCNN, ESPCN) provide the best trade-offs for higher α D and low–mid bitrates, offering strong quality uplift at low E tot. This is particularly visible at α D = 0.5, where FSRCNN and ESPCN remain part of the selected envelope for mid-bitrate ranges despite pruning, owing to their favorable balance between RD gains and energy cost. Moderate-complexity models excel only when α D is small, as their higher inference cost becomes prohibitive when energy regularization increases. For example, EDVR is only selected in the GPU-enabled case and primarily at low α D, where quality gains dominate the objective.

As α D varies, the dominant model in the envelope J α D ⋆ shifts predictably: lightweight models at low bitrates and high α D, moderate ones briefly at mid/high bitrates under low α D. This transition is clearly reflected in the BDCE trends: CPU-based configurations favor Bicubic under high α D, while GPU-based configurations continue to select FSRCNN and EDVR due to their reduced inference cost. Storage differences follow the same pattern, lightweight models drive ladders toward lower resolutions and smaller file sizes, especially after pruning.

Across both devices, E2-Stream consistently reshapes the RD envelope by exploiting enhancement-induced cross-overs. Lower-resolution encodes combined with VSR achieve near-native perceptual quality, shifting optimal operating points toward lower bitrates. However, the associated energy trade-offs diverge significantly: on Client A, higher α D primarily suppresses costly VSR configurations, whereas on Client B it enables synergies between efficient decoding and accelerated inference, leading to simultaneous improvements in bitrate, quality, and energy.

Across sequences, E2-Stream consistently reshapes the RD envelope via enhancement-aware encoding and decoding-energy regularization. VSR-induced cross-overs allow lower-resolution encodes to achieve near-native perceptual quality, while higher α D delivers large client-side energy reductions with minimal quality loss. Importantly, these benefits are only realized when the underlying hardware supports efficient execution of enhancement models. JND pruning further compresses the ladder, producing compact, enhancement-aware, and energy-efficient representations suitable for diverse devices.

6 Conclusion and future work

This work presented E2-Stream, an energy-empowered adaptive streaming framework that unifies VVC encoding, client-side VSR enhancement, and client-side decoding and enhancement energy profiling. By exhaustively evaluating all combinations of resolutions, bitrates, and VSR models on real hardware, E2-Stream constructs bitrate ladders grounded in empirical rate–quality–energy behaviour rather than heuristic assumptions. The integration of decoding and VSR inference energy into a tunable objective function enables the framework to explicitly navigate the quality–energy trade-off, while JND-aware pruning removes perceptually redundant steps and yields compact, storage-efficient ladders. Our experiments on the Inter-4K dataset show that VSR fundamentally reshapes the rate–distortion landscape of VVC streaming. Downscaled encodes enhanced with lightweight or moderate-complexity VSR frequently match or surpass the perceptual quality of native higher-resolution encodes, while offering 12%–28% bitrate savings across models. However, this quality uplift comes at a cost: client-side decoding and VSR inference energy increase by approximately 85% on a CPU-only device, underscoring the need to explicitly model device-level energy. In contrast, on GPU-enabled devices, hardware-accelerated decoding and VSR inference reduce total client-side energy by 25 % to 60%, demonstrating that enhancement-aware streaming can achieve simultaneous gains in bitrate efficiency and energy consumption under appropriate hardware conditions. The decoding-energy–aware objective mitigates these costs by shifting selection toward energy-lean representations, and JND pruning further reduces redundancy, achieving storage reductions of up to 58.98% without compromising adaptation smoothness or exceeding sub-JND perceptual differences. The key insight is that, enhancement-aware bitrate ladders are inherently device-dependent, with lightweight interpolation preferred on CPU-only systems and learned VSR models becoming advantageous on GPU-enabled platforms. These findings highlight that optimal streaming configurations can no longer be derived solely from codec-centric RD behavior. Instead, modern systems must incorporate hardware capabilities, enhancement models, and device-level energy constraints when constructing bitrate ladders. E2-Stream offers a principled foundation for such cross-layer, enhancement-aware, and energy-conscious optimization.

Future work includes extending E2-Stream to real-time adaptive pipelines, integrating device-side telemetry (e.g., thermal and DVFS states), and evaluating emerging mobile-optimized VSR architectures. Additionally, incorporating Common Media Client Data (CMCD) signaling may enable fully dynamic, device-aware bitrate-ladder selection during playback (Bentaleb et al., 2021). Together, these directions pave the way toward sustainable, perceptually aware, and energy-efficient next-generation streaming systems.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Yun Zhang, Sun Yat-sen University, China

Reviewed by: Daniel Joseph Ringis, The University of the West Indies St. Augustine, Trinidad and Tobago

Joao Da Mata Liborio Filho, University of the State of Amazonas, Brazil