SWEET: serving workload-balanced end-to-end efficient and tailored edge inference via quantization and partitioning

1 Introduction

Although deep learning has had tremendous success in computer vision (Krizhevsky et al., 2012), natural language processing (Vaswani et al., 2017), and autonomous transportation (Bojarski et al., 2016), the deployment of these already-trained and frontier models is still performed on server clusters, exposing the edge users to the limitation of high latency (Zhou et al., 2019), increased dependency on high transmission bandwidth (Shi et al., 2016), and potential privacy risks due to data transmission to centralized servers (Zhou et al., 2019). Many earlier efforts attempted to bring the frontier pre-trained machine learning models to the network’s edge. For instance, dedicated hardware accelerators such as Google’s Edge TPU and Nvidia’s Jetson are optimized for performing neural network inference on edge devices (Mittal, 2019). Within the network structure search (NAS) realm, several efforts have tried to discover neural networks optimized for deployment on edge devices (Sandler et al., 2018). Model compression is a common and low-cost method to deploy neural networks. Many schemes and frameworks, such as early exit (Prechelt, 2002), model pruning (Lin et al., 2022), autoencoders (Hu and Krishnamachari, 2020), and knowledge distillation (Chen et al., 2020), have been rigorously explored and integrated into the inference offloading problem within the distributed network. From the perspective of computation, inference offloading and model partitioning (Kang et al., 2017; Liu et al., 2019) have been promising tools for reducing the cost of deploying the machine learning models on the edge by offloading partial or all of the computation to more powerful servers.

While the performance and potential of certain technological advancements facilitating machine learning models on edge devices have been extensively explored in numerous prior studies, their constraints, such as hardware installation costs, model retraining expenses, and the costs associated with fine-tuning, remain significant considerations. Moreover, when deploying machine learning tasks on edge devices, the device type, computation capacity, expected accuracy, and channel conditions impact the inference performance and evolve over time, making it inappropriate to find a universal solution across all future inference queries. Specifically, there are a few factors to be considered before pushing the inference toward the edge of the network:

• Different hardware limitations on the edge: Edge devices such as cell phones, smart watches, network cameras, and VR/AR glasses generate machine learning inference queries but differ in computational capabilities. Different computation capacity comes from the variations in processor architecture, instruction sets, and clock rates (Wu et al., 2019; Bhattacharya and Lane, 2016). Devices with powerful processors are more capable of handling inference locally.
• Changing transmission conditions: The wireless transmission channel significantly impacts inference service (Amiri and Gündüz, 2020). According to Shannon’s capacity formula (Shannon, 1948), a higher signal-to-noise ratio (SNR) leads to greater bandwidth and lower latency. Hence, higher accuracy may be achieved by trading with latency. Considering channel capacity in inference or offloading makes solutions more robust against network fluctuations.
• Diverse accuracy requirement for applications: Desired accuracy varies by application. Critical tasks such as medical diagnosis (Miotto et al., 2018), autonomous vehicles (Bojarski et al., 2016), and power grid monitoring (Al-Omari et al., 2021) demand high accuracy. In contrast, applications such as recommendation systems (Hallinan and Striphas, 2016) and social media (Basarslan and Kayaalp, 2020) tolerate some accuracy loss. Accepting minor accuracy degradation can often save significant computing resources (Han et al., 2015).

To facilitate the deployment of already-trained neural network models on various edge devices without incurring high costs, this article proposes an accuracy-aware, model-less inference system. The system responds to inference requests from edge devices by automatically quantizing models and distributing computation workloads between the server and the device. As illustrated in Figure 1, the proposed system, serving workload-balanced end-to-end efficient and tailored edge inference (SWEET), functions as a machine learning inference infrastructure that maintains pre-trained models for a variety of edge applications on the edge server. When a neural network-based application with a specific accuracy requirement is initiated on an edge device, a model request is sent to SWEET, specifying the application task, required accuracy, and available local computation capacity. Upon receiving this request, SWEET responds with a quantized lower-bitwidth model, generated by solving a joint optimization problem constrained by the accuracy degradation, which is modeled as a layer-wise accuracy degradation measurement. Simultaneously, SWEET determines an optimal model partition point based on the current system conditions. The portion of the model before the partition point that is referred to as the local model segment is then transmitted to the edge device for local inference. After the edge device processes the raw input data through the local model segment, the intermediate output is sent back to the SWEET server for further inference. The final inference result is then returned to the edge device, if required. As inference requests continue to be generated from the device, SWEET periodically updates both the quantized model and the partition point, thereby balancing system performance with algorithmic overhead. Experimental results show that our framework can reduce communication payload, which is the dominant source of latency in many edge computing scenarios, by up to 80% (Kang et al., 2017), while keeping accuracy degradation below 1%. A comparative evaluation against existing model compression methods further confirms the efficiency of the proposed approach.

The contributions of this work are summarized as follows:

• SWEET, an inference serving system consisting of adaptive model quantization and automatic computation workload balancing, was proposed. Instead of scaling the pre-trained machine learning (ML) models for the edge manually, inference queries from the edge device are responded to by SWEET with a model for properly quantized lower latency, energy consumption, and cost, along with guaranteed accuracy.
• A joint optimization framework of layer-wise quantization and model partitioning is built for the SWEET system. The computation workload, time consumption, energy consumption, cost of server resources, and transmission latency are modeled within the inference request, forming an optimization problem that is well-solved with a closed-form solution of quantization bitwidth and partition point by introducing accuracy degradation measurement (Zhou et al., 2018) as the constraint, enabling SWEET to optimize quantization bitwidth and partition point without violating the accuracy requirement.
• Based on the solution, an offline quantization algorithm and online scheduling algorithm are proposed to optimize the partition point and layer-wise quantization in runtime.
• A simulation platform is built, and numerous experiments are carried out to show the efficiency of the proposed SWEET serving inference queries. The SWEET can reduce communication payloads by more than 80% while keeping accuracy degradation below 1%.

The article is organized as follows: Section 2 reviews the related work of inference offloading and model scaling. Section 3 outlines the target SWEET system settings and the modeling of inference task offloading, including model quantization and partitioning. Section 4 formulates the optimization problem and introduces the corresponding algorithm. The execution and discussion of simulations are covered in Section 5, and Section 6 presents the conclusion.

2 Background and related work

As an inference serving system, SWEET features model quantization and distributed inference, which locates the system at the intersection of model scaling and inference offloading methods. This section starts by elaborating on these two fundamental techniques that allow the pre-trained model to be deployed on the edge.

2.1 Deploying models toward the edge

Many modified neural networks or novel learning paradigms have been proposed to address the imbalance between mobile devices and server computational resources. Early exit, model pruning, and autoencoder, as techniques designed to enhance the computational efficiency of deep neural networks, are beneficial for devices with limited resources. Barbosa et al. (2022) proposed the decision early exit (DEEx) approach, which is implemented at a network’s first early exit point. DEEx aims to reduce total inference time by avoiding unnecessary evaluations at early exits that do not enhance model performance. Ren et al. (2023) proposed an offloading mechanism that, aided by model pruning, can significantly reduce latency and energy consumption compared to other algorithms. By proposing a two-step pruning framework for partitioning deep neural networks (DNNs) between mobile devices and multi-access edge computing (MEC) servers, Shi et al. (2019) effectively reduced the wireless transmission workload of the device or the total computation workload, highlighting the benefits of model pruning in cooperative inference between devices and MEC servers. Autoencoders were also utilized to facilitate inference on the edge. Hu and Krishnamachari (2020) investigated the streaming inference of CNN to simultaneously improve throughput and accuracy by communication compression. Specifically, an autoencoder was inserted into the CNN model to compress the communication payload.

Many machine learning-based inference offloading algorithms effectively reduce latency and maintain accuracy, but they often require modifications to the base model, hindering scalability and adaptability. For instance, early exit mechanisms (Li N. et al., 2022) necessitate additional training for each exit point, increasing training costs. Model pruning methods (Ren et al., 2023; Shi et al., 2019) risk accuracy loss when applied across various neural networks, and efforts to mitigate these issues, such as re-tuning, are labor-intensive. Integrating autoencoders adds unnecessary computational overhead (Hu and Krishnamachari, 2020). In contrast, our proposed model mitigates the computational burden by quantizing model parameters and intermediate outputs to reduced bitwidths, thus circumventing additional computations or modifications to the original neural networks.

2.2 Offloading the inference workload on servers

Inference offloading (Salmani et al., 2023; Ghafouri et al., 2025), the process of transferring data-intensive and computation-heavy tasks from resource-constrained devices to more capable computing nodes, faces significant bandwidth, latency, and energy consumption challenges.

Latency is a critical focus in inference offloading. Kang et al. (2017) introduced Neurosurgeon, which reduces inference latency, power consumption, and system throughput by partitioning computation across edge devices and cloud servers. Duan and Wu (2023) extended this by jointly optimizing DNN partitioning and pipeline scheduling to further reduce latency. Zhao et al. (2020) explored the trade-off between latency and accuracy in edge-cloud offloading, focusing on offloading raw data with reduced quality. The challenge of power efficiency in offloading due to limited power on edge devices has also been studied. Xu et al. (2020) optimized device and server energy consumption in 5G-enabled mobile edge clouds.

In this article, we proposed SWEET, which, to our best knowledge, is the first inference serving platform that combines automatic model quantization and adaptive inference offloading. By quantizing the model and optimizing the computation offloading pattern on the server, no model variants must be pre-tuned or stored.

Proposed by Zhang et al. (2020), Model-Switching is an inference serving scheduler that reacts to workload fluctuations solely by swapping among pre-trained model variants to trade accuracy for latency on a fixed set of machines. It is the most relevant work related to SWEET. In their approach, the model variant searching system responds to inference queries by identifying the optimal neural network variant within a set of manually-trained models, such as the ResNet family (He et al., 2016), including ResNet-18, ResNet-34, and ResNet-50. However, the SWEET responds with an optimal model variant based on accuracy-aware quantization on already-trained models, enabling SWEET to be easily adapted to a wide variety of deep learning models. Furthermore, as a centralized inference serving system, Model-Switching responds to inference requests by handling all computation workload on the server, which makes it less capable of maintaining satisfied latency under some circumstances. In contrast, the proposed SWEET system is a flexible, end-to-end inference serving platform for edge computing that not only scales the models but also distributes the inference workload adaptively. Hu et al. (2024) proposed a digital-twin-assisted device–edge DNN inference framework that adaptively decides when local execution should stop and intermediate features should be offloaded to the edge server. Zhang et al. (2025) proposed AMP4EC, a resource-aware edge inference framework that combines real-time monitoring with adaptive model partitioning and task scheduling in heterogeneous environments. Table 1 compares SWEET with related works.

TABLE 1. Comparison of related systems accommodating the neural networks on the edge. Latency: Is time consumption considered and optimized? Energy: Is power consumption considered and optimized? Accuracy: Is the inference accuracy considered? Scaling: Does the system scale the model for the edge devices? Offloading: Does the system support collaborative inference with the devices?

3 System model

In this section, we elaborate on our proposed inference serving system SWEET from the perspectives of system overview, inference offloading design, model quantization design, and communication model.

3.1 MEC system

First, we start with the whole picture of our proposed system shown in Figure 2. In edge computing systems, various neural networks can be deployed on each of the edge devices according to time-varying application tasks. At the beginning of the application task deployed on the edge device, an inference request r from a mobile device is sent to the edge server. Specifically, the query r is defined by a tuple r = ( θ , a ), where θ indicates the specific neural network model and a is the accuracy demand, which is defined as the maximal acceptable accuracy degradation compared to the initial accuracy of the model. Once the server receives the inference query, the optimized layer-wise quantization bitwidth and model partition point will be solved by the SWEET system. Specifically, model quantization, without degrading accuracy beyond requirements, effectively decreases the communication payload, thereby reducing communication latency, a primary contributor to overall inference latency in MEC networks (Li et al., 2019). Based on the optimal quantization bitwidth and partition point, model partitioning is used to split the inference task into two sequential segments, which are processed on the edge device and the server, respectively. Thereafter, the first model segment will be quantized and then sent back to the edge device. The edge device will perform the inference with a quantized model segment and send the intermediate output to the server for the inference of the second model segment. Once all inferences have been operated, the final results will be returned to the device or other nodes if needed. In order to reduce algorithmic overhead and communication rounds for more efficient inference, the configuration of quantization bitwidth and partition point will be updated periodically in the following inference queries.

3.2 Inference offloading

As illustrated in Figure 2, each inference query r originating from an edge device is divided into two distinct segments for distributed inference on the device and the server, respectively. We elaborate on the model partition process and analyze the elements contributing to time and energy consumption, including device-based and server-based inferences and communication-related overhead.

For request r with network model θ, the activation size of layer l ∈ { 1,2 , … , L } is indicated as z x, and the number of weight parameters in layer l is z l. Additionally, we quantify the computational demand of layer l in terms of multiply-accumulate (MAC) operations, referred to as o ( l ). To adapt our model to various scenarios, we present the formulation of o ( l ) for both linear (fully connected) and convolutional layers.

3.2.1 Linear layer computation cost

If layer l is a linear layer with input activation X ∈ R 1 × D and weight matrix W ∈ R D × G, the output activation can be expressed as Y = X ⋅ W , Y ∈ R 1 × G. Hence, the computation cost of layer l can be expressed in Equation 1; (Lahmer et al., 2022):

o l = D × G . (1)

3.2.2 Convolutional layer computation cost

A standard convolutional layer has the computation cost of (Lahmer et al., 2022) as shown in Equation 2 below:where C i n and C out are the numbers of input channels, output channels of layer l, F 1, and F 2 define the size of the convolutional filter, and U and V are the size of input activation.

o l = C i n × C out × F 1 × F 2 × U × V , (2)

Therefore, with the optimal model partition point p given for request r, the computational costs of two segments of the neural network s can be formulated in Equation 3 and Equation 4:

O 1 p = ∑ l = 1 p − 1 o l , (3)

O 2 p = ∑ l = p L o l . (4)

Let f local > 0 denote the clock rate of the edge device when inferring the first part of the request. The time consumption of local inference can be expressed as shown in Equation 5 below:where γ local is a constant, referring to the average number of clock cycles needed to finish one MAC operation on the mobile user device. Generally, it depends on the processor architecture, instruction set architecture, pipeline, etc.

T local = O 1 ⋅ γ local ⋅ 1 f local , (5)

According to De Vogeleer et al. (2014), the power consumption of a processor chip is proportional to V 2 ⋅ f clock, where V represents the circuit supply voltage and f clock is the clock rate. Burd and Brodersen (1996) observed that the clock frequency is approximately linearly proportional to the voltage supply, particularly when operating at low voltage limits. Therefore, the energy consumption per clock cycle can be defined as κ f clock 2, where κ denotes the energy efficiency parameter, which depends on the hardware architecture. Consequently, the energy consumption for local inference can be formulated as shown in Equation 6 below:where κ n is the energy efficiency parameter of device n. In a similar vein, the subsequent expression of time consumption of server-based inference can be obtained as shown in Equation 7 below:

E local = κ f local 2 ⋅ O 1 ⋅ γ local , (6)

T server = O 2 ⋅ γ server ⋅ 1 f server . (7)

The energy consumption of inference on the server side is not part of our model due to the data center’s continuous power supply.

As an inference serving platform, SWEET not only tailors the model variant for the edge devices but also shares the computation workload with the devices by forwarding the second model segment on the server. The edge devices are supposed to be charged for the computation resources on the server. The cost for occupying the computation resources on the server per second is denoted by ζ. Hence, the overall cost for inferencing the second segment on the server can be defined as Equation 8:

C = O 2 ⋅ γ server ⋅ ζ f server . (8)

3.3 Model quantization

Given the prevalence of hardware limitations in resource-constrained scenarios, such as embedded systems and wearable devices, coupled with the fact that many current neural network models are heavily overparameterized (Gholami et al., 2021), the focus on training and inference with limited precision has garnered increasing attention among researchers (Zhou et al., 2016). Model quantization, which involves reducing the precision of weights and biases in machine learning models, especially deep neural networks (DNNs), has emerged as a key technique. This process is integrated into our proposed SWEET because it significantly reduces a neural network’s memory footprint, facilitating its deployment on resource-constrained devices and reducing communication latency.

Figure 3 shows the layer-wise parameter reduction achieved by implementing our proposed algorithm. Across all layers, the sizes of the parameters are substantially reduced by 62%–84%, averaging 77%, which is referred to as the reduction ratio in the figure, while the accuracy degradation remains below 1%. This figure proves the effectiveness of model quantization in inference offloading.

We specifically focus on post-training quantization, which involves quantizing a model after it has been fully trained with full precision. By adopting this approach, our proposed quantized edge inference scheme can be seamlessly integrated into most existing edge inference applications without needing to retrain deep learning models. In the subsequent parts of this section, we introduce the notations used for quantization and quantized models and activations.

For a request r, we assume that the corresponding neural network model θ consists of L learnable layers in total and that every layer can be denoted as layer l , l ∈ { 1,2 , … , L }. The trained model parameters θ are stored with double floating-point precision to lower the precision of the model’s parameters and the intermediate activation to a more concise format while ensuring that the model’s accuracy or generalization capability remains largely unaffected, we first define the uniform asymmetric quantizer as follows:

Given a real value c, which can be a weight or activation in neural networks, the quantized value using b bits c q is selected from the following quantization set Q as shown in Equation 9 below:where μ and ϕ define the range of quantization set Q, [ μ : 1 2 b − 1 : ϕ ] denotes the uniform grid on [ μ , ϕ ] with interval 1 2 b − 1 between adjacent elements, and q z is the zero point. We define Q ( ⋅ ) as the quantization function; therefore, the quantized value c q is given by Equation 10:

Q = μ : 1 2 b − 1 : ϕ + q z , (9)

c q = Q c , b , q z = arg min q ∈ Q c − q . (10)

This article adopts layer-wise quantization and model partitioning techniques to serve the inference request for edge devices. We employ the notation p ∈ 1,2 , … , L to signify the partition point within the model, which serves as a strategic division for processing under our proposed optimization framework. The first neural network segment will be returned to the edge device for local inference. The quantization bitwidth for each layer in the first neural network segment is defined by a quantization bitwidth vector, b = [ b i , b x ], with i ∈ { 1,2 , … , p }, and b x is the bitwidth of activation. This vector b is essential as it determines the quantization granularity for each layer, thereby influencing the model’s computational efficiency and overall performance.

3.4 Communication model

Our proposed system uses a standard wireless communication framework to assess transmission latency. Once the model has been quantized and partitioned for the inference request on the server, the quantized model segment will be transmitted to the edge device through a wireless channel. In addition, the intermediate output of the model segment will be transmitted from the device to the server after local inference is completed. Assuming that the transmission time is comparable to the channel coherence period (Li X. et al., 2022), the channel gain of the wireless link from the mobile device to the server can be defined as shown in Equation 11:where h is the small-scale fading component, which is frequency-dependent and exponentially distributed with unit mean, and α is the large-scale fading component, which is frequency-independent, reflecting path loss and shadowing. Let π denote the transmit power of the device. We have the received SNR shown in Equation 12:

g = α ⋅ h , (11)

β k = π ⋅ g σ . (12)

Therefore, with a given channel bandwidth B, the channel capacity (Shannon, 1948) can be formulated using Equation 13:

r = B ⋅ l o g 2 1 + π ⋅ g σ . (13)

We can easily have the data size (in bits) of the quantized model parameters and activations to be transmitted, which make up most of the communication payload, as shown in Equation 14 below:where Z x and Z w are the activation and model parameter communication payload, respectively. Finally, we have the transmission latency T tran and energy consumption E tran of the inference request r n t expressed in Equation 15 and Equation 16:

Z = Z x + Z w = b x ⋅ z x + ∑ l = 1 p b l ⋅ z l , (14)

T tran = Z r , (15)

E tran = π ⋅ Z r . (16)

4 Problem formulation and proposed algorithm

In the inference serving scenario we consider, each edge device generates inference requests, which will be adaptively processed on the edge server and device according to the environmental conditions. Our primary objective is to develop a model quantization and computation sharing policy that minimizes inference latency, device power consumption, and service cost while ensuring the accuracy of each inference. Specifically, the optimization problem we aim to solve can be formulated as follows:where T queue is the task queueing time on the server and ω , τ , η ≥ 0 are the significance weights for time consumption, energy consumption, and cost, respectively. In our implementation, T queue is measured per request by the backend: the server logs the time between request arrival and start of service and exposes a moving average to the optimizer. This empirical definition lets T queue act as a real-time workload indicator that naturally captures burstiness without additional modeling assumptions. The weights offer the flexibility for each mobile device to tailor its preferences, striking a balance between performance and cost. Furthermore, by letting some weights be zero, the objective can be altered toward different expectations. For example, should a mobile device n exhibit a heightened sensitivity to inference latency coupled with a long battery life, increasing the value of ω would be beneficial.

m i n b , p ω ⋅ T local + T tran + T queue + T server + τ ⋅ E local + E tran + η ⋅ C s . t . a c c θ − a c c θ ′ ≤ a , (17)

Inspired by Zhou et al. (2018), we first define quantization noise and adversarial noise to quantify the impact of accuracy requirement a n t on the objective function. Specifically, we define σ l w and σ p x as the quantization noises on the last activation that are caused by the quantization of the lth layer’s weights and activation, respectively. They can be modeled as follows (Zhou et al., 2018):

‖ σ l w ‖ 2 2 = s l ⋅ e − l n 4 b l , (18)

‖ σ p x ‖ 2 2 = s p ⋅ e − l n 4 b p . (19)

Based on the noises we defined above, two measurements estimating the accuracy degradation caused by quantization of weights and activation in layer l can be defined (Zhou et al., 2018), as shown in Equation 20 and Equation 21: where ρ l ( a n t ) is the robustness parameter of layer l, which can be calculated with the given dataset D using Equation 22; (Zhou et al., 2018):where σ * is the adversarial noise on the last layer’s activation, which is the minimum noise that can change the classifying result of the neural network. As the measurement of the degradation of the accuracy caused by the quantification in layers, ψ l w = ψ l ′ w indicates that quantification at layers l and l ′ affects the accuracy of the model equally. Given that the linearity and additivity of this metric have already been established and considering the independence of these measurements across all layers, we can reformulate the optimization problem originally presented in Equation 17 into a new form by substituting terms in Equation 17 and adding up the accuracy degradation estimations across all layers, as shown in Equation 23:Here, T queue is maintained online as a runtime workload indicator and is included in the objective value used during the online serving stage. Δ is a constant related to model accuracy degradation a. Specifically, a higher Δ means a higher a. Additionally, local cost ξ, server cost δ, and transmission cost ϵ are defined for simplification, and they can be detailed as shown in Equations 24-26 below:

ψ l w = ‖ σ l w ‖ 2 2 ρ l a , (20)

ψ l x = ‖ σ l x ‖ 2 2 ρ l a . (21)

ρ l a = m e a n σ l w , σ l x m e a n σ * = 1 2 | D | ⋅ ∑ x ∈ D ‖ σ l w ‖ 2 2 + ‖ σ l x ‖ 2 2 1 | D | ∑ x ∈ D ‖ σ * ‖ 2 2 , (22)

m i n b , p ξ ⋅ O 1 + δ ⋅ O 2 + ϵ ⋅ b x ⋅ z x + ∑ l = 1 p b l ⋅ z l + ω T queue s . t . ψ p x + ∑ l = p L ψ l w ≤ Δ . (23)

ξ = ω ⋅ γ local f local + τ γ local κ n f local 2 , (24)

δ = ω + η ζ ⋅ γ server f server + τ γ server η m f server 2 , (25)

ϵ = ω + π n τ r m n . (26)

We solve the optimization problem in Equation 23 by solving the quantization bitwidth b for a fixed p and then enumerating p ∈ { 1,2 , ⋯ L } and selecting the solution with the minimum objective function value. By doing so, the quantization bitwidth can be solved with a closed-form solution, based on which we further propose a two-stage algorithm, distributing the algorithmic overhead for the real-time optimization.

With fixed partition point p, the optimal value of quantization bitwidth b and partition point p can be derived when

z 1 ρ 1 s 1 e − l n 4 b 1 = z 2 ρ 2 s 2 e − l n 4 b 2 = ⋯ = z p ρ p s p e − l n 4 b p = z x ρ x s x e − l n 4 b x . (27)

The proof of Equation 27 can be found in Supplementary Appendix S1.

From the solution in Equation 27, we can observe that for each possible partition point p ∈ { 1,2 , … , L }, layer-wise quantization bitwidth is solved by iteratively calculating the adversarial noise σ * ̄ and robustness parameter ρ l from the first layer to the last layer of the first model segment. Multiple forwarding operations are involved in this process of calculating adversarial noise, which renders it impractical for real-time deployment. This is a common challenge in many optimization problems, particularly in systems with strict real-time constraints. To address this, many works in the literature resort to approximate or heuristic-based methods that trade off some degree of optimality for practical applicability. In this case, we detach the model quantization from the runtime and design the inference serving strategy SWEET with the two following algorithms, namely, an offline quantization algorithm and an online serving algorithm.

• Offline quantization algorithm: The optimal layer-wise quantization bitwidth is solved by the offline quantization algorithm, and it is pre-computed under various system conditions. By optimizing the layer-wise quantization offline, most of the calculation overhead can be handled in advance, and the inference serving system can hence respond to the requests in real-time.
• Online serving algorithm: With quantization patterns being computed by offline algorithms, the online serving algorithm periodically optimizes quantization patterns and partition points for the inference requests within a certain time window, balancing between optimization overhead and the adaptivity to dynamic environments.

4.1 Offline quantization algorithm

The offline quantization algorithm is shown in Algorithm 1, which takes a parameterized model θ as input and generates a set of model variants for all possible partition points and several different accuracy levels. Following the convention of recent accuracy-scaling systems such as INFaaS (Romero et al., 2021), Proteus (Ahmad et al., 2024a), and Loki (Ahmad et al., 2024b), we select five representative QoS tiers for model quantization during the offline stage. Specifically, the proposed SWEET enumerates the partition point from the first layer to the last layer p ∈ { 1,2 , … , L }, with five different accuracy degradation requirements a ∈ { a 1 , a 2 , a 3 , a 4 , a 5 }, and optimizes the quantization bitwidth b a p accordingly. Then, the set of quantization patterns { ( b a p , p ) } θ will be stored for the online serving algorithm in runtime.

Algorithm 1 Offline Model Quantization Algorithm for Inference Offloading.

For each model quantization pattern b a p with given partition point p and accuracy degradation requirement a, to optimize the layer-wise quantization bitwidth, we start from the layer p. The bitwidth for the layer at the partition point can be directly derived from Equation 27. Subsequently, the bitwidth for the preceding layer, b l , l ∈ p − 1 , p − 2 , … , 1, is solved sequentially from layer p − 1 to layer 1 by substituting the parameters of the previously calculated layer b l + 1, ρ l + 1, s l + 1, and z l + 1 and the current layer ρ l, s l, and z l with actual values in Equation 27.

4.2 Online serving algorithm

In the online serving stage, the SWEET periodically optimizes the inference requests by searching for the optimal quantization and partitioning pattern ( b a * p * , p * ) θ within the set { ( b a p , p ) } θ, which meets the requirement of the accuracy degradation, and then quantizing the pre-trained model θ, shown in Algorithm 2.

Algorithm 2 Online Inference Serving Algorithm for Edge Devices.

5 Simulations and numerical results

In this section, we present the numerical results obtained from our simulation platform that validate the effectiveness of our proposed model quantization and partitioning algorithm for inference offloading. Specifically, we first use a compact six-layer DNN-based classifier as a controlled case study to expose the core behavior of SWEET under joint quantization and partitioning, including the trade-off among local computation, transmission payload, server execution, and queueing effects. This DNN comprises six fully connected layers, as illustrated in Figure 4. To gain a more comprehensive perspective on compression-based inference offloading schemes and thoroughly examine the efficacy of our proposed algorithm SWEET in the context of compression-based inference offloading, we have performed comparative tests with autoencoder-based offloading (Yao et al., 2020), model-pruning-based offloading (Ren et al., 2023; Shi et al., 2019), and direct offloading without applying any optimization methods. Additionally, we performed extensive evaluations of SWEET across multiple datasets, including MNIST (LeCun et al., 1998), SVHN (Netzer et al., 2011), CIFAR-10 and CIFAR-100 (Krizhevsky, 2009), and ImageNet (Deng et al., 2009), as well as various models such as the ResNet family (He et al., 2016), including ResNet-18, ResNet-34, and ResNet-50, to demonstrate its adaptability and robustness on more realistic CNN-based workloads. Finally, we evaluated SWEET under real-world trace conditions to verify its effectiveness in proactively balancing the workload between the edge device and server and to analyze the impact of the configuration update window size on overall system performance. Unless otherwise stated, each plotted result is averaged over five independent simulation runs per configuration with independent random seeds. We report mean values, and the observed trends remain consistent across repeats.

We set up the simulator with a Python script comprising three components: an executing module, a communication module, and a performance module. The executing module simulates the processing behaviors of edge devices and the server according to the parameters shown in Table 2 and executes the two model segments with different processing profiles. The communication module simulates the transmission of the model parameter and intermediate output between the edge device and the server based on the wireless communication model defined in Section 3. The performance module collects the performance metrics from the other two modules, such as time and energy consumption. Specific simulation parameters are summarized in Table 2:

TABLE 2. Simulation settings.

First, we enumerate the model partition point and calculate the quantization bitwidth for subsequent layers. As the partition point is incrementally shifted from the first layer toward the final layer, we plot the total time consumption, energy consumption, and server cost, which are shown in Figure 5.

Analysis of Figure 5 reveals a trend: As the partition point moves closer to the last layer, the time consumption and power consumption increase, while the server cost decreases. The reason for the increase in time consumption and energy consumption along the partition point moving toward the output layer is that more model parameters would be counted into the communication payload from the SWEET platform to the device. Meanwhile, the server cost is decreasing because the shared computation workload on the server decreases, while more calculations are done in the device. Therefore, a trade-off in computation workload sharing becomes evident: attempting to save time and energy by allocating more computation to the server increases server cost. Specifically, we compared the layer-wise performance metrics under two strategies, the SWEET and the inference service without optimization, which means the model segments are transmitted from the server to the devices without an intact, initial pre-trained model. From Figure 5, we can determine the extent to which SWEET reduces the power and time consumption significantly in all possible partition layers.

By appropriately quantizing the model parameter, the time consumption and power consumption are substantially reduced; however, the reduced bitwidth incurs a trade-off in accuracy. According to Equation 18 and Equation 19, the quantization noise will increase, while the quantization bitwidth decreases, which will degrade the inference accuracy. Figure 6 illustrates this trade-off between accuracy and model parameter size by establishing various accuracy degradation thresholds and testing the optimized parameter size. This relationship is discerned by aggregating the sizes of the parameters across all layers. It is evident from the figure that the overall model parameter size can decrease exponentially as the required accuracy level is lowered.

Figures 7–9 compare the proposed algorithm SWEET with autoencoder-based (Yao et al., 2020) and model pruning-based (Ren et al., 2023; Shi et al., 2019) inference task offloading algorithms. To ensure a fair comparison, all baselines use the same backbone model, dataset, partition-point sweep, and device/server/network parameters as SWEET. For the autoencoder baseline, an encoder–decoder pair is inserted at the split point, and the bottleneck dimension is tuned separately for each partition point so that the resulting accuracy degradation remains comparable to that of SWEET under the same target accuracy requirement. For the pruning baseline, pruning is applied only to the layers offloaded to the edge device, and the pruning threshold is likewise tuned for each partition point to satisfy the same accuracy budget. Figure 7 illustrates the objective function values of the four offloading schemes under these matched conditions, varying according to the partition point. The SWEET algorithm achieves the lowest overall cost. Notably, the autoencoder-based approach tends to incur a significantly higher overall cost than the other three schemes, a finding further substantiated by the subsequent figures.

Figures 8, 9 provide detailed breakdowns of the time and energy consumption for the four offloading schemes. It is readily apparent that the SWEET algorithm consistently exhibits the lowest energy and time consumption, regardless of the partition point. Due to the incorporation of additional encoding and decoding layers, the autoencoder-based method incurs greater time consumption and, consequently, higher energy consumption than the other three schemes.

Table 3 presents the inference accuracy associated with the above four offloading schemes. We evaluated the model’s accuracy using the MNIST test dataset, applying four distinct offloading strategies. In scenarios where no optimization algorithm is incorporated into the offloading scheme, the inference accuracy remains consistent with that of the unmodified neural network. The autoencoder-based method tends to demonstrate improved prediction accuracy when more layers are processed locally. This trend can be attributed to the layers closer to the input being more susceptible to the recovery noise introduced by the autoencoder. The proposed SWEET algorithm maintains accuracy successfully and surpasses the other two compression-based methods in performance.

TABLE 3. Accuracy of algorithms at different partition points.

We deployed deep learning models on the SVHN (Netzer et al., 2011), CIFAR10 and CIFAR100 (Krizhevsky, 2009), and ImageNet (Deng et al., 2009) datasets to validate our proposed framework across a broader range of datasets and baseline neural networks. In addition, we implemented ResNet (He et al., 2016) architectures, including ResNet18, ResNet34, and ResNet50, on the ImageNet dataset (Deng et al., 2009) to further benchmark our framework. Table 4 illustrates the efficacy of our proposed SWEET when applied across these datasets and models. Notably, we observed a significant reduction in communication payload, calculated by dividing the optimized parameter size by the initial parameter size, ranging from 11.88% to 18.12%, with negligible accuracy degradation, ranging from 0.08% to 0.66%. These results underscore the versatility and efficiency of our proposed model.

TABLE 4. Accuracy on baseline models and datasets.

Figure 10 reports the measured average end-to-end inference latency and the normalized partition point as functions of the number of concurrent edge devices N for four DNN architectures, 6-FC, ResNet-18, ResNet-34, and ResNet-50, on a physical edge–server testbed. In this setup, the edge side is represented by a Raspberry Pi 5, and the server side is a shared GPU server with an Nvidia L40S GPU. For each model, we first profile the real device-side execution latency and the per-partition transmission cost on the Raspberry Pi 5 and then use these measurements to drive concurrent end-to-end request generation against the live server. For each value of N, the workload spans 500 request generation slots with a per-device Poisson arrival rate of λ = 1.0 requests/slot, and the reported results are averaged over five independent runs.

In each subfigure, the left y-axis shows the mean latency under four partitioning strategies, namely, SWEET adaptive, edge-only, server-only, and fixed-midpoint, while the right y-axis (purple curve) shows the normalized partition point p * / L selected by SWEET. At low concurrency levels, SWEET routes all computation to the GPU server ( p * / L ≈ 0 ), achieving latencies of 29.5 ms, 45.4 ms, 60.4 ms, and 58.2 ms for the four models, respectively, corresponding to a 7.6 × to 5.5 × reduction compared to edge-only execution. As N increases beyond the server’s processing capacity (N ≳ 33,28,20,21 for the four models), the server-only strategy exhibits unbounded queue growth, with latency exceeding 1 s. In contrast, SWEET progressively shifts computation to the edge. For example, at N = 40, the normalized partition point increases to 0.47–0.55 for the ResNet variants, while the adaptive latency remains well below the edge-only baseline. Across all four models and over the full range of N, SWEET consistently achieves the lowest average latency among all strategies, demonstrating that the queue-length-aware partition optimizer effectively balances edge-side computation cost against server-side queueing delay across different model complexities.

To evaluate SWEET under realistic workload conditions, we conduct a trace-driven simulation using the Twitter-trace dataset (Archive Team, 2021). This dataset captures real-time tweet activity, reflecting fine-grained, high-variance request patterns. We use the volume of tweets over time to simulate inference request arrival rates, making it a suitable proxy for dynamic workloads in machine learning inference serving environments. Furthermore, we implement model switching following the same design philosophy proposed by Zhang et al. (2020), where the system adapts to fluctuating workloads by dynamically selecting among model variants with different latency-accuracy trade-offs. In our setup, we adopt the ResNet family (e.g., ResNet-18, ResNet-34, and ResNet-50) (He et al., 2016) as candidate variants, allowing the system to switch models at runtime based on current load conditions without scaling up computational resources. As shown in Figure 11, SWEET maintains stable latency across different workload levels. Specifically, under lighter workloads, SWEET exhibits slightly higher but still acceptable latency compared to model switching, a fully centralized inference system. This is because SWEET forwards more layers to the edge in light workload scenarios, leading to an expected increase in latency. However, under heavier workloads, SWEET effectively maintains stable performance by proactively partitioning and scheduling inference across edge and cloud resources. We evaluated SWEET under different accuracy degradation thresholds, specifically 1% and 2%. The results show that with more permissible accuracy degradation, SWEET can reduce latency further by applying lower-bitwidth model quantization.

To further study the effect of decision window size on latency, we simulate an inference system using the Twitter-trace dataset (Archive Team, 2021) to model request arrival rates. As shown in Figure 12, smaller windows achieve lower latency by adapting quickly to workload changes, albeit with higher algorithmic overhead, while larger windows reduce algorithmic overhead at the cost of higher delay due to outdated partitioning decisions. Our results show that using a moderate window size (e.g., 5–10 s) achieves near-optimal latency, suggesting that infrequent updates can offer a good balance between efficiency and responsiveness.

6 Conclusion

In this article, we have proposed SWEET, a neural network inference serving system, emphasizing the development of a flexible, accuracy-aware, and retraining-free system tailored for different edge devices. This was achieved by integrating model quantization and model partitioning. We have modeled the time and energy consumption associated with local inference, data transmission, and remote inference. Additionally, our approach includes an accuracy degradation measurement within the joint optimization problem, allowing us to identify the optimal quantization bitwidth and partition point while considering accuracy requirements. Our simulation results demonstrate a significant reduction in latency and energy consumption while meeting the strict accuracy degradation, underscoring the effectiveness of our proposed SWEET algorithm. Looking ahead, future work will explore global scheduling and allocation strategies. We aim to propose a comprehensive hierarchical optimization framework for inference offloading in mobile edge computing systems, which we anticipate will further enhance the system’s overall efficiency and effectiveness in managing complex computational tasks.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcpxs.2026.1801157/full#supplementary-material

Supplementary material Supplementary material at the source

Footnotes

Edited by: Chandra Krintz, University of California, Santa Barbara, United States

Reviewed by: Anurag Dutt, Stony Brook University, United States

Hesheng Sun, Nanjing University, China