Fairness-aware resource allocation for FSO/RF non-terrestrial networks

1 Introduction

The exponential growth of wireless devices and applications in 5G and beyond, together with the need for ubiquitous connectivity, has highlighted the importance of NTNs. NTNs provide wide-area coverage and enable connectivity in remote and underserved areas, such as inaccessible mountains, huge desert regions, dense forests and vast oceans (Liu et al., 2023), as well as support communication in emergency scenarios where terrestrial infrastructure is unavailable (Wondmagegn et al., 2025).

NTNs consist of multiple layers, including space, air, and ground segments, enabling three-dimensional network coverage. The space segment comprises satellites operating in different orbits, with Low Earth Orbit (LEO) satellites being particularly attractive due to their low orbital altitude, which enables reduced propagation delay, lower path loss, and high-capacity transmission capabilities (Bakhsh et al., 2025). As a result, LEO satellites can provide low-latency broadband services over wide areas while supporting connectivity in both remote regions and densely populated environments (Seong et al., 2025b).

Despite these advantages, LEO satellite systems are affected by large transmission distances, which can degrade link quality and limit direct communication with ground users. To address this issue, High-Altitude Platform Stations (HAPs) are introduced as intermediate aerial nodes in NTN architectures. HAPs are aerial platforms operating in the lower stratosphere at altitudes of approximately 17–25 km (Shah S. et al., 2021). By integrating LEO satellites with HAPs, the communication distance can be effectively reduced, leading to improved link quality, enhanced transmission capacity, and wider coverage. In addition, HAPs can provide direct wireless access to terrestrial users and act as reliable relay nodes between space and ground segments (Mashiko et al., 2025).

The limited spectral and power resources in LEO–HAP–terrestrial user networks make efficient resource management a critical challenge (Guo et al., 2024). In this context, multiple access techniques play a crucial role in enhancing transmission capacity by enabling efficient sharing of limited resources among multiple users. Conventional Orthogonal Multiple Access (OMA) schemes, such as Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), and Orthogonal Frequency Division Multiple Access (OFDMA), allocate resources orthogonally in time, frequency, or code domains, allowing low-complexity signal separation at the receiver (Shah et al., 2021a). However, OMA is fundamentally limited by the finite number of orthogonal resources, and its performance degrades under channel impairments such as fading and synchronization errors, making it unsuitable for massive connectivity scenarios (Kebede et al., 2022).

To overcome these limitations, Non-Orthogonal Multiple Access (NOMA) has been introduced, allowing multiple users to share the same time–frequency resources and improving spectral efficiency (Dai et al., 2018). In particular, power-domain NOMA (PD-NOMA) employs Successive Interference Cancellation (SIC) to separate users, but its performance degrades in large-scale systems due to increased interference and imperfect SIC (Hassan M. et al., 2025; Moltafet et al., 2019). To address these issues, code-domain NOMA techniques such as SCMA have been proposed. SCMA enables overloaded transmission, where the number of users exceeds the available orthogonal resources, while maintaining manageable receiver complexity (Bai et al., 2025). More recently, RSMA has emerged as a flexible multiple access framework that partially decodes interference, providing improved robustness and spectral efficiency under imperfect channel conditions. In addition, RSMA effectively supports a large number of users, making it a promising candidate for next-generation NTN systems (Seong et al., 2025a). However, their performance under fairness-aware power allocation in hybrid RF/FSO NTN systems remains insufficiently explored. In this context, efficient resource allocation, particularly power allocation, plays a key role in balancing system performance and user fairness under heterogeneous channel conditions (Minani et al., 2026).

To better position the adopted RF channel model within the broader fading-modelling literature, recent studies have proposed several generalized and composite fading distributions for complex and non-homogeneous wireless environments. The λ–κ–μ fading distribution captures multipath clustering, line-of-sight components, and correlation between in-phase and quadrature signal components (Wang H. et al., 2024). This model has also been extended to cascaded relaying scenarios through the double λ–κ–μ distribution (Liu et al., 2025). In addition, the λ–κ–μ shadowed distribution incorporates shadowing effects with multipath clusters, line-of-sight components, and I/Q correlation (Wang et al., 2026). Similarly, the α–η–μ/IGA distribution captures nonlinear multipath propagation with inverse-Gamma shadowing (Xu et al., 2026). Although these models provide flexible channel characterization, they introduce additional analytical and computational complexity. For RF access links, the Nakagami-m model remains a tractable and widely used representation of weak, moderate, and strong fading through a single fading-severity parameter (Gugliani et al., 2024).

2 Related work

Several studies have investigated multiple access techniques and resource allocation strategies in non-terrestrial networks, including systems employing radio frequency (RF) and free-space optical (FSO) links.

2.1 Power-domain NOMA-based resource allocation in NTNs

PD-NOMA-based resource allocation in NTNs has been extensively studied to improve spectral efficiency and support massive connectivity. A downlink land mobile satellite system was studied in (Yan et al., 2018), where PD-NOMA was applied with power allocation designed to maximize sum rate under user rate constraints. Analytical expressions for capacity, energy efficiency, and outage probability were derived. The study is limited to a two-user scenario without addressing scalable multi-user resource allocation. In multibeam LEO satellite systems, a PD-NOMA-based beam hopping and frequency allocation scheme was developed in (Zheng et al., 2023), where a joint resource allocation framework combining user pairing and frequency allocation was proposed. The approach is designed for interference mitigation and beam-level optimization, with limited consideration of system-wide resource allocation across heterogeneous NTN architectures. A PD-NOMA-enabled integrated satellite–HAP–terrestrial network was studied in (Guo et al., 2024), where a quality-of-service (QoS) based power allocation scheme was developed to mitigate channel estimation errors and imperfect SIC.

Resource allocation in integrated satellite–terrestrial networks has also been investigated under PD-NOMA frameworks. In (Xu et al., 2022), an uplink PD-NOMA-based model was considered, where subchannel and power allocation were jointly optimized to minimize latency and energy consumption. In (Xin et al., 2024), user association, power allocation, and beamforming were jointly optimized using a two-stage iterative approach to maximize the system sum rate, with QoS incorporated as a constraint rather than an explicit fairness objective. In (Wang et al., 2021), a cooperative space–terrestrial satellite network was studied, where user association, pairing, and power allocation were jointly optimized using convex optimization techniques for energy-efficient operation.

In addition, fairness-aware power allocation for PD-NOMA-based multibeam LEO satellite networks was studied in (Biyoghe and Balyan, 2025), where the max–min fairness problem was decomposed into intra- and inter-beam subproblems and solved using Offered-Capacity to Traffic-Request (OCTR) ratio convergence optimization techniques. The model is restricted to two users per beam, which limits its scalability in dense networks scenarios. In (Hojeij et al., 2017), a downlink PD-NOMA system was studied, where a proportional fairness scheduler was combined with a water-filling–based power allocation scheme for joint resource allocation across users and subbands. The approach is tailored to terrestrial cellular systems and does not consider NTN-specific characteristics.

Existing PD-NOMA-based approaches in NTNs primarily focus on optimizing system performance metrics such as sum rate, energy efficiency, and latency through joint resource allocation strategies. Although fairness is considered in some works using rate-based criteria such as max–min or proportional fairness, it is generally defined in terms of rate balancing rather than user connectivity.

2.2 SCMA-based resource allocation in NTNs

Unlike power-domain NOMA, SCMA employs sparse codebooks to enable user multiplexing without relying on strict power ordering. This provides greater flexibility in interference management and improves robustness under heterogeneous channel conditions commonly encountered in NTNs.

SCMA-based resource allocation has been explored to enhance system capacity in various NTN scenarios. A multi-gateway satellite system was studied in (Gao et al., 2025), where a three-stage optimization framework jointly optimized factor graph design, power allocation, and onboard processing to maximize the system sum rate, demonstrating higher spectral efficiency and improved user support compared to OFDM and PD-NOMA schemes. An SCMA-assisted UAV communication system was investigated in (Chaturvedi et al., 2024), where UAV deployment, trajectory design, subcarrier assignment, and power allocation were jointly optimized to maximize the system sum rate under energy constraints. An SCMA-based LEO–UAV communication framework was studied in (Chaturvedi et al., 2026), where UAV deployment, user association, subchannel assignment, power allocation, and bandwidth allocation were jointly optimized through a decomposition-based approach to maximize the system sum rate. In (Ferdouse et al., 2020), resource allocation in SCMA-based downlink cloud radio access networks was investigated, where a GWF-based power allocation scheme combined with NOMA-SIC was employed for efficient transmit power distribution across subcarriers. These approaches primarily focus on enhancing system capacity and spectral efficiency through joint optimization frameworks, with limited consideration of fairness in resource allocation.

2.3 RSMA-based resource allocation in NTNs

RSMA has emerged as a flexible framework for improving spectral efficiency and robustness in NTN systems, enabling more efficient handling of interference under diverse channel conditions. Accordingly, RSMA-based resource allocation has been investigated in various NTN scenarios. In (Liu et al., 2023), an RSMA-based beamforming framework was studied for multibeam GEO satellite systems, where energy efficiency was maximized using a robust optimization approach based on successive convex approximation, demonstrating improved robustness to channel uncertainty and higher efficiency compared to SDMA and WMMSE-based schemes. In (Cui et al., 2023), energy-efficient resource allocation under payload constraints was investigated, where power and rate allocation were jointly optimized to minimize unmet system capacity and energy consumption.

In LEO satellite systems, RSMA has been extended to address channel uncertainty and heterogeneous traffic demands. In (Seong et al., 2025a), an RSMA-based framework incorporating space–time coding was proposed, where power allocation and beamforming were jointly optimized based on a max–min fairness criterion to improve robustness under imperfect CSI. In (Seong et al., 2025b), traffic-aware RSMA optimization was investigated to reduce rate mismatch under heterogeneous traffic demands through joint power allocation.

RSMA has also been applied in aerial NTN scenarios. In (Truong et al., 2022), RSMA was integrated into HAP-assisted mobile edge computing systems, where joint optimization of task offloading and power allocation reduced latency and energy consumption. In (Dutta et al., 2024), resource allocation for cooperative multi-HAP networks was studied, where asynchronous non-orthogonal transmission with joint user association and power allocation improved system performance.

In cooperative NTN architectures, RSMA has been applied for joint broadcast and unicast transmission in (Han et al., 2024), achieving improved rate balancing among users. Furthermore, RSMA-based coordinated beamforming was investigated in (Yin and Clerckx, 2023), where a max–min fairness criterion enhanced robustness under channel uncertainty. In (Liu et al., 2024), RSMA-based integrated sensing and communication (ISAC) optimization was studied, where a max–min energy efficiency framework improved system performance compared to SDMA. Despite these advances, existing RSMA-based approaches primarily focus on improving energy efficiency, robustness, or rate-based fairness. Although fairness is considered in several works through criteria such as max–min rate optimization, it is typically defined in terms of rate balancing rather than explicitly addressing user connectivity under heterogeneous channel conditions.

2.4 Resource allocation in hybrid RF/FSO NTN systems

In addition to multiple access techniques, hybrid RF/FSO architectures have been investigated to exploit the complementary advantages of both technologies in NTN systems. In (Dang et al., 2024), resource allocation for UAV-mounted base stations backhauled by LEO satellites using hybrid RF/FSO links was studied, where dynamic rate adaptation and frame allocation significantly improved throughput and outage performance compared to FSO-only schemes. A HAPS-assisted dual-hop hybrid RF/FSO communication system was studied in (Deka et al., 2022), where transmit power at the source ground station and HAPS was jointly optimized under energy constraints. The proposed power allocation strategy minimized outage probability and improved BER performance compared to equal power allocation schemes. Similarly, in (Li et al., 2025), a hybrid FSO/RF satellite-UAV-terrestrial network was studied, where modulation and coding scheme selection, power allocation, and UAV trajectory were jointly optimized to maximize system throughput while ensuring transmission reliability under imperfect CSI and random fading.

Resource allocation with fairness considerations has also been explored in hybrid RF/FSO NTN systems. In (Vu et al., 2026), a mixed RF/FSO downlink NTN system was investigated, where UAV-BS placement, beamforming, power allocation, and power-splitting were jointly optimized to minimize transmitted power and maximize fairness under QoS and energy constraints. The iterative optimization framework improved minimum user rate and energy efficiency. Likewise, a hybrid FSO/RF multiuser UAV communication system was studied in (Zhu et al., 2019), where channel assignment, channel matching, and power allocation were jointly optimized to provide proportional fairness and improve transmission capacity under user rate requirements. The proposed heuristic resource allocation algorithm achieved near-optimal throughput while improving system efficiency under low transmit power conditions.

Existing hybrid RF/FSO NTN studies mainly focus on throughput enhancement, outage reduction, energy efficiency, and reliability through adaptive resource allocation across backhaul and access links. Although fairness is considered in some works through proportional fairness or minimum-rate optimization, these approaches primarily address rate distribution and transmission efficiency rather than fairness-aware connectivity provisioning across heterogeneous NTN users.

In summary, state-of-the-art resource allocation in NTNs has evolved from PD-NOMA-based power distribution to more advanced frameworks utilizing SCMA and RSMA to enhance spectral efficiency and robustness. Existing research primarily focuses on optimizing system-level metrics such as sum-rate maximization, energy efficiency, and latency reduction through joint optimization of power, codebooks, and beamforming. While fairness is often incorporated via max-min or proportional fairness criteria, these approaches are generally defined by rate balancing among active users rather than ensuring reliable connectivity for all users. A critical research gap remains in addressing the capacity–connectivity trade-off in hybrid FSO/RF architectures, where heterogeneous channel impairments such as atmospheric turbulence in FSO links and fading in RF links often lead to the disconnection of users with poor channel conditions under conventional allocation strategies. There is a clear need for a resource allocation framework that not only maximizes transmission capacity but also explicitly prioritizes fairness-aware connectivity to ensure all users meet a minimum service threshold regardless of their spatial location or environmental conditions. The following Table 1 provides a comprehensive comparison of recent literature, highlighting the specific techniques, objectives, and remaining gaps in current NTN resource allocation strategies.

TABLE 1. Comparison of state-of-the-art resource allocation schemes in NTNs.

2.5 Motivation and contributions

Despite significant progress in NTN resource allocation, existing approaches do not jointly address fairness in user connectivity with advanced multiple access frameworks in hybrid RF/FSO environments under heterogeneous channel conditions. In particular, current methods primarily focus on rate-based fairness or system-level performance metrics without explicitly addressing the trade-off between transmission capacity and user connectivity. To address this gap, this paper proposes a fairness-aware resource allocation framework for hybrid RF/FSO NTN systems that enhances transmission capacity while ensuring reliable connectivity for all users under realistic channel impairments.

The main contributions of this work are summarized as follows:

• A unified multi-layer NTN architecture is developed, integrating LEO satellite, HAP, and ground users while jointly incorporating RF and FSO communication links. Unlike existing works, such as (Dang et al., 2024; Guo et al., 2024) that considered individual layers or single-link technologies, the proposed framework enables cross-layer interaction between heterogeneous links. This provides a more realistic and comprehensive platform for analyzing next-generation NTN systems.
• A low-complexity fairness-aware geometric water-filling algorithm is developed for resource allocation in SCMA- and RSMA-based NTN systems. Unlike conventional capacity-driven water-filling approaches, the proposed method incorporates fairness into power allocation by enforcing a connectivity constraint that iteratively redistributes power to satisfy a minimum SNR threshold γ t h = 13.54 dB for all users. This shifts the optimization objective from pure sum-rate maximization toward fair user connectivity under adverse channel conditions.
• A comparative and unified analysis of SCMA and RSMA is conducted within the same NTN architecture and channel conditions. Unlike prior studies that evaluate these multiple access schemes independently, this work provides a direct and fair comparison, offering new insights into their performance trade-offs in terms of transmission capacity, fairness, and user connectivity.
• The proposed system is evaluated under practical and heterogeneous channel models, including Nakagami-m fading for RF links and Málaga turbulence for FSO links. The analysis provides practical insights into the impact of channel impairments on transmission capacity, fairness, and system reliability in NTN environments.

The rest of this paper is organized as follows. Section 3 presents the system model. Section 4 describes the multiple access schemes. Section 5 formulates the resource allocation problem and introduces the proposed FAGWF algorithm. Section 6 presents the simulation results and discussion. Finally, Section 7 concludes the paper.

3 System model

3.1 NTN architecture

The considered system adopts an NTN architecture consisting of a LEO satellite, a HAP, and ground users. LEO satellites offer significantly lower propagation delay and reduced path loss compared to Medium Earth Orbit (MEO) and Geostationary Earth Orbit (GEO) satellites (Yue et al., 2023). HAP serves as an intermediate relay layer due to its advantageous operating characteristics, including stable stratospheric conditions, wide-area coverage, lower latency compared to direct satellite links, and quasi-stationary positioning compared to low-altitude UAV platforms (Shah S. et al., 2021). Ground users are distributed within the service area and communicate only with the HAP.

The considered deployment scenario targets remote and hard-to-reach areas where conventional terrestrial communication infrastructure is unavailable or impractical due to geographical, economic, or environmental constraints. Ground users are randomly distributed in three-dimensional space with varying altitudes to represent realistic terrain elevation in mountainous or uneven landscapes. The HAP is positioned above the center of the coverage area. The distance between the HAP and each user is calculated using the Euclidean distance model as D = Δ x 2 + Δ y 2 + Δ z 2. Here Δx, Δy, and Δz denote the differences in the (x, y, z) coordinates between the HAP and the ground user.

For efficient resource allocation, users are grouped into clusters according to their distance from the HAP, since distance strongly affects channel quality primarily through path loss. After calculating the Euclidean distance between the HAP and each ground user, users are sorted in ascending order based on their distance values. Every six consecutive users are then assigned to one cluster; thus, each cluster contains users with relatively similar HAP–user propagation distances. A cluster size of six users is selected as a practical design choice to maintain manageable resource allocation complexity for both SCMA and RSMA under the considered system model. The same cluster structure is applied to both schemes to ensure a fair performance comparison. The network architecture considered in this work is illustrated in Figure 1.

3.2 Channel models

A hybrid communication scheme is adopted based on channel characteristics. The LEO–HAP link is implemented using FSO communication to exploit its high bandwidth and capacity under relatively stable atmospheric conditions at high altitude, where turbulence effects are relatively weak (Elamassie and Uysal, 2023). In contrast, RF links are used between the HAP and ground users due to their robustness to weather conditions, support for non-line-of-sight transmission, and suitability for multi-user access (Hassan H. et al., 2025). Perfect channel state information (CSI) is assumed at the transmitter for resource allocation and power allocation in both SCMA and RSMA schemes. The simulation parameters and assumptions used in this study are summarized in Table 2.

TABLE 2. Simulation parameters and assumptions.

3.2.1 RF link

The RF channel between the HAP and ground users is modeled using a composite fading model that includes small-scale Nakagami-m and large-scale path loss, which captures the effects of propagation conditions and distance-dependent attenuation.

3.2.1.1 Nakagami-m fading model

The Nakagami-m distribution is used due to its effectiveness in modeling the fading conditions ranging from no fading or weak fading to moderate and strong fading (Gugliani et al., 2024). For Nakagami-m distribution fading with signal amplitude r ≥ 0, the probability density function (PDF) is defined in Equation 1 as (Benamira et al., 2024):where Γ . is the gamma function, Ω = Ε r 2 is average fading power and m is the fading severity parameter. The parameter m controls channel fading severity, where lower values correspond to stronger fading and higher values indicate weaker fading. To evaluate system robustness under heterogeneous RF channel conditions, three representative fading scenarios are considered: strong fading with m = 0.5, moderate fading with m = 4, and weak fading with m = 8 (Pajala et al., 2006).

f R r = 2 m m r m − 1 Γ m Ω m e − m Ω r 2 (1)

3.2.1.2 Path loss model

The model captures the combined effects of free-space propagation and additional losses due to shadowing under Line-of-Sight (LoS) conditions, which are commonly assumed for HAP-to-ground communications. According to (Popoola et al., 2020; Wang Z. et al., 2024), the path loss model between the HAP and the ground user is expressed in Equation 2 as:where F S P L is the free space path loss given by F S P L = 20 ⁡ log 10 4 π D f c C. Here D m is the distance between the HAP and the ground users, f c H z is the frequency of the carrier, and C = 3 × 10 8 m / s is the speed of light. The term η L o S is the additional path loss of LoS link caused by shadow fading. It is characterized as a Gaussian random variable with zero mean and σ 2 variance (Popoola et al., 2020).

P L L o S = F S P L + η L o S , (2)

The combined effect of Nakagami-(m) fading and path-loss attenuation is represented by an effective scalar channel coefficient h c , k , u for each cluster c, subcarrier k, and user u. The corresponding channel-gain-to-noise ratio g c , k , u is then used in the SCMA and RSMA power allocation procedures.

3.2.2 FSO link

The FSO channel is modeled by considering three main factors: atmospheric turbulence I v , path loss I l , and pointing errors I p , which are essential for evaluating the reliability and performance of the optical channel. The overall channel gain can be expressed in Equation 3 as:

I = I v . I l . I p (3)

3.2.2.1 Malaga distribution model

The FSO link is modeled using Malaga distribution due to its ability to accurately represent optical turbulence conditions ranging from weak to strong regimes (Saber et al., 2024). Based on a Málaga distribution, the probability density function (PDF) of the received irradiance Iv is expressed in Equation 4 as (Arya and Chung, 2020; Chen et al., 2023):where A v = 2 α v α v 2 γ v 1 + α v 2 Γ α v γ v β v γ v β v + Ω v ′ β v + α v 2 and a v k = β v − 1 k − 1 γ v β v + Ω v ′ 1 − k 2 k − 1 ! Ω v ′ γ v k − 1 α v β v k 2.

f I v I v = A v ∑ k = 1 β v a v k I v α v + k 2 − 1 K α v − k 2 . α v β v k γ v β v + Ω v ′ (4)

The parameter α v denotes the effective number of large-scale eddies. β v is the effective number of small-scale eddies. Ω v is the average power of the deterministic line-of-sight (LoS) component. K α v − k . denotes the modified Bessel function of the second kind of order α v − k. The parameter γ v represents the average power of non-coupled scattered components and is given by γ v = 2 . b 0 1 − ρ, where b 0 is average power of the total scattered components and ρ represents the fraction of the total scattered power that is coupled to the LoS component (0 ≤ ρ ≤ 1). The effective average power term that combines LoS and scattered contributions is defined as Ω v ′ = Ω v + 2 b 0 ρ + 2 2 b 0 ρ Ω v cos ⁡ ∅ A − ∅ B, where ∅ A and ∅ B are the random phases of the LoS and scattered components, respectively.

3.2.2.2 Atmospheric attenuation

In optical communication, the atmospheric attenuation due to absorption and scattering along the propagation path can be expressed based on Beer–Lambert law in Equation 5 as follows (Yahia et al., 2022):where ψ = 8 × 10 − 3 km − 1 denotes the attenuation coefficient (Mashiko et al., 2025), and L L H is the FSO link length between LEO and HAPs.

I l = exp − ψ . L L H (5)

3.2.2.3 Pointing error

Pointing errors in the FSO link are modeled as misalignment between the transmitter and receiver, which reduces the received optical intensity. The pointing error loss is expressed in Equation 6 as (Mashiko et al., 2025):where A 0 = erf υ 2 denotes the fraction of the collected power at r = 0, r represents the misalignment distance at the receiver aperture, modeled as a random variable with zero mean and standard deviation σ s, known as the jitter standard deviation. The erf . is the error function. The parameter = a π ω d 2 , where a denotes the radius of the receiver aperture and ω d is the beam width at distance d, and ω e q is the equivalent beam waist, given by ω e q = ω d 2 π erf ⁡ υ 2 υ exp ⁡ − υ 2.

I p r ≈ A 0 ⁡ exp − 2 r 2 ω e q 2 (6)

4 Multiple access schemes

4.1 Rate splitting multiple access scheme

In this research, a 1-layer RSMA architecture is implemented to optimize the transmission capacity of the proposed system (Mao et al., 2022). The 1-layer RSMA structure provides a practical trade-off between performance and receiver complexity by bridging the gap between orthogonal and non-orthogonal multiple access schemes (Mao et al., 2018).

The total number of users U are grouped into C clusters, and RSMA is employed independently within each cluster. Each cluster contains U C = 6 users. The original message W u c intended for each user u c ∈ 1 , … . , U c is split into two parts: a common part W C o m , u c , which is combined with the common parts of other users in the same cluster to form a single joint common message W C o m = W Com , 1 , … , W Com , U c, that is decoded by all users, and a private part W P r i v , u c, which remains unique to user u c and is decoded only by that specific user (Yin and Clerckx, 2020).

The common and private messages are independently modulated using Quadrature Phase Shift Keying (QPSK). Let s C o m denote the common stream, and s P r i v , u c denote the private stream intended for user u c. The transmitted signal for each cluster can then be expressed in Equation 7 as:where P C o m represents the power allocated to the common stream and P P r i v , u c denotes the power allocated to the private stream of user u c. The total transmit power per cluster is constrained in Equation 8 as:

x = P C o m s C o m + ∑ u c = 1 U c P P r i v , u c s P r i v , u c (7)

P C o m + ∑ u c = 1 U c P P r i v , u c = P t o t a l (8)

The total transmit power P t o t a l is divided between the common and private streams according to a power splitting factor ρ ∈ 0 , 1. The power allocated to the common stream is P C o m, while the remaining power is distributed among the private streams of the users. In this work, ρ = 0.2, meaning that 80 % of the transmit power is allocated to private streams while the remaining 20 % is assigned to the common stream. The fixed 0.2/0.8 common–private power split is adopted as a baseline to maintain a controlled RSMA configuration and to enable a consistent comparison with SCMA under the considered system model. This split provides sufficient power for common-stream decoding while preserving most of the power for private-stream transmission. Accordingly, the reported RSMA results represent baseline performance under this fixed split, while adaptive common–private split optimization is left for future work.

After propagation through the channel, the received signal at user u c can be written in Equation 9 as:where h u c denotes the channel coefficient, and n u c represents additive white Gaussian noise (AWGN). Substituting Equation 7 into Equation 9, the received signal can be expressed in Equation 10 as:

y u c = h u c x + n u c , (9)

y u c = h u c P C o m s C o m + h u c ∑ j = 1 U c P P r i v , j s P r i v , j + n u c (10)

According to the adopted RSMA signal model, the common stream is decoded first by treating the private streams as interference. After the common-stream component is removed through SIC, the private stream is decoded while the remaining interference is treated as noise.

4.2 Sparse Code Multiple Access scheme

In the SCMA scheme, the total users U are grouped into clusters C and in each cluster a SCMA technique is employed. In each cluster, U C = 6 users share K C = 4 subcarriers, also referred to as resource elements (REs). Each user is assigned a predefined codebook (CB). The CB size is K c × M, where M = 2 B represents the modulation order and B denotes the number of coded bits transmitted per symbol. In this work, B = 2 corresponds to QPSK modulation. The SCMA encoder mapping is shown in Equation 11 as:where m is the symbol of the input bits m = 1 , … , M. The mapping between users and codebooks is represented by a binary matrix Q c , u , j. The value of one in Q indicates that user u in cluster c is assigned to codebook j.

x j m = C B : , m , j (11)

The mapping between subcarriers and codebooks is represented by the factor graph matrix F K C , J C. Each row matches to a single S C, and each column matches to one codebook. Each element in this binary matrix indicates whether subcarrier k C is assigned to codebook j C (Chaturvedi et al., 2022). The factor graph matrix is defined in Equation 12 as:

F 4 , 6 = 1 1 1 0 0 0 1 0 0 1 1 0 0 1 0 1 0 1 0 0 1 0 1 1 (12)

The received symbol at the receiver can be expressed in Equation 13 as:where H j = d i a g h 1 , j , . . . . , h k c , j is the effective channel matrix for the j t h user. h k c , j represents the channel fading coefficient at the k t h RE for the j t h user. n is a complex additive white Gaussian noise.

y = ∑ j = 1 6 H j x j m + n (13)

In practical SCMA receivers, multiuser detection is commonly carried out using the Message Passing Algorithm (MPA), which exploits the sparsity of the SCMA factor graph. In this work, the SCMA factor graph is used to define the user-codebook-subcarrier mapping and to evaluate power allocation, achievable rate, and energy efficiency.

5 Resource allocation optimization

5.1 Problem formulation

To optimize the transmission performance of the proposed NTN system, the resource allocation problem is formulated as a capacity maximization problem. The objective is to maximize the sum achievable rate of all users within each cluster by optimally allocating the transmit power under practical system constraints. For a given cluster c, the sum capacity maximization problem can be defined in Equation 14 as:where P c , u , k denotes the transmit power allocated to user u on subcarrier k, and R c , u represents the achievable rate. R c , u depends on the implemented multiple access scheme and is computed based on the corresponding signal-to-interference-plus-noise ratio (SINR) . The same objective function is used for SCMA and RSMA to ensure that both schemes are compared under the same NTN architecture, channel conditions, clustering method, and power allocation setting. However, the achievable rate is calculated according to the SINR expression of each scheme. In this way, the comparison uses the same evaluation criteria while keeping the different transmission characteristics of SCMA and RSMA. The comparison is therefore controlled at the system level using the same user clustering, spectral resources, channel conditions, and transmit-power constraints, while receiver-level implementation complexity is discussed qualitatively rather than used as a numerical performance metric.

max P c , u , k ∑ u = 1 U c R c , u (14)

In the SCMA system, the achievable rate is evaluated using an effective SINR expression based on the SCMA factor graph and the allocated transmit power. The SINR of user u on subcarrier k in cluster c is expressed as S I N R c , u , k S C M A = P c , u , k h c , u , k 2 ∑ u ′ = 1 u ′ ≠ u U c P c , u ′ , k h c , u , k 2 + σ 2, where h c , u , k denotes the channel coefficient between the transmitter and user u on subcarrier k and σ 2 represents the noise power. Consequently, the total achievable rate for user u under the SCMA scheme is the sum of the rates across all subcarriers, given by R c , u S C M A = ∑ k = 1 K c ⁡ log 2 1 + S I N R c , u , k S C M A.

In the RSMA system, the SINR of the common and private streams for user u on subcarrier k in cluster c are given by S I N R c , u , k C o m = P C o m h c , u , k 2 ∑ i = 1 U c P c , i , k h c , u , k 2 + σ 2, and S I N R c , u , k P r i v = P c , u , k h c , u , k 2 ∑ i = 1 i ≠ u U c P c , i , k h c , u , k 2 + σ 2 respectively, where P C o m denotes the power allocated to the common stream, and i denotes the index of interfering users in the same cluster. The common-stream rate decodable by user u in cluster c is first obtained as R c , u C o m = ∑ k = 1 K c ⁡ log 2 1 + S I N R c , u , k C o m. Since the common stream must be decoded by all users in the same cluster, the achievable common rate is limited by the weakest common stream user and is therefore defined as R c C o m = min u ϵ U c R c , u C o m. The private rate of user u in cluster c is given by R c , u P r i v = ∑ k = 1 K c ⁡ log 2 1 + S I N R c , u , k P r i v. The achievable rate of user u in cluster c is obtained by summing the assigned common rate portion and private rate component, i.e., R c , u R S M A = R c C o m U c + R c , u P r i v.

The total transmit power per cluster is limited by the available power budget. Using a unified notation for both SCMA and RSMA, the power constraint is expressed as P C o m + ∑ u = 1 U c ∑ k = 1 K c P c , u , k ≤ P t o t a l. Here, the common stream power P C o m is set to zero for SCMA. The allocated power must also satisfy the non-negativity constraint P c , u , k ≥ 0 for all users and subcarriers.

To guarantee reliable communication for all users and ensure that users with weaker channel conditions are protected from severe performance degradation, a minimum QoS requirement is imposed in the form of the SNR constraint in Equation 15:where γ t h denotes the target SNR threshold. The formulated problem is non-convex due to the coupling of power allocation variables in the SINR expressions, the presence of multiuser interference across subcarriers, and the dependency of the achievable rate on the common and private stream structure in RSMA. As a result, obtaining a closed-form optimal solution is computationally intractable.

S N R c , u ≥ γ t h , ∀ u (15)

5.2 Geometric water-filling algorithm

The GWF algorithm, as an advanced power allocation technique, is implemented to maximize the total transmission capacity of the proposed NTN system. Compared to conventional water-filling, GWF provides a structured and low-complexity geometric interpretation of optimal power allocation and has been widely applied in multiuser and multicarrier systems (Ferdouse et al., 2020; He et al., 2013). In this work, the GWF algorithm is applied to both SCMA and RSMA schemes in each cluster, where the users are ordered based on channel conditions following an SIC-inspired principle. For SCMA, this ordering is used only for power allocation purposes based on effective SINR and does not imply SIC-based signal detection.

The GWF-based power allocation is performed in three main steps: (i) Equal Power Allocation (EPA) for initialization, (ii) Weighted Power Allocation (WPA) to exploit channel conditions, and (iii) Geometric Water-Filling for optimal power redistribution.

To initialize the geometric water-filling procedure, the GWF starts with EPA, where the transmit power is uniformly distributed. In the SCMA-based system, the transmit power of each user is equally distributed among the active subcarriers associated with its assigned SCMA codebook. For a given cluster c, the power allocated to user u on subcarrier k under EPA is expressed in Equation 16 as:where P u S C M A denotes the total transmit power allocated to user u, which is distributed among its active subcarriers such that P u S C M A = ∑ k ∈ S j P c , u , k, N = 2 denotes the number of active subcarriers per SCMA codebook, and S j is the set of subcarriers used by codebook j derived from the SCMA factor graph matrix F.

P c , u , k E P A , S C M A = P u S C M A N , k ∈ S j , Q c , u , j = 1 0 , otherwise (16)

In the RSMA-based system, EPA is applied to the private streams by distributing the total private stream power equally among all users and subcarriers within the cluster. Therefore, the power allocated to the private stream of user u on subcarrier k in cluster c is given in Equation 17 as:where P P r i v a t e denotes the total power allocated to the private streams, which is distributed across users and subcarriers such that P P r i v a t e = ∑ u = 1 U c ∑ k = 1 K c P c , u , k. Following the EPA initialization, a WPA strategy is applied to exploit channel state information. In WPA, the available transmit power is distributed among subcarriers according to the channel gain-to-noise ratio (CNR), such that subcarriers with better channel conditions are allocated more power. This channel-aware allocation enhances spectral efficiency while maintaining low computational complexity and provides a more suitable input for the subsequent geometric water-filling optimization.

P c , u , k E P A , R S M A = P P r i v a t e K c U c (17)

In the SCMA scheme, the weight factor w c , u , k j for user u assigned to codebook j in cluster c is computed in Equation 18 as follows:where g c , k , u = h c , k , u 2 σ 2 denotes the channel gain to noise ratio of user u on subcarrierk in cluster c. This metric guides power allocation and directly influences the received SNR and achievable rate of each user. Using these normalized weights, the WPA-based power allocation for SCMA is expressed in Equation 19 as:

w c , u , k j = g c , k , u ∑ k ′ ∈ S j g c , k ′ , u (18)

P c , u , k W P A , S C M A = P u S C M A w c , u , k j , k ∈ S j , Q c , u , j = 1 0 , otherwise (19)

The total power allocated to each subcarrier within a cluster is computed in Equation 20 by summing the contributions from all users that transmit their data over that subcarrier:

P c , k S C M A = ∑ u = 1 U c P c , u , k W P A , S C M A (20)

In the RSMA scheme, the allocation depends on the channel gain to noise ratio g c , k , u. The weight w c , u , k for subcarrier k is computed in Equation 21 as follows:

w c , u , k = g c , k , u ∑ k ′ = 1 K c g c , k ′ , u (21)

The weights are normalized such that their sum over all subcarriers equals one. The WPA-based power allocation for private streams of user u on subcarrier k is given in Equation 22 as:

P c , u , k W P A , R S M A = w c , u , k P P r i v a t e U c (22)

The total private power per subcarrier k in cluster c is computed based on the Weighted Power Allocation for private streams in Equation 23 as follows:

P c , k R S M A = ∑ u = 1 U c P c , u , k W P A , R S M A (23)

Finally, the GWF algorithm redistributes power based on the ordered channel conditions. To reflect user priority in the GWF process, a depth parameter is introduced following an SIC-inspired ordering principle. Users with lower CNR are assigned higher depth values, while users with higher CNR are assigned lower depth values. Accordingly, more power is allocated to users with poor channel conditions. The inverse CNR is used to define the depth parameter such that users with weaker channel conditions are assigned larger depth values, thereby prioritizing them in the power allocation process. The depth parameter is calculated as the inverse of the normalized CNR in Equation 24 as:where ∑ u ′ = 1 U c g c , k , u ′ is the total gain over all users on the same cluster c and subcarrier k. The users on each subcarrier are arranged in descending order of their CNR for SIC method decoding, and the corresponding depth parameters are rearranged according to this ordering.

d c , k , u = ∑ u ′ = 1 U c g c , k , u ′ g c , k , u (24)

The GWF searches for the largest water level l = 1 , 2 , … , U c such that the residual power remains positive. The residual power at each water level l is computed in Equation 25 as:where P c , k = P c , k S C M A for SCMA scheme, which is calculated based on Equation 20, and P c , k = P c , k R S M A for RSMA scheme, which is calculated based on Equation 23.

P l = P c , k l = 1 max 0 , P c , k − ∑ i = 1 l − 1 d c , k , u l − d c , k , u i l > 1 (25)

The maximum water level l * is defined in Equation 26 as (He et al., 2013):

l * = max l | P l > 0 , 1 ≤ l ≤ U c (26)

Once the maximum water level l * is determined, the allocated power at l * level is computed in Equation 27 as:

P l * = P l * l * (27)

The allocated power for each user on the subcarrier k in cluster c under GWF is given in Equation 28 by:where users are indexed according to the descending order of their CNR. The overall GWF procedure is summarized in Algorithm 1 for clarity.

P c , u , k G W F = P l * + d c , k , u l * − d c , k , u u , u ≤ l * 0 , o t h e r w i s e (28)

Algorithm 1 GWF Algorithm in SCMA and RSMA

It is important to note that while the GWF algorithm maximizes the system sum capacity, it may completely allocate zero transmit power to users experiencing poor channel conditions, particularly under moderate and strong fading scenarios. This behavior is inherent to water-filling–based power allocation strategies that aim to maximize the system sum rate, where users with very low channel gain to noise ratios are excluded from transmission to favor users with stronger channels. Although this approach is capacity-efficient, it can severely degrade fairness and reliability, making it unsuitable for practical NTN systems where minimum service guarantees are required for all users. To address this limitation, a Fairness-Aware Geometric Water Filling algorithm is proposed. The FAGWF scheme ensures that all users receive a minimum level of transmit power sufficient for reliable communication, thereby preventing user starvation and maintaining system connectivity even under adverse channel conditions.

5.3 Proposed fairness aware geometric water filling algorithm

The FAGWF algorithm is proposed as a low-complexity fairness-aware extension of the conventional GWF scheme. It aims to improve the system sum rate while enhancing reliable user connectivity through the minimum QoS constraint. In this work, fairness is interpreted as QoS-based user-connectivity fairness rather than strict rate-equality among users. A user is considered supported when its received SNR satisfies the minimum threshold in Equation 15. Conventional GWF mainly follows channel conditions and may allocate very low or zero power to weak-channel users. Therefore, FAGWF modifies the conventional GWF allocation by introducing a minimum-power adjustment for users that do not satisfy the target SNR threshold, while preserving the total power constraint. After applying the GWF allocation, the power assigned to user u on subcarrier k in cluster c, as given in Equation 28, is considered the raw allocation in Equation 29:

P c , u , k r a w A l l o c = P c , u , k G W F (29)

The total received signal power for each user is calculated in Equation 30 by summing the contributions over all allocated subcarriers:

S i g n a l P o w e r u = ∑ k = 1 K c P c , u , k r a w A l l o c g c , k , u (30)

The corresponding SNR per user is expressed in Equation 31 as:

S N R u = S i g n a l P o w e r u σ 2 K c (31)

Next, the algorithm checks whether the SNR of each user satisfies the target threshold. A user is considered satisfied if SNR u ≥ γ t h; otherwise, the user is marked as unsatisfied.

If any user fails to satisfy this condition, the minimum power ratio for that user is iteratively increased. Specifically, when P u m i n R a t i o + Δ P min ≤ P u m a x R a t i o, the minimum power ratio is updated as P u m i n R a t i o = P u min R a t i o + Δ P min. The P u m i n R a t i o denotes the minimum power ratio, which is initially set to zero. P u m a x R a t i o and Δ P min represent the maximum power ratio and the step size, with fixed values of 0.3 and 0.01, respectively. These values are selected to balance fairness improvement and computational complexity. The maximum ratio prevents excessive power allocation to weak users, while the small step size ensures stable and gradual convergence. The chosen parameters were empirically determined and provide a good trade-off between fairness enhancement and system performance.

The minimum power allocated to user u on subcarrier k is computed in Equation 32 as:

P u , k min = P u m i n R a t i o P c , k (32)

The fair power allocation for each user is defined in Equation 33 as:

P ∼ f a i r u = max P c , u , k r a w A l l o c , P u , k min (33)

To maintain the total power constraint, the fair power allocation is normalized in Equation 34 as:

P f a i r u = P ∼ f a i r u ∑ u ′ = 1 U c P f a i r u ′ P c , k (34)

Equations 32–34 show how the fairness adjustment is applied. First, a minimum power floor is defined for each user as a fraction of the available subcarrier power. Then, the raw GWF allocation is compared with this minimum power floor, so that users receiving insufficient power are given additional protection. Finally, the adjusted allocation is normalized to maintain the total subcarrier power constraint. The SNR of each user is then checked against the target threshold, and the minimum-power ratio is increased iteratively for users that remain unsatisfied.

The allocation is then recomputed based on the GWF scheme. The final power assignment follows the SIC-inspired ordering and is given in Equation 35 for each user index i d x = 1 , 2 , … . , U c:

P c , u , k F A G W F = P f a i r i d x (35)

In cases where a user cannot reach the target SNR even after attaining the maximum allowable power ratio P u m a x R a t i o, the algorithm terminates further adjustment for that user and accepts a suboptimal SNR level. The FAGWF algorithm in SCMA and RSMA is presented in Algorithm 2.

To determine the minimum required SNR for reliable communication, a target bit error rate (BER) constraint is imposed. For a QPSK modulation, the BER as a function of the instantaneous received SNR can be approximated as B E R = Q 2 . S N R (Balaji and Prabu, 2018). Based on this relationship, the SNR threshold corresponding to a target BER of 10 − 6 is approximately γ t h = 13.54 dB. This threshold is employed as the QoS constraint in the proposed FAGWF algorithm.

To define a QoS connectivity criterion that is applied consistently to SCMA and RSMA, an SNR threshold is adopted as a reference value. The threshold is selected based on the baseline QPSK BER relation under an AWGN reference channel, where the BER can be approximated as B E R ≈ Q γ, with γ denoting the received symbol SNR. For a target BER of 10 − 6, the corresponding SNR threshold is approximately γ t h = 13.54 dB. In this work, this value is not used as an exact BER prediction for the SCMA and RSMA systems under composite fading and multiuser interference. Instead, it is used as a fixed QoS threshold to evaluate whether a user is considered connected under the effective received SNR or SINR after channel impairments and power allocation. Applying the same threshold to both schemes ensures a consistent comparison of user connectivity and fairness.

Algorithm 2 FAGWF Algorithm in SCMA and RSMA

The computational complexity of the proposed FAGWF algorithm mainly comes from the additional SNR checking and minimum power adjustment introduced after the conventional GWF allocation. In the GWF stage, users are ordered according to their channel-gain conditions, and the water level is computed for each subcarrier. FAGWF then checks whether each user satisfies the target SNR threshold and increases the minimum-power ratio only for users that remain unsatisfied. Since the minimum-power ratio is updated using a fixed step size and is bounded by a predefined maximum value, the number of adjustment steps is limited. This provides a controlled complexity-performance trade-off, where additional computation is introduced to improve user connectivity while maintaining the low-complexity structure of the GWF-based allocation. It should be noted that FAGWF is designed as a low-complexity extension of GWF for improving QoS-based user connectivity in the considered RF/FSO NTN system. The minimum-power ratio is updated using a fixed step size and capped by a predefined maximum value; therefore, the adjustment process terminates within a finite number of iterations. Feasibility depends on the available transmit power and channel conditions. When the target SNR cannot be satisfied after reaching the maximum power ratio, the algorithm accepts a suboptimal SNR level instead of claiming guaranteed feasibility under all possible conditions.

6 Results and discussion

This section evaluates the transmission capacity and resource allocation performance of the proposed NTN system across both RF and FSO links, considering SCMA and RSMA multiple access schemes. The RF link between the HAP and ground users is analyzed under Nakagami-m fading conditions (weak, moderate, and strong), while the FSO link between the LEO satellite and the HAP is modeled using the Málaga distribution under weak atmospheric turbulence. All results are obtained using Monte Carlo simulations over 100 independent channel realizations and user distributions, and the reported performance metrics are averaged to obtain representative performance trends. The number of realizations was selected to balance statistical averaging with computational cost, since each realization evaluates SCMA and RSMA under multiple power-allocation schemes across the considered RF and FSO channel scenarios. The same simulation procedure is applied to all compared multiple-access and power-allocation schemes, and the reported values represent averaged performance over the considered independent realizations to support a consistent relative comparison. The analysis focuses on key performance metrics, including energy efficiency, sum data rate, as well as user connectivity and fairness. The performance of SCMA and RSMA is compared using four power allocation schemes: EPA, WPA, GWF, and the proposed FAGWF algorithm.

6.1 Energy efficiency performance evaluation in the RF link

This section evaluates the cumulative energy efficiency (EE) performance of the RF link between the HAP and the ground users. It investigates the performance of SCMA and RSMA under variable fading conditions: weak, moderate, and strong Nakagami-m fading is considered to evaluate the robustness of each technique with considering the path loss. The path loss is dependent on the distance each user is from the HAP, in which User one in Cluster one is the closest and User 60 is the farthest. The performance of the proposed FAGWF resource allocation scheme is compared with the GWF, WPA and EPA techniques. Figure 2 shows the cumulative EE performance of SCMA and RSMA under weak, moderate, and strong Nakagami-m fading conditions with all power allocation schemes. It should be noted that Figure 2 represents the entire RF access system with 60 users distributed over 10 clusters, rather than a single cluster. The x-axis denotes the sorted user index, where every six consecutive users correspond to one cluster.

In the three fading scenarios, the cumulative EE decreases with increasing fading severity and user distance due to reduced channel gain and higher path loss. RSMA with FAGWF achieves the highest EE across all fading conditions, with values ranging from approximately 4.3 × 10 6 to 3 × 1 0 6 bits/J for the nearest user, and from 2.5 × 10 6 to 2.1 × 10 6 bits/J for the farthest user as fading varies from weak to strong. This degradation is mainly attributed to the reduction in effective SINR under severe fading conditions.

In the considered scenarios, RSMA offers higher EE than SCMA. This is due to its effective interference management via the message-splitting mechanism, which increases the effective SINR at the receiver by allowing part of the multiuser interference to be managed through the common-stream component and reducing the impact of the remaining interference. Unlike SCMA, which relies on fixed sparse codebooks and may face performance limitations under severe channel conditions, RSMA benefits from its rate-splitting structure under the adopted fixed split between common and private streams. In the adopted SINR and achievable-rate formulation, part of the interference is managed through the common stream, while the remaining interference is treated as noise, improving the achievable rate. This advantage becomes more pronounced under moderate and strong fading conditions, as well as for users located farther from the HAP, where channel disparities and interference effects are more significant. This results in more efficient power utilization and improved spectral efficiency, which directly enhances energy efficiency.

FAGWF and GWF deliver the best EE performance, and in both RSMA and SCMA they outperform the WPA and EPA schemes. This behavior is attributed to the more efficient power allocation strategy of GWF and FAGWF, which prioritizes users based on channel conditions, thereby improving the effective SINR. In contrast, WPA and EPA distribute power less adaptively, leading to higher interference and lower spectral efficiency. Under weak fading conditions, the proposed FAGWF algorithm achieves identical EE performance to the conventional GWF in both SCMA and RSMA schemes, as evidenced by the overlapping curves. Under moderate and strong fading conditions, the FAGWF algorithm achieves energy efficiency performance very close to that of the conventional GWF, with a marginal difference of approximately 1%–3% across all users. This small difference indicates that the fairness enhancement introduces only limited power redistribution while preserving the efficiency of the original GWF scheme. The FAGWF algorithm effectively balances energy efficiency and user fairness, maintaining high cumulative EE even under strong fading conditions. This indicates that FAGWF algorithm is robust and aligns with the goal of developing reliable and fair connectivity in non-terrestrial communication systems.

6.2 Sum data rate performance evaluation in the RF link

The sum data rate for both SCMA and RSMA systems was evaluated across 10 clusters under various Nakagami-m fading conditions in the RF link with four different power allocation schemes: FAGWF, GWF, WPA, and EPA.

In all fading scenarios for both RSMA and SCMA techniques, the sum data rate decreases as the cluster index increases from Cluster one to Cluster 10. This is due to being farther from the HAP and suffering from higher path loss. Furthermore, the overall sum rate for both SCMA in Figure 3, and RSMA in Figure 4 significantly degrades as the fading severity increases from weak to strong.

This demonstrates the negative impact of channel impairments on sum data rate. However, RSMA always achieves a higher sum data rate than SCMA among all fading conditions. Under weak fading, using FAGWF, the highest rate for RSMA is approximately 4.9 × 10 7 bps, while SCMA achieves about 4.1 × 10 7 bps. This performance gap is maintained even under strong fading, where RSMA achieves a higher sum rate of approximately 3.9 × 10 7 bps compared to about 3.3 × 10 7 bps for SCMA. This corresponds to an improvement of approximately 18%–20%, with similar performance gains observed across different fading conditions. This confirms that the interference management capabilities of the rate-splitting mechanism allow RSMA to utilize the channel more efficiently and achieve greater sum data rate compared to SCMA’s non-orthogonal codebook mapping, by improving the effective SINR and enabling more efficient bandwidth utilization.

Among the power-allocation techniques, GWF and FAGWF achieve identical sum-rate performance under weak fading conditions. Under moderate and strong fading conditions, the FAGWF algorithm achieves sum data rate performance very close to that of the conventional GWF, with a marginal difference of approximately 0%–2% across all users. This indicates that the fairness constraint introduced by FAGWF results in only minor power redistribution, preserving the overall sum-rate performance while improving user connectivity. They provide the highest sum-rate performance and outperform WPA and EPA in all clusters and fading conditions. This is because EPA assigns equal power to all users and does not consider channel conditions, while WPA prioritizes users with stronger channels but does not ensure fairness across users.

6.3 User connectivity and fairness evaluation

As discussed above, GWF and FAGWF achieve very similar sum-rate performance and GWF has a minor advantage under moderate and strong fading. The two schemes differ in their ability to maintain user connectivity, particularly under severe fading. GWF maximizes throughput by allocating more power to users with strong channel gains, which can result in some users receiving little or no power, effectively dropping them from service.

In contrast, the FAGWF approach modifies the GWF procedure with the aim of ensuring that every user receives at least a minimum transmission power, allowing users to satisfy the minimum SNR requirement for reliable communication (S N R ≥ γ t h). This difference is illustrated in Figure 5, which shows the total number of users that can communicate across the 10 clusters. Under strong fading, SCMA-GWF supports only 41 users and RSMA-GWF supports 46 users, leaving a notable portion of users disconnected. FAGWF, however, maintains full connectivity for all 60 users in both SCMA and RSMA systems, regardless of fading severity within the considered system configuration. These results highlight that while GWF slightly outperforms FAGWF in sum-rate and energy efficiency, FAGWF provides a significant improvement in fairness and connectivity, ensuring reliable service for all users with only a marginal reduction in the total sum-rate and EE. This occurs because FAGWF applies only a limited and targeted redistribution to users that fail the SNR threshold, while the normalization step preserves the total power budget and avoids large power reductions for users with strong channel conditions. Therefore, fairness in this work is evaluated in terms of QoS satisfaction rather than strict rate-based fairness.

In addition to the fairness evaluation based on connectivity, Jain’s Fairness Index is used to quantify the balance of user rates within each cluster. For cluster c, Jain’s Fairness Index is calculated as J F I c = ∑ u = 1 U c R c , u 2 U c ∑ u = 1 U c R c , u 2. A value close to one indicates a more balanced rate distribution among users, while a lower value indicates a larger rate imbalance. Figure 6 presents JFI for GWF and FAGWF under weak, moderate, and strong fading conditions. While Figure 5 evaluates fairness from the perspective of user connectivity, Figure 6 provides an additional indication of how evenly the achieved rates are distributed among users within each cluster. Thus, connectivity fairness and Jain’s Fairness Index are used as complementary metrics, where the former evaluates whether users satisfy the minimum QoS requirement, while the latter evaluates the balance of achieved rates among users. In Figure 6a, the SCMA results show that GWF maintains a high fairness index under weak fading, but its fairness decreases under moderate and strong fading, especially when some users receive insufficient power. In contrast, FAGWF keeps the fairness index close to one across all clusters and fading conditions. A similar trend is observed for RSMA in Figure 6b, where GWF shows reduced fairness under stronger fading, while FAGWF maintains a consistently high fairness index. These results confirm that FAGWF improves fairness not only by maintaining full user connectivity, as shown in Figure 5, but also by producing a more balanced user rate distribution across clusters.

6.4 Energy efficiency and sum data rate performance evaluation in FSO link

In this section, the performance of the FSO link connecting the LEO satellite and the HAP is evaluated. Unlike the RF access link, the FSO link operates in a high-altitude environment where atmospheric turbulence is significantly less severe. Therefore, the channel is modeled using the Malaga distribution under weak turbulence conditions. The simulation considers a scenario with six users over the FSO link for both SCMA and RSMA schemes. These users represent multiple data streams transmitted over the same FSO link, corresponding to aggregated user signals at the HAP. It should be noted that the LEO–HAP FSO link is physically a point-to-point optical link. In this study, SCMA and RSMA are used to combine multiple data streams before transmission over the same optical link toward the HAP, rather than representing multiple independent optical transmitters accessing the HAP.

The cumulative EE of RSMA and SCMA is evaluated under weak atmospheric turbulence using the Málaga model in the FSO link, as shown in Figure 7. The results demonstrate that RSMA achieves higher EE compared to SCMA for all users. It can also be observed that GWF and FAGWF show nearly identical performance in this scenario, as further discussed at the end of this section.

The RSMA system achieves a higher and more stable EE of approximately 2.3 × 10⁸ bits/Joule when using GWF and FAGWF, whereas the SCMA system with the same power allocation schemes achieves a lower EE of around 2.0 × 10⁸ bits/Joule. This indicates that RSMA is more efficient in delivering higher data rates for a given energy budget in the FSO link. This improvement is mainly due to the rate-splitting structure of RSMA under the adopted fixed split between common and private streams. By decoding part of the interference through the common stream and treating the remaining part as noise, RSMA improves the effective SINR. Both GWF and FAGWF outperform WPA and EPA for all users, this confirms the importance of adaptive power allocation in enhancing energy efficiency. In contrast, WPA and EPA achieve lower EE due to their limited ability to allocate power effectively based on the channel conditions.

The sum data rate performance of RSMA and SCMA in the FSO link between LEO and HAP is illustrated in Figure 8. Similar to the observations in the RF link, RSMA consistently achieves higher spectral efficiency than SCMA in all power allocation schemes. RSMA attains a maximum sum data rate of approximately 2.75 × 10⁹ bps when using the GWF and FAGWF algorithms, whereas SCMA reaches a peak of about 2.4 × 10⁹ bps. This performance advantage is due to RSMA’s flexible interference management, which allows it to effectively handle multiplexed streams even in the optical domain.

Both GWF and FAGWF show identical performance and achieve the highest sum rates for RSMA and SCMA. This demonstrates their effectiveness in optimizing power distribution under atmospheric turbulence. In contrast, the EPA scheme achieves the lowest performance, which confirms that equal power allocation without considering channel conditions fails to exploit the channel’s capacity efficiently.

It is important to note that the performance of FAGWF and GWF is nearly identical in this scenario due to the weak atmospheric turbulence in the FSO link. Under weak turbulence, channel conditions across users become more uniform, resulting in minimal variation in channel gain-to-noise ratios among users, which leads to nearly uniform power allocation across users. As a result, both FAGWF and GWF converge to similar power distribution strategies, leading to nearly identical energy efficiency and sum data rate performance.

7 Conclusion

This paper investigated transmission capacity enhancement and fairness-aware resource allocation in RF/FSO NTNs integrating LEO satellites, HAPs, and ground users. To support massive connectivity and improve spectral efficiency, two advanced multiple access schemes, SCMA and RSMA, were implemented and evaluated within the proposed NTN architecture. In addition, a FAGWF algorithm was proposed to balance transmission capacity and user fairness while maintaining reliable connectivity under heterogeneous channel conditions.

Simulation results show that when the proposed FAGWF algorithm is applied to both RSMA and SCMA, the RSMA-based system outperforms the SCMA-based system in terms of energy efficiency and sum data rate across both RF and FSO links under the considered scenarios. Under the adopted simulation configuration of 60 users grouped into 10 clusters with six users per cluster, RSMA achieves up to 19% higher sum data rate than SCMA in the RF access link under strong fading conditions, while in the FSO link it provides an improvement of approximately 15%. Moreover, in the considered configuration, the proposed FAGWF algorithm achieves performance very close to conventional GWF, with only 0%–2% difference in sum rate and 1%–3% difference in energy efficiency, while significantly improving fairness by maintaining 100% user connectivity and achieving a Jain’s Fairness Index close to one across the considered fading conditions, compared to 68.3% and 76.7% connectivity for SCMA and RSMA under conventional GWF, respectively.

Overall, the results indicate that FAGWF achieves near-optimal performance with improved user connectivity, providing an effective trade-off between system throughput and fairness. These results demonstrate that the combination of RSMA and fairness-aware power allocation provides an effective framework for enhancing transmission performance in RF/FSO NTN systems. Future work may extend this framework by considering different cluster sizes, user loading factors, user mobility, Doppler effects, CSI aging, imperfect channel state information (CSI), handover, and dynamic traffic demands, as well as incorporating broader benchmark comparisons and machine-learning-based resource allocation for large-scale NTN systems.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Simone Morosi, University of Florence, Italy

Reviewed by: Guanjun Xu, Hangzhou Dianzi University, China

Nawaf Q. Othman, Xi’an Fanyi University, China