5G-MANTRA: Multi-Access Network Testbed for Research on ATSSS

This paper presents a 5G multi-access core network testbed, which is designed to serve as an essential tool for research on Access Traffic Steering, Switching, and Splitting (ATSSS) in 4G and 5G networks. By integrating Non-3GPP Inter-Working Function (N3IWF) and 5G core from different open-source projects, our testbed enables multi-access sessions in 4G-LTE, 5G NSA, and 5G SA. Furthermore, our testbed incorporates a transport converter into the 5G core network, allowing User Equipment (UE) to utilize multi-access networks, even in the absence of multi-path TCP (MPTCP) support from external servers. We demonstrated the ability of our testbed to measure traffic steering, switching, and splitting across both lab-based and real-world networks.


INTRODUCTION
The 5G technology evolution has enabled user equipment (UE) to connect to the core network (CN) through non-3GPP access networks, such as WiFi [1].This has unlocked significant opportunities for users and operators to utilize the multi-access capabilities inherent to 5G networks as it allows them to offload traffic from traditional 3GPP access networks to alternative non-3GPP access networks [2].This strategic shift can optimize the network's performance and capacity, thereby enhancing the efficiency and versatility of 5G technology.
To further utilize these multi-access capabilities, the 3GPP 5G Release 16 specifications introduced "Access Traffic Steering, Switching, and Splitting" (ATSSS) as an optional feature [3].ATSSS includes "Traffic Steering, " "Traffic Switching, " and "Traffic Splitting." Traffic Steering enables dedicated use of either 3GPP or non-3GPP access networks based on current conditions and needs.Traffic Switching allows for seamless data migration from one network to another as performance changes.Lastly, Traffic Splitting permits simultaneous data transmission over both the 3GPP and non-3GPP access networks.
Despite these advancements in 3GPP standards, a significant shortcoming exists in the current open programmable software stacks for 5G technology: they lack the implementation of crucial multi-access and ATSSS features.Specifically, while open-source solutions like Open5GS [4] and Free5GC [5] are widely used in the 3GPP research community, each project has limitations that prevent its use as an independent testbed for 5G multi-access networks and ATSSS research.
Table 1 shows a summary of the different testbeds, comparing their capabilities for 5G multi-access networks and ATSSS research.Open5GS supports 4G-LTE, 5G NSA, and 5G SA deployments but lacks the Non-3GPP Inter-Working Function (N3IWF) implementation.Thus, it does not support the connection through non-3GPP access networks.On the other hand, Free5GC includes an N3IWF implementation but lacks 4G-LTE and 5G-NSA deployments, which are likely to be in use for years as 5G SA adoption expands.Moreover, both Open5GS and Free5GC do not support multi-access packet data unit (MA-PDU) sessions.This means that the UE can only use one of 3GPP access or non-3GPP access for each PDU session.These limitations of the existing open-source solutions hinder the execution of prototyping and advanced experiments in the multi-access and ATSSS research field.
In this paper, we present 5G Multi-access Network Testbed for Research on ATSSS (5G-MANTRA) as a crucial step towards enabling comprehensive multi-access communications and ATSSS operations in future research.Our testbed integrates the network functions of Open5GS and Free5GC to support the multi-access sessions for the active 3GPP generations, including 4G-LTE, 5G NSA, and 5G SA.Then, we implemented a transport converter within the core network of our testbed.The transport converter is one potential implementation strategy to implement ATSSS services.It functions as a multi-path TCP (MPTCP) proxy which enables multi-access data sessions and management of them [6], [7].We demonstrated the ability of our testbed to measure traffic steering, switching, and splitting across both lab-based and real-world networks.This provides a valuable resource for researchers to investigate the impact of these techniques.We also release 5G-MANTRA as open-source for future research [8] [9].
The rest of this paper is organized as follows.Section 2 presents a background and related work of multi-access capability and ATSSS in 5G networks.The design and implementation details are described in Section 3. Section 4 presents the performance evaluation of our testbed.Finally, we conclude this paper in Section 5.

BACKGROUND AND RELATED WORK 2.1 Core Network Functions
The 5G core network consists of various network functions (NFs) [10].Key NFs include the Access and Mobility Management Function (AMF) for user device authentication and mobility management; the Policy Control Function (PCF) for enforcing security, quality of service, and charging policies; the Session Management Function (SMF) for managing user sessions and allocating IP addresses; and the User Plane Function (UPF) for handling data traffic, performing packet routing and forwarding, and managing traffic.
In addition to the aforementioned NFs, a 5G core network can include Non-3GPP Inter-Working Function (N3IWF) that enables UE to connect to the 5G core network using non-3GPP access networks, such as WiFi.In particular, it supports session establishment and authentication through the non-3GPP access networks for the UE.Operators have the option to enable non-3GPP data sessions via either "trusted" or "untrusted" access networks.For trusted access, the operator uses a Trusted Network Access Point (TNAP) with keying material for secure session establishment on behalf of UE.In untrusted scenarios, the N3IWF holds the operator's keying material and aids in setting up a Security Association (SA) for a UE connecting through a standard access point.

ATSSS
ATSSS is designed to support Multi-Access PDU (MA-PDU) sessions, which enables traffic to be transmitted over one or more concurrent accesses, including 3GPP access, trusted non-3GPP access, and untrusted non-3GPP access.Especially, to utilize these multi-accesses, ATSSS provides three services: traffic steering, traffic switching, and traffic splitting.The concept of traffic steering involves selecting the most suitable network for data transmission at any given time.Traffic switching refers to the smooth handover between different access networks.Traffic splitting is the aggregation of network resources from multiple access networks.
The core operation of ATSSS is defined by a set of rules that are assigned to each UE.These rules allow traffic to be managed with fine granularity.Attributes of these rules include steering modes such as 1) load-balancing traffic across links by percentage, 2) priority-based steering, 3) and redundant mode steering.Traffic can be steered based on source and destination IP addresses.Furthermore, rules can have loss threshold percentages that determine what level of network congestion and packet loss is considered to be within acceptable bounds.

MPTCP and Transport Converter
In 3GPP Release 16, Multi-Path TCP (MPTCP) was selected as the primary transport layer protocol for implementing ATSSS.MPTCP's core functionality includes quality of service measurements along multi-accesses, active steering across optimal access However, MPTCP is not widely adopted in the real world due to the lack of support in operating systems and its deployment cost [11].To overcome this problem, the TS 25.301 standard mandates the inclusion of an ATSSS-compliant transport converter within the core network [1].The transport converter is defined in RFC 8803 (0-RTT TCP Convert Protocol) and capable of transforming MPTCP traffic into a standard TCP traffic between the ATSSS-capable UE and non-MPTCP servers [7].Moreover, it can offer vital TCP data including average Round Trip Time (RTT), acknowledged and lost packets, along with other pertinent TCP connection state data.
As a result, the transport converter has been recognized as a practical approach to realizing the ATSSS service as it not only facilitates seamless MPTCP communications but also provides invaluable TCP information for network optimization across multiple network interfaces [12].

Related Work
While there are limited studies on ATSSS in 5G networks, a few works have investigated the implementation of ATSSS.Ha et al. [13] investigated the protocol requirements for ATSSS MA-PDU sessions at the UPF level and suggested the possible implementation of MA-PDU sessions consisting of "children" of a "parent" PDU sessions.Even though the authors discussed the design issues of ATSSS operations, this study did not provide the actual implementation of ATSSS.Wu et al. [14] investigated the fundamentals of multi-access data sessions in a 5G-WLAN context and specifically concentrate on optimized schedulers for such sessions.However, the study employed the simulated environment based on Mininet for the experiments.Furthermore, the study utilized an MPQUIC implementation instead of MPTCP, thereby reducing the applicability of the results.
To sum up, to the best of our knowledge, no study has actually implemented ATSSS so far.In this paper, we introduce a tangible testbed that facilitates the real-world execution and investigation of ATSSS in 5G networks.

DESIGN AND IMPLEMENTATION
Figure 1 shows the software and hardware architecture of our testbed, which consists of a custom core network enhanced with 5G-MANTRA 5G-MeMU '23, September 10, 2023, New York, NY, USA a transport converter, a laptop-based UE with MPTCP capabilities, and a radio access network (RAN) and WiFi access points.In this section, we describe the design and implementation of each component in our testbed.

Core Network
3.1.1Integration of Network Functions.In order to facilitate multi-access PDU sessions across all current 3GPP standards (i.e., 4G-LTE, 5G NSA, and 5G SA), we developed a custom solution by merging Open5GS and Free5GC.Particularly, we incorporated the N3IWF from Free5GC into a custom build of the Open5GS core network.Then, we confirmed that the N3IWF could successfully connect to the core network.Please note that our custom core network supports both eNB and gNB as it includes the network functions of both EPC and 5GC.
We set up our core network on an Arch Linux machine running the latest Linux 6.2 kernel.We chose the newer kernel because it allows us to access detailed TCP information at the subflow level, which is crucial for validating our testbed.
3.1.2Transport Converter.Our testbed embodies a custom, lightweight implementation of a transport converter, written in Python.The key operation of the transport converter is "0-RTT redirection" defined in 8803 (0-RTT TCP Convert Protocol).The "0-RTT redirection" is performed based on the TCP fast open.During an MPTCP 3-way handshake, the TCP fast open nests the destination IP of the TCP SYN packet within the packet's payload.Then, it subsequently replaces the IP of the packet.
In our testbed, the transport converter is collocated on the same Linux 6.2 system where the UPF is deployed.Along with the UPF, our transport converter maintains a series of states for each proxied connection and logs TCP information.Consequently, this allows the core network can control the multi-access communication and collect the network metrics from TCP information.
To leverage the network metrics provided by the transport converter, we also implemented a performance logger module within our transport converter.The performance logger is designed as an independent module, allowing for easy extensibility.The performance logger relies on kernel-provided mptcp_info, mptcp_subflow_data, and mptcp_subflow_addrs objects.These objects contain information on MPTCP connections, such as the number of active subflows, source and destination addresses per subflow, and path management information.In addition, the performance logger can fetch tcp_info object per subflow by using the MPTCP_TCPINFO option for getsockopt().This tcp_info object includes the information on each subflow connection, such as smoothed RTT, the number of bytes sent, the number of bytes acknowledged, and so on.As a result, we could track the subflow status and its performance in real-time by using these objects.

User Equipment
We created a UE that allows more fine-grained experimentation for the multi-access data sessions.Especially, we utilized a second Arch Linux-based machine with 6.2 Linux kernel, connected to the core network through a USB WiFi and LTE dongles.This provides the machine with two network interfaces.
Furthermore, we integrated an open-source NWu protocol "dialer" into the testbed UE to establish a PDU session over an untrusted non-3GPP (WLAN) access network [15].This dialer encompasses the requisite functionalities for UE attachment over a non-3GPP access network, such as authentication (i.e., EAP-AKA and 5G-AKA), data encryption/integration, and the establishment of IKEv2/IPSec tunnels with N3IWF.This allows the UE to sustain data sessions with the core network, spanning both 3GPP and non-3GPP accesses, Lastly, to enable "0-RTT redirection" in the UE, we used a modified version of the libconvert library written by Tessares [16].This library implements the whole 0-RTT Convert Protocol and is loaded with LD_PRELOAD to intercept system calls made by our test C code.When our code makes calls to socket, connect, or read, the library intercepts these calls and automatically writes and parses Convert Protocol headers when redirecting the traffic to the transport converter.This allowed us to use sockets of type IPPROTO_MPTCP and have the connection proxied automatically without writing custom code.

PERFORMANCE EVALUATION
To demonstrate the feasibility and effectiveness of our testbed, we conducted two different experiments: 1) an ATSSS scenario experiment 2) and a real-world deployment experiment.In the ATSSS scenario experiment, we evaluated the network performance under different ATSSS traffic scenarios: traffic steering, switching, and splitting.Meanwhile, in the real-world deployment experiment, we deployed and tested our transport converter, which is a key component of our testbed, in the real-world environment.The aim of the real-world deployment experiment is to see the potential benefits and drawbacks of deploying ATSSS in the real-world.

ATSSS Scenario
In the ATSSS scenario experiment, the performance measurements were conducted in a controlled laboratory environment.To analyze the performance of ATSSS, we controlled network settings with three different traffic scenarios: traffic steering, switching, and splitting.Then, we used the performance logger to keep track of key network metrics such as RTT, bytes sent, and transfer rate for each subflow.
Traffic Steering: In the traffic steering scenario, the UE first establishes two active subflows with both 3GPP (LTE) and non-3GPP (WiFi) accesses.Then, 2 minutes after the start of the experiment, we introduced congestion on the non-3GPP access using the traffic shaping tool tc [17].We imposed a 100ms latency and 2% packet loss exclusively on the UE's WiFi interface for both ingress and egress traffic.
Figure 2 shows selected TCP measurements per subflow, including Round Trip Time (RTT), bytes sent, and transfer rate measured in the traffic steering scenario.In the figure, we can observe that the congestion is introduced at approximately two minutes, causing traffic to be steered away from the WiFi subflow.Accordingly, in Figure 2(b), the LTE subflow emerging as the "dominant" subflow, handling a larger portion of the data transmission after a few minutes of increased congestion.In addition, the transfer rate of WiFi decreases due to the congestion as shown in Figure 2(c).Traffic Switching: In the traffic switching scenario, we set the UE to establish two active subflows on both 3GPP and non-3GPP accesses.Then, we gradually moved our UE away from the WiFi access point until the WiFi subflow becomes completely non-functional.The purpose of this scenario of is to mimic a general situation where a user is engaged in a video call and begins moving away from a WiFi access point.Without ATSSS service, the call may experience interruptions or temporary drops as a new TCP session is established [18].However, traffic switching addresses this challenge by ensuring seamless mobility and preserving the active session.
Figure 3 shows the TCP measurements per subflow in the traffic switching scenario.In Figure 3(c), the transfer rate of WiFi drops to zero approximately at 1 minute and 30 seconds.After the WiFi's transfer rate reaches zero, the transfer rate over the LTE subflow increases dramatically, and the session maintains satisfactory performance even if the aggregate transfer rate is not as high as when both WiFi and LTE are utilized.In addition, Figures 3(b) reveals that the traffic is no longer sent over the WiFi interface after a short period of instability, and the subflow is eventually terminated altogether.
Traffic Splitting In the traffic splitting scenario, we focused on the offloading of traffic from the LTE to WiFi.We first set the UE to establish a single subflow over LTE.Then, after some time, we added a second subflow over WiFi.
Figure 3 shows the TCP measurements per subflow in the traffic splitting scenario.In Figure 4(c), the WiFi subflow becomes active at 2 minutes, and the LTE subflow experiences a noticeable decline in transfer rate.This redistribution of traffic was performed to

Real World Deployment
In the real-world deployment experiment, we deployed our transport converter in the real-world networks.The transport converter is a key component of our testbed that enables the multi-access data sessions and ATSSS functionalities.Thus, this experiment can provide us with insights into the potential benefits and drawbacks of ATSSS in the real-world.
Figure 5 shows the deployment of the real-world experiment.In the experiment, the UE was placed at a fixed location and equipped with a SIM card provided by a major US operator, referred to as "Operator N".This allowed our UE to connect to the real 5G network.Then, by examining the IP geolocation of the SIM card, we could determine that the traffic over the 5G network travels to Operator N's core network in Houston, Texas, which is approximately 150 km away from the UE's location.
Our transport converter was deployed in close proximity to the core network to ensure the hop-to-hop delay between the core network and the transport converter.To this end, we provisioned an AWS EC2 t3.xlarge instance within an AWS "Local Zone" in Houston.The instance was assigned a Houston-based IP address, and we could minimize the distance between the core network and our transport converter.
Furthermore, we deployed our web server on a second EC2 instance in Virginia.The web server is capable of functioning as both a TCP server and an MPTCP server.This flexibility allowed us to route traffic directly to the server (i.e., MPTCP mode) or through the transport converter (i.e., TCP mode).Please note that we did not have control over the routing of WiFi network.Therefore, we estimated that all WiFi traffic traverses a central routing location in Dallas, Texas.
Finally, to assess the performance of the transport converter in the real-world environment, we measured download time, uplink completion time, and time to first byte (TTFB) using various file sizes.Then, we compared the measurements under two cases: 1) using the transport converter; and 2) direct MPTCP.In the case of using the transport converter, we routed the traffic to the web server via the transport converter.This case represents the ATSSS installed in the real-world environment.On the other hand, in the case of direct MPTCP, we established a direct MPTCP connection between the UE and the web server.The measurement was iterated 50 times and the results were averaged.
Download Time: To assess the download performance, we conducted a series of experiments in which the UE downloads the files with various sizes: 50MB, 100MB, 200MB, and 500MB.Figure 6(a) illustrates the comparison of download times for different file sizes.The downlink performance achieved through the transport converter exhibits minimal variation, with the maximum difference observed to be slightly above 100ms.Considering the dynamic nature of network conditions in a real Mobile Network Operator (MNO) environment, we conclude that the transport converter introduces a negligible performance impact on downlink traffic.
Upload Time: In the experiment for the upload time, the UE uploaded the files with various sizes.Figure 6(b) illustrates the comparison of upload times for different file sizes.Uploading files through the transport converter as opposed to a direct MPTCP connection exhibits similar performance for smaller files.However, as the file size increases, uploading files through the transport converter shows a longer upload time compared to that of direct MPTCP.Our observations show that the aggregate transfer rate through the transport converter tends to remain higher, which may account for shorter upload times.
Time to First Byte: The Time to First Byte (TTFB) provides valuable insights into the session creation time between the client and server [19].Moreover, in the case of using transport converter, TTFB also contains the processing time introduced by the transport converter.
To measure TTFB, we included it as part of the downlink performance measurements (i.e., experiment for Download Time).In other words, we calculated the average TTFB when the UE downloaded the files with different sizes.Figure 6(c) compares TTFB depending on file sizes during our downlink experiments in Section.The TTFB achieved through a direct MPTCP is consistently smaller in all cases, albeit by only a few milliseconds.This marginal difference can likely be attributed to reduced processing times when communicating directly, bypassing the transport converter.Nevertheless, the minimal impact on TTFB reaffirms that the transport converter has a negligible effect on TTFB.

CONCLUSIONS
In this paper, we presented a 5G Multi-access Network Testbed for Research on ATSSS (5G-MATRA) for the future research on ATSSS in 4G and 5G networks.Our testbed incorporated the N3IWF from Free5GC into a custom build of the Open5GS core network to support the multi-access sessions.In addition, we implemented the transport converter within the core network of our testbed.Our transport converter enabled the ATSSS services by supporting multiaccess data sessions and their management.Finally, we performed the experiments to demonstrate the ability of our testbed.The experimental results showed that our testbed can enable the ATSSS services in the lab environment and provide us with insights into the potential benefits and drawbacks of deploying ATSSS in the real-world.

Figure 1 :
Figure 1: 5G-MANTRA software and hardware architectures networks, and traffic splitting across multi-access networks.These are essential for the successful implementation of ATSSS and make MPTCP a crucial component for enabling efficient utilization of available network interfaces.However, MPTCP is not widely adopted in the real world due to the lack of support in operating systems and its deployment cost[11].To overcome this problem, the TS 25.301 standard mandates the inclusion of an ATSSS-compliant transport converter within the core network [1].The transport converter is defined in RFC 8803 (0-RTT TCP Convert Protocol) and capable of transforming MPTCP traffic into a standard TCP traffic between the ATSSS-capable UE and non-MPTCP servers[7].Moreover, it can offer vital TCP data including average Round Trip Time (RTT), acknowledged and lost packets, along with other pertinent TCP connection state data.As a result, the transport converter has been recognized as a practical approach to realizing the ATSSS service as it not only facilitates seamless MPTCP communications but also provides invaluable TCP information for network optimization across multiple network interfaces[12].

Figure 5 :
Figure 5: Deployment for Real World Testing maintain the overall performance level while offloading a portion of traffic to the WiFi.

Figure 6 :
Figure 6: Real World Performance Testing