CARL-W: a Testbed for Empirical Analyses of 5G and Starlink Performance

The deployment of 5G networks, including 5G Non-Public Networks (5G-NPNs) for private use in several verticals, is rapidly taking place worldwide. However, deploying these networks in under-served areas, where there may be limited Internet access or wired backhauling capabilities, presents challenges. To address these challenges, there is a growing interest in using Low Earth Orbit (LEO) satellites, such as SpaceX's Starlink, which can provide high-throughput and low-latency Internet access via dense satellite constellations. In this paper, we present CARL-W, the Wireless module of the Communications Advanced Research Laboratory (CARL) at Karlstad University, which combines a 5G-NPN and a Starlink deployment. CARL-W serves as a platform for empirical analyses on both systems, thus contributing to the study of their possible integration. In particular, we outline the CARL-W experimentation framework and provide access to the CARL-W visualization and data exporting platform. We also open-source a 1-month Starlink dataset, facilitating further analyses of this relatively new technology.


INTRODUCTION
Since their standardization in 3GPP Release 15 (Rel-15), 5G networks have been rapidly deployed worldwide.Unlike previous generations, mostly focused on voice and general Internet access, 5G systems are designed to address diverse verticals in the context of enhanced Mobile Broadband (eMBB), Ultra-Reliable Low-Latency Communication (URLLC), and massive Machine-Type Communication (mMTC).
To meet the requirements of these services, 3GPP has standardized several innovative features.5G Non-Standalone (NSA) and Standalone (SA) deployment modes have been defined, both requiring a 5G New Radio (NR) Radio Access Network (RAN) but leveraging either a 4G or a 5G core network, respectively.Moreover, 3GPP has defined architectural solutions ranging from network slicing on a 5G public network (5G-PN) to 5G Non-Public Networks (5G-NPNs).
Via 5G-PN network slicing, Mobile Network Operators (MNOs) can allocate specific resources of their 5G-PNs (i.e., slices) to specific verticals, thus satisfying their service requirements while using a common infrastructure.On the other hand, a 5G-NPN is a 5G network for private use in a vertical [7,22].Compared to 5G-PNs, 5G-NPNs have additional benefits in terms of vertical-specific configurations, data security, and performance.Therefore, 5G-NPNs are crucial for several verticals including Industry 4.0, healthcare, education, and communication in under-served areas (ports, ships, extraction sites, and rural areas).
The deployment of a 5G wired backhaul is difficult in some of the above use cases, e.g., due to geographical topology, thus potentially hindering the use of 5G-NPNs, and of 5G in general.Hence, alternative solutions for Internet access, particularly satellite-based options, become significant.
Classical satellite communications for Internet access use the K a band  and Geostationary Equatorial Orbit (GEO) satellites.These satellites orbit at a distance of approximately 36 000 km from Earth, and can provide connectivity to a large area at the cost of high latency (several hundreds of milliseconds).Hence, Low Earth Orbit (LEO) satellite communications have recently emerged, using K u (12-18 GHz) and K a bands.LEO satellites orbit at distances from 180 to 2000 km, thus covering smaller areas compared to GEO satellites.Denser satellite constellations are thus used to cover large areas with, however, significant benefits in terms of latency and throughput.A key LEO satellite-based Internet Service Provider (ISP) is SpaceX with Starlink [35].
Aiming at providing Internet access and/or backhauling capabilities to remote 5G deployments via a satellite link, the integration of 5G and LEO satellite-based systems is being increasingly discussed across stakeholders [38], and within research and standardization fora [1,4,18,23,28].
In this paper, we describe our ongoing activities towards the design and implementation of an innovative testbed combining 5G and LEO satellite-based systems, and provide a validation of the currently deployed testbed.The Wireless module of the Communications Advanced Research Laboratory (CARL-W) at Karlstad University features a 5G-NPN (in SA mode) and a Starlink deployment.CARL-W includes a cross-technology experimentation framework that facilitates baseline experiments, and a data pipeline for measurement collection, storage, visualization, and export.This framework enables parallel characterization of 5G and Starlink technologies, as well as integration analysis and testing.
Considering joint empirical analyses of 5G and Starlink, preliminary work has been recently carried out [10] but further investigation is needed to advance the state of the art, which mostly addresses these two systems separately (see Section 2).Our contribution can be summarized as follows: • We present CARL-W, the wireless testbed deployed at Karlstad University that includes a 5G-NPN and a Starlink deployment.Together with the infrastructural components, we describe the experimentation framework instantiated onto the testbed, which includes a data visualization and exporting platform freely accessible at 5g.carl.kau.se.

RELATED WORK
Over the last few years, several measurement-based studies of 5G systems have emerged.Besides initial performance analyses on dedicated testbeds, e.g., those in the scopes of the EU 5G Public Private Partnership (5G-PPP) [2] and of the US Platforms for Advanced Wireless Research (PAWR) [31] programs, 5G performance was mostly analyzed on commercial 5G-PNs deployed in urban scenarios.
Performance baselines for commercial 5G-PNs were established in [25,27] for the US, [41] for China, and [15] for Europe (Italy), focusing on mid-band and/or high-band deployments, and NSA and/or SA modes, depending on the implementation choices of the MNOs in each country.The studies analyzed several aspects, including coverage, throughput, and latency.Further analyses were carried out on mobility management [9,16,19], video streaming services [33], edge deployments [17], and the use of machine learning (ML) [26].
Regarding 5G-NPNs, surveys on enabling technologies, deployment models, use cases, and research directions were provided in [7,22].A performance evaluation of 5G-NPNs was given in [34], where mid-band NSA and SA 5G-NPNs were compared in terms of end-to-end and core-only metrics, including one-way delay, packet loss, and packet delay variation.Results highlighted a significant impact of the packet generation rate on the above metrics and better performance for the SA mode.In [21], the 5G-NPN deployed at the University of Kaiserslautern (mid-band, SA mode) was analyzed in terms of coverage, throughput, latency, and jitter, showing high performance variability depending on the measurement location and adopted user equipment (UE).Results highlighted the importance of proper deployment planning and the need for further tests with different UEs, network configurations, and traffic patterns from different services.
Starlink is the first large-scale system providing Internet access via LEO satellites.It operates around 2500 satellites (nearly 40000 in the next years) and is commercially available in the US from 2020 and in many EU countries from 2021.
As a new option for Internet access, Starlink is attracting research interest.In [24], a Starlink vantage point was used to analyze throughput, latency, packet loss, and web browsing performance using TCP and QUIC protocols.Compared to a GEO satellite-based solution, Starlink achieved higher throughput and improved web browsing experience.In the same paper, Starlink was compared with the 4G performance reported in [14,32,40], showing a higher downlink throughput and comparable uplink throughput and web browsing performance.In [13], different vantage points were used to analyze Starlink performance.The authors deployed a browser extension that collected web performance from 18 Starlink and 10 non-Starlink users in 10 cities. Results showed slightly better web performance compared to WiFi and cellular ISPs.Starlink performance bottlenecks were also identified, with packet loss and throughput negatively affected by inter-satellite handovers and bad weather conditions, and latency affected by the bent-pipe architecture [5].Similar findings were discussed in [20], where measurements in urban and rural scenarios, and under mobility, showed that Starlink performance were affected by several environmental factors.A comparison against a cable-based ISP showed slightly higher latency and lower throughput.
The above literature highlights that further investigation is needed as the evolution of both 5G and Starlink systems progresses.A relevant aspect to be addressed empirically is the integration between 5G and Starlink systems.In this direction, an initial analysis was carried out in [10], where Starlink was used as the backhaul between a single 5G NR gNB and the core network.The integration was tested in terms of throughput, latency, web performance, voice over IP, and video streaming, and the results indicated Starlink as the bottleneck in uplink and 5G in downlink (the latter due to the low capability of the adopted 5G setup, as claimed by the authors).Being in the same research area, our work presents the CARL-W testbed at Karlstad University, where a high-performing 5G-NPN coexists with Starlink.CARL-W enables to evaluate the two technologies and thus advance the analysis of the use cases leveraging their integration, including but not limited to the backhaul case in [10].

THE CARL-W TESTBED AT KARLSTAD UNIVERSITY
The CARL-W testbed includes several hardware and software components that enable the assessment of advanced wireless features and novel use cases.In this section, we describe the 5G and Starlink deployments in CARL-W.

5G Deployment
The 5G deployment in CARL-W is illustrated in Figure 1 and comprises a complete 5G-NPN in SA mode (Rel-17 compliant), collaboratively operated by the Department of Computer Science at Karlstad University (CS@KaU), Ericsson, and Telia, in the context of the recently released Telia NorthStar network [39].
On the RAN side, the deployment includes a gNB managing two Ericsson 5G NR indoor radio units (IRUs) [6], deployed in two different buildings (CS@KaU main building and Karlstad Innovation Park, around 500 meters apart).The CS@KaU IRU connects four radio dots (antennas) installed at strategic locations in the building, while the Karlstad Innovation Park IRU manages a single radio dot.The radio coverage encompasses open, wide, indoor spaces physically accessible to the public, as well as controlled lab rooms suitable for long-term placement of experimental equipment.
On the client side, the deployment includes four stationary and one mobile Cradlepoint R1900 routers as 5G-capable devices, placed at different locations and thus experiencing different radio coverage conditions.These routers enable good experimentation flexibility, since measurements can be executed by either connecting further devices or directly within the routers (e.g., as Docker containers).Being operated by Telia, the deployment has access to Telia commercial spectrum in Band n78; however, it also has a dedicated spectrum allocation in the 3.8-4.2GHz band, allowing the implementation of a standalone 5G-NPN using open components, e.g., ORAN [30], srsRAN [36], and Amarisoft [3].
With regards to the core network, the deployment features a cloud-native microservice-based dual mode core with both 5G core (5GC) and 4G Evolved Packet Core (EPC), the latter for future NSA compatibility.Most of the 5GC functions, and particularly the ones for the control plane, are currently instantiated in Stockholm in the Telia NorthStar administrative domain, and are accessed via a dedicated, high performing, wired transport network.As regards to the data plane functions, our deployment features a local breakout (LBO) functionality, with a User Plane Function (UPF) locally deployed in Karlstad on a dedicated edge infrastructure.The LBO reduces the path length for the data traffic towards a Data Network (DN), thus enabling the analysis of time-sensitive services.By changing the Data Network Name (DNN) configuration, the IP traffic can still be redirected to external UPFs, e.g., via the operator's network, thus enabling experiments in non-LBO scenarios.In summary, different configurations enables the testing of several network scenarios, including the integration with Starlink.
Further capabilities of the deployment include advanced network slicing and orchestration solutions (e.g., the dynamic network slice selection provided by Ericsson 5GC) and several monitoring and observability tools based on network API service exposure.Functionalities under integration are the Low Latency Low Loss Scalable throughput technology (L4S), for the optimization of the RAN towards low-latency services, and an indoor cellular positioning systems, for supporting experimentation towards location-aware services.

Starlink Deployment
The Starlink deployment in CARL-W comprises a Gen-2 Starlink kit, which includes various components: a) a satellite dish known as Dishy McFlatface, functioning as an electronic phased antenna; b) a base that facilitates motorized self-orientation of the dish; c) a WiFi router with an Ethernet adapter; d) data and power cables.
The dish antenna is installed on the roof of the CS@KaU main building, and connected to the Starlink WiFi router.The router is further connected to a switch via Ethernet, to a virtual Local Area Network (LAN), and ultimately to the Karlstad University WiFi network.The configuration shown in Figure 2 allows to connect virtual machines to the Starlink virtual LAN, enabling continuous measurements.
Furthermore, CARL-W offers the capability to combine 5G and Starlink deployments, enabling the analysis of scenarios such as the forwarding of 5G traffic over Starlink.

DATA COLLECTION & VISUALIZATION
For tracking performance over time and ensuring that CARL-W is operating as expected, we continuously track throughput and latency in both 5G and Starlink deployments through active measurements and also perform passive measurements to collect metadata from the wireless modems and the Starlink dish.The collected data is visualized live online and is also available for download.We further detail our measurement and visualisation pipeline below.

Baseline measurements
4.1.1Active measurements.We continuously run baseline measurements of throughput and latency in CARL-W. 1 For both our 5G and Starlink deployments, throughput is evaluated using Ookla's Speedtest [29], utilizing TCP with multiple flows, while latency is measured via ping and the ICMP protocol.For Speedtest, tests are performed towards a server in the Swedish University Network (SUNET).The Speedtest measurements capture TCP downlink and uplink throughput as well as packet loss at the TCP layer.Speedtest also performs TCP ping measurements to capture the round-triptime (RTT) to the server.The ping responder for the dedicated ping measurements is hosted on a server located within our department.These measurements thus capture the RTT to an edge server and allow to precisely capture the latency of the 5G network and the benefits of the LBO.
For our 5G deployment we use a single modem, identified as Cradlepoint-c17, for the Speedtest and ping measurements.This modem is strategically positioned within one of our lab rooms, less than two meters away from one of the 5G radio dots, and has a direct line of sight.A virtual machine (vm-1) is connected over two virtual LANs (thus accessing both the Cradlepoint-c17 and the Starlink modem) to facilitate experiment orchestration.To ensure continuous latency measurements, 10 pings are sent at the start of each minute, over both 5G and Starlink, while the Speedtests are performed every 30 minutes over 5G and every hour over Starlink, taking into account the limitation of data quota for the latter.In addition to our focused 5G and Starlink deployments, vm-1 may also use another virtual LAN connection to perform measurements over a further commercial mobile network for comparison.Due to quota reasons, we do not perform continuous baseline measurements over this network.
In addition to the Speedtest and ping measurements over Cradlepoint-c17, we also perform Netperf throughput measurements from all four Cradlepoint modems in our 5G deployment every 30 minutes.The Cradlepoint modems have built-in support for Netperf, and we leverage Cradlepoint's management services to orchestrate these measurements.The Netperf measurements are coordinated in time with the Speedtest measurements to avoid overlap and run directly on the Cradlepoints.They are managed by a separate virtual machine that communicates with a central Cradlepoint API server to sequentially request Netperf tests from all four modems.The API server subsequently dispatches the tasks to the modems in order.The Netperf tests capture TCP download/upload throughput and TCP latency, with a server hosted in our department.
To efficiently collect, store, and visualize our baseline measurements, we utilize a Telegraf, InfluxDB, and Grafana (TIG) stack.To orchestrate an experiment, that is to collect, process, and aggregate various time series data such as throughput and latency, we employ Telegraf [12].Telegraf is a plugin-driven server agent part of InfluxDB [11], serving as an intermediary for gathering data and forwarding it to specified destinations.In our setup, upon completing a test and processing the data, Telegraf forwards it to InfluxDB.InfluxDB is a time series NoSQL database, optimized for storing and serving time series data through time:value pairs.Lastly, we use Grafana [8], a browser-based visualization tool, to present the data in a visually accessible manner.For Starlink, the dish provides us with valuable metadata on a number of features related to the satellite connection and the communication.We collect these metadata using a dedicated tool called Starlink-gprc-tools [37], which also runs on vm-1 and captures the metadata every 10 seconds.Collected features encompass satellite direction; satellite reachability information; SNR information; various alerts; latency, packet loss rate, and throughputs towards points of presence (POPs); user terminal state; uptime; and hardware/software versions.A comprehensive description of the features collected by the passive Starlink measurements is given in Appendix A.
As for the active measurements, the data collected from both the Telegraf script and the Starlink-gprc-tools are stored and archived in InfluxDB for further analysis and retention.

Visualization
As mentioned above, we use Grafana to present the collected data in a web interface.Grafana has various plugins that enable users to visualize the data in their preferred format, e.g., charts, tables, and graphs.Grafana also allows to customize the time granularity to display the desired feature data.
Figure 3 illustrates a Grafana dashboard that visualizes various time series data obtained from our live 5G measurements.By customizing the time granularity, we take a closer look at part of the dashboard from Figure 3 in Figure 4, and focus on the 5G downlink throughput obtained from Netperf.To enhance readability, from Figure 4, we display the graphs through plotted representations rather than presenting them as screenshots.
Figure 4 illustrates the achievable throughput from the four Cradlepoints and highlights the difference based on their position in relation to the radio dots.The top three graphs consistently  demonstrate high throughput, averaging above 750 Mb/s.The top two graphs exceed 1 Gb/s, as they correspond to modems positioned in close vicinity and in line of sight to a radio dot.In contrast, the bottom graph illustrates lower throughput, averaging below 200 Mb/s.This difference can be attributed to the modem's placement, which results significantly distant and in non-line of sight from any of the 5G radio dots.As illustrated by the results, the placement of the radio dots allows our baseline measurements, as well as any custom measurements, to capture 5G-NPN performance under different radio conditions.Next we consider Starlink performance as captured by the Speedtest measurements.Figure 5 illustrates the achieved throughput over time using Starlink for an 11-day period from March 2023.In particular, Figure 5a represents the downlink throughput, while Figure 5b represents the uplink throughput, both presented on the Grafana dashboard.Both figures also include throughput data (depicted by red lines) from a commercial 4G network for comparison purposes, illustrating also how baseline measurements from multiple technologies can be easily combined in the CARL-W visualisation platform.Clearly, Starlink averages a higher downlink throughput compared to 4G, but a lower uplink throughput than 4G, which aligns with previous literature results [24].Starlink also exhibits more rapid variations in throughput as compared to 4G, whereas the diurnal variations in throughput are more distinctly noticeable for 4G.
Figure 6 showcases a comparison of the latency and packet loss rate performance between Starlink and 4G as captured by the Speedtest measurements.Due to the inherent nature of satellite connections, Starlink exibits approximately double the latency of 4G, with greater variability (Figure 6a).Furthermore, Starlink also shows higher packet loss rates compared to 4G (Figure 6b).The outcomes of our Starlink baseline tests correspond with the previous study [10] in which primary integration was explored and assessed for various applications like web browsing and video streaming.These findings provide a solid foundation for progressing towards real-world integration and conducting a comprehensive evaluation.

Data Availability
As described above, the captured data can be visualized online.Additionally, the displayed datasets are also available for download in csv format from the online CARL-W visualization platform at https://5g.carl.kau.se.
Starlink is a relatively new technology, and there is a limited availability of Starlink dataset within the research community for data-driven analysis and research purposes.For convenience, we have therefore also compiled a comprehensive curated dataset from our active and passive Starlink measurements that span one month (March 2023) of measurements.This dataset is now available for   download from https://doi.org/10.5281/zenodo.8130936.It is worth noting that, to our knowledge, no other existing Starlink datasets include such an extensive range of metadata.

CONCLUSION & FUTURE WORK
In this paper, we presented CARL-W, the wireless part of the Communications Advanced Research Laboratory at Karlstad University, Sweden.CARL-W features a 5G-NPN in SA mode with LBO capability and a Starlink deployment, thus enabling empirical analyses on both 5G and Starlink technologies.We described the CARL-W experimentation framework and made available the corresponding visualization platform, where the research community can freely access the measurements continuously executed on 5G and Starlink deployments.To facilitate further analyses on Starlink, we also made available a 1-month long dataset with multiple features that allow for in-depth investigations.Future work include the analysis of possible integration solutions between 5G and Starlink, thus contributing to the ongoing research activities on terrestrial/nonterrestrial integrated communication systems.

4. 1 . 2
Passive Measurements.In addition to the active measurements, we also use passive measurements to collect data captured by the modems.For our 5G deployment, a Telegraf script collects various data related to radio coverage from each Cradlepoint modem every hour.The collected radio coverage data encompass Reference Signal Received Power (RSRP), Reference Signal Received Quality (RSRQ), and Signal-to-Interference-plus-Noise Ratio (SINR), as well as an additional feature called signal_percent, derived from the RSRP, RSRQ, and SINR by the Cradlepoint devices.

Figure 4 :
Figure 4: 5G measurements in CARL-W: Example time series of 5G downlink throughput for the four Cradlepoints.

Figure 5 :
Figure 5: Starlink measurements in CARL-W: Example time series of (a) downlink throughput and (b) uplink throughput compared to a 4G commercial network.

Figure 6 :
Figure 6: Starlink measurements in CARL-W: Example time series of (a) latency and (b) packet loss rate compared to a 4G commercial network.

•
We present a parallel evaluation of 5G-NPN and Starlink systems, characterizing both in terms of several performance metrics, e.g., latency and throughput.