SoK: A Systematic Review of TEE Usage for Developing Trusted Applications

Trusted Execution Environments (TEEs) are a feature of modern central processing units (CPUs) that aim to provide a high assurance, isolated environment in which to run workloads that demand both confidentiality and integrity. Hardware and software components in the CPU isolate workloads, commonly referred to as Trusted Applications (TAs), from the main operating system (OS). This article aims to analyse the TEE ecosystem, determine its usability, and suggest improvements where necessary to make adoption easier. To better understand TEE usage, we gathered academic and practical examples from a total of 223 references. We summarise the literature and provide a publication timeline, along with insights into the evolution of TEE research and deployment. We categorise TAs into major groups and analyse the tools available to developers. Lastly, we evaluate trusted container projects, test performance, and identify the requirements for migrating applications inside them.


INTRODUCTION
Often, sensitive data is processed on general-purpose operating systems (OSs) which are prone to compromise due to the large number of complex features and services they support.Typically, when an OS is compromised, the applications and their data are also compromised [61].For instance, if an adversary takes control of an Internet of Things (IoT) device or a cloud instance, the adversary can also access the processes running there [61,174].
To help mitigate these risks, modern central processing units (CPUs) support a mode of operation that isolates the applications which manage sensitive data from the rest of the system.These isolated environments generally only support enough functionality to enable the processing of this sensitive data.This reduced functionality leads to less code, hence, a smaller Trusted Computing Base (TCB), which in turn enables us to derive trust in those components.Thus, these modes are generally referred to as Trusted Execution Environments (TEEs).
A TEE is a component of the CPU that comprises both hardware and software features with the aim of ensuring the confidentiality and integrity of the code and data loaded inside.The code that runs inside the TEE is often referred to as a Trusted Application (TA), although it does not have to be a full application in the traditional sense; it may comprise only the parts of a larger application that process sensitive data.
The end user of an application or a system is becoming increasingly aware of the need for security, however, they lack the technical knowledge to make informed decisions.As such, the onus is on the developers and maintainers of software to make the correct choices for the user in the most transparent manner possible.For this reason, we take the software developer's perspective and review TEE software development kits (SDKs) and trusted containers (tcons) in order to determine their usability and, consequently, the likelihood of their adoption by applications.Our research questions (RQs) are: RQ1.Which use case classification describes TAs? RQ2.Which SDKs are available for TA development?RQ3.What types of tcons are available?RQ4.What are the usability implications of porting existing applications to tcons?TEE implementations are available from a variety of hardware vendors, including AMD Secure Encrypted Virtualization (SEV), Intel Software Guard Extensions (SGX), Intel Trusted Domain Extensions (TDX), ARM TrustZone, and RISC-V Keystone [153].In addition to TEEs, there are a number of solutions that utilise trusted co-processors: AMD Platform Security Processor (PSP), Google Titan M, and Apple Secure Enclave Processor (SEP) provide many of the same benefits, however, they are discrete from the main CPU.
Cryptographic primitives are utilised extensively to ensure the confidentiality and integrity of the TEE/TA throughout its lifecycle.Attestation of the TEE assures that it is in a known good state before code is loaded; signed binaries ensure that only approved code is loaded; encrypted and integrity-checked data protects it from being read or modified by untrusted parties.All of this is bound to the CPU which protects it from the main OS, potential attackers, and also from the user.As surprising as it may sound, the legitimate Figure 1: TEE literature falls under the categories of review, framework, container, and application.user of a system may be considered a potential adversary in certain cases, for example in the case of Digital Rights Management (DRM) or selected banking operations where there is the potential for financial gain by subverting the system's normal operation.
This separation of TEE and Rich Execution Environment (REE) allows for more rigorous verification to be applied to the securitycritical parts of a system [8].
The scope of this review is to understand practical TEE deployments.Figure 1 illustrates our references, categorised and ordered by year.In cases of multiple references for the same topic, we list the most peer-reviewed one.
Based on our methodology, we systematically collected 208 of 223 references for this article from 11 March 2022 to 7 June 2022, and only added 15 references after this date.On 13 February 2023, we updated preprint papers with published versions for references that were officially published after our systematic collection of references, but we did not search for additional articles published after that date.Therefore, it is probable that we are missing references published after May 2022.
Our TEE reference search methodology encompasses both academic and applied activity.We discover that the publications fit well under the categories of review, framework, container, and application.The selection of these categories is based on the references' natural fit within them.
We notice that publication velocity increased after 2015.Between 2013 and 2015, there were few publications, but the rate of publication began rising in 2016.We believe this is likely due to the introduction of Intel SGX in 2015, which sparked a surge in interest in TEE-related research.Most ARM TrustZone deployments, which are the most common commercially available TEEs, are proprietary and designed for embedded use cases which renders them inaccessible to most researchers.19 of the 20 containers listed in Table 6 support Intel SGX hardware.Moreover, 70 of the 103 applications listed in Table 4 support SGX.These facts support the argument that SGX technology is the cause of an increase in publications after 2015.
The number of publications in 2018 remained almost unchanged from the previous year.In 2019, the number of publications nearly doubled and continued to rise in 2020 and 2021.There are more publications in 2021 than ever before, and we speculate this trend will continue in the foreseeable future.
In Figure 1, 103 of the 154 academic publications demonstrate applications.43 articles cover frameworks and containers.8 publications are reviews, including surveys and systematisation of knowledge.
This provides the motivation for our article: systematisation of knowledge is required because there are numerous articles but few reviews.These reviews have a limited perspective, typically cover only one piece of hardware, and do not seek real-world usage examples.Our knowledge systematisation assists software developers and encompasses heterogeneous hardware.
This preliminary categorisation of TEE papers shows how academic literature focuses on application demonstrations and helps form a basis for our research questions.It is the initial beginning of our extensive systematisation of knowledge.We base our suggestions on the number of citations for publications and the number of stars for repositories, which we collected between 23 January 2023 and 25 January 2023.We collected the number of citations from the IEEE Xplore1 , Springer Link2 , and ACM Digital Library3 databases.This methodology has limitations when it comes to technical and research papers published elsewhere: the collection method does not take these publications into account.
Similarly, GitHub4 alone is utilised to determine the number of stars for each repository.
The chosen articles and repositories give a great overview of some of the most important real-world TEE use cases.

Related work
As there is prior work on systematising TEE knowledge, we began studying publications and resources that organise TEE utilisation.These data sources cover the following topics.

Software development kits (SDKs).
Each CPU vendor has its own TEE.To assist TA development, there are numerous SDKs to aid the software developer [110].Intel SGX SDK [78], OP-TEE [199], etc. intend to make the development easier.

Trusted containers (tcons).
To execute an application within a TEE, a developer must apply framework-specific modifications to the original application, which can be a time-consuming operation.Trusted containers solve this usability issue by allowing direct execution of unmodified binary code within a TEE, or by performing automated transformations on source code prior to loading it into a TEE executable [10].Certain tcons support multiple hardware backends, eliminating the need for a software developer to make hardware design selections at the code level [110].We utilise the existing work on tcons by Liu et al. [110] in our categorisations in Table 5 and Table 6.Their work provides a comprehensive analysis of 15 existing tcon solutions' designs and implementations, highlighting the most common security pitfalls.We are not evaluating containers in terms of security, but rather analysing the software wrapper stack and hardware support of 20 containers.Additionally, we check which containers are open source and active as of 2022.We conclude by comparing the active tcons, benchmarking the performance of various tcons, and discussing the usability of the tcons from our perspective.
Applications of TEEs.Tamrakar [189] covers several applications of TEEs, including attestation mechanisms and access control.Our categorisation of TEE utilisation in Table 4 is not based on said work, yet we included the applications presented therein.We also used the study of attestation mechanisms for TEEs by Ménétrey et al. [117] for systemising knowledge of TEE attestation applications.Dangwal et al. [39] explore how TEEs can be used in conjunction with security technologies such as homomorphic encryption and differential privacy for efficient software-hardware-security codesign.They propose that security techniques must be combined in order to overcome the inherent limitations of existing technologies.
Curated lists of TEE publications.Schiavoni [168] maintains a curated list of SGX papers while Novella [135] maintains a similar list for TrustZone publications.Whereas the former aims to list all peer-reviewed publications regarding SGX, the latter focuses on attacks against TrustZone-based TEEs and is primarily composed of technical reports, blog postings, and hacking conference presentations.TEE hardware security.Zhao et al. [219] systemise knowledge of hardware security support for TEEs.Schneider et al. [169] present a systematisation of knowledge pertaining to how various hardwarebased TEE solutions meet the security goals of verifiable launch, run-time isolation, trusted I/O, and secure storage.This survey is valuable for understanding how present TEE solution designs achieve their security goals and how existing knowledge can be applied to the development of future TEE solutions.
Attacks against TEEs.There are also other surveys on TEEs not directly relevant to our work.For example, presenting how TEEs reduce the attack surface but do not eliminate it.Numerous attacks have been launched against TEE protection mechanisms and TA implementations [57].Researchers and practitioners target security flaws and propose solutions for real-world applications, for example, Cerdeira et al. [24] and Koutroumpouchos et al. [95] present a security analysis of popular TrustZone-assisted TEE systems.Akram et al. [3] present a systematisation of knowledge pertaining existing TEEs, highlighting common mechanisms of security guarantees, and offering comparative analyses of different TEE proposals.They also bring up the current limitations of TEEs for high-performance computing systems.

TEE use cases
A TEE technology provides extra protection for various sensitive applications.The following are the most prevalent usage scenarios [189]: Digital rights management.Copyright holders frequently use TEEs to prevent consumers from copying video or audio [147].TEEs protect digitally encoded media on connected devices, including smartphones, tablets, and high-definition televisions [11,53].Along with the fact that the TEE and the device's display are connected via a protected hardware channel, this prevents the device's owner from reading stored secrets.
Online payments.Mobile wallets, peer-to-peer payments, cryptocurrency wallets, and the use of a mobile device as a point-of-sale terminal -all have well-defined security requirements.Blockchain systems use lightweight clients, which outsource the computational and storage load over full blockchain nodes [114].It is possible to use TEEs to protect the privacy of the light clients without compromising the performance of the assisting full nodes [114].TEEs can be used as trusted backend systems to provide the required security to facilitate financial transactions.This may necessitate the entry of a PIN, password, or biometric identifier by the user.
Authentication.TEEs are commonly used to implement biometric identity methods (facial recognition, fingerprint sensor, and voice authorisation).For instance, Android OS can save fingerprint biometrics in the TEE because it is inaccessible and encrypted from the ordinary OS environment [84].Often, biometric identifications are convenient to use and more difficult to steal than PINs and passwords.TEEs can be utilised to protect the biometric identification method.However, increasingly, biometric data is being stored and verified directly on the sensors and only an attestation is shared with the TEE.Similarly to biometric identification information, cryptographic private keys can also be stored in the TEE.Combining the biometric identification information and the private keys allows passwordless authentication standards such as Apple's passkeys [7].
Trusted cloud.Typically, when a cloud (the server or the backend) is compromised, the adversary gains access to the cloud's processes and data.TEEs provide protection against compromised infrastructure: the adversary is unable to access selective parts of the TA, which safeguards sensitive code and data.
Privacy-preserving data analysis.Machine learning has become an essential part of data processing in several application domains, such as healthcare, stock prediction, and artificial intelligence.Sometimes these applications process sensitive data, and to protect said data, a TEE-based solution can be used to maintain the integrity of the machine learning process and prevent attacks [32].
Runtime integrity.TEEs can be used for runtime integrity, such as real-time kernel protection.If an attacker targets kernel binaries, the security monitoring service can shut down if it is isolated in a secure environment [15].

Secure modular programming.
As it decouples functionalities into small, self-contained modules, modular programming is an efficient way to build software architectures for software assets that encourages reuse.In this instance, each module contains everything necessary to perform its intended function, and the TEE permits the execution of the module while protecting it from the vulnerabilities of other modules.

METHODOLOGY 2.1 Collecting references for the review
We began our search for scientific literature with Google Scholar 5 , arXiv open-access archive 6 , the DBLP computer science bibliography7 , Andor8 , ACM Digital Library, and IEEE Xplore using TEE-related search terms, such as "TEE", "Trusted Execution Environment", "OP-TEE", "TrustZone", "(Intel) SGX", "AMD SEV", "confidential computing", etc.While this paints an overall picture of TEE-related scholarly work, it does not cover more applied aspects, such as toolkits and deployments.
To address this gap, we then mined real source code using the Sourcegraph9 search engine, to find examples of practical TEE utilisation.Sourcegraph covers GitLab10 , GitHub, and BitBucket11 , as well as other public software source repositories.Table 3 details our search terms regarding Sourcegraph, with examples 12 .The most difficult aspect of the mining process was locating appropriate TEE applications, development frameworks, and container repositories.Typically, a keyword search yields thousands of repositories.These repositories contain OSs and kernels, as well as forks and projects with work-in-progress status.Furthermore, we specified "code" as the search type, then sorted and filtered the results to identify the most relevant ones, then finally, manually examined the results.
We based our selection of important phrases on the constants, variables, and functions utilised in the source code of each TEEbased application.The alternative method for picking specific search phrases was to consult the documentation of various TEE-based frameworks and containers, such as the GlobalPlatform API [68].It reveals applications and other frameworks, containers, and repositories.However, this required combing through each repository manually to obtain the desired results.

Dimensions for knowledge systematisation
Based on related work and our observations while gathering and reviewing the publications, we organise the TEE literature and practical work.
In Section 3 we address RQ1.Our goal is to assist the reader in comprehending TEEs, how they are utilised, and when, how, and why they could be used.To accomplish this, we tag the applications with 92 distinct keywords, which we merge into 21 primary categories and seven distinct security properties and mechanisms based on initial similarities.We discover that the primary 21 use cases for TEEs in application development are data analytics, cloud computing, access control, data protection, online payments, memory protection, attestation tools, secure storage, network security, secure channels, content sharing, secure code offloading, smart contracts, computer games, hardware accelerators, formal methods, medical data, secure system logging, web search, data trading, and digital contracts.Additionally, we discover that the main security properties and mechanisms related to the use cases are privacy, integrity, confidentiality, cryptography, attestation, blockchain, and decentralisation.This is the classification we utilise while reviewing existing TA demonstrations and practical implementations.Figure 2 illustrates our categorisation of the key use cases and the related security properties and mechanisms.Only the strongest relationships are shown in the figure.The size of a category, security property, or mechanism approximately corresponds to its prevalence in existing implementations.Table 4 shows which applications are related to each primary category.
In Section 4 we address RQ2.We compare the software frameworks targeting developers.It is difficult to compare TEE software development tools due to a lack of similar work and public information about their features.Hence, we compile Table 5 detailing the available tools, their supported programming languages, their software licences, the hardware architecture they support, and whether or not they are actively developed as of 2022.
In Section 5 we address RQ3.We organise the tcons for the developers.Again, it is difficult to directly compare tcon tools due to the absence of shared and unique characteristics.In addition, some containers are not actively developed, while others, such as the Enarx container [54], are updated every month with new features.In response, we compile Table 6, which details the available tools, interfaces, software licences, activity of the container project as of 2022, and hardware supported by each tcon.
In Section 6 we address RQ4.We compare the actively developed containers.Additionally, we present Figure 3, which exemplifies a required tcon-specific modification to existing code, demonstrating how challenging it can be to use tcons.Finally, we benchmark the performance of an existing test application using various tcons and present the findings in Table 7.

Limitations and bias
Lack of documentation of closed-source systems.Companies that own proprietary solutions utilising TEEs typically withhold information about their systems from the public.Therefore, it is difficult to find detailed information regarding closed-source solutions that employ TEEs.Because of this, the data we gathered might be biased towards open-source software and might not show the whole picture of reality.For example, there may be several more closed-source applications and development frameworks for ARM TrustZone than we present in this paper.
Lack of citation data.Certain venues or sources do not disclose the number of citations.This imposes restrictions on which technical and research papers can be listed in Table 1.
Date of initial release.It is often difficult to discover when a specific application, framework, or container was first released.Due to this, the publication year information in Figure 1 may not be entirely accurate.
Manual keyword search.The likelihood of omitting relevant repositories is the most significant shortcoming of a manual search.Although using Sourcegraph as a repository search engine simplifies the search process, it also generates a large number of irrelevant results.There is a chance of missing other applications, development frameworks, and containers that employ different keywords not on our list.

RQ1: Which use case classification describes TAs?
The TEE isolates and protects the TA code and data in terms of confidentiality and integrity.While we may be unaware, there are a large number of gadgets around us, most notably smartphones, set-top boxes, videogame consoles, and Smart TVs, that utilise a TEE.The number of gadgets utilising a TEE that are designed for many different purposes results in a wide range of use cases.These use cases vary from everyday user applications to backend services, such as mobile financial services and cloud services [189].To address RQ1, Table 4 combines TEE application scenarios based on our categorisation.
We gathered a total of 103 application use cases.The categories and the number of references corresponding to each category are the following: Data analytics (18), Cloud computing (14), Access control (14), Data protection (10), Online payments (8), Memory protection (8), Attestation tools (7), Secure storage (7), Network security (7), Secure channels (7), Content sharing (7), Secure code offloading (7), Smart contracts (5), Computer games (4), Hardware accelerators Previous Screens: Spatial-Temporal Recreation of Android App Displays from Memory Images [36,159] • Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware [195,196] towards model privacy at the edge using TEEs [126,127] Atlas: Automated Scale-out of Trust-Oblivious Systems to TEEs [13, 66] Exploring Widevine for Fun and Profit [147] • A ledger for confidential computing shims for tracking memory management system calls [56] •  According to Table 4, the vast majority of TEE applications operate on Intel SGX, ARM TrustZone, or both.Only a minority of applications operate on other platforms such as AMD SEV, RISC-V, or GPU TEEs.While most of the references we collected fit within the 21 categories outlined in Section 2, five applications did not fit into any of these categories.
On this basis, the majority of TEE applications aim to provide privacy-preserving data analysis (including machine learning applications).Cloud computing is frequently associated with machine learning applications and is the second-largest TEE usage category in our listing.Application domains surrounding access control, data protection, online payments, and memory protection are also among the most common use cases for TEEs.Albeit noticeably less prevalent than the use cases previously stated, attestation tools, secure storage, network security, secure channels, content sharing, and secure code offloading are all prominent use cases as well with seven references each.Smart contracts, computer games, hardware accelerators, formal methods, and medical data are also fairly prevalent use cases, with three to five references each.The remaining categories represent highly specific TEE use cases with few existing applications.Examples include web search data protection, digital contract signing, and secure system logging.
The number of applications utilising TEEs has steadily increased since 2015.52 of the 103 references we collected are from 2020 or after, and 48 applications have been deployed to actual users, according to our study.40 of these 48 applications deployed to actual users are licenced under an open-source licence.Notably, despite this, a large number of proprietary applications with closed-source licences comparable to the Widevine DRM component [147] utilise TEEs.Typically, these proprietary applications are not accompanied by any public documentation or scholarly studies, hence they are largely absent from our work.

TOOLS FOR DEVELOPING TEE SOFTWARE
RQ2: Which SDKs are available for TA development?
Numerous middleware frameworks are available to assist developers with TEE development, deployment, and maintenance.To address RQ2, Table 5 combines tools for developing TEE software.In Table 5, we highlight open-source SDKs that are currently being actively developed.Four of the open-source frameworks, such as Webinos [212], are deprecated and no longer under active development.Although there are minor updates, Open-TEE [115] is no longer undergoing substantial development.For our purposes, we consider a project active if there are software updates in 2022, which we assessed on 6 November 2022.
There is a wide selection of frameworks available to software developers for a variety of hardware architectures.The frameworks mentioned are available as open-source software or as brandfocused commercial solutions from various manufacturers, such as the Samsung Knox SDK for Samsung Android devices [162].11 of the 23 referred frameworks support Intel SGX, while 13 frameworks support ARM TrustZone, as Table 5 shows.Notably, 21 of the 23 referred frameworks support either Intel SGX or ARM TrustZone, or both.
We researched and compiled a list of supported software languages for active SDK projects.We obtained this information from the SDKs' documentation and examples.This is a limitation, as we can only include supported language information from documented open-source SDKs; thus, these SDKs might have wider non-documented language support.We found that the main languages supported by active SDKs are C and C++. 12 SDKs support at least one of these two languages.Four of them also work with Rust, four work with Java, one supports Go (Edgeless RT), and one supports JavaScript (Confidential Consortium).
The frameworks serve a variety of practical purposes in order to facilitate development efforts.Several frameworks focus on mobile devices and wearables, where the intent is to provide ready-made APIs to support application development [161,162,165].The framework references are also focused on IoT devices or web applications, but due to the wide range of programming language support, the frameworks cover also many other areas [121,133,201,212].Some of the frameworks are focused on or support very niche areas, like Trustonic's Kinibi-520a SDK [73], where Symmetric-Multi-Processing enables the development of biometric functions like fingerprint scanning and face recognition.
The choice of development framework by the developer is usually severely constrained by the hardware architecture.For example, developers of mobile applications can only use options that are compatible with ARM TrustZone.We find that open-source development frameworks, such as OP-TEE [199], Open Enclave SDK [141], Teaclave TrustZone SDK [192], and Trusty TEE [5] support TrustZone at least in some capacity and are still actively maintained.
These frameworks may provide open-source alternatives for mobile application developers, who have traditionally been limited to proprietary closed-source frameworks, such as the Samsung Knox SDK [162] or Trustonic's TEE SDKs [73,200,201].Nevertheless, many open-source frameworks only support specific platforms, so proprietary SDKs may remain the only option for developers on unsupported platforms.
For an application to function on any TEE technology, the development process must follow framework-specific design solutions.This makes the procedure difficult and time-intensive for application developers.In addition, a developer must implement attestation to trust the application.To address the usability issue with different TEE technologies, a set of tcons enables either the direct execution of unmodified binary code inside a TEE or automatic transformation of source code prior to loading it into a TEE executable [110].In order to address RQ3, Table 6 enumerates tcons.
We collected 20 distinct containers, identified the supported hardware and application middleware interfaces, and determined whether or not the project is open source and active.17 of the 20 referred tcons are open-source software.If there are software updates in 2022, we consider the tcon project to be active.We evaluated this on 13 October 2022.The open-source tcons saw development activity in the following years: MesaPy (2018); AccTEE (2020); Deflection, GoTEE, Ratel, Ryoan, SGX-LKL, Twine (2021); vSGX, Enarx, Apache Teaclave, EGo SDK, Fortanix EDP, Gramine, Mystikos, and Occlum (2022).Accordingly, there are eight active tcon projects.
We discover that 19 of the 20 tcons support Intel SGX and only three support AMD SEV.In addition, we find no tcons that support TrustZone TEE technology, confining mobile application developers to SDKs.A recent trend seems to be containers that support multiple hardware architectures.The objective is to allow developers to adapt the same program to many platforms without having to alter the source code.Enarx [54] is a good example of such a tcon.Recently published vSGX [220] supports directly running SGXenabled applications inside AMD SEV.
A system call is an interface between software and the OS through which applications can request services from the OS.Since Intel SGX restricts applications from making system calls, unmodified applications cannot be executed within an enclave.
Seven tcons utilise library OS (LibOS): the missing OS interface that either natively or transparently relays in-enclave system calls to the OS outside the enclave.The LibOS concept predates TEE technologies by at least a decade, motivated by applications in the embedded space due to severe resource constraints [152].LibOS is an approach to operating system design and implementation where the traditional functionality of an OS is provided by a set of libraries.These libraries are linked directly into the application to create a single address space executable.By encapsulating the operating system functionality within libraries, it becomes easier to define and enforce boundaries between different components, while reducing the TCB.This enables developers to implement security policies at the application level, restricting access to sensitive resources, and preventing unauthorised access to data or interference with other processes.In addition, because the OS primitives are included in the application, this removes the need to invoke system calls and hence, reduces the context switches between user space and kernel space, thus improving performance.
All of this makes LibOS an ideal candidate for use in a TEE, either as a set of standalone applications or as a wrapper around existing applications to reduce the porting effort, e.g., by intercepting system calls from the application and replacing them with LibOS-specific ones.
Six of the 20 tcons utilise wrappers around the C standard library (libc) as an application middleware interface.Executing a system call with libc wrappers, such as EGo SDK, is equivalent to requesting the untrusted OS to perform the corresponding operations outside of the enclave.
The WebAssembly System Interface (WASI) works in a similar fashion and restricts system calls.It provides a runtime for We-bAssembly (WASM) binary execution within a TEE [110].Of the referred 20 tcons, four utilise WASI: AccTEE, Apache Teaclave, Enarx, and Twine execute WASM binaries within a TEE.
From 13 October 2022 to 22 November 2022, we collected and compared the number of Linux system calls against the number of implemented system calls in Gramine, Occlum, Mystikos, Enarx, and Fortanix EDP.For reference, the Linux kernel has a total of 451 system calls, including outdated system calls 13 .Gramine implements 166 system calls 14 .Occlum has 159 implemented system calls 15 .Mystikos implements 102 system calls 16 .Enarx implements the sallyport 17 proxying service for service calls and executes WASM binary within a TEE Keep.The sallyport protocol implements 31 system calls from a Keep to the host 18 .Fortanix EDP -specifically, its user-call API -implements 16 system calls 19 , purposefully kept simple to facilitate security audits.
From a security point of view, these tcons increase the TCB.However, if we examine a realistic TA application like machine learning with Python, it has very few system interactions.The developer chooses between implementing the whole software stack from scratch with some SDK or using a tcon; both options have pros and cons.

TESTING TCONS
RQ4: What are the usability implications of porting existing applications to tcons?
We compare tcons that are actively in development.A project is deemed active if software updates are released in 2022.Based on the comparison, we deploy and run a benchmark application within the suitable tcons.Realistically, a software developer would choose between the eight active tcon projects based on the functionality they provide.

Are tcons easy to use?
The trusted containers that use WASM (and Wasmtime) promote the phrase "put your app in a container".When we tested this, we discovered restrictions imposed by WASM that contradict this statement.
While testing WASM containers such as Enarx, we discovered the following obstacles: (1) The selected programming language needs to have native support for WASM development -for instance, Rust.
(2) Even with a properly chosen programming language, routine standard library operations like threading and networking may require a redesign of the application.(3) As a result, the majority of Rust's libraries are inoperable by default because they depend on standard libraries.For example, a programmer cannot use existing HTTP libraries to execute an HTTP GET request.(4) Instead, lowlevel code may be a requirement for even a simple task where you would normally just use one line to call a library.(5) A programmer eventually needs to add .cargo/configconfigurations, macros, and dependencies that are unique to WASM.(6) Development requires mappings between software code and the tcon, for example, preopened sockets need to be defined in the Enarx.tomlconfiguration file.
In certain situations, standard libraries need to be replaced with alternatives that support WASM.For instance, Tokio 20 is an eventdriven, non-blocking I/O platform for developing asynchronous Rust applications, with unstable support for some extra WASM  features.However, not all methods are available.For example, new sockets cannot be created from within WASM.Instead, the code must catch the sockets that the tcon creates, as demonstrated by Figure 3.
As a test, we rewrote a TCP proxy application using Tokio with unstable WASM support.In the code, we define WASM build sections with macros and use them to catch the pre-opened sockets.We utilise the "current_thread" macro for threads instead of using a thread pool.Until Wasmtime supports a large number of standard library requirements, it is difficult to simply "put your app in a container".
Using LibOS-based tcons, such as Gramine, Mystikos, and Occlum, we were able to launch diverse applications without modifications.Therefore, it is easier to utilise LibOS-based tcons than those that require WASM.

Performance of tcons
We test the general-purpose containers Enarx, Gramine, Mystikos, and Occlum to benchmark and execute a Rust application.We select these tcons because they are actively developed, can execute Rust applications, and support Intel SGX.As a comparison, we execute the WASM binary without a secure enclave using Wasmtime.As a second comparison, we test Enarx using AMD SEV hardware instead of Intel SGX hardware.
We forked a paper-based backup scheme application 21 that generates encrypted backups and splits the secret key into multiple key shards that can be held independently by different users (Shamir's Secret Sharing).We select this application because it is utilised in the real world, is built in Rust, and Rust supports WASM builds.Due to the file interface constraints of tcons, we hard-coded the values in order to test the application.
The time to load a container varies greatly, however, we did not compare this metric because it depends on whether the container performs the attestation process.In order to enable attestation,  22 .We measured the real execution time of the application: the main function prints out the duration of the entire code execution.The variance was very low, so the average of 100 samples accurately reflects the execution time and the number of CPU cycles.Notice that CPU cycles take into account the full launch of the tcon and execution of the application, but for the time we measured only the execution time of the application inside the tcon.This means that the number of runtime seconds does not necessarily correlate with the number of CPU cycles.We repeated the Intel SGX tests and Wasmtime tests in the same environment.Unexpectedly, code execution on Occlum takes 2.34 seconds, whereas on Enarx it takes 0.39 seconds while utilising the same WASM binary.There is no obvious reason why Occlum execution is significantly slower.In comparison, without the enclave, the execution time is 0.81 seconds, which is noticeably longer than the Enarx execution time.We used the same build, wasm32-wasi target file, for both execution with Enarx and Wasmtime, yet Enarx is faster, so we suspect that its WASM runtime is lightweight.With Enarx, the application runtime is similar to that on AMD SEV and Intel SGX hardware, but the number of CPU cycles greatly differs.The number of CPU cycles is not comparable in this case because these are different pieces of hardware.As a result, as expected, using a tcon adds overhead to the execution and, when compared to Wasmtime, requires more CPU cycles.

CONCLUSION
This article organises TEE applications, frameworks, containers, and reviews in order to determine historic use and usability factors.Our key conclusions are as follows: Open-source TEE SDKs help TA creation.Typically, a developer must make laborious framework-specific modifications to the original application in order for it to run within a TEE.We listed 23 SDKs available to aid developers with TEE deployment, out of which 17 are open-source software.
Open-source tcons are gaining popularity.A trusted container (tcon) solves the usability issue raised in the previous paragraph by enabling either the direct execution of unmodified binary code within a TEE or the automatic transformation of source code prior to loading a TEE executable.We provided a list of 20 tcons that eliminate the need for software developers to use specific SDKs to write TEE-related code, out of which 17 are open-source software.
Current tcons are not as easy to use as advertised.Our experiments indicate that tcons are not as simple to utilise as advertised.Particularly WASM-based tcons impose strict limitations, necessitating a rewrite of the software's source code and the creation of separate configuration files.In addition, the application must be written in a language that supports WASM natively.
We benchmarked tcons.We benchmarked Enarx, Gramine, Mystikos, and Occlum tcons with the Intel SGX backend.As a comparison, we also ran a WASM binary without a TEE using Wasmtime and Enarx with an AMD SEV backend.Using the identical WASM binary, code execution varied from 2.34 seconds with Occlum to 0.39 seconds with Enarx.According to the measurements, tcons add overhead to the execution and need 1.4 to 16 times more CPU cycles than Wasmtime.
Intel SGX and ARM TrustZone are the most researched.Most of the publications demonstrate application use cases, and Intel SGX is the most popular hardware for applications.In fact, 93 out of 103 TEE applications utilise either Intel SGX, ARM TrustZone, or both.Only a small number of applications can run on other platforms such as AMD SEV, RISC-V, or GPU TEEs.
Current tcons support primarily Intel SGX or AMD SEV.The choice of SDK and tcon by the developer is severely constrained by the hardware architecture.For instance, mobile application developers are restricted to options that are compatible with ARM TrustZone, which means there are no tcons available and a limited number of SDKs to choose from, the majority of which are closed-source frameworks.Some recent tcons, such as Enarx [54] and vSGX [220], enable the execution of the same application within TEEs based on multiple hardware architectures without requiring source code modifications (in theory).Typically, though, tcons only support Intel SGX, AMD SEV, or both.
Data analytics is the most common application category for opensource TAs.Additionally, we examined the primary elements of the execution and the data people attempt to secure with their TAs.We gathered a total of 103 application use cases in Table 4. Data analytics (18 references), Cloud computing (14 references), and Access control (14 references) are the most common of the 21 primary drivers to use TEE.
RISC-V, Sanctum, and Keystone.According to academic references, RISC-V TEE technologies are interesting, but few publications are available about them.The scientific community is ideally suited to pursue the objective of open-source hardware, which is undeniably a concrete development step.The objective is to create a secure and trustworthy hardware-backed enclave for RISC-V.Sanctum [34] and Keystone [100] are seminal steps in this direction, yet we are unaware of any deployments.This lack of mainstream hardware inhibits the growth of the surrounding software ecosystem, somewhat analogous to TrustZone-based TEE technologies, such as On-board Credentials (ObC) [52] in the 2000s: it is clear that ObC predates unified TEE software architectures, such as the GlobalPlatform API [68], yet such standardisation and unification efforts arguably emerged too late to prevent fragmentation of the software ecosystem.In summary, as a community, we should steer TA software development in a consistent and narrow fashion, and to achieve this, we need mainstream hardware available with TEErelevant hardware-assisted security features that are open source and accessible to developers.

FUTURE WORK
RISC-V and applying lessons learned.Several factors have negatively influenced the adoption of TEEs in the past.Moreover, we identified a key research topic based on the analysis of real-world threats and effective mitigation techniques that are most relevant for TEE implementations.With RISC-V as an emerging technology, the standardisation of TEE mechanisms for RISC-V is an excellent opportunity to not only apply valuable lessons learned but to drive the development toward a secure and useable TEE.

Figure 2 :
Figure 2: We define which classification aggregates the usage examples after reviewing the TEE example applications.Only the most significant relationships between the categories and the security properties and mechanisms are depicted.

22 asyncFigure 3 :
Figure 3: This snippet of a TCP proxy application demonstrates WASM and tcon-specific modifications regarding sockets and threading.Line 31 catches the pre-opened socket.

Table 1 :
Most cited TEE-related articles.Since our survey comprises of 223 references, we wish to highlight some of the most essential ones.In Table1and Table2, we recommend 11 TEE-related articles and 10 repositories, respectively.

Table 3 :
Using the Sourcegraph search engine, we compile real-world applications of TEEs with the provided search terms.

Table 4 :
We classified TEE application scenarios into 21 groups.
Enha92]ng Security and Privacy of Tor's Ecosystem by Using TEEs[86,92] [23], and Fair Query Service for Light Clients on Public Blockchains[23]