Confidential Container Groups

The experiments presented here demonstrate that Parma, the architecture that drives confidential containers on Azure container instances, adds less than one percent additional performance overhead beyond that added by the underlying TEE. Importantly, Parma ensures a security invariant over all reachable states of the container group rooted in the attestation report. This allows external third parties to communicate securely with containers, enabling a wide range of containerized workflows that require confidential access to secure data. Companies obtain the advantages of running their most confidential workflows in the cloud without having to compromise on their security requirements. Tenants gain flexibility, efficiency, and reliability; CSPs get more business; and users can trust that their data is private, confidential, and secure.


S
ince the launch of large-scale IaaS (infrastructure as a service) offerings from Amazon (Amazon Web Services in 2006), Microsoft (Azure in 2008), and Google (Google Cloud Platform in 2008), there has been a continuous trend toward cloud computing, which allows customers to leverage capability and cost advantages through economies of scale.This was made possible through virtualization, 3 where VMs (virtual machines) allow the efficient use of large bare-metal compute architectures (hosts) by using a hypervisor to coordinate sharing between multiple tenants according to their expressed usage requirements.While VMs provide a way for users to obtain additional compute capacity quickly and maximize the utilization of existing hardware (and/or avoid the cost of maintaining peak capacity by using a public cloud), it is still necessary to configure, deploy, manage, and maintain them.
In recent years, container-based technologies such as Docker and Kubernetes have arisen to make it easier to manage software application lifecycles.They provide a lightweight solution for creating a set of machine configurations, called containers, which can be deployed onto hardware (virtualized or physical) as a Implementing confidential computing on Azure container instances group via an automated process.Container technology provides multiple separate user-space instances that are isolated from one another via kernel software.Unlike VMs, containers run directly on the host system (sharing its kernel) and as such, do not need to emulate devices or maintain large disk files.Further, container definitions adhering to the OCI (Open Containers Initiative) Distribution Specification 13 specify dependencies as layers that can be shared between different containers, making them amenable to caching and thus speeding up deployment while reducing storage costs for multiple containers.
The success of containerization technology for onpremises systems has led to major cloud providers developing their own CaaS (containers as a service) offerings, which provide customers with the ability to maintain and deploy containers in the public cloud.In CaaS offerings, containers run in a per-group UVM (utility virtual machine), which provides hypervisor-level isolation between containers running from different tenants on the same host.While the container manager and container shim running on the host are responsible for pulling images from the container registry, bringing up the UVM, and orchestrating container execution, an agent running in the UVM (the guest agent) coordinates the container workflow as directed by the host-side container shim.
Cloud computing poses challenges for systems that require confidentiality, 7 because the three groups involved have competing needs: 3 CSP (cloud service provider) -The owner and operator of a public cloud.
3 Tenant -A customer of the CSP, who uses the cloud to host applications.3 User -A customer or employee of the tenant, who uses the tenant's cloud applications.
Although VMs and VM-isolated container groups provide strong isolation between tenants, they are deployed by the CSP and coordinated by the CSP's hypervisor.As such, in existing cloud offerings, the host operating system (including the container manager and container shim) and the hypervisor lie in the tenant's TCB (trusted computing base) and consist of all the hardware and software a tenant is required to trust.As a rule, the TCB should be as small as possible.Therefore, research into confidential computing 22 proposed reducing the size of the TCB by leveraging hardware-enforced TEEs (trusted execution environments). 2,4,5,6,8,10,11,16,17EEs isolate the code and data of an application from the rest of the system, including privileged components.They protect the integrity and confidentiality of user workloads, even if the host's software is compromised or controlled by a malicious entity.TEEs available from major CPU vendors can be either process-based, such as Intel SGX; 19 or VMbased, such as AMD SEV-SNP, 9 Intel TDX, 20 and ARM CCA. 21M-based TEEs offer hardware-level isolation of the VM, preventing the host operating system and the hypervisor from having access to the VM's memory and registers.
With CaaS, container execution is orchestrated by a host-side shim that communicates with the guest agent, which coordinates the activity of the container group in the UVM.The UVM can be hardware-isolated in a TEE enclave, but the container images are still controlled by the host, as are the order in which they are mounted, the container environment variables, the commands that are sent to the containers via the bridge between the container shim and the guest agent, and so forth.This means that a compromised host can overcome the hardware isolation of the VM by injecting malicious containers.This risk of attack-be it from malicious or compromised employees of the CSP, or external threats-limits the extent to which containerization can be used in the cloud for sensitive workloads in industries such as finance and healthcare.
The naïve solution to this problem is to run the guest agent and container shim in the same VM-based TEE.This removes the CSP from the tenant's TCB, but it also takes away the CSP's ability to orchestrate and automate the container workflow.It also leaves the tenant in the user's TCB.This is because the container images are controlled by the tenant, as is the order in which they are mounted, the container environment variables, and the commands that are sent to the containers.The user of the confidential container (e.g., a bank customer or a patient providing data to a doctor) must then trust that the tenant has and will run only the expected commands.Finally, this leaves image integrity and data confidentiality/integrity unsolved.
To provide image and data integrity while keeping both the CSP and the tenant out of the user's TCB, we built Parma, which powers confidential containers on Azure container instances.It implements the confidential containers abstraction on a state-of-the-art container stack running on processors with VM-based TEE support (i.e., AMD SEV-SNP processors).Parma provides a liftand-shift experience and the ability to run unmodified containers pulled from (unmodified) container registries, while providing the following strong security guarantees: 3 Container attestation and integrity.Only customerspecified containers can run in the TCB, and any means of container tampering is detected by the TCB and will either cause deployment of the container group to fail (if the execution policy is intact) or remote attestation to be invalid (because the attested policy is different).3 User data confidentiality and integrity.Only the TCB has access to the user's data, and any means of data tampering is detected by the TCB via evidence made available to the user.Figure 1 shows how Parma uses execution policies to enforce user constraints and provide security to a container group.An execution policy is a component of a confidential VM that is attested at initialization time, and describes all the actions the tenant has explicitly allowed the guest agent (operated by the CSP) to take in the container group that runs on that VM. Figure 1(a) shows how Parma limits the user TCB to the container group itself, isolating it from the host operating system and hypervisor.Figure 1(b) shows an unsuccessful mount action (requested by the host and denied by the guest), in which a layer of a container image does not have a dm-verity (device-manager's "verity" target) root hash that matches a hash enumerated in the policy.
Parma provides strong protection from the CSP for the container's root file system (consisting of the container image layers and writable scratch space) and the user's data.For container image layers (pulled in plaintext by the untrusted container manager and stored in a host-side block FIGURE 1: execution policies device), Parma mounts the device as an integrity-protected read-only file system and relies on the file-system driver to enforce integrity checking on an access to the file system.For confidentiality and integrity of privacy-sensitive data stored in a block device (e.g., writable scratch space of the container's root file system) or blob storage (e.g., remote blobs holding user data), Parma relies on block-level encryption and integrity to decrypt memory-mapped blocks, guaranteeing that data appears in plaintext only in the VM's hardware-protected memory.
Finally, Parma provides container attestation rooted in security hardware by enforcing tenant-specified execution policies.The guest agent has been augmented to enforce the execution policy such that it executes only those commands (submitted by the untrusted container shim) that are explicitly allowed by the tenant, as seen in figure 1.The policy is attested by encoding its measurement in the attestation report as an immutable field at UVM initialization.
As a result of including the execution policy, the hardware-issued attestation describes the initial state of the container group and all its allowed state transitions.The attestation can then be used downstream by users for operations that require it to establish security guarantees.For example, remote verifiers may release keys (governing the user's encrypted data) to only those container groups that can present an attestation report encoding the expected execution policy and measurement of the UVM.which uses attested execution policies.First the container platform, which forms the basis of the Parma design and implementation, is described, followed by a detailed description of the threat model.This section also presents the security guarantees under the threat model and describes how Parma provides these guarantees via a collection of design principles.

ARCHITECTURE
The guiding principle of Parma is to provide an inductive proof over the state of a container group, rooted in the attestation report produced by the PSP (platform security processor).The standard components and lifecycle for the CPLAT (container platform) are largely unchanged, except for the guest agent, whose actions become circumscribed by the execution policy.Thus constrained, the future state of the system can be rooted in the measurement performed during guest initialization.CPLAT is a group of components built around the capabilities of containerd, a daemon that manages the complete container lifecycle as seen in figure 2, from image pull and storage to container execution and supervision to low-level storage and network attachments.containerd supports the OCI image and runtime specifications, 13 and provides the substrate for multiple container offerings such as Docker, Kubernetes, and various public cloud offerings.Clients interact with containerd via a client interface.containerd supports running bare-metal containers (i.e., those that run directly on the host) as well as those that run in a UVM.

Container platform
VM-isolated containers are the focus here.CPLAT interfaces with a custom container shim running in the host operating system.The container shim interacts with (i) host services to bring up a UVM required for launching a new pod, and (ii) the guest agent running in the UVM.The guest agent is responsible for creating containers and spawning a process for starting the container.In essence, the container shim to guest-agent path allows the CPLAT components running in the host operating system to execute containers in isolated guest VMs.The execution of a VM-isolated container on Linux using CPLAT involves four high-level steps (as seen in figure 2), equivalent to the four commands in figure 3.

Threat model
Parma is designed to address the case of a strong adversary who controls the entire host system, including the hypervisor and the host operating system, along with all services running in it.That said, we trust the CPU package, including the PSP and the AMD SEV-SNP implementation, which provides hardware-based isolation of the guest's address space from the system software.We also trust the firmware running on the PSP and its measurements of the guest VM, including the guest agent.
Such an adversary can: 3 Tamper with the container's OCI runtime specification.These capabilities would allow an adversary to gain access to the address space of the guest operating system without the protection of Parma.

Security architecture
Under this threat model, the goal is to provide strong confidentiality and integrity guarantees for the container and customer data.We have used the following principles to design a security architecture that provides those guarantees.

Hardware-based isolation of the UVM
The memory address space and disks of the VM must be protected from the CSP and other VMs by hardwarelevel isolation.Parma relies on the SEV-SNP hardware guarantee that the memory address space of the UVM cannot be accessed by host system software.

UVM measurement
The UVM, its operating system, and the guest agent are cryptographically measured during initialization by the TEE, and this measurement can be requested over a secure channel at any time by tenant containers.The AMD SEV-SNP hardware performs the measurement and encodes it in the signed attestation report. 1

Verifiable execution policy
The tenant must be provided with a mechanism to verify that the active execution policy (see later in the section on Execution Policy) in a container group is what it is expected to be.The execution policy is defined and measured independently by the tenant and is then provided to the CaaS deployment system.The host measures the policy (e.g., using SHA-512) and places this measurement in the immutable host data of the report. 1 The policy itself is passed to the UVM by the container shim, where it is measured again by the tenant and the user to ensure that its measurement matches the one encoded as host data in the report.

Integrity-protected read-only file systems
Any block device or blob storage is mounted in the UVM as an integrity-protected file system.The file-system driver enforces integrity checking on access to the file system, ensuring that the host-system software cannot tamper with the data and container images.In Parma, a container file system is expressed as an ordered sequence of layers, where each layer is mounted as a separate device and then assembled into an overlay file system.First, Parma verifies as each layer is mounted that the dm-verity root hash for the device matches a layer that is enumerated in the policy.Second, when the container shim requests the mounting of an overlay file system that assembles multiple layer devices, Parma verifies that the specific ordering of layers is explicitly laid out in the execution policy for one or more containers.

Encrypted and integrity-protected read/write file systems
Any block device or blob storage that holds privacy-sensitive data is mounted as an encrypted file system.The file-system driver decrypts the memory-mapped block upon access to the file system.The decrypted block is stored in hardwareisolated memory space, ensuring that host-system software cannot access the plaintext data.The writable scratch space of the container is mounted with dm-crypt and dmintegrity, and this is enforced by the execution policy.The encryption key for the writable scratch space is ephemeral and is provisioned initially in hardware-protected memory, and erased once the device is mounted.

Remote attestation
Remote verifiers (e.g., tenants, users, external services, attestation services) need to verify an attestation report so that they can establish trust in a secure communication channel with the container group running in the UVM.They also need to verify that the UVM has booted the expected operating system, the correct guest agent, and further that the guest agent is configured with the expected execution policy.
In Parma, the UVM (including privileged containers) can request an attestation report using the secure channel established between the PSP and the UVM. 1 The requester generates an ephemeral token-for example, TLS (Transport Layer Security) public-key pair or a sealing/ wrapping public key-which is presented as a runtime claim in the report.The token's cryptographic digest is encoded as report data.A remote verifier can then verify that: 3 The report has been signed by a genuine AMD processor using a key rooted to AMD's root certificate authority.3 The guest launch measurement and host data match the expected VM measurement and the digest of the expected execution policy.3 The report data matches the hash digest of the runtime claim presented as additional evidence.
Once the verification is complete, the remote verifier that trusts the UVM (including the guest operating system, guest agent, and execution policy) then trusts that the UVM and the container group within it will not reveal the private keys from which the public tokens have been generated (e.g., TLS private key, sealing/wrapping private key).The remote verifier can use the runtime claim accordingly.For example: 3 A TLS public key can be used for establishing a TLS connection with the attested container group.As such, the remote verifier can trust there is no replay or manin-the-middle attack.3 A sealing public key can be used to seal (via encryption) a request or response intended only for the attested containers.3 A wrapping public key can be used by a key-management service to wrap and release encryption keys required by the VM's container group for decrypting remote blob storage.As such, the remote verifier can hold that only trustworthy and attested container groups can unwrap the encryption keys.Figure 4 illustrates this process, a typical attestation workflow.
In the figure, a container (key in circle) attempts to obtain and decrypt the user's data for use by other containers in the group.The key has been previously provisioned into a key-management service with a defined key-release policy.It follows these steps: (1) The container in the UVM requests that the PSP issue an attestation report, including an RSA (Rivest-Shamir-Adleman) public key as a runtime claim.(2) The report and additional attestation evidence are provided to the attestation service, which verifies that the report is valid and then (3) Finally, the attestation token is provided to the key-management service, which returns the user's key to the container wrapped using the provided RSA public key only if the token's claims satisfy the key-release policy statement.

Execution policy
As discussed earlier in this article, we do not trust the container shim, as it could be under the control of an attacker.This implies that any action that the container shim requests the guest agent undertake inside the UVM is suspect (see the Threat Model section for a list of malicious host actions).Even if the current state of the container group is valid, the host may choose to compromise it in the future, and thus the attestation report cannot be used as a gate for access to secure user data.The attestation report on its own simply records the UVM operating system, guest agent, and the container runtime versions in use.It makes no claims about the container group the host will subsequently orchestrate.
For example, the host can start the user container group in a manner expected by an attestation service until it acquires some desired secure information.It can then load a series of containers that open the container group to a remote code execution attack.The attestation report, obtained during initialization, cannot protect against this.Even updating it by providing additional runtime data to the PSP 1 does not help, because the vulnerability is added by the host after the attestation report has been consumed by the external service.
The concept of an execution policy addresses this vulnerability.Written by the tenant, the execution policy describes which actions the guest agent is allowed to take throughout the lifecycle of the container group.The guest agent is altered to consult this before taking any of the actions in table 1 and provides information to the policy that is used to make decisions.Each of these actions has a corresponding enforcement point in the execution policy, which will either allow or deny the action.In our implementation of Parma for Confidential ACI, the policy is defined using the Rego policy language. 14 5.
The policy actions shown in table 1 are those that need to be enforced by the execution policy.Each action is provided with the listed properties of the command as part of the enforcement query.The list is specific to our implementation but given standardization around containerd, it should be applicable to most scenarios.First, some actions pertain to the creation of containers.By ensuring that any device mounted by the guest has a dm-verity root hash that is listed in the policy, and that "name": "devices", 14 "action": "add", 15 "key": input.target,16 "value": input.deviceHash,17 } they are combined into overlay file systems in layer orders that coincide with specific containers, we first establish that the container file systems are correct.The container can then be started, further ensuring that the environment variables and start command comply with policy and that mounts from the UVM to the container are as expected (along with other container-specific properties).Other actions proceed in a similar manner, constraining the control that the container shim has over the guest agent.
A novel feature of our implementation is the ability for a policy to manipulate its own metadata state (maintained by the guest agent).This provides an attested mechanism for the execution policy to build a representation of the state of the container group, which allows for more complex interactions.For example, in the rule shown in figure 5, the enforcement point for mounting a device creates a metadata entry for the device, which will be used to prevent other devices from being mounted to the same target path.
The result of making this small change to the guest agent is that the state space of the container group is bounded by a state machine, in which transitions between states correspond to the actions described earlier.Each transition is executed atomically and comes with an enforcement point.

Security invariant
The state machine starts as a system that is fully measured and attested, including the execution policy with all its enforcement points, with the root of trust being the PSP hardware (n = 1).The set of enforced commands in A novel feature of our implementation is the ability for a policy to manipulate its own metadata state.
table 1 are designed to govern all state transitions in the container group.Since this policy is part of the attestation, the security properties of that initial state are preserved with each transition.
This security comes with some caveats.First, Parma cannot protect the user against compromised containers that the tenant has measured and included in its policies.It protects the tenant against malicious actions on the part of the CSP and forces the tenant to take only those actions that are explicitly listed in the execution policy made available to the user for measurement.Parma cannot protect the tenant from itself.Second, the security provided is worthwhile only to the degree that the tenant and its users make use of the attestation report.That is, the user must verify that the attestation report received from Parma is bound to a UVM and an execution policy that uses the data in an acceptable manner.EVALUATION Figure 6 displays latency histograms from the Nginx benchmark.The results in table 2 are from redis, SPEC2017, and Triton benchmarks.The redis values are geometric means in units of thousands of requests per second-higher is better.The SPEC2017 values reported here are the base ratio from reportable benchmark runs-higher is better.The reported values for Nvidia Triton are inferences per second-higher is better.In all cases, introducing Parma has a negligible effect when compared to SEV-SNP on its own.Benchmarking tools were used to evaluate several typical containerized workloads for reductions in computation throughput, network throughput, and database transaction rates to ensure that Parma does not introduce significant computational overhead.In all cases Parma provided confidentiality for containerized workloads with minimal costs to performance (typically less than one percent additional overhead versus running in an enclave).
Experiments were performed using three configurations, shown in table 2: 3 Base -a baseline benchmark container running outside the SEV-SNP enclave.The results are shown in figure 6.The histograms were composed of all latency samples gathered.There was an increase in latency by the introduction of SEV-SNP, as expected, as well as a minor increase in latency when the execution policy was added.

redis
The in-memory key/value database redis provides another useful benchmark for containerized computation.It supports a diverse set of data structures, such as hashtables, sets, and lists.The benchmarking used the provided redis-benchmark with 25 parallel clients and a subset of tests, the results of which can be seen in table 2. Looking at the geometric mean over all actions, we see a performance overhead of 18 percent added by operating in the AMD SEV-SNP enclave, and a further one percent when using Parma.The performance overhead can be attributed to increased TLB pressure arising from large working sets, which exhibit poor temporal reuse in TLBs and trigger page-table walks.In SEV-SNP-enabled systems, page-table walks incur additional metadata checks (in the reverse page table) to ensure that the page is indeed owned by the VM.

SPEC2017
Parma was also evaluated by measuring the computation performance overhead using the SPEC2017 intspeed benchmarks. 15These benchmark programs were compiled and run on the bare-metal hardware.When containerized, they were provided with 32 cores and 32 GB of memory.As can be seen in table 2, AMD SEV-SNP adds a performance overhead of 13 percent on average, and Parma adds less than one percent on top of this.

Nvidia Triton Inference Server
Finally, Parma was evaluated by running a ML (machine learning) inference workload based on Nvidia's Triton Inference Server. 12Models (trained offline) and their parameters were used to serve requests via a REST API.The confidential ML inference server was deployed in a container group comprising two containers: an (unmodified) Triton inference container and a sidecar container that mounted an encrypted remote blob (holding the ML model) using dm-crypt and dm-integrity.(The sidecar container also implemented the attestation workflow described in figure 4 to release the encryption key.)The file system (and the precomputed ML model) were made available to the Triton inference container.Nvidia's perf-analyzer system was used to evaluate the inference servers.This allowed measurement of the overhead introduced by SEV-SNP and Parma, as shown in table 2. For each of the three configurations (Base, SNP, and Parma), four experiments were run with one to four concurrent clients making continuous inference requests.As before, the additional overhead from Parma is approximately one percent.The overheads share the same root cause in the increased TLB pressure as previously described in the section on redis.

CONCLUSION
The experiments presented here demonstrate that Parma, the architecture that drives confidential containers on Azure container instances, adds less than one percent additional performance overhead beyond that added by the underlying TEE (i.e., AMD SEV-SNP).Importantly, Parma ensures a security invariant over all reachable states of the container group rooted in the attestation report.This allows external third parties to communicate securely (via remote attestation) with containers, enabling a wide range of containerized workflows that require confidential access to secure data.Companies obtain the advantages of running their most confidential workflows in the cloud without having to compromise on their security requirements.Tenants gain flexibility, efficiency, and reliability; CSPs get more business; and users can trust that their data is private, confidential, and secure.
This section describes the Parma architecture that implements this new confidential containers abstraction,

Figure 2
illustrates container flow.The sequence diagram at the top shows the process that results in a VM-isolated container.The pentagons correspond to the following steps: (i) Pull the image; (ii) launch a pod; (iii) create a container; (iv) start a container.The circles in this figure outline multiple points of attack in the workflow: The container shim may pass a compromised (1) UVM image or (2) guest agent during UVM creation.The container manager can (3) alter or fabricate malicious layer VHDs (virtual hard disks) and/or (4) mount any combination of layers onto the UVM.The container shim can pass(5) any set of layers to use for creating a container file system, as well as (6) any combination of environment variables or

FIGURE 5
FIGURE 5: Enforcement point 3 SEV-SNP -the same container running in the SEV-SNP enclave.3Parma -the same container again in the enclave and

3
Tamper with block devices storing the read-only container image layers and the writable scratch layer.3Tamper with container definitions, including the overlay file system (i.e., changing the order or injecting rogue layers); add, alter, or remove environment variables; alter the tenant command, and the mount sources and destinations from the UVM. 3 Add, delete, and make arbitrary modifications to network messages (i.e., fully control the network).3 Request execution of arbitrary commands in the UVM and in individual containers.
3Request debugging information from the UVM and running containers such as access to I/O, the stack, or container properties.

TABLE 1 :
A sample Policy actions enforcement point implemented in Rego can be seen in figure

TABLE 2 :
Benchmarking 18benchmarking tool.Each test ran for 60 seconds on 10 threads, simulating 100 concurrent users making 200 requests per second (for a total of 12,000 requests per test).The tests were repeated 20 times for each of the three configurations and measured the latency.