GuardML: Efficient Privacy-Preserving Machine Learning Services Through Hybrid Homomorphic Encryption

Machine Learning (ML) has emerged as one of data science's most transformative and influential domains. However, the widespread adoption of ML introduces privacy-related concerns owing to the increasing number of malicious attacks targeting ML models. To address these concerns, Privacy-Preserving Machine Learning (PPML) methods have been introduced to safeguard the privacy and security of ML models. One such approach is the use of Homomorphic Encryption (HE). However, the significant drawbacks and inefficiencies of traditional HE render it impractical for highly scalable scenarios. Fortunately, a modern cryptographic scheme, Hybrid Homomorphic Encryption (HHE), has recently emerged, combining the strengths of symmetric cryptography and HE to surmount these challenges. Our work seeks to introduce HHE to ML by designing a PPML scheme tailored for end devices. We leverage HHE as the fundamental building block to enable secure learning of classification outcomes over encrypted data, all while preserving the privacy of the input data and ML model. We demonstrate the real-world applicability of our construction by developing and evaluating an HHE-based PPML application for classifying heart disease based on sensitive ECG data. Notably, our evaluations revealed a slight reduction in accuracy compared to inference on plaintext data. Additionally, both the analyst and end devices experience minimal communication and computation costs, underscoring the practical viability of our approach. The successful integration of HHE into PPML provides a glimpse into a more secure and privacy-conscious future for machine learning on relatively constrained end devices.


INTRODUCTION
Machine Learning (ML) has increasingly become one of the most impactful fields of data science in recent years, allowing various users to classify and make predictions based on multi-dimensional data.One of the main metrics for ML is the accuracy of the prediction or classification results.However, to achieve this, the results should be accompanied by a large amount of high-quality training data requiring the collaboration of several organizations.Currently, regulations such as the General Data Protection Regulations (GDPR) forbid the sharing and processing of sensitive data without the data subject's consent.It has, therefore, become crucial to uphold data privacy and confidentiality when obtaining data from other organizations.One solution to this problem is employing Privacy-Preserving Machine Learning (PPML).PPML ensures that the use of data protects user privacy and that data is utilized in a safe fashion, avoiding leakage of confidential and private information.To this end, researchers have proposed and implemented various PPMLachieving techniques, varying from secure cryptographic schemes to distributed, hybrid, and data modification approaches.This work focuses on cryptographic approaches, the most commonly used one being Homomorphic Encryption (HE) [14,28], which exhibits a high potential in ML applications.
HE allows users to perform computations such as addition or multiplication on encrypted data [28].One of the first fully HE (FHE) schemes was proposed by C. Gentry [14].FHE allows infinite operations on encrypted data while still producing a valid result.Since Gentry's seminal work [14], multiple HE schemes have been proposed to improve the efficiency and applicability of HE.Currently, the most popular schemes used are CKKS [7], TFHE [8] and BFV [5,13].Because these schemes allow arbitrary computation on encrypted data, FHE has created exciting new applications in several areas, such as Machine Learning as a Service (MLaaS) [18][19][20][21][22][23][24]27].In MLaaS applications, data is encrypted under an HE scheme and transferred to the cloud for processing.Furthermore, due to the versatility of HE, it can also be used to keep the model private through model parameter encryption.Despite the various advances in HE, it has yet to find mainstream use because of high computational complexity and extended ciphertext expansion, resulting in very large ciphertexts.To address these issues, researchers have turned to the Hybrid Homomorphic Encryption (HHE) concept [3,12], which uses symmetric ciphers to make HE more accessible for users.HHE in a Nutshell: In an HHE scheme, rather than encrypt data with only an HE scheme, users locally encrypt data via a symmetric key encryption scheme.Subsequently, they homomorphically encrypt the symmetric key used in the encryption process with an HE scheme.The two ciphertexts resulting from the abovementioned encryptions are then forwarded to the server.Upon receiving both ciphertexts, the server transforms the symmetric ciphertext into a homomorphic ciphertext using the homomorphically encrypted symmetric key.The use of HHE produces ciphertexts of a substantially smaller size compared to using only HE schemes because of the symmetric encryption.The reduced size drastically lowers the communication overhead a user would typically incur using only a traditional HE approach.However, due to high multiplicative depth, not all symmetric key encryption schemes are compatible with HHE.To this end, researchers have designed a number of HE-friendly symmetric ciphers for use in HHE schemes, such as HERA/Rubato [10,16], Elisabeth [11] and PASTA [12].Aside from reducing ciphertext size, HHE also provides a way for most consumer-grade devices to benefit from the advantages of HE schemes by transferring computationally expensive operations to the server or cloud.
For this work, we adopt the concept of HHE, as implemented in PASTA [12], to address the privacy problem of machine learning prediction as a service.Apart from the novelty of incorporating HHE in PPML, we aim to smooth out the big hurdles of implementing strong PPML models in a wide range of devices.Contributions: We provide a realistic solution that carefully considers the vagaries of PPML, all while exploring the use of new technologies that might unleash creative ideas in the decades to come.Additionally, the paper makes the following contributions: C1.We first demonstrate the effective utilization of HHE to tackle the challenges of PPML.By extending the use of PASTA to ML applications, our approach introduces HHE as a key element in PPML, unlocking new possibilities.Our primary goal is to overcome the significant obstacles in implementing robust PPML models across diverse devices.C2.We present two formally designed protocols enabling an authorized entity (e.g., an analyst) to process encrypted data as if it were unencrypted efficiently.Rigorous security proofs demonstrate that our protocols preserve user privacy, ensuring no leakage that could compromise confidentiality.C3.Through extensive experiments, we showcase the practicality of our protocol in a real-world ML scenario using a sensitive medical dataset.The experimental results indicate that our PPML protocol achieves nearly comparable accuracy to the plaintext version while safeguarding both the dataset and the neural network's privacy.Additionally, most of the computation cost is effectively outsourced to a CSP.

RELATED WORKS
Homomorphic Encryption: Various recent works have proposed using HE schemes in implementing PPML.Gentry's work [14] revolutionized the field of HE and paved the way for multiple modern schemes, such as TFHE [9], BFV [13], and CKKS [7] in PPML applications.BFV was one of the first improvements to the original HE scheme proposed by Gentry and works by limiting expensive bootstrapping operations.BFV is, therefore, referred to as a Somewhat Homomorphic Encryption (SHE) scheme and allows a limited amount of operation to be performed on integer ciphertexts.Cheon et al. also proposed the CKKS scheme, which also allows a limited number of operations to be performed on ciphertext but allows computations on floating point data [7].Chillotti et al. then proposed the TFHE [8] scheme.TFHE greatly improves the efficiency of bootstrapping and allows an unlimited amount of bitwise operations on binary data.Each of the aforementioned schemes has been applied to ML applications, requiring different techniques to implement non-linear activation functions.For example, TFHE relies on fast and efficient LUT searches to compute non-linear activations [24], while BFV and CKKS require polynomial approximations [17,23].Each HE scheme has been applied to PPML with high-accuracy results.Examples of TFHE-based PPML works are TAPAS [29], FHE-DiNN [4] and Glyph [24].An example of a CKKSbased PPML protocol is POSEIDON [30], while an example of a BFV-based PPML work is HCNN [2].HE schemes suffer from large ciphertext sizes and high computational complexities, which make them unsuitable for every environment.Enter HHE.
Hybrid Homomorphic Encryption: The first approaches to designing HHE schemes relied on existing and well-established symmetric ciphers such as AES [15].However, AES has been proven to not be a good fit for HHE schemes, primarily due to its large multiplicative depth [12].Thus, research in the field of HHE took a different approach, where the main focus shifted to the design of symmetric ciphers with different optimization criteria, such as eliminating the ciphertext expansion [6] or using filter permutators [25].However, to date, HHE has seen limited practical application [3] in real-world applications, and only a handful of works exist in the field of PPML.To the best of our knowledge, the main HHE schemes currently are HERA [10], Elisabeth [11] and PASTA [12].The authors of HERA also proposed Rubato [16]; however, the specifications remain largely the same as in HERA.These proposed approaches have different specifications and can be applied to different use cases.HERA [10] is a stream cipher based on the CKKS HE scheme and allows computations on floating point data types.In comparison, Elisabeth [11] is designed to utilize the TFHE scheme, while PASTA [12] is based on BFV for integer data types.HERA and PASTA are defined over Z q , where q = 2 16 + 1, and can store up to 16-bit inputs.Meanwhile, Elisabeth is defined over Z q , where q = 2 4 , and can store up to 4 bits of data.Each approach achieves the same security level of 128-bits.Additionally, HERA also provides tests for a security level of 80 bits.As HERA allows computations on floating point data types, it does not require quantization on certain inputs.Elisabeth and PASTA, on the other hand, require quantization to operate on floating point numbers, which introduces a rounding error, which can reduce the accuracy of certain applications, i.e.ML.To the best of our knowledge, HERA has yet to be applied to an ML application.One of the use cases provided by Elisabeth is a CNN classification task on the Fashion-MNIST dataset and shows equivalent accuracy (84.18%) to the cleartext model without HE.Table 1 provides an overview of the different popular HHE schemes.For this work, we aim to evaluate the suitability of PASTA when applied to PPML and investigate the feasibility of applying PASTA to various real-world ML datasets, particularly medical datasets, with the potential to achieve comparable accuracy and higher efficiency than conventional HE techniques.

PRELIMINARIES 3.1 Homomorphic Encryption
Definition 3.1 (Homomorphic Encryption).Let HE be a (publickey) homomorphic encryption scheme with a quadruple of PPT algorithms HE = (KeyGen,Enc, Dec, Eval) such that: • HE.KeyGen : The key generation algorithm (pk, evk, sk) ← HE.KeyGen(1  ) takes as input a unary representation of the security parameter , and outputs a public key pk, an evaluation key evk and a private key sk.

Hybrid Homomorphic Encryption
Definition 3.2 (Hybrid Homomorphic Encryption).Let HE be a Homomorphic Encryption scheme and SKE = (Gen, Enc, Dec) be a symmetric-key encryption scheme.Moreover, let X = ( 1 , . . .,   ) be the message space and  the security parameter.An HHE scheme consists of five PPT algorithms HHE = (KeyGen, Enc, Decomp, Eval, Dec) such that: • HHE.KeyGen: The key generation algorithm takes as input a security parameter  and outputs a HE public/private key pair (pk/sk) and a HE evaluation key (evk).• HHE.Enc: The encryption algorithm consists of three steps: -SKE.Gen: The SKE generation algorithm takes as input the security parameter  and outputs a symmetric key K. -HE.Enc: An HE encryption algorithm that takes as input pk and K, and outputs  K -a homomorphically encrypted representation of the symmetric key K. -SKE.Enc: The SKE encryption algorithm takes as input a message  and K and outputs a ciphertext .• HHE.Decomp: This algorithm takes as an input the evaluation key evk, the symmetrically encrypted ciphertext , and the homomorphically encrypted symmetric key  K , and outputs  ′ -a homomorphic encryption of the message .• HHE.Eval: This algorithm takes as input  homomorphic ciphertexts  ′  , where  ≥ 2, the evaluation key evk and a homomorphic function  , and outputs a ciphertext  ′  of the evaluation results.
• HHE.Dec: The decryption algorithm takes as input a private key sk and the evaluated ciphertext  ′  and outputs  ().
The correctness of an HHE scheme follows directly from the correctness of the underlying public-key HE scheme.

Machine Learning
ML is a set of algorithms that leverage already-available training data as input to train a model  ( ).The process of training is to find the optimal parameters  = (, ), where  are weights matrices and  are biases that can provide accurate predictions on the training data.Once trained,  ( ) can be used to provide predictions on unseen input data, which is called the "inference" or "prediction" phase.In this work, we leverage HHE to build PPML protocols that preserve data and model privacy in inference phase.

SYSTEM MODEL
In this section, we introduce our system model by explicitly describing our protocol's main entities and their capabilities.
• User: Let U = { 1 , . . .,   } be the set of all users.Each user generates a unique symmetric key K i locally and encrypts their data.Then the generated ciphertexts are outsourced to the CSP along with an HE encryption  K i of the underlying symmetric key.• Cloud Service Provider (CSP): Primarily responsible for gathering symmetrically encrypted data from multiple users.
The CSP is tasked with converting the symmetrically encrypted data into homomorphic ciphertexts and, upon request, performing blind operations on them.• Analyst (A): In one of our proposed protocols, there exists an analyst who owns an ML model and is interested in learning the output of ML operations on the encrypted data stored at the CSP.In this protocol, A decrypts the encrypted data from the HE evaluation of collected encrypted data and, thus, may gain insights from user data.

GUARDML
In this section, we discuss in detail the construction of GuardMLour Hybrid Homomorphic Privacy-Preserving protocols that constitute the core of this paper.GuardML is comprised of two protocols, 2GML and 3GML.The primary differences between the protocols mentioned above lie in the presence of an analyst and support for a multi-user scenario.Each protocol may be suitable for different use cases.For example, 2GML can be used in commercial ML applications where the CSP, such as AWS, Azure, etc., owns the ML models.While 3GML, on the other hand, would be ideal for a setting where the analyst is owner of the model and does not wish to reveal the contents to CSP.In this setting, the CSP's role is reduced to simply performing operations on encrypted data and models.
Building Blocks: Before proceeding to describe each protocol, we first define the building blocks used in our constructions.
Additionally, to provide secure communication, we define a public key encryption scheme, which supports message encryption and decryption, a signature scheme used for message signing and verification, and a secure cryptographic hash function to verify message integrity.We make the following assumptions: • A CCA2 secure public-key encryption scheme PKE = (Gen, Enc, Dec).• An EUF-CMA secure signature scheme  = (sign, ver).
• A first and second pre-image resistant cryptographic hash function H(•).

GuardML: 2-Party Setting
High On receiving  2 , the CSP verifies the signature by using   's verification key and the freshness of the message through the timestamp.
If the verification fails, the CSP aborts the protocol and outputs ⊥.
Otherwise, the CSP continues to the secure evaluation phase.2GML.Eval: The secure evaluation phase begins with the CSP transforming the received symmetric ciphertext    into HE ciphertext  ′   by running the HHE.Decomp algorithm.HHE.Decomp uses HE.Eval which takes as an input (evk u i , SKE.Dec,  K i ,    ), where SKE.Dec is the symmetric cipher decryption algorithm to transform the ciphertext.Afterwards, the CSP takes  ′   and their ML model parameters (, ) and inputs both of them into the HHE.Eval along with the evk u i to compute an encrypted prediction   from the encrypted data.Finally, the CSP securely sends the encrypted result back to   via  3 : Upon reception,   verifies the integrity and the freshness of the message.If the verification fails,   aborts the protocol and outputs ⊥.Otherwise, continues to the final phase.
2GML.Classify: In the final phase of the protocol,   decrypts the received ciphertext to gain insight into the data.This is done by   running the HHE.Dec algorithm with inputs sk u i ,   and outputs the prediction .

CSP User
Run HHE.KeyGen

GuardML: 3-Party Setting
High-Level Overview: We note that 2GML is unsuitable for multiuser scenarios, where data is collected from multiple users and stored at the CSP.In such a scenario, each user would be required to generate a unique set of HHE keys, while the CSP would need to store the evk u i of each user to run the protocol successfully.To resolve this, we present 3GML -an extended version of 2GML for the multi-client model.3GML consists of three parties: a set of users U, CSP, and an analyst A. The protocol's steps are primarily the same as in 2GML (subsection 5.1).The fundamental distinction is that under the three-party setting, the HHE keys are generated by A instead of   .In the 3GML setting, we assume that an analyst A owns the ML model with parameters (, ), while a user   provides the data   .A generates the required HHE keys (pk A , sk A , evk A ), publishes pk A , and sends evk A to the CSP.Each   generates a symmetric key K i and encrypts their data   locally to output a symmetric ciphertext    .Additionally,   also generates  K i -a homomorphic encryption of the symmetric key K i using A's public key and sends both the encrypted data    and  K i to the CSP.Upon reception, the CSP stores the values locally.In the evaluation phase, A can request a prediction to be performed on the stored  ′   by first sending to the CSP their homomorphically encrypted pretrained ML model parameters (  ,   ).On receiving the request from A, the CSP transforms    into homomorphic cihpertext  ′   .CSP produces an encrypted result   and sends the results back to A. Finally, A decrypts the prediction result  using sk A .

CSP Analyst
Run HHE.KeyGen  Formal Construction: The overall flow of the protocol remains the same as in the two-party protocol.Figure 2 provides a visual overview of the protocol.3GML.Setup: Each party generates their respective signing/verification key pair for the signature scheme  and publishes the verification keys.A executes the HHE.KeyGen algorithm to output the public, private and evaluation keys (pk A , sk A , evk A ).The steps of the phase are summarized below.
A then publishes pk A and sends evk A to CSP via  1 .

THREAT MODEL AND SECURITY ANALYSIS
In this section, we define the threat model used to prove the security of GuardML by formalizing the capabilities of an adversary ADV.
The PASTA [12] scheme we adopt as our underlying cryptographic scheme has been proven to be resilient against differential and linear statistical attacks and their variations.The cipher construction incorporates changing linear layers during encryption, which ensures defence against statistical attacks.The authors provide rigorous proof that this layer instantiation is secure, ensuring full diffusion throughout the entire scheme, even in the best-case scenario for an attacker [12].PASTA is also secure against algebraic attacks, such as Linearization and Gröbner Basis Attacks.To this end, we define a threat model focusing on the communication between entities in our protocols and not the underlying scheme itself.We consider a powerful adversary ADV capable of performing a variety of attacks aiming at breaking the security and privacy of the protocols.In general, ADV is capable of corrupting any number of users and the CSP.However, for 3GML, we assume that A does not collude with the CSP.Additionally, we assume that each entity can verify the owner of a public key.With this assumption, we eliminate the possibility of basic man-in-the-middle attacks.From these definitions and assumptions, we present the following possible attacks: Attack 1 (Ciphertext Substitution Attack).Let ADV be a malicious adversary.ADV successfully launches a Ciphertext Substitution Attack if she manages to replace the generated ciphertexts sent by any entity in an indistinguishable way.
Attack 2 (ML Model Unauthorized Access Attack).Let ADV be a malicious adversary.ADV successfully launches the ML model unauthorized access attack if she manages to learn information about the underlying ML model utilized by either the CSP or the analyst A.

Security Analysis
We now prove the security of our protocols in the presence of the adversary defined in 6. Proposition 6.1 (Ciphertext Substitution Attack Soundness).Let  be an EUF-CMA secure signature scheme and PKE an INC-CPA public key encryption scheme.Then ADV, cannot successfully launch the Ciphertext Substitution Attack against GuardML.
Proof.To successfully perform the Ciphertext Substitution Attack, ADV needs to successfully attack the 2GML/3GML.Upload, 2GML.Eval or 3GML.Eval phase of GuardML by substituting the actual ciphertexts with a sequence of ADV generated ciphertexts.To this end, we categorize the attacks into the following cases:

EXPERIMENTS SECTION
In this section, we report the process and results of building a privacy-preserving inference protocol on sensitive medical data (ECG) based on our 3GML construction subsection 7.1.Subsequently, we extensively evaluated the computational performance of the 2GML and 3GML protocols (subsection 7.2).For these evaluations, we utilized a dummy dataset where each data input was a vector of four random integers, and the weights and biases were also integer vectors of length four.Finally, to provide concrete evidence of the efficiency of our protocols, we compared the performance of our constructions against a basic BFV HE scheme subsection 7.3.For these experiments, our primary testbed was a commercial desktop with a 12th Generation Intel i7-12700 CPU with 20 cores and 32GB of RAM running on an Ubuntu 20.04 operating system.Furthermore, in all the evaluations, we utilized the SEAL cryptographic library1 for basic HE operations and PASTA library2 to implement the secure symmetric cipher.To ensure statistical significance, each experiment was repeated 50 times, with the average results considered to provide a comprehensive overview of the performance of each algorithm under evaluation.

PPML Application
In the first phase of our evaluations, we demonstrated the realworld applicability of our 3GML protocol by applying it to a PPML application with a sensitive heartbeat dataset to classify whether a heartbeat is subjected to heart disease or not.More specifically, we employed the MIT-BIH dataset [26], which is a dataset of human heartbeats obtained from 47 subjects from 1975 to 1979.7.1.1MIT-BIH dataset.In total, the dataset contains 48 half-hour excerpts of two-channel ECG recordings.We used the processed ECG data from [1], which contains a train split and a test split, each comprising 13,245 ECG examples that belong to five classes: normal heartbeat (N), right bundle branch block (R), left bundle branch block (L), atrial premature contraction (A), ventricular premature contraction (V).Each ECG example is a float 1D time series signal of length 128 with values in the range of [0, 1].We further grouped all ECG examples from the later four classes (L, R, A, V) into one super class called "diseased heartbeats" (Figure 3).By doing so, we simplified our problem into a binary classification problem.Based on the input ECG signal, we tried to classify if it is a normal or diseased heartbeat.As previously discussed, our HHE protocols are based on the BFV scheme, which only works with integer data and arithmetic.However, our ECG data are floating-point numbers in the range of [0, 1], and normally, training neural networks also produces models with weights and biases in floating-point numbers.
To this end, we first quantized the ECG data into 4-bit integer data with values in the range of [0, 15] (Figure 4).On preparing the data, we trained a simple neural network with one fully connected (FC) layer with a sigmoid activation function on the floating-point ECG dataset.The neural network can be written as  ( ) = sigmoid( + ), where  = (, ) are the model's weights and biases, and  is the training data.For the integer ECG data, we used the PocketNN framework [31]  To begin our evaluations, we ran the encrypted inference protocol on a different number of examples from the test split, then compared the predictions with the ground-truth outputs to get the encrypted test accuracy.For each experiment, we also made inferences on plaintext ECG data in floating point and integer arithmetic to compare the results.Table 2 shows the encrypted and plaintext accuracies on a different number of data inputs.Overall, the accuracy of plaintext inference in integer arithmetic is very similar to encrypted inference accuracies.When the number of input examples is low (1-500), integer arithmetic and encrypted accuracies are comparable or even higher than plaintext floating-point inference.When the number of data inputs increases (1000-2000 examples), plaintext integer and encrypted inference produce slightly lower accuracies (0.5-0.8% lower).For 1000-2000 input data samples, encrypted inference had higher accuracy than plaintext integer inference but with a minimal margin (0.1-0.15%).We note that this is due to HE noise which helps make a few more correct predictions.Subsequently, we evaluated the computational overhead of the ecgPPML protocol (Table 3).In the integer and float plaintext inference protocol, the client and analyst are not required to perform any computation as they outsource all their data and neural network model to the CSP.Therefore, in Table 3, the client and analyst only have a single column for the encrypted inference results.Looking at these results (i.e.across all data input examples for the encrypted inference protocol), we observed that the CSP is responsible for the most computational overhead (99% or even more), which increased linearly with the number of data inputs.Compared to plaintext inference, encrypted inference is more computationally expensive.However, encrypted inference for one data sample takes 12.18 seconds on a commercial desktop, which is a promising result.Furthermore, these experimental results align with our goal and expectation for the HHE protocol, as we want the client and analyst to do minimal work, and most of the computations take place in the CSP.Finally, we analyzed the communication cost of the ecgPPML protocol and reported the results in Table 4.For the communication between the client and the CSP, we observed that when the number of data inputs was low (1-500), the encrypted communication cost was very high compared to the plaintext costs due to the size of the HE ciphertext of the symmetric key (1.8 Mb).However, once the number of input examples increased, the size of the symmetrically encrypted data being sent to the CSP increased linearly, similar to the plaintext size, making this difference unimportant.There is no communication between the client and the analyst in plaintext protocols, while in the encrypted inference protocol, the client only transfers the HE public key to the analyst, which has a fixed size of 2.06 Mb.Overall, the communication cost for the client was minimal and increased linearly with the number of data inputs submitted to the CSP.The communication cost between the analyst and CSP was also minimal for plaintext inference since the plaintext weights, biases, and results were small.On the other hand, the majority of the communication cost for the encrypted inference protocol was incurred between the analyst and CSP.This increased cost is caused by the HE-encrypted output of the linear layer that needs to be sent from the CSP to the analyst.Hence, if the number of data inputs increases, the communication cost between the analyst and CSP will increase linearly.We observed that the communication cost for 2000 data input examples is 5548.21Mb, or about 5 Gb of data, which is a reasonable result for today's internet bandwidth.These experimental results show that our ecgPPML protocol produces comparable results in accuracy compared to inference on plaintext data.Furthermore, the CSP is responsible for most of the computation costs, and the majority of communication cost also occurs between the CSP and the analyst.These results align with our vision of using HHE for PPML applications and show the potential for HHE when applied in real-world PPML applications.

Computational Analysis
This subsection focused on the computational performance of the core algorithms executed by each entity in the 2GML and 3GML protocols.More precisely, we measured the time taken to execute each HHE algorithm in each protocol phase by the responsible party.
For both the 2GML.Setup and 3GML.Setup phases, we observed the time taken to generate a set of HHE keys at the user and the analyst, respectively.For implementation purposes, the HHE.KeyGen algorithm involved the generation of encryption parameters parms, secret key sk, public key pk, relinkey rk, and a Galois key gk, and took 243 milliseconds to execute.Subsequently, in the 2GML.Upload and 3GML.Upload phases, we measured the time taken to homomorphically encrypt a symmetric key K using the HE.Enc and the time taken to execute the SKE.Enc for various inputs ranging from 1 to 300 (each input is an integer vector of length 4).HE.Enc took 7 milliseconds to run.For a single data input, SKE.Enc executed in 2 milliseconds and 600 milliseconds for 300 data inputs (Table 5).
When evaluating the 2GML.Eval phase; we measured the cost of executing the HHE.Decomp algorithm for various numbers of symmetric ciphertexts from 1 to 300, and the cost of executing the HHE.Eval algorithm for various homomorphic ciphertexts (1 to 300).For a single input, HHE.Decomp took 11.9 seconds, while HHE.Eval took 7 milliseconds.On the other hand, for 300 inputs,  Finally, in both 2GML.Classify and 3GML.Classify phases, we focused primarily on measuring the cost of HHE.Dec for a range of homomorphic ciphertext inputs from 1 to 300.For a single input, HHE.Dec ran in 3 milliseconds, and for 300 inputs, it ran in 900 milliseconds (Table 5).

Comparison with plain BFV
To provide concrete evidence of the efficiency of our proposed construction, we implemented a plain BFV scheme with a similar architecture to 3GML and compared the results.More precisely, we measured the performance of a plain BFV scheme, where a user continuously encrypts data input homomorphically before outsourcing them to the CSP.The same encryption parameters were used for all implementations.For these experiments, we only focused on comparing the total computational and communication costs of running the 3GML.Upload phase of our protocol, with the cost of continuously using HE encryption in the plain BFV.We varied the number of data inputs from 1 to 300 (each data input is an integer vector of length 4).
For a single data input, 3GML.Upload took 9 milliseconds to execute, while the plain BFV scheme took 7 milliseconds to perform a single HE encryption.It is worth noting that the plain BFV scheme is marginally faster for a single data value.However, this is due  to the fact that 3GML.Upload involves two operations (a symmetric encryption operation and an HE encryption operation), while the plain BFV scheme involves just one HE encryption operation.However, when the number of data values was increased to 300, 3GML.Upload ran in 0.608 seconds, while the plain BFV scheme ran in 2.2 seconds.Figure 5 provides an overview of the computational comparison results obtained from this phase of our experiments.It is worth pointing out the fact that, in most cases, uploading just one single input is unrealistic since most PPML services will require a plethora of data to properly evaluate a problem.Subsequently, we compared the communication expenses by measuring the total size of transferable ciphertext data in bytes from a user   to CSP.Overall, 3GML.Upload sent approximately 1.8 MB of ciphertext data for both a single input and 300 inputs.This is primarily because the size of a symmetric ciphertext is almost negligible as compared to that of a homomorphic ciphertext.The plain BFV, on the other hand, sent approximately 1.82 MB of ciphertext data for a single data input and 547.8 MB for 300 data inputs.Figure 6 provides an overview of the comparison of the communication costs for 1 to 300 different inputs.From these results, it is evident that GuardML reduces the communication and computational burden of   and transfers them to CSP.

Open Science & Reproducible Research
To support open science and reproducible research and provide other researchers with the opportunity to use, test, and hopefully extend our work, the source codes used for the evaluations have been made available online 3,4 .

CONCLUSION
This paper is one of the first attempts to effectively use the novel concept of HHE to address the problem of privacy-preserving machine learning.We have provided a realistic solution that carefully considers the vagaries of PPML.The designed approach is able to carefully balance ML functionality and privacy so as to allow the use of PPML techniques in a wide range of areas, such as pervasive computing, where, in many cases, the underlying infrastructure presents certain inbuilt limitations.By using HHE, we managed to overcome the main difficulties of PPML application in real-life scenarios, where the majority of data is collected and processed by constraint devices.Certain that the future of cryptography goes hand in hand with ML, we believe we have made the first step towards implementing PPML services with strong security guarantees, which operate efficiently in a wide range of architectures.
to train the same neural network in integer arithmetic.In both cases, we trained until the models' predictive accuracy on the test data split no longer improved.Training in floating-point arithmetic produced the best train accuracy of 89.36% at epoch 482 and the best test accuracy of 88.93% at epoch 500.With respect to training in integer arithmetic, we got the best train accuracy of 86.65% at epoch 42 and the best test accuracy of 87.06% at epoch 31.From these results, we observed that the test accuracy of the integer neural network was only 1.87% lower than the test accuracy produced by the floatingpoint neural network.Note that when training in integer arithmetic, we constrained the values of the weight and biases to be in the range of [−2047, 2048], or [−2 11 + 1, 2 11 ].The reason for this is that the HHE protocol works in Z  where  = 2 16 + 1; therefore, all computation results need to be in the range of [−2 15 + 1, 2 15 ]; otherwise they will be wrapped around by the modulo operation and produce incorrect predictions.As mentioned above, our ECG data are 4-bit integers and in the range of [0, 2 4 − 1].In the 1 FC neural network, we needed to do one encrypted element-wise matrix multiplication between the data and the weights, and hence, if the weights are in [−2 11 + 1, 2 11 ], the results will be constrained in [−2 11 + 1, 2 11 ] × 2 4 = [−2 15 + 2 4 , 2 15 ] ≈ [−2 15 + 1, 2 15 ], which is what we needed.After training and getting the trained integer model, we built ecgPPML -a privacy-preserving inference protocol on the quantized integer ECG data based on our 3GML protocol construction and ran the experiments with the results reported below.

Table 1 :
Comparison of different HHE schemes • HE.Enc : The encryption algorithm  ← HE.Enc(pk, ) takes as input the public key pk and a message  and outputs a ciphertext .• HE.Eval : The algorithm   ← HE.Eval(evk,  ,  1 , . . .,   ) takes as input the evaluation key evk, a function  , and a set of  ciphertexts, and outputs a ciphertext   .• HE.Dec : The decryption algorithm HE.Dec(sk, ) → , takes as input the secret key sk and a ciphertext , and outputs a plaintext .
-Level Overview: The first version of GuardML is 2GML -a 2-party protocol that consists of a CSP and a user   .In this setting, we assume that the CSP is the owner of a trained ML model with parameters (, ), while   provides the input data   to the model.Initially, the user   generates the necessary HHE keys (pk u i , sk u i , evk u i ).It then publishes the public key pk u i and sends the evaluation key evk u i to the CSP.Subsequently,   generates a unique symmetric key K i .On completing the key generation phase,   begins the data upload phase by first generating  K where    is a signature created by   .Upon reception, the CSP verifies the signature by using   's verification key and the freshness of the message through the timestamp.If the verification fails, the CSP aborts the protocol and outputs ⊥.Otherwise, the CSP stores evk u i locally.2GML.Upload: In this phase,   runs the HHE.Enc algorithm to generate a symmetric encryption key K i through SKE.Gen and then encrypts the plaintext data   into ciphertext    using the SKE.Enc algorithm, which takes (K i ,   ) as an input.After encrypting the data,   also runs HE.Enc to homomorphically encrypt K i into  K i with their pk u i .After both encryptions are finished   sends both ciphertexts to the CSP with  2 : i , a homomorphic encryption of the symmetric key K i , and then calculates a symmetric encryption of the data   with K i .The data upload phase concludes with the sending of both ciphertext values to the CSP.Upon reception, the CSP begins the secure evaluation phase by first transforming the symmetrically encrypted data    into a homomorphic ciphertext  ′   .On successful run, CSP uses evk u i ,  ′   and  to produce an encrypted prediction   .The encrypted result is then sent to   for decryption.Formal Construction: Figure1provides an overview of the 2GML protocol, which is divided into four distinct phases, namely 2GML.Se tup, 2GML.Upload, 2GML.Eval, and 2GML.Classify.2GML.Setup: In the setup phase, both parties generate their respective signing/verification key pairs for the signature scheme  and publish the verification keys.The CSP runs the PKE.Gen algorithm to generate a public/private key pair (pk CSP , sk CSP ).On the other hand,   runs the HHE.KeyGen algorithm to generate the public, private and evaluation keys (pk u i , sk u i , evk u i ) used for HHE operations.Finally,   publishes pk u i and outsources evk u i to the CSP through  1 . 1 = ⟨ 1 , Enc(pk CSP , evk u i ),    (H( 1 ||evk u i ))⟩, to generate a unique symmetric key K i through SKE.Gen.Then,   encrypts their data by computing SKE.Enc, which takes K i and   as input and outputs    .Additionally, each   homomorphically encrypts its symmetric key K i using HE.Enc with inputs pk A and K i and output  K i .The steps of this phase are summarized as:K i ← SKE.Gen(1  ), SKE.Enc(K i ,   ) →    ,  K i ← HE.Enc(pk A , K i )Finally,   sends the encrypted values (   and  K i ) through  2 . 2 = ⟨ 2 ,    ,  K i ,    (H( 2 ||   || K i ))⟩3GML.Eval: The secure evaluation phase in the three-party protocol is initiated by A to gain insight into the data provided by any user   .First, A homomorphically encrypts their ML model parameters (, ) by running the HE.Enc algorithm, which takes as input (pk A , (, )) and outputs encrypted parameters (  ,   ).A then sends  3 to the CSP (Figure2).Upon receiving the encrypted model, the CSP transforms    into homomorphic ciphertext  ′   by running the HHE.Decomp algorithm which takes as input evk A ,    , and  K i , and outputs  ′   .Subsequently, the CSP runs the HHE.Eval algorithm, which takes as input (evk A , (  ,   ),  ′   ) to output an encrypted prediction   .Afterwards,   is sent through  4 .The phase is summarized below.  ) ← HE.Enc(pk A , (,  ) ), Enc(pk CSP , evk A ),   (H( 1 ||evk A ))⟩ 3GML.Upload: With the three-party setting, there can be multiple users, namely   ∈ U. Hence, each   independently runs HHE.Enc,  ′   ← HHE.Decomp(evk A ,   ,  K i ),  ← HHE.Eval(evk A , (,   ),  ′   )  3 = ⟨ 3 , (  ,   ),   (H( 3 ||(  ,   )))⟩  4 = ⟨ 4 ,   ,   (H( 4 ||  ))⟩ Option 1 (2GML/3GML.Upload): When a user   outsources data to the CSP via  2 = ⟨ 2 ,    ,  K i ,    (H( 2 ||   || K i ))⟩, ADV needs to replace    with  ′   and  K i with  ′ Tamper with  2 in an indistinguishable way.The first three tasks are straightforward to achieve.Additionally, substituting    with  ′   and  K i with  ′  2 is trivial.However, both    and  ′   are included in the signature; hence, successfully substituting the terms is equivalent to forging   's signature.Given the EUF-CMA security of the signature scheme , this can only happen with negligible probability in the security parameter  of .Option 2 (2GML.Eval): When CSP returns the results of a secure classification to   via  3 = ⟨ 3 ,   ,   (H( 3 ||  ))⟩, ADV needs to replace   with  ′  for this instance of the attack to be successful.More specifically, since   is encrypted with pk u i , ADV simply encrypts fictitious results  ′ with pk u i to produce  ′  .However,   is included in the signature part of  3 ; hence tampering with  3 requires forging the signature of CSP.Given the EUF-CMA security of the signature scheme , this can only happen with negligible probability.Option 3 (3GML.Eval):When A initiates secure classification via  3 = ⟨ 3 , (  ,   ),   (H( 3 || (  ,   )))⟩, and CSP responds with the secure evaluation via  4 = ⟨ 4 ,   ,   (H( 4 ||  ⟩, ADV needs to replace (  ,   ) with ( ′  ,  ′  ) and   with  ′  to successfully perform this attack.The proof for this attack is similar to the one provided for the attack on 2GML.Eval with the only difference being that ADV attacks both  3 and  4 instead of just  3 .Using the same reasoning (i.e., the negligible probability of forging the signature), we conclude that ADV has a negligible probability of tampering with  3 and  4 ; hence this attack fails.□ Proposition 6.2 (ML Model Unauthorized Access Attack Soundness).Let  be a multi-layered ML model and HE semantically secure encryption scheme.Then ADV cannot successfully launch the ML Model Unauthorised Access attack for any of the GuardML protocols.Proof.To successfully launch the ML Model Unauthorized Access attack, ADV must collude or corrupt multiple entities in GuardML depending on the use case.More specifically, ADV can either attack 2GML by colluding with multiple users (  )  ∈ where  ⊆ [] or attack 3GML by colluding with a user   and the CSP.To this end, we distinguish the attack into the following; Option 1 (Attacking 2GML): Let  be a multi-layered ML model owned by CSP, and assume that ADV colludes with • Generate a symmetric key K ADV ; • Use K ADV to generate a series of ciphertexts  ′ ; • Encrypt K ADV with pk A or pk u i to get  ′ K ADV ; • ′ =  () users.On completion of the secure evaluation phase (Figure 1), each user receives   =  (  ), where   is the input from the user.ADV successfully launches this attack if with the help of the  ′ colluding users, ADV can solve for  given (  )  ∈ and (  )  ∈ .With the assumption that  is a multi-layered ML model, the likelihood of this attack is negligible.Option 2 (Attacking 3GML): Let  be a multi-layered ML model owned by A with (  ,   ) as the HE encrypted parameters sent to CSP via  3 = ⟨ 3 , (  ,   ),   (H( 3 || (  ,   )))⟩.ADV successfully launches this attack if given a corrupt user   and a corrupt CSP, ADV successfully retrieves  in the form of (, ).HE has been proven semantically secure; hence, the likelihood of ADV decrypting the ciphertext (  ,   ) is considered negligible.□

Table 5 :
Computational AnalysisHHE.Decomp took 3595.6 seconds, while HHE.Eval took 2100 milliseconds (Table5).When evaluating 3GML.Eval, we first measured the performance of HE.Enc at A and then the performance of the HHE.Decomp and HHE.Eval algorithms on various number of symmetric ciphertext inputs from 1 to 300 at the CSP.HE.Enc took 16 milliseconds to execute, while the results for HHE.Decomp were similar to those from 2GML.Eval.For a single input, HHE.Eval took 38 milliseconds to execute and 11.6 seconds for 300 inputs.

Table 6 :
Total Computation Cost -2GML for 300 data inputs

Table 6 and
Table 7provide the computational analysis of 2GML and 3GML protocols respectively for 300 inputs.

Table 7 :
Total Computation Cost-3GML for 300 data inputs