Make Split, not Hijack: Preventing Feature-Space Hijacking Attacks in Split Learning

The popularity of Machine Learning (ML) makes the privacy of sensitive data more imperative than ever. Collaborative learning techniques like Split Learning (SL) aim to protect client data while enhancing ML processes. Though promising, SL has been proved to be vulnerable to a plethora of attacks, thus raising concerns about its effectiveness on data privacy. In this work, we introduce a hybrid approach combining SL and Function Secret Sharing (FSS) to ensure client data privacy. The client adds a random mask to the activation map before sending it to the servers. The servers cannot access the original function but instead work with shares generated using FSS. Consequently, during both forward and backward propagation, the servers cannot reconstruct the client's raw data from the activation map. Furthermore, through visual invertibility, we demonstrate that the server is incapable of reconstructing the raw image data from the activation map when using FSS. It enhances privacy by reducing privacy leakage compared to other SL-based approaches where the server can access client input information. Our approach also ensures security against feature space hijacking attack, protecting sensitive information from potential manipulation. Our protocols yield promising results, reducing communication overhead by over 2x and training time by over 7x compared to the same model with FSS, without any SL. Also, we show that our approach achieves>96% accuracy and remains equivalent to the plaintext models.


INTRODUCTION
In recent years, Machine Learning (ML) has developed into a highly valuable field used in everyday situations.Various ML applications, like large language models [30], have become more accessible than ever before.However, ML requires a suitable amount of processing power to train.As such, various companies have started offering Machine-Learning-as-a-Service (MLaaS) to allow clients to use maintained servers to outsource resource-intensive ML tasks.Despite its benefits, MLaaS raises privacy concerns, as it requires sharing client data to a company during model training [14,34].To address this privacy issue, researchers have begun developing Privacy-Preserving Machine Learning (PPML) techniques to protect client data [16][17][18].These techniques encompass cryptographic approaches like Homomorphic Encryption (HE) [8] and Multi-Party Computation (MPC) [39] as well as collaborative learning approaches such as Federated Learning (FL) [22,27] and Split Learning (SL) [1].These techniques have been used in a number of works and successfully reduce the privacy leakage of ML [1, 12, 19-21, 29, 33].
FL and SL are two methods which allow to train a model collectively from various distributed data sources without sharing raw data.FL, introduced by Google AI Blog, allow each user to run a copy of the entire model on its data.The server receives updated weights from each user and aggregates them.However, in FL, the users' model weights are shared with the server, which can leak sensitive information [13].The SL approach proposed by Gupta and Raskar [11] offers a number of significant advantages over FL.With SL, multiple parties can train an ML model together while keeping their respective parts private.Similar to FL, SL does not share raw data but it has the benefit of not disclosing the model's architecture and weights and clients can train an ML model without sharing their data with a server, which has the remaining part of the model.Hence, in this work, our focus is on the SL technique, where low-cost operations are performed on the client-side while, computationally expensive operations are outsourced to a powerful server, expediting the models training process.However, SL without any additional privacy-preserving measures is prone to privacy leakage through Feature Space Hijacking Attack (FSHA) [31] or data analysis techniques, such as Visual Invertibility (VI) [1].
Through VI a malicious server is able to infer information of the underlying input data by analysing the visual similarity between the original image and the produced activation map after a number of convolutional layers.Looking into improving the security and efficiency of individual approaches, researchers have lately started exploring solutions that combine several privacy-preserving techniques [20,33,36].As such, researchers proposed to use differential privacy (DP) [1] with SL, but noted, that it greatly reduces the utility of the trained model.Others have proposed encrypting the intermediate results using HE before sending them to the server [20].This allows the server to compute the final layers of the ML model and get the correct outputs without leaking additional information to the server.Despite the increased security, the authors report, that the computational overhead even for a single linear layer using HE increases the training time drastically, when compared to the unencrypted models.Regarding the FSHA, existing work in literature does not address this attack in SL.FSHA allows a malicious server to hijack the model and steer it towards a specific target feature space.Once the model maps into the target feature space, the malicious server can recover the private training instances by reversing the known feature space.In short, FSHA allows a malicious server to take control of the model and extract private information from training data.
In this work, to secure SL against FSHA and visual analysis, we use Function Secret Sharing (FSS) [5] -an MPC technique which allows two parties to perform computations on a public input using a private, secret shared function.We show that FSS can be used to eliminate the privacy leakage of SL while at the same time keeping data safe from FSHA.In addition, it greatly reduces the computational complexity, when compared to existing techniques [20,29,32].Furthermore, we show that combining FSS with SL results in lower communications overhead than combining HE with SL.

Contributions:
The main contributions of this work, can be summarized as follows: C1.We designed an efficient PPML protocol using SL and FSS.
Our contribution lies in being the first to incorporate FSS into SL, aiming at enhancing the privacy of SL.C2.We have conducted experiments on the MNIST dataset to demonstrate privacy leakage in SL after the first and second convolution layers.We have used FSS to mitigate this issue, offering a more efficient alternative to the computationally expensive HE method mentioned in prior work by Nguyen et al. [29].Our approach enables running multiple layers on the server-side, unlike [29], which considered only one layer on the server-side.C3.We designed an FSS based vanilla SL (refer to as private vanilla SL model) protocol and compared it with AriaNN [32] (private local).Our findings demonstrated that our approach is more efficient in terms of communication cost and complexity while maintaining the same level of accuracy and privacy.C4.To date, VI has not been considered an attack in research; rather, it has been seen as a privacy leakage metric in SL.In our work, we use VI as an attack, specifically a VI Inference Attack (VIIA) in SL.C5.We demonstrated the security of our approach against FSHA.Also, our VI analysis demonstrates that the server cannot reconstruct raw image data from the activation maps.

Organization
The rest of the paper is organized as follows, in section 2, we provide the necessary background information on Convolution Neural Network (CNN), SL, and FSS.In section 3, we define the notation used throughout the paper, discuss FSS in a higher detail, and describe its core primitives.In section 4, we present important published works in the area of SL and FSS.The architectures of the two local models: one without a split and another with a split are presented in section 5, where we elaborate on the private local model and introduce the key actors in our protocols.A detailed description of our private vanilla SL protocol is described in section 6, followed by the considered threat model & security analysis in section 7, while section 8 provides a detailed explanation of the experimental setup and provides extensive experimental results for our proposed approach.Finally, we conclude the paper in section 9.

PRELIMINARIES
In this section, we cover the underlying ML and cryptographic techniques we utilize in our design.

Convolutional neural network
CNN is a special type of neural network which have shown great results in various ML classification and recognition tasks for two dimensional (2D) images data [23].For our task, we employ a 2D CNN which consists of the following layers: • 2D Convolution (Conv2D): It is used in CNN to perform convolution operation on the input data, that is, this layer applies a set of kernels, to a sliding window of input data, which extracts local features from the data (see Figure 1) [23].• ReLU: It is a non-linear activation function that is commonly used in CNN to compute a simple function: () = max(0, ).• Max-pooling: It is used in CNN to create a downsampled activation map () with a lower resolution than the input.This operation helps to reduce the computational cost of the network and prevent overfitting by compressing information in the .
• Fully Connected (FC): In FC layers, each neuron in the layer is connected to every other neuron in the previous layer.This means that the output of each neuron in the layer is computed as a weighted sum of the outputs of all the neurons in the previous layer.

Split learning
It is a collaborative learning technique [11] in which an ML model is split into two parts, the client part (  ), comprises the first few layers of model ( computing the loss, the server performs the backward propagation, only returning to the client the gradients to complete the backward propagation.This process is repeated until the model converges and learns a suitable set of parameters. The aim of SL is to protect client privacy by allowing clients to train part of the model, and share  (instead of their raw data) with a server running the remaining part of the model.It is also utilized to reduce the client's computational overhead by merely running a few layers rather than the entire model.

Function secret sharing
FSS provides a way for additively secret-sharing a function  from a given function family  .More concretely, a two-party FSS scheme splits a function  : (0, 1)  → G, for some abelian group , into functions described by keys such that  = f 0 + f 1 and every strict subset of the keys hides  .An FSS scheme for some class  has two algorithms (KeyGen, EvalAll) [5]: • KeyGen(1  ,  ) → (f 0 , f 1 ): The KeyGen algorithm that takes the security parameter 1  and some secret function  to be shared and outputs two different function shares (f 0 , f 1 ) (also called function keys).• EvalAll(, f  ,   ) → f  (  ): The EvalAll algorithm takes three parameters, the input bit  ∈ {0, 1}, the key f  , and the public data   and outputs the shares f  (  ).
An FSS scheme should satisfy the following two properties: • Secrecy: A single key (f  ) hides the original function  .
As can be seen in Figure 2, function  is first split into two shares f 0 and f 1 and then each party locally evaluates the function on the input   .In FSS, adding the locally evaluated functions (f 0 (  )+ f 1 (  )), gives the additive reconstruction of the original function applied on the input  (  ).

FUNCTION SECRET SHARING PRIMITIVES
Before going into the details of the FSS primitives, first we define all the notation used in the paper.After that, we go into the details of the FSS primitives taking into consideration ReLU as an example.
Notation.Table 1 summarizes the core notation of this paper.In addition to that,    denotes the client part of the model, while    0 and    1 , represent the split part of the remaining model associated with Server 1 ( 0 ) and Server 2 ( 1 ).
As mentioned earlier, the ReLU function returns 0 for negative inputs and keeps positive value as is.To create shares of the ReLU function, we have employed two FSS tests that fit our needs: (1) the equality test and (2) the comparison test [4][5][6][7].Before going into the details of these tests, let's briefly cover the concepts of the Pseudo-random Number Generator (PRNG) and Correction Word (CW).Pseudo-random number generator.A PRNG is an algorithm that takes a short, random seed and produces a longer sequence of pseudo-random bits.In FSS, it is used to generate the keys for the FSS protocol.Each party starts with a random seed, and uses the PRNG to generate a sequence of keys.These keys are then used to generate CW for the FSS protocol.Correction word.It is a special value used in FSS to correct the private shares being compared between two parties.The CW shifts a pseudo-random string so that it matches another random string on one side and remains independent on the other side.Each node in the binary decision tree generates its own CW, which is computed by applying a PRNG to the private shares being compared at that node.The resulting value is then used to correct the shares by computing an XOR operation on them with the CW.This ensures that the shares are equal if the input bits are the same, and different if the input bits are different.The CW is passed down to the child nodes of the current node in the binary decision tree, where it is used to correct the shares at those nodes as well.This process continues until the final output of the binary decision tree is computed.Equality test.A test that involves comparing   to a private value  and the goal is to determine whether   is equal to  or Figure 3: Function secret sharing for equality test not.The random mask  and the private input  are combined using modular arithmetic and published to obtain a public value   .The equality test is performed using a binary decision tree of depth , where  is the number of bits in the input   .The tree evaluates the input using f 0 , f 1 , which are generated using FSS.The path from the root to  is called the special path.The two servers ( 0 ,  1 ) start from the root and update their state depending on the bit , they stay on the special path and output 1, otherwise, they output 0. The main idea is that the servers on the special path have specific states ( 0 ,  0 ) and ( 1 ,  1 ), where  0 and  1 are independent and identically distributed and  0 ⊕  1 = 1 (see the dotted lines in Figure 3).When the servers are out of the special path, the CW ensures that  0 =  1 but remain random and indistinguishable, and  0 =  1 , guaranteeing  0 ⊕  1 = 0. To reconstruct the result, each server outputs its value   , and the final result is obtained by  0 ⊕  1 .
Comparison test.The comparison test involves a binary decision tree with a depth of .In this test, the path from the root down to the greater value is called the special path.Each server starts at the root of the binary decision tree and updates their state based on the bits using CW from f 0 and f 1 .The goal is to evaluate all the paths simultaneously.When a server deviates from the special path, they either fall on the left side for   <  or on the right side for   >  (see Figure 4).Hence, at each node, an additional step is taken where a leaf label is output depending on the bit value and all previous bits are same.Only one label, either the final label (which corresponds to   = ) or the leaf label can be equal to 1, because only one path is taken.Therefore, the servers return the sum of all the labels to obtain the final output.

RELATED WORK
SL provides significant advantages in terms of reducing the computational cost of training deep neural networks.However, a potential vulnerability arises during end-to-end training, where the exchange of gradients at the cut layer could inadvertently encode private features or labels, posing a risk to data privacy.Abuadbba et al. [1] proposed using SL when training and classifying 1-dimensional data on CNN models.In their work, they use two CNN models, one with two Conv2D layers and another with three Conv2D layers to see the applicability of SL on time-series data, when compared Figure 4: Function secret sharing for comparison test to a centralized system.They identified that SL achieves comparable accuracy to centralized models, but noted that SL by itself has multiple privacy leakages, which hamper the confidentiality of the input data.Moreover, researchers have found that SL is susceptible to different types of attacks, such FSHA [31].
The leakages highlight the importance of addressing data privacy concerns when implementing SL techniques.As such, researchers have attempted combining different cryptographic techniques with SL to address these attacks.For example, Abuadbba et al. [1] tried two privacy mitigation techniques -adding more hidden layers on the client-side and using DP.However, both of these techniques indicate a trade-off between reducing privacy leakage and experiencing a decline in model accuracy, particularly when DP is used.Thapa et al. [35] proposed to combine SL with FL to reduce the computational complexity of SL.To address the remaining privacy leakage, the authors made use of DP-enhanced client model training and PixelDP [25], however faced the same drawback of greatly reduced model utility.Some other works like [19][20][21] make use of HE to encrypt the intermediate  of SL before outsourcing them to the server.This way, the model retains equivalent accuracy when compared to plaintext SL, and addresses the privacy leakages known in SL.However, a key limitation of this method is identified during backward propagation.By analyzing the gradients sent from the client, the server gains valuable information about the client's input data, leading to potential privacy leaks.To tackle this, an improved protocol is proposed by Nguyen et al. [29], aiming to minimize privacy leakage.The authors [29] claim that their proposed approach not only helps maintain privacy but also accelerates the training time and reduces the communication overhead compared to [19]).Despite these advancements, the training time and communication cost in [29] continue to pose challenges, even for a simple model featuring two convolutional layers on the client-side (processing plaintext data) and only one fully connected layer on the server-side (processing encrypted data).Specifically, it takes a substantial amount of time and communication cost, which poses a challenge for practical implementation.
There is a plethora of works, which employ FSS in PPML, such as AriaNN [32] which is the first work that makes use of FSS for secure neural network training and inference against a semi-honest adversary.Pika [37] introduces a 3PC PPML solution against either a semi-honest or a malicious adversary.Another recent FSS work is LLAMA [10], which implements a low-latency library for computing non-linear mathematical functions, useful to PPML, such as sigmoid, tanh, reciprocal square root, using FSS primitives.Orca [15] directly uses the library proposed in LLAMA [10] performs secure training on neural network architectures.SIGMA [9] demonstrates how FSS can be used for secure inference of generative pre-trained transformers, which contain GeLU, Softmax, and layer normalizations, and shows that the secure inference can be accelerated through GPUs.Agarwal et al. [2] designs a secure logistic regression training construction with FSS.FssNN [38] proposes a two-party FSS framework for secure NN training and makes improvements to various aspects in prior arts.
In all of the covered FSS works, the researchers employ FSS on the entire model which in turn increases computational complexity and communication overhead.However, to the best of our knowledge, the idea of combining SL with FSS has not been explored in the literature.To this end, our approach provides new insights into how the use of these techniques can improve the privacy guarantees of ML applications, solve the privacy leakage caused by SL and reduce both the communication and computation costs of FSS.

ARCHITECTURE
This section provides an overview of the ML model used for classification.We begin with the description of the initial non-split or local model followed by a brief explanation of a related FSS model from a previous work [32].Then, we explain how we convert the local model into a split model.Finally, we introduce our private vanilla SL protocl.Additionally, we introduce the key entities involved in the training process of the split model, namely the client and the server, elaborating on their respective roles and assigned parameters during the training phase.Let |D| = {(  ,   )| = 1, 2, . . ., } be a training dataset consisting of  data samples.Each data sample  has a corresponding encoded label vector  that represents its ground-truth class.Then,   is a function representing the local model with  being a set of adjustable parameters. is first initialized to small random values denoted as Φ.The goal is to find the optimal parameters  to map  to a predicted output vector ŷ, where ŷ is as close as possible to . ŷ is a vector of probabilities (called posteriors), and we use the highest probability to determine the class to which  belongs.To find the closest value of ŷ with respect to , we try to minimize a loss function  = L( ŷ, ).Training the local model is an iterative process to find the best  to minimize  .This process involves two steps: forward and backward propagation.The forward propagation computes each layer in the network (Figure 5) on input , reaching the output layer to predict ŷ.Backward propagation starts from the output layer and goes back to the input layer to calculate the gradients of the loss function  w.r.t the network's weights  .These weights are then updated according to the calculated gradients.We train the local model with numerous samples

Local model with vanilla split
In this protocol, we split the local model shown in Figure 5 between the client and the server in such a way that the starting layers, Conv2D, max-pooling and the first two ReLU layers are executed on client-side while two FC layers and the last two ReLU layers are executed on the server-side.As can be seen in Figure 6, the training input data is residing on the client-side while the final prediction of the model is on the server-side.

Private local model
In [32], the authors proposed AriaNN, an efficient protocol for achieving private equality and comparison through FSS.The model used in this study, as depicted in Figure 5, consists of two Conv2D layers, two Max-pooling, four ReLUs and two FC layers.Among these layers, ReLU is supported as a direct application of the equality and comparison protocol, while the Conv2D layers are computed using beaver triples.More details, about the computations of these layers can be found in [32].We reproduced the results for this variant of the protocol, achieving the best test accuracy of 97.29%.

Private vanilla SL
In our private vanilla SL protocol, the client handles the initial layers of the model, while the server takes care of the rest (see Figure 7).The client, owning the data, executes their part of the model without splitting.On the server-side, as depicted in Figure 7, With FSS on the server-side, we now have two servers, each working on its private shared function and evaluating it on the public input   .This protocol allows client-server collaboration in ML model training exposing client's input data.Additionally, with some of the layers on the server-side, computational complexity is reduced, making the model more efficient.More details on the private vanilla SL will be discussed in subsection 6.1.

Actors in the model
Now, we define the main actors participating in the protocol and detail their abilities: • Client: The client provides the training data and is capable of computing the initial layer of the model in plaintext.The client uses their private data to compute the initial layers of the model and shares the intermediate  and the remaining layers with the servers.In our approach, the client has to share the labels with the server.• Servers: Our protocol requires two servers to perform computations on the secret shares.Based on the received   , each server make computations on the underlying ML model.Both servers can compute the final outputs and compare it to the truth labels received from the client.We assume that there is no collusion between the two servers and the servers are hosted independently to avoid reconstructing the original data from the shares.This assumption is realistic and consistent with other 2PC works in the area [3,28,32].

FSS BASED SL PROTOCOL
This section provides a more detailed explanation of the algorithms for training the private vanilla SL model.

Private vanilla SL training protocol
We have used algorithm 1 and algorithm 2 to train the private vanilla SL model reported in subsection 5.4.This model consists of the initialization, forward and backward propagation.The initialization phase only takes place once at the beginning of the procedure, whereas the other phases continue until the model iterates through all epochs.The initialization phase consists of socket and random weight initialization Φ.The client first establishes a socket connection and synchronize the hyperparameters , ,  , .  () and the gradient are initially set to zero ∅.These parameters must be synchronized on both sides to be trained in the same way.There are two phases in training the whole protocol, forward and backward propagation phase.During the forward propagation, the client forward propagates the input  through    to generate an activation map .The private input which is  is first masked using random mask  to construct the public input   =  + before sending it to the server.Once   is constructed, the next step is to send   to the server.As can be seen in Figure 8, we have two servers labelled as  0 and  1 .Each server receives the public input   and initiates forward propagation by executing their respective portion of the model.On the server-side, we employ FSS, allowing us to compute the function keys.Specifically, these keys are generated for the ReLU layers.We employ the FSS KeyGen algorithm, to create these keys for each server.Additionally, to compute the FC layers on the server-side, we used the beaver triples.
As in private vanilla SL, the final output layer is executed on the server-side, hence both servers execute their respective share of the layer to produce the final output ŷ .Both the servers calculate the loss using the equation  ← L( ŷ ,   ).During the loss calculation, the intermediate results and the final loss remain in the form of additive shares.Specifically,   is the additive shares of the target value while ŷ are the predicted output computed by each server.
As the loss computation is performed by each servers, now, in order to get the final predicted output, the two servers must collaborate to add their secret shares.In this setting, the servers must know the actual target values as well as the predicted output of the model.After completing the forward propagation,each server starts the backward propagation by first computing    ŷj and continue the backward propagation algorithm until layer  + 1 and update the weights and biases of the layers using the given equations.
During backward propagation, each server also calculates: and sends   ( ) to the client.After receiving   ( ) , the client calculates the gradients of  w.r.t the weights and biases of its part of layers using the chain-rule, which can generally be described as: (5)  Finally, after calculating the gradients   () ,   () , the client updates  () and  () .The forward and backward propagation continue until the model converges to learn a suitable set of parameters.

THREAT MODEL & SECURITY ANALYSIS
In this section we first define the threat model we consider.Then, we proceed by proving the security of our protocol.

Threat model
In SL, collaborative learning of local models introduces potential risk due to interactions among the participants [26].In this study, we examine a scenario involving a semi-honest adversary, denoted as ADV, capable of corrupting one of the two servers engaged in our protocol.In this context, the corrupted parties adhere to the protocol's specifications, all the while attempting to gather as much information as possible about the inputs and function shares of other participants.Within our specific framework, secure protocols are employed to facilitate collaborative CNN model training by three parties: a client and two servers.The client, who also serves as the data owner (note that considering a malicious client in this context is pointless), initiates the process by executing the initial layers of the model.This action produces an intermediary result, denoted as , which is subsequently kept confidential through secret sharing between two independent servers.Following this, these two servers proceed to jointly train the remaining segments of the model using the client's data, achieved by employing the FSS protocol.Apart from that, the behavior of ADV can be summarized as follows: • ADV does not possess information about the client participating in the distributed training, except the necessary details required to execute the SL protocol.
• ADV is familiar with a shadow dataset, which captures the same domain as the clients' training sets.• ADV has no knowledge of the architecture and weights of the whole model.• ADV is unaware of the specific task on which the distributed model is trained.Finally, we extend the above threat model by defining two attacks available to ADV who wishes to launch an FSHA or perform data analysis through VI.
Attack 1 (Feature-Space Hijacking Attack).Let ADV be an adversary that has compromised at most one of the servers ( 0 or  1 ).ADV successfully performs an FSHA if she manages to substitute the original learning task chosen by the client with a new function intentionally designed to mold the feature space of  .Attack 2 (Visual Invertibility Inference Attack).Let ADV be an adversary, that has compromised at most one of the servers ( 0 or  1 ).ADV successfully performs a Visual Invertibility Inference Attack (VIIA) if she manages to infer sensitive information of the original input data using the acquired convolutions from the client.ADV attempts to identify the original images characteristics and truth labels using intermediate data received during the SL process.

Feature-space hijacking attack
Here, we describe the main concept behind an FSHA.In such an attack, the client owns its private data and is responsible for training the    .Instead of running its part of the SL protocol, the server runs the code provided by the ADV.To carry out the attack ADV relies on three networks: The first two f and f −1 are used to build an autoencoder which is trained with a shadow dataset.Specifically, f maps the data space to a feature space whereas f −1 act as a decoder that maps back the feature space into the original data space.The third one is Discriminator (D), the network that substitutes     ,  ∈ {0, 1}, and is trained to distinguish between the output of    and the output of network f .It is used to caste the gradients for    during training and bring it in an insecure state.D indirectly guides    to learn a mapping between the private data and the feature space defined from f .More specifically, D is used to train    to produce an output that is as close as possible to the one of f .Ultimately, this is the network that substitutes server network in the protocol, and that defines the gradient which is sent to the client during the distributed training process.After certain number of iterations, the feature space learned by    overlaps with one defined from f .Once this happens, f −1 is used to map back this smashed data sent by the client to do the training and recover the approximation of the original private data.

Visual invertibility inference attack
VI serves as a privacy assessment metric designed to measure the correlations between the split layer  and original raw data samples.The underlying concept of VIIA involves the client initially training its part of the model to create the .Afterwards, the client sends the  to the server.Upon reception, if the server is malicious, it could exploit the received  to extract private  information by inferring the client's raw data and the corresponding truth label.In essence, VIIA assesses the risk for unintended data exposure during the model's collaborative learning process.In our SL setup, the client computes    before outsourcing the remaining operations to a server.The client must share  to the server for computing    .However,  have patterns that are very similar to the raw image data and cause a privacy leakage.This similarity is visible in Figure 9, where the first row shows the raw image data and the second and third rows show intermediate outputs after the first and second Conv2D layers respectively.Even though the similarities are reduced after the second Conv2D layer, the ADV could extract some information and features of the private data, compromising the client's data privacy.

Security analysis
With our threat model established, we can now move forward to demonstrate the security of our protocol.
Proposition 1 (FSHA Soundness).Let ADV be a semi-honest adversary that corrupts at most one of the two servers ( 0 or  1 ) involved in the protocol.Then ADV cannot launch a successful Feature-Space Hijacking attack.
Proof.Let ADV be a semi-honest adversary that has compromised  0 or  1 .In addition to that, assume f  ,  ∈ [0, 1] be the key distribution of an arbitrary function  .Then, if ADV can only distinguish the key distribution f  from another f ′  (for some other function  ′ ), with negligible probability then our protocol is secure against the Feature-Space Hijacking attack.
To show that ADV can only distinguish between these values with negligible probability, we will use a proof by contradiction.Let's assume the opposite, that ADV can distinguish from which function ( or  ′ ) the key came from.This means that ADV has a non-negligible advantage to correctly determine whether the key came from  or  ′ .
To test this assumption, let's consider a security game, where ADV has access to both  and  ′ and receives f  from a challenger CH AL: Following this game, the ADV would have a method to correctly guess which function the key f  represents, through the bit  ′ , so that  ′ == .However, the key f  is generated in such a way, that it does not reveal any information about the original function  subsection 2.3.As such, the ADV has a 50% chance of successfully guessing the original function, just like flipping a fair coin with a negligible advantage.This reaches a contradiction because the ADV cannot accurately guess the original function with a non-negligible advantage from the obtained function key f  .Now, in order to show the soundness of our protocol against FSHA, we will focus on a scenario where the ADV takes control of one of the servers and leaves the other server and client untouched.The ADV tries to manipulate the feature space or the model's output and to accomplish this the ADV would need to understand the underlying feature space of    .However, through the use of FSS,    is split between two servers which makes understanding the model and feature space improbable.Each server holds a share of the function but not the entire function itself.This makes it more challenging for an ADV to compromise the entire model by taking control of one server.Additionally, guessing the function share of the non-compromised server from the share of the compromised server is not possible (1).Hence, we prove that our proposed private vanilla SL protocol is secure against FSHA.Let ADV be a semi-honest adversary that corrupts at most one of the two servers ( 0 or  1 ) involved in the protocol.Then ADV cannot successfully infer the raw input data through VIIA.
Proof.Let ADV be a semi-honest adversary that has compromised one of the two servers   ,  ∈ [0, 1].In addition, assume a noisy intermediate map   =  +, where  is the original intermediate activation map and  is a set amount of noise.Then, if ADV cannot acquire  from   , then our protocol is secure against raw data inference through Visual Invertibility Inference Attack.
The use of FSS addresses the privacy leakage caused by VIIA as the  is masked with a specific amount of noise (  = +) for both servers   .Following this a single corrupted server   would not be able to discern any additional information from the obtained   without guessing the function share held by the other, non-corrupted server as proven in Proposition 1.Additionally, if the ADV is unable to obtain , then it is improbable that  could be reconstructed through   , when an adequate amount of noise is added.To this extent, we also showcase a visualization of the masked  from both servers in Figure 10a and Figure 10b to show that   becomes unrecognisable to the original convolutions .

PERFORMANCE ANALYSIS
This section covers our experimental setup and the results of our private vanilla SL protocol.

Dataset
MNIST.To test the applicability of our approach we used the MNIST dataset [24].It consists of a total of 70,000 images of which 60,000 are in the training set and the remaining 10,000 are in the test set.All of the images are 28x28 pixels and display a black and white handwritten number ranging from 0 to 9.

Experimental setup
We test and compare our SL protocol with AriaNN [32].For the experiments, we used a machine running Ubuntu 20.04 LTS, processor 12th generation Intel Core i7-12700, 32 GB RAM mesa Intel graphics (ADL-S GT1).We implemented our code using Python 3.7 and it is made available on GitHub1 .FSS implemented using the PySyft framework2 and a modified version of AriaNN3 .To get a more consistent and reproducible result, we ran our experiments a total of 10 times for each experiment, totaling 60 executions of the code with different settings.However, in our study, we are reporting the top three results for each model (refer to Table 2).In the experiments we measured the communication costs, training times, and accuracy of the model after each epoch and report the total costs after full training.The communication costs are calculated by measuring the byte length of serialized objects sent through packets.Time measurements are taken by making use of the in-built time module from Python.The loss is calculated using mean squared error.
In terms of hyperparameters, we train all networks with  = 10 epochs,  = 0.002 learning rate,  = 0.9 momentum, a batch size of  = 128, and the total training data samples  = 59904 (as the incomplete final batch is dropped).These parameters were chosen as they showed best results during preliminary testing and are equivalent to the parameters set for the AriaNN implementation [32].This lets us accurately compare our results with AriaNN without the parameters making a large impact on the comparison.

Evaluation
This section provides an overview of the outcomes derived from our experiments.It is organized into three distinct parts, with each part dedicated to a specific architectural framework as defined in section 5. First, the local models with and without SL are covered.These models run in plaintext without making use of FSS to create a baseline to compare further results and also measure the effects of FSS on the performance of the models.Second, the reproduced results of AriaNN [32], which we use to compare to our proposed models, are provided.Finally, the results of the private vanilla SL model and its comparison with the results of AriaNN are presented.
Plaintext models.As previously mentioned, it's important to note that training the models on plaintext data does not account for FSS.These models consist of both a local model and a standard SL model.We have successfully implemented and replicated the results for these models.Regarding the communication cost, we are considering costs on both the client-side and server-side, as well as the total costs during training and testing.In Table 2 we show the variation between the client-side and server-side communication costs between the local and SL model.We observe that the SL approach has slightly higher server-side communication costs, than the local model, but client-side communications are about 3× larger on the local model.This difference arises from the size of the data being shared as in the local model, the client must send 28 × 28 images, while in the SL model, the client send the intermediate , which is only 8 × 8.Even though the vanilla SL model provides similar accuracy and has the same training and communication cost when compared to the locally computed model, it is essential to note that the vanilla SL model introduce a privacy leakage, which we mitigate using FSS.
FSS without SL.For our benchmarking we used AriaNN [32] a promising solution for doing PPML on sensitive data.The authors have shown that AriaNN provides competitive performance compared to existing work, and have demonstrated its practicality by running private training on MNIST using models like LeNet.Given the promising outcomes achieved by AriaNN in comparison to other works, we adopted it as the base paper for our study.
In our experimentation, we replicated the results for AriaNN, which we refer to as the private local model.To thoroughly assess and compare the impact of FSS on performance, we initialized two models: one before applying FSS (referred to as the "public local model") and one after applying FSS, both with the same set of initial weights.Our analysis, clearly indicates that the accuracy obtained before and after applying FSS remains nearly identical.In simpler terms, the implementation of FSS exhibits no noticeable effect on the performance of the models.As can be seen in Table 2, the private local model achieves an impressive test accuracy of approximately 97.29%, closely followed by accuracies of 97.04% and 97.61%.This showcases that the private local model attains a commendable level of accuracy, lacking only by a modest 2% to 3% when compared to the public local models.
However, when we examine the training time for private local model over 10 epochs, a significant contrast emerges.Private local model necessitates approximately 1288 minutes, which is roughly 900× more time-consuming than the training of the public local models.Additionally, the training communication cost incurred by private local model is 2× higher, and the testing communication cost is 4× greater compared to the public local models.The communication costs reported in Table 2 do not take into account the preprocessing communication costs for transmitting the FSS keys and securely computing each function.The preprocessing costs for the private local model is 10.572851 MB per batch.
In conclusion, private local model demonstrates its potential as a promising solution for maintaining data privacy while achieving commendable accuracy in ML.Nonetheless, it is important to remember that using private local model comes with significant drawbacks in terms of the time it takes to train models and the extra communication needed.This should be carefully weighed when deciding if it is the right choice when selecting a privacy-preserving framework for specific use cases.Private vanilla SL.The private vanilla SL model addresses privacy concerns in public while maintaining nearly the same accuracy, experiencing only a 2% reduction compared to both public local and public vanilla models (see Table 2).This is because, as previously discussed, both SL and FSS can be implemented without a notable reduction in model accuracy.It also achieves comparable accuracy to the private local model in around 192 minutes, a significant advantage.The shorter training time is due to executing all client-side layers in plain, unlike the private local model, which employs various techniques such as beaver triples (for convolution layers) and FSS (for ReLU).Additionally, the total communication cost of private vanilla SL is 2.9× lower than that of the private local model.Notably, the server-side communication cost for private vanilla SL is remarkably low (similar to the public vanilla SL model), compared to the private local model.It is important to note that private vanilla SL additionally requires 0.119634 MB of preprocessing communications per batch, which is a 88.3× reduction when compared to private local FSS (10.572851MB per batch).This is because we execute client-side layers on plaintext, whereas private local model involves more interaction between parties, increasing communication costs.In summary, private vanilla SL model outperforms the private local model in terms of training time and communication cost while maintaining similar accuracy.Moreover, it offers better privacy than either public models, though with a slight accuracy trade-off.It is important to mention that our private vanilla SL model has higher training time and communication costs compared to the public models.

Open Science and Reproducible Research:
To support open science and reproducible research, and provide researchers with opportunity to use, test, and extend our work, source code used for the evaluations is publicly available4 .

CONCLUSION
Although split learning was once viewed as a promising collaborative learning method for safeguarding the privacy of client data, recent studies have highlighted concerns about its effectiveness in preventing potential privacy leaks from client data.In response to these concerns, we have introduced a hybrid approach that combines split learning with another privacy-preserving technique known as function secret sharing.Our research demonstrates how the utilization of function secret sharing can effectively mitigate privacy breaches in split learning while still maintaining efficiency.
Our findings indicate that our proposed approach, which combines function secret sharing with vanilla split learning, outperforms existing protocols (such as private local models) in terms of communication efficiency and complexity.Notably, it achieves the same level of accuracy while offering improved privacy protections.

Figure 5
Figure5illustrates our local model, which consists of two Conv2D layers, two max-pooling layers, two FC layers and four ReLU layers.These layers operate without any split between client and server, with one party performing computations on the entire model.The training process of this model can be described as follows.Let |D| = {(  ,   )| = 1, 2, . . ., } be a training dataset consisting of  data samples.Each data sample  has a corresponding encoded label vector  that represents its ground-truth class.Then,   is a function representing the local model with  being a set of adjustable parameters. is first initialized to small random values denoted as Φ.The goal is to find the optimal parameters  to map  to a predicted output vector ŷ, where ŷ is as close as possible to . ŷ is a vector of probabilities (called posteriors), and we use the highest probability to determine the class to which  belongs.To find the closest value of ŷ with respect to , we try to minimize a loss function  = L( ŷ, ).Training the local model is an iterative process to find the best  to minimize  .This process involves two steps: forward and backward propagation.The forward propagation computes each layer in the network (Figure5) on input , reaching the output layer to predict ŷ.Backward propagation starts from the output layer and goes back to the input layer to calculate the gradients of the loss function  w.r.t the network's weights  .These weights are then updated according to the calculated gradients.We train the local model with numerous samples

Figure 8 :
Figure 8: Utilizing function secret sharing between two servers

Figure 9 :
Figure 9: Visual invertibility inference attack: First row shows the original images (image size: 28 × 28) from the MNIST dataset.Second row shows the  after the first Conv2D layer before the first max-pooling operation (image size: 24 × 24).Third row displays the  after the second Conv2D layer before the second max-pooling operation (image size: 8 × 8).

Figure 10 : 8 □Conference' 17 ,
Figure 10: Visual invertibility inference attack with FSS applied: First row shows the original  after the first Conv2D (before the max-pooling operations) from the MNIST dataset.Second row shows what  0 receives when the FSS protocol is applied.Third row what  1 receives when the FSS protocol is applied.(a) Image Size: 24 × 24; (b) Image Size: 8 × 8

Table 1 :
Parameters and description in the algorithms |D| is the size of the dataset.Once the model goes through all the training batches, it has completed one training  and this process continues for a total of  epochs.We implement and reproduce the results for this model.The best test accuracy that we obtained for this model is 99.29%.
of  and corresponding , through many iterations of forward and backward propagation.We do not train the network on each single data example, but use a number of them at a time (using batch size ).The total number of training batches is  = |D|  , where 11   ← L( ŷ ,   ) ()  ,  ()  18 end 19 .send   (+1) After training our local model for approximately 10 epochs, we achieved the highest test accuracy at 99.36%, with the second and third highest test accuracies at 99.33% and 99.28%, respectively.Furthermore, our experiments indicate that training the vanilla SL model on plaintext data yield similar accuracy results when compared to training the local model.As illustrated in Table 2, the top three results for the vanilla SL model are 99.29%,99.28%, and 99.21%.These accuracy values closely align, suggesting that the SL model can be effectively applied to CNN models on the MNIST dataset without experiencing a significant degradation in classification accuracy.We also consider the training time and communication cost of the vanilla SL model and compare it to the local model.The total training time for 10 epochs is nearly identical, ranging between 1 and 2 minutes.Interestingly, even though the vanilla model involve communication between the client and server to exchange  and gradients, their training times remain similar to that of the local model.This similarity arises because the local model's training time encompasses the duration during which the client transmits images to the server.Consequently, both models have nearly the same training times.

Table 2 :
Performance comparison: training and testing results for various privacy-preserving models