Optimizing Privacy in Federated Learning with MPC and Differential Privacy

With the proliferation of information technologies, data has become a fundamental driver of societal development. However, issues related to data regulation, data rights confirmation, and data protection hinder data circulation, leading to "data islands" and raising concerns about data privacy and security. In this context, achieving a balance between data circulation and privacy protection is crucial. Federated Learning (FL) is a distributed machine learning methodology that allows multiple client devices to collaboratively construct a model without transmitting their local data to a cloud server. However, privacy concerns persist due to potential disclosure of clients’ private information through their submitted updates. To mitigate these risks, researchers have integrated differential privacy (DP) techniques into FL, where clients enhance the privacy of their local parameter updates by introducing random noise. This paper presents a novel approach that combines Secure Multi-Party Computation (MPC) with DP to ensure privacy without compromising accuracy. Our approach is resilient to collusion attacks and eliminates potential leakage of any input in the output. Unlike previous solutions, our approach prevents the server from inferring the noisy weights of any specific user, making the system fully private. Our contributions include a more practical assumption of network connectivity and a simplified communication structure of MPC, reducing the required MPC communication overhead and enhancing efficiency. This work provides a significant step towards achieving a balance between data circulation and privacy protection in the era of big data.


ABSTRACT
With the proliferation of information technologies, data has become a fundamental driver of societal development.However, issues related to data regulation, data rights confirmation, and data protection hinder data circulation, leading to "data islands" and raising concerns about data privacy and security.In this context, achieving a balance between data circulation and privacy protection is crucial.Federated Learning (FL) is a distributed machine learning methodology that allows multiple client devices to collaboratively construct a model without transmitting their local data to a cloud server.However, privacy concerns persist due to potential disclosure of clients' private information through their submitted updates.To mitigate these risks, researchers have integrated differential privacy (DP) techniques into FL, where clients enhance the privacy of their local parameter updates by introducing random noise.This paper presents a novel approach that combines Secure Multi-Party Computation (MPC) with DP to ensure privacy without compromising accuracy.Our approach is resilient to collusion attacks and eliminates potential leakage of any input in the output.Unlike previous solutions, our approach prevents the server from inferring the noisy weights of any specific user, making the system fully private.Our contributions include a more practical assumption of network connectivity and a simplified communication structure of MPC, reducing the required MPC communication overhead and enhancing efficiency.This work provides a significant step towards achieving a balance between data circulation and privacy protection in the era of big data.

INTRODUCTION
With the rapid development of information technologies such as mobile internet, cloud computing, artificial intelligence, and 5G, a variety of new service models and data applications are continuously emerging and diversifying.Data, exhibiting massive and explosive growth across all sectors, has become one of the most important and fundamental elements driving societal development.However, the current state of data circulation is hindered by issues related to data regulation, data rights confirmation, and data protection, preventing the large-scale application of traditional data circulation models.
On one hand, due to competitive considerations and data security, it is challenging to achieve data resource integration and sharing within and between industries, exacerbating the problem of "data islands".On the other hand, with the increasing prominence of data privacy and security issues, regulatory compliance worldwide is tightening.While stricter data regulation promotes data privacy protection to some extent, it also raises concerns about the compliance of data circulation.Moreover, frequent incidents of user privacy data breaches in recent years pose significant challenges to data circulation.Therefore, in the context where data interconnection has become a development trend, the question of how to effectively break the "data island" barriers while ensuring privacy security, promote the "soft landing" of policy regulation "hard institutions" in a more flexible manner, and achieve a balance between data circulation and privacy protection, has become a focal point of widespread concern in the industry.
Federated Learning (FL) [5,8,9] is a distributed methodology in machine learning where a multitude of client devices collaboratively construct a model, circumventing the need to transmit their local data to a cloud server.During each round of federated learning, all chosen clients download the global model from the server, perform local training with their private data, and share their local model updates with the server via encrypted communication.The server, in turn, amalgamates these local updates to formulate a fresh global model, adhering to a pre-established aggregation rule, and disseminates it to the clients for the ensuing round.Contrasting with centralized machine learning, FL significantly attenuates the privacy concerns of clients by obviating the necessity for data centralization.
However, the privacy concerns associated with FL remain significant due to the potential disclosure of clients' private information through the analysis of their submitted updates, such as gradients and model parameters.Specifically, even when participants only share gradients, malicious attackers could potentially infer participants' private data by analyzing these gradients.Recent studies have shown that attackers can employ membership inference attacks [7,13,16] to ascertain if a particular sample belongs to a client's training set.Meanwhile, class representative reconstruction attacks [2,6,15] can deduce class representatives, and property inference attacks [3,11] can expose sensitive attributes.Moreover, the most severe privacy threat arises from gradient leakage attacks (GLAs) [4,18,20,22,23].By minimizing the disparity between dummy gradients of the global model when fed with dummy data and the actual gradients shared by clients, attackers can approximate the original private training data.This allows them to retrieve private training data from the gradients that clients publicly share.A clear trade-off exists: the incorporation of randomness into the collected data safeguards user privacy, albeit at the expense of data accuracy.
To mitigate privacy risks in FL, researchers have integrated differential privacy (DP) techniques [10,14,17,19,21] into FL where clients can enhance the privacy of their local parameter updates by introducing random noise into gradients during the implementation of stochastic gradient descent (SGD) for local training, thereby obfuscating the updates before sharing them.As in the DP-based method, the larger the injected DP noise, the stronger the privacy clause; nevertheless excessive noise seriously degrades the model performance and reduce the training efficiency.
Secure Multi-Party Computation (MPC) [12] presents a promising approach to address the aforementioned trade-off, offering privacy assurance without sacrificing accuracy.In an MPC setup, multiple entities collaboratively execute a computation of shared interest, while keeping their individual inputs concealed.An MPC protocol is deemed secure if the entities gain no knowledge beyond the final computation result.While MPC-based solutions appear ideal as they only disclose information about the final outcome, they are vulnerable to collusion attacks.By integrating differential privacy with MPC, we can engineer a system robust against such sophisticated collusion attacks.By introducing noise to each input, we maintain the accuracy of the final computation within a known boundary, while eliminating potential leakage of any individual input.In contrast to previous solutions that solely employed differential privacy, where the server had access to the "noisy" private weights of each user, our combined MPC and differential privacy solution encrypts the noisy weights sent to the server, thereby preventing the server from deducing the noisy weights of any particular user.Consequently, the system achieves complete privacy.
Our contributions, compared to previous work [1], are as follows.Firstly, previous work made overly idealistic and stringent assumptions, assuming network connectivity between any two users to exchange necessary information, which may not be feasible in practice.In our work, any user only needs to form a communication ring structure with two other users.Simultaneously, due to the simplified communication structure of secure multi-party computation, the required MPC communication overhead is reduced, making our approach more efficient than previous work .

PRELIMINARIES
In this section, we briefly recall differential privacy and Laplacian Mechanism respectively.

Differential Privacy
In the realm of privacy preservation research, -differential privacy is a robust privacy protection mechanism.It is defined as follows: A randomized algorithm  satisfies -differential privacy if for any pair of adjacent datasets  1 and  2 (i.e.,  1 and  2 differ by only one record), and any possible output subset , it holds that: This implies that an adversary cannot determine whether a specific record is present in the input dataset by observing the algorithm's output.
Global sensitivity is a key concept in differential privacy.It measures the maximum change in the output of a function due to the change of a single record in the input dataset.Formally, for a realvalued query function  :  →   , the global sensitivity Δ of  is defined as: Here, 1 and  2 are any pair of datasets that differ by exactly one record (i.e., they are adjacent), and || • || 1 denotes the L1 norm (sum of absolute values).
The global sensitivity of a function is used to calibrate the amount of noise added to the function's output in order to ensure differential privacy.The higher the global sensitivity, the more noise is needed to preserve privacy.

Laplacian Mechanism
The essence of differential privacy is probabilistic randomness, that is, all differential privacy mechanisms are random.To ensure the differential privacy of a random algorithm, a noise mechanism is introduced to add noise to the data, thereby meeting the requirements of differential privacy.
The Laplace mechanism protects user privacy by adding Laplace noise to the original data to change the true value, making it satisfy differential privacy before and after adding noise.For the query data result function (),  is added randomly to get () + .The probability density function of the noise which is Laplacedistributed with zero mean and scale  defined as:

𝑏
This formula characterizes the Laplace distribution, frequently employed in differential privacy for data perturbation.The scale parameter, , is contingent upon the global sensitivity of the query function and the specified privacy level.Given the global sensitivity, Δ, of the query function , and the privacy parameter , the Laplacian mechanism  introduces random perturbation  following the Laplacian distribution with scale  = Δ  .Consequently, the Laplacian mechanism ensures -differential privacy.In logistic regression, we aim to predict a binary outcome based on a set of features.The logistic regression model can be represented as:

ENHANCING PRIVACY AND EFFICIENCY IN FL THROUGH SECURE MPC AND DP
where  ( = 1|) is the probability of the target variable  being 1 given the features , and  and   are the model parameters to be estimated from the training data.
In the context of FL, each client  has a local dataset   and computes the local model updates using its own data.The local updates are represented as Δ  () =   () −   (), where   () is the model parameters at the beginning of round , and   ()  is the model parameters after  local iterations.
After computing the local updates, each client sends Δ  () to the server.The server aggregates these updates from all participating clients to update the global model: where   is the number of clients participating in round , and  () is the global model parameters at round .
This process continues for  rounds until the model converges.The final model is a result of collaborative learning from all clients, and it ensures privacy as the raw data never leaves each client's local device.

Federated Learning with Local Differential Privacy
Federated For a 1-Lipschitz function, the global sensitivity for a multi-party setting is 2       , where   is the size of the smallest dataset among the   clients, and  is the regularization parameter.Therefore,  = Laplace( 2     ), where  is the privacy loss parameter.

Secure Model Update Protocol
To safeguard each client's model weights from the server, we employ secure multi-party computation (MPC).This method enables clients to encrypt their individual updates, ensuring their privacy before transmitting them to the server.Our secure weighted average protocol is designed such that clients share a unique source of randomness.During the process of weight communication, clients form a communication ring structure.Each client in the ring adds the shared random number provided by the preceding client to the shared randomness and subtracts the shared random number given by the subsequent client.This mechanism ensures that the server's averaged model aligns with a model produced without the use of MPC, but each weight from each client is obfuscated by the addition or subtraction of two large random numbers.This effectively prevents the server from accurately reconstructing any client's actual model weights, thereby enhancing the security of the system.Our protocol allows for the reuse of common randomness in each logistic regression iteration, requiring pairwise communication only once at the protocol's start.In later iterations, each client communicates solely with the server.An example with  parties is illustrated in Figure 1, where weights are encrypted using common randomness  .The values  do not disclose anything about the weights .
While our protocol conceals all information about each client's weights from the server, the shared learned model could still reveal some information about individual client weights and their local data set.To prevent this, even in cases of collusion among clients, we incorporate differential privacy within the MPC protocol.Each client independently generates and adds random "noise" to each of Our secure model update protocol refers to the algorithm outlined in Algorithm 1, for secure logistic regression performed by a set of clients ( 1 , ...,   ) and a server .All operations are performed modulo some bound .
During setup, each neighbour pair of parties   and  +1 will share some common randomness  , =  , .In the online weighted average phase, client   sends its weights masked with these common random strings, adding  ,+1 and subtracting  , −1 .That is,   sends to server  the following message for its data   : mod  In this formula,   represents the weights of the -th client,  ,+1 and  , −1 are the common random strings shared between the -th client and the  + 1-th and  − 1-th clients respectively, and  is the modulo bound.

EXPERIMENTS
To assess our proposed method, we have incorporated it into ABIDES, an open-source, agent-based interactive discrete event simulation framework.ABIDES operates in a single-threaded mode, enabling deterministic re-simulation even in the presence of stochastic components.It simulates the parallel operations of tens of thousands of agents.The simulation Kernel, which manages time in nanoseconds, enforces "simulation physics" that include computation delays for agents and noisy latency in pairwise communication between agents.These computation delays can be preset or adjusted during execution to match the actual computation time.
All communication between agents is managed by the Kernel, which uses a priority queue for timestamped messages.The discrete event simulation nature of the system allows for efficient computation of sparse activity patterns at a high time resolution.Unlike many previous studies on federated learning that overlook the server communication time when calculating their protocol's running time, our implementation allows us to simulate the latency of distributed client communication.All experiments are conducted on an Intel Xeon E5-4214 (2.20 GHz) machine with an Nvidia Tesla V100 GPU.
We evaluated the protocol using simulations, which allowed us to accurately build a model of how long it would take to run such a protocol in the real world.To achieve this, each simulated client times each part of its protocol section, capturing the running the key exchange phase (once), the encryption and privacy steps (each iteration), and the local model training step (each iteration) The actual time required.
In Table 1, we provide timing results with the previous work [1] on different phases running time.The experimental settings basically follow the previous work, except for the network environment where the experiment is located.From the experimental results, it can be concluded that our method is better than previous work, reducing the communication complexity from  ( 2 ) to  ().To further demonstrate the effectiveness and efficiency of our method, we compare it with other methods.Our approach outperforms previous work in terms of total protocol time, average server time per iteration, and client key exchange.These results demonstrate the superiority of our approach in reducing communication complexity and improving computational efficiency.Furthermore, our approach is also better scalable and can handle larger-scale agent networks without significantly increasing the computational burden.This gives our approach a greater advantage when dealing with large-scale distributed systems.

CONCLUSION
In this paper, we has presented a novel approach to address the challenges of data privacy and security in the context of Federated Learning (FL).By integrating differential privacy (DP) techniques and Secure Multi-Party Computation (MPC), we have proposed a system that not only ensures the privacy of individual data but also maintains the accuracy of the final computation.Our approach significantly mitigates the risk of privacy breaches, as it obfuscates the local parameter updates before sharing them, thereby preventing potential attackers from inferring private data.Furthermore, by introducing noise to each input, we have managed to maintain the accuracy of the final computation within a known boundary, while eliminating potential leakage of any individual input.Unlike previous solutions that solely employed differential privacy, where the server had access to the "noisy" private weights of each user, our combined MPC and differential privacy solution encrypts the noisy weights sent to the server, thereby preventing the server from deducing the noisy weights of any particular user.This ensures complete privacy.Moreover, our approach simplifies the communication structure of secure multi-party computation, reducing the required MPC communication overhead and making our system more efficient than previous work.We have also relaxed the stringent assumptions made in previous work about network connectivity, making our approach more practical and feasible for real-world applications.In the future, we aim to further optimize our system and explore its potential applications in various industries.We believe that our work contributes significantly to the ongoing efforts to balance data circulation and privacy protection, and we hope that it will inspire further research in this important area.

Federated learning operates on
a publish-subscribe model where the server broadcasts a federated learning task, encompassing data structure, data modality, neural network model, and key hyperparameters.A client subscribes to a task, communicates its local training dataset size to the server, and commits to downloading the joint training model.The client then adheres to the federated learning protocol: for each round, it retrieves the current global model parameters from the server and performs local model training on its private data.The client then sends the local training parameter updates back to the server.The server aggregates these updates from all clients to produce the global model parameter updates, triggering the next round of joint training.This process continues until all rounds are completed.To accommodate fluctuating client availability, only a subset of clients are selected to participate in each round of federated learning.

Figure 1 :
Figure 1: Secure n-party Model Update Protocol

Table 1 :
Timing results on different phases running time.