Abstract
Because of the powerful computing and storage capability in cloud computing, machine learning as a service (MLaaS) has recently been valued by the organizations for machine learning training over some related representative datasets. When these datasets are collected from different organizations and have different distributions, multi-task learning (MTL) is usually used to improve the generalization performance by scheduling the related training tasks into the virtual machines in MLaaS and transferring the related knowledge between those tasks. However, because of concerns about privacy breaches (e.g., property inference attack and model inverse attack), organizations cannot directly outsource their training data to MLaaS or share their extracted knowledge in plaintext, especially the organizations in sensitive domains. In this article, we propose a novel privacy-preserving mechanism for distributed MTL, namely NOInfer, to allow several task nodes to train the model locally and transfer their shared knowledge privately. Specifically, we construct a single-server architecture to achieve the private MTL, which protects task nodes’ local data even if \(n-1\) out of \(n\) nodes colluded. Then, a new protocol for the Alternating Direction Method of Multipliers (ADMM) is designed to perform the privacy-preserving model training, which resists the inference attack through the intermediate results and ensures that the training efficiency is independent of the number of training samples. When releasing the trained model, we also design a differentially private model releasing mechanism to resist the membership inference attack. Furthermore, we analyze the privacy preservation and efficiency of NOInfer in theory. Finally, we evaluate our NOInfer over two testing datasets and evaluation results demonstrate that NOInfer efficiently and effectively achieves the distributed MTL.
- [1] . 2016. Deep learning with differential privacy. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. 308–318. Google Scholar
Digital Library
- [2] . 2015. Machine learning classification over encrypted data. In Proceedings of the 22nd Annual Network and Distributed System Security Symposium. 1–14.Google Scholar
Cross Ref
- [3] . 2011. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning 3, 1 (2011), 1–122. Google Scholar
Digital Library
- [4] . 2011. Differentially private empirical risk minimization. Journal of Machine Learning Research 12 (2011), 1069–1109. Google Scholar
Digital Library
- [5] . 2006. Differential privacy. In Proceedings of Automata, Languages and Programming, 33rd International Colloquium, Part II. 1–12. Google Scholar
Digital Library
- [6] . 2016. Calibrating noise to sensitivity in private data analysis. Journal of Privacy and Confidentiality 7, 3 (2016), 17–51.Google Scholar
Cross Ref
- [7] . 2020. Privacy-preserving Gaussian process regression—A modular approach to the application of homomorphic encryption. In Proceedings of the 34th AAAI Conference on Artificial Intelligence. 3866–3873.Google Scholar
Cross Ref
- [8] . 2013. TrPF: A trajectory privacy-preserving framework for participatory sensing. IEEE Transactions on Information Forensics and Security 8, 6 (2013), 874–887. Google Scholar
Digital Library
- [9] . 2004. The Foundations of Cryptography—Volume 2, Basic Applications. Cambridge University Press. Google Scholar
Digital Library
- [10] . 2019. Social community detection and message propagation scheme based on personal willingness in social network. Soft Computing 23, 15 (2019), 6267–6285. Google Scholar
Digital Library
- [11] . 2020. DP-ADMM: ADMM-based distributed learning with differential privacy. IEEE Transactions on Information Forensics and Security 15 (2020), 1002–1012.Google Scholar
Digital Library
- [12] . 2016. Federated learning: Strategies for improving communication efficiency. CoRR abs/1610.05492 (2016).Google Scholar
- [13] . 2017. Multi-key privacy-preserving deep learning in cloud computing. Future Generation Computer Systems 74 (2017), 76–85. Google Scholar
Digital Library
- [14] . 2020. NPMML: A framework for non-interactive privacy-preserving multi-party machine learning. IEEE Transactions on Dependable and Secure Computing. Early access, February 4, 2020.Google Scholar
Cross Ref
- [15] . 2017. Distributed multi-task relationship learning. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, 937–946. Google Scholar
Digital Library
- [16] . 2016. An efficient privacy-preserving outsourced calculation toolkit with multiple keys. IEEE Transactions on Information Forensics and Security 11, 11 (2016), 2401–2414.Google Scholar
Digital Library
- [17] . 2020. Towards fair and privacy-preserving federated deep models. IEEE Transactions on Parallel and Distributed Systems 31, 11 (2020), 2524–2541.Google Scholar
Cross Ref
- [18] . 2017. APPLET: A privacy-preserving framework for location-aware recommender system. Science China Information Sciences 60, 9 (2017), 092101.Google Scholar
Cross Ref
- [19] . 2018. ARMOR: A trust-based privacy-preserving framework for decentralized friend recommendation in online social networks. Future Generation Computer Systems 79 (2018), 82–94.Google Scholar
Cross Ref
- [20] . 2018. PDLM: Privacy-preserving deep learning model on cloud with multiple keys. IEEE Transactions on Services Computing 14, 4 (2018), 1251–1263.Google Scholar
Cross Ref
- [21] . 2018. Inference attacks against collaborative learning. CoRR abs/1805.04049 (2018).Google Scholar
- [22] . 2019. Exploiting unintended feature leakage in collaborative learning. In Proceedings of the 2019 IEEE Symposium on Security and Privacy. 691–706.Google Scholar
Cross Ref
- [23] . 2017. SecureML: A system for scalable privacy-preserving machine learning. In Proceedings of the IEEE Symposium on Security and Privacy. 19–38.Google Scholar
Cross Ref
- [24] . 2018. Machine learning with membership privacy using adversarial regularization. In Proceedings of ACM SIGSAC Conference on Computer and Communications Security. 634–646. Google Scholar
Digital Library
- [25] . 2018. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security 13, 5 (2018), 1333–1345. Google Scholar
Digital Library
- [26] . 2019. Privacy-preserving deep learning via weight transmission. IEEE Transactions on Information Forensics and Security 14, 11 (2019), 3003–3015. Google Scholar
Digital Library
- [27] . 2019. Trident: Efficient 4PC framework for privacy preserving machine learning. CoRR abs/1912.02631 (2019).Google Scholar
- [28] . 2015. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. 1310–1321. Google Scholar
Digital Library
- [29] . 2017. Membership inference attacks against machine learning models. In Proceedings of IEEE Symposium on Security and Privacy. 3–18.Google Scholar
Cross Ref
- [30] . 2019. Effects of differential privacy and data skewness on membership inference vulnerability. CoRR abs/1911.09777 (2019).Google Scholar
- [31] . 2020. FALCON: Honest-majority maliciously secure framework for private deep learning. CoRR abs/2004.02229 (2020).Google Scholar
- [32] . 2018. Dynamic trust relationships aware data privacy protection in mobile crowd-sensing. IEEE Internet of Things Journal 5, 4 (2018), 2958–2970.Google Scholar
Cross Ref
- [33] . 2017. Privacy-preserving distributed multi-task learning with asynchronous updates. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1195–1204. Google Scholar
Digital Library
- [34] . 2020. VerifyNet: Secure and verifiable federated learning. IEEE Transactions on Information Forensics and Security 15 (2020), 911–926.Google Scholar
Digital Library
- [35] . 2020. Multi-user multi-keyword rank search over encrypted data in arbitrary language. IEEE Transactions on Dependable and Secure Computing 17, 2 (2020), 320–334.Google Scholar
Digital Library
- [36] . 2020. A privacy-preserving multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. Frontiers in Neurorobotics 13 (2020), 112.Google Scholar
Cross Ref
- [37] . 2016. Privacy preserving deep computation model on cloud for big data feature learning. IEEE Transactions on Computers 65, 5 (2016), 1351–1362. Google Scholar
Digital Library
- [38] . 2017. Dynamic differential privacy for ADMM-based distributed classification learning. IEEE Transactions on Information Forensics and Security 12, 1 (2017), 172–187. Google Scholar
Digital Library
- [39] . 2018. Privacy-preserving collaborative deep learning with irregular participants. CoRR abs/1812.10113 (2018).Google Scholar
- [40] . 2019. Helen: Maliciously secure coopetitive learning for linear models. In Proceedings of the 2019 IEEE Symposium on Security and Privacy. 724–738.Google Scholar
Cross Ref
Index Terms
Privacy-Preserving Distributed Multi-Task Learning against Inference Attack in Cloud Computing
Recommendations
Privacy-Preserving Distributed Multi-Task Learning with Asynchronous Updates
KDD '17: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data MiningMany data mining applications involve a set of related learning tasks. Multi-task learning (MTL) is a learning paradigm that improves generalization performance by transferring knowledge among those tasks. MTL has attracted so much attention in the ...
A review of privacy preserving models for multi-party data release framework
WIR '16: Proceedings of the ACM Symposium on Women in Research 2016Nowadays, with the improvement of internet technology and advancement in distributed computing data is increasing rapidly. There is a need of information sharing between organizations. Ideally, we wish to share data from multiple private databases and ...
Privacy-preserving data sharing in cloud computing
Storing and sharing databases in the cloud of computers raise serious concern of individual privacy. We consider two kinds of privacy risk: presence leakage, by which the attackers can explicitly identify individuals in (or not in) the database, and ...






Comments