research-article

Public Access

FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy

Authors:

Jianqing Zhang,

Haibing GuanAuthors Info & Claims

KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Pages 3249 - 3261

https://doi.org/10.1145/3580305.3599345

Published: 04 August 2023 Publication History

Abstract

Recently, personalized federated learning (pFL) has attracted increasing attention in privacy protection, collaborative learning, and tackling statistical heterogeneity among clients, e.g., hospitals, mobile smartphones, etc. Most existing pFL methods focus on exploiting the global information and personalized information in the client-level model parameters while neglecting that data is the source of these two kinds of information. To address this, we propose the Federated Conditional Policy (FedCP) method, which generates a conditional policy for each sample to separate the global information and personalized information in its features and then processes them by a global head and a personalized head, respectively. FedCP is more fine-grained to consider personalization in a sample-specific manner than existing pFL methods. Extensive experiments in computer vision and natural language processing domains show that FedCP outperforms eleven state-of-the-art methods by up to 6.69%. Furthermore, FedCP maintains its superiority when some clients accidentally drop out, which frequently happens in mobile settings. Our code is public at https://github.com/TsingZ0/FedCP.

Supplementary Material

MP4 File (rtfp0722-2min-promo.mp4)

Promotional video

Download
45.18 MB

MP4 File (rtfp0722-20min-video.mp4)

The presentation video for our paper: FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy

Download
216.04 MB

References

[1]

Mohammad Al-Rubaie and J Morris Chang. 2016. Reconstruction attacks against mobile-based continuous authentication systems in the cloud. IEEE Transactions on Information Forensics and Security, Vol. 11, 12 (2016), 2648--2663.

[2]

Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. 2019. Federated Learning with Personalization Layers. arXiv preprint arXiv:1912.00818 (2019).

[3]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer Normalization. arXiv preprint arXiv:1607.06450 (2016).

[4]

Hong-You Chen and Wei-Lun Chao. 2021. On Bridging Generic and Personalized Federated Learning for Image Classification. In International Conference on Learning Representations (ICLR).

[5]

Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. 2017. A Downsampled Variant of Imagenet as an Alternative to the Cifar Datasets. arXiv preprint arXiv:1707.08819 (2017).

[6]

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. 2021. Exploiting Shared Representations for Personalized Federated Learning. In International Conference on Machine Learning (ICML).

[7]

Lydia de la Torre. 2018. A guide to the california consumer privacy act of 2018. Available at SSRN 3275571 (2018).

[8]

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. 2020. Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[9]

Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. 2020. Inverting gradients-how easy is it to break privacy in federated learning?. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[10]

Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. 2006. A Kernel Method for the Two-Sample-Problem. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[11]

Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. 2019. Spottune: Transfer Learning through Adaptive Fine-Tuning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]

Seok-Ju Hahn, Minwoo Jeong, and Junghye Lee. 2022. Connecting Low-Loss Subspace for Personalized Federated Learning. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD).

[13]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]

Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531, Vol. 2, 7 (2015).

[15]

Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. 2021. Personalized Cross-Silo Federated Learning on Non-IID Data. In AAAI Conference on Artificial Intelligence (AAAI).

[16]

Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning (ICML).

Digital Library

[17]

Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical Reparameterization with Gumbel-Softmax. arXiv preprint arXiv:1611.01144 (2016).

[18]

Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of Tricks for Efficient Text Classification. In Conference of the European Chapter of the Association for Computational Linguistics (EACL).

[19]

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. 2019. Advances and Open Problems in Federated Learning. arXiv preprint arXiv:1912.04977 (2019).

[20]

Alex Krizhevsky and Hinton Geoffrey. 2009. Learning Multiple Layers of Features From Tiny Images. Technical Report (2009).

[21]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep Learning. Nature, Vol. 521, 7553 (2015), 436--444.

[22]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-Based Learning Applied to Document Recognition. Proc. IEEE, Vol. 86, 11 (1998), 2278--2324.

[23]

Qinbin Li, Bingsheng He, and Dawn Song. 2021a. Model-Contrastive Federated Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. 2021b. Ditto: Fair and Robust Federated Learning Through Personalization. In International Conference on Machine Learning (ICML).

[25]

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020a. Federated Learning: Challenges, Methods, and Future Directions. IEEE Signal Processing Magazine, Vol. 37, 3 (2020), 50--60.

[26]

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020b. Federated Optimization in Heterogeneous Networks. In Conference on Machine Learning and Systems (MLSys).

[27]

Xin-Chun Li, De-Chuan Zhan, Yunfeng Shao, Bingshuai Li, and Shaoming Song. 2021c. FedPHP: Federated Personalization with Inherited Private Models. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML).

[28]

Yuanzhi Li and Yang Yuan. 2017. Convergence Analysis of Two-Layer Neural Networks with Relu Activation. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[29]

Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. 2020. Ensemble Distillation for Robust Model Fusion in Federated Learning. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[30]

Lanlan Liu and Jia Deng. 2018. Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-Offs by Selective Execution. In AAAI Conference on Artificial Intelligence (AAAI).

[31]

Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. 2021. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[32]

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In International Conference on Artificial Intelligence and Statistics (AISTATS).

[33]

Kevin P Murphy. 2012. Machine learning: a probabilistic perspective. MIT press.

Digital Library

[34]

Dinh C Nguyen, Ming Ding, Pubudu N Pathirana, Aruna Seneviratne, Jun Li, and H Vincent Poor. 2021. Federated learning for internet of things: A comprehensive survey. IEEE Communications Surveys & Tutorials, Vol. 23, 3 (2021), 1622--1658.

[35]

Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. 2018. Tadam: Task Dependent Adaptive Metric for Improved Few-Shot Learning. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[36]

Florent Perronnin, Jorge Sánchez, and Thomas Mensink. 2010. Improving the Fisher Kernel for Large-scale Image Classification. In European Conference on Computer Vision (ECCV).

[37]

Can Qin, Haoxuan You, Lichen Wang, C-C Jay Kuo, and Yun Fu. 2019. Pointdan: A Multi-Scale 3D Domain Adaption Network for Point Cloud Representation. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[38]

Protection Regulation. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council. Regulation (eu), Vol. 679 (2016), 2016.

[39]

Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. 2020. Balanced Meta-softmax for Long-tailed Visual Recognition. International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[40]

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual Explanations from Deep Networks via Gradient-based Localization. In IEEE International Conference on Computer Vision (ICCV).

[41]

Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. 2021. Personalized Federated Learning using Hypernetworks. In International Conference on Machine Learning (ICML).

[42]

Canh T Dinh, Nguyen Tran, and Tuan Dung Nguyen. 2020. Personalized Federated Learning with Moreau Envelopes. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[43]

Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. 2022. Towards Personalized Federated Learning. IEEE Transactions on Neural Networks and Learning Systems (2022).

[44]

Andreas Veit and Serge Belongie. 2018. Convolutional Networks with Adaptive Inference Graphs. In European Conference on Computer Vision (ECCV).

[45]

Chi Wang, Yang Hua, Zheng Lu, Jian Gao, and Neil Robertson. 2021. Temporal Meta-Adaptor for Video Object Detection. In British Machine Vision Conference (BMVC).

[46]

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. 2020. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[47]

Yuxin Wu and Kaiming He. 2018. Group Normalization. In European Conference on Computer Vision (ECCV).

[48]

Yuezhou Wu, Yan Kang, Jiahuan Luo, Yuanqin He, Lixin Fan, Rong Pan, and Qiang Yang. 2022. FedCG: Leverage Conditional GAN for Protecting Privacy and Maintaining Competitive Performance in Federated Learning. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[49]

Liu Yang, Ben Tan, Vincent W Zheng, Kai Chen, and Qiang Yang. 2020. Federated Recommendation Systems. In Federated Learning. 225--239.

[50]

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated Machine Learning: Concept and Applications. ACM Transactions on Intelligent Systems and Technology, Vol. 10, 2 (2019), 1--19.

Digital Library

[51]

Rui Ye, Mingkai Xu, Jianyu Wang, Chenxin Xu, Siheng Chen, and Yanfeng Wang. 2023. FedDisco: Federated Learning with Discrepancy-Aware Collaboration. arXiv preprint arXiv:2305.19229 (2023).

[52]

Lei Yu and Huan Liu. 2003. Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution. In International Conference on Machine Learning (ICML).

[53]

Honglei Zhang, Fangyuan Luo, Jun Wu, Xiangnan He, and Yidong Li. 2023 b. LightFR: Lightweight federated recommendation with privacy-preserving matrix factorization. ACM Transactions on Information Systems (2023), 1--28.

[54]

Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. 2023 a. FedALA: Adaptive Local Aggregation for Personalized Federated Learning. In AAAI Conference on Artificial Intelligence (AAAI).

[55]

Jianqing Zhang, Dongjing Wang, and Dongjin Yu. 2021. TLSAN: Time-aware long-and short-term attention network for next-item recommendation. Neurocomputing, Vol. 441 (2021), 179--191.

[56]

Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. 2020. Personalized Federated Learning with First Order Model Optimization. In International Conference on Learning Representations (ICLR).

[57]

Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recommender system: A survey and new perspectives. ACM Computing Surverys, Vol. 52, 1 (2019), 1--38.

[58]

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-Level Convolutional Networks for Text Classification. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

[59]

Ligeng Zhu, Zhijian Liu, and Song Han. 2019. Deep Leakage from Gradients. In International Conference on Advances in Neural Information Processing Systems (NeurIPS).

Cited By

Yu HYang XGao XKang YWang HZhang JLi TBaeza-Yates RBonchi F(2024)Personalized Federated Continual Learning via Multi-Granularity PromptProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671948(4023-4034)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671948
Kiani SBoenisch FDraper S(2024)Controlled Privacy Leakage Propagation Throughout Overlapping Grouped LearningIEEE Journal on Selected Areas in Information Theory10.1109/JSAIT.2024.34160895(442-463)Online publication date: 2024
https://doi.org/10.1109/JSAIT.2024.3416089
Zhang JLiu YHua YCao J(2024)An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01151(12109-12119)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01151
Show More Cited By

Index Terms

FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy
1. Computing methodologies

Recommendations

Personalized Federated Recommendation via Joint Representation Learning, User Clustering, and Model Adaptation
CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management

Federated recommendation applies federated learning techniques in recommendation systems to help protect user privacy by exchanging models instead of raw user data between user devices and the central server. Due to the heterogeneity in user's ...
Towards fair and personalized federated recommendation
Abstract
Recommender systems have gained immense popularity in recent years for predicting users’ interests by learning embeddings. The majority of existing recommendation approaches, represented by graph neural network-based recommendation algorithms, ...
Highlights
- We design a novel federated recommendation framework, which takes fairness and personalization of recommendation into consideration simultaneously.
- In the proposed framework, user local model, cluster-level model, and global model are ...
Connecting Low-Loss Subspace for Personalized Federated Learning
KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Due to the curse of statistical heterogeneity across clients, adopting a personalized federated learning method has become an essential choice for the successful deployment of federated learning-based services. Among diverse branches of personalization ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 2023

5996 pages

ISBN:9798400701030

DOI:10.1145/3580305

General Chairs:
Ambuj Singh
UC Santa Barbara, USA
,
Yizhou Sun
UC Los Angeles, USA
,
Program Chairs:
Leman Akoglu
Carnegie Mellon University, USA
,
Dimitrios Gunopulos
University of Athens, Greece
,
Xifeng Yan
UC Santa Barbara, USA
,
Ravi Kumar
Google, USA
,
Fatma Ozcan
Google, USA
,
Jieping Ye
Alibaba DAMO Academy

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 August 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Intel Corporation
Wuxi IoT Innovation Promotion Center
NSF (National Science Foundation)
National Key R&D Program of China
Eighth Research Institute in China Aerospace Science and Technology Corporation (Shanghai)

Conference

KDD '23

Sponsor:

KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 6 - 10, 2023

CA, Long Beach, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
920
Total Downloads

Downloads (Last 12 months)844
Downloads (Last 6 weeks)79

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yu HYang XGao XKang YWang HZhang JLi TBaeza-Yates RBonchi F(2024)Personalized Federated Continual Learning via Multi-Granularity PromptProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671948(4023-4034)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671948
Kiani SBoenisch FDraper S(2024)Controlled Privacy Leakage Propagation Throughout Overlapping Grouped LearningIEEE Journal on Selected Areas in Information Theory10.1109/JSAIT.2024.34160895(442-463)Online publication date: 2024
https://doi.org/10.1109/JSAIT.2024.3416089
Zhang JLiu YHua YCao J(2024)An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01151(12109-12119)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01151
Zhong LZeng JWang ZZhou WWen J(2024)SCFL: Spatio-temporal consistency federated learning for next POI recommendationInformation Processing & Management10.1016/j.ipm.2024.10385261:6(103852)Online publication date: Nov-2024
https://doi.org/10.1016/j.ipm.2024.103852
Zhu GLiu XNiu JWei YTang SZhang J(2024)Learning by imitating the classics: Mitigating class imbalance in federated learning via simulated centralized learningExpert Systems with Applications10.1016/j.eswa.2024.124755255(124755)Online publication date: Dec-2024
https://doi.org/10.1016/j.eswa.2024.124755
Yin KJi XWang YWang Z(2024)FedCLCC: A personalized federated learning algorithm for edge cloud collaboration based on contrastive learning and conditional computingDefence Technology10.1016/j.dt.2024.08.015Online publication date: Aug-2024
https://doi.org/10.1016/j.dt.2024.08.015
Zhang JHua YCao JWang HSong TXue ZMa RGuan HOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Eliminating domain bias for federated learning in representation spaceProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666747(14204-14227)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3666747
Chen HChen XPeng LBai Y(2023)Personalized Fair Split Learning for Resource-Constrained Internet of ThingsSensors10.3390/s2401008824:1(88)Online publication date: 23-Dec-2023
https://doi.org/10.3390/s24010088
Ye RNi ZXu CWang JChen SEldar Y(2023)FedFM: Anchor-Based Feature Matching for Data Heterogeneity in Federated LearningIEEE Transactions on Signal Processing10.1109/TSP.2023.331427771(4224-4239)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TSP.2023.3314277
Li ZShang XHe RLin TWu C(2023)No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00490(5296-5306)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00490

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents