research-article

Free access

DM-PFL: Hitchhiking Generic Federated Learning for Efficient Shift-Robust Personalization

Authors:

Yongxin TongAuthors Info & Claims

KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Pages 3396 - 3408

https://doi.org/10.1145/3580305.3599311

Published: 04 August 2023 Publication History

Abstract

Personalized federated learning collaboratively trains client-specific models, which holds potential for various mobile and IoT applications with heterogeneous data. However, existing solutions are vulnerable to distribution shifts between training and test data, and involve high training workloads on local devices. These two shortcomings hinder the practical usage of personalized federated learning on real-world mobile applications. To overcome these drawbacks, we explore efficient shift-robust personalization for federated learning. The principle is to hitchhike the global model to improve the shift-robustness of personalized models with minimal extra training overhead. To this end, we present DM-PFL, a novel framework that utilizes a dual masking mechanism to train both global and personalized models with weight-level parameter sharing and end-to-end sparse training. Evaluations on various datasets show that our methods not only improve the test accuracy in presence of test-time distribution shifts but also save the communication and computation costs compared to state-of-the-art personalized federated learning schemes.

Supplementary Material

MP4 File (rtfp0409-2min-promo.mp4)

Promotional Video for "DM-PFL: Hitchhiking Generic Federated Learning for Efficient Shift-Robust Personalization"

Download
56.08 MB

MP4 File (rtfp0409-20min-video.mp4)

Presentation video for "DM-PFL: Hitchhiking Generic Federated Learning for Efficient Shift-Robust Personalization"

Download
165.12 MB

References

[1]

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. 2020. Federated learning based on dynamic regularization. In Proceedings of the International Conference on Learning Representations.

[2]

Durmus Alp Emre Acar, Yue Zhao, Ruizhao Zhu, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, and Venkatesh Saligrama. 2021. Debiasing model updates for improving personalized federated training. In Proceedings of the International Conference on Machine Learning. PMLR, Virtual Event, 21--31.

[3]

Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz, et al. 2013. A public domain dataset for human activity recognition using smartphones. In Esann, Vol. 3. 3.

[4]

Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. 2019. Federated learning with personalization layers. CoRR, Vol. abs/1912.00818 (2019).

[5]

Sameer Bibikar, Haris Vikalo, Zhangyang Wang, and Xiaohan Chen. 2022. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, Vancouver, British Columbia, Canada (Virtual Conference), 6080--6088.

[6]

Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konevc nỳ, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. 2018. Leaf: a benchmark for federated settings. CoRR, Vol. abs/1812.01097 (2018).

[7]

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. 2021. Exploiting shared representations for personalized federated learning. In Proceedings of the International Conference on Machine Learning. PMLR, Virtual Event, 2089--2099.

[8]

Rong Dai, Li Shen, Fengxiang He, Xinmei Tian, and Dacheng Tao. 2022. Dispfl: Towards communication-efficient personalized federated learning via decentralized sparse training. In Proceedings of the International Conference on Machine Learning. PMLR, Baltimore, Maryland, USA, 4587--4604.

[9]

Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. 2020. Adaptive personalized federated learning. CoRR, Vol. abs/2003.13461 (2020).

[10]

Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. 2020. Rigging the lottery: Making all tickets winners. In Proceedings of the International Conference on Machine Learning. PMLR, Virtual Event, 2943--2952.

[11]

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. 2020. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 3557--3568.

[12]

Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. 2020. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 19586--19597.

[13]

Yves Grandvalet and Yoshua Bengio. 2004. Semi-supervised Learning by entropy minimization. In Advances in Neural Information Processing Systems. Vancouver, British Columbia, Canada, 529--536.

[14]

Dan Hendrycks and Thomas G. Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the International Conference on Learning Representations. New Orleans, LA, USA.

[15]

Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the International Conference on Learning Representations. Toulon, France.

[16]

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2019. Measuring the effects of non-identical data distribution for federated visual classification. CoRR, Vol. abs/1909.06335 (2019).

[17]

Yihan Jiang, Jakub Konevc ný, Keith Rush, and Sreeram Kannan. 2019. Improving federated learning personalization via model agnostic meta learning. CoRR, Vol. abs/1909.12488 (2019).

[18]

Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, and Leandros Tassiulas. 2022. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems (2022).

[19]

Peter Kairouz, H. Brendan McMahan, Brendan Avent, Auré lien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A. Bonawitz, Zachary Charles, et al. 2021. Advances and open problems in federated learning. Found. Trends Mach. Learn., Vol. 14, 1--2 (2021), 1--210.

Digital Library

[20]

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. 2020. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the International Conference on Machine Learning. PMLR, Virtual Event, 5132--5143.

[21]

Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. 2016. Federated learning: strategies for improving communication efficiency. CoRR, Vol. abs/1610.05492 (2016).

[22]

Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).

[23]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE, Vol. 86, 11 (1998), 2278--2324.

[24]

David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, and Joseph Dureau. 2019. Federated learning for keyword spotting. In IEEE international conference on acoustics, speech and signal processing. IEEE, 6341--6345.

[25]

Ang Li, Jingwei Sun, Pengcheng Li, Yu Pu, Hai Li, and Yiran Chen. 2021c. Hermes: an efficient federated learning framework for heterogeneous mobile clients. In Proceedings of the Annual International Conference on Mobile Computing and Networking. ACM, New Orleans, Louisiana, USA, 420--437.

Digital Library

[26]

Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. 2021d. Fedmask: joint computation and communication-efficient personalized federated learning via heterogeneous masking. In Proceedings of the ACM Conference on Embedded Networked Sensor Systems. ACM, Coimbra, Portugal, 42--55.

Digital Library

[27]

Chenglin Li, Di Niu, Bei Jiang, Xiao Zuo, and Jianming Yang. 2021b. Meta-har: Federated representation learning for human activity recognition. In Proceedings of the Web Conference 2021. 912--922.

Digital Library

[28]

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. 2021a. Ditto: Fair and robust federated learning through personalization. In Proceedings of the International Conference on Machine Learning. PMLR, Virtual Event, 6357--6368.

[29]

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org.

[30]

Paul Pu Liang, Terrance Liu, Ziyin Liu, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020. Think locally, act globally: federated learning with local and global Representations. CoRR, Vol. abs/2001.01523 (2020).

[31]

Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. 2020. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 2351--2363.

[32]

Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. 2021. Do we actually need dense over-parameterization? In-time over-parameterization in sparse training. In Proceedings of the International Conference on Machine Learning. PMLR, Virtual Event, 6989--7000.

[33]

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 1273--1282.

[34]

Hesham Mostafa and Xin Wang. 2019. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In Proceedings of the International Conference on Machine Learning. PMLR, Long Beach, California, USA, 4646--4655.

[35]

Xinchi Qiu, Javier Fernández-Marqués, Pedro Porto Buarque de Gusmão, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. 2022. Zerofl: efficient on-device training for federated learning with local sparsity. In Proceedings of the International Conference on Learning Representations.

[36]

Md Aamir Raihan and Tor Aamodt. 2020. Sparse weight activation training. Advances in Neural Information Processing Systems, 15625--15638.

[37]

Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. 2020. FetchSGD: communication-efficient federated learning with sketching. In Proceedings of the International Conference on Machine Learning. PMLR, Virtual Event, 8253--8265.

[38]

Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. 2021. Clustered federated learning: model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, Vol. 32, 8 (2021), 3710--3722.

[39]

Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. 2017. Federated multi-task learning. In Advances in Neural Information Processing Systems. Long Beach, CA, USA, 4424--4434.

[40]

Benyuan Sun, Hongxing Huo, Yi Yang, and Bo Bai. 2021. Partialfed: Cross-domain personalized federated learning via partial initialization. Advances in Neural Information Processing Systems, 23309--23320.

[41]

Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. 2020. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the International Conference on Machine Learning. PMLR, Virtual Event, 9229--9248.

[42]

Canh T Dinh, Nguyen Tran, and Josh Nguyen. 2020. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 21394--21405.

[43]

Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. 2022. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems (2022).

[44]

Florian Tramèr and Dan Boneh. 2019. Adversarial training and robustness for multiple perturbations. Advances in Neural Information Processing Systems, 5858--5868.

[45]

Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos, and Yasaman Khazaeni. 2020. Federated learning with matched averaging. In Proceedings of the International Conference on Learning Representations.

[46]

Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Francc oise Beaufays, and Daniel Ramage. 2019. Federated evaluation of on-device personalization. CoRR, Vol. abs/1910.10252 (2019).

[47]

Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, and Graham Neubig. 2021. Efficient test time adapter ensembling for low-resource language varieties. In Findings of the Association for Computational Linguistics: EMNLP. Association for Computational Linguistics, Virtual Event, 730--737.

[48]

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated machine learning: concept and applications. ACM Transactions on Intelligent Systems and Technology, Vol. 10, 2 (2019), 1--19.

Digital Library

[49]

Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and Xiang Chen. 2021. Fed2: Feature-aligned federated learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2066--2074.

Digital Library

[50]

Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. 2020. Salvaging federated learning by local adaptation. CoRR, Vol. abs/2002.04758 (2020).

[51]

Honglin Yuan, Warren Richard Morningstar, Lin Ning, and Karan Singhal. 2022. What do we mean by generalization in federated learning?. In Proceedings of the International Conference on Learning Representations.

[52]

Marvin Zhang, Sergey Levine, and Chelsea Finn. 2021a. MEMO: Test time robustness via adaptation and augmentation. CoRR, Vol. abs/2110.09506 (2021).

[53]

Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. 2021b. Personalized federated learning with first order model optimization. In Proceedings of the International Conference on Learning Representations.

Cited By

Li XLiu SZhou ZGuo BXu YYu Z(2024)EchoPFLProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435608:1(1-22)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643560
Liu BMa YZhou ZShi YLi STong YBaeza-Yates RBonchi F(2024)CASA: Clustered Federated Learning with Asynchronous ClientsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671979(1851-1862)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671979

Index Terms

DM-PFL: Hitchhiking Generic Federated Learning for Efficient Shift-Robust Personalization
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms

Recommendations

PFL-MoE: Personalized Federated Learning Based on Mixture of Experts
Web and Big Data
Abstract
Federated learning (FL) is an emerging distributed machine learning paradigm that avoids data sharing among training nodes so as to protect data privacy. Under the coordination of the FL server, each client conducts model training using its own ...
PerFedRec++: Enhancing Personalized Federated Recommendation with Self-Supervised Pre-Training
Federated recommendation systems employ federated learning techniques to safeguard user privacy by transmitting model parameters instead of raw user data between user devices and the central server. Nevertheless, the current federated recommender system ...
Towards federated unsupervised representation learning
EdgeSys '20: Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking

Making deep learning models efficient at inferring nowadays requires training with an extensive number of labeled data that are gathered in a centralized system. However, gathering labeled data is an expensive and time-consuming process, centralized ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 2023

5996 pages

ISBN:9798400701030

DOI:10.1145/3580305

General Chairs:
Ambuj Singh
UC Santa Barbara, USA
,
Yizhou Sun
UC Los Angeles, USA
,
Program Chairs:
Leman Akoglu
Carnegie Mellon University, USA
,
Dimitrios Gunopulos
University of Athens, Greece
,
Xifeng Yan
UC Santa Barbara, USA
,
Ravi Kumar
Google, USA
,
Fatma Ozcan
Google, USA
,
Jieping Ye
Alibaba DAMO Academy

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 August 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation of China (NSFC)
WeBank Scholars Program
the Funding
CityU APRC grant
Beihang University Basic Research Funding

Conference

KDD '23

Sponsor:

KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 6 - 10, 2023

CA, Long Beach, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
475
Total Downloads

Downloads (Last 12 months)420
Downloads (Last 6 weeks)21

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li XLiu SZhou ZGuo BXu YYu Z(2024)EchoPFLProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435608:1(1-22)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643560
Liu BMa YZhou ZShi YLi STong YBaeza-Yates RBonchi F(2024)CASA: Clustered Federated Learning with Asynchronous ClientsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671979(1851-1862)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671979

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents