skip to main content
research-article

DANCE: Distributed Generative Adversarial Networks with Communication Compression

Authors Info & Claims
Published:22 October 2021Publication History
Skip Abstract Section

Abstract

Generative adversarial networks (GANs) have shown great success in deep representations learning, data generation, and security enhancement. With the development of the Internet of Things, 5th generation wireless systems (5G), and other technologies, the large volume of data collected at the edge of networks provides a new way to improve the capabilities of GANs. Due to privacy, bandwidth, and legal constraints, it is not appropriate to upload all the data to the cloud or servers for processing. Therefore, this article focuses on deploying and training GANs at the edge rather than converging edge data to the central node. To address this problem, we designed a novel distributed learning architecture for GANs, called DANCE. DANCE can adaptively perform communication compression based on the available bandwidth, while supporting both data and model parallelism training of GANs. In addition, inspired by the gossip mechanism and Stackelberg game, a compatible algorithm, AC-GAN is proposed. The theoretical analysis guarantees the convergence of the model and the existence of approximate equilibrium in AC-GAN. Both simulation and prototype system experiments show that AC-GAN can achieve better training effectiveness with less communication overhead than the SOTA algorithms, i.e., FL-GAN and MD-GAN.

REFERENCES

  1. [1] Akhtar Z., Nam Y. S., Govindan R., Rao S., Chen J., Katz-Bassett E., Ribeiro B., Zhan J., and Zhang H.. 2018. Oboe: Auto-tuning video ABR algorithms to network conditions. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication. 4458. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Alistarh D., Grubic D., Li J., Tomioka R., and Vojnovic M.. 2017. QSGD: Communication-Efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems, Vol. 30, 17091720. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Boyd S., Ghosh A., Prabhakar B., and Shah D.. 2006. Randomized gossip algorithms. IEEE Trans. Inf. Theory 52, 6 (2006), 25082530.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. [4] Cao X., Tang G., Guo D., Li Y., and Zhang W.. 2020. Edge federation: Towards an integrated service provisioning model. IEEE/ACM Transactions on Networking 28, 3 (2020), 1116–1129.Google ScholarGoogle Scholar
  5. [5] Dai W., Doyle J., Liang X., Zhang H., Dong N., Li Y., and Xing E. P.. 2017. Scan: Structure correcting adversarial network for chest x-rays organ segmentation. arXiv:1703.08770. Retrieved from https://arxiv.org/abs/1703.08770.Google ScholarGoogle Scholar
  6. [6] Dereziński M., Warmuth M. K., and Hsu D.. 2019. Unbiased estimators for random design regression. arXiv:1907.03411. Retrieved from https://arxiv.org/abs/1907.03411.Google ScholarGoogle Scholar
  7. [7] Durugkar I., Gemp I., and Mahadevan S.. 2017. Generative multi-adversarial networks. In Proceedings of the International Conference on Learning Representations.Google ScholarGoogle Scholar
  8. [8] Ferdowsi A. and Saad W.. 2019. Generative adversarial networks for distributed intrusion detection in the internet of things. In Proceedings of the IEEE Global Communications Conference (GLOBECOM’19). IEEE, 16.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Ghosh A., Kulharia V., Namboodiri V. P., Torr P. H. S., and Dokania P. K.. 2018. Multi-agent diverse generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 85138521.Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Goodfellow I. J., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley D., Ozair S., Courville A., and Bengio Y.. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems, Vol. 3, 26722680. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Hardy C., Merrer E. L., and Sericola B.. 2018. Gossiping GANs position paper. In Proceedings of the 2nd Workshop on Distributed Infrastructures for Deep Learning, Part of Middleware, 2528. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Hardy C., Merrer E. L., and Sericola B.. 2019. MD-GAN: Multi-discriminator generative adversarial networks for distributed datasets. In Proceedings of the IEEE 33rd International Parallel and Distributed Processing Symposium. 866877.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Heusel M., Ramsauer H., Unterthiner T., Nessler B., and Hochreiter S.. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems. 66266637. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Hoang Q., Nguyen T. D., Le T., and Phung D.. 2017. Multi-generator generative adversarial nets. arXiv:1708.02556. Retrieved from https://arxiv.org/abs/1708.02556.Google ScholarGoogle Scholar
  15. [15] Karras T., Aila T., Laine S., and Lehtinen J.. 2018. Progressive growing of gans for improved quality, stability, and variation. In Proceedings of the International Conference on Learning Representations, 73547363.Google ScholarGoogle Scholar
  16. [16] Kingma D. P. and Ba J.. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980. Retrieved from https://arxiv.org/abs/1412.6980.Google ScholarGoogle Scholar
  17. [17] Koloskova A., Stich S. U., and Jaggi M.. 2019. Decentralized stochastic optimization and gossip algorithms with compressed communication. In Proceedings of the International Conference on Machine Learning. 34783487.Google ScholarGoogle Scholar
  18. [18] Konečný J., McMahan H. B., Yu F. X., Richtárik P., Suresh A. T., and Bacon D.. 2017. Federated learning: Strategies for improving communication efficiency. arXiv:1610.05492. Retrieved from https://arxiv.org/abs/1610.05492.Google ScholarGoogle Scholar
  19. [19] Krizhevsky A., Nair V., and Hinton G.. 2014. The cifar-10 Dataset. Retrieved from http://www.cs.toronto.edu/kriz/cifar.html.Google ScholarGoogle Scholar
  20. [20] LeCun Y., Bottou L., Bengio Y., and Haffner P.. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 22782324.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Li J., Cao X., Guo D., Xie J., and Chen H.. 2020. Task Scheduling with UAV-assisted Vehicular Cloud for Road Detection in Highway Scenario. IEEE Internet of Things Journal 7, 8 (2020), 7702–7713.Google ScholarGoogle Scholar
  22. [22] Li M., Andersen D. G., Park J. W., Smola A. J., Ahmed A., Josifovski V., Long J., Shekita E. J., and Su B.. 2014. Scaling distributed machine learning with the parameter server. In Proceedings of the 11th Symposium on Operating Systems Design and Implementation. 583598. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Lin Y., Han S., Mao H., Wang Y., and Dally B.. 2018. Deep gradient compression: Reducing the communication bandwidth for distributed training. In Proceedings of the International Conference on Learning Representations.Google ScholarGoogle Scholar
  24. [24] Mao X., Li Q., Xie H., Lau R. Y. K., Wang Z., and Smolley S. P.. 2017. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 27942802.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Mao X., Li Q., Xie H., Lau R. Y. K., Wang Z., and Smolley S. P.. 2018. On the effectiveness of least squares generative adversarial networks. IEEE Trans. Pattern Anal. Mach. Intell. 41, 12 (2018), 29472960.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] McMahan H. B., Moore E., Ramage D., and Arcas B. A.. 2016. Federated learning of deep networks using model averaging. arXiv:1602.05629. Retrieved from https://arxiv.org/abs/1602.05629.Google ScholarGoogle Scholar
  27. [27] Melis L., Song C., Cristofaro E. D., and Shmatikov V.. 2019. Exploiting unintended feature leakage in collaborative learning. In Proceedings of the IEEE Symposium on Security and Privacy (SP’19). IEEE, 691706.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Nguyen T., Le T., Vu H., and Phung D.. 2017. Dual discriminator generative adversarial nets. In Advances in Neural Information Processing Systems. 26702680. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Odena A., Olah C., and Shlens J.. 2017. Conditional image synthesis with auxiliary classifier gans. In Proceedings of the 34th International Conference on Machine Learning, Vol. 70. 26422651. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Raei H. and Yazdani N.. 2017. Performability analysis of cloudlet in mobile cloud computing. Inf. Sci.388 (2017), 99117. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Robbins H. and Monro S.. 1951. A stochastic approximation method. The Annals of Mathematical Statistics, 400407.Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Salimans T., Goodfellow I. J., Zaremba W., Cheung V., Radford A., and Chen X.. 2016. Improved techniques for training gans. In Advances in Neural Information Processing Systems. 22342242. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Seide F., Fu H., Droppo J., Li G., and Yu D.. 2014. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In Proceedings of the Annual Conference of the International Speech Communication Association (2014), 10581062.Google ScholarGoogle ScholarCross RefCross Ref
  34. [34] Shi H., Dong J., Wang W., Qian Y., and Zhang X.. 2017. SSGAN: Secure steganography based on generative adversarial networks. In Proceedings of the Pacific Rim Conference on Multimedia. Springer, 534544.Google ScholarGoogle Scholar
  35. [35] Speedtest.net. 2019. Speedtest Report of the U.S. Retrieved February 14, 2020 fromhttps://www.speedtest.net/reports/united-states/.Google ScholarGoogle Scholar
  36. [36] Stich S. U., Cordonnier J. B., and Jaggi M.. 2018. Sparsified SGD with memory. In Advances in Neural Information Processing Systems, 44474458. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Wang C., Xu C., Wang C., and Tao D.. 2018. Perceptual adversarial networks for image-to-image transformation. IEEE Trans. Image Process. 27, 8 (2018), 40664079.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Xiao H., Rasul K., and Vollgraf R.. 2017. Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747. Retrieved from https://arxiv.org/abs/1708.07747.Google ScholarGoogle Scholar
  39. [39] Yonetani R., Takahashi T., Hashimoto A., and Ushiku Y.. 2019. Decentralized learning of generative adversarial networks from multi-client non-iid data. arXiv:1905.09684. Retrieved from https://arxiv.org/abs/1905.09684.Google ScholarGoogle Scholar
  40. [40] Zhang H., Goodfellow I. J., Metaxas D., and Odena A.. 2019. Self-attention generative adversarial networks. In Proceedings of the International Conference on Machine Learning, 73547363.Google ScholarGoogle Scholar
  41. [41] Zhang H., Xu S., Jiao J., Xie P., Salakhutdinov R., and Xing E. P.. 2018. Stackelberg GAN: Towards provable minimax equilibrium via multi-generator architectures. arXiv:1811.08010. Retrieved from https://arxiv.org/abs/1811.08010.Google ScholarGoogle Scholar

Index Terms

  1. DANCE: Distributed Generative Adversarial Networks with Communication Compression

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Internet Technology
          ACM Transactions on Internet Technology  Volume 22, Issue 2
          May 2022
          582 pages
          ISSN:1533-5399
          EISSN:1557-6051
          DOI:10.1145/3490674
          • Editor:
          • Ling Liu
          Issue’s Table of Contents

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 22 October 2021
          • Accepted: 1 March 2021
          • Revised: 1 October 2020
          • Received: 1 May 2020
          Published in toit Volume 22, Issue 2

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Refereed
        • Article Metrics

          • Downloads (Last 12 months)128
          • Downloads (Last 6 weeks)14

          Other Metrics

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        View Full Text

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!