skip to main content
research-article

D3T-GAN: Data-Dependent Domain Transfer GANs for Image Generation with Limited Data

Authors Info & Claims
Published:15 March 2023Publication History
Skip Abstract Section

Abstract

As an important and challenging problem, image generation with limited data aims at generating realistic images through training a GAN model given few samples. A typical solution is to transfer a well-trained GAN model from a data-rich source domain to the data-deficient target domain. In this paper, we propose a novel self-supervised transfer scheme termed D3T-GAN, addressing the cross-domain GANs transfer in limited image generation. Specifically, we design two individual strategies to transfer knowledge between generators and discriminators, respectively. To transfer knowledge between generators, we conduct a data-dependent transformation, which projects target samples into the latent space of source generator and reconstructs them back. Then, we perform knowledge transfer from transformed samples to generated samples. To transfer knowledge between discriminators, we design a multi-level discriminant knowledge distillation from the source discriminator to the target discriminator on both the real and fake samples. Extensive experiments show that our method improves the quality of generated images and achieves the state-of-the-art FID scores on commonly used datasets.

REFERENCES

  1. [1] Abdal Rameen, Qin Yipeng, and Wonka Peter. 2019. Image2StyleGAN: How to embed images into the StyleGAN latent space?. In Proc. ICCV. 44324441.Google ScholarGoogle Scholar
  2. [2] Antoniou Antreas, Storkey Amos, and Edwards Harrison. 2017. Data augmentation generative adversarial networks. In Proc. ICLR.Google ScholarGoogle Scholar
  3. [3] Arjovsky Martin, Chintala Soumith, and Bottou Léon. 2017. Wasserstein generative adversarial networks. In Proc. ICML. PMLR, 214223.Google ScholarGoogle Scholar
  4. [4] Bartunov Sergey and Vetrov Dmitry. 2018. Few-shot generative modelling with generative matching networks. In Proc. AISTATS. PMLR, 670678.Google ScholarGoogle Scholar
  5. [5] Brock Andrew, Donahue Jeff, and Simonyan Karen. 2019. Large scale GAN training for high fidelity natural image synthesis. In Proc. ICLR.Google ScholarGoogle Scholar
  6. [6] Clouâtre Louis and Demers Marc. 2019. FIGR: Few-shot image generation with reptile. In arXiv preprint arXiv:1901.02199.Google ScholarGoogle Scholar
  7. [7] Dosovitskiy Alexey and Brox Thomas. 2016. Generating images with perceptual similarity metrics based on deep networks. In Proc. NeurIPS, Vol. 29. 658666.Google ScholarGoogle Scholar
  8. [8] Fan Chenyou and Huang Jianwei. 2021. Federated few-shot learning with adversarial learning. In arXiv preprint arXiv:2104.00365.Google ScholarGoogle Scholar
  9. [9] Gao Zan, Guo Leming, Guan Weili, Liu An-An, Ren Tongwei, and Chen Shengyong. 2020. A pairwise attentive adversarial spatiotemporal network for cross-domain few-shot action recognition-R2. In IEEE Trans. Image Process., Vol. 30. IEEE, 767782.Google ScholarGoogle Scholar
  10. [10] Goodfellow Ian J., Pouget-Abadie Jean, Mirza Mehdi, Xu Bing, Warde-Farley David, Ozair Sherjil, Courville Aaron, and Bengio Yoshua. 2014. Generative adversarial networks. In Proc. NeurIPS.Google ScholarGoogle Scholar
  11. [11] Gu Jinjin, Shen Yujun, and Zhou Bolei. 2020. Image processing using multi-code GAN prior. In Proc. CVPR. 30123021.Google ScholarGoogle Scholar
  12. [12] Gulrajani Ishaan, Ahmed Faruk, Arjovsky Martin, Dumoulin Vincent, and Courville Aaron. 2017. Improved training of Wasserstein GANs. In Proc. NeurIPS.Google ScholarGoogle Scholar
  13. [13] Guo Zonghui, Zhang Liqiang, Jiang Yufeng, Niu Wenjie, Gu Zhaorui, Zheng Haiyong, Wang Guoyu, and Zheng Bing. 2020. Few-shot fish image generation and classification. In Global Oceans. IEEE, 16.Google ScholarGoogle Scholar
  14. [14] Ho Trang-Thi, Virtusio John Jethro, Chen Yung-Yao, Hsu Chih-Ming, and Hua Kai-Lung. 2020. Sketch-guided deep portrait generation. In ACM Trans. Multimedia Comput. Commun. Appl., Vol. 16. ACM New York, NY, USA, 118.Google ScholarGoogle Scholar
  15. [15] Hong Yan, Niu Li, Zhang Jianfu, and Zhang Liqing. 2020. MatchingGAN: Matching-based few-shot image generation. In Proc. IEEE ICME. IEEE, 16.Google ScholarGoogle Scholar
  16. [16] Hong Yan, Niu Li, Zhang Jianfu, Zhao Weijie, Fu Chen, and Zhang Liqing. 2020. F2GAN: Fusing-and-filling GAN for few-shot image generation. In Proc. ACM MM. 25352543.Google ScholarGoogle Scholar
  17. [17] Huang Xun and Belongie Serge. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proc. ICCV. 15011510.Google ScholarGoogle Scholar
  18. [18] Jiang Shuqiang, Min Weiqing, Lyu Yongqiang, and Liu Linhu. 2020. Few-shot food recognition via multi-view representation learning. In ACM Trans. Multimedia Comput. Commun. Appl., Vol. 16. ACM New York, NY, USA, 120.Google ScholarGoogle Scholar
  19. [19] Karras Tero, Aila Timo, Laine Samuli, and Lehtinen Jaakko. 2018. Progressive growing of GANs for improved quality, stability, and variation. In Proc. ICLR.Google ScholarGoogle Scholar
  20. [20] Karras Tero, Aittala Miika, Hellsten Janne, Laine Samuli, Lehtinen Jaakko, and Aila Timo. 2020. Training generative adversarial networks with limited data. In Proc. NeurIPS.Google ScholarGoogle Scholar
  21. [21] Karras Tero, Laine Samuli, and Aila Timo. 2019. A style-based generator architecture for generative adversarial networks. In Proc. CVPR. 44014410.Google ScholarGoogle Scholar
  22. [22] Karras Tero, Laine Samuli, Aittala Miika, Hellsten Janne, Lehtinen Jaakko, and Aila Timo. 2020. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR. 81108119.Google ScholarGoogle Scholar
  23. [23] Kingma Diederik P. and Ba Jimmy. 2015. Adam: A method for stochastic optimization. In Proc. ICLR.Google ScholarGoogle Scholar
  24. [24] Li Yu, Tang Sheng, Zhang Rui, Zhang Yongdong, Li Jintao, and Yan Shuicheng. 2019. Asymmetric GAN for unpaired image-to-image translation. In IEEE Trans. Image Process., Vol. 28. IEEE, 58815896.Google ScholarGoogle Scholar
  25. [25] Li Yijun, Zhang Richard, Lu Jingwan, and Shechtman Eli. 2020. Few-shot image generation with elastic weight consolidation. In Proc. NeurIPS.Google ScholarGoogle Scholar
  26. [26] Liang Weixin, Liu Zixuan, and Liu Can. 2020. DAWSON: A domain adaptive few shot generation framework. In arXiv preprint arXiv:2001.00576.Google ScholarGoogle Scholar
  27. [27] Mangla Puneet, Kumari Nupur, Singh Mayank, Balasubramanian Vineeth N., and Krishnamurthy Balaji. 2020. Data instance prior for transfer learning in GANs. In arXiv preprint arXiv:2012.04256.Google ScholarGoogle Scholar
  28. [28] Mescheder Lars, Geiger Andreas, and Nowozin Sebastian. 2018. Which training methods for GANs do actually converge?. In Proc. ICML. PMLR, 34813490.Google ScholarGoogle Scholar
  29. [29] Miyato Takeru, Kataoka Toshiki, Koyama Masanori, and Yoshida Yuichi. 2018. Spectral normalization for generative adversarial networks. In Proc. ICLR.Google ScholarGoogle Scholar
  30. [30] Mo Sangwoo, Cho Minsu, and Shin Jinwoo. 2020. Freeze discriminator: A simple baseline for fine-tuning GANs. In Proc. CVPR Workshop.Google ScholarGoogle Scholar
  31. [31] Nichol Alex, Achiam Joshua, and Schulman John. 2018. On first-order meta-learning algorithms. In arXiv preprint arXiv:1803.02999.Google ScholarGoogle Scholar
  32. [32] Noguchi Atsuhiro and Harada Tatsuya. 2019. Image generation from small datasets via batch statistics adaptation. In Proc. ICCV. 27502758.Google ScholarGoogle Scholar
  33. [33] Nowozin Sebastian, Cseke Botond, and Tomioka Ryota. 2016. F-GAN: Training generative neural samplers using variational divergence minimization. In Proc. NeurIPS.Google ScholarGoogle Scholar
  34. [34] Phaphuangwittayakul Aniwat, Guo Yi, and Ying Fangli. 2021. Fast adaptive meta-learning for few-shot image generation. In IEEE Trans. Multimedia. IEEE.Google ScholarGoogle Scholar
  35. [35] Radford Alec, Metz Luke, and Chintala Soumith. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proc. ICLR.Google ScholarGoogle Scholar
  36. [36] Robb Esther, Chu Wen-Sheng, Kumar Abhishek, and Huang Jia-Bin. 2020. Few-shot adaptation of generative adversarial networks. In arXiv preprint arXiv:2010.11943.Google ScholarGoogle Scholar
  37. [37] Si Zhangzhang and Zhu Song-Chun. 2011. Learning hybrid image templates (HIT) by information projection. In IEEE Trans. Pattern Anal. Mach. Intell., Vol. 34. IEEE, 13541367.Google ScholarGoogle Scholar
  38. [38] Szegedy Christian, Vanhoucke Vincent, Ioffe Sergey, Shlens Jon, and Wojna Zbigniew. 2016. Rethinking the inception architecture for computer vision. In Proc. CVPR. 28182826.Google ScholarGoogle Scholar
  39. [39] Tang Hao, Liu Hong, and Sebe Nicu. 2020. Unified generative adversarial networks for controllable image-to-image translation. In IEEE Trans. Image Process., Vol. 29. IEEE, 89168929.Google ScholarGoogle Scholar
  40. [40] Tran Ngoc-Trung, Tran Viet-Hung, Nguyen Ngoc-Bao, Nguyen Trung-Kien, and Cheung Ngai-Man. 2021. On data augmentation for GAN training. In IEEE Trans. Image Process.IEEE, 18821897.Google ScholarGoogle Scholar
  41. [41] Wang Ting-Chun, Liu Ming-Yu, Tao Andrew, Liu Guilin, Kautz Jan, and Catanzaro Bryan. 2019. Few-shot video-to-video synthesis. In Proc. NeurIPS.Google ScholarGoogle Scholar
  42. [42] Wang Xueping, Wang Yunhong, and Li Weixin. 2019. U-Net conditional GANs for photo-realistic and identity-preserving facial expression synthesis. In ACM Trans. Multimedia Comput. Commun. Appl., Vol. 15. ACM New York, NY, USA, 123.Google ScholarGoogle Scholar
  43. [43] Wang Yaxing, Gonzalez-Garcia Abel, Berga David, Herranz Luis, Khan Fahad Shahbaz, and Weijer Joost van de. 2020. MineGAN: Effective knowledge transfer from GANs to target domains with few images. In Proc. CVPR. 93329341.Google ScholarGoogle Scholar
  44. [44] Wang Yaxing, Wu Chenshen, Herranz Luis, Weijer Joost van de, Gonzalez-Garcia Abel, and Raducanu Bogdan. 2018. Transferring GANs: Generating images from limited data. In Proc. ECCV. 218234.Google ScholarGoogle Scholar
  45. [45] Wang Yaqing and Yao Quanming. 2019. Few-shot learning: A survey. In arXiv preprint arXiv:1904.05046.Google ScholarGoogle Scholar
  46. [46] Yang Ceyuan, Shen Yujun, Xu Yinghao, and Zhou Bolei. 2021. Data-efficient instance generation from instance discrimination. In arXiv preprint arXiv:2106.04566.Google ScholarGoogle Scholar
  47. [47] Yu Fisher, Zhang Yinda, Song Shuran, Seff Ari, and Xiao Jianxiong. 2015. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. In arXiv preprint arXiv:1506.03365.Google ScholarGoogle Scholar
  48. [48] Zhang Han, Goodfellow Ian, Metaxas Dimitris, and Odena Augustus. 2019. Self-attention generative adversarial networks. In Proc. ICML. PMLR, 73547363.Google ScholarGoogle Scholar
  49. [49] Zhang Han, Zhang Zizhao, Odena Augustus, and Lee Honglak. 2019. Consistency regularization for generative adversarial networks. In Proc. ICLR.Google ScholarGoogle Scholar
  50. [50] Zhao Junbo, Mathieu Michael, and LeCun Yann. 2017. Energy-based generative adversarial network. In Proc. ICLR.Google ScholarGoogle Scholar
  51. [51] Zhao Miaoyun, Cong Yulai, and Carin Lawrence. 2020. On leveraging pretrained GANs for generation with limited data. In Proc. ICML. PMLR, 1134011351.Google ScholarGoogle Scholar
  52. [52] Zhao Shengyu, Liu Zhijian, Lin Ji, Zhu Jun-Yan, and Han Song. 2020. Differentiable augmentation for data-efficient GAN training. In Proc. NeurIPS.Google ScholarGoogle Scholar
  53. [53] Zhao Zhengli, Singh Sameer, Lee Honglak, Zhang Zizhao, Odena Augustus, and Zhang Han. 2021. Improved consistency regularization for GANs. In Proc. AAAI.Google ScholarGoogle Scholar
  54. [54] Zhao Zhengli, Zhang Zizhao, Chen Ting, Singh Sameer, and Zhang Han. 2020. Image augmentations for GAN training. In arXiv preprint arXiv:2006.02595.Google ScholarGoogle Scholar
  55. [55] Zhu Jiapeng, Shen Yujun, Zhao Deli, and Zhou Bolei. 2020. In-domain GAN inversion for real image editing. In Proc. ECCV. Springer, 592608.Google ScholarGoogle Scholar
  56. [56] Zhu Jun-Yan, Krähenbühl Philipp, Shechtman Eli, and Efros Alexei A.. 2016. Generative visual manipulation on the natural image manifold. In Proc. ECCV. Springer, 597613.Google ScholarGoogle Scholar
  57. [57] Zhu Linchao, Fan Hehe, Luo Yawei, Xu Mingliang, and Yang Yi. 2021. Few-shot common-object reasoning using common-centric localization network. In IEEE Trans. Image Process., Vol. 30. IEEE, 42534262.Google ScholarGoogle Scholar

Index Terms

  1. D3T-GAN: Data-Dependent Domain Transfer GANs for Image Generation with Limited Data

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 4
        July 2023
        263 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3582888
        • Editor:
        • Abdulmotaleb El Saddik
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 15 March 2023
        • Online AM: 6 February 2023
        • Accepted: 29 November 2022
        • Revised: 26 October 2022
        • Received: 25 June 2022
        Published in tomm Volume 19, Issue 4

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
      • Article Metrics

        • Downloads (Last 12 months)150
        • Downloads (Last 6 weeks)17

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!