Abstract
Generative Adversarial Networks (GANs) have the ability to generate images that are visually indistinguishable from real images. However, recent studies have revealed that generated and real images share significant differences in the frequency domain. In this article, we argue that the frequency gap is caused by the high-frequency sensitivity of the discriminator. According to our observation, during the training of most GANs, severe high-frequency differences make the discriminator focus on high-frequency components excessively, which hinders the generator from fitting the low-frequency components that are important for learning images’ content. Then, we propose two simple yet effective image pre-processing operations in the frequency domain for eliminating the side effects caused by high-frequency differences in GANs training: High-frequency Confusion (HFC) and High-frequency Filter (HFF). The proposed operations are general and can be applied to most existing GANs at a fraction of the cost. The advanced performance of the proposed operations is verified on multiple loss functions, network architectures, and datasets. Specifically, the proposed HFF achieves significant improvements of 42.5% FID on CelebA (128*128) unconditional generation based on SNGAN, 30.2% FID on CelebA unconditional generation based on SSGAN, and 69.3% FID on CelebA unconditional generation based on InfoMAXGAN. Furthermore, we also adopt HFF as the first attempt at data augmentation in the frequency domain for contrastive learning, achieving state-of-the-art performance on unconditional generation. Code is available at https://github.com/iceli1007/HFC-and-HFF.
- [1] . 2017. Wasserstein GAN. Retrieved from https://arXiv:1701.07875.Google Scholar
- [2] . 2019. The convergence rate of neural networks for learned functions of different frequencies. Retrieved from https://arXiv:1906.00425.Google Scholar
- [3] . 2018. Demystifying MMD GANs. Retrieved from https://arXiv:1801.01401.Google Scholar
- [4] . 2019. Towards understanding the spectral bias of deep learning. Retrieved from https://arXiv:1912.01198.Google Scholar
- [5] . 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning. PMLR, 1597–1607.Google Scholar
- [6] . 2019. Self-supervised GANs via auxiliary rotation loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12154–12163.Google Scholar
Cross Ref
- [7] . 2021. SSD-GAN: Measuring the realness in the spatial and spectral domains. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 1105–1112.Google Scholar
Cross Ref
- [8] . 2011. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics. 215–223.Google Scholar
- [9] . 2022. A novel multi-sample generation method for adversarial attacks. ACM Trans. Multimedia Comput. Commun. Appl. 18, 4 (2022), 1–21.Google Scholar
Digital Library
- [10] . 2020. Watch your up-convolution: CNN-based generative deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7890–7899.Google Scholar
Cross Ref
- [11] . 2019. Fourier spectrum discrepancies in deep network generated images. Retrieved from https://arXiv:1911.06465.Google Scholar
- [12] . 2020. Leveraging frequency analysis for deep fake image recognition. Retrieved from https://arXiv:2003.08685.Google Scholar
- [13] . 2021. Fourier space losses for efficient perceptual image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2360–2369.Google Scholar
Cross Ref
- [14] . 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. MIT Press, 2672–2680.Google Scholar
Digital Library
- [15] . 2014. Explaining and harnessing adversarial examples. Retrieved from https://arXiv:1412.6572.Google Scholar
- [16] . 2020. Image processing using multi-code GAN prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3012–3021.Google Scholar
Cross Ref
- [17] . 2017. Improved training of wasserstein GANs. In Advances in Neural Information Processing Systems. MIT Press, 5767–5777.Google Scholar
Digital Library
- [18] . 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9729–9738.Google Scholar
Cross Ref
- [19] . 2017. Gans trained by a two time-scale update rule converge to a local Nash equilibrium. Adv. Neural Info. Process. Syst. 30 (2017), 6626–6637.Google Scholar
- [20] . 2021. Training GANs with stronger augmentations via contrastive discriminator. Retrieved from https://arXiv:2103.09742.Google Scholar
- [21] . 2021. Deceive D: Adaptive pseudo augmentation for GAN training with limited data. In Proceedings of the 35th Conference on Neural Information Processing Systems.Google Scholar
- [22] . 2017. Progressive growing of GANs for improved quality, stability, and variation. Retrieved from https://arXiv:1710.10196.Google Scholar
- [23] . 2020. Training generative adversarial networks with limited data. Adv. Neural Info. Process. Syst. 33 (2020).Google Scholar
- [24] . 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4401–4410.Google Scholar
Cross Ref
- [25] . 2020. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8110–8119.Google Scholar
Cross Ref
- [26] . 2020. Spatial frequency bias in convolutional generative adversarial networks. Retrieved from https://arXiv:2010.01473.Google Scholar
- [27] . 2009. Learning multiple layers of features from tiny images. http://www.cs.utoronto.ca/kriz/learning-features-2009-TR.pdf.Google Scholar
- [28] . 2020. InfoMax-GAN: Improved adversarial image generation via information maximization and contrastive learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3942–3952.Google Scholar
- [29] . 2021. Learning to fool the speaker recognition. ACM Trans. Multimedia Comput. Commun. Appl. 17, 3s (2021), 1–21.Google Scholar
Digital Library
- [30] . 2021. Interpreting the latent space of GANs via measuring decoupling. IEEE Trans. Artific. Intell. 2, 1 (2021), 58–70. Google Scholar
Cross Ref
- [31] . 2022. A systematic survey of regularization and normalization in GANs. ACM Comput. Surv. (
Nov. 2022). Google ScholarDigital Library
- [32] . 2022. FakeCLR: Exploring contrastive learning for solving latent discontinuity in data-efficient GANs. In Proceedings of the European Conference on Computer Vision. Springer, 598–615.Google Scholar
Digital Library
- [33] . 2022. A comprehensive survey on data-efficient GANs in image generation. Retrieved from https://arXiv:2204.08329.Google Scholar
- [34] . 2022. A new perspective on stabilizing GANs training: Direct adversarial training. In IEEE Transactions on Emerging Topics in Computational Intelligence. 1–12.
DOI: 10.1109/TETCI.2022.3193373Google Scholar - [35] . 2017. Geometric GAN. Retrieved from https://arXiv:1705.02894.Google Scholar
- [36] . 2017. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 2794–2802.Google Scholar
Cross Ref
- [37] . 2018. Spectral normalization for generative adversarial networks. Retrieved from https://arXiv:1802.05957.Google Scholar
- [38] . 2016. f-GAN: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems. MIT Press, 271–279.Google Scholar
- [39] . 2018. Representation learning with contrastive predictive coding. Retrieved from https://arXiv:1807.03748.Google Scholar
- [40] . 2021. Lt-GAN: Self-supervised GAN with latent transformation detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3189–3198.Google Scholar
Cross Ref
- [41] . 2019. CM-GANs: Cross-modal generative adversarial networks for common representation learning. ACM Trans. Multimedia Comput. Commun. Appl. 15, 1 (2019), 1–24.Google Scholar
Digital Library
- [42] . 2019. On the spectral bias of neural networks. In Proceedings of the International Conference on Machine Learning. PMLR, 5301–5310.Google Scholar
- [43] . 2016. Improved techniques for training GANs. Adv. Neural Info. Process. Syst. 29 (2016), 2234–2242.Google Scholar
- [44] . 2019. Synthesizing facial photometries and corresponding geometries using generative adversarial networks. ACM Trans. Multimedia Comput. Commun. Appl. 15, 3s (2019), 1–24.Google Scholar
Digital Library
- [45] . 2020. Interpreting the latent space of GANs for semantic face editing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9243–9252.Google Scholar
Cross Ref
- [46] . 2020. Pepsi++: Fast and lightweight network for image inpainting. IEEE Trans. Neural Netw. Learn. Syst. 32, 1 (2020), 252–265.Google Scholar
- [47] . 2022. Response generation by jointly modeling personalized linguistic styles and emotions. ACM Trans. Multimedia Comput. Commun. Appl. 18, 2 (2022), 1–20.Google Scholar
Digital Library
- [48] . 2020. Fourier features let networks learn high frequency functions in low dimensional domains. Retrieved from https://arXiv:2006.10739.Google Scholar
- [49] . 2019. ResAttr-GAN: Unpaired deep residual attributes learning for multi-domain face image translation. IEEE Access 7 (2019), 132594–132608.Google Scholar
Cross Ref
- [50] . 2021. On data augmentation for GAN training. IEEE Trans. Image Process. 30 (2021), 1882–1897.Google Scholar
Digital Library
- [51] . 2018. Perceptual adversarial networks for image-to-image transformation. IEEE Trans. Image Process. 27, 8 (2018), 4066–4079.Google Scholar
Cross Ref
- [52] . 2020. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8684–8694.Google Scholar
Cross Ref
- [53] . 2019. U-Net conditional GANs for photo-realistic and identity-preserving facial expression synthesis. ACM Trans. Multimedia Comput. Commun. Appl. 15, 3s (2019), 1–23.Google Scholar
Digital Library
- [54] . 2022. View vertically: A hierarchical network for trajectory prediction via fourier spectrums. In Proceedings of the European Conference on Computer Vision. Springer, 682–700.Google Scholar
Digital Library
- [55] . 2021. Gradient normalization for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 6373–6382.Google Scholar
Cross Ref
- [56] . 2022. CSCNet: Contextual semantic consistency network for trajectory prediction in crowded spaces. Pattern Recogn. 126 (2022), 108552.Google Scholar
Digital Library
- [57] . 2018. Understanding training and generalization in deep learning by fourier analysis. Retrieved from https://arXiv:1808.04295.Google Scholar
- [58] . 2018. Frequency principle in deep learning with general loss functions and its potential application. Retrieved from https://arXiv:1811.10146.Google Scholar
- [59] . 2019. Frequency principle: Fourier analysis sheds light on deep neural networks. Retrieved from https://arXiv:1901.06523.Google Scholar
- [60] . 2019. Training behavior of deep neural network in frequency domain. In Proceedings of the International Conference on Neural Information Processing. Springer, 264–274.Google Scholar
Digital Library
- [61] . 2021. F-Drop&Match: GANs with a dead zone in the high-frequency domain. Retrieved from https://arXiv:2106.02343.Google Scholar
- [62] . 2021. Data-efficient instance generation from instance discrimination. Adv. Neural Info. Process. Syst. 34 (2021).Google Scholar
- [63] . 2022. WaveGAN: Frequency-aware GAN for high-fidelity few-shot image generation. In Proceedings of the European Conference on Computer Vision. Springer, 1–17.Google Scholar
Digital Library
- [64] . 2015. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. Retrieved from https://arXiv:1506.03365.Google Scholar
- [65] . 2021. Conditional LSTM-GAN for melody generation from lyrics. ACM Trans. Multimedia Comput. Commun. Appl. 17, 1 (2021), 1–20.Google Scholar
Digital Library
- [66] . 2019. Detecting and simulating artifacts in GAN fake images. In Proceedings of the IEEE International Workshop on Information Forensics and Security (WIFS’19). IEEE, 1–6.Google Scholar
Cross Ref
- [67] . 2019. Explicitizing an implicit bias of the frequency principle in two-layer neural networks. Retrieved from https://arXiv:1905.10264.Google Scholar
- [68] . 2020. Differentiable augmentation for data-efficient GAN training. Adv. Neural Info. Process. Syst. 33 (2020).Google Scholar
- [69] . 2022. JoT-GAN: A framework for jointly training GAN and person re-identification model. ACM Trans. Multimedia Comput. Commun. Appl. 18, 1s (2022), 1–18.Google Scholar
Digital Library
- [70] . 2020. Image augmentations for GAN training. Retrieved from https://arXiv:2006.02595.Google Scholar
Index Terms
Exploring the Effect of High-frequency Components in GANs Training
Recommendations
Data augmentation using fast converging CIELAB-GAN for efficient deep learning dataset generation
The commercial deep learning applications require large training datasets with many samples from different classes. The generative adversarial networks (GAN) are able to create new data samples for training these machine learning models. However, the low ...
Effect of regularity on learning in GANs
CCRIS '21: Proceedings of the 2021 2nd International Conference on Control, Robotics and Intelligent SystemGenerative Adversarial Networks (GANs) are algorithmic architectures that use two neural networks, pitting one against the opposite (thus the “adversarial”) so as to come up with new, synthetic instances of data that can pass for real data. GANs have ...
GIU-GANs: Global Information Utilization for Generative Adversarial Networks
AbstractRecently, with the rapid development of artificial intelligence, image generation based on deep learning has advanced significantly. Image generation based on Generative Adversarial Networks (GANs) is a promising study. However, ...






Comments