skip to main content
research-article

Distill-DBDGAN: Knowledge Distillation and Adversarial Learning Framework for Defocus Blur Detection

Published:17 February 2023Publication History
Skip Abstract Section

Abstract

Defocus blur detection (DBD) aims to segment the blurred regions from a given image affected by defocus blur. It is a crucial pre-processing step for various computer vision tasks. With the increasing popularity of small mobile devices, there is a need for a computationally efficient method to detect defocus blur accurately. We propose an efficient defocus blur detection method that estimates the probability of each pixel being focused or blurred in resource-constraint devices. Despite remarkable advances made by the recent deep learning-based methods, they still suffer from several challenges such as background clutter, scale sensitivity, indistinguishable low-contrast focused regions from out-of-focus blur, and especially high computational cost and memory requirement. To address the first three challenges, we develop a novel deep network that efficiently detects blur map from the input blurred image. Specifically, we integrate multi-scale features in the deep network to resolve the scale ambiguities and simultaneously modeled the non-local structural correlations in the high-level blur features. To handle the last two issues, we eventually frame our DBD algorithm to perform knowledge distillation by transferring information from the larger teacher network to a compact student network. All the networks are adversarially trained in an end-to-end manner to enforce higher order consistencies between the output and the target distributions. Experimental results demonstrate the state-of-the-art performance of the larger teacher network, while our proposed lightweight DBD model imitates the output of the teacher network without significant loss in accuracy. The codes, pre-trained model weights, and the results will be made publicly available.

REFERENCES

  1. [1] Bae Soonmin and Durand Frédo. 2007. Defocus magnification. In Computer Graphics Forum, Vol. 26. 571579.Google ScholarGoogle Scholar
  2. [2] Carvalho Marcela, Saux Bertrand Le, Trouvé-Peloux Pauline, Almansa Andrés, and Champagnat Frédéric. 2018. Deep depth from defocus: How can defocus blur improve 3D estimation using dense neural networks? In Proceedings of the European Conference on Computer Vision (ECCV’18) Workshops.Google ScholarGoogle Scholar
  3. [3] Cheng Ming-Ming, Mitra Niloy J., Huang Xiaolei, Torr Philip H. S., and Hu Shi-Min. 2014. Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37, 3 (2014), 569582.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. [4] Cun X. and Pun C. M.. 2020. Defocus blur detection via depth distillation. In Proceedings of the European Conference Computer Vision (ECCV’20), Vol. 12358. 747763.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Deng J., Dong W., Socher R., Li L. J., Li Kai, and Fei-Fei Li. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’09). 248255.Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Golestaneh S. A. and Karam L. J.. 2017. Spatially-varying blur detection based on multiscale fused and sorted transform coefficients of gradient magnitudes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 596605.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Goodfellow I., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley D., Ozair S., Courville A., and Bengio Y.. 2014. Generative adversarial nets. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’14). 26722680.Google ScholarGoogle Scholar
  8. [8] Guo Wenliang, Xiao Xiao, Hui Yilong, Yang Wenming, and Sadovnik Amir. 2021. Heterogeneous attention nested U-shaped network for blur detection. IEEE Signal Process. Lett. 29 (2021), 140–144.Google ScholarGoogle Scholar
  9. [9] Heusel Martin, Ramsauer Hubert, Unterthiner Thomas, Nessler Bernhard, and Hochreiter Sepp. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS’17), Vol. 30.Google ScholarGoogle Scholar
  10. [10] Hu J., Shen L., and Sun G.. 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). 71327141.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Huang R., Feng W., Fan M., Wan L., and Sun J.. 2018. Multiscale blur detection by learning discriminative deep features. Neurocomputing 285 (2018), 154166.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Jin-Yu Zhang, Yan Chen, and Xian-Xiang Huang. 2009. Edge detection of images based on improved Sobel operator and genetic algorithms. In Proceedings of the International Conference on Image Analysis and Signal Processing. 3135.Google ScholarGoogle Scholar
  13. [13] Jolicoeur-Martineau Alexia. 2019. The relativistic discriminator: A key element missing from standard GAN. In Proceedings of the International Conference on Learning Representations (ICLR’19).Google ScholarGoogle Scholar
  14. [14] Kingma Diederick P. and Ba Jimmy. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR’15).Google ScholarGoogle Scholar
  15. [15] Li Boyi, Wu Felix, Weinberger Kilian Q., and Belongie Serge J.. 2019. Positional normalization. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS’19). 16201632.Google ScholarGoogle Scholar
  16. [16] Li Jinxing, Fan Dandan, Yang Lingxiao, Gu Shuhang, Lu Guangming, Xu Yong, and Zhang David. 2021. Layer-output guided complementary attention learning for image defocus blur detection. IEEE Trans. Image Process. 30 (2021), 37483763.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Lin Zinan, Sekar Vyas, and Fanti Giulia. 2020. Why spectral normalization stabilizes GANs: Analysis and improvements. Advances in Neural Information Processing Systems (NeurIPS) 34 (2020), 9652–9638.Google ScholarGoogle Scholar
  18. [18] Ma K., Fu H., Liu T., Wang Z., and Tao D.. 2018. Deep blur mapping: Exploiting high-level semantics by deep neural networks. IEEE Trans. Image Process. 27, 10 (2018), 51555166.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Mao X., Li Q., Xie H., Lau R. Y. K., Wang Z., and Smolley S. P.. 2017. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). 28132821.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Mao X., Li Q., Xie H., Lau R. Y. K., Wang Z., and Smolley S. P.. 2019. On the effectiveness of least squares generative adversarial networks. IEEE Trans. Pattern Anal. Mach. Intell. 41, 12 (2019), 29472960.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Mirza Mehdi and Osindero Simon. 2014. Conditional generative adversarial nets. arXiv:1411.1784. Retrieved from https://arxiv.org/abs/1411.1784.Google ScholarGoogle Scholar
  22. [22] Miyato Takeru, Kataoka Toshiki, Koyama Masanori, and Yoshida Yuichi. 2018. Spectral normalization for generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR’18).Google ScholarGoogle Scholar
  23. [23] Mokhtarian Farzin and Suomela Riku. 1998. Robust image corner detection through curvature scale space. IEEE Trans. Pattern Anal. Mach. Intell. 20, 12 (1998), 13761381.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Pang Y., Zhu H., Li X., and Li X.. 2016. Classifying discriminative features for blur detection. IEEE Trans. Cybern. 46, 10 (2016), 22202227.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Park J., Tai Y. W., Cho D., and Kweon I. S.. 2017. A unified approach of multi-scale deep and hand-crafted features for defocus estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 27602769.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Radford Alec, Metz Luke, and Chintala Soumith. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR’16), Bengio Yoshua and LeCun Yann (Eds.).Google ScholarGoogle Scholar
  27. [27] Ronneberger O., Fischer P., and Brox T.. 2015. U-Net: Convolutional networks for biomedical image segmentation. In Proceedings of the Annual Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI’15). 234241.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Sandler Mark, Howard Andrew, Zhu Menglong, Zhmoginov Andrey, and Chen Liang-Chieh. 2018. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Shi J., Xu L., and Jia J.. 2014. Discriminative blur detection features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). 29652972.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Shi J., Xu L., and Jia J.. 2015. Just noticeable defocus blur detection and estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 657665.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Sun Xiaoli, Zhang Xiujun, Xiao Mingqing, and Xu Chen. 2020. Blur detection via deep pyramid network with recurrent distinction enhanced modules. Neurocomputing 414 (2020), 278290.Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Tan M. and Le Q. V.. 2019. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning (ICML’19).Google ScholarGoogle Scholar
  33. [33] Tang C., Liu X., An S., and Wang P.. 2021. BR\(^2\)Net: Defocus blur detection via a bidirectional channel attention residual refining network. IEEE Trans. Multimedia 23 (2021), 624635.Google ScholarGoogle ScholarCross RefCross Ref
  34. [34] Tang C., Liu X., Zheng X., Li W., Xiong J., Wang L., Zomaya A., and Longo A.. 2022. DeFusionNET: Defocus blur detection via recurrently fusing and refining discriminative multi-scale deep features. IEEE Trans. Pattern Anal. Mach. Intell. 44, 2 (2022), 955–968.Google ScholarGoogle Scholar
  35. [35] Tang Chang, Liu Xinwang, Zhu Xinzhong, Zhu En, Sun Kun, Wang Pichao, Wang Lizhe, and Zomaya Albert. 2020. R\(^2\)MRF: Defocus blur detection via recurrently refining multi-scale residual features. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 1206312070.Google ScholarGoogle Scholar
  36. [36] Tang C., Wu J., Hou Y., Wang P., and Li W.. 2016. A spectral and spatial approach of coarse-to-fine blurred image region detection. IEEE Sign. Process. Lett. 23, 11 (2016), 16521656.Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Tang Chang, Zhu Xinzhong, Liu Xinwang, Wang Lizhe, and Zomaya Albert. 2019. DeFusionNET: Defocus blur detection via recurrently fusing and refining multi-scale deep features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19).Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Yang M., Yu K., Zhang C., Li Z., and Yang K.. 2018. DenseASPP for semantic segmentation in street scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). 36843692.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Yi X. and Eramian M.. 2016. LBP-Based segmentation of defocus blur. IEEE Trans. Image Process. 25, 4 (2016), 16261638.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. [40] Zeng K., Wang Y., Mao J., Liu J., Peng W., and Chen N.. 2019. A local metric for defocus blur detection based on CNN feature learning. IEEE Trans. Image Process. 28, 5 (2019), 21072115.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. [41] Zhai Yongping, Wang Junhua, Deng Jinsheng, Yue Guanghui, Zhang Wei, and Tang Chang. 2021. Global context guided hierarchically residual feature refinement network for defocus blur detection. Sign. Process. 183 (2021), 107996.Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] Zhang Ning and Yan Junchi. 2020. Rethinking the defocus blur detection problem and a real-time deep DBD model. In Proceedings of the European Conference Computer Vision (ECCV’20). 617632.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. [43] Zhao Wenda, Hou Xueqing, He You, and Lu Huchuan. 2021. Defocus blur detection via boosting diversity of deep ensemble networks. IEEE Trans. Image Process. 30 (2021), 54265438.Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Zhao Wenda, Shang Cai, and Lu Huchuan. 2021. Self-generated defocus blur detection via dual adversarial discriminators. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’21). 69336942.Google ScholarGoogle ScholarCross RefCross Ref
  45. [45] Zhao Wenda, Zhao Fan, Wang Dong, and Lu Huchuan. 2018. Defocus blur detection via multi-stream bottom-top-bottom fully convolutional network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Zhao W., Zhao F., Wang D., and Lu H.. 2019. Defocus blur detection via multi-stream bottom-top-bottom network. IEEE Trans. Pattern Anal. Mach. Intell. 42, 8 (2019), 18841897.Google ScholarGoogle Scholar
  47. [47] Zhao Wenda, Zheng Bowen, Lin Qiuhua, and Lu Huchuan. 2019. Enhancing diversity of defocus blur detectors via cross-ensemble network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19).Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Distill-DBDGAN: Knowledge Distillation and Adversarial Learning Framework for Defocus Blur Detection

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 2s
      April 2023
      545 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3572861
      • Editor:
      • Abdulmotaleb El Saddik
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 17 February 2023
      • Online AM: 22 August 2022
      • Accepted: 3 August 2022
      • Revised: 30 June 2022
      • Received: 27 January 2022
      Published in tomm Volume 19, Issue 2s

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
    • Article Metrics

      • Downloads (Last 12 months)215
      • Downloads (Last 6 weeks)8

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!