skip to main content
research-article

Semantic Guided Single Image Reflection Removal

Authors Info & Claims
Published:01 November 2022Publication History
Skip Abstract Section

Abstract

Reflection is common when we see through a glass window, which not only is a visual disturbance but also influences the performance of computer vision algorithms. Removing the reflection from a single image, however, is highly ill-posed since the color at each pixel needs to be separated into two values belonging to the clear background and the reflection, respectively. To solve this, existing methods use additional priors such as reflection layer smoothness, double reflection effect, and color consistency to distinguish the two layers. However, these low-level priors may not be consistently valid in real cases. In this paper, inspired by the fact that human beings can separate the two layers easily by recognizing the objects and understanding the scene, we propose to use the object semantic cue, which is high-level information, as the guidance to help reflection removal. Based on the data analysis, we develop a multi-task end-to-end deep learning method with a semantic guidance component, to solve reflection removal and semantic segmentation jointly. Extensive experiments on different datasets show significant performance gain when using high-level object-oriented information. We also demonstrate the application of our method to other computer vision tasks.

Skip Supplemental Material Section

Supplemental Material

REFERENCES

  1. [1] Agrawal Amit, Raskar Ramesh, Nayar Shree K., and Li Yuanzhen. 2005. Removing photography artifacts using gradient projection and flash-exposure sampling. TOG 24, 3 (2005), 828835.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Arvanitopoulos Nikolaos, Achanta Radhakrishna, and Susstrunk Sabine. 2017. Single image reflection suppression. In CVPR.Google ScholarGoogle Scholar
  3. [3] Baslamisli Anil S., Groenestege Thomas T., Das Partha, Le Hoang-An, Karaoglu Sezer, and Gevers Theo. 2018. Joint learning of intrinsic images and semantic segmentation. In ECCV.Google ScholarGoogle Scholar
  4. [4] Carion Nicolas, Massa Francisco, Synnaeve Gabriel, Usunier Nicolas, Kirillov Alexander, and Zagoruyko Sergey. 2020. End-to-end object detection with transformers. In ECCV. Springer.Google ScholarGoogle Scholar
  5. [5] Chen Liang-Chieh, Zhu Yukun, Papandreou George, Schroff Florian, and Adam Hartwig. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV.Google ScholarGoogle Scholar
  6. [6] Collobert R., Kavukcuoglu K., and Farabet C.. 2011. Torch7: A MATLAB-like environment for machine learning. In NIPS Workshop.Google ScholarGoogle Scholar
  7. [7] Deng J., Dong W., Socher R., Li L.-J., Li K., and Fei-Fei L.. 2009. ImageNet: A large-scale hierarchical image database. In CVPR.Google ScholarGoogle Scholar
  8. [8] Dong Hang, Pan Jinshan, Xiang Lei, Hu Zhe, Zhang Xinyi, Wang Fei, and Yang Ming-Hsuan. 2020. Multi-scale boosted dehazing network with dense feature fusion. In CVPR.Google ScholarGoogle Scholar
  9. [9] Everingham M., Gool L. Van, Williams C. K. I., Winn J., and Zisserman A.. [n. d.]. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.Google ScholarGoogle Scholar
  10. [10] Fan Qingnan, Yang Jiaolong, Hua Gang, Chen Baoquan, and Wipf David. 2017. A generic deep architecture for single image reflection removal and image smoothing. In ICCV.Google ScholarGoogle Scholar
  11. [11] Guo Xiaojie, Cao Xiaochun, and Ma Yi. 2014. Robust separation of reflection from multiple images. In CVPR.Google ScholarGoogle Scholar
  12. [12] Hao Zhixiang, You Shadi, Li Yu, and Lu Feng. 2019. Learning from synthetic photorealistic raindrop for single image raindrop removal. In ICCV Workshop.Google ScholarGoogle Scholar
  13. [13] He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian. 2016. Deep residual learning for image recognition. In CVPR.Google ScholarGoogle Scholar
  14. [14] Hong Yuchen, Zheng Qian, Zhao Lingran, Jiang Xudong, Kot Alex C., and Shi Boxin. 2021. Panoramic image reflection removal. In CVPR.Google ScholarGoogle Scholar
  15. [15] Hu Jie, Shen Li, and Sun Gang. 2018. Squeeze-and-excitation networks. In CVPR.Google ScholarGoogle Scholar
  16. [16] Huang Xun, Liu Ming-Yu, Belongie Serge, and Kautz Jan. 2018. Multimodal unsupervised image-to-image translation. In ECCV.Google ScholarGoogle Scholar
  17. [17] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the importance of initialization and momentum in deep learning. In ICML.Google ScholarGoogle Scholar
  18. [18] Kim Soomin, Huo Yuchi, and Yoon Sung-Eui. 2020. Single image reflection removal with physically-based training images. In CVPR.Google ScholarGoogle Scholar
  19. [19] Kim Suhong, RahmaniKhezri Hamed, Nourbakhsh Seyed Mohammad, and Hefeeda Mohamed. 2020. Unsupervised single-image reflection separation using perceptual deep image priors. arXiv preprint arXiv:2009.00702 (2020).Google ScholarGoogle Scholar
  20. [20] Kong N., Tai Y. W., and Shin J. S.. 2014. A physically-based approach to reflection separation: From physical modeling to constrained optimization. TPAMI 36, 2 (2014), 209221.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Lei Chenyang and Chen Qifeng. 2021. Robust reflection removal with reflection-free flash-only cues. In CVPR.Google ScholarGoogle Scholar
  22. [22] Lei Chenyang, Huang Xuhua, Zhang Mengdi, Yan Qiong, Sun Wenxiu, and Chen Qifeng. 2020. Polarized reflection removal with perfect alignment in the wild. In CVPR.Google ScholarGoogle Scholar
  23. [23] Li Boyi, Ren Wenqi, Fu Dengpan, Tao Dacheng, Feng Dan, Zeng Wenjun, and Wang Zhangyang. 2018. Benchmarking single-image dehazing and beyond. TIP (2018).Google ScholarGoogle Scholar
  24. [24] Li Chao, Yang Yixiao, He Kun, Lin Stephen, and Hopcroft John E.. 2020. Single image reflection removal through cascaded refinement. In CVPR.Google ScholarGoogle Scholar
  25. [25] Li Rui, Qiu Simeng, Zang Guangming, and Heidrich Wolfgang. 2020. Reflection separation via multi-bounce polarization state tracing. In ECCV. Springer.Google ScholarGoogle Scholar
  26. [26] Li Yu and Brown Michael S.. 2013. Exploiting reflection change for automatic reflection removal. In ICCV.Google ScholarGoogle Scholar
  27. [27] Li Yu and Brown M. S.. 2014. Single image layer separation using relative smoothness. In CVPR.Google ScholarGoogle Scholar
  28. [28] Li Yu, Liu Ming, Yi Yaling, Li Qince, Ren Dongwei, and Zuo Wangmeng. 2020. Two-Stage single image reflection removal with reflection-aware guidance. arXiv preprint arXiv:2012.00945 (2020).Google ScholarGoogle Scholar
  29. [29] Liu Ding, Wen Bihan, Jiao Jianbo, Liu Xianming, Wang Zhangyang, and Huang Thomas S.. 2020. Connecting image denoising and high-level vision tasks via deep learning. TIP (2020).Google ScholarGoogle Scholar
  30. [30] Liu Yunfei, Hao Zhixiang, You Shadi, Li Yu, and Lu Feng. 2019. PBRR: Physically Based Raindrop Rendering. https://liuyunfei.net/project/pbrr/.Google ScholarGoogle Scholar
  31. [31] Liu Yunfei and Lu Feng. 2020. Separate in latent space: Unsupervised single image layer separation. In AAAI.Google ScholarGoogle Scholar
  32. [32] Liu Yunfei, Ma Xingjun, Bailey James, and Lu Feng. 2020. Reflection backdoor: A natural backdoor attack on deep neural networks. In ECCV. Springer.Google ScholarGoogle Scholar
  33. [33] Liu Yu-Lun, Lai Wei-Sheng, Yang Ming-Hsuan, Chuang Yung-Yu, and Huang Jia-Bin. 2020. Learning to see through obstructions. In CVPR.Google ScholarGoogle Scholar
  34. [34] Ma Daiqian, Wan Renjie, Shi Boxin, Kot Alex C., and Duan Ling-Yu. 2019. Learning to jointly generate and separate reflections. In ICCV.Google ScholarGoogle Scholar
  35. [35] MATLAB. 2010. version R2017b. The MathWorks Inc., Natick, Massachusetts.Google ScholarGoogle Scholar
  36. [36] Nandoriya Ajay, Elgharib Mohamed, Kim Changil, Hefeeda Mohamed, and Matusik Wojciech. 2017. Video reflection removal through spatio-temporal optimization. In ICCV.Google ScholarGoogle Scholar
  37. [37] Niklaus Simon, Zhang Xuaner Cecilia, Barron Jonathan T., Wadhwa Neal, Garg Rahul, Liu Feng, and Xue Tianfan. 2021. Learned dual-view reflection removal. In WACV.Google ScholarGoogle Scholar
  38. [38] Paszke Adam, Gross Sam, Chintala Soumith, Chanan Gregory, Yang Edward, DeVito Zachary, Lin Zeming, Desmaison Alban, Antiga Luca, and Lerer Adam. 2017. Automatic differentiation in PyTorch. In NIPS Workshop.Google ScholarGoogle Scholar
  39. [39] Punnappurath Abhijith and Brown Michael S.. 2019. Reflection removal using a dual-pixel sensor. In CVPR.Google ScholarGoogle Scholar
  40. [40] Quan Yuhui, Deng Shijie, Chen Yixin, and Ji Hui. 2019. Deep learning for seeing through window with raindrops. In ICCV.Google ScholarGoogle Scholar
  41. [41] Rad Mohammad Saeed, Bozorgtabar Behzad, Marti Urs-Viktor, Basler Max, Ekenel Hazim Kemal, and Thiran Jean-Philippe. 2019. SROBB: Targeted perceptual loss for single image super-resolution. In ICCV.Google ScholarGoogle Scholar
  42. [42] Redmon Joseph and Farhadi Ali. 2018. YOLOv3: An incremental improvement. arXiv (2018).Google ScholarGoogle Scholar
  43. [43] Ren Wenqi, Zhang Jingang, Xu Xiangyu, Ma Lin, Cao Xiaochun, Meng Gaofeng, and Liu Wei. 2018. Deep video dehazing with semantic segmentation. TIP (2018).Google ScholarGoogle Scholar
  44. [44] Sandhan Tushar and Jin Young Choi. 2017. Anti-glare: Tightly constrained optimization for eyeglass reflection removal. In CVPR.Google ScholarGoogle Scholar
  45. [45] Schechner Y. Y., Kiryati N., and Basri R.. 1998. Separation of transparent layers using focus. In ICCV.Google ScholarGoogle Scholar
  46. [46] Shih YiChang, Krishnan Dilip, Durand Fredo, and Freeman William T.. 2015. Reflection removal using ghosting cues. In CVPR.Google ScholarGoogle Scholar
  47. [47] Simon Christian and Park In Kyu. 2015. Reflection removal for in-vehicle black box videos. In CVPR.Google ScholarGoogle Scholar
  48. [48] Sinha Sudipta N., Kopf Johannes, Goesele Michael, Scharstein Daniel, and Szeliski Richard. 2012. Image-based rendering for scenes with reflections. TOG 31, 4 (2012), 110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. [49] Wan Renjie, Shi Boxin, Duan Ling Yu, Tan Ah Hwee, and Kot Alex C.. 2017. Benchmarking single-image reflection removal algorithms. In ICCV.Google ScholarGoogle Scholar
  50. [50] Wan Renjie, Shi Boxin, Duan Ling-Yu, Tan Ah-Hwee, and Kot Alex C.. 2018. CRRN: Multi-scale guided concurrent reflection removal network. In CVPR.Google ScholarGoogle Scholar
  51. [51] Wan Renjie, Shi Boxin, Li Haoliang, Duan Ling-Yu, Tan Ah-Hwee, and Kot Alex C.. 2019. CoRRN: Cooperative reflection removal network. TPAMI (2019).Google ScholarGoogle Scholar
  52. [52] Wang Guoqing, Sun Changming, and Sowmya Arcot. 2020. Cascaded attention guidance network for single rainy image restoration. TIP (2020).Google ScholarGoogle Scholar
  53. [53] Wei Kaixuan, Yang Jiaolong, Fu Ying, Wipf David, and Huang Hua. 2019. Single image reflection removal exploiting misaligned training data and network enhancements. In CVPR.Google ScholarGoogle Scholar
  54. [54] Wen Qiang, Tan Yinjie, Qin Jing, Liu Wenxi, Han Guoqiang, and He Shengfeng. 2019. Single image reflection removal beyond linearity. In CVPR.Google ScholarGoogle Scholar
  55. [55] Wen Sijia, Zheng Yingqiang, and Lu Feng. 2021. Polarization guided specular reflection separation. TIP (2021).Google ScholarGoogle Scholar
  56. [56] Wieschollek Patrick, Gallo Orazio, Gu Jinwei, and Kautz Jan. 2018. Separating reflection and transmission images in the wild. In ECCV.Google ScholarGoogle Scholar
  57. [57] Xue Tianfan, Rubinstein Michael, Liu Ce, and Freeman William T.. 2015. A computational approach for obstruction-free photography. TOG 34, 4 (2015), 111.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. [58] Yang Jie, Gong Dong, Liu Lingqiao, and Shi Qinfeng. 2018. Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal. In ECCV.Google ScholarGoogle Scholar
  59. [59] Yang Jiaolong, Li Hongdong, Dai Yuchao, and Tan Robby T.. 2016. Robust optical flow estimation of double-layer images under transparency or reflection. In CVPR.Google ScholarGoogle Scholar
  60. [60] Yang Yang, Ma Wenye, Zheng Yin, Cai Jian-Feng, and Xu Weiyu. 2019. Fast single image reflection suppression via convex optimization. In CVPR.Google ScholarGoogle Scholar
  61. [61] Yun Jae-Seong and Sim Jae-Young. 2018. Reflection removal for large-scale 3D point clouds. In CVPR.Google ScholarGoogle Scholar
  62. [62] Zhang Xuaner, Ng Ren, and Chen Qifeng. 2018. Single image reflection separation with perceptual losses. In CVPR.Google ScholarGoogle Scholar
  63. [63] Zhao Yongqiang, Peng Qunnie, Xue Jize, and Kong Seong G.. 2015. Specular reflection removal using local structural similarity and chromaticity consistency. ICIP (2015).Google ScholarGoogle Scholar
  64. [64] Zheng Qian, Shi Boxin, Chen Jinnan, Jiang Xudong, Duan Ling-Yu, and Kot Alex C.. 2021. Single image reflection removal with absorption effect. In CVPR.Google ScholarGoogle Scholar
  65. [65] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2014. Learning deep features for scene recognition using places database. In NeurIPS.Google ScholarGoogle Scholar
  66. [66] Zhou Bolei, Zhao Hang, Puig Xavier, Fidler Sanja, Barriuso Adela, and Torralba Antonio. 2017. Scene parsing through ADE20K dataset. In CVPR.Google ScholarGoogle Scholar
  67. [67] Zhou Wang, Conrad Bovik Alan, Rahim Sheikh Hamid, and Simoncelli Eero P.. 2004. Image quality assessment: From error visibility to structural similarity. TIP (2004).Google ScholarGoogle Scholar

Index Terms

  1. Semantic Guided Single Image Reflection Removal

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 3s
      October 2022
      381 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3567476
      • Editor:
      • Abdulmotaleb El Saddik
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 1 November 2022
      • Online AM: 18 February 2022
      • Accepted: 7 January 2022
      • Revised: 1 January 2022
      • Received: 1 July 2021
      Published in tomm Volume 18, Issue 3s

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!