Abstract
Reflection is common when we see through a glass window, which not only is a visual disturbance but also influences the performance of computer vision algorithms. Removing the reflection from a single image, however, is highly ill-posed since the color at each pixel needs to be separated into two values belonging to the clear background and the reflection, respectively. To solve this, existing methods use additional priors such as reflection layer smoothness, double reflection effect, and color consistency to distinguish the two layers. However, these low-level priors may not be consistently valid in real cases. In this paper, inspired by the fact that human beings can separate the two layers easily by recognizing the objects and understanding the scene, we propose to use the object semantic cue, which is high-level information, as the guidance to help reflection removal. Based on the data analysis, we develop a multi-task end-to-end deep learning method with a semantic guidance component, to solve reflection removal and semantic segmentation jointly. Extensive experiments on different datasets show significant performance gain when using high-level object-oriented information. We also demonstrate the application of our method to other computer vision tasks.
Supplemental Material
Available for Download
Supplementary appendix
- [1] . 2005. Removing photography artifacts using gradient projection and flash-exposure sampling. TOG 24, 3 (2005), 828–835.Google Scholar
Digital Library
- [2] . 2017. Single image reflection suppression. In CVPR.Google Scholar
- [3] . 2018. Joint learning of intrinsic images and semantic segmentation. In ECCV.Google Scholar
- [4] . 2020. End-to-end object detection with transformers. In ECCV. Springer.Google Scholar
- [5] . 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV.Google Scholar
- [6] . 2011. Torch7: A MATLAB-like environment for machine learning. In NIPS Workshop.Google Scholar
- [7] . 2009. ImageNet: A large-scale hierarchical image database. In CVPR.Google Scholar
- [8] . 2020. Multi-scale boosted dehazing network with dense feature fusion. In CVPR.Google Scholar
- [9] . [n. d.]. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.Google Scholar
- [10] . 2017. A generic deep architecture for single image reflection removal and image smoothing. In ICCV.Google Scholar
- [11] . 2014. Robust separation of reflection from multiple images. In CVPR.Google Scholar
- [12] . 2019. Learning from synthetic photorealistic raindrop for single image raindrop removal. In ICCV Workshop.Google Scholar
- [13] . 2016. Deep residual learning for image recognition. In CVPR.Google Scholar
- [14] . 2021. Panoramic image reflection removal. In CVPR.Google Scholar
- [15] . 2018. Squeeze-and-excitation networks. In CVPR.Google Scholar
- [16] . 2018. Multimodal unsupervised image-to-image translation. In ECCV.Google Scholar
- [17] . 2013. On the importance of initialization and momentum in deep learning. In ICML.Google Scholar
- [18] . 2020. Single image reflection removal with physically-based training images. In CVPR.Google Scholar
- [19] . 2020. Unsupervised single-image reflection separation using perceptual deep image priors. arXiv preprint arXiv:2009.00702 (2020).Google Scholar
- [20] . 2014. A physically-based approach to reflection separation: From physical modeling to constrained optimization. TPAMI 36, 2 (2014), 209–221.Google Scholar
Digital Library
- [21] . 2021. Robust reflection removal with reflection-free flash-only cues. In CVPR.Google Scholar
- [22] . 2020. Polarized reflection removal with perfect alignment in the wild. In CVPR.Google Scholar
- [23] . 2018. Benchmarking single-image dehazing and beyond. TIP (2018).Google Scholar
- [24] . 2020. Single image reflection removal through cascaded refinement. In CVPR.Google Scholar
- [25] . 2020. Reflection separation via multi-bounce polarization state tracing. In ECCV. Springer.Google Scholar
- [26] . 2013. Exploiting reflection change for automatic reflection removal. In ICCV.Google Scholar
- [27] . 2014. Single image layer separation using relative smoothness. In CVPR.Google Scholar
- [28] . 2020. Two-Stage single image reflection removal with reflection-aware guidance. arXiv preprint arXiv:2012.00945 (2020).Google Scholar
- [29] . 2020. Connecting image denoising and high-level vision tasks via deep learning. TIP (2020).Google Scholar
- [30] . 2019. PBRR: Physically Based Raindrop Rendering. https://liuyunfei.net/project/pbrr/.Google Scholar
- [31] . 2020. Separate in latent space: Unsupervised single image layer separation. In AAAI.Google Scholar
- [32] . 2020. Reflection backdoor: A natural backdoor attack on deep neural networks. In ECCV. Springer.Google Scholar
- [33] . 2020. Learning to see through obstructions. In CVPR.Google Scholar
- [34] . 2019. Learning to jointly generate and separate reflections. In ICCV.Google Scholar
- [35] . 2010. version R2017b. The MathWorks Inc., Natick, Massachusetts.Google Scholar
- [36] . 2017. Video reflection removal through spatio-temporal optimization. In ICCV.Google Scholar
- [37] . 2021. Learned dual-view reflection removal. In WACV.Google Scholar
- [38] . 2017. Automatic differentiation in PyTorch. In NIPS Workshop.Google Scholar
- [39] . 2019. Reflection removal using a dual-pixel sensor. In CVPR.Google Scholar
- [40] . 2019. Deep learning for seeing through window with raindrops. In ICCV.Google Scholar
- [41] . 2019. SROBB: Targeted perceptual loss for single image super-resolution. In ICCV.Google Scholar
- [42] . 2018. YOLOv3: An incremental improvement. arXiv (2018).Google Scholar
- [43] . 2018. Deep video dehazing with semantic segmentation. TIP (2018).Google Scholar
- [44] . 2017. Anti-glare: Tightly constrained optimization for eyeglass reflection removal. In CVPR.Google Scholar
- [45] . 1998. Separation of transparent layers using focus. In ICCV.Google Scholar
- [46] . 2015. Reflection removal using ghosting cues. In CVPR.Google Scholar
- [47] . 2015. Reflection removal for in-vehicle black box videos. In CVPR.Google Scholar
- [48] . 2012. Image-based rendering for scenes with reflections. TOG 31, 4 (2012), 1–10.Google Scholar
Digital Library
- [49] . 2017. Benchmarking single-image reflection removal algorithms. In ICCV.Google Scholar
- [50] . 2018. CRRN: Multi-scale guided concurrent reflection removal network. In CVPR.Google Scholar
- [51] . 2019. CoRRN: Cooperative reflection removal network. TPAMI (2019).Google Scholar
- [52] . 2020. Cascaded attention guidance network for single rainy image restoration. TIP (2020).Google Scholar
- [53] . 2019. Single image reflection removal exploiting misaligned training data and network enhancements. In CVPR.Google Scholar
- [54] . 2019. Single image reflection removal beyond linearity. In CVPR.Google Scholar
- [55] . 2021. Polarization guided specular reflection separation. TIP (2021).Google Scholar
- [56] . 2018. Separating reflection and transmission images in the wild. In ECCV.Google Scholar
- [57] . 2015. A computational approach for obstruction-free photography. TOG 34, 4 (2015), 1–11.Google Scholar
Digital Library
- [58] . 2018. Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal. In ECCV.Google Scholar
- [59] . 2016. Robust optical flow estimation of double-layer images under transparency or reflection. In CVPR.Google Scholar
- [60] . 2019. Fast single image reflection suppression via convex optimization. In CVPR.Google Scholar
- [61] . 2018. Reflection removal for large-scale 3D point clouds. In CVPR.Google Scholar
- [62] . 2018. Single image reflection separation with perceptual losses. In CVPR.Google Scholar
- [63] . 2015. Specular reflection removal using local structural similarity and chromaticity consistency. ICIP (2015).Google Scholar
- [64] . 2021. Single image reflection removal with absorption effect. In CVPR.Google Scholar
- [65] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2014. Learning deep features for scene recognition using places database. In NeurIPS.Google Scholar
- [66] . 2017. Scene parsing through ADE20K dataset. In CVPR.Google Scholar
- [67] . 2004. Image quality assessment: From error visibility to structural similarity. TIP (2004).Google Scholar
Index Terms
Semantic Guided Single Image Reflection Removal
Recommendations
A Model-Guided Unfolding Network for Single Image Reflection Removal
MMAsia '21: ACM Multimedia AsiaRemoving undesirable reflections from a single image captured through a glass surface is of broad application to various image processing and computer vision tasks, but it is an ill-posed and challenging problem. Existing traditional single image ...
Face Image Reflection Removal
AbstractFace images captured through glass are usually contaminated by reflections. The low-transmitted reflections make the reflection removal more challenging than for general scenes because important facial features would be completely occluded. In ...
Single Image Reflection Removal Based on GAN with Gradient Constraint
Pattern RecognitionAbstractWhen we take a picture through glass windows, the photographs are often degraded by undesired reflections. To separate reflection layer and background layer is an important problem for enhancing image quality. However, single-image reflection ...






Comments