Abstract
Residual image and illumination estimation have been proven to be helpful for image enhancement. In this article, we propose a general framework, called RI-GAN, that exploits residual and illumination using generative adversarial networks (GANs). The proposed framework detects and removes shadows in a coarse-to-fine fashion. At the coarse stage, we employ three generators to produce a coarse shadow-removal result, a residual image, and an inverse illumination map. We also incorporate two indirect shadow-removal images via the residual image and the inverse illumination map. With the residual image, the illumination map, and the two indirect shadow-removal images as auxiliary information, the refinement stage estimates a shadow mask to identify shadow regions in the image, and then refines the coarse shadow-removal result to the fine shadow-free image. We introduce a cross-encoding module to the refinement generator, in which the use of feature-crossing can provide additional details to promote the shadow mask and the high-quality shadow-removal result. In addition, we apply data augmentation to the discriminator to reduce the dependence between representations of the discriminator and the quality of the predicted image. Experiments for shadow detection and shadow removal demonstrate that our method outperforms state-of-the-art methods. Furthermore, RI-GAN exhibits good performance in terms of image dehazing, rain removal, and highlight removal, demonstrating the effectiveness and flexibility of the proposed framework.
- [1] Y. Akashi and T. Okatani. 2016. Separation of reflection components by sparse non-negative matrix factorization. Computer Vision and Image Understanding 146 (2016), 77–85.Google Scholar
Digital Library
- [2] . 2021. Robust representation learning with feedback for single image deraining. In Conference on Computer Vision and Pattern Recognition (CVPR’21).Google Scholar
Cross Ref
- [3] . 2019. Self-supervised GANs via auxiliary rotation loss. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 12146–12155.Google Scholar
Cross Ref
- [4] . 2021. CANet: A context-aware network for shadow removal. In IEEE International Conference on Computer Vision (ICCV’21). 4743–4752.Google Scholar
Cross Ref
- [5] . 2020. A multi-task mean teacher for semi-supervised shadow detection. In Conference on Computer Vision and Pattern Recognition (CVPR’20).Google Scholar
Cross Ref
- [6] . 2002. Improving shadow suppression in moving object detection with HSV color information. In IEEE Intelligent Transportation Systems. 334–339.Google Scholar
- [7] . 2020. Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting GAN. In AAAI. 10680–10687.Google Scholar
- [8] . 2020. Fast deep multi-patch hierarchical network for nonhomogeneous image dehazing. In Conference on Computer Vision and Pattern Recognition (CVPR’20).Google Scholar
- [9] . 2019. ARGAN: Attentive recurrent generative adversarial network for shadow detection and removal.. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 10212–10221.Google Scholar
Cross Ref
- [10] . 2008. Texture-consistent shadow removal. In European Conference on Computer Vision (ECCV’08). 437–450.Google Scholar
- [11] . 2005. On the removal of shadows from images. T-PAMI (2005).Google Scholar
- [12] . 2021. Auto-exposure fusion for single-image shadow removal. In Conference on Computer Vision and Pattern Recognition (CVPR’21).Google Scholar
Cross Ref
- [13] . 2016. A weighted variational model for simultaneous reflectance and illumination estimation. In Conference on Computer Vision and Pattern Recognition (CVPR’16). 2782–2790.Google Scholar
Cross Ref
- [14] . 2021. A multi-task network for joint specular highlight detection and removal. In Conference on Computer Vision and Pattern Recognition (CVPR’21).Google Scholar
- [15] . 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS’14).Google Scholar
- [16] . 2015. Learning to Remove Soft Shadows, Vol. 34. 1–15.Google Scholar
Digital Library
- [17] . 2011. Single-image shadow detection and removal using paired regions. In Conference on Computer Vision and Pattern Recognition (CVPR’11). 2033–2040.Google Scholar
Digital Library
- [18] . 2017. LIME: Low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing 26, 2 (2017), 982–993.Google Scholar
Digital Library
- [19] . 2011. Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 12 (2011), 2341–2353.Google Scholar
Digital Library
- [20] . 2018. Densely connected pyramid dehazing network. In Conference on Computer Vision and Pattern Recognition (CVPR’18). 3194–3203.Google Scholar
- [21] . 2020. Direction-aware spatial context features for shadow detection and removal. 42, 11 (2020), 2795–2808.Google Scholar
- [22] . 2018. Collaborative active visual recognition from crowds: A distributed ensemble approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 3 (2018), 582–594.Google Scholar
Cross Ref
- [23] . 2016. Automatic shadow detection and removal from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 3 (2016), 431–446.Google Scholar
Digital Library
- [24] . 2018. A+ D net: Training a shadow detector with adversarial shadow attenuation. In European Conference on Computer Vision (ECCV’18). 680–696.Google Scholar
Digital Library
- [25] . 2017. AOD-Net: All-in-one dehazing network. In IEEE International Conference on Computer Vision (ICCV’17). 4780–4788.Google Scholar
Cross Ref
- [26] . 2008. Texture-consistent shadow removal. In European Conference on Computer Vision (ECCV’08). (2008).Google Scholar
- [27] . 2019. GridDehazeNet: Attention-based multi-scale network for image dehazing. In International Conference on Computer Vision (ICCV’19). 7313–7322.Google Scholar
Cross Ref
- [28] . 2015. Multi-class multi-annotator active learning with robust Gaussian process for visual recognition. In International Conference on Computer Vision (ICCV’15). 2839–2847.Google Scholar
Digital Library
- [29] . 2017. Correlational gaussian processes for cross-domain visual recognition. In Conference on Computer Vision and Pattern Recognition (CVPR”17). 4932–4940.Google Scholar
Cross Ref
- [30] . 2014. Accurate object detection with location relaxation and regionlets re-localization. In Asia Conference on Computer Vision.Google Scholar
- [31] . 2020. End-to-end active object tracking and its real-world deployment via reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 6 (2020), 1317–1332.Google Scholar
Cross Ref
- [32] . 2000. Moving shadow and object detection in traffic scenes. In Conference on International Conference on Pattern Recognition.Google Scholar
- [33] . 2018. Spectral normalization for generative adversarial networks. International Conference on Learning Representations (2018).Google Scholar
- [34] . 2017. Shadow detection with conditional generative adversarial networks. In International Conference on Computer Vision (ICCV’19).Google Scholar
Cross Ref
- [35] . 2020. Cumulative rain density sensing network for single image derain. IEEE Signal Processing Letters 27 (2020), 406–410.Google Scholar
Cross Ref
- [36] . 2017. DeshadowNet: A multi-context embedding deep network for shadow removal. In Conference on Computer Vision and Pattern Recognition (CVPR’17). 2308–2316.Google Scholar
Cross Ref
- [37] . 2018. Single image haze removal using a generative adversarial network. In Conference on Computer Vision and Pattern Recognition (CVPR’18). 37–42.Google Scholar
- [38] . 2020. Single image deraining using bilateral recurrent network. IEEE Transactions on Image Processing 29, 99 (2020), 6852–6863.Google Scholar
Cross Ref
- [39] H. Shen and Z. Zheng. 2013. Real-time highlight removal using intensity ratio. Applied Optics 52, 19 (2013), 4483–4493.Google Scholar
Cross Ref
- [40] . 2008. The shadow meets the mask: Pyramid-based shadow removal. Computer Graphics Forum 27, 2 (2008), 577–586.Google Scholar
Cross Ref
- [41] . 2019. Conditional GANs for multi-illuminant color constancy: Revolution or yet another approach? In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’19). 1748–1758.Google Scholar
Cross Ref
- [42] . 2016. Large-scale training of shadow detectors with noisily-annotated shadow examples. In Conference on European Conference on Computer Vision (ECCV’16).Google Scholar
- [43] . 2020. A model-driven deep neural network for single image rain removal. In Conference on Computer Vision and Pattern Recognition (CVPR’20).Google Scholar
- [44] . 2018. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In Conference on Computer Vision and Pattern Recognition (CVPR’18). 1788–1797.Google Scholar
Cross Ref
- [45] . 2019. Underexposed photo enhancement using deep illumination estimation. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 6842–6850.Google Scholar
Cross Ref
- [46] . 2020. Instance shadow detection. In Conference on Computer Vision and Pattern Recognition (CVPR’20). 1877–1886.Google Scholar
Cross Ref
- [47] . 2019. Shadow inpainting and removal using generative adversarial networks with slice convolutions. Computer Graphics Forum 38, 7 (2019), 381–392.Google Scholar
Cross Ref
- [48] . 2007. Natural shadow matting. ACM Transactions on Graphics 26, 2 (2007), 8.Google Scholar
Digital Library
- [49] . 2021. Single UHD image dehazing via interpretable pyramid network. In Conference on Computer Vision and Pattern Recognition (CVPR’21).Google Scholar
- [50] . 2013. Fast shadow removal using adaptive multi-scale illumination transfer. Computer Graphics Forum 32, 8 (2013), 207–218.Google Scholar
Cross Ref
- [51] . 2013. Efficient shadow removal using subregion matching illumination transfer. Computer Graphics Forum 32, 7 (2013), 421–430.Google Scholar
Cross Ref
- [52] . 2019. General improvement method of specular component separation using high-emphasis filter and similarity function. ITE Transactions on Media Technology Applications 7, 2 (2019), 92–102.Google Scholar
Cross Ref
- [53] . 2015. Efficient and robust specular highlight removal. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 6 (2015), 1304–1311.Google Scholar
Digital Library
- [54] . 2020. BEDSR-net: A deep shadow removal network from a single document image. In Conference on Computer Vision and Pattern Recognition (CVPR’20) (2020), 12902–12911.Google Scholar
- [55] . 2020. RIS-GAN: Explore residual and illumination with generative adversarial networks for shadow removal. In Proceedings of the AAAI Conference on Artificial Intelligence (2020), 12829–12836.Google Scholar
Cross Ref
- [56] . 2019. Effective shadow removal via multi-scale image decomposition. The Visual Computer 35, 6–8 (2019), 1091–1104.Google Scholar
Digital Library
- [57] . 2015. Shadow remover: Image shadow removal based on illumination recovering optimization. IEEE Transactions on Image Processing 24, 11 (2015), 4623–4636.Google Scholar
Digital Library
- [58] . 2019. Distraction-aware shadow detection. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 5162–5171.Google Scholar
Cross Ref
- [59] . 2018. Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In European Conference on Computer Vision (ECCV’18). 122–137.Google Scholar
Digital Library
- [60] . 2021. Mitigating intensity bias in shadow detection via feature decomposition and reweighting. In International Conference on Computer Vision (ICCV’21). 4682–4691.Google Scholar
Cross Ref
Index Terms
Exploiting Residual and Illumination with GANs for Shadow Detection and Shadow Removal
Recommendations
Simple shadow removal using shadow depth map and illumination-invariant feature
AbstractShadows included in images provide useful information for visual scene analysis, but are also factors that negatively affect digital image analysis. Therefore, shadow detection and removal must be considered essential in the preprocessing of the ...
Shadow Remover: Image Shadow Removal Based on Illumination Recovering Optimization
In this paper, we present a novel shadow removal system for single natural images as well as color aerial images using an illumination recovering optimization method. We first adaptively decompose the input image into overlapped patches according to the ...
Shadow Removal Using Intensity Surfaces and Texture Anchor Points
Removal of shadows from a single image is a challenging problem. Producing a high-quality shadow-free image which is indistinguishable from a reproduction of a true shadow-free scene is even more difficult. Shadows in images are typically affected by ...






Comments