Abstract
Background: Underwater images, in general, suffer from low contrast and high color distortions due to the non-uniform attenuation of the light as it propagates through the water. In addition, the degree of attenuation varies with the wavelength, resulting in the asymmetric traversing of colors. Despite the prolific works for underwater image restoration (UIR) using deep learning, the above asymmetricity has not been addressed in the respective network engineering.
Contributions: As the first novelty, this article shows that attributing the right receptive field size (context) based on the traversing range of the color channel may lead to a substantial performance gain for the task of UIR. Further, it is important to suppress the irrelevant multi-contextual features and increase the representational power of the model. Therefore, as a second novelty, we have incorporated an attentive skip mechanism to adaptively refine the learned multi-contextual features. The proposed framework, called Deep WaveNet, is optimized using the traditional pixel-wise and feature-based cost functions. An extensive set of experiments have been carried out to show the efficacy of the proposed scheme over existing best-published literature on benchmark datasets. More importantly, we have demonstrated a comprehensive validation of enhanced images across various high-level vision tasks, e.g., underwater image semantic segmentation and diver’s 2D pose estimation. A sample video to exhibit our real-world performance is available at https://tinyurl.com/yzcrup9n. Also, we have open-sourced our framework at https://github.com/pksvision/Deep-WaveNet-Underwater-Image-Restoration.
- [1] . 2013. Wavelet neural networks: A practical guide. Neural Networks 42 (2013), 1–27.
DOI: Google ScholarDigital Library
- [2] . 2012. Enhancing underwater images and videos by fusion. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. 81–88.
DOI: Google ScholarCross Ref
- [3] . 2018. Color balance and fusion for underwater image enhancement. IEEE Transactions on Image Processing 27, 1 (2018), 379–393.
DOI: Google ScholarCross Ref
- [4] . 2022. Patches are All you need? In Submitted to the 10th International Conference on Learning Representations. https://openreview.net/forum?id=TVHS5Y4dNvM.
Under review. Google Scholar - [5] . 2020. Underwater image enhancement based on global and local equalization of histogram and dual-image multi-scale fusion. IEEE Access 8 (2020), 128973–128990.
DOI: Google ScholarCross Ref
- [6] . 2020. Underwater single image color restoration using haze-lines and a new quantitative dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020), 1–1.
DOI: Google ScholarCross Ref
- [7] . 2019. OpenPose: Realtime multi-person 2D pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019).Google Scholar
Digital Library
- [8] . 2017. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR.Google Scholar
- [9] . 2020. Perceptual underwater image enhancement with deep learning and physical priors. IEEE Transactions on Circuits and Systems for Video Technology (2020), 1–1.
DOI: Google ScholarCross Ref
- [10] . 2020. A wavelet based deep learning method for underwater image super resolution reconstruction. IEEE Access 8 (2020), 117759–117769.
DOI: Google ScholarCross Ref
- [11] . 2012. Underwater image enhancement by wavelength compensation and dehazing. IEEE Transactions on Image Processing 21, 4 (2012), 1756–1769.
DOI: Google ScholarDigital Library
- [12] . 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248–255.
DOI: Google ScholarCross Ref
- [13] . 2017. Image denoising via CNNs: An adversarial approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.Google Scholar
Cross Ref
- [14] . 2015. Compression artifacts reduction by a deep convolutional network. In 2015 IEEE International Conference on Computer Vision (ICCV’15). 576–584.
DOI: Google ScholarDigital Library
- [15] . 2016. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 2 (2016), 295–307.
DOI: Google ScholarDigital Library
- [16] . 2020. Deep underwater image restoration and beyond. IEEE Signal Processing Letters 27 (2020), 675–679.
DOI: Google ScholarCross Ref
- [17] . 2018. Enhancing underwater imagery using generative adversarial networks. In 2018 IEEE International Conference on Robotics and Automation (ICRA’18).
DOI: Google ScholarDigital Library
- [18] . 2014. A retinex-based enhancing approach for single underwater image. In 2014 IEEE International Conference on Image Processing (ICIP’14).
DOI: Google ScholarCross Ref
- [19] . Generative adversarial nets. In Advances in Neural Information Processing Systems, , , , , and (Eds.). Curran Associates, Inc.Google Scholar
Digital Library
- [20] . 2020. Underwater image enhancement using a multiscale dense generative adversarial network. IEEE Journal of Oceanic Engineering 45, 3 (2020), 862–870.
DOI: Google ScholarCross Ref
- [21] . 2020. Underwater image enhancement based on a spiral generative adversarial framework. IEEE Access 8 (2020), 218838–218852.
DOI: Google ScholarCross Ref
- [22] . 2011. Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 12 (2011), 2341–2353.
DOI: Google ScholarDigital Library
- [23] . 2020. Benchmarking underwater image enhancement and restoration, and beyond. IEEE Access 8 (2020), 122078–122091.
DOI: Google ScholarCross Ref
- [24] . 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 (ICML’15). JMLR.org.Google Scholar
- [25] . 2020. Semantic segmentation of underwater imagery: Dataset and benchmark. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’20). IEEE/RSJ.Google Scholar
Digital Library
- [26] . 2020. Simultaneous enhancement and super-resolution of underwater imagery for improved visual perception. In Robotics: Science and Systems (RSS’20). Corvalis, Oregon.
DOI: Google ScholarCross Ref
- [27] . 2020. Underwater image super-resolution using deep residual multipliers. In 2020 IEEE International Conference on Robotics and Automation (ICRA’20).
DOI: Google ScholarCross Ref
- [28] . 2020. Fast underwater image enhancement for improved visual perception. IEEE Robotics and Automation Letters 5, 2 (2020), 3227–3234.
DOI: Google ScholarCross Ref
- [29] . 2020. Fast underwater image enhancement for improved visual perception. IEEE Robotics and Automation Letters 5, 2 (2020).
DOI: Google ScholarCross Ref
- [30] . 2020. A flexible deep CNN framework for image restoration. IEEE Transactions on Multimedia 22, 4 (2020), 1055–1068.
DOI: Google ScholarDigital Library
- [31] . 2016. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision.Google Scholar
Cross Ref
- [32] . 2016. Accurate image super-resolution using very deep convolutional networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR Oral’16).Google Scholar
Cross Ref
- [33] . 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR’15), Conference Track Proceedings, and (Eds.).Google Scholar
- [34] . 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1 (NIPS’12). 1097–1105.Google Scholar
Digital Library
- [35] . 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).Google Scholar
Cross Ref
- [36] . 2021. Underwater image enhancement via medium transmission-guided multi-color space embedding. IEEE Transactions on Image Processing 30 (2021).
DOI: Google ScholarCross Ref
- [37] . 2020. An underwater image enhancement benchmark dataset and beyond. IEEE Transactions on Image Processing 29 (2020), 4376–4389.
DOI: Google ScholarCross Ref
- [38] . 2016. Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior. IEEE Transactions on Image Processing 25, 12 (2016), 5664–5677.
DOI: Google ScholarDigital Library
- [39] . 2018. Emerging from water: Underwater image color correction based on weakly supervised color transfer. IEEE Signal Processing Letters 25, 3 (2018), 323–327.
DOI: Google ScholarCross Ref
- [40] . 2016. Single underwater image restoration by blue-green channels dehazing and red channel correction. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’16). 1731–1735.
DOI: Google ScholarDigital Library
- [41] . 2018. Structure-revealing low-light image enhancement via robust retinex model. IEEE Transactions on Image Processing 27, 6 (2018), 2828–2841.
DOI: Google ScholarCross Ref
- [42] . 2019. Underwater image high definition display using the multilayer perceptron and color feature-based SRCNN. IEEE Access 7 (2019), 83721–83728.
DOI: Google ScholarCross Ref
- [43] . 2019. Enhance visual recognition under adverse conditions via deep networks. IEEE Transactions on Image Processing 28, 9 (2019), 4401–4412.
DOI: Google ScholarCross Ref
- [44] . 2018. When image denoising meets high-level vision tasks: A deep learning approach. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI’18). AAAI Press, 842–848. Google Scholar
Digital Library
- [45] . 2019. Underwater image enhancement with a deep residual framework. IEEE Access 7 (2019), 94614–94629.
DOI: Google ScholarCross Ref
- [46] . 2018. Multi-level wavelet-CNN for image restoration. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’18).
DOI: Google ScholarCross Ref
- [47] . 2020. MLFcGAN: Multilevel feature fusion-based conditional GAN for underwater image color correction. IEEE Geoscience and Remote Sensing Letters 17, 9 (2020).
DOI: Google ScholarCross Ref
- [48] . 2015. Fully convolutional networks for semantic segmentation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 3431–3440.
DOI: Google ScholarCross Ref
- [49] . 2017. Underwater image super-resolution by descattering and fusion. IEEE Access 5 (2017), 670–679.
DOI: Google ScholarCross Ref
- [50] . 2018. Contrast limited adaptive histogram equalization-based fusion in YIQ and HSI color spaces for underwater image enhancement. International Journal of Pattern Recognition and Artificial Intelligence 32, 7 (2018), 1854018.
DOI: arXiv:https://doi.org/10.1142/S0218001418540186 Google ScholarCross Ref
- [51] . 2013. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters 20, 3 (2013), 209–212.
DOI: Google ScholarCross Ref
- [52] . 2016. Human-visual-system-inspired underwater image quality measures. IEEE Journal of Oceanic Engineering 41, 3 (2016), 541–551.
DOI: Google ScholarCross Ref
- [53] . 2019. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 8024–8035.Google Scholar
- [54] . 2017. Large kernel matters - improve semantic segmentation by global convolutional network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 1743–1751.
DOI: Google ScholarCross Ref
- [55] . 2017. Underwater image restoration based on image blurriness and light absorption. IEEE Transactions on Image Processing 26, 4 (2017), 1579–1594.
DOI: Google ScholarDigital Library
- [56] . 2015. Single underwater image enhancement using depth estimation based on blurriness. In 2015 IEEE International Conference on Image Processing (ICIP’15).
DOI: Google ScholarDigital Library
- [57] . 2018. Generalization of the dark channel prior for single image restoration. IEEE Transactions on Image Processing 27, 6 (2018), 2856–2868.
DOI: Google ScholarCross Ref
- [58] . 2020. Scale-aware conditional generative adversarial network for image dehazing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV’20).Google Scholar
- [59] . 2019. Dual-domain single image de-raining using conditional generative adversarial network. In 2019 IEEE International Conference on Image Processing (ICIP’19).
DOI: Google ScholarCross Ref
- [60] . 2006. Image information and visual quality. IEEE Transactions on Image Processing 15, 2 (2006).
DOI: Google ScholarDigital Library
- [61] . 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16).Google Scholar
Cross Ref
- [62] . 2017. Hand keypoint detection in single images using multiview bootstrapping. In CVPR.Google Scholar
- [63] . n.d. Very deep convolutional networks for large-scale image recognition. CoRR (n.d.).Google Scholar
- [64] . 2014. Intriguing properties of neural networks. In International Conference on Learning Representations.Google Scholar
- [65] . 2021. Coarse-to-fine CNN for image super-resolution. IEEE Transactions on Multimedia 23 (2021), 1489–1502.
DOI: Google ScholarDigital Library
- [66] . Attention is all you need. In Advances in Neural Information Processing Systems, , , , , , , and (Eds.). Curran Associates, Inc.Google Scholar
- [67] . 2020. CA-GAN: Class-condition attention GAN for underwater image enhancement. IEEE Access 8 (2020), 130719–130728.
DOI: Google ScholarCross Ref
- [68] . 2019. Single underwater image enhancement based on \( L_{P} \)-norm decomposition. IEEE Access 7 (2019), 145199–145213.
DOI: Google ScholarCross Ref
- [69] . 2020. Dual super-resolution learning for semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20).Google Scholar
Cross Ref
- [70] . 2017. Underwater image restoration via maximum attenuation identification. IEEE Access 5 (2017).
DOI: Google ScholarCross Ref
- [71] . 2018. Understanding convolution for semantic segmentation. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV’18). IEEE Computer Society, Los Alamitos, CA, 1451–1460.
DOI: Google ScholarCross Ref
- [72] . 2015. A patch-structure representation method for quality assessment of contrast changed images. IEEE Signal Processing Letters 22, 12 (2015).
DOI: Google ScholarCross Ref
- [73] . 2017. Single underwater image restoration using attenuation-curve prior. In 2017 IEEE International Symposium on Circuits and Systems (ISCAS’17). 1–4.
DOI: Google ScholarCross Ref
- [74] . 2017. A deep CNN method for underwater image enhancement. In 2017 IEEE International Conference on Image Processing (ICIP’17). 1382–1386.
DOI: Google ScholarDigital Library
- [75] . 2004. Image quality assessment: From error visibility to structural similarity. 13, 4 (2004). Google Scholar
- [76] . 2016. Studying very low resolution recognition using deep networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16).
DOI: Google ScholarCross Ref
- [77] . 2016. Convolutional pose machines. In CVPR.Google Scholar
- [78] . 2013. Single underwater image enhancement with a new optical model. In 2013 IEEE International Symposium on Circuits and Systems (ISCAS’13). 753–756.
DOI: Google ScholarCross Ref
- [79] . 2018. CBAM: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV’18).Google Scholar
Digital Library
- [80] . 2006. Non-linear algorithm for contrast enhancement for image using wavelet neural network. In 2006 9th International Conference on Control, Automation, Robotics and Vision. 1–6.
DOI: Google ScholarCross Ref
- [81] . 2017. Underwater image enhancement based on structure-texture decomposition. In 2017 IEEE International Conference on Image Processing (ICIP’17).
DOI: Google ScholarDigital Library
- [82] . 2015. An underwater color image quality evaluation metric. IEEE Transactions on Image Processing 24, 12 (2015), 6062–6071.
DOI: Google ScholarDigital Library
- [83] . 2019. Underwater image enhancement using scene depth-based adaptive background light estimation and dark channel prior algorithms. IEEE Access 7 (2019).
DOI: Google ScholarCross Ref
- [84] . 2019. Deep learning for single image super-resolution: A brief review. Trans. Multi. 21, 12 (
Dec. 2019), 3106–3121.DOI: Google ScholarDigital Library
- [85] . 2016. Multi-scale context aggregation by dilated convolutions. In International Conference on Learning Representations (ICLR’16).Google Scholar
- [86] . 2016. Deep convolution networks for compression artifacts reduction. CoRR abs/1608.02778 (2016).
arxiv:1608.02778 Google Scholar - [87] . 2021. Mask guided matting via progressive refinement network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21). 1154–1163.Google Scholar
Cross Ref
- [88] . 2011. Adaptive deconvolutional networks for mid and high level feature learning. In 2011 International Conference on Computer Vision. 2018–2025.
DOI: Google ScholarDigital Library
- [89] . 2018. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google Scholar
Cross Ref
- [90] . 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In 2017 IEEE International Conference on Computer Vision (ICCV’17).Google Scholar
Cross Ref
Index Terms
Wavelength-based Attributed Deep Neural Network for Underwater Image Restoration
Recommendations
Single-shot underwater image restoration: A visual quality-aware method based on light propagation model
Highlights- A new approach to restore images acquired from underwater scenes.
- The approach ...
AbstractIn this paper, we present a novel method to restore the visual quality of images from scenes immersed in participating media, in particular water. Our method builds upon existing physics-based model and estimates the scene radiance by ...
Underwater Image Restoration Based on Image Blurriness and Light Absorption
Underwater images often suffer from color distortion and low contrast, because light is scattered and absorbed when traveling through water. Such images with different color tones can be shot in various lighting conditions, making restoration and ...
DeepRecog: Threefold underwater image deblurring and object recognition framework for AUV vision systems
AbstractUnderwater explorations and probes have now become frequent for marine discovery and endangered resources protection. The decrease in natural light with an increase in water depth and the characteristic of the medium to absorb and scatter light ...






Comments