Abstract
The capability of image semantic segmentation may be deteriorated due to the noisy input image, where image denoising prior to segmentation may help. Both image denoising and semantic segmentation have been developed significantly with the advance of deep learning. In this work, we are interested in the synergy between these two tasks by using a holistic deep model. We observe that not only denoising helps combat the drop of segmentation accuracy due to the noisy input, but also pixel-wise semantic information boosts the capability of denoising. We then propose a boosting network to perform denoising and segmentation alternately. The proposed network is composed of multiple segmentation and denoising blocks (SDBs), each of which estimates a semantic map and then uses the map to regularize denoising. Experimental results show that the denoised image quality is improved substantially and the segmentation accuracy is improved to close to that on clean images, and segmentation and denoising are both boosted as the number of SDBs increases. On the Cityscapes dataset, using three SDBs improves the denoising quality to 34.42 dB in PSNR, and the segmentation accuracy to 66.5 in mIoU, when the additive white Gaussian noise level is 50.
- [1] . 2018. A high-quality denoising dataset for smartphone cameras. In CVPR. IEEE, 1692–1700.Google Scholar
- [2] . 2021. Unsupervised hyperspectral stimulated Raman microscopy image enhancement: Denoising and segmentation via one-shot deep learning. arXiv preprint arXiv:2104.08338 (2021).Google Scholar
- [3] . 2019. Real image denoising with feature attention. In ICCV. IEEE, 3155–3164.Google Scholar
- [4] . 2017. Category-specific object image denoising. IEEE Trans. Image Process. 26, 11 (2017), 5506–5518.Google Scholar
Digital Library
- [5] . 2010. Contour detection and hierarchical image segmentation. IEEE Trans. Patt. Anal. Mach. Intell. 33 (2010), 898–916.Google Scholar
Digital Library
- [6] . 2017. SegNet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Patt. Anal. Mach. Intell. 39, 12 (2017), 2481–2495.Google Scholar
Cross Ref
- [7] . 2020. DenoiSeg: Joint denoising and segmentation. In ECCV. Springer, 324–337.Google Scholar
- [8] . 2006. A general iterative regularization framework for image denoising. In CISS. IEEE, 452–457.Google Scholar
- [9] . 2019. Real-world image denoising with deep boosting. IEEE Trans. Patt. Anal. Mach. Intell. 42, 12 (2019), 3071–3087.Google Scholar
Cross Ref
- [10] . 2021. HINet: Half instance normalization network for image restoration. In CVPR. 182–192.Google Scholar
- [11] . 2014. Semantic image segmentation with deep convolutional nets and fully connected CRFs. arXiv preprint arXiv:1412.7062 (2014).Google Scholar
- [12] . 2017. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Patt. Anal. Mach. Intell. 40, 4 (2017), 834–848.Google Scholar
Cross Ref
- [13] . 2016. The cityscapes dataset for semantic urban scene understanding. In CVPR. IEEE, 3213–3223.Google Scholar
- [14] . 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16, 8 (2007), 2080–2095.Google Scholar
Digital Library
- [15] . 2011. Sparsity-based image denoising via dictionary learning and structural clustering. In CVPR. IEEE, 457–464.Google Scholar
- [16] . 2006. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15, 12 (2006), 3736–3745.Google Scholar
Digital Library
- [17] . 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 249–256.Google Scholar
- [18] . 2019. Toward convolutional blind denoising of real photographs. In CVPR. IEEE, 1712–1722.Google Scholar
- [19] . 2016. Incremental boosting convolutional neural network for facial action unit recognition. In NIPS. 109–117.Google Scholar
- [20] . 2015. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In ICCV. 1026–1034.Google Scholar
- [21] . 2016. Deep residual learning for image recognition. In CVPR. IEEE, 770–778.Google Scholar
- [22] . 2015. Image denoising with edge-preserving and segmentation based on mask NHA. IEEE Trans. Image Process. 24 (2015), 6025–6033.Google Scholar
Digital Library
- [23] . 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML. 448–456.Google Scholar
- [24] . 2016. Perceptual losses for real-time style transfer and super-resolution. In ECCV. Springer, 694–711.Google Scholar
- [25] . 2020. Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. In CVPR. IEEE, 3482–3492.Google Scholar
- [26] . 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- [27] . 2019. Noise2Void-learning denoising from single noisy images. In CVPR. IEEE, 2129–2137.Google Scholar
- [28] . 2019. Anatomical priors for image segmentation via post-processing with denoising autoencoders. In MICCAI. Springer, 585–593.Google Scholar
- [29] . 2018. Deep CNN based MR image denoising for tumor segmentation using watershed transform. Int. J. Eng. Technol. 7 (2018), 37–42.Google Scholar
Cross Ref
- [30] . 2018. Noise2Noise: Learning image restoration without clean data. In ICML. 2965–2974.Google Scholar
- [31] . 2021. SwinIR: Image restoration using Swin transformer. In ICCV. 1833–1844.Google Scholar
- [32] . 2017. RefineNet: Multi-path refinement networks for high-resolution semantic segmentation. In CVPR. IEEE, 1925–1934.Google Scholar
- [33] . 2014. Microsoft COCO: Common objects in context. In ECCV. Springer, 740–755.Google Scholar
- [34] . 2021. Learning degraded image classification with restoration data fidelity. arXiv preprint arXiv:2101.09606 (2021).Google Scholar
- [35] . 2020. Connecting image denoising and high-level vision tasks via deep learning. IEEE Trans. Image Process. 29 (2020), 3695–3706.Google Scholar
Digital Library
- [36] . 2018. When image denoising meets high-level vision tasks: A deep learning approach. In IJCAI. 842–848.Google Scholar
- [37] . 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV. 10012–10022.Google Scholar
- [38] . 2015. Fully convolutional networks for semantic segmentation. In CVPR. IEEE, 3431–3440.Google Scholar
- [39] . 2009. Non-local sparse models for image restoration. In ICCV. IEEE, 2272–2279.Google Scholar
- [40] . 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In NIPS. 2802–2810.Google Scholar
- [41] . 2016. Boosted convolutional neural networks. In BMVC. 1–6.Google Scholar
- [42] . 2019. PyTorch: An imperative style, high-performance deep learning library. In NeurIPS. 8024–8035.Google Scholar
- [43] . 2017. Benchmarking denoising algorithms with real photographs. In CVPR. IEEE, 1586–1595.Google Scholar
- [44] . 2018. Class-aware fully convolutional Gaussian and Poisson denoising. IEEE Trans. Image Process. 27, 11 (2018), 5707–5722.Google Scholar
Cross Ref
- [45] . 2017. Video deblurring via semantic segmentation and pixel-wise non-linear kernel. In ICCV. IEEE, 1077–1085.Google Scholar
- [46] . 2015. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI. Springer, 234–241.Google Scholar
- [47] . 2020. Is normalization indispensable for training deep neural network? In NeurIPS. 13434–13444.Google Scholar
- [48] . 2018. Classification-driven dynamic image enhancement. In CVPR. IEEE, 4033–4041.Google Scholar
- [49] . 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
- [50] . 1999. Segmentation based denoising using multiple compaction domains. In ICIP. IEEE, 372–375.Google Scholar
- [51] . 2021. Segmenter: Transformer for semantic segmentation. In ICCV. 7262–7272.Google Scholar
- [52] . 2012. How to SAIF-ly boost denoising performance. IEEE Trans. Image Process. 22, 4 (2012), 1470–1485.Google Scholar
Digital Library
- [53] . 2018. Deep image prior. In CVPR. IEEE, 9446–9454.Google Scholar
- [54] . 2017. Attention is all you need. Adv. Neural Inf. Process. 30 (2017).Google Scholar
- [55] . 2009. Denoising and segmentation of 3D brain images.Image Process. Comput. Vis. Patt. Recog. 9 (2009), 561–567.Google Scholar
- [56] . 2020. Deep high-resolution representation learning for visual recognition. IEEE Trans. Patt. Anal. Mach. Intell. (2020).
DOI: Google ScholarCross Ref
- [57] . 2020. Dual super-resolution learning for semantic segmentation. In CVPR. IEEE, 3774–3783.Google Scholar
- [58] . 2019. Segmentation-aware image denoising without knowing true segmentation. arXiv preprint arXiv:1905.08965 (2019).Google Scholar
- [59] . 2021. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV. 568–578.Google Scholar
- [60] . 2018. Recovering realistic texture in image super-resolution by deep spatial feature transform. In CVPR. IEEE, 606–615.Google Scholar
- [61] . 2021. Uformer: A general u-shaped transformer for image restoration. arXiv preprint arXiv:2106.03106 (2021).Google Scholar
- [62] . 2014. Segmentation based denoising of PET images: An iterative approach via regional means and affinity propagation. In MICCAI. Springer, 698–705.Google Scholar
- [63] . 2018. Joint solution for PET image segmentation, denoising, and partial volume correction. Med. Image Anal. 46 (2018), 229–243.Google Scholar
Cross Ref
- [64] . 2018. DenseASPP for semantic segmentation in street scenes. In CVPR. IEEE, 3684–3692.Google Scholar
- [65] . 2015. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 (2015).Google Scholar
- [66] . 2020. Object-contextual representations for semantic segmentation. In ECCV. Springer, 173–190.Google Scholar
- [67] . 2021. Restormer: Efficient transformer for high-resolution image restoration. arXiv preprint arXiv:2111.09881 (2021).Google Scholar
- [68] . 2021. Multi-stage progressive image restoration. In CVPR. 14821–14831.Google Scholar
- [69] . 2019. Two-stream action recognition-oriented video super-resolution. In ICCV. IEEE, 8799–8808.Google Scholar
- [70] . 2017. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26, 7 (2017), 3142–3155.Google Scholar
Digital Library
- [71] . 2017. Learning deep CNN denoiser prior for image restoration. In CVPR. IEEE, 3929–3938.Google Scholar
- [72] . 2018. FFDNet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 27, 9 (2018), 4608–4622.Google Scholar
- [73] . 2018. Joint task-recursive learning for semantic segmentation and depth estimation. In ECCV. Springer, 235–251.Google Scholar
- [74] . 2017. Pyramid scene parsing network. In CVPR. IEEE, 2881–2890.Google Scholar
- [75] . 2021. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR. 6881–6890.Google Scholar
- [76] . 2017. Scene parsing through ADE20K dataset. In CVPR. IEEE, 633–641.Google Scholar
Index Terms
Synergy between Semantic Segmentation and Image Denoising via Alternate Boosting
Recommendations
Image denoising with complex ridgelets
In this paper, we propose a novel image denoising method by incorporating the dual-tree complex wavelets into the ordinary ridgelet transform. The approximate shift invariant property of the dual-tree complex wavelet and the high directional sensitivity ...
Image denoising using SVM classification in nonsubsampled contourlet transform domain
For image denoising, the main challenge is how to preserve the information-bearing structures such as edges and textures to get satisfactory visual quality when improving the signal-to-noise-ratio (SNR). Edge-preserving image denoising has become a very ...
Image Denoising Using Neighbouring Contourlet Coefficients
ISNN '08: Proceedings of the 5th international symposium on Neural Networks: Advances in Neural Networks, Part IIThe denoising of a natural image corrupted by Gaussian white noise is a classical problem in image processing. In this paper, a new image denoising method is proposed by using the contourlet transform. The thresholding process employs a small ...






Comments