Abstract
The block-based coding structure in the hybrid video coding framework inevitably introduces compression artifacts such as blocking, ringing, and so on. To compensate for those artifacts, extensive filtering techniques were proposed in the loop of video codecs, which are capable of boosting the subjective and objective qualities of reconstructed videos. Recently, neural network-based filters were presented with the power of deep learning from a large magnitude of data. Though the coding efficiency has been improved from traditional methods in High-Efficiency Video Coding (HEVC), the rich features and information generated by the compression pipeline have not been fully utilized in the design of neural networks. Therefore, in this article, we propose the Residual-Reconstruction-based Convolutional Neural Network (RRNet) to further improve the coding efficiency to its full extent, where the compression features induced from bitstream in form of prediction residual are fed into the network as an additional input to the reconstructed frame. In essence, the residual signal can provide valuable information about block partitions and can aid reconstruction of edge and texture regions in a picture. Thus, more adaptive parameters can be trained to handle different texture characteristics. The experimental results show that our proposed RRNet approach presents significant BD-rate savings compared to HEVC and the state-of-the-art CNN-based schemes, indicating that residual signal plays a significant role in enhancing video frame reconstruction.
- [1] . 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 126–135.Google Scholar
Cross Ref
- [2] . 2019. FFmpeg software, A complete, cross-platform solution to record, convert, and stream audio and video. Retrieved from http://ffmpeg.org/.Google Scholar
- [3] . 2001. Calculation of average PSNR differences between RD-curves. Document VCEG-M33, Austin, Texas.Google Scholar
- [4] . 2019. Versatile video coding (draft 4). Document ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 JVET-M1001-v6, Marrakech, MA.Google Scholar
- [5] . 2016. Variational lossy autoencoder. arXiv preprint arXiv:1611.02731 (2016).Google Scholar
- [6] . 2018. Recent advances in efficient computation of deep convolutional neural networks. Front. Inf. Technol. Electron. Eng. 19 (2018), 64–77.Google Scholar
Cross Ref
- [7] . 2017. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282 (2017).Google Scholar
- [8] . 2018. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Sig. Process. Mag. 35, 1 (2018), 126–136.Google Scholar
Cross Ref
- [9] . 2009. Adaptive filter based on combination of sum-modified Laplacian filter indexing and quadtree partitioning. ITU-T/ISO/IEC JCT-VC Document VCEG-AL27.Google Scholar
- [10] . 2017. A convolutional neural network approach for post-processing in HEVC intra coding. In Proceedings of the IEEE International Conference on Multimedia Modeling. 28–39.Google Scholar
Cross Ref
- [11] . 2015. Compression artifacts reduction by a deep convolutional network. In Proceedings of the IEEE International Conference on Computer Vision. 576–584. Google Scholar
Digital Library
- [12] . 2012. Sample adaptive offset in the HEVC standard. IEEE Trans. Circ. Syst. Vid. Technol. 22, 12 (2012), 1755–1764. Google Scholar
Digital Library
- [13] . 2010. TE10 subtest 3: Quadtree-based adaptive offset. ITU-T/ISO/IEC JCT-VC Document JCTVC-C147.Google Scholar
- [14] . 2011. CE8 subset 3: Picture quadtree adaptive offset. ITU-T/ISO/IEC JCT-VC Document JCTVC-D122.Google Scholar
- [15] . 2011. CE13: Sample adaptive offset with LCU-independent decoding. ITU-T/ISO/IEC JCT-VC Document JCTVC-E049.Google Scholar
- [16] . 2014. Quadtree-based non-local Kuan’s filtering in video compression. J. Vis. Commun. Image Repres. 25, 5 (2014), 1044–1055.Google Scholar
Cross Ref
- [17] . 2015. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE International Conference on Computer Vision. 1026–1034. Google Scholar
Digital Library
- [18] . 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.Google Scholar
Cross Ref
- [19] . 2018. Enhancing HEVC compressed videos with a partition-masked convolutional neural network. In Proceedings of the 25th IEEE International Conference on Image Processing (ICIP). 216–220.Google Scholar
Cross Ref
- [20] . 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504–507.Google Scholar
Cross Ref
- [21] . 2010. In-loop adaptive restoration. ITU-T/ISO/IEC JCT-VC Document JCTVC-B077.Google Scholar
- [22] . 2019. PIRM challenge on perceptual image enhancement on smartphones: Report. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops.Google Scholar
Cross Ref
- [23] . 2019. Content-aware convolutional neural network for in-loop filtering in high efficiency video coding. IEEE Trans. Image Process. 28, 7 (2019).Google Scholar
Digital Library
- [24] . 2017. Spatial-temporal residue network based in-loop filter for video coding. In Proceedings of the IEEE Visual Communications and Image Processing Conference (VCIP). 1–4.Google Scholar
Cross Ref
- [25] . 2020. Residual guided deblocking with deep learning. In Proceedings of the IEEE International Conference on Image Processing (ICIP).Google Scholar
Cross Ref
- [26] . 2020. A flexible deep CNN framework for image restoration. IEEE Trans. Multimedia 22, 4 (2020), 1055–1068.Google Scholar
Digital Library
- [27] . 2015. Learning-based joint super-resolution and deblocking for a highly compressed image. IEEE Trans. Multimedia 17, 7 (2015), 921–934.Google Scholar
Digital Library
- [28] . 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- [29] . 2018. An iterative framework of cascaded deblocking and superresolution for compressed images. IEEE Trans. Multimedia 20, 6 (2018), 1305–1320.Google Scholar
Cross Ref
- [30] . 2019. Methodology and reporting template for neural network coding tool testing. Document ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 JVET-M1006-v1, Marrakech, MA.Google Scholar
- [31] . 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 136–144.Google Scholar
Cross Ref
- [32] . 2011. A ringing-artifact reduction method for block-DCT-based image resizing. IEEE Trans. Circ. Syst. Vid. Technol. 21, 7 (2011), 879–889. Google Scholar
Digital Library
- [33] . 2019. Partition-aware adaptive switching neural networks for post-processing in HEVC. IEEE Trans. Multimedia 22, 11 (2019).Google Scholar
- [34] . 2003. Adaptive deblocking filter. IEEE Trans. Circ. Syst. Vid. Technol. 13, 7 (2003), 614–619. Google Scholar
Digital Library
- [35] . 2007. An in/post-loop deblocking filter with hybrid filtering schedule. IEEE Trans. Circ. Syst. Vid. Technol. 17, 7 (2007), 937–943. Google Scholar
Digital Library
- [36] . 2018. Deep Kalman filtering network for video compression artifact reduction. In Proceedings of the European Conference on Computer Vision (ECCV). 568–584.Google Scholar
Cross Ref
- [37] . 2016. Nonlocal in-loop filter: The way toward next-generation video coding? IEEE MultiMedia 23, 2 (2016), 16–26.Google Scholar
Cross Ref
- [38] . 2010. Samsung’s response to the call for proposals on video compression technology. ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 JCT-VC Document JCTVC-A124, Dresden, DE.Google Scholar
- [39] . 2012. HEVC deblocking filter. IEEE Trans. Circ. Syst. Vid. Technol. 22, 12 (2012), 1746–1754. Google Scholar
Digital Library
- [40] . 2016. CNN-based in-loop filtering for coding efficiency improvement. In Proceedings of the IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP). 1–5.Google Scholar
Cross Ref
- [41] . 2015. U-Net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention. 234–241.Google Scholar
Cross Ref
- [42] . 2018. Common test conditions. Document ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 JCTVC-AE1100, San Diego, US.Google Scholar
- [43] . 2012. Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. Circ. Syst. Vid. Technol. (2012), 1649–1668. Google Scholar
Digital Library
- [44] . 2017. Detail-revealing deep video super-resolution. In Proceedings of the IEEE International Conference on Computer Vision. 4472–4480.Google Scholar
Cross Ref
- [45] . 2013. Adaptive loop filtering for video coding. IEEE J. Select. Topics Sig. Process. 7, 6 (2013), 934–945.Google Scholar
Cross Ref
- [46] . 2016. Residual networks behave like ensembles of relatively shallow networks. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 550–558. Google Scholar
Digital Library
- [47] . 2014. Mathematical Statistics with Applications. Cengage Learning.Google Scholar
- [48] . 2017. A novel deep learning-based method of improving coding efficiency from the decoder-end for HEVC. In Proceedings of the Data Compression Conference (DCC). 410–419.Google Scholar
Cross Ref
- [49] . 2018. Dense residual convolutional neural network based in-loop filter for HEVC. In Proceedings of the IEEE Visual Communications and Image Processing (VCIP). 1–4.Google Scholar
Cross Ref
- [50] . 2016. D3: Deep dual-domain based fast restoration of JPEG-compressed images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2764–2772.Google Scholar
Cross Ref
- [51] . 2003. Overview of the H. 264/AVC video coding standard. IEEE Trans. Circ. Syst. Vid. Technol. 13, 7 (2003), 560–576. Google Scholar
Digital Library
- [52] . 2017. Video enhancement with task-oriented flow. Int. J. Comput. Vis. 127, 8 (2017), 1–20. Google Scholar
Digital Library
- [53] . 2018. Enhancing quality for HEVC compressed videos. IEEE Trans. Circ. Syst. Vid. Technol. 29, 7 (2018), 2039–2054.Google Scholar
- [54] . 2017. Decoder-side HEVC quality enhancement with scalable convolutional neural network. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). 817–822.Google Scholar
Cross Ref
- [55] . 2010. Deconvolutional networks. In Proceedings of the Conference on Computer Vision and Pattern Recognition.Google Scholar
Cross Ref
- [56] . 2016. Low-rank-based nonlocal adaptive loop filter for high-efficiency video compression. IEEE Trans. Circ. Syst. Vid. Technol. 27, 10 (2016), 2177–2188.Google Scholar
Digital Library
- [57] . 2018. Residual highway convolutional neural networks for in-loop filtering in HEVC. IEEE Trans. Image Process. 27, 8 (2018), 3827–3841.Google Scholar
Cross Ref
- [58] . 2012. Efficient parallel framework for H.264/AVC deblocking filter on many-core platform. IEEE Trans. Multimedia 14, 3 (2012), 510–524. Google Scholar
Digital Library
Index Terms
Residual-guided In-loop Filter Using Convolution Neural Network
Recommendations
NR-CNN: Nested-Residual Guided CNN In-loop Filtering for Video Coding
Recently, deep learning for video coding, such as deep predictive coding, deep transform coding, and deep in-loop filtering, has been an emerging research area. The coding gain of hybrid coding framework could be extensively promoted by the data-driven ...
A nonlocal HEVC in-loop filter using CNN-based compression noise estimation
AbstractHigh-efficiency video coding (HEVC) effectively reduces the amount of video data while unavoidably introducing compression noise. The in-loop filter can enhance the reconstructed frames at the encoder to prevent compression noise from transmitting ...
Convolutional neural network based low complexity HEVC intra encoder
AbstractVideo coding is one of the key technologies of visual sensors. As the state-of-art video coding standard, High Efficiency Video Coding (HEVC) achieves a significant high compression ratio for video. However, it also introduces heavy computational ...






Comments