Abstract
Recently, deep learning for video coding, such as deep predictive coding, deep transform coding, and deep in-loop filtering, has been an emerging research area. The coding gain of hybrid coding framework could be extensively promoted by the data-driven models. However, previous deep coding tools especially deep in-loop filtering mainly consider the performance improvement while pay less attention to the reliability, usability, and adaptivity of the networks. In this article, a nested-residual guided convolutional neural network (NR-CNN) structure with cascaded global shortcut and configurable residual blocks is proposed for in-loop filtering. By taking advantage of the correlation between different color components, we further extend the NR-CNN by utilizing luminance as textural and structural guidance for chrominance filtering, which significantly improves the filtering performance. To fully exploit the proposed network into codec integration, we subsequently introduce an efficient and adaptive framework consisting of an adaptive granularity optimization and a parallel inference pipeline for deep learning based filtering. The former contributes to the coding performance improvement through an adaptive decision-making based on rate-distortion analysis at various granularities. The latter reduces the running time of network inference. The extensive experimental results show the superiority of the proposed method, achieving 8.2%, 14.9%, and 13.2% BD-rate savings on average under random access (RA) configuration. Meanwhile, the proposed method also obtains better subjective quality.
- [1] . 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 126–135.Google Scholar
Cross Ref
- [2] . 2019. The common test conditions of AVS3. AVS-Doc, N2727.Google Scholar
- [3] . 2020. Algorithm description for versatile video coding and test model 10 (VTM’10). JVET-S2002.Google Scholar
- [4] . 2017. A convolutional neural network approach for post-processing in HEVC intra coding. In Proceedings of the International Conference on Multimedia Modeling. Springer, 28–39.Google Scholar
Cross Ref
- [5] . 2015. Compression artifacts reduction by a deep convolutional network. In Proceedings of the IEEE International Conference on Computer Vision. 576–584.Google Scholar
Digital Library
- [6] . 2012. Sample adaptive offset in the HEVC standard. IEEE Transactions on Circuits and Systems for Video Technology 22, 12 (2012), 1755–1764.Google Scholar
Digital Library
- [7] . 2016. Adaptive clipping in JEM2.0. JVET-C0040.Google Scholar
- [8] . 2021. Digital retina: A way to make the city brain more efficient by visual coding. IEEE Transactions on Circuits and Systems for Video Technology 31, 11 (2021), 4147–4161.Google Scholar
Digital Library
- [9] . 2010. Guided image filtering. In Proceedings of the European Conference on Computer Vision. Springer, 1–14.Google Scholar
Digital Library
- [10] . 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.Google Scholar
Cross Ref
- [11] . 2018. Enhancing HEVC compressed videos with a partition-masked convolutional neural network. In Proceedings of the 2018 25th IEEE International Conference on Image Processing. IEEE, 216–220.Google Scholar
Cross Ref
- [12] . 2018. Enhanced intra prediction with recurrent neural network in video coding. In Proceedings of the 2018 Data Compression Conference. IEEE, 413–413.Google Scholar
Cross Ref
- [13] . 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4700–4708.Google Scholar
Cross Ref
- [14] . 2018. Convolutional neural network-based motion compensation refinement for video coding. In Proceedings of the 2018 IEEE International Symposium on Circuits and Systems. IEEE, 1–4.Google Scholar
Cross Ref
- [15] . 2019. Content-aware convolutional neural network for in-loop filtering in high efficiency video coding. IEEE Transactions on Image Processing 28, 7 (2019), 3343–3356.Google Scholar
Digital Library
- [16] . 2017. Spatial-temporal residue network based in-loop filter for video coding. In Proceedings of the 2017 IEEE Visual Communications and Image Processing. IEEE, 1–4.Google Scholar
Cross Ref
- [17] . 2021. Residual-guided in-loop filter using convolution neural network. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 4 (2021), 1–19.Google Scholar
Digital Library
- [18] . 2020. An enhanced sample adaptive offset filtering method for AVS3. AVS-M5373.Google Scholar
- [19] . 2016. Geometry transformation-based adaptive in-loop filter. In Proceedings of the 2016 Picture Coding Symposium. IEEE, 1–5.Google Scholar
Cross Ref
- [20] . 2015. Adam: A method for stochastic optimization. In ICLR (Poster).Google Scholar
- [21] . 2018. Deep learning based HEVC in-loop filtering for decoder quality enhancement. In Proceedings of the 2018 Picture Coding Symposium. IEEE, 164–168.Google Scholar
Cross Ref
- [22] . 2019. Adaptive CU mode selection in HEVC intra prediction: A deep learning approach. Circuits, Systems, and Signal Processing 38, 11 (2019), 5081–5102.Google Scholar
Digital Library
- [23] . 2021. Deep learning based HEVC in-loop filter and noise reduction. Signal Processing: Image Communication 99 (2021), 116409.Google Scholar
Cross Ref
- [24] . 2020. Cross-component sample adaptive offset. AVS-M5800.Google Scholar
- [25] . 2019. An in-loop filter based on low-complexity CNN using residuals in intra video coding. In Proceedings of the 2019 IEEE International Symposium on Circuits and Systems. IEEE, 1–5.Google Scholar
Cross Ref
- [26] . 2018. Fully connected network-based intra prediction for image coding. IEEE Transactions on Image Processing 27, 7 (2018), 3236–3247.Google Scholar
Cross Ref
- [27] . 2020. Sub-sampled cross-component prediction for chroma component coding. In Proceedings of the 2020 Data Compression Conference. IEEE, 203–212.Google Scholar
Cross Ref
- [28] . 2019. Residual in residual based convolutional neural network in-loop filter for AVS3. In Proceedings of the 2019 Picture Coding Symposium. IEEE, 1–5.Google Scholar
Cross Ref
- [29] . 2021. BVI-DVC: A training database for deep video compression. IEEE Transactions on Multimedia.Google Scholar
- [30] . 2015. AVS2? Making video coding smarter [standards in a nutshell]. IEEE Signal Processing Magazine 32, 2 (2015), 172–183.Google Scholar
Cross Ref
- [31] . 2019. Image and video compression with neural networks: A review. IEEE Transactions on Circuits and Systems for Video Technology 30, 6 (2019), 1683–1698.Google Scholar
- [32] . 2019. On cross component adaptive loop filter for video compression. In Proceedings of the 2019 Picture Coding Symposium. IEEE, 1–5.Google Scholar
Cross Ref
- [33] . 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning. 807–814.Google Scholar
Digital Library
- [34] . 2012. HEVC deblocking filter. IEEE Transactions on Circuits and Systems for Video Technology 22, 12 (2012), 1746–1754.Google Scholar
Digital Library
- [35] . 2018. In-loop reshaping for SDR video. JVET-K0309.Google Scholar
- [36] . 2018. A practical convolutional neural network as loop filter for intra frame. In Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 1133–1137.Google Scholar
Cross Ref
- [37] . 2012. Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on Circuits and Systems for Video Technology 22, 12 (2012), 1649–1668.Google Scholar
Digital Library
- [38] . 2013. Adaptive loop filtering for video coding. IEEE Journal of Selected Topics in Signal Processing 7, 6 (2013), 934–945.Google Scholar
Cross Ref
- [39] . 2019. Extended coding unit partitioning for future video coding. IEEE Transactions on Image Processing 29, (2019), 2931–2946Google Scholar
- [40] . 2018. Dense residual convolutional neural network based in-loop filter for HEVC. In Proceedings of the 2018 IEEE Visual Communications and Image Processing. IEEE, 1–4.Google Scholar
Cross Ref
- [41] . 2017. Bilateral filtering for video coding. In Proceedings of the 2017 IEEE Visual Communications and Image Processing. IEEE, 1–4.Google Scholar
Cross Ref
- [42] . 2003. Rate-constrained coder control and comparison of video coding standards. IEEE Transactions on Circuits and Systems for Video Technology 13, 7 (2003), 688–703.Google Scholar
Digital Library
- [43] . 2019. Switch mode based deep fractional interpolation in video coding. In Proceedings of the 2019 IEEE International Symposium on Circuits and Systems. IEEE, 1–5.Google Scholar
Cross Ref
- [44] . 2018. Reducing complexity of HEVC: A deep learning approach. IEEE Transactions on Image Processing 27, 10 (2018), 5044–5059.Google Scholar
Cross Ref
- [45] . 2019. Recent development of AVS video coding standard: AVS3. In Proceedings of the 2019 Picture Coding Symposium. IEEE, 1–5.Google Scholar
Cross Ref
- [46] . 2015. Non-local structure-based filter for video coding. In Proceedings of the 2015 IEEE International Symposium on Multimedia. IEEE, 301–306.Google Scholar
Cross Ref
- [47] . 2018. Enhanced cross-component linear model for chroma intra-prediction in video coding. IEEE Transactions on Image Processing 27, 8 (2018), 3983–3997.Google Scholar
Cross Ref
- [48] . 2017. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing 26, 7 (2017), 3142–3155.Google Scholar
Digital Library
- [49] . 2012. Adaptive loop filter with temporal prediction. In Proceedings of the 2012 Picture Coding Symposium. IEEE, 437–440.Google Scholar
Cross Ref
- [50] . 2018. Residual highway convolutional neural networks for in-loop filtering in HEVC. IEEE Transactions on Image Processing 27, 8 (2018), 3827–3841.Google Scholar
Cross Ref
- [51] . 2019. Enhanced motion-compensated video coding with deep virtual reference frame generation. IEEE Transactions on Image Processing 28, 10 (2019), 4832–4844.Google Scholar
Cross Ref
- [52] . 2018. Enhanced bi-prediction with convolutional neural network for high efficiency video coding. IEEE Transactions on Circuits and Systems for Video Technology 29, 11 (2018), 3291–3301.Google Scholar
- [53] . 2020. Residual convolutional neural network based in-loop filter with intra and inter frames processed respectively for Avs3. In Proceedings of the 2020 IEEE International Conference on Multimedia & Expo Workshops. IEEE, 1–6.Google Scholar
Cross Ref
Index Terms
NR-CNN: Nested-Residual Guided CNN In-loop Filtering for Video Coding
Recommendations
Residual-guided In-loop Filter Using Convolution Neural Network
The block-based coding structure in the hybrid video coding framework inevitably introduces compression artifacts such as blocking, ringing, and so on. To compensate for those artifacts, extensive filtering techniques were proposed in the loop of video ...
A nonlocal HEVC in-loop filter using CNN-based compression noise estimation
AbstractHigh-efficiency video coding (HEVC) effectively reduces the amount of video data while unavoidably introducing compression noise. The in-loop filter can enhance the reconstructed frames at the encoder to prevent compression noise from transmitting ...
PTR-CNN for in-loop filtering in video coding
AbstractA deep learning method called PTR-CNN (Predicted frame with Transform unit partition and prediction Residual aided CNN) is proposed for in-loop filtering in video compression. To reduce the computational complexity of an end-to-end CNN ...
Highlights- A non-learning image quality assessment method is used to select a reference frame.






Comments