Abstract
In the 6G network, lots of edge devices facilitate the low-latency transmission of video. However, with limited processing and storage capabilities, the edge devices cannot afford to reconstruct the vast amount of video data. On the condition of edge computing in the 6G network, this article fuses a self-similarity-based context feature into Frame Rate Up-Conversion (FRUC) to generate the pseudo-true video sequences at high frame rate, and its core is the extraction of the context layer for each video frame. First, we extract the patch centered at each pixel and use the self-similarity descriptor to generate the correlation surface. Then, the expectation or skewness of the correlation surface in statistics is computed to represent its context feature. By attaching an expectation or a skewness to each pixel, the context layer is constructed and added to the video frame as a new channel. According to the context layer, we predict the motion vector field of the absent frame by using the bidirectional context match and finally produce the interpolated frame. From the experimental results, it can be seen that by deploying the proposed FRUC algorithm on edge devices, the output pseudo-true video sequences have satisfying objective and subjective qualities.
- [1] . 2007. Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation. IEEE Transactions on Circuits and Systems for Video Technology 17, 4 (2007), 407–416.
DOI: Google ScholarDigital Library
- [2] . 2017. A new motion estimation method for motion-compensated frame interpolation using a convolutional neural network. In 2017 IEEE International Conference on Image Processing (ICIP’17). 800–804.
DOI: Google ScholarDigital Library
- [3] . 2019. Triple-frame-based bi-directional motion estimation for motion-compensated frame interpolation. IEEE Transactions on Circuits and Systems for Video Technology 29, 5 (2019), 1251–1258.
DOI: Google ScholarDigital Library
- [4] . 2013. Novel true-motion estimation algorithm and its application to motion-compensated temporal frame interpolation. IEEE Transactions on Image Processing 22, 8 (2013), 2931–2945.
DOI: Google ScholarCross Ref
- [5] . 2019. Robust localization of interpolated frames by motion-compensated frame interpolation based on an artifact indicated map and Tchebichef moments. IEEE Transactions on Circuits and Systems for Video Technology 29, 7 (2019), 1893–1906.
DOI: Google ScholarCross Ref
- [6] . 2016. Frame rate up-conversion using linear quadratic motion estimation and trilateral filtering motion smoothing. Journal of Display Technology 12, 1 (2016), 89–98.
DOI: Google ScholarCross Ref
- [7] . 2020. Spatio-temporal saliency-based motion vector refinement for frame rate up-conversion. ACM Trans. Multimedia Comput. Commun. Appl. 16, 2, Article
55 (May 2020), 18 pages.DOI: Google ScholarDigital Library
- [8] . 2021. MV2Flow: Learning motion representation for fast compressed video action recognition. ACM Trans. Multimedia Comput. Commun. Appl. 16, 3s, Article
102 (Dec. 2021), 19 pages.DOI: Google ScholarDigital Library
- [9] . 2022. Deep inter prediction with error-corrected auto-regressive network for video coding. ACM Trans. Multimedia Comput. Commun. Appl. (
March 2022).DOI: Just Accepted .Google ScholarDigital Library
- [10] . 2010. A novel approach to fruc using discriminant saliency and frame segmentation. IEEE Transactions on Image Processing 19, 11 (2010), 2924–2934.
DOI: Google ScholarDigital Library
- [11] . 2021. Run your visual-inertial odometry on NVIDIA Jetson: Benchmark tests on a micro aerial vehicle. IEEE Robotics and Automation Letters 6, 3 (2021), 5332–5339.
DOI: Google ScholarCross Ref
- [12] . 2013. Motion-compensated frame interpolation based on multihypothesis motion estimation and texture optimization. IEEE Transactions on Image Processing 22, 11 (2013), 4497–4509.
DOI: Google ScholarDigital Library
- [13] . 2021. RaspMI: Raspberry pi assisted embedded system for monitoring and recording of seismic ambient noise. IEEE Sensors Journal 21, 5 (2021), 6306–6313.
DOI: Google ScholarCross Ref
- [14] . 2013. Iterative true motion estimation for motion-compensated frame interpolation. IEEE Transactions on Circuits and Systems for Video Technology 23, 3 (2013), 445–454.
DOI: Google ScholarDigital Library
- [15] . 2014. Bi-directional trajectory tracking with variable block-size motion estimation for frame rate up-convertor. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 4, 1 (2014), 29–42.
DOI: Google ScholarCross Ref
- [16] . 2016. Bilateral frame rate up-conversion algorithm based on the comparison of texture complexity. Electronics Letters 52, 5 (2016), 354–355.
DOI: arXiv:https://ietresearch.onlinelibrary. wiley.com/doi/pdf/10.1049/el.2015.3612 Google ScholarCross Ref
- [17] . 2022. MA-NET: Multi-scale attention-aware network for optical flow estimation. In 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’22). 2844–2848.
DOI: Google ScholarCross Ref
- [18] . 2020. Toward edge intelligence: Multiaccess edge computing for 5G and internet of things. IEEE Internet of Things Journal 7, 8 (2020), 6722–6747.
DOI: Google ScholarCross Ref
- [19] . 2016. Motion-compensated frame interpolation with multiframe-based occlusion handling. Journal of Display Technology 12, 1 (2016), 45–54.
DOI: Google ScholarCross Ref
- [20] . 2018. Adaptive fractional-pixel motion estimation skipped algorithm for efficient HEVC motion estimation. ACM Trans. Multimedia Comput. Commun. Appl. 14, 1, Article
12 (Jan. 2018), 19 pages.DOI: Google ScholarDigital Library
- [21] . 2019. Deep learning-based luma and chroma fractional interpolation in video coding. IEEE Access 7 (2019), 112535–112543.
DOI: Google ScholarCross Ref
- [22] . 2020. Federated learning meets blockchain at 6G edge: A drone-assisted networking for disaster response. In Proceedings of the 2nd ACM MobiCom Workshop on Drone Assisted Wireless Communications for 5G and Beyond (DroneCom’20). Association for Computing Machinery, New York, NY, 49–54.
DOI: Google ScholarDigital Library
- [23] . 2016. Con-patch: When a patch meets its context. IEEE Transactions on Image Processing 25, 9 (2016), 3967–3978.
DOI: Google ScholarDigital Library
- [24] . 2007. Matching local self-similarities across images and videos. In 2007 IEEE Conference on Computer Vision and Pattern Recognition. 1–8.
DOI: Google ScholarCross Ref
- [25] . 2018. Motion compensated frame interpolation of occlusion and motion ambiguity regions using color-plus-depth information. In 2018 25th IEEE International Conference on Image Processing (ICIP’18). 1478–1482.
DOI: Google ScholarCross Ref
- [26] . 2020. Adaptive temporal frame interpolation algorithm for frame rate up-conversion. IEEE Consumer Electronics Magazine 9, 3 (2020), 17–21.
DOI: Google ScholarCross Ref
- [27] . 2021. Hedonic pricing of cloud computing services. IEEE Transactions on Cloud Computing 9, 1 (2021), 182–196.
DOI: Google ScholarCross Ref
- [28] . 2018. Hierarchical extended bilateral motion estimation-based frame rate upconversion using learning-based linear mapping. IEEE Transactions on Image Processing 27, 12 (2018), 5918–5932.
DOI: Google ScholarDigital Library
- [29] . 2020. Refined TV- L1 optical flow estimation using joint filtering. IEEE Transactions on Multimedia 22, 2 (2020), 349–364.
DOI: Google ScholarDigital Library
- [30] . 2020. Weighted convolutional motion-compensated frame rate up-conversion using deep residual network. IEEE Transactions on Circuits and Systems for Video Technology 30, 1 (2020), 11–22.
DOI: Google ScholarCross Ref
- [31] . 2010. A motion-aligned auto-regressive model for frame rate up conversion. IEEE Transactions on Image Processing 19, 5 (2010), 1248–1258.
DOI: Google ScholarDigital Library
- [32] . 2019. Frame rate up-conversion based on edge information. In 2019 7th International Conference on Information, Communication and Networks (ICICN’19). 158–162.
DOI: Google ScholarCross Ref
Index Terms
Context-aware Pseudo-true Video Interpolation at 6G Edge
Recommendations
Quality improvement of motion-compensated frame interpolation by self-similarity based context feature
AbstractBlock Matching Algorithm (BMA) is the core of Motion-Compensated Frame Interpolation (MCFI), and its accuracy greatly affects the interpolation quality of MCFI. To improve BMA accuracy, this paper proposes the use of a self-similarity based ...
A Frame Rate Up-Conversion Algorithm for 3-D Video
UIC-ATC '12: Proceedings of the 2012 9th International Conference on Ubiquitous Intelligence and Computing and 9th International Conference on Autonomic and Trusted ComputingIn this paper, we present an improved multi-pass true motion estimation algorithm for frame rate up-conversion. In the proposed motion estimation algorithm, the motion vectors of different objects which are parted with depth information will be refined ...
Quality Enhancement of Frame Rate Up-Converted Video by Adaptive Frame Skip and Reliable Motion Extraction
Frame rate up-conversion is a postprocessing tool to convert the frame rate from a lower number to a higher one. It is a useful technique for a lot of practical applications, such as display format conversion, low bit rate video coding and slow motion ...






Comments