skip to main content
research-article

Spatio-temporal Saliency-based Motion Vector Refinement for Frame Rate Up-conversion

Published:22 May 2020Publication History
Skip Abstract Section

Abstract

A spatio-temporal saliency-based frame rate up-conversion (FRUC) approach is proposed, which achieves better quality of interpolated frames and invalidates existing texture variation-based FRUC detectors. A spatio-temporal saliency model is designed to select salient frames. After obtaining initial motion vector field by texture- and color-based bilateral motion estimation, two motion vector refining (MVR) schemes are adopted for high and low saliency frames to hierarchically refine the motion vectors, respectively. To produce high-quality interpolated frames, image enhancement are performed for salient frames after frame interpolation. Due to distinct MVR schemes, there are different degrees of texture information in interpolated frames. Some edge and texture information is supplemented into salient frames as post-processing, which can invalidate existing texture variation-based FRUC detectors. Experimental results show that the proposed approach outperforms state-of-the-art works in both objective and subjective qualities of interpolated frames, and achieves the purpose of FRUC anti-forensics.

References

  1. Khosro Bahrami and Alex C. Kot. 2014. A fast approach for no-reference image sharpness assessment based on maximum local variation. IEEE Signal Processing Letters 21, 6 (2014), 751--755. https://ieeexplore.ieee.xilesou.top/abstract/document/6780989/Google ScholarGoogle ScholarCross RefCross Ref
  2. Wenbo Bao, Xiaoyun Zhang, Li Chen, Lianghui Ding, and Zhiyong Gao. 2018. High-order model and dynamic filtering for frame rate up-conversion. IEEE Transactions on Image Processing 27, 8 (2018), 3813--3826. https://ieeexplore.ieee.xilesou.top/abstract/document/8334253/Google ScholarGoogle Scholar
  3. Bryce E. Bayer. 1976. Color imaging array. https://patents.glgoo.top/patent/US3971065/en US Patent 3,971,065.Google ScholarGoogle Scholar
  4. Jenny Benois-Pineau and Henri Nicolas. 2002. A new method for region-based depth ordering in a video sequence: application to frame interpolation. Journal of Visual Communication and Image Representation 13, 3 (2002), 363--385. https://sciencedirect.xilesou.top/science/article/pii/S1047320301904900Google ScholarGoogle ScholarCross RefCross Ref
  5. Byeong-Doo Choi, Jong-Woo Han, Chang-Su Kim, and Sung-Jea Ko. 2007. Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation. IEEE Transactions on Circuits and Systems for Video Technology 17, 4 (2007), 407--416. https://ieeexplore.ieee.xilesou.top/abstract/document/4162523/Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Dooseop Choi, Wonseok Song, Hyuk Choi, and Taejeong Kim. 2016. MAP-based motion refinement algorithm for block-based motion-compensated frame interpolation. IEEE Transactions on Circuits and Systems for Video Technology 26, 10 (2016), 1789--1804. https://ieeexplore.ieee.xilesou.top/abstract/document/7225154Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Xiangling Ding, Yue Li, Ming Xia, Jiale He, and Gaobo Yang. 2019. Detection of motion compensated frame interpolation via motion-aligned temporal difference. Multimedia Tools and Applications 78, 6 (2019), 7453--7477. https://link.springer.xilesou.top/article/10.1007/s11042-018-6504-5Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Xiangling Ding, Gaobo Yang, Ran Li, Lebing Zhang, Yue Li, and Xingming Sun. 2018. Identification of motion-compensated frame rate up-conversion based on residual signals. IEEE Transactions on Circuits and Systems for Video Technology 28, 7 (2018), 1497--1512. https://ieeexplore.ieee.xilesou.top/abstract/document/7869361/Google ScholarGoogle ScholarCross RefCross Ref
  9. Xiangling Ding, Ningbo Zhu, Leida Li, Yue Li, and Gaobo Yang. 2018. Robust localization of interpolated frames by motion-compensated frame-interpolation based on artifact indicated map and Tchebichef moments. IEEE Transactions on Circuits and Systems for Video Technology 29, 7 (2018), 1893--1906. https://ieeexplore.ieee.xilesou.top/abstract/document/8403308/Google ScholarGoogle ScholarCross RefCross Ref
  10. Dashan Gao, Vijay Mahadevan, and Nuno Vasconcelos. 2008. On the plausibility of the discriminant center-surround hypothesis for visual saliency. Journal of Vision 8, 7 (2008), 13--30. https://jov.arvojournals.org/article.aspx?articleid=2193585Google ScholarGoogle ScholarCross RefCross Ref
  11. Yong Guo, Li Chen, Zhiyong Gao, and Xiaoyun Zhang. 2016. Frame rate up-conversion using linear quadratic motion estimation and trilateral filtering motion smoothing. Journal of Display Technology 12, 1 (2016), 89--98. https://www.osapublishing.org/abstract.cfm?uri=jdt-12-1-89Google ScholarGoogle Scholar
  12. Jiale He, Gaobo Yang, Jingyu Song, Xiangling Ding, and Ran Li. 2018. Hierarchical prediction-based motion vector refinement for video frame-rate up-conversion. Journal of Real-Time Image Processing (2018), 1--15. https://link.springer.xilesou.top/article/10.1007/s11554-018-0767-yGoogle ScholarGoogle Scholar
  13. Seong-Gyun Jeong, Chul Lee, and Chang-Su Kim. 2012. Exemplar-based frame rate up-conversion with congruent segmentation. In 2012 19th IEEE International Conference on Image Processing. IEEE, 845--848. https://ieeexplore.ieee.xilesou.top/abstract/document/6466992/Google ScholarGoogle ScholarCross RefCross Ref
  14. Seong-Gyun Jeong, Chul Lee, and Chang-Su Kim. 2013. Motion-compensated frame interpolation based on multihypothesis motion estimation and texture optimization. IEEE Transactions on Image Processing 22, 11 (2013), 4497--4509. https://ieeexplore.ieee.xilesou.top/abstract/document/6567908/Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Suk-Ju Kang, Kyoung-Rok Cho, and Young Hwan Kim. 2007. Motion compensated frame rate up-conversion using extended bilateral motion estimation. IEEE Transactions on Consumer Electronics 53, 4 (2007), 1759--1767. https://ieeexplore.ieee.xilesou.top/abstract/document/4429281/Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Suk-Ju Kang, Sungjoo Yoo, and Young Hwan Kim. 2010. Dual motion estimation for frame rate up-conversion. IEEE Transactions on Circuits and Systems for Video Technology 20, 12 (2010), 1909--1914. https://ieeexplore.ieee.xilesou.top/abstract/document/5604667/Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Xiangui Kang, Jingxian Liu, Hongmei Liu, and Z. Jane Wang. 2016. Forensics and counter anti-forensics of video inter-frame forgery. Multimedia Tools and Applications 75, 21 (2016), 13833--13853. https://link.springer.xilesou.top/article/10.1007/s11042-015-2762-7Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Un Seob Kim and Myung Hoon Sunwoo. 2014. New frame rate up-conversion algorithms with low computational complexity. IEEE Transactions on Circuits and Systems for Video Technology 24, 3 (2014), 384--393. https://ieeexplore.ieee.xilesou.top/abstract/document/6578124/Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Won Hee Lee, Kyuha Choi, and Jong Beom Ra. 2014. Frame rate up conversion based on variational image fusion. IEEE Transactions on Image Processing 23, 1 (2014), 399--412. https://ieeexplore.ieee.xilesou.top/abstract/document/6651823/Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Ran Li, Hongbing Liu, Jie Chen, and Zongliang Gan. 2016. Wavelet pyramid based multi-resolution bilateral motion estimation for frame rate up-conversion. IEICE Transactions on Information and Systems 99, 1 (2016), 208--218. https://www.jstage.jst.go.jp/article/transinf/E99.D/1/E99.D_2015EDP7027/_article/-char/ja/Google ScholarGoogle ScholarCross RefCross Ref
  21. Ran Li, Hongbing Liu, Zhenghui Liu, Yanling Li, and Zhangjie Fu. 2017. Motion-compensated frame interpolation using patch-based sparseland model. Signal Processing: Image Communication 54 (2017), 36--48. https://sciencedirect.xilesou.top/science/article/pii/S0923596517300267Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Ran Li, Yongfeng Lv, and Zhenghui Liu. 2018. Multi-scheme frame rate up-conversion using space-time saliency. IEEE Access 6, 99 (2018), 1905--1915. https://ieeexplore.ieee.xilesou.top/abstract/document/8169029/Google ScholarGoogle ScholarCross RefCross Ref
  23. Yanli Li, Wendan Ma, and Yue Han. 2019. A spatial prediction-based motion-compensated frame rate up-conversion. Future Internet 11, 2 (2019), 26--35. https://www.mdpi.xilesou.top/1999-5903/11/2/26Google ScholarGoogle ScholarCross RefCross Ref
  24. Yue Li, Gaobo Yang, Yapei Zhu, Xiangling Ding, and Xingming Sun. 2017. Adaptive inter CU depth decision for HEVC using optimal selection model and encoding parameters. IEEE Transactions on Broadcasting 63, 3 (2017), 535--546. https://ieeexplore.ieee.xilesou.top/abstract/document/7940104/Google ScholarGoogle ScholarCross RefCross Ref
  25. Yue Li, Gaobo Yang, Yapei Zhu, Xiangling Ding, and Xingming Sun. 2017. Unimodal stopping model-based early SKIP mode decision for high-efficiency video coding. IEEE Transactions on Multimedia 19, 7 (2017), 1431--1441. https://ieeexplore.ieee.xilesou.top/abstract/document/7857045/Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Hongbin Liu, Ruiqin Xiong, Debin Zhao, Siwei Ma, and Wen Gao. 2012. Multiple hypotheses Bayesian frame rate up-conversion by adaptive fusion of motion-compensated interpolations. IEEE Transactions on Circuits and Systems for Video Technology 22, 8 (2012), 1188--1198. https://ieeexplore.ieee.xilesou.top/abstract/document/6193418/Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Jacobson Natan, Yenlin Lee, and Mahadevan Vijay. 2010. Motion vector refinement for FRUC using saliency and segmentation. In Proceedings of the 11th IEEE International Conference on Multimedia and Expo. IEEE, 778--783. https://ieeexplore.ieee.xilesou.top/abstract/document/5582574/Google ScholarGoogle Scholar
  28. Jacobson Natan and Nguyen Truong Q. 2012. Scale-aware saliency for application to frame rate upconversion. IEEE Transactions on Image Processing 21, 4 (2012), 2198--2206. https://ieeexplore.ieee.xilesou.top/abstract/document/6099617/Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Michael T. Orchard and Gary J. Sullivan. 1994. Overlapped block motion compensation: An estimation-theoretic approach. IEEE Transactions on Image Processing 3, 5 (1994), 693--699. https://ieeexplore.ieee.xilesou.top/abstract/document/334974/Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Aixi Qu, Ju Liu, Wenbo Wan, and Yifan Xiao. 2016. A frame rate up-conversion method with quadruple motion vector post-processing. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1686--1690. https://ieeexplore.ieee.xilesou.top/abstract/document/7471964/Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Milan Sonka, Vaclav Hlavac, and Roger Boyle. 2014. Image Processing, Analysis, and Machine Vision. Cengage Learning.Google ScholarGoogle Scholar
  32. Matthew C. Stamm, W. Sabrina Lin, and K. J. Ray Liu. 2012. Temporal forensics and anti-forensics for motion compensated video. IEEE Transactions on Information Forensics and Security 7, 4 (2012), 1315--1329. https://ieeexplore.ieee.xilesou.top/abstract/document/6222325/Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Tsung-Han Tsai and Hsueh-Yi Lin. 2012. High visual quality particle based frame rate up conversion with acceleration assisted motion trajectory calibration. Journal of Display Technology 8, 6 (2012), 341--351. https://www.osapublishing.org/jdt/abstract.cfm?uri=jdt-8-6-341Google ScholarGoogle ScholarCross RefCross Ref
  34. Tsung-Han Tsai, An-Ting Shi, and Ko-Ting Huang. 2016. Accurate frame rate up-conversion for advanced visual quality. IEEE Transactions on Broadcasting 62, 2 (2016), 426--435. https://ieeexplore.ieee.xilesou.top/abstract/document/7464837/Google ScholarGoogle ScholarCross RefCross Ref
  35. Ci Wang, Lei Zhang, Yuwen He, and Yap-Peng Tan. 2010. Frame rate up-conversion using trilateral filtering. IEEE Transactions on Circuits and Systems for Video Technology 20, 6 (2010), 886--893. https://ieeexplore.ieee.xilesou.top/abstract/document/5433045/Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Min Xia, Gaobo Yang, Leida Li, Ran Li, and Xingming Sun. 2017. Detecting video frame rate up-conversion based on frame-level analysis of average texture variation. Multimedia Tools and Applications 76, 6 (2017), 8399--8421. https://link.springer.xilesou.top/article/10.1007/s11042-016-3468-1Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Yuxuan Yao, Gaobo Yang, Xingming Sun, and Leida Li. 2016. Detecting video frame-rate up-conversion based on periodic properties of edge-intensity. Journal of Information Security and Applications 26 (2016), 39--50. https://sciencedirect.xilesou.top/science/article/pii/S2214212615000691Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Dong-Gon Yoo, Suk-Ju Kang, and Young Hwan Kim. 2013. Direction-select motion estimation for motion-compensated frame rate up-conversion. Journal of Display Technology 9, 10 (2013), 840--850. https://www.osapublishing.org/jdt/abstract.cfm?uri=jdt-9-10-840Google ScholarGoogle ScholarCross RefCross Ref
  39. Sung-Jun Yoon, Hyun-Ho Kim, and Munchurl Kim. 2018. Hierarchical extended bilateral motion estimation-based frame rate upconversion using learning-based linear mapping. IEEE Transactions on Image Processing 27, 12 (2018), 5918--5932. https://ieeexplore.ieee.xilesou.top/abstract/document/8423719/Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Yun Zhai and Mubarak Shah. 2006. Visual attention detection in video sequences using spatiotemporal cues. In Proceedings of the 14th ACM International Conference on Multimedia. ACM, 815--824. https://dl.acm.org/doi/abs/10.1145/1180639.1180824Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Hanling Zhang, Chenxing Xia, and Xiuju Gao. 2017. Robust saliency detection via corner information and an energy function. IET Computer Vision 11, 6 (2017), 379--388. https://digital-library.theiet.org/content/journals/10.1049/iet-cvi.2016.0492Google ScholarGoogle ScholarCross RefCross Ref
  42. Lebing Zhang, Fei Peng, Le Qin, and Min Long. 2018. Face spoofing detection based on color texture Markov feature and support vector machine recursive feature elimination. Journal of Visual Communication and Image Representation 51 (2018), 56--59. https://sciencedirect.xilesou.top/science/article/pii/S1047320318300014Google ScholarGoogle ScholarCross RefCross Ref
  43. Yongbing Zhang, Debin Zhao, Siwei Ma, Ronggang Wang, and Wen Gao. 2010. A motion-aligned auto-regressive model for frame rate up conversion. IEEE Transactions on Image Processing 19, 5 (2010), 1248--1258. https://ieeexplore.ieee.xilesou.top/abstract/document/5357389/Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Spatio-temporal Saliency-based Motion Vector Refinement for Frame Rate Up-conversion

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!