Abstract
A spatio-temporal saliency-based frame rate up-conversion (FRUC) approach is proposed, which achieves better quality of interpolated frames and invalidates existing texture variation-based FRUC detectors. A spatio-temporal saliency model is designed to select salient frames. After obtaining initial motion vector field by texture- and color-based bilateral motion estimation, two motion vector refining (MVR) schemes are adopted for high and low saliency frames to hierarchically refine the motion vectors, respectively. To produce high-quality interpolated frames, image enhancement are performed for salient frames after frame interpolation. Due to distinct MVR schemes, there are different degrees of texture information in interpolated frames. Some edge and texture information is supplemented into salient frames as post-processing, which can invalidate existing texture variation-based FRUC detectors. Experimental results show that the proposed approach outperforms state-of-the-art works in both objective and subjective qualities of interpolated frames, and achieves the purpose of FRUC anti-forensics.
- Khosro Bahrami and Alex C. Kot. 2014. A fast approach for no-reference image sharpness assessment based on maximum local variation. IEEE Signal Processing Letters 21, 6 (2014), 751--755. https://ieeexplore.ieee.xilesou.top/abstract/document/6780989/Google Scholar
Cross Ref
- Wenbo Bao, Xiaoyun Zhang, Li Chen, Lianghui Ding, and Zhiyong Gao. 2018. High-order model and dynamic filtering for frame rate up-conversion. IEEE Transactions on Image Processing 27, 8 (2018), 3813--3826. https://ieeexplore.ieee.xilesou.top/abstract/document/8334253/Google Scholar
- Bryce E. Bayer. 1976. Color imaging array. https://patents.glgoo.top/patent/US3971065/en US Patent 3,971,065.Google Scholar
- Jenny Benois-Pineau and Henri Nicolas. 2002. A new method for region-based depth ordering in a video sequence: application to frame interpolation. Journal of Visual Communication and Image Representation 13, 3 (2002), 363--385. https://sciencedirect.xilesou.top/science/article/pii/S1047320301904900Google Scholar
Cross Ref
- Byeong-Doo Choi, Jong-Woo Han, Chang-Su Kim, and Sung-Jea Ko. 2007. Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation. IEEE Transactions on Circuits and Systems for Video Technology 17, 4 (2007), 407--416. https://ieeexplore.ieee.xilesou.top/abstract/document/4162523/Google Scholar
Digital Library
- Dooseop Choi, Wonseok Song, Hyuk Choi, and Taejeong Kim. 2016. MAP-based motion refinement algorithm for block-based motion-compensated frame interpolation. IEEE Transactions on Circuits and Systems for Video Technology 26, 10 (2016), 1789--1804. https://ieeexplore.ieee.xilesou.top/abstract/document/7225154Google Scholar
Digital Library
- Xiangling Ding, Yue Li, Ming Xia, Jiale He, and Gaobo Yang. 2019. Detection of motion compensated frame interpolation via motion-aligned temporal difference. Multimedia Tools and Applications 78, 6 (2019), 7453--7477. https://link.springer.xilesou.top/article/10.1007/s11042-018-6504-5Google Scholar
Digital Library
- Xiangling Ding, Gaobo Yang, Ran Li, Lebing Zhang, Yue Li, and Xingming Sun. 2018. Identification of motion-compensated frame rate up-conversion based on residual signals. IEEE Transactions on Circuits and Systems for Video Technology 28, 7 (2018), 1497--1512. https://ieeexplore.ieee.xilesou.top/abstract/document/7869361/Google Scholar
Cross Ref
- Xiangling Ding, Ningbo Zhu, Leida Li, Yue Li, and Gaobo Yang. 2018. Robust localization of interpolated frames by motion-compensated frame-interpolation based on artifact indicated map and Tchebichef moments. IEEE Transactions on Circuits and Systems for Video Technology 29, 7 (2018), 1893--1906. https://ieeexplore.ieee.xilesou.top/abstract/document/8403308/Google Scholar
Cross Ref
- Dashan Gao, Vijay Mahadevan, and Nuno Vasconcelos. 2008. On the plausibility of the discriminant center-surround hypothesis for visual saliency. Journal of Vision 8, 7 (2008), 13--30. https://jov.arvojournals.org/article.aspx?articleid=2193585Google Scholar
Cross Ref
- Yong Guo, Li Chen, Zhiyong Gao, and Xiaoyun Zhang. 2016. Frame rate up-conversion using linear quadratic motion estimation and trilateral filtering motion smoothing. Journal of Display Technology 12, 1 (2016), 89--98. https://www.osapublishing.org/abstract.cfm?uri=jdt-12-1-89Google Scholar
- Jiale He, Gaobo Yang, Jingyu Song, Xiangling Ding, and Ran Li. 2018. Hierarchical prediction-based motion vector refinement for video frame-rate up-conversion. Journal of Real-Time Image Processing (2018), 1--15. https://link.springer.xilesou.top/article/10.1007/s11554-018-0767-yGoogle Scholar
- Seong-Gyun Jeong, Chul Lee, and Chang-Su Kim. 2012. Exemplar-based frame rate up-conversion with congruent segmentation. In 2012 19th IEEE International Conference on Image Processing. IEEE, 845--848. https://ieeexplore.ieee.xilesou.top/abstract/document/6466992/Google Scholar
Cross Ref
- Seong-Gyun Jeong, Chul Lee, and Chang-Su Kim. 2013. Motion-compensated frame interpolation based on multihypothesis motion estimation and texture optimization. IEEE Transactions on Image Processing 22, 11 (2013), 4497--4509. https://ieeexplore.ieee.xilesou.top/abstract/document/6567908/Google Scholar
Digital Library
- Suk-Ju Kang, Kyoung-Rok Cho, and Young Hwan Kim. 2007. Motion compensated frame rate up-conversion using extended bilateral motion estimation. IEEE Transactions on Consumer Electronics 53, 4 (2007), 1759--1767. https://ieeexplore.ieee.xilesou.top/abstract/document/4429281/Google Scholar
Digital Library
- Suk-Ju Kang, Sungjoo Yoo, and Young Hwan Kim. 2010. Dual motion estimation for frame rate up-conversion. IEEE Transactions on Circuits and Systems for Video Technology 20, 12 (2010), 1909--1914. https://ieeexplore.ieee.xilesou.top/abstract/document/5604667/Google Scholar
Digital Library
- Xiangui Kang, Jingxian Liu, Hongmei Liu, and Z. Jane Wang. 2016. Forensics and counter anti-forensics of video inter-frame forgery. Multimedia Tools and Applications 75, 21 (2016), 13833--13853. https://link.springer.xilesou.top/article/10.1007/s11042-015-2762-7Google Scholar
Digital Library
- Un Seob Kim and Myung Hoon Sunwoo. 2014. New frame rate up-conversion algorithms with low computational complexity. IEEE Transactions on Circuits and Systems for Video Technology 24, 3 (2014), 384--393. https://ieeexplore.ieee.xilesou.top/abstract/document/6578124/Google Scholar
Digital Library
- Won Hee Lee, Kyuha Choi, and Jong Beom Ra. 2014. Frame rate up conversion based on variational image fusion. IEEE Transactions on Image Processing 23, 1 (2014), 399--412. https://ieeexplore.ieee.xilesou.top/abstract/document/6651823/Google Scholar
Digital Library
- Ran Li, Hongbing Liu, Jie Chen, and Zongliang Gan. 2016. Wavelet pyramid based multi-resolution bilateral motion estimation for frame rate up-conversion. IEICE Transactions on Information and Systems 99, 1 (2016), 208--218. https://www.jstage.jst.go.jp/article/transinf/E99.D/1/E99.D_2015EDP7027/_article/-char/ja/Google Scholar
Cross Ref
- Ran Li, Hongbing Liu, Zhenghui Liu, Yanling Li, and Zhangjie Fu. 2017. Motion-compensated frame interpolation using patch-based sparseland model. Signal Processing: Image Communication 54 (2017), 36--48. https://sciencedirect.xilesou.top/science/article/pii/S0923596517300267Google Scholar
Digital Library
- Ran Li, Yongfeng Lv, and Zhenghui Liu. 2018. Multi-scheme frame rate up-conversion using space-time saliency. IEEE Access 6, 99 (2018), 1905--1915. https://ieeexplore.ieee.xilesou.top/abstract/document/8169029/Google Scholar
Cross Ref
- Yanli Li, Wendan Ma, and Yue Han. 2019. A spatial prediction-based motion-compensated frame rate up-conversion. Future Internet 11, 2 (2019), 26--35. https://www.mdpi.xilesou.top/1999-5903/11/2/26Google Scholar
Cross Ref
- Yue Li, Gaobo Yang, Yapei Zhu, Xiangling Ding, and Xingming Sun. 2017. Adaptive inter CU depth decision for HEVC using optimal selection model and encoding parameters. IEEE Transactions on Broadcasting 63, 3 (2017), 535--546. https://ieeexplore.ieee.xilesou.top/abstract/document/7940104/Google Scholar
Cross Ref
- Yue Li, Gaobo Yang, Yapei Zhu, Xiangling Ding, and Xingming Sun. 2017. Unimodal stopping model-based early SKIP mode decision for high-efficiency video coding. IEEE Transactions on Multimedia 19, 7 (2017), 1431--1441. https://ieeexplore.ieee.xilesou.top/abstract/document/7857045/Google Scholar
Digital Library
- Hongbin Liu, Ruiqin Xiong, Debin Zhao, Siwei Ma, and Wen Gao. 2012. Multiple hypotheses Bayesian frame rate up-conversion by adaptive fusion of motion-compensated interpolations. IEEE Transactions on Circuits and Systems for Video Technology 22, 8 (2012), 1188--1198. https://ieeexplore.ieee.xilesou.top/abstract/document/6193418/Google Scholar
Digital Library
- Jacobson Natan, Yenlin Lee, and Mahadevan Vijay. 2010. Motion vector refinement for FRUC using saliency and segmentation. In Proceedings of the 11th IEEE International Conference on Multimedia and Expo. IEEE, 778--783. https://ieeexplore.ieee.xilesou.top/abstract/document/5582574/Google Scholar
- Jacobson Natan and Nguyen Truong Q. 2012. Scale-aware saliency for application to frame rate upconversion. IEEE Transactions on Image Processing 21, 4 (2012), 2198--2206. https://ieeexplore.ieee.xilesou.top/abstract/document/6099617/Google Scholar
Digital Library
- Michael T. Orchard and Gary J. Sullivan. 1994. Overlapped block motion compensation: An estimation-theoretic approach. IEEE Transactions on Image Processing 3, 5 (1994), 693--699. https://ieeexplore.ieee.xilesou.top/abstract/document/334974/Google Scholar
Digital Library
- Aixi Qu, Ju Liu, Wenbo Wan, and Yifan Xiao. 2016. A frame rate up-conversion method with quadruple motion vector post-processing. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1686--1690. https://ieeexplore.ieee.xilesou.top/abstract/document/7471964/Google Scholar
Digital Library
- Milan Sonka, Vaclav Hlavac, and Roger Boyle. 2014. Image Processing, Analysis, and Machine Vision. Cengage Learning.Google Scholar
- Matthew C. Stamm, W. Sabrina Lin, and K. J. Ray Liu. 2012. Temporal forensics and anti-forensics for motion compensated video. IEEE Transactions on Information Forensics and Security 7, 4 (2012), 1315--1329. https://ieeexplore.ieee.xilesou.top/abstract/document/6222325/Google Scholar
Digital Library
- Tsung-Han Tsai and Hsueh-Yi Lin. 2012. High visual quality particle based frame rate up conversion with acceleration assisted motion trajectory calibration. Journal of Display Technology 8, 6 (2012), 341--351. https://www.osapublishing.org/jdt/abstract.cfm?uri=jdt-8-6-341Google Scholar
Cross Ref
- Tsung-Han Tsai, An-Ting Shi, and Ko-Ting Huang. 2016. Accurate frame rate up-conversion for advanced visual quality. IEEE Transactions on Broadcasting 62, 2 (2016), 426--435. https://ieeexplore.ieee.xilesou.top/abstract/document/7464837/Google Scholar
Cross Ref
- Ci Wang, Lei Zhang, Yuwen He, and Yap-Peng Tan. 2010. Frame rate up-conversion using trilateral filtering. IEEE Transactions on Circuits and Systems for Video Technology 20, 6 (2010), 886--893. https://ieeexplore.ieee.xilesou.top/abstract/document/5433045/Google Scholar
Digital Library
- Min Xia, Gaobo Yang, Leida Li, Ran Li, and Xingming Sun. 2017. Detecting video frame rate up-conversion based on frame-level analysis of average texture variation. Multimedia Tools and Applications 76, 6 (2017), 8399--8421. https://link.springer.xilesou.top/article/10.1007/s11042-016-3468-1Google Scholar
Digital Library
- Yuxuan Yao, Gaobo Yang, Xingming Sun, and Leida Li. 2016. Detecting video frame-rate up-conversion based on periodic properties of edge-intensity. Journal of Information Security and Applications 26 (2016), 39--50. https://sciencedirect.xilesou.top/science/article/pii/S2214212615000691Google Scholar
Digital Library
- Dong-Gon Yoo, Suk-Ju Kang, and Young Hwan Kim. 2013. Direction-select motion estimation for motion-compensated frame rate up-conversion. Journal of Display Technology 9, 10 (2013), 840--850. https://www.osapublishing.org/jdt/abstract.cfm?uri=jdt-9-10-840Google Scholar
Cross Ref
- Sung-Jun Yoon, Hyun-Ho Kim, and Munchurl Kim. 2018. Hierarchical extended bilateral motion estimation-based frame rate upconversion using learning-based linear mapping. IEEE Transactions on Image Processing 27, 12 (2018), 5918--5932. https://ieeexplore.ieee.xilesou.top/abstract/document/8423719/Google Scholar
Digital Library
- Yun Zhai and Mubarak Shah. 2006. Visual attention detection in video sequences using spatiotemporal cues. In Proceedings of the 14th ACM International Conference on Multimedia. ACM, 815--824. https://dl.acm.org/doi/abs/10.1145/1180639.1180824Google Scholar
Digital Library
- Hanling Zhang, Chenxing Xia, and Xiuju Gao. 2017. Robust saliency detection via corner information and an energy function. IET Computer Vision 11, 6 (2017), 379--388. https://digital-library.theiet.org/content/journals/10.1049/iet-cvi.2016.0492Google Scholar
Cross Ref
- Lebing Zhang, Fei Peng, Le Qin, and Min Long. 2018. Face spoofing detection based on color texture Markov feature and support vector machine recursive feature elimination. Journal of Visual Communication and Image Representation 51 (2018), 56--59. https://sciencedirect.xilesou.top/science/article/pii/S1047320318300014Google Scholar
Cross Ref
- Yongbing Zhang, Debin Zhao, Siwei Ma, Ronggang Wang, and Wen Gao. 2010. A motion-aligned auto-regressive model for frame rate up conversion. IEEE Transactions on Image Processing 19, 5 (2010), 1248--1258. https://ieeexplore.ieee.xilesou.top/abstract/document/5357389/Google Scholar
Digital Library
Index Terms
Spatio-temporal Saliency-based Motion Vector Refinement for Frame Rate Up-conversion
Recommendations
Adaptive frame rate up-conversion based on motion classification
In this paper, a new technique on video frame rate up-conversion (FRUC) is presented by combining the adaptive motion classification (AMC) for image sequences and the mixed motion estimation (ME). In the proposed FRUC scheme, the AMC classifies ...
Multiframe-based bilateral motion estimation with emphasis on stationary caption processing for frame rate up-conversion
In this paper, we present a new motion compensated frame rate up-conversion algorithm that uses multiframes to enhance the accuracy of motion estimation. We also develop adaptive motion vector smoothing to correct outliers in a motion vector field. In ...
Region-based motion-compensated frame rate up-conversion by homography parameter interpolation
ICIP'09: Proceedings of the 16th IEEE international conference on Image processingA new region-based frame interpolation algorithm is proposed based on the segmented motion layers with planar perspective motion models. It is shown that performing the motion model interpolation in the homography parameter space is equivalent to ...






Comments