Abstract
Video Frame Interpolation (VFI) is a fascinating and challenging problem in the computer vision (CV) field, aiming to generate non-existing frames between two consecutive video frames. In recent years, many algorithms based on optical flow, kernel, or phase information have been proposed. In this article, we provide a comprehensive review of recent developments in the VFI technique. We first introduce the history of VFI algorithms’ development, the evaluation metrics, and publicly available datasets. We then compare each algorithm in detail, point out their advantages and disadvantages, and compare their interpolation performance and speed on different remarkable datasets. VFI technology has drawn continuous attention in the CV community, some video processing applications based on VFI are also mentioned in this survey, such as slow-motion generation, video compression, video restoration. Finally, we outline the bottleneck faced by the current video frame interpolation technology and discuss future research work.
- [1] . 2019. A fast 4K video frame interpolation using a hybrid task-based convolutional neural network. Symmetry 11, 5 (2019), 619.Google Scholar
Cross Ref
- [2] . 2021. Motion-blurred video interpolation and extrapolation. In Proceedings of the AAAI Conference on Artificial Intelligence.Google Scholar
Cross Ref
- [3] . 2011. A database and evaluation methodology for optical flow. International Journal of Computer Vision 92, 1 (2011), 1–31.Google Scholar
Digital Library
- [4] . 2019. Depth-aware video frame interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3703–3712.Google Scholar
Cross Ref
- [5] . 2019. Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019).Google Scholar
- [6] . 2018. High-order model and dynamic filtering for frame rate up-conversion. IEEE Transactions on Image Processing 27, 8 (2018), 3813–3826.Google Scholar
Cross Ref
- [7] . 2019. Deep frame interpolation for video compression. In Proceedings of the DCC 2019-Data Compression Conference. IEEE, 1–10.Google Scholar
Cross Ref
- [8] . 2019. Learning to synthesize motion blur. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6840–6848.Google Scholar
Cross Ref
- [9] . 2020. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision. Springer, 213–229.Google Scholar
Digital Library
- [10] . 1996. A method for motion adaptive frame rate up-conversion. IEEE Transactions on Circuits and Systems for Video Technology 6, 5 (1996), 436–446.Google Scholar
Digital Library
- [11] . 2020. Generative pretraining from pixels. In Proceedings of the International Conference on Machine Learning. (PMLR), 1691–1703.Google Scholar
- [12] . 2016. Single-image depth perception in the wild. In Advances in Neural Information Processing Systems. 730–738.Google Scholar
- [13] . 2021. PDWN: Pyramid deformable warping network for video interpolation. IEEE Open Journal of Signal Processing 2 (2021), 413–424.Google Scholar
Cross Ref
- [14] . 2019. A multi-scale position feature transform network for video frame interpolation. IEEE Transactions on Circuits and Systems for Video Technology 30, 11 (2019), 3968–3981.Google Scholar
Cross Ref
- [15] . 2020. Video frame interpolation via deformable separable convolution. In Proceedings of the AAAI Conference on Artificial Intelligence 34, 10607–10614.Google Scholar
Cross Ref
- [16] . 2021. Multiple video frame interpolation via enhanced deformable separable convolution. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).Google Scholar
- [17] . 2020. All at once: Temporally adaptive multi-frame interpolation with advanced motion modeling. In European Conference on Computer Vision. Springer, 107–123.Google Scholar
Digital Library
- [18] . 2007. Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation. IEEE Transactions on Circuits and Systems for Video Technology 17, 4 (2007), 407–416.Google Scholar
Digital Library
- [19] . 2019. Deep frame prediction for video coding. IEEE Transactions on Circuits and Systems for Video Technology (2019).Google Scholar
- [20] . 2020. Scene-adaptive video frame interpolation via meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9444–9453.Google Scholar
Cross Ref
- [21] . 2021. Test-time adaptation for video frame interpolation via meta-learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021), 1–1.Google Scholar
- [22] . 2020. Channel attention is all you need for video frame interpolation. In Proceedings of the AAAI. 10663–10671.Google Scholar
Cross Ref
- [23] . 2021. Motion-aware dynamic architecture for efficient frame interpolation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 13839–13848.Google Scholar
Cross Ref
- [24] . 2021. Multi-scale warping for video frame interpolation. IEEE Access 9 (2021), 150470–150479.Google Scholar
Cross Ref
- [25] . 2017. Deformable convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision. 764–773.Google Scholar
Cross Ref
- [26] . 2019. Self-reproducing video frame interpolation. In Proceedings of the 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). IEEE, 193–198.Google Scholar
Cross Ref
- [27] . 2021. CDFI: Compression-driven network design for frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8001–8011.Google Scholar
Cross Ref
- [28] . 2021. Detection of deep video frame interpolation via learning dual-stream fusion CNN in the compression domain. In Proceedings of the 2021 IEEE International Conference on Multimedia and Expo (ICME). 1–6.Google Scholar
Cross Ref
- [29] . 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).Google Scholar
- [30] . 2015. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision. 2758–2766.Google Scholar
Digital Library
- [31] . 2021. ReFIn: A refinement approach for video frame interpolation. In Proceedings of the NeurIPS 2021 Workshop on Deep Learning and Inverse Problems.Google Scholar
- [32] . 2021. Efficient space-time video super resolution using low-resolution flow and mask upsampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 314–323.Google Scholar
Cross Ref
- [33] . 2016. Deepstereo: Learning to predict new views from the world’s imagery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5515–5524.Google Scholar
Cross Ref
- [34] . 2015. Optical flow modeling and computation: A survey. Computer Vision and Image Understanding 134 (2015), 1–21.Google Scholar
Digital Library
- [35] . 2017. Residual conv-deconv grid network for semantic segmentation. arXiv preprint arXiv:1707.07958 (2017).Google Scholar
- [36] . 2020. Intelligent cooperative edge computing in internet of things. IEEE Internet of Things Journal 7, 10 (2020), 9372–9382.Google Scholar
Cross Ref
- [37] . 2019. Continuous bidirectional optical flow for video frame sequence interpolation. In Proceedings of the 2019 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 1768–1773.Google Scholar
Cross Ref
- [38] . 2020. FeatureFlow: Robust video interpolation via structure-to-texture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14004–14013.Google Scholar
Cross Ref
- [39] . 2020. A spatiotemporal volumetric interpolation network for 4D dynamic medical image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4726–4735.Google Scholar
Cross Ref
- [40] . 2004. Motion compensated frame interpolation by new block-based motion estimation algorithm. IEEE Transactions on Consumer Electronics 50, 2 (2004), 752–759.Google Scholar
Digital Library
- [41] . 2020. Space-time-aware multi-resolution video enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2859–2868.Google Scholar
Cross Ref
- [42] . 2020. Motion feedback design for video frame interpolation. In Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020). IEEE, 4347–4351.Google Scholar
Cross Ref
- [43] . 2009. Correlation-based motion vector processing with adaptive interpolation scheme for motion-compensated frame interpolation. IEEE Transactions on Image Processing 18, 4 (2009), 740–752.Google Scholar
Digital Library
- [44] . 2008. A multistage motion vector processing method for motion-compensated frame interpolation. IEEE Transactions on Image Processing 17, 5 (2008), 694–708.Google Scholar
Digital Library
- [45] . 2018. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8981–8989.Google Scholar
Cross Ref
- [46] . 2017. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2462–2470.Google Scholar
Cross Ref
- [47] . 2018. Occlusions, motion and depth boundaries with a generic network for disparity, optical flow or scene flow estimation. In Proceedings of the European Conference on Computer Vision (ECCV). 614–630.Google Scholar
Digital Library
- [48] . 2003. Coarse-to-fine frame interpolation for frame rate up-conversion using pyramid structure. IEEE Transactions on Consumer Electronics 49, 3 (2003), 499–508.Google Scholar
Digital Library
- [49] . 2021. Consistent WCE video frame interpolation based on endoscopy image motion estimation. In Proceedings of the 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). 334–338.Google Scholar
Cross Ref
- [50] . 2018. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9000–9008.Google Scholar
Cross Ref
- [51] . 2019. Learning to extract flawless slow motion from blurry videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8112–8121.Google Scholar
Cross Ref
- [52] . 2021. Sequence image interpolation via separable convolution network. Remote Sensing 13, 2 (2021), 296.Google Scholar
Cross Ref
- [53] . 2020. What matters in unsupervised optical flow. In Proceedings of the European Conference on Computer Vision. Springer, 557–572.Google Scholar
Digital Library
- [54] . 2016. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1–10.Google Scholar
Digital Library
- [55] . 2020. FLAVR: Flow-agnostic video representations for fast frame interpolation. arXiv preprint arXiv:2012.08512 (2020).Google Scholar
- [56] . 2007. Motion compensated frame rate up-conversion using extended bilateral motion estimation. IEEE Transactions on Consumer Electronics 53, 4 (2007), 1759–1767.Google Scholar
Digital Library
- [57] . 2020. FISR: Deep joint frame interpolation and super-resolution with a multi-scale temporal loss. In Proceedings of the AAAI. 11278–11286.Google Scholar
Cross Ref
- [58] . 2017. Frame interpolation using generative adversarial networks.Google Scholar
- [59] . 2021. Direct video frame interpolation with multiple latent encoders. IEEE Access 9 (2021), 32457–32466.Google Scholar
Cross Ref
- [60] . 2014. Ultra high definition HEVC DASH data set. In Proceedings of the 5th ACM Multimedia Systems Conference. 7–12.Google Scholar
Digital Library
- [61] . 2020. AdaCoF: Adaptive collaboration of flows for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5316–5325.Google Scholar
Cross Ref
- [62] . 2019. Edge AI: On-demand accelerating deep neural network inference via edge computing. IEEE Transactions on Wireless Communications 19, 1 (2019), 447–457.Google Scholar
Cross Ref
- [63] . 2018. Learning IoT in edge: Deep learning for the Internet of Things with edge computing. IEEE Network 32, 1 (2018), 96–101.Google Scholar
Cross Ref
- [64] . 2017. Multimedia processing pricing strategy in GPU-accelerated cloud computing. IEEE Transactions on Cloud Computing (2017).Google Scholar
- [65] . 2019. Fi-net: A lightweight video frame interpolation network using feature-level flow. IEEE Access 7 (2019), 118287–118296.Google Scholar
- [66] . 2020. Video frame interpolation via residue refinement. In Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020). IEEE, 2613–2617.Google Scholar
Cross Ref
- [67] . 2021. Analysis of coronary angiography video interpolation methods to reduce x-ray exposure frequency based on deep learning. Cardiovascular Innovations and Applications (2021).Google Scholar
- [68] . 2020. Learning event-driven video deblurring and interpolation. In Proceedings of the ECCV (8). 695–710.Google Scholar
Digital Library
- [69] . 2008. A novel spatial and temporal correlation integrated based motion-compensated interpolation for frame rate up-conversion. IEEE Transactions on Consumer Electronics 54, 2 (2008), 863–869.Google Scholar
Digital Library
- [70] . 2020. Enhanced quadratic video interpolation. In Proceedings of the European Conference on Computer Vision. Springer, 41–56.Google Scholar
Digital Library
- [71] . 2019. Deep video frame interpolation using cyclic frame generation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33. 8794–8802.Google Scholar
Digital Library
- [72] . 2020. ConvTransformer: A convolutional transformer network for video frame synthesis. arXiv preprint arXiv:2011.10185 (2020).Google Scholar
- [73] . 2017. Video frame synthesis using deep voxel flow. In Proceedings of the IEEE International Conference on Computer Vision. 4463–4471.Google Scholar
Cross Ref
- [74] . 2016. Learning image matching by simply watching video. In Proceedings of the European Conference on Computer Vision. Springer, 434–450.Google Scholar
Cross Ref
- [75] . 1981. An iterative image registration technique with an application to stereo vision In. Proceedings of theIJCAI (IJCAI81) (1981), 674–679.Google Scholar
- [76] . 2018. Unflow: Unsupervised learning of optical flow with a bidirectional census loss. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.Google Scholar
Cross Ref
- [77] . 2020. Visual quality assessment for interpolated slow-motion videos based on a novel database. In Proceedings of the 2020 12th International Conference on Quality of Multimedia Experience (QoMEX). 1–6.Google Scholar
Cross Ref
- [78] . 2018. Phasenet for video frame interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 498–507.Google Scholar
Cross Ref
- [79] . 2015. Phase-based frame interpolation for video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1410–1418.Google Scholar
Cross Ref
- [80] . 2020. Medical image interpolation based on 3D lanczos filtering. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization 8, 3 (2020), 294–300.Google Scholar
Cross Ref
- [81] . 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3883–3891.Google Scholar
Cross Ref
- [82] . 2018. Context-aware synthesis for video frame interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1701–1710.Google Scholar
Cross Ref
- [83] . 2020. Softmax splatting for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5437–5446.Google Scholar
Cross Ref
- [84] . 2017. Video frame interpolation via adaptive convolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 670–679.Google Scholar
- [85] . 2017. Video frame interpolation via adaptive separable convolution. In Proceedings of the IEEE International Conference on Computer Vision. 261–270.Google Scholar
Cross Ref
- [86] . 2020. Revisiting adaptive convolutions for video frame interpolation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1099–1109.Google Scholar
- [87] . 2021. DeMFI: Deep joint deblurring and multi-frame interpolation with flow-guided attentive correlation and recursive boosting. arXiv preprint arXiv:2111.09985 (2021).Google Scholar
- [88] . 2021. EFI-net: Video frame interpolation from fusion of events and frames. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1291–1301.Google Scholar
Cross Ref
- [89] . 2021. A comprehensive survey on video frame interpolation techniques. The Visual Computer (2021), 1–25.Google Scholar
- [90] . 2020. BRUBC: Bilateral motion estimation with bilateral cost volume for video interpolation. In Proceedings of the European Conference on Computer Vision. Springer, 109–125.Google Scholar
- [91] . 2021. Asymmetric bilateral motion estimation for video frame interpolation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 14539–14548.Google Scholar
Cross Ref
- [92] . 2019. IM-net for high resolution video frame interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2398–2407.Google Scholar
Cross Ref
- [93] . 2016. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 724–732.Google Scholar
Cross Ref
- [94] . 2020. Dain-app: Application for video interpolations. (2020).Google Scholar
- [95] . 2012. Motion compensated frame interpolation with a symmetric optical flow constraint. In Proceedings of the International Symposium on Visual Computing. Springer, 447–457.Google Scholar
Cross Ref
- [96] . 2017. Optical flow estimation using a spatial pyramid network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4161–4170.Google Scholar
Cross Ref
- [97] . 2019. Unsupervised video interpolation using cycle consistency. In Proceedings of the IEEE International Conference on Computer Vision. 892–900.Google Scholar
Cross Ref
- [98] . 2015. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention. Springer, 234–241.Google Scholar
Cross Ref
- [99] . 2018. Generative compression. In Proceedings of the2018 Picture Coding Symposium (PCS). 258–262.Google Scholar
Cross Ref
- [100] . 2020. Optical flow estimation with deep learning, a survey on recent advances. In Deep Biometrics. Springer, 257–287.Google Scholar
Cross Ref
- [101] . 2020. A recurrent transformer network for novel view action synthesis. ECCV (27) (2020), 410–426.Google Scholar
- [102] . 2010. Motion tuned spatio-temporal quality assessment of natural videos. IEEE Transactions on Image Processing 19, 2 (2010), 335–350.Google Scholar
Digital Library
- [103] . 2020. Blurry video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5114–5123.Google Scholar
Cross Ref
- [104] . 2020. Video frame interpolation and enhancement via pyramid recurrent framework. IEEE Transactions on Image Processing 30 (2020), 277–292.Google Scholar
Cross Ref
- [105] . 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1874–1883.Google Scholar
Cross Ref
- [106] . 2021. Video frame interpolation via generalized deformable convolution. IEEE Transactions on Multimedia (2021).Google Scholar
- [107] . 2021. XVFI: eXtreme video frame interpolation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 14489–14498.Google Scholar
Cross Ref
- [108] . 2021. Deep animation video interpolation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6587–6595.Google Scholar
Cross Ref
- [109] . 2020. AIM 2020 challenge on video temporal super-resolution. In European Conference on Computer Vision. Springer, 23–40.Google Scholar
Digital Library
- [110] . 2013. The SJTU 4K video sequence dataset. In Proceedings of the 2013 5th International Workshop on Quality of Multimedia Experience (QoMEX). IEEE, 34–35.Google Scholar
Cross Ref
- [111] . 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012).Google Scholar
- [112] . 2017. Deep video deblurring for hand-held cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1279–1288.Google Scholar
Cross Ref
- [113] . 2018. PWC-NET: CRNS for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8934–8943.Google Scholar
Cross Ref
- [114] . 2019. Super resolution reconstruction of images based on interpolation and full convolutional neural network and application in medical fields. IEEE Access 7 (2019), 186470–186479.Google Scholar
Cross Ref
- [115] . 2012. SimpleFlow: A non-iterative, sublinear optical flow algorithm. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 345–353.Google Scholar
- [116] . 2020. RAFT: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision. Springer, 402–419.Google Scholar
Digital Library
- [117] . 2020. TDAN: Temporally-deformable alignment network for video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3360–3369.Google Scholar
Cross Ref
- [118] . 2021. FineNet: Frame interpolation and enhancement for face video deblurring. arXiv preprint arXiv:2103.00871 (2021).Google Scholar
- [119] . 2020. Efficient video frame interpolation using generative adversarial networks. Applied Sciences 10, 18 (2020), 6245.Google Scholar
Cross Ref
- [120] . 2019. A survey of variational and CNN-based optical flow techniques. Signal Processing: Image Communication 72 (2019), 9–24.Google Scholar
Digital Library
- [121] . 2021. Time lens: Event-based video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16155–16164.Google Scholar
Cross Ref
- [122] . 2016. Frame interpolation for cloud-based mobile video streaming. IEEE Transactions on Multimedia 18, 5 (2016), 831–839.Google Scholar
Digital Library
- [123] . 2017. Frame interpolation with multi-scale deep loss functions and generative adversarial networks. arXiv preprint arXiv:1711.06045 (2017).Google Scholar
- [124] . 2017. Attention is all you need. arXiv preprint arXiv:1706.03762 (2017).Google Scholar
- [125] . 2010. Motion-compensated frame rate up-conversion-Part II: New algorithms for frame interpolation. IEEE Transactions on Broadcasting 56, 2 (2010), 142–149.Google Scholar
Cross Ref
- [126] . 2021. MaX-DeepLab: End-to-end panoptic segmentation with mask transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5463–5474.Google Scholar
Cross Ref
- [127] . 2019. EDVR: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 0–0.Google Scholar
Cross Ref
- [128] . 2021. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8741–8750.Google Scholar
Cross Ref
- [129] . 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600–612.Google Scholar
Digital Library
- [130] . 2021. Sparse self-attention aggregation networks for neural sequence slice interpolation. BioData Mining 14, 1 (2021), 1–19.Google Scholar
Cross Ref
- [131] . 2021. Temporal spatial-adaptive interpolation with deformable refinement for electron microscopic images. arXiv preprint arXiv:2101.06771 (2021).Google Scholar
- [132] . 2013. DeepFlow: Large displacement optical flow with deep matching. In Proceedings of the IEEE International Conference on Computer Vision. 1385–1392.Google Scholar
Digital Library
- [133] . 2019. Generating realistic videos from keyframes with concatenated GANs. IEEE Transactions on Circuits and Systems for Video Technology 29, 8 (2019), 2337–2348. Google Scholar
Cross Ref
- [134] . 2011. Optical flow guided TV-L 1 video interpolation and restoration. In Proceedings of the International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition. Springer, 273–286.Google Scholar
Cross Ref
- [135] . 2018. Video compression through image interpolation. In Proceedings of the European Conference on Computer Vision (ECCV). 416–431.Google Scholar
Digital Library
- [136] . 2015. Modeling and optimization of high frame rate video transmission over wireless networks. IEEE Transactions on Wireless Communications 15, 4 (2015), 2713–2726.Google Scholar
Digital Library
- [137] . 2021. DRVI: Dual refinement for video interpolation. IEEE Access 9 (2021), 113566–113576.Google Scholar
Cross Ref
- [138] . 2020. Inter-slice image augmentation based on frame interpolation for boosting medical image segmentation accuracy. arXiv preprint arXiv:2001.11698 (2020).Google Scholar
- [139] . 2020. Zooming slow-mo: Fast and accurate one-stage space-time video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3370–3379.Google Scholar
Cross Ref
- [140] . 2021. Flow-aware synthesis: A generic motion model for video frame interpolation. Computational Visual Media (2021), 1–13.Google Scholar
- [141] . 2019. Quadratic video interpolation. Advances in Neural Information Processing Systems 32 (2019), 1647–1656.Google Scholar
- [142] . 2021. BWIN: A bilateral warping method for video frame interpolation. In Proceedings of the 2021 IEEE International Conference on Multimedia and Expo (ICME). 1–6.Google Scholar
Cross Ref
- [143] . 2019. Video enhancement with task-oriented flow. International Journal of Computer Vision 127, 8 (2019), 1106–1125.Google Scholar
Digital Library
- [144] . 2020. Frame-GAN: Increasing the frame rate of gait videos with generative adversarial networks. Neurocomputing 380 (2020), 95–104.Google Scholar
Digital Library
- [145] . 2020. Fine-grained motion estimation for video frame interpolation. IEEE Transactions on Broadcasting (2020).Google Scholar
- [146] . 2008. A new objective quality metric for frame interpolation used in video compression. IEEE Transactions on Broadcasting 54, 3 (2008), 680–11.Google Scholar
Cross Ref
- [147] . 2019. PoSNet: 4x video frame interpolation using position-specific flow. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). IEEE, 3503–3511.Google Scholar
Cross Ref
- [148] . 2013. Multi-level video frame interpolation: Exploiting the interaction among different levels. IEEE Transactions on Circuits and Systems for Video Technology 23, 7 (2013), 1235–1248.Google Scholar
Digital Library
- [149] . 2019. Zoom-in-to-check: Boosting video interpolation via instance-level discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12183–12191.Google Scholar
Cross Ref
- [150] . 2019. Multi-frame pyramid refinement network for video frame interpolation. IEEE Access 7 (2019), 130610–130621.Google Scholar
Cross Ref
- [151] . 2020. A flexible recurrent residual pyramid network for video frame interpolation. In European Conference on Computer Vision. Springer, 474–491.Google Scholar
Digital Library
- [152] . 2014. VSI: A visual saliency-induced index for perceptual image quality assessment. IEEE Transactions on Image Processing 23, 10 (2014), 4270–4281.Google Scholar
Cross Ref
- [153] . 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 586–595.Google Scholar
Cross Ref
- [154] . 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV). 286–301.Google Scholar
Digital Library
- [155] . 2020. Video frame interpolation without temporal priors. Advances in Neural Information Processing Systems 33 (2020), 13308–13318.Google Scholar
- [156] . 2021. EA-Net: Edge-aware network for flow-based video frame interpolation. arXiv preprint arXiv:2105.07673 (2021).Google Scholar
- [157] . 2018. Enhanced CTU-level inter-prediction with deep frame rate up-conversion for high efficiency video coding. In Proceedings of the2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 206–210.Google Scholar
Cross Ref
- [158] . 2020. End-to-end object detection with adaptive clustering transformer. arXiv preprint arXiv:2011.09315 (2020).Google Scholar
- [159] . 2021. How Video Super-Resolution and Frame Interpolation Mutually Benefit. Association for Computing Machinery, New York, NY, USA, 5445–5453.Google Scholar
- [160] . 2019. AAIoT: Accelerating artificial intelligence in IoT systems. IEEE Wireless Communications Letters 8, 3 (2019), 825–828.Google Scholar
Cross Ref
- [161] . 2017. To prune, or not to prune: Exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 (2017).Google Scholar
- [162] . 2019. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9308–9316.Google Scholar
Cross Ref
- [163] . 2021. Deep inter prediction via reference frame interpolation for blurry video coding. In Proceedings of the 2021 International Conference on Visual Communications and Image Processing (VCIP). 1–5.Google Scholar
Cross Ref
Index Terms
Video Frame Interpolation: A Comprehensive Survey
Recommendations
Progressive Spatial-temporal Collaborative Network for Video Frame Interpolation
MM '22: Proceedings of the 30th ACM International Conference on MultimediaMost video frame interpolation (VFI) algorithms infer the intermediate frame with the help of adjacent frames through the cascaded motion estimation and content refinement.However, the intrinsic correlations between motion and content are barely ...
Multi-Scale Coarse-to-Fine Transformer for Frame Interpolation
MM '22: Proceedings of the 30th ACM International Conference on MultimediaThe majority of prevailing video interpolation methods compute flows to estimate the intermediate motion. However, accurate estimation of the intermediate motion is difficult with low-order motion model hypothesis, which induces enormous difficulties ...
A comprehensive survey on video frame interpolation techniques
AbstractVideo frame interpolation is an important area in the computer vision research activities for video post-processing, surveillance, and video restoration tasks. It aims toward increasing the frame rate of a video sequence by calculating ...






Comments