Video Extrapolation Using Neighboring Frames

Abstract

With the popularity of immersive display systems that fill the viewer’s field of view (FOV) entirely, demand for wide FOV content has increased. A video extrapolation technique based on reuse of existing videos is one of the most efficient ways to produce wide FOV content. Extrapolating a video poses a great challenge, however, due to the insufficient amount of cues and information that can be leveraged for the estimation of the extended region. This article introduces a novel framework that allows the extrapolation of an input video and consequently converts a conventional content into one with wide FOV. The key idea of the proposed approach is to integrate the information from all frames in the input video into each frame. Utilizing the information from all frames is crucial because it is very difficult to achieve the goal with a two-dimensional transformation based approach when parallax caused by camera motion is apparent. Warping guided by three-dimensnional scene points matches the viewpoints between the different frames. The matched frames are blended to create extended views. Various experiments demonstrate that the results of the proposed method are more visually plausible than those produced using state-of-the-art techniques.

References

1. Aseem Agarwala, Ke Colin Zheng, Chris Pal, Maneesh Agrawala, Michael Cohen, Brian Curless, David Salesin, and Richard Szeliski. 2005. Panoramic video textures. In ACM Transactions on Graphics, Vol. 24. ACM, 821--827.
2. Amit Aides, Tamar Avraham, and Yoav Y. Schechner. 2011. Multiscale ultrawide foveated video extrapolation. In Proceedings of the 2011 IEEE International Conference on Computational Photography (ICCP’11). IEEE, 1--8.
3. Tamar Avraham and Yoav Y. Schechner. 2011. Ultrawide foveated video extrapolation. IEEE J. Select. Top. Sign. Process. 5, 2 (2011), 321--334.
4. Barco. 2014. Barco Escape. Retrieved from http://www.ready2escape.com.
5. Matthew Brown and David G. Lowe. 2007. Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74, 1 (2007), 59--73.
6. Che-Han Chang, Yuuki Sato, and Yung-Yu Chuang. 2014. Shape-preserving half-projective warps for image stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3254--3261.
7. Sunghyun Cho, Jue Wang, and Seungyong Lee. 2012. Video deblurring for hand-held cameras using patch-based synthesis. ACM Trans. Graph. 31, 4 (2012), 64.
8. Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13, 9 (2004), 1200--1212.
9. Carolina Cruz-Neira, Daniel J. Sandin, and Thomas A. DeFanti. 1993. Surround-screen projection-based virtual reality: The design and implementation of the CAVE. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques. ACM, 135--142.
10. Andrew J. Davison, Ian D. Reid, Nicholas D. Molton, and Olivier Stasse. 2007. MonoSLAM: Real-time single camera SLAM. IEEE Trans. Pattern Anal. Mach. Intell. 29, 6 (2007).
11. Abhijeet Ghosh, Matthew Trentacoste, Helge Seetzen, and Wolfgang Heidrich. 2005. Real illumination from virtual environments. In ACM SIGGRAPH 2005 Sketches. ACM, 41.
12. Miguel Granados, Kwang Kim, James Tompkin, Jan Kautz, and Christian Theobalt. 2012. Background inpainting for videos with dynamic objects and a free-moving camera. In Proceedings of the Computer Vision (ECCV’12). 682--695.
13. Jan Herling and Wolfgang Broll. 2014. High-quality real-time video inpaintingwith PixMix. IEEE Trans. Vis. Comput. Graph. 20, 6 (2014), 866--879.
14. Takeo Igarashi, Tomer Moscovich, and John F. Hughes. 2005. As-rigid-as-possible shape manipulation. ACM Trans. Graph. 24, 3 (2005), 1134--1141.
15. Wei Jiang and Jinwei Gu. 2015. Video stitching with spatial-temporal content-preserving warping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 42--48.
16. Brett Jones, Rajinder Sodhi, Michael Murdock, Ravish Mehra, Hrvoje Benko, Andrew Wilson, Eyal Ofek, Blair MacIntyre, Nikunj Raghuvanshi, and Lior Shapira. 2014. Roomalive: Magical experiences enabled by scalable, adaptive projector-camera units. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology. ACM, 637--644.
17. Brett R. Jones, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2013. IllumiRoom: Peripheral projected illusions for interactive experiences. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 869--878.
18. Georg Klein and David Murray. 2009. Parallel tracking and mapping on a camera phone. In Proceedings of the 8th IEEE International Symposium on Mixed and Augmented Reality (ISMAR’09). IEEE, 83--86.
19. Felix Klose, Oliver Wang, Jean-Charles Bazin, Marcus Magnor, and Alexander Sorkine-Hornung. 2015. Sampling based scene-space video processing. ACM Trans. Graph. 34, 4 (2015), 67.
20. Johannes Kopf, Michael F. Cohen, and Richard Szeliski. 2014. First-person hyper-lapse videos. ACM Trans. Graph. 33, 4 (2014), 78.
21. Jungjin Lee, Bumki Kim, Kyehyun Kim, Younghui Kim, and Junyong Noh. 2016. Rich360: Optimized spherical representation from structured panoramic camera arrays. ACM Trans. Graph. 35, 4 (2016), 63.
22. Jungjin Lee, Sangwoo Lee, Younghui Kim, and Junyong Noh. 2017. ScreenX: Public immersive theatres with uniform movie viewing experiences. IEEE Trans. Vis. Comput. Graph. 23, 2 (2017), 1124--1138.
23. Shiwei Li, Lu Yuan, Jian Sun, and Long Quan. 2015. Dual-feature warping-based motion model estimation. In Proceedings of the IEEE International Conference on Computer Vision. 4283--4291.
24. Jing Liao, Mark Finch, and Hugues Hoppe. 2015. Fast computation of seamless video loops. ACM Trans. Graph. 34, 6 (2015), 197.
25. Chung-Ching Lin, Sharathchandra U. Pankanti, Karthikeyan Natesan Ramamurthy, and Aleksandr Y. Aravkin. 2015. Adaptive as-natural-as-possible image stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1155--1163.
26. Kaimo Lin, Shuaicheng Liu, Loong-Fah Cheong, and Bing Zeng. 2016. Seamless video stitching from hand-held camera inputs. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 479--487.
27. Wen-Yan Lin, Siying Liu, Yasuyuki Matsushita, Tian-Tsong Ng, and Loong-Fah Cheong. 2011. Smoothly varying affine stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 345--352.
28. Feng Liu, Michael Gleicher, Hailin Jin, and Aseem Agarwala. 2009. Content-preserving warps for 3D video stabilization. ACM Trans. Graph. 28, 3 (2009), 44.
29. Abhimitra Meka, Michael Zollhöfer, Christian Richardt, and Christian Theobalt. 2016. Live intrinsic video. ACM Trans. Graph. 35, 4 (2016), 109.
30. Peter Mills, Alia Sheikh, Graham Thomas, and Paul Debenham. 2011. BBC surround video. Retrieved from http://www.bbc.com/news/magazine-22315685.
31. Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez. 2014. Video inpainting of complex scenes. SIAM J. Imag. Sci. 7, 4 (2014), 1993--2019.
32. Daniel Novy and Bove V. Michael. 2012. Infinity by nine. Retrieved from http://video.mit.edu/watch/infinity-by-nine-11565/.
33. F. Perazzi, A. Sorkine-Hornung, H. Zimmer, P Kaufmann, O. Wang, S. Watson, and M. Gross. 2015. Panoramic video from unstructured camera arrays. Comput. Graph. Forum 34, 2 (2015), 57--68.
34. Wen Qu, Yifei Zhang, Daling Wang, Shi Feng, and Ge Yu. 2015. Semantic movie summarization based on string of IE-RoleNets. Comput. Vis. Media 1, 2 (2015), 129--141.
35. Arno Schödl, Richard Szeliski, David H. Salesin, and Irfan Essa. 2000. Video textures. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. ACM Press/Addison-Wesley Publishing Co., 489--498.
36. Heung-Yeung Shum and Richard Szeliski. 2000. Systems and experiment paper: Construction of panoramic image mosaics with global and local alignment. Int. J. Comput. Vis. 36, 2 (2000), 101--130.
37. Jaewon Song. 2017. Personal communication with jaewon song, CTO, Digital Idea Corp. (2017).
38. Richard Szeliski. 2006. Image alignment and stitching: A tutorial. Found. Trends Comput. Graph. Vis. 2, 1 (2006), 1--104.
39. Laura Turban, Fabrice Urban, and Philippe Guillotel. 2017. Extrafoveal video extension for an immersive viewing experience. IEEE Trans. Visu. Comput. Graph. 23, 5 (2017), 1520--1533.
40. Miao Wang, Yukun Lai, Yuan Liang, Ralph Robert Martin, and Shi-Min Hu. 2014. BiggerPicture: Data-driven image extrapolation using graph matching. ACM Trans. Graph. 33, 6 (2014).
41. A Weffers-Albu, S. de Waele, W. Hoogenstraaten, and C. Kwisthout. 2011. Immersive TV viewing with advanced ambilight. In Proceedings of the 2011 IEEE International Conference on Consumer Electronics (ICCE’11). IEEE, 753--754.
42. Jordi Zaragoza, Tat-Jun Chin, Quoc-Huy Tran, Michael S. Brown, and David Suter. 2014. As-projective-as-possible image stitching with moving DLT. IEEE Trans. Pattern Anal. Mach. Intell. 36, 7 (2014), 1285--1298.
43. Fan Zhang and Feng Liu. 2014. Parallax-tolerant image stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3262--3269.
44. Fang-Lue Zhang, Jue Wang, Han Zhao, Ralph R. Martin, and Shi-Min Hu. 2015. Simultaneous camera path optimization and distraction removal for improving amateur video. IEEE Trans. Image Process. 24, 12 (2015), 5982--5994.
45. Fang-Lue Zhang, Xian Wu, Hao-Tian Zhang, Jue Wang, and Shi-Min Hu. 2016. Robust background identification for dynamic video editing. ACM Trans. Graph. 35, 6 (2016), 197.
46. Yinda Zhang, Jianxiong Xiao, James Hays, and Ping Tan. 2013. Framebreak: Dramatic image extrapolation by guided shift-maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1171--1178.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

  • Computer graphics

    • Image manipulation

      • Image-based rendering