Smooth Loops from Unconstrained Video

Abstract

Converting unconstrained video sequences into videos that loop seamlessly is an extremely challenging problem. In this work, we take the first steps towards automating this process by focusing on an important subclass of videos containing a single dominant foreground object. Our technique makes two novel contributions over previous work: first, we propose a correspondence-based similarity metric to automatically identify a good transition point in the video where the appearance and dynamics of the foreground are most consistent. Second, we develop a technique that aligns both the foreground and background about this transition point using a combination of global camera path planning and patch-based video morphing. We demonstrate that this allows us to create natural, compelling, loopy videos from a wide range of videos collected from the internet.

References

1. <label>{APS*14}</label> Arev I., Park H.S., Sheikh Y., Hodgins J., Shamir A.: Automatic editing of footage from multiple social cameras. ACM Trans. on Graph. Proc. SIGGRAPH Volume 33, Issue 4 July 2014, pp.81:1-81:11. 9
2. <label>{AZP*05}</label> Agarwala A., Zheng K.C., Pal C., Agrawala M., Cohen M., Curless B., Salesin D., Szeliski R.: Panoramic video textures. ACM Trans. on Graph. Proc. SIG-GRAPH Volume 24, Issue 3 July 2005, pp.821-827. 2
3. <label>{BAAR12}</label> Bai J., Agarwala A., Agrawala M., Ramamoorthi R.: Selectively de-animating video. ACM Trans. on Graph. Proc. SIGGRAPH Volume 31, Issue 4 July 2012, pp.66:1-66:10. 2
4. <label>{BBPP10}</label> Ballan L., Brostow G.J., Puwein J., Pollefeys M.: Unstructured video-based rendering: Interactive exploration of casually captured videos. ACM Trans. on Graph. Proc. SIGGRAPH Volume 29, Issue 4 July 2010, pp.87:1-87:11. 9
5. <label>{BLA12}</label> Berthouzoz F., Li W., Agrawala M.: Tools for placing cuts and transitions in interview video. ACM Trans. Graph. Volume 31, Issue 4 July 2012, pp.67:1-67:8. 2, 9
6. <label>{Car}</label> CARRÉ L. :. URL: "http://lillicarre.tumblr.com/". 1
7. <label>{DCWS03}</label> Doretto G., Chiuso A., Wu Y., Soatto S.: Dynamic textures. IJCV Volume 51, Issue 2 2003, pp.91-109. 2
8. <label>{FS11}</label> Freeman J., Simoncelli E.P.: Metamers of the ventral stream. Nature neuroscience Volume 14, Issue 9 2011, pp.1195-1201. 1
9. <label>{Gol07}</label> Goldman D.R.: A framework for video annotation, visualization, and interaction. PhD thesis, University of Washington, 2007. 2
10. <label>{Gom99}</label> Gomes J.: Warping and Morphing of Graphical Objects. Morgan Kaufmann, 1999. 2
11. <label>{HSGL11}</label> HaCohen Y., Shechtman E., Goldman D.B., Lischinski D.: Non-rigid dense correspondence with applications for image enhancement. ACM Trans. on Graph. Proc. SIGGRAPH Volume 30, Issue 4 2011, pp.70:1-70:9. 2, 4, 8
12. <label>{Inc}</label> Inci E. :. URL: "http://erdalinci.tumblr.com/". 1
13. <label>{JMD*12}</label> Joshi N., Mehta S., Drucker S., Stollnitz E., Hoppe H., Uyttendaele M., Cohen M.: Cliplets: Juxtaposing still and dynamic imagery</otherTitle>. In <otherTitle>Proc. UIST 2012, pp. pp.251-260. 2
14. <label>{KSE*03}</label> Kwatra V., Schödl A., Essa I., Turk G., Bobick A.: Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. on Graph. Proc. SIGGRAPH Volume 22, Issue 3 July 2003, pp.277-286. 2
15. <label>{KSSGS11}</label> Kemelmacher-Shlizerman I., Shechtman E., Garg R., Seitz S.M.: Exploring photobios. ACM Trans. on Graph. Proc. SIGGRAPH Volume 30, Issue 4 July 2011, pp.61:1-61:10. 2
16. <label>{LJH13}</label> Liao Z., Joshi N., Hoppe H.: Automated video looping with progressive dynamism. ACM Trans. on Graph. Proc. SIGGRAPH Volume 32, Issue 4 July 2013, pp.77:1-77:10. 2, 3
17. <label>{LLN*14}</label> Liao J., Lima R.S., Nehab D., Hoppe H., Sander P.V.: Semi-automated video morphing. Comp. Graph. Forum Proc. EGSR Volume 33, Issue 4 2014, pp.51-60. 2
18. <label>{LTK12}</label> Levieux P., Tompkin J., Kautz J.: Interactive viewpoint video textures</otherTitle>. In <otherTitle>Proc. CVMP December 2012. 3
19. <label>{PF13}</label> Papazoglou A., Ferrari V.: Fast object segmentation in unconstrained video</otherTitle>. <otherTitle>ICCV 2013. 3, 8
20. <label>{Raj}</label> Rajkovic M. :. URL: "http://milosrajk.ovic.tumblr.com/". 1
21. <label>{RKB04}</label> Rother C., Kolmogorov V., Blake A.: GrabCut: Interactive foreground extraction using iterated graph cuts</otherTitle>. In <otherTitle>ACM Trans. on Graph. Proc. SIGGRAPH 2004, pp. pp.309-314. 3
22. <label>{RWSG13}</label> Rãijegg J., Wang O., Smolic A., Gross M.: Ducttake: Spatiotemporal video compositing. Comp. Graph. Forum Volume 32, Issue 2pt1 2013, pp.51-61. 2
23. <label>{SRAIS10}</label> Shechtman E., Rav-Acha A., Irani M., Seitz S.M.: Regenerative morphing</otherTitle>. In <otherTitle>CVPR 2010, pp. pp.615-622. 2, 6
24. <label>{SRDB10}</label> Sun D., Roth S., Darmstadt T., Black M.: Secrets of optical Flow Estimation and Their Principles</otherTitle>. <otherTitle>CVPR 2010.
25. <label>{SSSE00}</label> Schödl A., Szeliski R., Salesin D.H., Essa I.: Video Textures</otherTitle>. In <otherTitle>ACM Trans. on Graph. Proc. SIGGRAPH 2000, pp. pp.489-498. 1, 2, 3, 4, 5
26. <label>{TPSK11}</label> Tompkin J., Pece F., Subr K., Kautz J.: Towards moment images: Automatic cinemagraphs</otherTitle>. In <otherTitle>Proc. CVMP November 2011, pp. pp.87-93. 2
27. <label>{Vin14}</label> Vine statistics, Aug 2014. URL: "http://blog.vine.co/post/95288683756/new-vine-camera-shoot-import-edit-and-share". 1
28. <label>{ZCA*09}</label> Zheng K.C., Colburn A., Agarwala A., Agrawala M., Curless B., Salesin D., Cohen M.: Parallax photography: Creating 3D cinematic effects from stills. In Graph. Interface 2009 May 2009. 2