Gigapixel Panorama Video Loops

Abstract

We present the first technique to create wide-angle, high-resolution looping panoramic videos. Starting with a 2D grid of registered videos acquired on a robotic mount, we formulate a combinatorial optimization to determine for each output pixel the source video and looping parameters that jointly maximize spatiotemporal consistency. This optimization is accelerated by reducing the set of source labels using a graph-coloring scheme. We parallelize the computation and implement it out-of-core by partitioning the domain along low-importance paths. The merged panorama is assembled using gradient-domain blending and stored as a hierarchy of video tiles. Finally, an interactive viewer adaptively preloads these tiles for responsive browsing and allows the user to interactively edit and improve local regions. We demonstrate these techniques on gigapixel-sized looping panoramas.

References

1. Aseem Agarwala. 2007. Efficient gradient-domain compositing using quadtrees. ACM Trans. Graph. 26, 3 (2007), 94.
2. A. Agarwala, M. Dontcheva, M. Agrawala, S. Drucker, A. Colburn, B. Curless, D. Salesin, and M. Cohen. 2004. Interactive Digital Photomontage. ACM Trans. Graph. 23, 3 (2004).
3. Aseem Agarwala, Ke Colin Zheng, Chris Pal, Maneesh Agrawala, Michael Cohen, Brian Curless, David Salesin, and Richard Szeliski. 2005. Panoramic video textures. ACM Trans. Graph. 24, 3 (July 2005), 7.
4. Jiamin Bai, Aseem Agarwala, Maneesh Agrawala, and Ravi Ramamoorthi. 2012. Selectively De-Animating Video. ACM Trans. Graph. 31, 4 (2012).
5. Jiamin Bai, Aseem Agarwala, Maneesh Agrawala, and Ravi Ramamoorthi. 2013. Automatic cinemagraph portraits. Computer Graphics Forum 32, 4 (2013), 17--25.
6. Jamie Beck and Kevin Burg. 2012. Cinemagraphs. http://cinemagraphs.com/. (2012).
7. Jean-Yves Bouguet. 2001. Pyramidal implementation of the affine Lucas-Kanade feature tracker: Description of the algorithm. Intel Corporation 5, 1--10 (2001), 4.
8. Matthew Brown and David G. Lowe. 2007. Automatic panoramic image stitching using invariant features. IJCV 74, 1 (2007), 59--73.
9. V. Couture, M. Langer, and S. Roy. 2011. Panoramic stereo video textures. ICCV (2011), 1251--1258.
10. Vincent Couture, Michael S. Langer, and Sébastien Roy. 2012. Perception of blending in stereo motion panoramas. ACM Trans. Applied Perception 9, 3 (2012), 15.
11. Dan B. Goldman. 2010. Vignette and exposure calibration and compensation. IEEE transactions on pattern analysis and machine intelligence 32, 12 (2010), 2276--2288.
12. João F. Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista. 2015. High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 3 (2015), 583--596.
13. Chris Hermans, Cedric Vanaken, Tom Mertens, Frank Van Reeth, and Philippe Bekaert. 2008. Augmented panoramic video. In Computer Graphics Forum, Vol. 27.
14. Cheuk Yiu Ip and Amitabh Varshney. 2011. Saliency-assisted navigation of very large landscape images. IEEE TVCG 17, 12 (2011).
15. Neel Joshi, Sisil Mehta, Steven Drucker, Eric Stollnitz, Hugues Hoppe, Matt Uyttendaele, and Michael Cohen. 2012. Cliplets: Juxtaposing still and dynamic imagery. Proc. of UIST (2012).
16. George Karypis. 2003. Multi-constraint mesh partitioning for contact/impact computations. In Proc. of ACM/IEEE conference on Supercomputing. ACM, 56.
17. M. Kazhdan and H. Hoppe. 2008. Streaming Multigrid for Gradient-Domain Operations on Large Images. ACM Trans. Graph. 27, 3 (2008).
18. Michael Kazhdan, Dinoj Surendran, and Hugues Hoppe. 2010. Distributed gradient-domain processing of planar and spherical images. ACM Trans. Graph. 29, 2 (2010).
19. V. Kolmogorov and R. Zabih. 2004. What Energy Functions can be Minimized via Graph Cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26, 2 (2004).
20. Johannes Kopf, Matt Uyttendaele, Oliver Deussen, and Michael F. Cohen. 2007. Capturing and viewing gigapixel images. ACM Trans. Graph. 26, 3 (2007), 93.
21. Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. 2003. Graphcut textures: image and video synthesis using graph cuts. In ACM Trans. Graph., Vol. 22. 277--286.
22. Jing Liao, Mark Finch, and Hugues Hoppe. 2015. Fast computation of seamless video loops. ACM Trans. Graph. 34, 6 (2015), 197.
23. Jing Liao, Neel Joshi, and Hugues Hoppe. 2013. Automated video looping with progressive dynamism. ACM Trans. Graph. 32, 4 (2013).
24. Jiangyu Liu and Jian Sun. 2010. Parallel graph-cuts by adaptive bottom-up merging. In Proc. CVPR. IEEE, 2181--2188.
25. Qing Luan, Steven M. Drucker, Johannes Kopf, Ying-Qing Xu, and Michael F. Cohen. 2008. Annotating gigapixel images. In Proc. ACM UIST. 33--36.
26. Microsoft Research. 2016. Image Composite Editor. http://research.microsoft.com/en-us/um/redmond/groups/ivm/ICE/. (2016).
27. Federico Perazzi, Alexander Sorkine-Hornung, Henning Zimmer, Peter Kaufmann, Oliver Wang, S. Watson, and Markus H. Gross. 2015. Panoramic video from unstructured camera arrays. Computer Graphics Forum 34, 2 (2015), 57--68.
28. Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson image editing. ACM Trans. Graph. 22, 3 (2003).
29. Sujin Philip, Brian Summa, Valerio Pascucci, and P.-T. Bremer. 2011. Hybrid CPU-GPU solver for gradient domain processing of massive images. In Intl. Conf. on Parallel and Distributed Systems.
30. Sujin Philip, Brian Summa, Julien Tierny, Peer-Timo Bremer, and Valerio Pascucci. 2015. Distributed Seams for Gigapixel Panoramas. IEEE TVCG 21, 3 (2015), 350--362.
31. Sören Pirk, Michael F. Cohen, Oliver Deussen, Matt Uyttendaele, and Johannes Kopf. 2012. Video enhanced gigapixel panoramas. SIGGRAPH Asia 2012 Technical Briefs (2012).
32. Alex Rav-Acha, Yael Pritch, Dani Lischinski, and Shmuel Peleg. 2007. Dynamosaicing: Mosaicing of Dynamic Scenes. IEEE Trans. on Pattern Anal. Mach. Intell. 29, 10 (2007).
33. Arno Schödl, Richard Szeliski, David H. Salesin, and Irfan Essa. 2000. Video textures. In Proc. SIGGRAPH. 489--498.
34. Laura Sevilla-Lara, Jonas Wulff, Kalyan Sunkavalli, and Eli Shechtman. 2015. Smooth Loops from Unconstrained Video. Computer Graphics Forum 34, 4 (2015), 99--107.
35. Petter Strandmark and Fredrik Kahl. 2010. Parallel and distributed graph cuts by dual decomposition. In Proc. CVPR. IEEE, 2085--2092.
36. Brian Summa, Giorgio Scorzelli, Ming Jiang, Peer-Timo Bremer, and Valerio Pascucci. 2011. Interactive editing of massive imagery made simple: Turning Atlanta into Atlantis. ACM Transactions on Graphics (TOG) 30, 2 (2011), 7.
37. Brian Summa, Julien Tierny, and Valerio Pascucci. 2012. Panorama weaving: Fast and flexible seam processing. ACM Trans. Graph. 31, 4 (2012).
38. Richard Szeliski. 2006. Image alignment and stitching: A tutorial. Foundations and Trends in Computer Graphics and Vision 2, 1 (2006).
39. Richard Szeliski and Heung-Yeung Shum. 1997. Creating full view panoramic image mosaics and environment maps. In Proc. ACM SIGGRAPH.
40. James Tompkin, Fabrizio Pece, Rajvi Shah, Shahram Izadi, Jan Kautz, and Christian Theobalt. 2013. Video collections in panoramic contexts. In Proc. ACM UIST.
41. James Tompkin, Fabrizio Pece, Kartic Subr, and Jan Kautz. 2011. Towards Moment Images: Automatic Cinemagraphs. In Proc. of the 8th European Conference on Visual Media Production (CVMP 2011).

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation

      • Computational photography