Cliplets: juxtaposing still and dynamic imagery

ABSTRACT

We explore creating ""cliplets"", a form of visual media that juxtaposes still image and video segments, both spatially and temporally, to expressively abstract a moment. Much as in ""cinemagraphs"", the tension between static and dynamic elements in a cliplet reinforces both aspects, strongly focusing the viewer's attention. Creating this type of imagery is challenging without professional tools and training. We develop a set of idioms, essentially spatiotemporal mappings, that characterize cliplet elements, and use these idioms in an interactive system to quickly compose a cliplet from ordinary handheld video. One difficulty is to avoid artifacts in the cliplet composition without resorting to extensive manual input. We address this with automatic alignment, looping optimization and feathering, simultaneous matting and compositing, and Laplacian blending. A key user-interface challenge is to provide affordances to define the parameters of the mappings from input time to output time while maintaining a focus on the cliplet being created. We demonstrate the creation of a variety of cliplet types. We also report on informal feedback as well as a more structured survey of users.

References

1. A. Adams. Camera. Number v. 1. Houghton Mifflin Harcourt P, 1980.
2. Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, and Michael Cohen. Interactive digital photomontage. ACM Trans. Graph., 23(3):294--302, 2004.
3. Aseem Agarwala, Ke Colin Zheng, Chris Pal, Maneesh Agrawala, Michael Cohen, Brian Curless, David Salesin, and Richard Szeliski. Panoramic video textures. ACM Trans. Graph., 24(3):821--827, July 2005.
4. Jiamin Bai, Aseem Agarwala, Maneesh Agrawala, and Ravi Ramamoorthi. Selectively de-animating video. ACM Transactions on Graphics, 31(4), 2012.
5. Xue Bai, Jue Wang, David Simons, and Guillermo Sapiro. Video snapcut: robust video object cutout using localized classifiers. ACM Trans. Graph., 28(3), 2009.
6. Jamie Beck and Kevin Burg. Cinemagraphs. http://http://cinemagraphs.com/.
7. Peter J. Burt and Edward H. Adelson. A multiresolution spline with application to image mosaics. ACM Trans. Graph., 2(4):217--236, October 1983.
8. Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua. Brief: binary robust independent elementary features. In Proceedings of the 11th European Conference on Computer vision: Part IV, ECCV'10, pages 778--792, 2010.
9. Yung-Yu Chuang, Aseem Agarwala, Brian Curless, David H. Salesin, and Richard Szeliski. Video matting of complex scenes. ACM Trans. Graph., 21(3), 2002.
10. Yung-Yu Chuang, Dan B Goldman, Ke Colin Zheng, Brian Curless, David H. Salesin, and Richard Szeliski. Animating pictures with stochastic motion textures. ACM Trans. Graph., 24(3):853--860, 2005.
11. Martin A. Fischler and Robert C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6):381--395, June 1981.
12. William T. Freeman, Edward H. Adelson, and David J. Heeger. Motion without movement. In SIGGRAPH Proceedings, pages 27--30, 1991.
13. Michael L. Gleicher and Feng Liu. Re-cinematography: Improving the camerawork of casual video. ACM Trans. Multimedia Comput. Commun. Appl., 5(1):2:1--2:28, October 2008.
14. Dan B. Goldman, Chris Gonterman, Brian Curless, David Salesin, and Steven M. Seitz. Video object annotation, navigation, and composition. In Proceedings of ACM Symposium on User Interface Software and Technology, UIST '08, pages 3--12, 2008.
15. Matthias Grundmann, Vivek Kwatra, and Irfan Essa. Auto-directed video stabilization with robust L1 optimal camera paths. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011), 2011.
16. Randy Kennedy. The still-life mentor to a filmmaking generation. The New York Times, October 6, 2006.
17. Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. Graphcut textures: image and video synthesis using graph cuts. ACM Trans. Graph., 22(3):277--286, July 2003.
18. Ce Liu, Antonio Torralba, William T. Freeman, Frédo Durand, and Edward H. Adelson. Motion magnification. ACM Trans. Graph., 24(3):519--526, July 2005.
19. David G. Lowe. Object recognition from local scale-invariant features. IEEE International Conference on Computer Vision, 2:1150, 1999.
20. Alex Rav-Acha, Yael Pritch, Dani Lischinski, and Shmuel Peleg. Spatio-temporal video warping. In ACM SIGGRAPH 2005 Sketches, SIGGRAPH '05, 2005.
21. Alex Rav-Acha, Yael Pritch, Dani Lischinski, and Shmuel Peleg. Dynamosaicing: Mosaicing of dynamic scenes. IEEE Trans. Pattern Anal. Mach. Intell., 29(10):1789--1801, October 2007.
22. Arno Schödl and Irfan A. Essa. Controlled animation of video sprites. In ACM SIGGRAPH/Eurographics Symposium on Computer animation, pages 121--127, 2002.
23. Arno Schödl, Richard Szeliski, David H. Salesin, and Irfan Essa. Video textures. In SIGGRAPH Proceedings, pages 489--498, 2000.
24. Richard Szeliski. Locally adapted hierarchical basis preconditioning. ACM Trans. Graph., 25(3):1135--1143, 2006.
25. Richard Szeliski, Matt Uyttendaele, and Drew Steedly. Fast Poisson blending using multi-splines. In IEEE International Conference on Computational Photography, April 2011.
26. James Tompkin, Fabrizio Pece, Kartic Subr, and Jan Kautz. Towards moment images: Automatic cinemagraphs. In Proceedings of the 8th European Conference on Visual Media Production (CVMP 2011), November 2011.
27. Jue Wang and Michael F. Cohen. Simultaneous matting and compositing. In IEEE Computer Vision and Pattern Recognition, pages 1--8, June 2007.
28. Ke Colin Zheng, Alex Colburn, Aseem Agarwala, Maneesh Agrawala, David Salesin, Brian Curless, and Michael F. Cohen. Parallax photography: creating 3D cinematic effects from stills. In Proceedings of Graphics Interface, pages 111--118, 2009.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)