Video collections in panoramic contexts

ABSTRACT

Video collections of places show contrasts and changes in our world, but current interfaces to video collections make it hard for users to explore these changes. Recent state-of-the-art interfaces attempt to solve this problem for 'outside->in' collections, but cannot connect 'inside->out' collections of the same place which do not visually overlap. We extend the focus+context paradigm to create a video-collections+context interface by embedding videos into a panorama. We build a spatio-temporal index and tools for fast exploration of the space and time of the video collection. We demonstrate the flexibility of our representation with interfaces for desktop and mobile flat displays, and for a spherical display with joypad and tablet controllers. We study with users the effect of our video-collection+context system to spatio-temporal localization tasks, and find significant improvements to accuracy and completion time in visual search tasks compared to existing systems. We measure the usability of our interface with System Usability Scale (SUS) and task-specific questionnaires, and find our system scores higher.

References

1. Agarwala, A., Zheng, K., Pal, C., Agrawala, M., Cohen, M., Curless, B., Salesin, D., and Szeliski, R. Panoramic Video Textures. ACM Trans. Graph. (TOG) 24, 3 (2005), 821--827.
2. Ballan, L., Brostow, G. J., Puwein, J., and Pollefeys, M. Unstructured Video-based Rendering: Interactive Exploration of Casually Captured Videos. ACM Trans. Graph. 29, 4 (2010).
3. Benosman, R., and Kang, S. B. Panoramic Vision: Sensors, Theory, and Applications. Springer, 2001.
4. Brooke, J. SUS: A Quick and Dirty Usability Scale. In Usability Evaluation in Industry. Taylor & Francis, 1996.
5. Brown, M., and Lowe, D. G. Automatic Panoramic Image Stitching using Invariant Features. International Journal of Computer Vision 74, 1 (2006).
6. Cockburn, A., Karlson, A., and Bederson, B. B. A Review of Overview+detail, Zooming, and Focus+context Interfaces. ACM Comput. Surv. 41, 1 (2009), 2:1--2:31.
7. Dale, K., Shechtman, E., Avidan, S., and Pfister, H. Multi-video browsing and summarization. In CVPR Workshops (2012).
8. de Haan, G., Scheuer, J., de Vries, R., and Post, F. H. Egocentric Navigation for Video Surveillance in 3D Virtual Environments. 2009 IEEE Symposium on 3D User Interfaces (2009).
9. DeCamp, P., Shaw, G., Kubat, R., and Roy, D. An Immersive System for Browsing and Visualizing Surveillance Video. Proc. MM '10 (2010).
10. Farin, D. S. Automatic Video Segmentation employing Object/Camera Modeling Techniques. PhD thesis, Technische Universiteit Eindhoven, 2005.
11. Hartley, R., and Zisserman, A. Multiple View Geometry in Computer Vision, 2 ed. Cambridge University Press, New York, NY, USA, 2003.
12. Hermans, C., Vanaken, C., Mertens, T., Van Reeth, F., and Bekaert, P. Augmented Panoramic Video. Computer Graphics Forum 27, 2 (2008).
13. Hu, J., You, S., and Neumann, U. Texture Painting from Video. Journal of WSCG (2005).
14. Joshi, N., Kar, A., and Cohen, M. Looking at You: Fused Gyro and Face Tracking for Viewing Large Imagery on Mobile Devices. In Proc. SIGCHI '12, ACM (New York, NY, USA, 2012), 2211--2220.
15. Kim, K., Oh, S., Lee, J., and Essa, I. Augmenting Aerial Earth Maps with Dynamic Information from Videos. Virtual Reality (2011), 1--16.
16. Kroepfl, M., Wexler, Y., and Ofek, E. Efficiently Locating Photographs in Many Panoramas. In Proc. SIGSPATIAL '10, ACM (New York, NY, USA, 2010).
17. Lewis, J., and Sauro, J. The Factor Structure of the System Usability Scale. In Proc. Human Centered Design, Springer (2009), 94--103.
18. Lowe, D. G. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60, 2 (2004).
19. McCurdy, N. RealityFlythrough: A System for Ubiquitous Video. Ph.d., University of California, San Diego, 2007.
20. Mulloni, A., Seichter, H., Dünser, A., Baudisch, P., and Schmalstieg, D. 360 degrees: Panoramic Overviews for Location-Based Services. Proc. SIGCHI '12.
21. Naimark, M. Time Binoculars. http://www.naimark.net/projects/pending/timebinoculars.htm, 2010.
22. Neumann, U., You, S., and Hu, J. Augmented Virtual Environments (AVE): Dynamic Fusion of Imagery and 3D Models. IEEE Virtual Reality '03 (2003), 3--9.
23. Norris, J., Schnädelbach, H., and Qiu, G. CamBlend: An Object Focused Collaboration Tool. In Proc. SIGCHI'12, ACM (New York, NY, USA, 2012).
24. Pece, F., Steptoe, W., Wanner, F., Julier, S., Weyrich, T., Kautz, J., and Steed, A. Panoinserts: Practical spatial teleconferencing. In Proc. SIGCHI '13, ACM (New York, NY, USA, 2013).
25. Pirk, S., Cohen, M. F., Deussen, O., Uyttendaele, M., and Kopf, J. Video Enhanced Gigapixel Panoramas. SIGGRAPH Asia 2012 Technical Briefs (2012).
26. Pongnumkul, S., Wang, J., and Cohen, M. Creating Map-based Storyboards for Browsing Tour Videos. Proc. UIST '08 (2008).
27. Silva, J. R., Santos, T. T., and Morimoto, C. H. Automatic Camera Control in Virtual Environments augmented using Multiple Sparse Videos. Computers & Graphics 35, 2 (2011).
28. Snavely, N., Garg, R., Seitz, S. M., and Szeliski, R. Finding Paths Through the World's Photos. ACM Trans. Graph. 27, 3 (2008).
29. Snavely, N., Seitz, S., and Szeliski, R. Photo Tourism: Exploring Photo Collections in 3D. ACM Trans. Graph. 25, 3 (2006).
30. Szeliski, R. Computer Vision: Algorithms and Applications, 1st ed. Springer-Verlag, 2010.
31. Tompkin, J., Kim, K. I., Kautz, J., and Theobalt, C. Videoscapes: Exploring Sparse, Unstructured Video Collections. ACM Trans. Graph. 31, 4 (2012).
32. Wu, F., and Tory, M. PhotoScope: Visualizing Spatiotemporal Coverage of Photos for Construction Management. Proc. SIGCHI '09.

Index Terms

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

      • Virtual reality

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Mixed / augmented reality

      • Virtual reality