Schematic storyboarding for video visualization and editing

Abstract

We present a method for visualizing short video clips in a single static image, using the visual language of storyboards. These schematic storyboards are composed from multiple input frames and annotated using outlines, arrows, and text describing the motion in the scene. The principal advantage of this storyboard representation over standard representations of video -- generally either a static thumbnail image or a playback of the video clip in its entirety -- is that it requires only a moment to observe and comprehend but at the same time retains much of the detail of the source video. Our system renders a schematic storyboard layout based on a small amount of user interaction. We also demonstrate an interaction technique to scrub through time using the natural spatial dimensions of the storyboard. Potential applications include video editing, surveillance summarization, assembly instructions, composition of graphic novels, and illustration of camera technique for film studies.

References

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Transactions on Graphics (Proc. SIGGRAPH) 23, 4, 294--301.
2. Assa, J., Caspi, Y., and Cohen-Or, D. 2005. Action synopsis: Pose selection and illustration. ACM Transactions on Graphics (Proc. SIGGRAPH) 24, 3, 667--676.
3. Cheong, L., and Huo, H. 2001. Shot change detection using scene-based constraint. Multimedia Tools and Applications 14, 2 (June), 175--186.
4. Cutting, J. E. 2002. Representing motion in a static image: constraints and parallels in art, science, and popular culture. Perception 31, 1165--1193.
5. Freeman, W. T., and Zhang, H. 2003. Shape-time photography. In Proc. Computer Vision and Pattern Recognition, 151--157.
6. Hart, J. 1999. The art of the storyboard: storyboarding for film, TV and animation. Focal Press.
7. Heng, W., and Ngan, K. 2001. An object-based shot boundary detection using edge tracing and tracking. Journal of Visual Communication and Image Representation 12, 3 (September), 217--239.
8. Horn, B., Hilden, H., and Negahdaripour, S. 1988. Closed-form solution of absolute orientation using orthonormal matrices. Journal of the Optical Society of America A 5, 7, 1127--1135.
9. Irani, M., and Anandan, P. 1998. Video indexing based on mosaic representations. IEEE Transactions on Pattern Analysis and Machine Intelligence 86, 5 (May), 905--921.
10. Jia, J., and Tang, C.-K. 2005. Eliminating structure and intensity misalignment in image stitching. In Proc. ICCV.
11. Jojic, N., Basu, S., Petrovic, N., Frey, B., and Huang, T., 2003. Joint design of data analysis algorithms and user interface for video applications. presented at NIPS 2003 workshop on Machine Learning in User Interface, extended abstract at: http://research.microsoft.com/workshops/MLUI03/jojic.html.
12. Katz, S. D. 1991. Film directing shot by shot: visualizing from concept to screen. Michael Wiese Productions.
13. Kawagishi, Y., Hatsuyama, K., and Kondo, K. 2003. Cartoon blur: nonphotorealistic motion blur. In Proc. Comp. Graph. Intl., 276--281.
14. Kim, B., and Essa, I. 2005. Video-based nonphotorealistic and expressive illustration of motion. In Proc. Comp. Graph. Intl., 32--35.
15. Kumar, M. P., Torr, P. H. S., and Zisserman, A. 2005. Learning layered motion segmentations of video. In Proc. ICCV, 33--40.
16. Lee, M., Yang, Y., and Lee, S. 2001. Automatic video parsing using shot boundary detection and camera operation analysis. Pattern Recognition 34, 3 (March), 711--719.
17. Li, Y., Zhang, T., and Tretter, D. 2001. An overview of video abstraction techniques. Tech. Rep. HPL-2001-191, HP Laboratories.
18. Massey, M., and Bender, W. 1996. Salient stills: process and practice. IBM Systems Journal 35, 3, 4, 557--574.
19. Masuch, M., Schlechtweg, S., and Schultz, R. 1999. Speedlines: depicting motion in motionless pictures. In ACM SIGGRAPH 99 Conference abstracts and applications, 277.
20. McCloud, S. 1993. Understanding Comics: The Invisible Art. Harper-Collins.
21. Nicolas, H., Manaury, A., Benois-Pineau, J., Dupuy, W., and Barba, D. 2004. Grouping video shots into scenes based on 1d mosaic descriptors. In Proc. Intl. Conf. on Image Proc., I: 637--640.
22. Nienhaus, M., and Döllner, J. 2003. Dynamic glyphs -- depicting dynamics in images of 3D scenes. In Third International Symposium on Smart Graphics, 102--111.
23. Rother, C., Kolmogorov, V., and Blake, A. 2004. "GrabCut" -- interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics (Proc. SIGGRAPH) 23, 3, 309--314.
24. Rubin, M. 2005. Droidmaker: George Lucas and the digital revolution. Triad Publishing. 327, 338.
25. Sangster, C., 2005. Personal Communication.
26. Taniguchi, Y., Akutsu, A., and Tonomura, Y. 1997. PanoramaExcerpts: extracting and packing panoramas for video browsing. In Proc. ACM Intl. Conf. on Multimedia, 427--436.
27. Teodosio, L., and Bender, W. 1993. Salient video stills: Content and context preserved. In Proc. ACM Intl. Conf. on Multimedia, 39--46.
28. Umeyama, S. 1991. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 13, 4, 376--380.
29. Ward, J. 1979. Perception and Pictorial Representation, vol. 1. Praeger, New York, ch. 13, "A piece of the action: Moving figures in still pictures", 246--271.
30. Wexler, Y., and Simakov, D. 2005. Space-time scene manifolds. In Proc. ICCV, 858--863.
31. Wood, D. N., Finkelstein, A., Hughes, J. F., Thayer, C. E., and Salesin, D. H. 1997. Multiperspective panoramas for cel animation. In Proc. SIGGRAPH, 243--250.
32. Zelnik-Manor, L., Peters, G., and Perona, P. 2005. Squaring the circle in panoramas. In Proc. ICCV, 1292--1299.

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Graphics recognition and interpretation

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision tasks

  • Computer graphics