skip to main content
research-article

Re-cinematography: Improving the camerawork of casual video

Published:30 October 2008Publication History
Skip Abstract Section

Abstract

This article presents an approach to postprocessing casually captured videos to improve apparent camera movement. Re-cinematography transforms each frame of a video such that the video better follows cinematic conventions. The approach breaks a video into shorter segments. Segments of the source video where there is no intentional camera movement are made to appear as if the camera is completely static. For segments with camera motions, camera paths are keyframed automatically and interpolated with matrix logarithms to give velocity-profiled movements that appear intentional and directed. Closeups are inserted to provide compositional variety in otherwise uniform segments. The approach automatically balances the tradeoff between motion smoothness and distortion to the original imagery. Results from our prototype show improvements to poor quality home videos.

References

  1. Achanta, R., Yan, W.-Q., and Kankanhalli, M. 2006. Modeling intent for home video repurposing. IEEE Multimedia 13, 46--55. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Adams, B., Venkatesh, S., and Jain, R. 2005. IMCE: Integrated media creation environment. ACM Trans. Multimed. Comput. Comm. Appl 1, 211--247. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Alexa, M. 2002. Linear combinations of transformations. In Proceedings of the ACM SIGGRAPH International Conference on Computer Graphics and Interactive Techniques. 380--387. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Ang, T. 2005. Digital Video Handbook. Dorling Kindersley. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Arijon, D. 1991. Grammar of the Film Language. Silman-James Press.Google ScholarGoogle Scholar
  6. Bennett, E. P. and McMillan, L. 2003. Proscenium: A framework for spatio-temporal video editing. In Proceedings of the 11th ACM International Conference on Multimedia (MULTIMEDIA'03). ACM, New York, NY, 177--184. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Bevilacqua, A. and Azzari, P. 2006. High-quality real time motion detection using ptz cameras. In Proceedings of the IEEE International Conference on Video and Signal Based Surveillance (AVSS'06). 23. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Block, B. A. 2001. The Visual Story: Seeing the Structure of Film, TV, and New Media. Focal Press.Google ScholarGoogle Scholar
  9. Bordwell, D. and Thompson, K. 1997. Film Art: An Introduction. McGraw-Hill.Google ScholarGoogle Scholar
  10. Brandon, B. 2005. The Complete Digital Video Guide. Reader's Digest. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Brown, B. 2002. Cinematography: Theory and Practice: Imagemaking for Cinematographers, Directors & Videographers. Butterworth-Heinemann.Google ScholarGoogle Scholar
  12. Buehler, C., Bosse, M., and McMillan, L. 2001. Non-metric image-based rendering for video stabilization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Vol. 2. 609--614.Google ScholarGoogle ScholarCross RefCross Ref
  13. Casares, J., Long, A. C., Myers, B. A., Bhatnagar, R., Stevens, S. M., Dabbish, L., Yocum, D., and Corbett, A. 2002. Simplifying video editing using metadata. In Proceedings of the 4th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques. ACM, London, England, 157--166. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Chalfen, R. 1987. Snapshot Versions of Life. Bowling Green State University Press.Google ScholarGoogle Scholar
  15. Chen, L.-Q., Xie, X., Fan, X., Ma, W.-Y., Zhang, H.-J., and Zhou, H.-Q. 2003. A visual attention model for adapting images on small displays. Multimed. Syst. 9, 4, 353--364.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Christie, M., Machap, R., Normand, J.-M., Olivier, P., and Pickering, J. 2005. Virtual camera planning: A survey. In Proceedings of Smart Graphics. 40--52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Crow, F. C. 1984. Summed-area tables for texture mapping. In Proceedings of the ACM SIGGRAPH International Conference on Computer Graphics and Interactive Techniques. 207--212. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Dony, R., Mateer, J., and Robinson, J. 2005. Techniques for automated reverse storyboarding. IEE J. Vision, Image Signal Process. 152, 4, 425--436.Google ScholarGoogle ScholarCross RefCross Ref
  19. Fischler, M. A. and Bolles, R. C. 1981. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Comm. ACM 24, 6, 381--395. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Girgensohn, A., Boreczky, J., Chiu, P., Doherty, J., Foote, J., Golovchinsky, G., Uchihashi, S., and Wilcox, L. 2000. A semi-automatic approach to home video editing. In Proceedings of the Annual ACM Symposium on User Interface Software and Technology. 81--89. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Gleicher, M. 1997. Projective registration with difference decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 331--337. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Gleicher, M. and Liu, F. 2007. Re-cinematography: Improving the camera dynamics of casual video. In Proceedings of the 15th International Conference on Multimedia. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Govindu, V. M. 2004. Lie-algebraic averaging for globally consistent motion estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarGoogle ScholarCross RefCross Ref
  24. Hansen, M. and Mcdowel, L. 2001. Apparatus and method for removing blank areas from real-time stabilized images by inserting background information. U.S. Patent 6211913.Google ScholarGoogle Scholar
  25. Heck, R., Wallick, M., and Gleicher, M. 2007. Virtual videography. ACM Trans. Multimed. Comput. Comm. Appl. 3, 1, 4. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Hua, X.-S., Lu, L., and Zhang, H.-J. 2004. Optimization-based automated home video editing system. IEEE Trans. Circ. Syst. Video Tech. 14, 5 (May), 572--583. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Irani, M. and Anandan, P. 1998. Video indexing based on mosaic representations. Proc. IEEE 86, 5, 905--921.Google ScholarGoogle ScholarCross RefCross Ref
  28. Irani, M., Anandan, P., and Hsu, S. 1995. Mosaic based representations of video sequences and their applications. In Proceedings of the International Conference on Computer Vision. 605--611. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Irani, M., Rousso, B., and Peleg, S. 1994. Recovery of ego-motion using image stabilization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 454--460.Google ScholarGoogle Scholar
  30. Johnston, O. and Thomas, F. 1981. The Illusion of Life: Disney Animation. Abbeville Press.Google ScholarGoogle Scholar
  31. Katz, S. D. 1991. Film Directing Shot by Shot: Visualizing from Concept to Screen. Michael Wiese Productions.Google ScholarGoogle Scholar
  32. Kavan, L., Collins, S., O'Sullivan, C., and Zara, J. 2006. Dual quaternions for rigid transformation blending. Tech. Rep. TCD-CS-2006-46, Trinity College Dublin.Google ScholarGoogle Scholar
  33. Kender, J. and Yeo, B.-L. 2000. On the structure and analysis of home video. In Proceedings of ACCV.Google ScholarGoogle Scholar
  34. Kirk, D., Sellen, A., Harper, R., and Wood, K. 2007. Understanding videowork. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 61--70. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Litvin, A., Konrad, J., and Karl, W. 2003. Probabilistic video stabilization using kalman filtering and mosaicking. In Proceedings of the IS&T/SPIE Symposium on Electronic Imaging, Image, and Video Comm. 663--674.Google ScholarGoogle Scholar
  36. Liu, F. and Gleicher, M. 2006. Video retargeting: Automating pan and scan. In ACM Multimed. 241--250. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60, 2, 91--110. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Matsushita, Y., Ofek, E., Ge, W., Tang, X., and Shum, H.-Y. 2006. Full-frame video stabilization with motion inpainting. IEEE Trans. Pattern Anal. Mech. Intel. 28, 7, 1150--1163. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Mei, T., Hua, X.-S., and Zhou, H.-Q. 2005. Tracking users' capture intention: A novel complementary view for home video content analysis. In Proceedings of ACM Multimedia. 531--534. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Mikolajczyk, K. and Schmid, C. 2005. A performance evaluation of local descriptors. IEEE Trans. Pattern Anal. Mech. Intel. 27, 10, 1615--1630. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Nistér, D. 2005. Preemptive RANSAC for live structure and motion estimation. Machine Vision Appl. 16, 5, 321--329. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Nocedal, J. and Wright, S. J. 2006. Numerical Optimization, 2nd ed. Springer.Google ScholarGoogle Scholar
  43. Osian, M. and Van Gool, L. 2004. Video shot characterization. Machine Vision Appl. 15, 172--177. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Pan, Z. and Ngo, C.-W. 2004. Structuring home video by snippet detection and pattern parsing. In Proceedings of the ACM SIGMM Workshop on Multimedia Information Retrieval. 69--76. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Rosenholtz, R. 1999. A simple saliency model predicts a number of motion popout phenomena. Vision Research 39, 19, 3157--3163.Google ScholarGoogle ScholarCross RefCross Ref
  46. Suh, B., Ling, H., Bederson, B. B., and Jacobs, D. W. 2003. Automatic thumbnail cropping and its effectiveness. In Proceedings of the Annual ACM Symposium on User Interface Software and Technology. 95--104. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Szeliski, R. 1996. Video mosaics for virtual environments. IEEE Comput. Graph. Appl. 16, 2, 22--30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Szeliski, R. 2006. Image alignment and stitching: A tutorial. Tech. rep. MSR-TR-2004-92, Microsoft Research.Google ScholarGoogle Scholar
  49. Teodosio, L. and Bender, W. 2005. Salient stills. ACM Trans. Multimed. Comput. Commun. Appl. 1, 1, 16--36. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Viola, P. and Jones, M. 2001. Rapid object detection using a boosted cascade of simple features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 511--518.Google ScholarGoogle Scholar
  51. Wexler, Y., Shechtman, E., and Irani, M. 2007. Space-time completion of video. IEEE Trans. Pattern Anal. Mech. Intel. 29, 3, 463--476. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Wood, D. N., Finkelstein, A., Hughes, J. F., Thayer, C. E., and Salesin, D. H. 1997. Multiperspective panoramas for cel animation. In Proceedings of the ACM SIGGRAPH International Conference on Computer Graphics and Interactive Techniques. 243--250. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Yan, W.-Q. and Kankanhalli, M. S. 2002. Detection and removal of lighting and shaking artifacts in home videos. In Proceedings of ACM Multimedia. 107--116. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Re-cinematography: Improving the camerawork of casual video

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Multimedia Computing, Communications, and Applications
          ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 5, Issue 1
          October 2008
          201 pages
          ISSN:1551-6857
          EISSN:1551-6865
          DOI:10.1145/1404880
          Issue’s Table of Contents

          Copyright © 2008 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 30 October 2008
          • Revised: 1 May 2008
          • Accepted: 1 May 2008
          • Received: 1 January 2008
          Published in tomm Volume 5, Issue 1

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!