Automatic cinemagraph portraits

ABSTRACT

Cinemagraphs are a popular new type of visual media that lie in-between photos and video; some parts of the frame are animated and loop seamlessly, while other parts of the frame remain completely still. Cinemagraphs are especially effective for portraits because they capture the nuances of our dynamic facial expressions. We present a completely automatic algorithm for generating portrait cinemagraphs from a short video captured with a hand-held camera. Our algorithm uses a combination of face tracking and point tracking to segment face motions into two classes: gross, large-scale motions that should be removed from the video, and dynamic facial expressions that should be preserved. This segmentation informs a spatially-varying warp that removes the large-scale motion, and a graph-cut segmentation of the frame into dynamic and still regions that preserves the finer-scale facial expression motions. We demonstrate the success of our method with a variety of results and a comparison to previous work.

References

1. URL: http://cinemagraphs.com. 1
2. URL: http://www.filmindustrynetwork.biz/nyc-photographer-jamie-beck-cinemagraph/12173. 1
3. URL: http://kinotopic.com. 1
4. URL: http://cinemagr.am. 1
5. URL: http://www.icinegraph.com. 1
6. {ADA*04} Agarwala A., Dontcheva M., Agrawala M., Drucker S., Colburn A., Curless B., Salesin D., Cohen M.: Interactive digital photomontage. ACM Transactions on Graphics 23, 3 (Aug. 2004), 294--302. 2, 6
7. {BAAR12} Bai J., Agarwala A., Agrawala M., Ramamoorthi R.: Selectively de-animating video. ACM Transactions on Graphics (2012). 1, 2, 3, 5, 7
8. {BVZ01} Boykov Y., Veksler O., Zabih R.: Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11 (Nov. 2001), 1222--1239. 2, 7
9. {FB81} Fischler M. A., Bolles R. C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6 (June 1981), 381--395. 2
10. {FH04} Felzenszwalb P. F., Huttenlocher D. P.: Distance transforms of sampled functions. Tech. rep., Cornell Computing and Information Science, 2004. 6
11. {JMD*12} Joshi N., Mehta S., Drucker S., Stollnitz E., Hoppe H., Uyttendaele M., Cohen M.: Cliplets: juxtaposing still and dynamic imagery. In Proceedings of the 25th annual ACM symposium on User interface software and technology (2012), UIST '12, pp. 251--260. 1
12. {KSE*03} Kwatra V., Schödl A., Essa I., Turk G., Bobick A.: Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics 22, 3 (July 2003), 277--286. 2
13. {LGJA09} Liu F., Gleicher M., Jin H., Agarwala A.: Content-preserving warps for 3d video stabilization. ACM Transactions on Graphics 28, 3 (July 2009), 44:1--44:9. 1, 2
14. {Liu09} Liu C.: Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. PhD thesis, Massachusetts Institute of Technology, May 2009. 5
15. {LJH13} Liao Z., Joshi N., Hoppe H.: Automated video looping with progressive dynamism. ACM Transactions on Graphics (2013). 1
16. {LK81} Lucas B. D., Kanade T.: An iterative image registration technique with an application to stereo vision. International Joint Conference on Artificial Intelligence (1981). 1
17. {SLC11} Saragih J. M., Lucey S., Cohn J. F.: Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vision 91, 2 (Jan. 2011), 200--215. 1, 3
18. {SSSE00} Schödl A., Szeliski R., Salesin D. H., Essa I.: Video textures. In Proceedings of ACM SIGGRAPH 2000 (July 2000), Computer Graphics Proceedings, Annual Conference Series, pp. 489--498. 5
19. {TPSK11} Tompkin J., Pece F., Subr K., Kautz J.: Towards moment imagery: Automatic cinemagraphs. Visual Media Production, Conference for 0 (2011), 87--93. 1
20. {YL12} Yeh M.-C., Li P.-Y.: A tool for automatic cinemagraphs. In Proceedings of the 20th ACM international conference on Multimedia (2012), MM '12, pp. 1259--1260. 1

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Graphics recognition and interpretation

• Computing methodologies

  • Computer graphics