ABSTRACT

Illustrations of human movements are used to communicate ideas and convey instructions in many domains, but creating them is time-consuming and requires skill. We introduce DemoDraw, a multi-modal approach to generate these illustrations as the user physically demonstrates the movements. In a Demonstration Interface, DemoDraw segments speech and 3D joint motion into a sequence of motion segments, each characterized by a key pose and salient joint trajectories. Based on this sequence, a series of illustrations is automatically generated using a stylistically rendered 3D avatar annotated with arrows to convey movements. During demonstration, the user can navigate using speech and amend or re-perform motions if needed. Once a suitable sequence of steps has been created, a Refinement Interface enables fine control of visualization parameters. In a three-part evaluation, we validate the effectiveness of the generated illustrations and the usability of DemoDraw. Our results show 4 to 7-step illustrations can be created in 5 or 10 minutes on average.
References
- J.K. Aggarwal and M.S. Ryoo. 2011. Human Activity Analysis: A Review. ACM Comput. Surv. 43, 3, Article 16 (April 2011), 43 pages. DOI: http://dx.doi.org/10.1145/1922649.1922653 Google Scholar
Digital Library
- Maneesh Agrawala, Wilmot Li, and Floraine Berthouzoz. 2011. Design Principles for Visual Communication. Commun. ACM 54, 4 (April 2011), 60--69. DOI: http://dx.doi.org/10.1145/1924421.1924439 Google Scholar
Digital Library
- Fraser Anderson, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2013. YouMove: enhancing movement training with an augmented reality mirror. In Proceedings of UIST '13. ACM, 311--320. Google Scholar
Digital Library
- Robert Anderson and Jean Anderson. 2010. Before and After Weight Training. In Stretching. Shelter Publications, Bolinas, California, 210--211.Google Scholar
- Jackie Assa, Yaron Caspi, and Daniel Cohen-Or. 2005. Action synopsis: pose selection and illustration. ACM Transactions on Graphics (TOG) 24, 3 (2005), 667--676. Google Scholar
Digital Library
- Jackie Assa, Daniel Cohen-Or, I-Cheng Yeh, Tong-Yee Lee, and others. 2008. Motion overview of human actions. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 115. Google Scholar
Digital Library
- Ronald M. Baecker. 1969. Picture-driven animation. In Proceedings of the May 14--16, 1969, spring joint computer conference. ACM, Boston, Massachusetts, 273--288. DOI: http://dx.doi.org/10.1145/1476793.1476838 Google Scholar
Digital Library
- Connelly Barnes, David E. Jacobs, Jason Sanders, Dan B Goldman, Szymon Rusinkiewicz, Adam Finkelstein, and Maneesh Agrawala. 2008. Video Puppetry: A Performative Interface for Cutout Animation. In ACM SIGGRAPH Asia 2008. ACM, New York, NY, USA, 124:1--124:9. DOI: http://dx.doi.org/10.1145/1457515.1409077 Google Scholar
Digital Library
- Lawrence Bergman, Vittorio Castelli, Tessa Lau, and Daniel Oblinger. 2005. DocWizards: A System for Authoring Follow-me Documentation Wizards. In Proceedings of UIST '05. ACM, New York, NY, USA, 191--200. DOI: http://dx.doi.org/10.1145/1095034.1095067 Google Scholar
Digital Library
- Richard A. Bolt. 1980. Put-that-there: Voice and Gesture at the Graphics Interface. In Proceedings of SIGGRAPH '80. ACM, New York, NY, USA, 262--270. DOI: http://dx.doi.org/10.1145/800250.807503 Google Scholar
Digital Library
- Simon Bouvier-Zappa, Victor Ostromoukhov, and Pierre Poulin. 2007. Motion cues for illustration of skeletal motion capture data. In Proceedings of the 5th international symposium on Non-photorealistic animation and rendering. ACM, 133--140. Google Scholar
Digital Library
- Aaron Bryden, George Phillips Jr., and Michael Gleicher. 2012. Automated Illustration of Molecular Flexibility. IEEE Transactions on Visualization and Computer Graphics 18, 1 (Jan. 2012), 132--145. DOI: http://dx.doi.org/10.1109/TVCG.2010.250 Google Scholar
Digital Library
- Bill Buxton. 2007. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. Google Scholar
Digital Library
- Pei-Yu Chi, Sally Ahn, Amanda Ren, Mira Dontcheva, Wilmot Li, and Björn Hartmann. 2012. MixT: automatic generation of step-by-step mixed media tutorials. In Proceedings of UIST '12. ACM, 93--102. Google Scholar
Digital Library
- Pei-Yu Chi, Joyce Liu, Jason Linder, Mira Dontcheva, Wilmot Li, and Bjoern Hartmann. 2013. Democut: generating concise instructional videos for physical demonstrations. In Proceedings of UIST '13. ACM, 141--150. Google Scholar
Digital Library
- Myung Geol Choi, Kyungyong Yang, Takeo Igarashi, Jun Mitani, and Jehee Lee. 2012. Retrieval and visualization of human motion data via stick figures. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 2057--2065. Google Scholar
Digital Library
- Gabe Cohn, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Humantenna: using the body as an antenna for real-time whole-body interaction. In Proceedings of CHI '12. ACM, 1901--1910. Google Scholar
Digital Library
- Jonathan Corum. 2012. Drawing Science in Sign. (2012). http://style.org/sign/Google Scholar
- James E. Cutting. 2002. Representing motion in a static image: constraints and parallels in art, science, and popular culture. Perception 31, 10 (2002), 1165--93. DOI: http://dx.doi.org/10.1068/p3318 Google Scholar
Cross Ref
- Mira Dontcheva, Gary Yngve, and Zoran Popović. 2003. Layered Acting for Character Animation. In ACM SIGGRAPH '03. ACM, New York, NY, USA, 409--416. DOI:http://dx.doi.org/10.1145/1201775.882285 Google Scholar
Digital Library
- Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, Pushmeet Kohli, Vladimir Tankovich, and Shahram Izadi. 2016. Fusion4D: Real-time Performance Capture of Challenging Scenes. In ACM SIGGRAPH '16. Google Scholar
Digital Library
- Michael Gleicher. 1998. Retargetting motion to new characters. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. ACM, 33--42. Google Scholar
Digital Library
- Dan B Goldman, Brian Curless, David Salesin, and Steven M Seitz. 2006. Schematic storyboarding for video visualization and editing. In ACM Transactions on Graphics (TOG), Vol. 25. ACM, 862--871. Google Scholar
Digital Library
- Amy Gooch, Bruce Gooch, Peter Shirley, and Elaine Cohen. 1998. A non-photorealistic lighting model for automatic technical illustration. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. ACM, 447--452. Google Scholar
Digital Library
- Floraine Grabler, Maneesh Agrawala, Wilmot Li, Mira Dontcheva, and Takeo Igarashi. 2009. Generating photo manipulation tutorials by demonstration. ACM Transactions on Graphics (TOG) 28, 3 (2009), 66. Google Scholar
Digital Library
- Saul Greenberg, Sheelagh Carpendale, Nicolai Marquardt, and Bill Buxton. 2012. Sketching User Experiences: The Workbook. Elsevier/Morgan Kaufmann. https://books.google.com/books?id=c-RAUXk3gbkC Google Scholar
Digital Library
- Ankit Gupta, Maneesh Agrawala, Brian Curless, and Michael Cohen. 2014. MotionMontage: A System to Annotate and Combine Motion Takes for 3D Animations. In Proceedings of CHI '14 (CHI '14). ACM, New York, NY, USA, 2017--2026. DOI: http://dx.doi.org/10.1145/2556288.2557218 Google Scholar
Digital Library
- Ankit Gupta, Dieter Fox, Brian Curless, and Michael Cohen. 2012. DuploTrack: A Reatime System for Authoring and Guiding Duplo Model Assembly. In Proceedings of UIST '12. ACM, New York, NY, USA, 13. Google Scholar
Digital Library
- Robert Held, Ankit Gupta, Brian Curless, and Maneesh Agrawala. 2012. 3D puppetry: a kinect-based interface for 3D animation.. In Proceedings of UIST '12. Citeseer, 423--434. Google Scholar
Digital Library
- Natasha Kholgade, Tomas Simon, Alexei Efros, and Yaser Sheikh. 2014. 3D object manipulation in a single photograph using stock 3D models. ACM Transactions on Graphics (TOG) 33, 4 (2014), 127. Google Scholar
Digital Library
- Wilmot Li, Maneesh Agrawala, Brian Curless, and David Salesin. 2008. Automated generation of interactive 3D exploded view diagrams. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 101. Google Scholar
Digital Library
- David Lumb. 2013. "Manga Generator" Uses The Kinect To Put Your Smooth Moves In A Custom Comic. (Sept. 2013). http://www.fastcolabs.com/3016870Google Scholar
- Scott McCloud. 1994. Understanding Comics (reprint edition ed.). Avon, New York.Google Scholar
- P. Mijksenaar and P. Westendorp. 1999. Open here: the art of instructional design. Joost Elffers Books. https://books.google.com/books?id=fsJVAAAAMAAJGoogle Scholar
- Niloy J Mitra, Yong-Liang Yang, Dong-Ming Yan, Wilmot Li, and Maneesh Agrawala. 2010. Illustrating how mechanical assemblies work. ACM Transactions on Graphics-TOG 29, 4 (2010), 58. Google Scholar
Digital Library
- Peter Mohr, Bernhard Kerbl, Michael Donoser, Dieter Schmalstieg, and Denis Kalkofen. 2015. Retargeting Technical Documentation to Augmented Reality. In Proceedings of CHI '15. ACM, New York, NY, USA, 3337--3346. DOI: http://dx.doi.org/10.1145/2702123.2702490 Google Scholar
Digital Library
- Dorée Duncan Seligmann and Steven Feiner. 1991. Automated Generation of Intent-based 3D Illustrations. SIGGRAPH Comput. Graph. 25, 4 (July 1991), 123--132. DOI:http://dx.doi.org/10.1145/127719.122732 Google Scholar
Digital Library
- Richard Tang, Hesam Alizadeh, Anthony Tang, Scott Bateman, and Joaquim A.P. Jorge. 2014. [email protected]: Design Explorations to Support Movement Guidance. In CHI EA '14. ACM, New York, NY, USA, 1651--1656. DOI:http://dx.doi.org/10.1145/2559206.2581197 Google Scholar
Digital Library
- Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Arran Topalian, Erroll Wood, Sameh Khamis, Pushmeet Kohli, Toby Sharp, Shahram Izadi, Richard Banks, Andrew Fitzgibbon, and Jamie Shotton. 2016. Efficient and Precise Interactive Hand Tracking through Joint, Continuous Optimization of Pose and Correspondences. In ACM SIGGRAPH '16. Google Scholar
Digital Library
- Edward Tufte. 1990. Layering and Separation. Graphics Press, Cheshire, Connecticut.Google Scholar
- Ross Yeager. 2013. An Automated Physiotherapy Exercise Generator. Master's thesis. EECS Department, University of California, Berkeley. http://www.eecs. berkeley.edu/Pubs/TechRpts/2013/EECS-2013-91.htmlGoogle Scholar
- Yupeng Zhang, Teng Han, Zhimin Ren, Nobuyuki Umetani, Xin Tong, Yang Liu, Takaaki Shiratori, and Xiang Cao. 2013. BodyAvatar: Creating Freeform 3D Avatars Using First-person Body Gestures. In Proceedings of UIST '13 (UIST '13). ACM, New York, NY, USA, 387--396. DOI: http://dx.doi.org/10.1145/2501988.2502015 Google Scholar
Digital Library
Supplemental Material
Index Terms
Authoring Illustrations of Human Movements by Iterative Physical Demonstration


Pei-Yu (Peggy) Chi
Björn Hartmann


Comments