10.1145/2984511.2984559acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedings
research-article
Open Access

Authoring Illustrations of Human Movements by Iterative Physical Demonstration

ABSTRACT

Illustrations of human movements are used to communicate ideas and convey instructions in many domains, but creating them is time-consuming and requires skill. We introduce DemoDraw, a multi-modal approach to generate these illustrations as the user physically demonstrates the movements. In a Demonstration Interface, DemoDraw segments speech and 3D joint motion into a sequence of motion segments, each characterized by a key pose and salient joint trajectories. Based on this sequence, a series of illustrations is automatically generated using a stylistically rendered 3D avatar annotated with arrows to convey movements. During demonstration, the user can navigate using speech and amend or re-perform motions if needed. Once a suitable sequence of steps has been created, a Refinement Interface enables fine control of visualization parameters. In a three-part evaluation, we validate the effectiveness of the generated illustrations and the usability of DemoDraw. Our results show 4 to 7-step illustrations can be created in 5 or 10 minutes on average.

References

  1. J.K. Aggarwal and M.S. Ryoo. 2011. Human Activity Analysis: A Review. ACM Comput. Surv. 43, 3, Article 16 (April 2011), 43 pages. DOI: http://dx.doi.org/10.1145/1922649.1922653 Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Maneesh Agrawala, Wilmot Li, and Floraine Berthouzoz. 2011. Design Principles for Visual Communication. Commun. ACM 54, 4 (April 2011), 60--69. DOI: http://dx.doi.org/10.1145/1924421.1924439 Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Fraser Anderson, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2013. YouMove: enhancing movement training with an augmented reality mirror. In Proceedings of UIST '13. ACM, 311--320. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Robert Anderson and Jean Anderson. 2010. Before and After Weight Training. In Stretching. Shelter Publications, Bolinas, California, 210--211.Google ScholarGoogle Scholar
  5. Jackie Assa, Yaron Caspi, and Daniel Cohen-Or. 2005. Action synopsis: pose selection and illustration. ACM Transactions on Graphics (TOG) 24, 3 (2005), 667--676. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Jackie Assa, Daniel Cohen-Or, I-Cheng Yeh, Tong-Yee Lee, and others. 2008. Motion overview of human actions. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 115. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Ronald M. Baecker. 1969. Picture-driven animation. In Proceedings of the May 14--16, 1969, spring joint computer conference. ACM, Boston, Massachusetts, 273--288. DOI: http://dx.doi.org/10.1145/1476793.1476838 Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Connelly Barnes, David E. Jacobs, Jason Sanders, Dan B Goldman, Szymon Rusinkiewicz, Adam Finkelstein, and Maneesh Agrawala. 2008. Video Puppetry: A Performative Interface for Cutout Animation. In ACM SIGGRAPH Asia 2008. ACM, New York, NY, USA, 124:1--124:9. DOI: http://dx.doi.org/10.1145/1457515.1409077 Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Lawrence Bergman, Vittorio Castelli, Tessa Lau, and Daniel Oblinger. 2005. DocWizards: A System for Authoring Follow-me Documentation Wizards. In Proceedings of UIST '05. ACM, New York, NY, USA, 191--200. DOI: http://dx.doi.org/10.1145/1095034.1095067 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Richard A. Bolt. 1980. Put-that-there: Voice and Gesture at the Graphics Interface. In Proceedings of SIGGRAPH '80. ACM, New York, NY, USA, 262--270. DOI: http://dx.doi.org/10.1145/800250.807503 Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Simon Bouvier-Zappa, Victor Ostromoukhov, and Pierre Poulin. 2007. Motion cues for illustration of skeletal motion capture data. In Proceedings of the 5th international symposium on Non-photorealistic animation and rendering. ACM, 133--140. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Aaron Bryden, George Phillips Jr., and Michael Gleicher. 2012. Automated Illustration of Molecular Flexibility. IEEE Transactions on Visualization and Computer Graphics 18, 1 (Jan. 2012), 132--145. DOI: http://dx.doi.org/10.1109/TVCG.2010.250 Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Bill Buxton. 2007. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Pei-Yu Chi, Sally Ahn, Amanda Ren, Mira Dontcheva, Wilmot Li, and Björn Hartmann. 2012. MixT: automatic generation of step-by-step mixed media tutorials. In Proceedings of UIST '12. ACM, 93--102. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Pei-Yu Chi, Joyce Liu, Jason Linder, Mira Dontcheva, Wilmot Li, and Bjoern Hartmann. 2013. Democut: generating concise instructional videos for physical demonstrations. In Proceedings of UIST '13. ACM, 141--150. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Myung Geol Choi, Kyungyong Yang, Takeo Igarashi, Jun Mitani, and Jehee Lee. 2012. Retrieval and visualization of human motion data via stick figures. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 2057--2065. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Gabe Cohn, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Humantenna: using the body as an antenna for real-time whole-body interaction. In Proceedings of CHI '12. ACM, 1901--1910. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Jonathan Corum. 2012. Drawing Science in Sign. (2012). http://style.org/sign/Google ScholarGoogle Scholar
  19. James E. Cutting. 2002. Representing motion in a static image: constraints and parallels in art, science, and popular culture. Perception 31, 10 (2002), 1165--93. DOI: http://dx.doi.org/10.1068/p3318 Google ScholarGoogle ScholarCross RefCross Ref
  20. Mira Dontcheva, Gary Yngve, and Zoran Popović. 2003. Layered Acting for Character Animation. In ACM SIGGRAPH '03. ACM, New York, NY, USA, 409--416. DOI:http://dx.doi.org/10.1145/1201775.882285 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, Pushmeet Kohli, Vladimir Tankovich, and Shahram Izadi. 2016. Fusion4D: Real-time Performance Capture of Challenging Scenes. In ACM SIGGRAPH '16. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Michael Gleicher. 1998. Retargetting motion to new characters. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. ACM, 33--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Dan B Goldman, Brian Curless, David Salesin, and Steven M Seitz. 2006. Schematic storyboarding for video visualization and editing. In ACM Transactions on Graphics (TOG), Vol. 25. ACM, 862--871. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Amy Gooch, Bruce Gooch, Peter Shirley, and Elaine Cohen. 1998. A non-photorealistic lighting model for automatic technical illustration. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. ACM, 447--452. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Floraine Grabler, Maneesh Agrawala, Wilmot Li, Mira Dontcheva, and Takeo Igarashi. 2009. Generating photo manipulation tutorials by demonstration. ACM Transactions on Graphics (TOG) 28, 3 (2009), 66. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Saul Greenberg, Sheelagh Carpendale, Nicolai Marquardt, and Bill Buxton. 2012. Sketching User Experiences: The Workbook. Elsevier/Morgan Kaufmann. https://books.google.com/books?id=c-RAUXk3gbkC Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Ankit Gupta, Maneesh Agrawala, Brian Curless, and Michael Cohen. 2014. MotionMontage: A System to Annotate and Combine Motion Takes for 3D Animations. In Proceedings of CHI '14 (CHI '14). ACM, New York, NY, USA, 2017--2026. DOI: http://dx.doi.org/10.1145/2556288.2557218 Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Ankit Gupta, Dieter Fox, Brian Curless, and Michael Cohen. 2012. DuploTrack: A Reatime System for Authoring and Guiding Duplo Model Assembly. In Proceedings of UIST '12. ACM, New York, NY, USA, 13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Robert Held, Ankit Gupta, Brian Curless, and Maneesh Agrawala. 2012. 3D puppetry: a kinect-based interface for 3D animation.. In Proceedings of UIST '12. Citeseer, 423--434. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Natasha Kholgade, Tomas Simon, Alexei Efros, and Yaser Sheikh. 2014. 3D object manipulation in a single photograph using stock 3D models. ACM Transactions on Graphics (TOG) 33, 4 (2014), 127. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Wilmot Li, Maneesh Agrawala, Brian Curless, and David Salesin. 2008. Automated generation of interactive 3D exploded view diagrams. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 101. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. David Lumb. 2013. "Manga Generator" Uses The Kinect To Put Your Smooth Moves In A Custom Comic. (Sept. 2013). http://www.fastcolabs.com/3016870Google ScholarGoogle Scholar
  33. Scott McCloud. 1994. Understanding Comics (reprint edition ed.). Avon, New York.Google ScholarGoogle Scholar
  34. P. Mijksenaar and P. Westendorp. 1999. Open here: the art of instructional design. Joost Elffers Books. https://books.google.com/books?id=fsJVAAAAMAAJGoogle ScholarGoogle Scholar
  35. Niloy J Mitra, Yong-Liang Yang, Dong-Ming Yan, Wilmot Li, and Maneesh Agrawala. 2010. Illustrating how mechanical assemblies work. ACM Transactions on Graphics-TOG 29, 4 (2010), 58. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Peter Mohr, Bernhard Kerbl, Michael Donoser, Dieter Schmalstieg, and Denis Kalkofen. 2015. Retargeting Technical Documentation to Augmented Reality. In Proceedings of CHI '15. ACM, New York, NY, USA, 3337--3346. DOI: http://dx.doi.org/10.1145/2702123.2702490 Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Dorée Duncan Seligmann and Steven Feiner. 1991. Automated Generation of Intent-based 3D Illustrations. SIGGRAPH Comput. Graph. 25, 4 (July 1991), 123--132. DOI:http://dx.doi.org/10.1145/127719.122732 Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Richard Tang, Hesam Alizadeh, Anthony Tang, Scott Bateman, and Joaquim A.P. Jorge. 2014. [email protected]: Design Explorations to Support Movement Guidance. In CHI EA '14. ACM, New York, NY, USA, 1651--1656. DOI:http://dx.doi.org/10.1145/2559206.2581197 Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Arran Topalian, Erroll Wood, Sameh Khamis, Pushmeet Kohli, Toby Sharp, Shahram Izadi, Richard Banks, Andrew Fitzgibbon, and Jamie Shotton. 2016. Efficient and Precise Interactive Hand Tracking through Joint, Continuous Optimization of Pose and Correspondences. In ACM SIGGRAPH '16. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Edward Tufte. 1990. Layering and Separation. Graphics Press, Cheshire, Connecticut.Google ScholarGoogle Scholar
  41. Ross Yeager. 2013. An Automated Physiotherapy Exercise Generator. Master's thesis. EECS Department, University of California, Berkeley. http://www.eecs. berkeley.edu/Pubs/TechRpts/2013/EECS-2013-91.htmlGoogle ScholarGoogle Scholar
  42. Yupeng Zhang, Teng Han, Zhimin Ren, Nobuyuki Umetani, Xin Tong, Yang Liu, Takaaki Shiratori, and Xiang Cao. 2013. BodyAvatar: Creating Freeform 3D Avatars Using First-person Body Gestures. In Proceedings of UIST '13 (UIST '13). ACM, New York, NY, USA, 387--396. DOI: http://dx.doi.org/10.1145/2501988.2502015 Google ScholarGoogle ScholarDigital LibraryDigital Library

Supplemental Material

p809-chi.mp4

Index Terms

  1. Authoring Illustrations of Human Movements by Iterative Physical Demonstration

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!