skip to main content
research-article

GestureCards: A Hybrid Gesture Notation

Published:13 June 2019Publication History
Skip Abstract Section

Abstract

Describing gestures in detail has various advantages for project teams: communication is simplified, interaction concepts are documented, and technical decisions are supported. Common gesture notations focus on textual or graphical elements only, but we argue that hybrid approaches have various advantages, especially because some gesture traits are easier to describe with text and others with arrows or icons. We present GestureCards, a hybrid gesture notation mixing graphical and textual elements we developed to describe multi-touch gestures. To evaluate our approach, we compared how users perceive and are affected by different notations. First, we compared GestureCards with a textual notation and observed advantages in terms of speed, correctness, and confidence. Second, we asked participants to compare GestureCards, a textual, and a graphical notation and rate them. The results indicate that the participants' perception depends on the gesture, but GestureCards received consistently good ratings. Third, we monitored several participants working with GestureCards solving practical development tasks for gesture-based applications and they felt well supported by GestureCards.

References

  1. Bashar Altakrouri, Jan Gröschner, and Andreas Schrader. 2013. Documenting Natural Interactions. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). ACM, New York, NY, USA, 1173--1178. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Shahriyar Amini and Yang Li. 2013. CrowdLearner: rapidly creating mobile recognizers using crowdsourcing. In Proceedings of the 26th annual ACM symposium on User interface software and technology - UIST '13. ACM Press, New York, New York, USA, 163--172. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Lisa Anthony and Jacob O. Wobbrock. 2010. A Lightweight Multistroke Recognizer for User Interface Prototypes. In Proceedings of Graphics Interface 2010 (GI '10). Canadian Information Processing Society, Toronto, Ont., Canada, 245--252. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Lisa Anthony and Jacob O. Wobbrock. 2012. N-protractor: A Fast and Accurate Multistroke Recognizer. In Proceedings of Graphics Interface 2012 (GI '12). Canadian Information Processing Society, Toronto, Ont., Canada, 117--120. http://dl.acm.org/citation.cfm?id=2305276.2305296 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Ann Blandford. 2013. Engineering works: What is (and is not) "Engineering" for Interactive Computer Systems?. In Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems (EICS '13). ACM Press, New York, New York, USA, 285. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Matthias Book, Volker Gruhn, and Rüdiger Striemer. 2016. Tamed Agility: Pragmatic Contracting and Collaboration in Agile Software Projects. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Eunjung Choi, Heejin Kim, and Min K Chung. 2014. A Taxonomy and Notation Method for Three-Dimensional Hand Gestures. International Journal of Industrial Ergonomics, Vol. 44, 1 (2014), 171--188.Google ScholarGoogle Scholar
  8. Mauricio Cirelli and Ricardo Nakamura. 2014. A Survey on Multi-touch Gesture Recognition and Multi-touch Frameworks. In Proceedings of the 9th ACM International Conference on Interactive Tabletops and Surfaces (ITS '14). ACM Press, New York, NY, USA, 35--44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Tulio de Souza Alcantara, Jennifer Ferreira, and Frank Maurer. 2013. Interactive Prototyping of Tabletop and Surface Applications. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM Press, New York, NY, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Lucas Silva Figueiredo, Benjamin Livshits, David Molnar, and Margus Veanes. 2016. Prepose: Privacy, Security, and Reliability for Gesture-Based Programming. In 2016 IEEE Symposium on Security and Privacy (SP '16). 122--137.Google ScholarGoogle ScholarCross RefCross Ref
  11. Simon Fothergill, Helena Mentis, Pushmeet Kohli, and Sebastian Nowozin. 2012. Instructing people for training gestural interactive systems. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI '12. ACM Press, New York, New York, USA, 1737.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Roman Ganhö r and Wolfgang Spreicer. 2014. Monox: Extensible Gesture Notation for Mobile Devices. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services (MobileHCI '14). ACM, New York, NY, USA, 203--212.Google ScholarGoogle Scholar
  13. Emilien Ghomi, Guillaume Faure, Stéphane Huot, Olivier Chapuis, and Michel Beaudouin-Lafon. 2012. Using rhythmic patterns as an input method. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI '12. ACM Press, New York, NY, USA, 1253--1262. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Tomasz Hachaj and Marek R. Ogiela. 2014. Rule-Based Approach to Recognizing Human Body Poses and Gestures in Real Time. Multimedia Systems, Vol. 20, 1 (2014), 81--99. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Chris Harrison and Scott E. Hudson. 2008. Scratch Input: Creating Large, Inexpensive, Unpowered and Mobile Finger Input Surfaces. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 205--208. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Chris Harrison, Julia Schwarz, and Scott E Hudson. 2011. TapSense: Enhancing Finger Interaction on Touch Surfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 627--636. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. J. Herold and T. F. Stahovich. 2012. The 1¢ Recognizer: A Fast, Accurate, and Easy-to-Implement Handwritten Gesture Recognition Technique. In Proceedings of the International Symposium on Sketch-Based Interfaces and Modeling (SBIM '12). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 39--46. http://dl.acm.org/citation.cfm?id=2331067.2331074 Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Marc Hesenius, Tobias Griebe, Stefan Gries, and Volker Gruhn. 2014b. Automating UI Tests for Mobile Applications with Formal Gesture Descriptions. In Proceedings of the 16th International Conference on Human-computer Interaction with Mobile Devices & Services (MobileHCI '14). ACM, New York, NY, USA, 213--222.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Marc Hesenius, Tobias Griebe, and Volker Gruhn. 2014. Towards a Behavior-Oriented Specification and Testing Language for Multimodal Applications. In Proceedings of the 2014 ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '14). ACM, New York, NY, USA, 117--122.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Marc Hesenius and Volker Gruhn. 2016. Introducing GestureCards: A Prototyping Gesture Notation. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction (NordiCHI '16). ACM, New York, NY, USA, Article 111, bibinfonumpages6 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Marc Hesenius, Markus Kleffmann, and Volker Gruhn. 2016. Sketching Gesture-Based Applications in a Collaborative Working Environment with Wall-Sized Displays. In 2016 International Conference on Collaboration Technologies and Systems (CTS). IEEE Conference Publications, 327--336.Google ScholarGoogle ScholarCross RefCross Ref
  22. Marc Hesenius, Simon Sternal, and Volker Gruhn. 2017. A Multi-Touch-Recognizer for GestureCards. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '17). ACM, New York, NY, USA, 75--80.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Martin Höst, Björn Regnell, and Claes Wohlin. 2000. Using Students as Subjects--A Comparative Study of Students and Professionals in Lead-Time Impact Assessment. Empirical Software Engineering, Vol. 5, 3 (01 Nov 2000), 201--214. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Lode Hoste and Beat Signer. 2014. Criteria, Challenges and Opportunities for Gesture Programming Languages. In Proc. 1st Workshop on Engineering Gestures for Multimodal Interfaces. 22--29. http://ceur-ws.org/Vol-1190/paper4.pdfGoogle ScholarGoogle Scholar
  25. Ann Hutchinson Guest. 2005. Labanotation: The System of Analyzing and Recording Movement .Routledge.Google ScholarGoogle Scholar
  26. Ann Hutchinson Guest and Joukje Kolff. 2002. Hands, Fingers (Advanced Labanotation) .Dance Books Ltd.Google ScholarGoogle Scholar
  27. Andreas Jedlitschka, Marcus Ciolkowski, and Dietmar Pfahl. 2008. Reporting Experiments in Software Engineering. Springer London, London, 201--228.Google ScholarGoogle Scholar
  28. Dietrich Kammer, Dana Henkens, C. Henzen, and Rainer Groh. 2015. Gesture Formalization for Multitouch. Software: Practice and Experience, Vol. 45, 4 (2015), 527--548.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Dietrich Kammer, Mandy Keck, Georg Freitag, and Markus Wacker. 2010. Taxonomy and Overview of Multi-Touch Frameworks: Architecture, Scope and Features. In Workshop on Engineering Patterns for Multitouch Interfaces.Google ScholarGoogle Scholar
  30. Ju-Whan Kim, Han-Jong Kim, and Tek-Jin Nam. 2016. M.Gesture: An Acceleration-Based Gesture Authoring System on Multiple Handheld and Wearable Devices. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 2307--2318. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Ju-Whan Kim and Tek-Jin Nam. 2013. EventHurdle: Supporting Designers' Exploratory Interaction Prototyping with Gesture-based Sensors. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 267--276.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Kenrick Kin, Björn Hartmann, Tony DeRose, and Maneesh Agrawala. 2012. Proton+: A Customizable Declarative Multitouch Framework. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (2012). ACM, 477--486. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Kenrick Kin, Björn Hartmann, Tony DeRose, and Maneesh Agrawala. 2012b. Proton: Multitouch Gestures As Regular Expressions. In Proc. SIGCHI Conf. on Human Factors in Computing Systems (2012). ACM, 2885--2894. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, and J. Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. Software Engineering, IEEE Transactions on, Vol. 28, 8 (2002), 721--734.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Markus Kleffmann. 2014. AugIR: The Conceptual Design and Evaluation of an Augmented Interaction Room. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE '14). ACM, New York, NY, USA, 883--886. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Markus Kleffmann, Marc Hesenius, and Volker Gruhn. 2015a. Connecting UI and Business Processes in a Collaborative Sketching Environment. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '15). ACM, New York, NY, USA, 200--205. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Markus Kleffmann, Sebastian Röhl, Matthias Book, and Volker Gruhn. 2018. Evaluation of a Traceability Approach for Informal Freehand Sketches. Automated Software Engineering, Vol. 25, 1 (2018), 1--43. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Markus Kleffmann, Sebastian Röhl, Volker Gruhn, and Matthias Book. 2015b. Establishing and Navigating Trace Links between Elements of Informal Diagram Sketches. In 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability. 1--7. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Bö rge Kordts, Bashar Altakrouri, and Andreas Schrader. 2015. Capturing and analysing movement using depth sensors and Labanotation. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS '15. ACM Press, New York, New York, USA, 132--141. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Sven Kratz and Michael Rohs. 2010. A $3 Gesture Recognizer -- Simple Gesture Recognition for Devices Equipped with 3D Acceleration Sensors. In Proceedings of the 15th International Conference on Intelligent User Interfaces (IUI '10). ACM Press, New York, New York, USA, 341. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Effie Lai-Chong Law, Virpi Roto, Marc Hassenzahl, Arnold P.O.S. Vermeeren, and Joke Kort. 2009. Understanding, Scoping and Defining User Experience: A Survey Approach. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 719--728. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Hao Lü, James A. Fogarty, and Yang Li. 2014. Gesture Script: recognizing gestures and their structure using rendering scripts and interactively trained parts. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI '14. ACM Press, New York, New York, USA, 1685--1694. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Meredith Ringel Morris, Anqi Huang, Andreas Paepcke, and Terry Winograd. 2006. Cooperative gestures: multi-user gestural interactions for co-located groupware. In Proceedings of the SIGCHI conference on Human Factors in computing systems - CHI '06. ACM Press, New York, New York, USA, 1201.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Antti Oulasvirta, Tino Weinkauf, Myroslav Bachynskyi, and Gregorio Palmas. 2014. Gestikulieren mit Stil. Informatik-Spektrum, Vol. 37, 5 (2014), 449--453.Google ScholarGoogle ScholarCross RefCross Ref
  45. Otto Parra Gonzá lez, Sergio Espa n a, and Oscar Pastor. 2015. Including Multi-Stroke Gesture-based Interaction in User Interfaces Using a Model-Driven Method. In Proceedings of the XVI International Conference on Human Computer Interaction (Interacció n '15). ACM, New York, NY, USA, 6:1---6:8. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Gonzalo Ramos, Matthew Boulos, and Ravin Balakrishnan. 2004. Pressure Widgets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 487--494. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Christian Rendl, Patrick Greindl, Kathrin Probst, Martin Behrens, and Michael Haller. 2014. Presstures: Exploring Pressure-Sensitive Multi-Touch Gestures on Trackpads. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 431--434. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Julie Rico and Stephen Brewster. 2010. Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 887--896. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Dean Rubine. 1991. Specifying gestures by example. ACM SIGGRAPH Computer Graphics, Vol. 25, 4 (jul 1991), 329--337. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Lucio Davide Spano, Antonio Cisternino, Fabio Paternò, and Gianni Fenu. 2013. GestIT: A Declarative and Compositional Framework for Multiplatform Gesture Definition. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM, New York, NY, USA, 187--196.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Eugene M Taranta II and Joseph J LaViola Jr. 2015. Penny Pincher: A Blazing Fast, Highly Accurate $-family Recognizer. In Proceedings of the 41st Graphics Interface Conference (GI '15). Canadian Information Processing Society, Toronto, Ont., Canada, 195--202. http://dl.acm.org/citation.cfm?id=2788890.2788925 Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. G. Tassey. 2002. The Economic Impacts of Inadequate Infrastructure for Software Testing. Technical Report. National Institute of Standards and Technology .Google ScholarGoogle Scholar
  53. Radu-Daniel Vatavu, Lisa Anthony, and Jacob O. Wobbrock. 2012. Gestures as point clouds. In Proc. 14th ACM intl. conf. on Multimodal interaction - ICMI '12. ACM Press, New York, New York, USA, 273.Google ScholarGoogle Scholar
  54. Daniel Leite Viana and André Lu'is de Medeiros Santos. 2017. A Domain-Specific Language for the Specification of Gesture-based Applications. In Proceedings of the 21st Brazilian Symposium on Programming Languages (SBLP 2017). ACM, New York, NY, USA, Article 11, bibinfonumpages7 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Geoff Walker. 2012. A Review of Technologies for Sensing Contact Location on the Surface of a Display. Journal of the Society for Information Display, Vol. 20, 8 (2012), 413--440.Google ScholarGoogle ScholarCross RefCross Ref
  56. Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-Defined Gestures for Surface Computing. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 1083. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures Without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes. In Proc. 20th Annual ACM Symp. User Interface Software and Technology (UIST '07). ACM, New York, NY, USA, 159--168. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Yang Zhang and Chris Harrison. 2015. Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 167--173. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. SkinTrack: Using the Body As an Electrical Waveguide for Continuous Finger Tracking on the Skin. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1491--1503. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. GestureCards: A Hybrid Gesture Notation

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!