Abstract
Describing gestures in detail has various advantages for project teams: communication is simplified, interaction concepts are documented, and technical decisions are supported. Common gesture notations focus on textual or graphical elements only, but we argue that hybrid approaches have various advantages, especially because some gesture traits are easier to describe with text and others with arrows or icons. We present GestureCards, a hybrid gesture notation mixing graphical and textual elements we developed to describe multi-touch gestures. To evaluate our approach, we compared how users perceive and are affected by different notations. First, we compared GestureCards with a textual notation and observed advantages in terms of speed, correctness, and confidence. Second, we asked participants to compare GestureCards, a textual, and a graphical notation and rate them. The results indicate that the participants' perception depends on the gesture, but GestureCards received consistently good ratings. Third, we monitored several participants working with GestureCards solving practical development tasks for gesture-based applications and they felt well supported by GestureCards.
- Bashar Altakrouri, Jan Gröschner, and Andreas Schrader. 2013. Documenting Natural Interactions. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). ACM, New York, NY, USA, 1173--1178. Google Scholar
Digital Library
- Shahriyar Amini and Yang Li. 2013. CrowdLearner: rapidly creating mobile recognizers using crowdsourcing. In Proceedings of the 26th annual ACM symposium on User interface software and technology - UIST '13. ACM Press, New York, New York, USA, 163--172. Google Scholar
Digital Library
- Lisa Anthony and Jacob O. Wobbrock. 2010. A Lightweight Multistroke Recognizer for User Interface Prototypes. In Proceedings of Graphics Interface 2010 (GI '10). Canadian Information Processing Society, Toronto, Ont., Canada, 245--252. Google Scholar
Digital Library
- Lisa Anthony and Jacob O. Wobbrock. 2012. N-protractor: A Fast and Accurate Multistroke Recognizer. In Proceedings of Graphics Interface 2012 (GI '12). Canadian Information Processing Society, Toronto, Ont., Canada, 117--120. http://dl.acm.org/citation.cfm?id=2305276.2305296 Google Scholar
Digital Library
- Ann Blandford. 2013. Engineering works: What is (and is not) "Engineering" for Interactive Computer Systems?. In Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems (EICS '13). ACM Press, New York, New York, USA, 285. Google Scholar
Digital Library
- Matthias Book, Volker Gruhn, and Rüdiger Striemer. 2016. Tamed Agility: Pragmatic Contracting and Collaboration in Agile Software Projects. Springer. Google Scholar
Digital Library
- Eunjung Choi, Heejin Kim, and Min K Chung. 2014. A Taxonomy and Notation Method for Three-Dimensional Hand Gestures. International Journal of Industrial Ergonomics, Vol. 44, 1 (2014), 171--188.Google Scholar
- Mauricio Cirelli and Ricardo Nakamura. 2014. A Survey on Multi-touch Gesture Recognition and Multi-touch Frameworks. In Proceedings of the 9th ACM International Conference on Interactive Tabletops and Surfaces (ITS '14). ACM Press, New York, NY, USA, 35--44. Google Scholar
Digital Library
- Tulio de Souza Alcantara, Jennifer Ferreira, and Frank Maurer. 2013. Interactive Prototyping of Tabletop and Surface Applications. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM Press, New York, NY, USA. Google Scholar
Digital Library
- Lucas Silva Figueiredo, Benjamin Livshits, David Molnar, and Margus Veanes. 2016. Prepose: Privacy, Security, and Reliability for Gesture-Based Programming. In 2016 IEEE Symposium on Security and Privacy (SP '16). 122--137.Google Scholar
Cross Ref
- Simon Fothergill, Helena Mentis, Pushmeet Kohli, and Sebastian Nowozin. 2012. Instructing people for training gestural interactive systems. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI '12. ACM Press, New York, New York, USA, 1737.Google Scholar
Digital Library
- Roman Ganhö r and Wolfgang Spreicer. 2014. Monox: Extensible Gesture Notation for Mobile Devices. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services (MobileHCI '14). ACM, New York, NY, USA, 203--212.Google Scholar
- Emilien Ghomi, Guillaume Faure, Stéphane Huot, Olivier Chapuis, and Michel Beaudouin-Lafon. 2012. Using rhythmic patterns as an input method. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI '12. ACM Press, New York, NY, USA, 1253--1262. Google Scholar
Digital Library
- Tomasz Hachaj and Marek R. Ogiela. 2014. Rule-Based Approach to Recognizing Human Body Poses and Gestures in Real Time. Multimedia Systems, Vol. 20, 1 (2014), 81--99. Google Scholar
Digital Library
- Chris Harrison and Scott E. Hudson. 2008. Scratch Input: Creating Large, Inexpensive, Unpowered and Mobile Finger Input Surfaces. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 205--208. Google Scholar
Digital Library
- Chris Harrison, Julia Schwarz, and Scott E Hudson. 2011. TapSense: Enhancing Finger Interaction on Touch Surfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 627--636. Google Scholar
Digital Library
- J. Herold and T. F. Stahovich. 2012. The 1¢ Recognizer: A Fast, Accurate, and Easy-to-Implement Handwritten Gesture Recognition Technique. In Proceedings of the International Symposium on Sketch-Based Interfaces and Modeling (SBIM '12). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 39--46. http://dl.acm.org/citation.cfm?id=2331067.2331074 Google Scholar
Digital Library
- Marc Hesenius, Tobias Griebe, Stefan Gries, and Volker Gruhn. 2014b. Automating UI Tests for Mobile Applications with Formal Gesture Descriptions. In Proceedings of the 16th International Conference on Human-computer Interaction with Mobile Devices & Services (MobileHCI '14). ACM, New York, NY, USA, 213--222.Google Scholar
Digital Library
- Marc Hesenius, Tobias Griebe, and Volker Gruhn. 2014. Towards a Behavior-Oriented Specification and Testing Language for Multimodal Applications. In Proceedings of the 2014 ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '14). ACM, New York, NY, USA, 117--122.Google Scholar
Digital Library
- Marc Hesenius and Volker Gruhn. 2016. Introducing GestureCards: A Prototyping Gesture Notation. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction (NordiCHI '16). ACM, New York, NY, USA, Article 111, bibinfonumpages6 pages.Google Scholar
Digital Library
- Marc Hesenius, Markus Kleffmann, and Volker Gruhn. 2016. Sketching Gesture-Based Applications in a Collaborative Working Environment with Wall-Sized Displays. In 2016 International Conference on Collaboration Technologies and Systems (CTS). IEEE Conference Publications, 327--336.Google Scholar
Cross Ref
- Marc Hesenius, Simon Sternal, and Volker Gruhn. 2017. A Multi-Touch-Recognizer for GestureCards. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '17). ACM, New York, NY, USA, 75--80.Google Scholar
Digital Library
- Martin Höst, Björn Regnell, and Claes Wohlin. 2000. Using Students as Subjects--A Comparative Study of Students and Professionals in Lead-Time Impact Assessment. Empirical Software Engineering, Vol. 5, 3 (01 Nov 2000), 201--214. Google Scholar
Digital Library
- Lode Hoste and Beat Signer. 2014. Criteria, Challenges and Opportunities for Gesture Programming Languages. In Proc. 1st Workshop on Engineering Gestures for Multimodal Interfaces. 22--29. http://ceur-ws.org/Vol-1190/paper4.pdfGoogle Scholar
- Ann Hutchinson Guest. 2005. Labanotation: The System of Analyzing and Recording Movement .Routledge.Google Scholar
- Ann Hutchinson Guest and Joukje Kolff. 2002. Hands, Fingers (Advanced Labanotation) .Dance Books Ltd.Google Scholar
- Andreas Jedlitschka, Marcus Ciolkowski, and Dietmar Pfahl. 2008. Reporting Experiments in Software Engineering. Springer London, London, 201--228.Google Scholar
- Dietrich Kammer, Dana Henkens, C. Henzen, and Rainer Groh. 2015. Gesture Formalization for Multitouch. Software: Practice and Experience, Vol. 45, 4 (2015), 527--548.Google Scholar
Digital Library
- Dietrich Kammer, Mandy Keck, Georg Freitag, and Markus Wacker. 2010. Taxonomy and Overview of Multi-Touch Frameworks: Architecture, Scope and Features. In Workshop on Engineering Patterns for Multitouch Interfaces.Google Scholar
- Ju-Whan Kim, Han-Jong Kim, and Tek-Jin Nam. 2016. M.Gesture: An Acceleration-Based Gesture Authoring System on Multiple Handheld and Wearable Devices. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 2307--2318. Google Scholar
Digital Library
- Ju-Whan Kim and Tek-Jin Nam. 2013. EventHurdle: Supporting Designers' Exploratory Interaction Prototyping with Gesture-based Sensors. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 267--276.Google Scholar
Digital Library
- Kenrick Kin, Björn Hartmann, Tony DeRose, and Maneesh Agrawala. 2012. Proton+: A Customizable Declarative Multitouch Framework. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (2012). ACM, 477--486. Google Scholar
Digital Library
- Kenrick Kin, Björn Hartmann, Tony DeRose, and Maneesh Agrawala. 2012b. Proton: Multitouch Gestures As Regular Expressions. In Proc. SIGCHI Conf. on Human Factors in Computing Systems (2012). ACM, 2885--2894. Google Scholar
Digital Library
- B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, and J. Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. Software Engineering, IEEE Transactions on, Vol. 28, 8 (2002), 721--734.Google Scholar
Digital Library
- Markus Kleffmann. 2014. AugIR: The Conceptual Design and Evaluation of an Augmented Interaction Room. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE '14). ACM, New York, NY, USA, 883--886. Google Scholar
Digital Library
- Markus Kleffmann, Marc Hesenius, and Volker Gruhn. 2015a. Connecting UI and Business Processes in a Collaborative Sketching Environment. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '15). ACM, New York, NY, USA, 200--205. Google Scholar
Digital Library
- Markus Kleffmann, Sebastian Röhl, Matthias Book, and Volker Gruhn. 2018. Evaluation of a Traceability Approach for Informal Freehand Sketches. Automated Software Engineering, Vol. 25, 1 (2018), 1--43. Google Scholar
Digital Library
- Markus Kleffmann, Sebastian Röhl, Volker Gruhn, and Matthias Book. 2015b. Establishing and Navigating Trace Links between Elements of Informal Diagram Sketches. In 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability. 1--7. Google Scholar
Digital Library
- Bö rge Kordts, Bashar Altakrouri, and Andreas Schrader. 2015. Capturing and analysing movement using depth sensors and Labanotation. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS '15. ACM Press, New York, New York, USA, 132--141. Google Scholar
Digital Library
- Sven Kratz and Michael Rohs. 2010. A $3 Gesture Recognizer -- Simple Gesture Recognition for Devices Equipped with 3D Acceleration Sensors. In Proceedings of the 15th International Conference on Intelligent User Interfaces (IUI '10). ACM Press, New York, New York, USA, 341. Google Scholar
Digital Library
- Effie Lai-Chong Law, Virpi Roto, Marc Hassenzahl, Arnold P.O.S. Vermeeren, and Joke Kort. 2009. Understanding, Scoping and Defining User Experience: A Survey Approach. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 719--728. Google Scholar
Digital Library
- Hao Lü, James A. Fogarty, and Yang Li. 2014. Gesture Script: recognizing gestures and their structure using rendering scripts and interactively trained parts. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI '14. ACM Press, New York, New York, USA, 1685--1694. Google Scholar
Digital Library
- Meredith Ringel Morris, Anqi Huang, Andreas Paepcke, and Terry Winograd. 2006. Cooperative gestures: multi-user gestural interactions for co-located groupware. In Proceedings of the SIGCHI conference on Human Factors in computing systems - CHI '06. ACM Press, New York, New York, USA, 1201.Google Scholar
Digital Library
- Antti Oulasvirta, Tino Weinkauf, Myroslav Bachynskyi, and Gregorio Palmas. 2014. Gestikulieren mit Stil. Informatik-Spektrum, Vol. 37, 5 (2014), 449--453.Google Scholar
Cross Ref
- Otto Parra Gonzá lez, Sergio Espa n a, and Oscar Pastor. 2015. Including Multi-Stroke Gesture-based Interaction in User Interfaces Using a Model-Driven Method. In Proceedings of the XVI International Conference on Human Computer Interaction (Interacció n '15). ACM, New York, NY, USA, 6:1---6:8. Google Scholar
Digital Library
- Gonzalo Ramos, Matthew Boulos, and Ravin Balakrishnan. 2004. Pressure Widgets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 487--494. Google Scholar
Digital Library
- Christian Rendl, Patrick Greindl, Kathrin Probst, Martin Behrens, and Michael Haller. 2014. Presstures: Exploring Pressure-Sensitive Multi-Touch Gestures on Trackpads. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 431--434. Google Scholar
Digital Library
- Julie Rico and Stephen Brewster. 2010. Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 887--896. Google Scholar
Digital Library
- Dean Rubine. 1991. Specifying gestures by example. ACM SIGGRAPH Computer Graphics, Vol. 25, 4 (jul 1991), 329--337. Google Scholar
Digital Library
- Lucio Davide Spano, Antonio Cisternino, Fabio Paternò, and Gianni Fenu. 2013. GestIT: A Declarative and Compositional Framework for Multiplatform Gesture Definition. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM, New York, NY, USA, 187--196.Google Scholar
Digital Library
- Eugene M Taranta II and Joseph J LaViola Jr. 2015. Penny Pincher: A Blazing Fast, Highly Accurate $-family Recognizer. In Proceedings of the 41st Graphics Interface Conference (GI '15). Canadian Information Processing Society, Toronto, Ont., Canada, 195--202. http://dl.acm.org/citation.cfm?id=2788890.2788925 Google Scholar
Digital Library
- G. Tassey. 2002. The Economic Impacts of Inadequate Infrastructure for Software Testing. Technical Report. National Institute of Standards and Technology .Google Scholar
- Radu-Daniel Vatavu, Lisa Anthony, and Jacob O. Wobbrock. 2012. Gestures as point clouds. In Proc. 14th ACM intl. conf. on Multimodal interaction - ICMI '12. ACM Press, New York, New York, USA, 273.Google Scholar
- Daniel Leite Viana and André Lu'is de Medeiros Santos. 2017. A Domain-Specific Language for the Specification of Gesture-based Applications. In Proceedings of the 21st Brazilian Symposium on Programming Languages (SBLP 2017). ACM, New York, NY, USA, Article 11, bibinfonumpages7 pages. Google Scholar
Digital Library
- Geoff Walker. 2012. A Review of Technologies for Sensing Contact Location on the Surface of a Display. Journal of the Society for Information Display, Vol. 20, 8 (2012), 413--440.Google Scholar
Cross Ref
- Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-Defined Gestures for Surface Computing. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 1083. Google Scholar
Digital Library
- Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures Without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes. In Proc. 20th Annual ACM Symp. User Interface Software and Technology (UIST '07). ACM, New York, NY, USA, 159--168. Google Scholar
Digital Library
- Yang Zhang and Chris Harrison. 2015. Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 167--173. Google Scholar
Digital Library
- Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. SkinTrack: Using the Body As an Electrical Waveguide for Continuous Finger Tracking on the Skin. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1491--1503. Google Scholar
Digital Library
Index Terms
GestureCards: A Hybrid Gesture Notation
Recommendations
Introducing GestureCards: A Prototyping Gesture Notation
NordiCHI '16: Proceedings of the 9th Nordic Conference on Human-Computer InteractionGesture-based applications are common nowadays, but developing gestures is still error-prone and causes software defects as well as misunderstandings and frustration among project team members and stakeholders. Precise specification and documentation is ...
User-defined gestures for surface computing
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsMany surface computing prototypes have employed gestures created by system designers. Although such gestures are appropriate for early investigations, they are not necessarily reflective of user behavior. We present an approach to designing tabletop ...
UML Everywhere
A standardized and widely used diagramming notation is a sign of a profession's maturity. Yet in software engineering, we still use diverse and ad hoc notations. Here, the author proposes that engineers should put their differences aside and make a ...






Comments