Abstract

Supporting many gestures on small surfaces allows users to interact remotely with complex environments such as smart homes, large remote displays, or virtual reality environments, and switching between them (e.g., AR setup in a smart home). Providing eyes-free gestures in these contexts is important as this avoids disrupting the user's visual attention. However, very few techniques enable large sets of commands on small wearable devices supporting the user's mobility and even less provide eyes-free interaction. We present Side-Crossing Menus (SCM), a gestural technique enabling large sets of gestures on %small interactive surfaces like a smartwatch. Contrary to most gestural techniques, SCM relies on broad and shallow menus that favor small and rapid gestures. We demonstrate with a first experiment that users can efficiently perform these gestures eyes-free aided with tactile cues; 95% accuracy after training 20 minutes on a representative set of 30 gestures among 172. In a second experiment, we focus on the learning of SCM gestures and do not observe significant differences with conventional Multi-stroke Marking Menus in gesture accuracy and recall rate. As both techniques utilize contrasting menu structures, our results indicate that SCM is a compelling alternative for enhancing the input capabilities of small surfaces.
Supplemental Material
- Johnny Accot and Shumin Zhai. 2002. More Than Dotting the I's -- Foundations for Crossing-based Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02). ACM, 73--80. https://doi.org/10.1145/503376.503390Google Scholar
Digital Library
- Michel Amberg, Frédéric Giraud, Betty Semail, Paolo Olivo, Géry Casiez, and Nicolas Roussel. 2011. STIMTAC: A Tactile Input Device with Programmable Friction. In Proceedings of the 24th Annual ACM Symposium Adjunct on User Interface Software and Technology (UIST '11 Adjunct ). ACM, 7--8. https://doi.org/10.1145/2046396.2046401Google Scholar
Digital Library
- Georg Apitz and Franccois Guimbretière. 2004. CrossY: A Crossing-based Drawing Application. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, 3--12. https://doi.org/10.1145/1029632.1029635Google Scholar
Digital Library
- Caroline Appert and Shumin Zhai. 2009. Using Strokes As Command Shortcuts: Cognitive Benefits and Toolkit Support. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, 2289--2298. https://doi.org/10.1145/1518701.1519052Google Scholar
Digital Library
- Gilles Bailly, Eric Lecolinet, and Laurence Nigay. 2007. Wave Menus: Improving the Novice Mode of Hierarchical Marking Menus. In Human-Computer Interaction -- INTERACT 2007. Springer, 475--488. https://doi.org/10.1007/978--3--540--74796--3_45Google Scholar
Digital Library
- Gilles Bailly, Eric Lecolinet, and Laurence Nigay. 2008. Flower Menus: A New Type of Marking Menu with Large Menu Breadth, Within Groups and Efficient Expert Mode Memorization. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '08). ACM, 15--22. https://doi.org/10.1145/1385569.1385575Google Scholar
Digital Library
- Gilles Bailly, Eric Lecolinet, and Laurence Nigay. 2016. Visual Menu Techniques. ACM Comput. Surv. , Vol. 49, 4, Article 60 (Dec. 2016), bibinfonumpages41 pages. https://doi.org/10.1145/3002171Google Scholar
Digital Library
- Robert Ball, Chris North, and Doug A. Bowman. 2007. Move to improve: promoting physical navigation to increase user performance with large displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, 191--200. https://doi.org/10.1145/1240624.1240656Google Scholar
- Olivier Bau, Ivan Poupyrev, Ali Israr, and Chris Harrison. 2010. TeslaTouch: Electrovibration for Touch Surfaces. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST '10). ACM, 283--292. https://doi.org/10.1145/1866029.1866074Google Scholar
Digital Library
- Gabor Blasko and Steven Feiner. 2004. An Interaction System for Watch Computers Using Tactile Guidance and Bidirectional Segmented Strokes. In Proceedings of the Eighth International Symposium on Wearable Computers (ISWC '04). IEEE, 120--123. https://doi.org/10.1109/ISWC.2004.6Google Scholar
Digital Library
- Andrew Bragdon, Eugene Nelson, Yang Li, and Ken Hinckley. 2011. Experimental Analysis of Touch-screen Gesture Designs in Mobile Environments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, 403--412. https://doi.org/10.1145/1978942.1979000Google Scholar
Digital Library
- William A. S. Buxton. 1995. Chunking and Phrasing and the Design of Human-Computer Dialogues. In Human-Computer Interaction: Toward the Year 2000. Morgan Kaufmann, San Francisco, CA, USA, 494--499.Google Scholar
Digital Library
- Maria Claudia Buzzi, Marina Buzzi, Francesco Donini, Barbara Leporini, and Maria Teresa Paratore. 2013. Haptic Reference Cues to Support the Exploration of Touchscreen Mobile Devices by Blind Users. In Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI (CHItaly '13). ACM, Article 28, bibinfonumpages8 pages. https://doi.org/10.1145/2499149.2499156Google Scholar
Digital Library
- Tom Chandler, Maxime Cordeil, Tobias Czauderna, Tim Dwyer, Jaroslaw Glowacki, Cagatay Goncu, Matthias, Klapperstueck, Karsten Klein, Kim Marriott, Falk Schreiber, and Elliot Wilson. 2015. Immersive Analytics. In Big Data Visual Analytics (BDVA '15). IEEE, 1--8. https://doi.org/10.1109/BDVA.2015.7314296Google Scholar
Cross Ref
- Xiang 'Anthony' Chen, Tovi Grossman, and George Fitzmaurice. 2014. Swipeboard: A Text Entry Technique for Ultra-small Interfaces That Supports Novice to Expert Transitions. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, 615--620. https://doi.org/10.1145/2642918.2647354Google Scholar
Digital Library
- Christian Corsten, Christian Cherek, Thorsten Karrer, and Jan Borchers. 2015. HaptiCase: Back-of-Device Tactile Landmarks for Eyes-Free Absolute Indirect Touch. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, 2171--2180. https://doi.org/10.1145/2702123.2702277Google Scholar
Digital Library
- Raimund Dachselt and Anett Hübner. 2007. Three-dimensional Menus: A Survey and Taxonomy. Comput. Graph. , Vol. 31, 1 (Jan. 2007), 53--65. https://doi.org/10.1016/j.cag.2006.09.006Google Scholar
Digital Library
- Jules Franccoise, Olivier Chapuis, Sylvain Hanneton, and Frédéric Bevilacqua. 2016. SoundGuides: Adapting Continuous Auditory Feedback to Users. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, 2829--2836. https://doi.org/10.1145/2851581.2892420Google Scholar
- Bruno Fruchard, Eric Lecolinet, and Olivier Chapuis. 2017. MarkPad: Augmenting Touchpads for Command Selection. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, 5630--5642. https://doi.org/10.1145/3025453.3025486Google Scholar
Digital Library
- Bruno Fruchard, Eric Lecolinet, and Olivier Chapuis. 2018a. How Memorizing Positions or Directions Affects Gesture Learning?. In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces (ISS '18). ACM, 107--114. https://doi.org/10.1145/3279778.3279787Google Scholar
Digital Library
- Bruno Fruchard, Eric Lecolinet, and Olivier Chapuis. 2018b. Impact of Semantic Aids on Command Memorization for On-body Interaction and Directional Gestures. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces (AVI '18). ACM, Article 14, bibinfonumpages9 pages. https://doi.org/10.1145/3206505.3206524Google Scholar
Digital Library
- Tovi Grossman, Xiang Anthony Chen, and George Fitzmaurice. 2015. Typing on Glasses: Adapting Text Entry to Smart Eyewear. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, 144--152. https://doi.org/10.1145/2785830.2785867Google Scholar
Digital Library
- Franccois Guimbretiére and Terry Winograd. 2000. FlowMenu: Combining Command, Text, and Data Entry. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology (UIST '00). ACM, 213--216. https://doi.org/10.1145/354401.354778Google Scholar
Digital Library
- Carl Gutwin, Andy Cockburn, Joey Scarr, Sylvain Malacria, and Scott C. Olson. 2014. Faster Command Selection on Tablets with FastTap. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, 2617--2626. https://doi.org/10.1145/2556288.2557136Google Scholar
- Teng Han, Jiannan Li, Khalad Hasan, Keisuke Nakamura, Randy Gomez, Ravin Balakrishnan, and Pourang Irani. 2018. PageFlip: Leveraging Page-Flipping Gestures for Efficient Command and Value Selection on Smartwatches. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, Article 529, bibinfonumpages12 pages. https://doi.org/10.1145/3173574.3174103Google Scholar
Digital Library
- Tom Horak, Sriram Karthik Badam, Niklas Elmqvist, and Raimund Dachselt. 2018. When David Meets Goliath: Combining Smartwatches with a Large Vertical Display for Visual Data Exploration. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, Article 19, bibinfonumpages13 pages. https://doi.org/10.1145/3173574.3173593Google Scholar
Digital Library
- Mohit Jain and Ravin Balakrishnan. 2012. User Learning and Performance with Bezel Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, 2221--2230. https://doi.org/10.1145/2207676.2208376Google Scholar
Digital Library
- Maria Karam and m. c. schraefel. 2005. A Taxonomy of Gestures in Human Computer Interactions. (2005). https://eprints.soton.ac.uk/261149/ Project Report.Google Scholar
- Yuki Kubo, Buntarou Shizuki, and Jiro Tanaka. 2016. B2B-Swipe: Swipe Gesture for Rectangular Smartwatches from a Bezel to a Bezel. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, 3852--3856. https://doi.org/10.1145/2858036.2858216Google Scholar
Digital Library
- Gordon Kurtenbach. 1993. The Design and Evaluation of Marking Menus . Ph.D. Dissertation. University of Toronto, Canada. http://autodeskresearch.com/pdf/theses/kurtenbach-phd.pdfGoogle Scholar
- Gordon Kurtenbach and William Buxton. 1993. The Limits of Expert Performance Using Hierarchic Marking Menus. In Proceedings of the INTERCHI '93 Conference on Human Factors in Computing Systems (INTERCHI '93). IOS Press, 482--487. http://dl.acm.org/citation.cfm?id=164632.164977Google Scholar
Digital Library
- Benjamin Lafreniere, Carl Gutwin, and Andy Cockburn. 2017. Investigating the Post-Training Persistence of Expert Interaction Techniques. ACM Trans. Comput.-Hum. Interact. , Vol. 24, 4, Article 29 (Aug. 2017), bibinfonumpages46 pages. https://doi.org/10.1145/3119928Google Scholar
Digital Library
- Benjamin Lafreniere, Carl Gutwin, Andy Cockburn, and Tovi Grossman. 2016. Faster Command Selection on Touchscreen Watches. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, 4663--4674. https://doi.org/10.1145/2858036.2858166Google Scholar
Digital Library
- Yang Li. 2010. Gesture Search: A Tool for Fast Mobile Data Access. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST '10). ACM, 87--96. https://doi.org/10.1145/1866029.1866044Google Scholar
Digital Library
- K. Marriott, F. Schreiber, T. Dwyer, K. Klein, N. Henry Riche, T. Itoh, W. Stuerzlinger, and B.H Thomas (Eds.). 2018. Immersive Analytics . LNCS, Vol. 11190. Springer, Cham. https://doi.org/10.1007/978--3-030-01388--2Google Scholar
Cross Ref
- Mathieu Nancel and Michel Beaudouin-Lafon. 2008. Extending Marking Menus with Integral Dimensions . Technical Report 1503. LRI, Univ. Paris-Sud, France.Google Scholar
- Uran Oh, Shaun K. Kane, and Leah Findlater. 2013. Follow That Sound: Using Sonification and Corrective Verbal Feedback to Teach Touchscreen Gestures. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '13). ACM, Article 13, bibinfonumpages8 pages. https://doi.org/10.1145/2513383.2513455Google Scholar
Digital Library
- Simon T. Perrault, Eric Lecolinet, Yoann Pascal Bourse, Shengdong Zhao, and Yves Guiard. 2015. Physical Loci: Leveraging Spatial, Object and Semantic Memory for Command Selection. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, 299--308. https://doi.org/10.1145/2702123.2702126Google Scholar
Digital Library
- Simon T. Perrault, Eric Lecolinet, James Eagan, and Yves Guiard. 2013. Watchit: Simple Gestures and Eyes-free Interaction for Wristwatches and Bracelets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, 1451--1460. https://doi.org/10.1145/2470654.2466192Google Scholar
Digital Library
- Stuart Pook, Eric Lecolinet, Guy Vaysseix, and Emmanuel Barillot. 2000. Control Menus: Excecution and Control in a Single Interactor. In CHI '00 Extended Abstracts on Human Factors in Computing Systems (CHI EA '00). ACM, 263--264. https://doi.org/10.1145/633292.633446Google Scholar
Digital Library
- Arnaud Prouzeau, Anastasia Bezerianos, and Olivier Chapuis. 2018. Awareness Techniques to Aid Transitions Between Personal and Shared Workspaces in Multi-Display Environments. In Proceedings of the ACM International Conference on Interactive Surfaces and Spaces (ISS '18). ACM, 291--304. https://doi.org/10.1145/3279778.3279780Google Scholar
Digital Library
- Yosra Rekik, Radu-Daniel Vatavu, and Laurent Grisoni. 2014. Understanding Users' Perceived Difficulty of Multi-Touch Gesture Articulation. In Proceedings of the 16th International Conference on Multimodal Interaction (ICMI '14). ACM, 232--239. https://doi.org/10.1145/2663204.2663273Google Scholar
Digital Library
- Volker Roth and Thea Turner. 2009. Bezel Swipe: Conflict-free Scrolling and Multiple Selection on Mobile Touch Screen Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, 1523--1526. https://doi.org/10.1145/1518701.1518933Google Scholar
Digital Library
- Anne Roudaut, Gilles Bailly, Eric Lecolinet, and Laurence Nigay. 2009. Leaf Menus: Linear Menus with Stroke Shortcuts for Small Handheld Devices. In Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT '09). Springer-Verlag, 616--619. https://doi.org/10.1007/978--3--642-03655--2_69Google Scholar
Digital Library
- Joey Scarr, Andy Cockburn, Carl Gutwin, and Andrea Bunt. 2012. Improving Command Selection with CommandMaps. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, 257--266. https://doi.org/10.1145/2207676.2207713Google Scholar
Digital Library
- Joey Scarr, Andy Cockburn, Carl Gutwin, and Sylvain Malacria. 2013. Testing the Robustness and Performance of Spatially Consistent Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, 3139--3148. https://doi.org/10.1145/2470654.2466430Google Scholar
Digital Library
- Katherine Schramm, Carl Gutwin, and Andy Cockburn. 2016. Supporting Transitions to Expertise in Hidden Toolbars. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, 4687--4698. https://doi.org/10.1145/2858036.2858412Google Scholar
Digital Library
- Marcos Serrano, Khalad Hasan, Barrett Ens, Xing-Dong Yang, and Pourang Irani. 2015. SmartwatchesGoogle Scholar
- Head-Worn Displays: the "New" Smartphone. In ACM Workshop on Mobile Collocated Interaction (W28 - CHI 2015). ACM, 1--5. http://oatao.univ-toulouse.fr/15288/Google Scholar
- Marcos Serrano, Eric Lecolinet, and Yves Guiard. 2013. Bezel-Tap Gestures: Quick Activation of Commands from Sleep Mode on Tablets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, 3027--3036. https://doi.org/10.1145/2470654.2481421Google Scholar
Digital Library
- Marcos Serrano, Anne Roudaut, and Pourang Irani. 2017. Visual Composition of Graphical Elements on Non-Rectangular Displays. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, 4405--4416. https://doi.org/10.1145/3025453.3025677Google Scholar
Digital Library
- Stefan Tilkov and Steve Vinoski. 2010. Node. js: Using JavaScript to build high-performance network programs. IEEE Internet Computing , Vol. 14, 6 (2010), 80--83.Google Scholar
Digital Library
- Md. Sami Uddin, Carl Gutwin, and Benjamin Lafreniere. 2016. HandMark Menus: Rapid Command Selection and Large Command Sets on Multi-Touch Displays. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, 5836--5848. https://doi.org/10.1145/2858036.2858211Google Scholar
Digital Library
- Daniel Vogel and Patrick Baudisch. 2007. Shift: A Technique for Operating Pen-based Interfaces Using Touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, 657--666. https://doi.org/10.1145/1240624.1240727Google Scholar
Digital Library
- Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-defined Gestures for Surface Computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, 1083--1092. https://doi.org/10.1145/1518701.1518866Google Scholar
Digital Library
- Jacob O. Wobbrock, Brad A. Myers, and John A. Kembel. 2003. EdgeWrite: A Stylus-based Text Entry Method Designed for High Accuracy and Stability of Motion. In Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology (UIST '03). ACM, 61--70. https://doi.org/10.1145/964696.964703Google Scholar
Digital Library
- Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures Without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07). ACM, 159--168. https://doi.org/10.1145/1294211.1294238Google Scholar
Digital Library
- Pui Chung Wong, Kening Zhu, Xing-Dong Yang, and Hongbo Fu. 2020. Exploring Eyes-Free Bezel-Initiated Swipe on Round Smartwatches. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, 1--11. https://doi.org/10.1145/3313831.3376393Google Scholar
Digital Library
- Robert Xiao, Teng Cao, Ning Guo, Jun Zhuo, Yang Zhang, and Chris Harrison. 2018. LumiWatch: On-Arm Projected Graphics and Touch Input. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, Article 95, bibinfonumpages11 pages. https://doi.org/10.1145/3173574.3173669Google Scholar
Digital Library
- Shumin Zhai, Per Ola Kristensson, Caroline Appert, Tue Andersen, and Xiang Cao. 2012. Foundational Issues in Touch-Surface Stroke Gesture Design: An Integrative Review. Found. Trends Hum.-Comput. Interact. , Vol. 5, 2 (Feb. 2012), 97--205. https://doi.org/10.1561/1100000012Google Scholar
Digital Library
- Cheng Zhang, Junrui Yang, Caleb Southern, Thad E. Starner, and Gregory D. Abowd. 2016. WatchOut: Extending Interactions on a Smartwatch with Inertial Sensing. In Proceedings of the 2016 ACM International Symposium on Wearable Computers (ISWC '16). ACM, 136--143. https://doi.org/10.1145/2971763.2971775Google Scholar
- Shengdong Zhao, Maneesh Agrawala, and Ken Hinckley. 2006. Zone and Polygon Menus: Using Relative Position to Increase the Breadth of Multi-stroke Marking Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, 1077--1086. https://doi.org/10.1145/1124772.1124933Google Scholar
Digital Library
- Shengdong Zhao and Ravin Balakrishnan. 2004. Simple vs. Compound Mark Hierarchical Marking Menus. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, 33--42. https://doi.org/10.1145/1029632.1029639Google Scholar
Digital Library
- Jingjie Zheng, Xiaojun Bi, Kun Li, Yang Li, and Shumin Zhai. 2018. M3 Gesture Menu: Design and Experimental Analyses of Marking Menus for Touchscreen Mobile Interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, Article 249, bibinfonumpages14 pages. https://doi.org/10.1145/3173574.3173823Google Scholar
Digital Library
Index Terms
Side-Crossing Menus: Enabling Large Sets of Gestures for Small Surfaces
Recommendations
Touch+Finger: Extending Touch-based User Interface Capabilities with "Idle" Finger Gestures in the Air
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and TechnologyIn this paper, we present Touch+Finger, a new interaction technique that augments touch input with multi-finger gestures for rich and expressive interaction. The main idea is that while one finger is engaged in a touch event, a user can leverage the ...
MarkPad: Augmenting Touchpads for Command Selection
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsWe present MarkPad, a novel interaction technique taking advantage of the touchpad. MarkPad allows creating a large number of size-dependent gestural shortcuts that can be spatially organized as desired by the user. It relies on the idea of using visual ...
T(ether): spatially-aware handhelds, gestures and proprioception for multi-user 3D modeling and animation
SUI '14: Proceedings of the 2nd ACM symposium on Spatial user interactionT(ether) is a spatially-aware display system for multi-user, collaborative manipulation and animation of virtual 3D objects. The handheld display acts as a window into virtual reality, providing users with a perspective view of 3D data. T(ether) tracks ...






Comments