skip to main content
research-article
Public Access

Gesto: Mapping UI Events to Gestures and Voice Commands

Authors Info & Claims
Published:13 June 2019Publication History
Skip Abstract Section

Abstract

Gesto is a system that enables task automation for Android apps using gestures and voice commands. Using Gesto, a user can record a UI action sequence for an app, choose a gesture or a voice command to activate the UI action sequence, and later trigger the UI action sequence by the corresponding gesture/voice command. Gesto enables this for existing Android apps without requiring their source code or any help from their developers. In order to make such capability possible, Gesto combines bytecode instrumentation and UI action record-and-replay. To show the applicability of Gesto, we develop four use cases using real apps downloaded from Google Play-Bing, Yelp, AVG Cleaner, and Spotify. For each of these apps, we map a gesture or a voice command to a sequence of UI actions. According to our measurement, Gesto incurs modest overhead for these apps in terms of memory usage, energy usage, and code size increase. We evaluate our instrumentation capability and overhead using 1,000 popular apps downloaded from Google Play. Our result shows that Gesto is able to instrument 94.9% of the apps without any significant overhead. In addition, since our prototype currently supports 6 main UI elements of Android, we evaluate our coverage and measure what percentage of UI element uses we can cover. Our result shows that our 6 UI elements can cover 96.4% of all statically-declared UI element uses in the 1,000 Google Play apps.

References

  1. Cited May 2018. Accessibility Services. https://developer.android.com/reference/android/accessibilityservice/AccessibilityService.html.Google ScholarGoogle Scholar
  2. Cited May 2018. Apktool. https://ibotpeaches.github.io/Apktool/.Google ScholarGoogle Scholar
  3. Cited May 2018. appetizer-toolkit. https://github.com/appetizerio/appetizer-toolkit.Google ScholarGoogle Scholar
  4. Cited May 2018. Bot-bot. http://imaginea.github.io/bot-bot/index.html.Google ScholarGoogle Scholar
  5. Cited May 2018. Culebra. https://github.com/dtmilano/AndroidViewClient/wiki/culebra.Google ScholarGoogle Scholar
  6. Cited May 2018. Espresso Test Recorder, 2017. h ps://developer.android.com/studio/test/espresso-test-recorder.html.Google ScholarGoogle Scholar
  7. Cited May 2018. Frep. http://strai.x0.com/frep/.Google ScholarGoogle Scholar
  8. Cited May 2018. monkeyrunner. https://developer.android.com/studio/test/monkeyrunner/index.html.Google ScholarGoogle Scholar
  9. Cited May 2018. Ranorex. http://www.ranorex.com/mobile-automation-testing.html.Google ScholarGoogle Scholar
  10. Cited May 2018. Robotium Recorder. https://robotium.com/products/robotium-recorder.Google ScholarGoogle Scholar
  11. Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Romit Roy Choudhury, Kevin Caves, and Frank De Ruyter. 2011. Using mobile phones to write in air. In Proceedings of the 9th international conference on Mobile systems, applications, and services. ACM, 15--28. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Amazon. Cited December 2017. Alexa. https://developer.amazon.com/alexa.Google ScholarGoogle Scholar
  13. Amazon. Cited December 2017. Alexa SDK. https://developer.amazon.com/alexa-voice-service/sdk.Google ScholarGoogle Scholar
  14. Apple. Cited December 2017. Siri. https://www.apple.com/ios/siri/.Google ScholarGoogle Scholar
  15. Apple. Cited December 2017. SiriKit. https://developer.apple.com/sirikit/.Google ScholarGoogle Scholar
  16. Tanzirul Azim, Oriana Riva, and Suman Nath. 2016. uLink: Enabling User-De ned Deep Linking to App Content. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services (In MobiSys). ACM, New York, NY, USA. http://dl.acm.org/citation.cfmid=2906416&CFID=758371532&CFTOKEN=77725750 Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Ke-Yu Chen, Daniel Ashbrook, Mayank Goel, Sung-Hyuck Lee, and Shwetak Patel. 2015. AirLink: sharing files between multiple devices using in-air gestures. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 565--569. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Xiang 'Anthony' Chen and Yang Li. 2017. Improv: An Input Framework for Improvising Cross-Device Interaction by Demonstration. ACM Trans. Comput.-Hum. Interact. 24, 2, Article 15 (April 2017), 21 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Jason Flinnand Z Morley Mao. 2011. Can deterministic replay be an enabling tool for mobile computing?. In Proceedings of the 12th Workshop on Mobile Computing Systems and Applications. ACM, 84--89. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Lorenzo Gomez, Iulian Neamtiu, Tanzirul Azim, and Todd Millstein. 2013. Reran: Timing-and touch-sensitive record and replay for android. In So ware Engineering (ICSE), 2013 35th International Conference on. IEEE, 72--81.Google ScholarGoogle Scholar
  21. Google. Cited December 2017. Class android.view. View. https://developer.android.com/reference/android/view/ View.html.Google ScholarGoogle Scholar
  22. Google. Cited December 2017. Google Assistant. https://assistant.google.com/.Google ScholarGoogle Scholar
  23. Google. Cited December 2017. Google Assistant SDK. https://developers.google.com/assistant/sdk/.Google ScholarGoogle Scholar
  24. Ma hew Halpern, Yuhao Zhu, Ramesh Peri, and Vijay Janapa Reddi. 2015. Mosaic: cross-platform user-interaction record and replay for the fragmented android ecosystem. In Performance Analysis of Systems and So ware (ISPASS), 2015 IEEE International Symposium on. IEEE, 215--224.Google ScholarGoogle Scholar
  25. Yongjian Hu, Tanzirul Azim, and Iulian Neamtiu. 2015. Versatile yet lightweight record-and-replay for android. In ACM SIGPLAN Notices, Vol. 50. ACM, 349--366. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Intel. Cited December 2017. Intel Context Sensing SDK | Intel Software. https://software.intel.com/en-us/context-sensing-sdk.Google ScholarGoogle Scholar
  27. Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing Gesture Recognition to All Devices. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI'14). USENIX Association, Berkeley, CA, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Sven Kratz and Jason Wiese. 2016. GestureSeg: Developing a Gesture Segmentation System Using Gesture Execution Phase Labeling by Crowd Workers. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '16). ACM, New York, NY, USA.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Wing Lam, Zhengkai Wu, Dengfeng Li, Wenyu Wang, Haibing Zheng, Hui Luo, Peng Yan, Yuetang Deng, and Tao Xie. 2017. Record and replay for Android: are we there yet in industrial cases?. In Proceedings of the 2017 11th Joint Meeting on Foundations of So ware Engineering. ACM, 854--859. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Tessa Lau, Julian Cerruti, Guillermo Manzato, Mateo Bengualid, Jeffrey P. Bigham, and Jeffrey Nichols. 2010. A Conversational Interface to Web Automation. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST '10). ACM, New York, NY, USA, 229--238. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, 21 New York, NY, USA, 6038--6049.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Jiayang Liu, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. 2009. uWave: Accelerometer-based personalized gesture recognition and its applications. Pervasive and Mobile Computing 5, 6 (2009), 657--675.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Google LLC and Open Handset Alliance. Cited December 2017. SpeechRecognizer | Android Developers. https://developer.android.com/reference/android/speech/SpeechRecognizer.html.Google ScholarGoogle Scholar
  34. Yun Ma, Xuanzhe Liu, Ziniu Hu, Dian Yang, Gang Huang, Yunxin Liu, and Tao Xie. 2017. Aladdin: automating release of Android deep links to in-app content. In Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, 139--140. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Microsoft. Cited December 2017. Cortana. https://www.microsoft.com/en-us/windows/cortana.Google ScholarGoogle Scholar
  36. Microsoft. Cited December 2017. Cortana SDK. https://developer.microsoft.com/en-us/cortana.Google ScholarGoogle Scholar
  37. AVG Mobile. Cited December 2017. AVG Cleaner. https://play.google.com/store/apps/details?id=com.avg.cleaner&hl=en.Google ScholarGoogle Scholar
  38. Abhinav Parate, Meng-Chieh Chiu, Chaniel Chadowitz, Deepak Ganesan, and Evangelos Kalogerakis. 2014. Risq: Recognizing smoking gestures with inertial sensors on a wristband. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services. ACM, 149--161. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Taiwoo Park, Jinwon Lee, Inseok Hwang, Chungkuk Yoo, Lama Nachman, and Junehwa Song. 2011. E-gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices. In Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems. ACM, 260--273.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Zhengrui Qin, Yutao Tang, Ed Novak, and n Li. 2016. MobiPlay: A Remote Execution Based Record-and-replay Tool for Mobile Applications. In Proceedings of the 38th International Conference on So ware Engineering (ICSE '16). ACM, New York, NY, USA.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Giuseppe Raa, Jinwon Lee, and Lama Nachman. 2010. Don't slow me down: Bringing energy efficiency to continuous gesture recognition (Wearable Computers (ISWC), 2010 International Symposium). IEEE.Google ScholarGoogle Scholar
  42. Margaret Rouse. Cited November 2017. What is deep link? http://searchmicroservices.techtarget.com/definition/deep-link.Google ScholarGoogle Scholar
  43. Haichen Shen, Aruna Balasubramanian, Anthony LaMarca, and David Wetherall. 2015. Enhancing Mobile Apps to Use Sensor Hubs Without Programmer Effort. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Sheng Shen, He Wang, and Romit Roy Choudhury. 2016. I am a Smartwatch and I can Track my User's Arm. In Proceedings of the 14th annual international conference on Mobile systems, applications, and services. ACM, 85--96. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Lucio Davide Spano, Antonio Cisternino, Fabio Paternoff, and Gianni Fenu. 2013. GestIT: A Declarative and Com- positional Framework for Multiplatform Gesture Definition. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM, New York, NY, USA.Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Nishant Srivastava. 2016. GitHub - nisrulz/sensey: {Android Library} Play with sensor events & detect gestures in a snap. https://github.com/nisrulz/sensey.Google ScholarGoogle Scholar
  47. Raja Valle e-Rai, Phong Co, Etienne Gagnon, Laurie Hendren, Patrick Lam, and Vijay Sundaresan. 1999. Soot - a Java bytecode Optimization Framework. In Proceedings of the 1999 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON '99).Google ScholarGoogle Scholar
  48. Chao Xu, Parth H Pathak, and Prasant Mohapatra. {n. d.}. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications. ACM, 9--14.Google ScholarGoogle Scholar
  49. Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a mobile device into a mouse in the air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 15--29. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Gesto: Mapping UI Events to Gestures and Voice Commands

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!