Abstract

Gesto is a system that enables task automation for Android apps using gestures and voice commands. Using Gesto, a user can record a UI action sequence for an app, choose a gesture or a voice command to activate the UI action sequence, and later trigger the UI action sequence by the corresponding gesture/voice command. Gesto enables this for existing Android apps without requiring their source code or any help from their developers. In order to make such capability possible, Gesto combines bytecode instrumentation and UI action record-and-replay. To show the applicability of Gesto, we develop four use cases using real apps downloaded from Google Play-Bing, Yelp, AVG Cleaner, and Spotify. For each of these apps, we map a gesture or a voice command to a sequence of UI actions. According to our measurement, Gesto incurs modest overhead for these apps in terms of memory usage, energy usage, and code size increase. We evaluate our instrumentation capability and overhead using 1,000 popular apps downloaded from Google Play. Our result shows that Gesto is able to instrument 94.9% of the apps without any significant overhead. In addition, since our prototype currently supports 6 main UI elements of Android, we evaluate our coverage and measure what percentage of UI element uses we can cover. Our result shows that our 6 UI elements can cover 96.4% of all statically-declared UI element uses in the 1,000 Google Play apps.
- Cited May 2018. Accessibility Services. https://developer.android.com/reference/android/accessibilityservice/AccessibilityService.html.Google Scholar
- Cited May 2018. Apktool. https://ibotpeaches.github.io/Apktool/.Google Scholar
- Cited May 2018. appetizer-toolkit. https://github.com/appetizerio/appetizer-toolkit.Google Scholar
- Cited May 2018. Bot-bot. http://imaginea.github.io/bot-bot/index.html.Google Scholar
- Cited May 2018. Culebra. https://github.com/dtmilano/AndroidViewClient/wiki/culebra.Google Scholar
- Cited May 2018. Espresso Test Recorder, 2017. h ps://developer.android.com/studio/test/espresso-test-recorder.html.Google Scholar
- Cited May 2018. Frep. http://strai.x0.com/frep/.Google Scholar
- Cited May 2018. monkeyrunner. https://developer.android.com/studio/test/monkeyrunner/index.html.Google Scholar
- Cited May 2018. Ranorex. http://www.ranorex.com/mobile-automation-testing.html.Google Scholar
- Cited May 2018. Robotium Recorder. https://robotium.com/products/robotium-recorder.Google Scholar
- Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Romit Roy Choudhury, Kevin Caves, and Frank De Ruyter. 2011. Using mobile phones to write in air. In Proceedings of the 9th international conference on Mobile systems, applications, and services. ACM, 15--28. Google Scholar
Digital Library
- Amazon. Cited December 2017. Alexa. https://developer.amazon.com/alexa.Google Scholar
- Amazon. Cited December 2017. Alexa SDK. https://developer.amazon.com/alexa-voice-service/sdk.Google Scholar
- Apple. Cited December 2017. Siri. https://www.apple.com/ios/siri/.Google Scholar
- Apple. Cited December 2017. SiriKit. https://developer.apple.com/sirikit/.Google Scholar
- Tanzirul Azim, Oriana Riva, and Suman Nath. 2016. uLink: Enabling User-De ned Deep Linking to App Content. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services (In MobiSys). ACM, New York, NY, USA. http://dl.acm.org/citation.cfmid=2906416&CFID=758371532&CFTOKEN=77725750 Google Scholar
Digital Library
- Ke-Yu Chen, Daniel Ashbrook, Mayank Goel, Sung-Hyuck Lee, and Shwetak Patel. 2015. AirLink: sharing files between multiple devices using in-air gestures. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 565--569. Google Scholar
Digital Library
- Xiang 'Anthony' Chen and Yang Li. 2017. Improv: An Input Framework for Improvising Cross-Device Interaction by Demonstration. ACM Trans. Comput.-Hum. Interact. 24, 2, Article 15 (April 2017), 21 pages. Google Scholar
Digital Library
- Jason Flinnand Z Morley Mao. 2011. Can deterministic replay be an enabling tool for mobile computing?. In Proceedings of the 12th Workshop on Mobile Computing Systems and Applications. ACM, 84--89. Google Scholar
Digital Library
- Lorenzo Gomez, Iulian Neamtiu, Tanzirul Azim, and Todd Millstein. 2013. Reran: Timing-and touch-sensitive record and replay for android. In So ware Engineering (ICSE), 2013 35th International Conference on. IEEE, 72--81.Google Scholar
- Google. Cited December 2017. Class android.view. View. https://developer.android.com/reference/android/view/ View.html.Google Scholar
- Google. Cited December 2017. Google Assistant. https://assistant.google.com/.Google Scholar
- Google. Cited December 2017. Google Assistant SDK. https://developers.google.com/assistant/sdk/.Google Scholar
- Ma hew Halpern, Yuhao Zhu, Ramesh Peri, and Vijay Janapa Reddi. 2015. Mosaic: cross-platform user-interaction record and replay for the fragmented android ecosystem. In Performance Analysis of Systems and So ware (ISPASS), 2015 IEEE International Symposium on. IEEE, 215--224.Google Scholar
- Yongjian Hu, Tanzirul Azim, and Iulian Neamtiu. 2015. Versatile yet lightweight record-and-replay for android. In ACM SIGPLAN Notices, Vol. 50. ACM, 349--366. Google Scholar
Digital Library
- Intel. Cited December 2017. Intel Context Sensing SDK | Intel Software. https://software.intel.com/en-us/context-sensing-sdk.Google Scholar
- Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing Gesture Recognition to All Devices. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI'14). USENIX Association, Berkeley, CA, USA. Google Scholar
Digital Library
- Sven Kratz and Jason Wiese. 2016. GestureSeg: Developing a Gesture Segmentation System Using Gesture Execution Phase Labeling by Crowd Workers. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '16). ACM, New York, NY, USA.Google Scholar
Digital Library
- Wing Lam, Zhengkai Wu, Dengfeng Li, Wenyu Wang, Haibing Zheng, Hui Luo, Peng Yan, Yuetang Deng, and Tao Xie. 2017. Record and replay for Android: are we there yet in industrial cases?. In Proceedings of the 2017 11th Joint Meeting on Foundations of So ware Engineering. ACM, 854--859. Google Scholar
Digital Library
- Tessa Lau, Julian Cerruti, Guillermo Manzato, Mateo Bengualid, Jeffrey P. Bigham, and Jeffrey Nichols. 2010. A Conversational Interface to Web Automation. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST '10). ACM, New York, NY, USA, 229--238. Google Scholar
Digital Library
- Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, 21 New York, NY, USA, 6038--6049.Google Scholar
Digital Library
- Jiayang Liu, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. 2009. uWave: Accelerometer-based personalized gesture recognition and its applications. Pervasive and Mobile Computing 5, 6 (2009), 657--675.Google Scholar
Digital Library
- Google LLC and Open Handset Alliance. Cited December 2017. SpeechRecognizer | Android Developers. https://developer.android.com/reference/android/speech/SpeechRecognizer.html.Google Scholar
- Yun Ma, Xuanzhe Liu, Ziniu Hu, Dian Yang, Gang Huang, Yunxin Liu, and Tao Xie. 2017. Aladdin: automating release of Android deep links to in-app content. In Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, 139--140. Google Scholar
Digital Library
- Microsoft. Cited December 2017. Cortana. https://www.microsoft.com/en-us/windows/cortana.Google Scholar
- Microsoft. Cited December 2017. Cortana SDK. https://developer.microsoft.com/en-us/cortana.Google Scholar
- AVG Mobile. Cited December 2017. AVG Cleaner. https://play.google.com/store/apps/details?id=com.avg.cleaner&hl=en.Google Scholar
- Abhinav Parate, Meng-Chieh Chiu, Chaniel Chadowitz, Deepak Ganesan, and Evangelos Kalogerakis. 2014. Risq: Recognizing smoking gestures with inertial sensors on a wristband. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services. ACM, 149--161. Google Scholar
Digital Library
- Taiwoo Park, Jinwon Lee, Inseok Hwang, Chungkuk Yoo, Lama Nachman, and Junehwa Song. 2011. E-gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices. In Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems. ACM, 260--273.Google Scholar
Digital Library
- Zhengrui Qin, Yutao Tang, Ed Novak, and n Li. 2016. MobiPlay: A Remote Execution Based Record-and-replay Tool for Mobile Applications. In Proceedings of the 38th International Conference on So ware Engineering (ICSE '16). ACM, New York, NY, USA.Google Scholar
Digital Library
- Giuseppe Raa, Jinwon Lee, and Lama Nachman. 2010. Don't slow me down: Bringing energy efficiency to continuous gesture recognition (Wearable Computers (ISWC), 2010 International Symposium). IEEE.Google Scholar
- Margaret Rouse. Cited November 2017. What is deep link? http://searchmicroservices.techtarget.com/definition/deep-link.Google Scholar
- Haichen Shen, Aruna Balasubramanian, Anthony LaMarca, and David Wetherall. 2015. Enhancing Mobile Apps to Use Sensor Hubs Without Programmer Effort. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA. Google Scholar
Digital Library
- Sheng Shen, He Wang, and Romit Roy Choudhury. 2016. I am a Smartwatch and I can Track my User's Arm. In Proceedings of the 14th annual international conference on Mobile systems, applications, and services. ACM, 85--96. Google Scholar
Digital Library
- Lucio Davide Spano, Antonio Cisternino, Fabio Paternoff, and Gianni Fenu. 2013. GestIT: A Declarative and Com- positional Framework for Multiplatform Gesture Definition. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM, New York, NY, USA.Google Scholar
Digital Library
- Nishant Srivastava. 2016. GitHub - nisrulz/sensey: {Android Library} Play with sensor events & detect gestures in a snap. https://github.com/nisrulz/sensey.Google Scholar
- Raja Valle e-Rai, Phong Co, Etienne Gagnon, Laurie Hendren, Patrick Lam, and Vijay Sundaresan. 1999. Soot - a Java bytecode Optimization Framework. In Proceedings of the 1999 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON '99).Google Scholar
- Chao Xu, Parth H Pathak, and Prasant Mohapatra. {n. d.}. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications. ACM, 9--14.Google Scholar
- Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a mobile device into a mouse in the air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 15--29. Google Scholar
Digital Library
Index Terms
Gesto: Mapping UI Events to Gestures and Voice Commands
Recommendations
Layout and Image Recognition Driving Cross-Platform Automated Mobile Testing
ICSE '21: Proceedings of the 43rd International Conference on Software EngineeringThe fragmentation problem has extended from Android to different platforms, such as iOS, mobile web, and even mini-programs within some applications (app), like WeChat1. In such a situation, recording and replaying test scripts is one of the most ...
Prototyping iPhone apps: realistic experiences on the device
NordiCHI '10: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending BoundariesIn this paper we evaluate Touch Application Prototype - a tool for designers to quickly create interactive and realistic prototypes of Apple® iPhone® apps and test them on the device. We define 5 requirements such as Speed, Practicality and Realism, and ...
Gestural Interfaces Touchscreen: Thinking Interactions beyond the Button from Interaction Design for Gmail Android App
Proceedings of the Third International Conference on Design, User Experience, and Usability. User Experience Design for Diverse Interaction Platforms and Environments - Volume 8518The design of interfaces for mobile devices (smartphones, tablets) is still very tied to the WIMP paradigm (abbreviation for "windows, icons, menus and pointers"). Thus, the potential use of gestures (touch screen) is often not used efficiently. ...






Comments