Wontae Choi
ABSTRACT
In recent years, several automated GUI testing techniques for Android apps have been proposed. These tools have been shown to be effective in achieving good test coverage and in finding bugs without human intervention. Being automated, these tools typically run for a long time (say, for several hours), either until they saturate test coverage or until a testing time budget expires. Thus, these automated tools are not good at generating concise regression test suites that could be used for testing in incremental development of the apps and in regression testing.
We propose a heuristic technique that helps create a small regression test suite for an Android app from a large test suite generated by an automated Android GUI testing tool. The key insight behind our technique is that if we can identify and remove some common forms of redundancies introduced by existing automated GUI testing tools, then we can drastically lower the time required to minimize a GUI test suite. We have implemented our algorithm in a prototype tool called DetReduce. We applied DetReduce to several Android apps and found that DetReduce reduces a test-suite by an average factor of 16.9× in size and 14.7× in running time. We also found that for a test suite generated by running SwiftHand and a randomized test generation algorithm for 8 hours, DetReduce minimizes the test suite in an average of 14.6 hours.
References
- Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Bryan Dzung Ta, and Atif M Memon. 2015. MobiGUITAR: Automated model-based testing of mobile apps. IEEE Software 32, 5 (2015), 53--59.Google Scholar
- Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of Android apps. In ACM SIGPLAN Notices, Vol. 48. ACM, 641--660. Google Scholar
- Egon Balas. 1989. The prize collecting traveling salesman problem. Networks 19, 6 (1989), 621--636.Google Scholar
- Ravi Bhoraskar, Seungyeop Han, Jinseong Jeon, Tanzirul Azim, Shuo Chen, Jaeyeon Jung, Suman Nath, Rui Wang, and David Wetherall. 2014. Brahmastra: Driving Apps to Test the Security of Third-Party Components.. In USENIX Security. 1021--1036. Google Scholar
- Wontae Choi, George Necula, and Koushik Sen. 2013. Guided GUI testing of Android apps with minimal restart and approximate learning. In ACM SIGPLAN Notices, Vol. 48. ACM, 623--640. Google Scholar
- Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated test input generation for Android: Are we there yet?(e). In Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on. IEEE, 429--440.Google Scholar
- Lazaro Clapp, Osbert Bastani, Saswat Anand, and Alex Aiken. 2016. Minimizing GUI event traces. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 422--434. Google Scholar
- Markus Ermuth and Michael Pradel. 2016. Monkey see, monkey do: effective generation of GUI tests with inferred macro events. In Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, 82--93. Google Scholar
- Mattia Fazzini, Eduardo Noronha de A Freitas, Shauvik Roy Choudhary, and Alessandro Orso. 2016. From Manual Android Tests to Automated and Platform Independent Test Scripts. arXivpreprint arXiv:1608.03624 (2016).Google Scholar
- Lorenzo Gomez, Iulian Neamtiu, Tanzirul Azim, and Todd Millstein. 2013. Reran: Timing-and touch-sensitive record and replay for Android. In Software Engineering (ICSE), 2013 35th International Conference on. IEEE, 72--81. Google Scholar
- Google Inc. 2008. Google Play. https://play.google.com/store?hl=en. (2008). Accessed: 2017-04-11.Google Scholar
- Google Inc. 2008. UI/Application Exerciser Monkey. https://developer.android.com/studio/test/monkey.html. (2008). Accessed: 2017-04-11.Google Scholar
- Google Inc. 2010. Monkeyrunner. https://developer.android.com/studio/test/monkeyrunner/index.html. (2010). Accessed: 2017-04-11.Google Scholar
- Google Inc. 2011. Espresso. https://google.github.io/android-testing-support-library/docs/espresso/. (2011). Accessed: 2017-04-11.Google Scholar
- Alex Groce, Mohammed Amin Alipour, Chaoqiang Zhang, Yang Chen, and John Regehr. 2014. Cause reduction for quick testing. In Software Testing, Verification and Validation (ICST), 2014 IEEE Seventh International Conference on. IEEE, 243--252. Google Scholar
- Matthew Halpern, Yuhao Zhu, Ramesh Peri, and Vijay Janapa Reddi. 2015. Mosaic: cross-platform user-interaction record and replay for the fragmented Android ecosystem. In Performance Analysis of Systems and Software (ISPASS), 2015 IEEE International Symposium on. IEEE, 215--224.Google Scholar
- Mouna Hammoudi, Brian Burg, Gigon Bae, and Gregg Rothermel. 2015. On the use of delta debugging to reduce recordings and facilitate debugging of web applications. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 333--344. Google Scholar
- Dan Hao, Lu Zhang, Xingxia Wu, Hong Mei, and Gregg Rothermel. 2012. On-demand test suite reduction. In Proceedings of the 34th International Conference on Software Engineering. IEEE Press, 738--748. Google Scholar
- Hwa-You Hsu and Alessandro Orso. 2009. MINTS: A general framework and tool for supporting test-suite minimization. In Software Engineering, 2009. ICSE 2009. IEEE 31st International Conference on. IEEE, 419--429. Google Scholar
- Yongjian Hu, Tanzirul Azim, and Iulian Neamtiu. 2015. Versatile yet lightweight record-and-replay for Android. In ACM SIGPLAN Notices. ACM. Google Scholar
- Dennis Jeffrey and Neelam Gupta. 2005. Test suite reduction with selective redundancy. In Software Maintenance, 2005. ICSM'05. Proceedings of the 21st IEEE International Conference on. IEEE, 549--558. Google Scholar
- Dennis Jeffrey and Neelam Gupta. 2007. Improving fault detection capability by selectively retaining test cases during test suite reduction. IEEE Transactions on software Engineering 33, 2 (2007). Google Scholar
- Casper S Jensen, Mukul R Prasad, and Anders Møller. 2013. Automated testing with targeted event sequence generation. In Proceedings of the 2013 International Symposium on Software Testing and Analysis. ACM, 67--77. Google Scholar
- James A Jones and Mary Jean Harrold. 2003. Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Transactions on software Engineering 29, 3 (2003), 195--209. Google Scholar
- Chieh-Jan Mike Liang, Nicholas D Lane, Niels Brouwers, Li Zhang, Börje F Karlsson, Hao Liu, Yan Liu, Jun Tang, Xiang Shan, Ranveer Chandra, and others. 2014. Caiipa: Automated large-scale mobile app testing through contextual fuzzing. In Proceedings of the 20th annual international conference on Mobile computing and networking. ACM, 519--530. Google Scholar
- Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for Android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 224--234. Google Scholar
- Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. Evodroid: Segmented evolutionary testing of Android apps. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 599--609. Google Scholar
- Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for Android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, 94--105. Google Scholar
- Leonardo Mariani, Mauro Pezze, Oliviero Riganelli, and Mauro Santoro. 2012. Autoblacktest:Automaticblack-boxtestingofinteractiveapplications.In Software Testing, Verification and Validation (ICST), 2012 IEEE Fifth International Conference on. IEEE, 81--90. Google Scholar
- Scott McMaster and Atif Memon. 2008. Call-stack coverage for GUI test suite reduction. IEEE Transactions on Software Engineering 34, 1 (2008), 99--115. Google Scholar
- Ali Mesbah, Arie Van Deursen, and Stefan Lenselink. 2012. Crawling Ajax-based web applications through dynamic analysis of user interface state changes. ACM Transactions on the Web (TWEB) 6, 1 (2012), 3. Google Scholar
- Amin Milani Fard, Mehdi Mirzaaghaei, and Ali Mesbah. 2014. Leveraging existing tests in automated test generation for web applications. In Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. ACM, 67--78. Google Scholar
- Nariman Mirzaei, Hamid Bagheri, Riyadh Mahmood, and Sam Malek. 2015. Sigdroid: Automated system input generation for Android applications. In Software Reliability Engineering (ISSRE), 2015 IEEE 26th International Symposium on. IEEE. Google Scholar
- Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing combinatorics in GUI testing of Android applications. In Proceedings of the 38th International Conference on Software Engineering. ACM. Google Scholar
- Ghassan Misherghi and Zhendong Su. 2006. HDD: hierarchical delta debugging. In Proceedings of the 28th international conference on Software engineering. ACM, 142--151. Google Scholar
- Zhengrui Qin, Yutao Tang, Ed Novak, and Qun Li. 2016. Mobiplay: A remote execution based record-and-replay tool for mobile applications. In Proceedings of the 38th International Conference on Software Engineering. ACM, 571--582. Google Scholar
- Siegfried Rasthofer, Steven Arzt, Stefan Triller, and Michael Pradel. 2017. Making Malory Behave Maliciously: Targeted Fuzzing of Android Execution Environments. In Proceedings of the 39th International Conference on So ware Engineering. ACM, To appear. Google Scholar
- Vaibhav Rastogi, Yan Chen, and William Enck. 2013. AppsPlayground: automatic security analysis of smartphone applications. In Proceedings of the third ACM conference on Data and application security and privacy. ACM, 209--220. Google Scholar
- Renas Reda. 2010. Robotium, User scenario testing for Android. https://github.com/RobotiumTech/robotium. (2010). Accessed: 2017-04-11.Google Scholar
- Colin Scott, Andreas Wundsam, Barath Raghavan, Aurojit Panda, Andrew Or, Jefferson Lai, Eugene Huang, Zhi Liu, Ahmed El-Hassany, Sam Whitlock, and others. 2015. Troubleshooting blackbox SDN control software with minimal causal sequences. ACM SIGCOMM Computer Communication Review 44, 4 (2015), 395--406. Google Scholar
- August Shi, Alex Gyori, Milos Gligoric, Andrey Zaytsev, and Darko Marinov. 2014. Balancing trade-offs in test-suite reduction. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 246--256. Google Scholar
- August Shi, Tifany Yung, Alex Gyori, and Darko Marinov. 2015. Comparing and combining test-suite reduction and regression test selection. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 237--247. Google Scholar
- StackOverFlow. 2014. Stack Overflow: what would be the base optimal throttle and seed for an application using monkey test? http://stackoverflow.com/questions/9778881/what-would-be-the-base-optimal-throttle-and-seed-for-an-application-using-monkey. (2014). Accessed: 2017-04-14.Google Scholar
- Shengqian Yang, Dacong Yan, Haowei Wu, Yan Wang, and Atanas Rountev. 2015. Static control-flow analysis of user-driven callbacks in Android applications. In Software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Conference on, Vol. 1. IEEE, 89--99. Google Scholar
- Wei Yang, Mukul R Prasad, and Tao Xic. 2013. A grey-box approach for automated GUI-model generation of mobile applications. In International Conference on Fundamental Approaches to Software Engineering. Springer, 250--265. Google Scholar
- Shin Yoo and Mark Harman. 2012. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability 22, 2 (2012), 67--120. Google Scholar
- Andreas Zeller. 1999. Yesterday, my program worked. Today, it does not. Why?. In Software Engineering - ESEC/FSE '99. Springer, 253--267. Google Scholar
- Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and isolating failure-inducing input. IEEE Transactions on Software Engineering 28, 2 (2002), 183--200. Google Scholar
- Hailong Zhang, Haowei Wu, and Atanas Rountev. 2016. Automated test generation for detection of leaks in Android applications. In Automation of Software Test (AST), 2016 IEEE/ACM 11th International Workshop in. IEEE, 64--70. Google Scholar
- Hao Zhong, Lu Zhang, and Hong Mei. 2008. An experimental study of four typical test suite reduction techniques. Information and Software Technology 50, 6 (2008), 534--546. Google Scholar
Index Terms
DetReduce
Comments