Rico: A Mobile App Dataset for Building Data-Driven Design Applications

ABSTRACT

Data-driven models help mobile app designers understand best practices and trends, and can be used to make predictions about design performance and support the creation of adaptive UIs. This paper presents Rico, the largest repository of mobile app designs to date, created to support five classes of data-driven applications: design search, UI layout generation, UI code generation, user interaction modeling, and user perception prediction. To create Rico, we built a system that combines crowdsourcing and automation to scalably mine design and interaction data from Android apps at runtime. The Rico dataset contains design data from more than 9.7k Android apps spanning 27 categories. It exposes visual, textual, structural, and interactive design properties of more than 72k unique UI screens. To demonstrate the kinds of applications that Rico enables, we present results from training an autoencoder for UI layout similarity, which supports query- by-example search over UIs.

References

1. Android Activities, 2016. https://developer.android.com/guide/components/activities.html.
2. Database of Android Apps on Kaggle, 2016. https://www.kaggle.com/orgesleka/android-apps.
3. UI Overview, 2016. https://developer.android.com/guide/topics/ui/overview.html.
4. Alharbi, K., and Yeh, T. Collect, decompile, extract, stats, and diff: Mining design pattern changes in Android apps. In Proc. MobileHCI (2015).
5. Amini, S. Analyzing Mobile App Privacy Using Computation and Crowdsourcing. PhD thesis, Carnegie Mellon University, 2014.
6. Azim, T., and Neamtiu, I. Targeted and depth-first exploration for systematic testing of android apps. In ACM SIGPLAN Notices (2013).
7. Bell, S., and Bala, K. Learning visual similarity for product design with convolutional neural networks. ACM TOG (2015).
8. Bengio, Y. Learning deep architectures for ai. Foundations and Trends in Machine Learning 2, 1 (2009).
9. Bhoraskar, R., Han, S., Jeon, J., Azim, T., Chen, S., Jung, J., Nath, S., Wang, R., and Wetherall, D. Brahmastra: Driving apps to test the security of third-party components. In Proc. SEC (2014).
10. Brandt, J., Dontcheva, M., Weskamp, M., and Klemmer, S. R. Example-centric programming: integrating web search into the development environment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2010), 513--522.
11. Deka, B., Huang, Z., and Kumar, R. ERICA: Interaction mining mobile apps. In Proc. UIST (2016).
12. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proc. CVPR (2009).
13. Eckert, C., and Stacey, M. Sources of inspiration: A language of design. Design Studies 21, 5 (2000), 523--538.
14. Eckert, C., Stacey, M., and Earl, C. References to past designs. Studying Designers 5 (2005), 3--21.
15. Frank, M., Dong, B., Felt, A. P., and Song, D. Mining permission request patterns from android and facebook applications. In Proc. ICDM (2012).
16. Fu, B., Lin, J., Li, L., Faloutsos, C., Hong, J., and Sadeh, N. Why people hate your app: Making sense of user feedback in a mobile app store. In Proc. KDD (2013).
17. Funkhouser, T., Min, P., Kazhdan, M., Chen, J., Halderman, A., Dobkin, D., and Jacobs, D. A search engine for 3D models. ACM TOG (2003).
18. Koch, J., and Oulasvirta, A. Computational layout perception using gestalt laws. In Proc. CHI (Extended Abstracts) (2016).
19. Kumar, R., Satyanarayan, A., Torres, C., Lim, M., Ahmad, S., Klemmer, S. R., and Talton, J. O. Webzeitgeist: Design mining the web. In Proc. CHI (2013).
20. Lee, K., Flinn, J., Giuli, T., Noble, B., and Peplin, C. Amc: Verifying user interface properties for vehicular applications. In Proc. Mobisys (2013).
21. Leuthold, S., Schmutz, P., Bargas-Avila, J. A., Tuch, A. N., and Opwis, K. Vertical versus dynamic menus on the world wide web: Eye tracking study measuring the influence of menu design and task complexity on user performance and subjective preference. Computers in human behavior 27, 1 (2011), 459--472.
22. McAuley, J., Targett, C., Shi, Q., and Van Den Hengel, A. Image-based recommendations on styles and substitutes. In Proc SIGIR, ACM (2015), 43--52.
23. Miller, S. R., and Bailey, B. P. Searching for inspiration: An in-depth look at designers example finding practices. In ASME 2014 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (2014).
24. Miniukovich, A., and De Angeli, A. Visual impressions of mobile app interfaces. In Proc. Nordic CHI (2014).
25. Miniukovich, A., and De Angeli, A. Computation of interface aesthetics. In Proc. CHI (2015).
26. Miniukovich, A., and De Angeli, A. Visual diversity and user interface quality. In Proc. British HCI (2015).
27. Miniukovich, A., and De Angeli, A. Pick me!: Getting noticed on google play. In Proc. CHI (2016).
28. Nair, V., and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proc. ICML (2010), 807--814.
29. Nguyen, T. A., and Csallner, C. Reverse engineering mobile application user interfaces with REMAUI. In Proc. ASE (2015).
30. O'Donovan, P., Agarwala, A., and Hertzmann, A. Learning layouts for single-page graphic designs. IEEE TVCG (2014).
31. Reinecke, K., Yeh, T., Miratrix, L., Mardiko, R., Zhao, Y., Liu, J., and Gajos, K. Z. Predicting users' first impressions of website aesthetics with a quantification of perceived visual complexity and colorfulness. In Proc. CHI (2013).
32. Ritchie, D., Kejriwal, A. A., and Klemmer, S. R. d. tour: Style-based exploration of design example galleries. In Proc. UIST (2011).
33. Sahami Shirazi, A., Henze, N., Schmidt, A., Goldberg, R., Schmidt, B., and Schmauder, H. Insights into layout patterns of mobile user interfaces by an automatic analysis of Android apps. In Proc. EICS (2013).
34. Szydlowski, M., Egele, M., Kruegel, C., and Vigna, G. Challenges for dynamic analysis of iOS applications. In Open Problems in Network Security. Springer, 2012, 65--77.
35. Talton, J., Yang, L., Kumar, R., Lim, M., Goodman, N., and Mech, R. Learning design patterns with bayesian grammar induction. In Proc. UIST, ACM (2012).
36. Tractinsky, N., Inbar, O., Tsimhoni, O., and Seder, T. Slow down, you move too fast: Examining animation aesthetics to promote eco-driving. In Proc. AutoUI, ACM (2011), 193--202.
37. van der Geest, T., and Loorbach, N. Testing the visual consistency of web sites. Technical communication 52, 1 (2005), 27--36.
38. Viennot, N., Garcia, E., and Nieh, J. A measurement study of google play. In ACM SIGMETRICS Performance Evaluation Review, vol. 42, ACM (2014), 221--233.
39. Yi, L., Guibas, L., Hertzmann, A., Kim, V. G., Su, H., and Yumer, E. Learning hierarchical shape segmentation and labeling from online repositories. In Proc. SIGGRAPH (2017).

Index Terms

• Software and its engineering

  • Software creation and management

    • Designing software

      • Software implementation planning

        • Software design techniques

    • Software development process management