skip to main content
research-article
Open Access

Robust relational layout synthesis from examples for Android

Published:24 October 2018Publication History
Skip Abstract Section

Abstract

We present a novel approach for synthesizing robust relational layouts from examples. Given an application design consisting of a set of views and their location on the screen, we synthesize a relational layout that when rendered, places the components at that same location.

We present an end-to-end system, called InferUI, that addresses the above challenge in the context of Android. The system is based on the following technical contributions: (i) a formalization of the latest and most efficient ConstraintLayout class, capturing a rich set of relational constraints, (ii) a set of robustness properties designed to prevent common layout generalization errors, (iii) a synthesis algorithm that produces relational layouts that generalize across multiple screen sizes and resolutions, and (iv) a probabilistic model of constraints that guides the synthesizer towards layouts preferred by developers.

Our evaluation shows that InferUI is practically effective: it successfully synthesizes real world complex layouts obtained from top 500 GitHub and top 500 Google Play Store applications, succeeds in 100% of the cases when synthesizing layouts for a single device, and correctly generalizes 92% of the views across multiple devices, all without requiring additional specifications.

Skip Supplemental Material Section

Supplemental Material

a156-bielik.webm

References

  1. Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles A. Sutton. 2015. Suggesting accurate method and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE ’15). ACM, New York, NY, USA, 38–49. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. 2018. A Survey of Machine Learning for Big Code and Naturalness. ACM Comput. Surv. 51, 4, Article 81 (2018), 37 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Amigo. 2018. Amigo. https://play.google.com/store/apps/details?id=com.amigotrip.androidGoogle ScholarGoogle Scholar
  4. Asobimasu. 2018. Asobimasu. https://github.com/DipanshKhandelwal/AsobimasuGoogle ScholarGoogle Scholar
  5. Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to write programs. In 5th International Conference on Learning Representations (ICLR ’17).Google ScholarGoogle Scholar
  6. Tony Beltramelli. 2017. pix2code: Generating Code from a Graphical User Interface Screenshot. CoRR abs/1705.07962 (2017). arXiv: 1705.07962 http://arxiv.org/abs/1705.07962Google ScholarGoogle Scholar
  7. Pavol Bielik, Veselin Raychev, and Martin Vechev. 2016. PHOG: Probabilistic Model for Code. In Proceedings of The 33rd International Conference on Machine Learning (ICML ’16) , Vol. 48. PMLR, New York, NY, USA, 2933–2942. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Candid. 2018. Candid. https://play.google.com/store/apps/details?id=in.voiceme.app.voicemeGoogle ScholarGoogle Scholar
  9. Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18) . ACM, New York, NY, USA, 665–676. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Shauvik Roy Choudhary, Mukul R. Prasad, and Alessandro Orso. 2012. CrossCheck: Combining Crawling and Differencing to Better Detect Cross-browser Incompatibilities in Web Applications. In Proceedings of the IEEE 5th International Conference on Software Testing, Verification and Validation (ICST ’12) . IEEE Computer Society, Washington, DC, USA, 171–180. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob Albers. 2016. Programmatic and Direct Manipulation, Together at Last. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’16) . ACM, New York, NY, USA, 341–354. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Alex Corrado, Avery Lamp, Brendan Walsh, Edward Aryee, Erica Yuen, George Matthews, Jen Madiedo, Jeremie Laval, Luis Torres, Maddy Leger, Paris Hsu, Patrick Chen, Tim Rait, Seth Chong, Wjdan Alharthi, and Xiao Tu. 2018. Ink To Code. https://www.microsoft.com/en-us/garage/profiles/ink-to-code/Google ScholarGoogle Scholar
  13. Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’08/ETAPS’08) . Springer-Verlag, Berlin, Heidelberg, 337–340. http://dl.acm.org/citation.cfm?id=1792734.1792766 Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2017. Learning to Infer Graphics Programs from Hand-Drawn Images. CoRR abs/1707.09627 (2017). arXiv: 1707.09627 http://arxiv.org/abs/1707.09627Google ScholarGoogle Scholar
  15. Kevin Ellis, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2016. Sampling for Bayesian Program Learning. In Advances in Neural Information Processing Systems 29 . Curran Associates, Inc., Barcelona, Spain, 1297–1305. http: //papers.nips.cc/paper/6082-sampling-for-bayesian-program-learning.pdf Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. FBook. 2018. FBook. https://play.google.com/store/apps/details?id=com.framgia.bookGoogle ScholarGoogle Scholar
  17. Ruozi Huang, Yonghao Long, and Xiangping Chen. 2016. Automatically Generating Web Page From A Mockup. In The 28th International Conference on Software Engineering and Knowledge Engineering (SEKE ’16) . KSI Research Inc. and Knowledge Systems Institute Graduate School, Redwood City, San Francisco Bay, USA, 589–594.Google ScholarGoogle Scholar
  18. Geoffrey Irving, Christian Szegedy, Alexander A Alemi, Niklas Een, Francois Chollet, and Josef Urban. 2016. DeepMath - Deep Sequence Models for Premise Selection. In Advances in Neural Information Processing Systems 29. Curran Associates, Inc., Barcelona, Spain, 2235–2243. http://papers.nips.cc/paper/6280-deepmath-deep-sequence-models-for-premise-selection. pdf Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. W. E. Johnson. 1932. Probability: The Deductive and Inductive Problems. Mind 41, 164 (1932), 409–423. http://www.jstor. org/stable/2250183Google ScholarGoogle Scholar
  20. Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gulwani. 2018. NeuralGuided Deductive Search for Real-Time Program Synthesis from Examples. In 6th International Conference on Learning Representations (ICLR ’18) .Google ScholarGoogle Scholar
  21. Omer Katz, Ran El-Yaniv, and Eran Yahav. 2016. Estimating Types in Binaries Using Predictive Modeling. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’16) . ACM, New York, NY, USA, 313–326. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Woosuk Lee, Kihong Heo, Rajeev Alur, and Mayur Naik. 2018. Accelerating Search-based Program Synthesis Using Learned Probabilistic Models. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’18) . ACM, New York, NY, USA, 436–449. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. George James Lidstone. 1920. Note on the general case of the bayes-laplace formula for inductive or a posteriori probabilities. Transactions of the Faculty of Actuaries 8 (1920), 182–192.Google ScholarGoogle Scholar
  24. Fan Long and Martin Rinard. 2016. Automatic Patch Generation by Learning Correct Code. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’16) . ACM, New York, NY, USA, 298–312. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Chris Maddison and Daniel Tarlow. 2014. Structured Generative Models of Natural Source Code. In Proceedings of the 31st International Conference on Machine Learning (ICML ’14) , Vol. 32-II. PMLR, New York, NY, USA, 649–657. http: //dl.acm.org/citation.cfm?id=3044805.3044965 Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Sonal Mahajan, Bailan Li, Pooyan Behnamghader, and William G. J. Halfond. 2016. Using Visual Symptoms for Debugging Presentation Failures in Web Applications. In Proceedings of the IEEE 9th International Conference on Software Testing, Verification and Validation (ICST ’16) . IEEE Computer Society, Washington, DC, USA, 191–201.Google ScholarGoogle Scholar
  27. Aditya Menon, Omer Tamuz, Sumit Gulwani, Butler Lampson, and Adam Kalai. 2013. A Machine Learning Framework for Programming by Example. In Proceedings of The 30rd International Conference on Machine Learning (ICML ’13), Vol. 28-I. PMLR, New York, NY, USA, 187–195. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated Reporting of GUI Design Violations for Mobile Apps. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18). ACM, New York, NY, USA, 165–175. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI (T). In Proceedings of the 30th ACM/IEEE International Conference on Automated Software Engineering (ASE ’15). IEEE Computer Society, Washington, DC, USA, 248–259.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Pavel Panchekha, Adam T. Geller, Michael D. Ernst, Zachary Tatlock, and Shoaib Kamil. 2018. Verifying That Web Pages Have Accessible Layout. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’18) . ACM, New York, NY, USA, 1–14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting Program Properties from "Big Code". In Proceedings of the 42Nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’15) . ACM, New York, NY, USA, 111–124. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Steven P. Reiss. 2014. Seeking the User Interface. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE ’14) . ACM, New York, NY, USA, 103–114. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Shauvik Roy Choudhary, Mukul R. Prasad, and Alessandro Orso. 2013. X-PERT: Accurate Identification of Cross-browser Issues in Web Applications. In Proceedings of the 2013 International Conference on Software Engineering (ICSE ’13). IEEE Press, Piscataway, NJ, USA, 702–711. http://dl.acm.org/citation.cfm?id=2486788.2486881 Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Amanda Swearngin, Mira Dontcheva, Wilmot Li, Joel Brandt, Morgan Dixon, and Andrew J. Ko. 2018. Rewire: Interface Design Assistance from Examples. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18) . ACM, New York, NY, USA, Article 504, 12 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Clemens Zeidler, Christof Lutteroth, Wolfgang Stuerzlinger, and Gerald Weber. 2013a. Evaluating Direct Manipulation Operations for Constraint-Based Layout. In 14th IFIP Conference on Human-Computer Interaction (INTERACT ’13). Springer, Berlin, Heidelberg, 513–529.Google ScholarGoogle Scholar
  36. Clemens Zeidler, Christof Lutteroth, Wolfgang Sturzlinger, and Gerald Weber. 2013b. The Auckland Layout Editor: An Improved GUI Layout Specification Process. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST ’13) . ACM, New York, NY, USA, 343–352. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Clemens Zeidler, Gerald Weber, Wolfgang Stuerzlinger, and Christof Lutteroth. 2017. Automatic Generation of User Interface Layouts for Alternative Screen Orientations. In 16th IFIP Conference on Human-Computer Interaction (INTERACT ’17). Springer, Berlin, Heidelberg, 13–35.Google ScholarGoogle Scholar

Index Terms

  1. Robust relational layout synthesis from examples for Android

                Recommendations

                Comments

                Login options

                Check if you have access through your login credentials or your institution to get full access on this article.

                Sign in

                Full Access

                PDF Format

                View or Download as a PDF file.

                PDF

                eReader

                View online with eReader.

                eReader
                About Cookies On This Site

                We use cookies to ensure that we give you the best experience on our website.

                Learn more

                Got it!