Abstract
We present a novel approach for synthesizing robust relational layouts from examples. Given an application design consisting of a set of views and their location on the screen, we synthesize a relational layout that when rendered, places the components at that same location.
We present an end-to-end system, called InferUI, that addresses the above challenge in the context of Android. The system is based on the following technical contributions: (i) a formalization of the latest and most efficient ConstraintLayout class, capturing a rich set of relational constraints, (ii) a set of robustness properties designed to prevent common layout generalization errors, (iii) a synthesis algorithm that produces relational layouts that generalize across multiple screen sizes and resolutions, and (iv) a probabilistic model of constraints that guides the synthesizer towards layouts preferred by developers.
Our evaluation shows that InferUI is practically effective: it successfully synthesizes real world complex layouts obtained from top 500 GitHub and top 500 Google Play Store applications, succeeds in 100% of the cases when synthesizing layouts for a single device, and correctly generalizes 92% of the views across multiple devices, all without requiring additional specifications.
Supplemental Material
- Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles A. Sutton. 2015. Suggesting accurate method and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE ’15). ACM, New York, NY, USA, 38–49. Google Scholar
Digital Library
- Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. 2018. A Survey of Machine Learning for Big Code and Naturalness. ACM Comput. Surv. 51, 4, Article 81 (2018), 37 pages. Google Scholar
Digital Library
- Amigo. 2018. Amigo. https://play.google.com/store/apps/details?id=com.amigotrip.androidGoogle Scholar
- Asobimasu. 2018. Asobimasu. https://github.com/DipanshKhandelwal/AsobimasuGoogle Scholar
- Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to write programs. In 5th International Conference on Learning Representations (ICLR ’17).Google Scholar
- Tony Beltramelli. 2017. pix2code: Generating Code from a Graphical User Interface Screenshot. CoRR abs/1705.07962 (2017). arXiv: 1705.07962 http://arxiv.org/abs/1705.07962Google Scholar
- Pavol Bielik, Veselin Raychev, and Martin Vechev. 2016. PHOG: Probabilistic Model for Code. In Proceedings of The 33rd International Conference on Machine Learning (ICML ’16) , Vol. 48. PMLR, New York, NY, USA, 2933–2942. Google Scholar
Digital Library
- Candid. 2018. Candid. https://play.google.com/store/apps/details?id=in.voiceme.app.voicemeGoogle Scholar
- Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18) . ACM, New York, NY, USA, 665–676. Google Scholar
Digital Library
- Shauvik Roy Choudhary, Mukul R. Prasad, and Alessandro Orso. 2012. CrossCheck: Combining Crawling and Differencing to Better Detect Cross-browser Incompatibilities in Web Applications. In Proceedings of the IEEE 5th International Conference on Software Testing, Verification and Validation (ICST ’12) . IEEE Computer Society, Washington, DC, USA, 171–180. Google Scholar
Digital Library
- Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob Albers. 2016. Programmatic and Direct Manipulation, Together at Last. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’16) . ACM, New York, NY, USA, 341–354. Google Scholar
Digital Library
- Alex Corrado, Avery Lamp, Brendan Walsh, Edward Aryee, Erica Yuen, George Matthews, Jen Madiedo, Jeremie Laval, Luis Torres, Maddy Leger, Paris Hsu, Patrick Chen, Tim Rait, Seth Chong, Wjdan Alharthi, and Xiao Tu. 2018. Ink To Code. https://www.microsoft.com/en-us/garage/profiles/ink-to-code/Google Scholar
- Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’08/ETAPS’08) . Springer-Verlag, Berlin, Heidelberg, 337–340. http://dl.acm.org/citation.cfm?id=1792734.1792766 Google Scholar
Digital Library
- Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2017. Learning to Infer Graphics Programs from Hand-Drawn Images. CoRR abs/1707.09627 (2017). arXiv: 1707.09627 http://arxiv.org/abs/1707.09627Google Scholar
- Kevin Ellis, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2016. Sampling for Bayesian Program Learning. In Advances in Neural Information Processing Systems 29 . Curran Associates, Inc., Barcelona, Spain, 1297–1305. http: //papers.nips.cc/paper/6082-sampling-for-bayesian-program-learning.pdf Google Scholar
Digital Library
- FBook. 2018. FBook. https://play.google.com/store/apps/details?id=com.framgia.bookGoogle Scholar
- Ruozi Huang, Yonghao Long, and Xiangping Chen. 2016. Automatically Generating Web Page From A Mockup. In The 28th International Conference on Software Engineering and Knowledge Engineering (SEKE ’16) . KSI Research Inc. and Knowledge Systems Institute Graduate School, Redwood City, San Francisco Bay, USA, 589–594.Google Scholar
- Geoffrey Irving, Christian Szegedy, Alexander A Alemi, Niklas Een, Francois Chollet, and Josef Urban. 2016. DeepMath - Deep Sequence Models for Premise Selection. In Advances in Neural Information Processing Systems 29. Curran Associates, Inc., Barcelona, Spain, 2235–2243. http://papers.nips.cc/paper/6280-deepmath-deep-sequence-models-for-premise-selection. pdf Google Scholar
Digital Library
- W. E. Johnson. 1932. Probability: The Deductive and Inductive Problems. Mind 41, 164 (1932), 409–423. http://www.jstor. org/stable/2250183Google Scholar
- Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gulwani. 2018. NeuralGuided Deductive Search for Real-Time Program Synthesis from Examples. In 6th International Conference on Learning Representations (ICLR ’18) .Google Scholar
- Omer Katz, Ran El-Yaniv, and Eran Yahav. 2016. Estimating Types in Binaries Using Predictive Modeling. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’16) . ACM, New York, NY, USA, 313–326. Google Scholar
Digital Library
- Woosuk Lee, Kihong Heo, Rajeev Alur, and Mayur Naik. 2018. Accelerating Search-based Program Synthesis Using Learned Probabilistic Models. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’18) . ACM, New York, NY, USA, 436–449. Google Scholar
Digital Library
- George James Lidstone. 1920. Note on the general case of the bayes-laplace formula for inductive or a posteriori probabilities. Transactions of the Faculty of Actuaries 8 (1920), 182–192.Google Scholar
- Fan Long and Martin Rinard. 2016. Automatic Patch Generation by Learning Correct Code. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’16) . ACM, New York, NY, USA, 298–312. Google Scholar
Digital Library
- Chris Maddison and Daniel Tarlow. 2014. Structured Generative Models of Natural Source Code. In Proceedings of the 31st International Conference on Machine Learning (ICML ’14) , Vol. 32-II. PMLR, New York, NY, USA, 649–657. http: //dl.acm.org/citation.cfm?id=3044805.3044965 Google Scholar
Digital Library
- Sonal Mahajan, Bailan Li, Pooyan Behnamghader, and William G. J. Halfond. 2016. Using Visual Symptoms for Debugging Presentation Failures in Web Applications. In Proceedings of the IEEE 9th International Conference on Software Testing, Verification and Validation (ICST ’16) . IEEE Computer Society, Washington, DC, USA, 191–201.Google Scholar
- Aditya Menon, Omer Tamuz, Sumit Gulwani, Butler Lampson, and Adam Kalai. 2013. A Machine Learning Framework for Programming by Example. In Proceedings of The 30rd International Conference on Machine Learning (ICML ’13), Vol. 28-I. PMLR, New York, NY, USA, 187–195. Google Scholar
Digital Library
- Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated Reporting of GUI Design Violations for Mobile Apps. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18). ACM, New York, NY, USA, 165–175. Google Scholar
Digital Library
- Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI (T). In Proceedings of the 30th ACM/IEEE International Conference on Automated Software Engineering (ASE ’15). IEEE Computer Society, Washington, DC, USA, 248–259.Google Scholar
Digital Library
- Pavel Panchekha, Adam T. Geller, Michael D. Ernst, Zachary Tatlock, and Shoaib Kamil. 2018. Verifying That Web Pages Have Accessible Layout. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’18) . ACM, New York, NY, USA, 1–14. Google Scholar
Digital Library
- Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting Program Properties from "Big Code". In Proceedings of the 42Nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’15) . ACM, New York, NY, USA, 111–124. Google Scholar
Digital Library
- Steven P. Reiss. 2014. Seeking the User Interface. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE ’14) . ACM, New York, NY, USA, 103–114. Google Scholar
Digital Library
- Shauvik Roy Choudhary, Mukul R. Prasad, and Alessandro Orso. 2013. X-PERT: Accurate Identification of Cross-browser Issues in Web Applications. In Proceedings of the 2013 International Conference on Software Engineering (ICSE ’13). IEEE Press, Piscataway, NJ, USA, 702–711. http://dl.acm.org/citation.cfm?id=2486788.2486881 Google Scholar
Digital Library
- Amanda Swearngin, Mira Dontcheva, Wilmot Li, Joel Brandt, Morgan Dixon, and Andrew J. Ko. 2018. Rewire: Interface Design Assistance from Examples. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18) . ACM, New York, NY, USA, Article 504, 12 pages. Google Scholar
Digital Library
- Clemens Zeidler, Christof Lutteroth, Wolfgang Stuerzlinger, and Gerald Weber. 2013a. Evaluating Direct Manipulation Operations for Constraint-Based Layout. In 14th IFIP Conference on Human-Computer Interaction (INTERACT ’13). Springer, Berlin, Heidelberg, 513–529.Google Scholar
- Clemens Zeidler, Christof Lutteroth, Wolfgang Sturzlinger, and Gerald Weber. 2013b. The Auckland Layout Editor: An Improved GUI Layout Specification Process. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST ’13) . ACM, New York, NY, USA, 343–352. Google Scholar
Digital Library
- Clemens Zeidler, Gerald Weber, Wolfgang Stuerzlinger, and Christof Lutteroth. 2017. Automatic Generation of User Interface Layouts for Alternative Screen Orientations. In 16th IFIP Conference on Human-Computer Interaction (INTERACT ’17). Springer, Berlin, Heidelberg, 13–35.Google Scholar
Index Terms
Robust relational layout synthesis from examples for Android
Recommendations
Synthesis of web layouts from examples
ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringWe present a new technique for synthesizing dynamic, constraint-based visual layouts from examples. Our technique tackles two major challenges of layout synthesis. First, realistic layouts, especially on the web, often contain hundreds of elements, so ...
Conflict-Driven Synthesis for Layout Engines
Modern web browsers rely on layout engines to convert HTML documents to layout trees that specify color, size, and position. However, existing layout engines are notoriously difficult to maintain because of the complexity of web standards. This is ...
Synthesis of layout engines from relational constraints
OOPSLA 2015: Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and ApplicationsWe present an algorithm for synthesizing efficient document layout engines from compact relational specifications. These specifications are compact in that a single specification can produce multiple engines, each for a distinct layout situation, i.e., ...






Comments