ABSTRACT
SUGILITE is a new programming-by-demonstration (PBD) system that enables users to create automation on smartphones. SUGILITE uses Android's accessibility API to support automating arbitrary tasks in any Android app (or even across multiple apps). When the user gives verbal commands that SUGILITE does not know how to execute, the user can demonstrate by directly manipulating the regular apps' user interface. By leveraging the verbal instructions, the demonstrated procedures, and the apps? UI hierarchy structures, SUGILITE can automatically generalize the script from the recorded actions, so SUGILITE learns how to perform tasks with different variations and parameters from a single demonstration. Extensive error handling and context checking support forking the script when new situations are encountered, and provide robustness if the apps change their user interface. Our lab study suggests that users with little or no programming knowledge can successfully automate smartphone tasks using SUGILITE.
References
- James Allen, Nathanael Chambers, George Ferguson, et al. 2007. Plow: A collaborative task learning agent. In Proceedings of the National Conference on Artificial Intelligence, 1514. Google Scholar
Digital Library
- V. Antila, J. Polet, A. Lämsä, and J. Liikka. 2012. RoutineMaker: Towards end-user automation of daily routines using smartphones. In 2012 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), 399--402.Google Scholar
- Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. 2009. A survey of robot learning from demonstration. Robotics and autonomous systems 57, 5: 469--483. Google Scholar
Digital Library
- Amos Azaria, Jayant Krishnamurthy, and Tom M. Mitchell. 2016. Instructable intelligent personal agent. In Proc. The 30th AAAI Conference on Artificial Intelligence (AAAI). Google Scholar
Digital Library
- Lawrence Bergman, Vittorio Castelli, Tessa Lau, and Daniel Oblinger. 2005. DocWizards: A System for Authoring Follow-me Documentation Wizards. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (UIST '05), 191--200. Google Scholar
Digital Library
- Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. 2008. Robot programming by demonstration. In Springer handbook of robotics. Springer, 1371-- 1394.Google Scholar
- Michael Bolin, Matthew Webber, Philip Rha, Tom Wilson, and Robert C. Miller. 2005. Automation and customization of rendered web pages. In Proceedings of the 18th annual ACM symposium on User interface software and technology, 163--172. Google Scholar
Digital Library
- David L. Chen and Raymond J. Mooney. 2011. Learning to Interpret Natural Language Navigation Instructions from Observations. In AAAI, 1--2. Google Scholar
Digital Library
- Jiun-Hung Chen and Daniel S. Weld. 2008. Recovering from Errors During Programming by Demonstration. In Proceedings of the 13th International Conference on Intelligent User Interfaces (IUI '08), 159--168. Google Scholar
Digital Library
- Yun-Nung Chen, Ming Sun, and Alexander I. Rudnicky. 2015. Matrix factorization with domain knowledge and behavioral patterns for intent modeling. In NIPS Workshop on Machine Learning for SLU and Interaction.Google Scholar
- Allen Cypher and Daniel Conrad Halbert. 1993. Watch what I do: programming by demonstration. MIT press. Google Scholar
Digital Library
- Martin R. Frank and James D. Foley. 1993. Model-based User Interface Design by Example and by Interview. In Proceedings of the 6th Annual ACM Symposium on User Interface Software and Technology (UIST '93), 129--137. Google Scholar
Digital Library
- Martin R. Frank and James D. Foley. 1994. A Pure Reasoning Engine for Programming by Demonstration. In Proceedings of the 7th Annual ACM Symposium on User Interface Software and Technology (UIST '94), 95--101. Google Scholar
Digital Library
- Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: automatically generating user interfaces. In Proceedings of the 9th international conference on Intelligent user interfaces, 93--100. Google Scholar
Digital Library
- Floraine Grabler, Maneesh Agrawala, Wilmot Li, Mira Dontcheva, and Takeo Igarashi. 2009. Generating Photo Manipulation Tutorials by Demonstration. In ACM SIGGRAPH 2009 Papers (SIGGRAPH '09), 66:1--66:9. Google Scholar
Digital Library
- Ting-Hao Kenneth Huang, Amos Azaria, and Jeffrey P. Bigham. 2016. InstructableCrowd: Creating IF-THEN Rules via Conversations with the Crowd. 1555--1562. Google Scholar
Digital Library
- IFTTT. IFTTT. IFTTT / Connect the apps you love.Google Scholar
- Jiepu Jiang, Ahmed Hassan Awadallah, Rosie Jones, et al. 2015. Automatic Online Evaluation of Intelligent Assistants. In Proceedings of the 24th International Conference on World Wide Web (WWW '15), 506--516. Google Scholar
Digital Library
- Ken Kahn. 1996. Toontalk TM - an animated programming environment for children. Journal of Visual Languages & Computing 7, 2: 197--217.Google Scholar
Cross Ref
- Simon Khalaf. Seven Years Into The Mobile Revolution: Content is King... Again. Yahoo Developer Network.Google Scholar
- Tessa Lau. 2009. Why Programming-By-Demonstration Systems Fail: Lessons Learned for Usable AI. AI Magazine 30, 4: 65.Google Scholar
Cross Ref
- Tessa A. Lau and Daniel S. Weld. 1999. Programming by Demonstration: An Inductive Learning Formulation. In Proceedings of the 4th International Conference on Intelligent User Interfaces (IUI '99), 145--152. Google Scholar
Digital Library
- Gilly Leshed, Eben M. Haber, Tara Matthews, and Tessa Lau. 2008. CoScripter: Automating & Sharing How-to Knowledge in the Enterprise. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08), 1719--1728. Google Scholar
Digital Library
- Henry Lieberman. 2001. Your wish is my command: Programming by example. Morgan Kaufmann.Google Scholar
- Pattie Maes. 1994. Agents That Reduce Work and Information Overload. Commun. ACM 37, 7: 30--40. Google Scholar
Digital Library
- Rodrigo de A. Maués and Simone Diniz Junqueira Barbosa. 2013. Keep Doing What I Just Did: Automating Smartphones by Demonstration. In Proceedings of the 15th International Conference on Human-computer Interaction with Mobile Devices and Services (MobileHCI '13), 295--303. Google Scholar
Digital Library
- Richard G. McDaniel and Brad A. Myers. 1997. Gamut: Demonstrating Whole Applications. In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology (UIST '97), 81--82. Google Scholar
Digital Library
- Richard G. McDaniel and Brad A. Myers. 1999. Getting More out of Programming-by-demonstration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99), 442--449. Google Scholar
Digital Library
- Francesmary Modugno and Brad A. Myers. 1994. Pursuit: Graphically Representing Programs in a Demonstrational Visual Shell. In Conference Companion on Human Factors in Computing Systems (CHI '94), 455-- 456. Google Scholar
Digital Library
- Brad. A. Myers. 1986. Visual Programming, Programming by Example, and Program Visualization: A Taxonomy. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '86), 59-- 66. Google Scholar
Digital Library
- Brad A. Myers. 1990. Creating user interfaces using programming by example, visual programming, and constraints. ACM Transactions on Programming Languages and Systems (TOPLAS) 12, 2: 143--177. Google Scholar
Digital Library
- Brad A. Myers. 1991. Graphical techniques in a spreadsheet for specifying user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 243--249. Google Scholar
Digital Library
- Brad A. Myers and Richard McDaniel. 2001. Sometimes you need a little intelligence, sometimes you need a lot. Your Wish is My Command: Programming by Example. San Francisco, CA: Morgan Kaufmann Publishers: 45-- 60. Google Scholar
Digital Library
- Brad A. Myers, Brad Vandcr Zanden, and Roger B. Dannenberg. 1989. Creating graphical interactive application objects by demonstration. In Proceedings of the 2nd annual ACM SIGGRAPH symposium on User interface software and technology, 95--104. Google Scholar
Digital Library
- Shin 'ichiro Nakaoka, Atsushi Nakazawa, Fumio Kanehiro, et al. 2007. Learning from observation paradigm: Leg task models for enabling a biped humanoid robot to imitate human dances. The International Journal of Robotics Research 26, 8: 829--844. Google Scholar
Digital Library
- A. Namoun, A. Daskalopoulou, N. Mehandjiev, and Z. Xun. 2016. Exploring Mobile End User Development: Existing Use and Design Factors. IEEE Transactions on Software Engineering PP, 99: 1--1. Google Scholar
Digital Library
- Nielsen. 2015. So Many Apps, So Much More Time for Entertainment.Google Scholar
- Lenin Ravindranath, Arvind Thiagarajan, Hari Balakrishnan, and Samuel Madden. 2012. Code in the Air: Simplifying Sensing and Coordination Tasks on Smartphones. In Proceedings of the Twelfth Workshop on Mobile Computing Systems & Applications (HotMobile '12), 4:1--4:6. Google Scholar
Digital Library
- André Rodrigues. 2015. Breaking Barriers with Assistive Macros. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS '15), 351--352. Google Scholar
Digital Library
- Ben Shneiderman, Catherine Plaisant, Maxine Cohen, Steven Jacobs, Niklas Elmqvist, and Nicholas Diakopoulos. 2016. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Pearson, Boston. Google Scholar
Digital Library
- Ming Sun, Yun-Nung Chen, and Alexander I. Rudnicky. 2016. HELPR: A Framework to Break the Barrier across Domains in Spoken Dialog Systems. In International Workshop on Spoken Dialog Systems.Google Scholar
- Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. 2015. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the Twenty-Fourth international joint conference on Artificial Intelligence (IJCAI). Google Scholar
Digital Library
- Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. 2009. Sikuli: Using GUI Screenshots for Search and Automation. In Proceedings of the 22Nd Annual ACM Symposium on User Interface Software and Technology (UIST '09), 183--192. Google Scholar
Digital Library
- Sha Zhao, Julian Ramos, Jianrong Tao, et al. 2016. Discovering Different Kinds of Smartphone Users Through Their Application Usage Behaviors. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16), 498--509. Google Scholar
Digital Library
- SiriKit - Apple Developer. https://developer.apple.com/sirikit/Google Scholar
- Automate everyday automation for Android LlamaLab. http://llamalab.com/automate/Google Scholar
- Workato - Connect your apps. Automate your work. Workato. https://www.workato.com/Google Scholar
Supplemental Material
Index Terms
SUGILITE


Brad A. Myers


Comments