Abstract
Large-scale data centers and cloud computing have turned system configuration into a challenging problem. Several widely-publicized outages have been blamed not on software bugs, but on configuration bugs. To cope, thousands of organizations use system configuration languages to manage their computing infrastructure. Of these, Puppet is the most widely used with thousands of paying customers and many more open-source users. The heart of Puppet is a domain-specific language that describes the state of a system. Puppet already performs some basic static checks, but they only prevent a narrow range of errors. Furthermore, testing is ineffective because many errors are only triggered under specific machine states that are difficult to predict and reproduce. With several examples, we show that a key problem with Puppet is that configurations can be non-deterministic. This paper presents Rehearsal, a verification tool for Puppet configurations. Rehearsal implements a sound, complete, and scalable determinacy analysis for Puppet. To develop it, we (1) present a formal semantics for Puppet, (2) use several analyses to shrink our models to a tractable size, and (3) frame determinism-checking as decidable formulas for an SMT solver. Rehearsal then leverages the determinacy analysis to check other important properties, such as idempotency. Finally, we apply Rehearsal to several real-world Puppet configurations.
- Paul Anderson. Towards a High-Level Machine Configuration System. USENIX Large Installation System Administration Conference (LISA), 1994. Google Scholar
Digital Library
- Paul Anderson and Herry Herry. A Formal Semantics for the SmartFrog Configuration Language. Journal of Network and Systems Management, 24(2):309–345, 2016. Google Scholar
Digital Library
- Esben Andreasen and Anders Møller. Determinacy in Static Analysis for jQuery. ACM SIGPLAN Conference on Object Oriented Programming, Systems, Languages and Applications (OOPSLA), 2014. Google Scholar
Digital Library
- Konstantine Arkoudas, Karen Zee, Viktor Kuncak, and Martin Rinard. Verifying a file system implementation. International Conference on Formal Engineering Methods (ICFEM), 2004.Google Scholar
Cross Ref
- Augeas. Retrieved Apr 15 2016 from http://augeas.net.Google Scholar
- Robert L. Bocchino, Jr., Vikram S. Adve, Danny Dig, Sarita V. Adve, Stephen Heumann, Rakesh Komuravelli, Jeffrey Overbey, Patrick Simmons, and Hyojin Sung. A Type and Effect System for Deterministic Parallel Java. ACM SIGPLAN Conference on Object Oriented Programming, Systems, Languages and Applications (OOPSLA), 2009. Google Scholar
Digital Library
- Aaron Bohannon, J. Nathan Foster, Benjamin C. Pierce, Alexandre Pilkiewicz, and Alan Schmitt. Boomerang: Resourceful Lenses for String Data. ACM SIGPLANSIGACT Symposium on Principles of Programming Languages (POPL), 2008. Google Scholar
Digital Library
- Jacob Burnim and Koushik Sen. Asserting and Checking Determinism for Multithreaded Programs. Joint Meeting of the European Software Engineering Conference (ESEC) and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE), 2009. Google Scholar
Digital Library
- Maria Christakis, K. Rustan M. Leino, and Wolfram Schulte. Formalizing and Verifying a Modern Build Language. International Symposium on Formal Methods (FM), 2014.Google Scholar
Digital Library
- Thomas Delaet, Wouter Joosen, and Bart Vanbrabant. A survey of system configuration tools. USENIX Large Installation System Administration Conference (LISA), 2010. Google Scholar
Digital Library
- Leonardo De Moura and Nikolaj Bjørner. Z3: An Efficient SMT Solver. Tools and Algorithms for the Construction and Analysis of Systems (TACAS), 2008. Google Scholar
Digital Library
- Eelco Dolstra, Andreas Löh, and Nicholas Pierron. NixOS: A Purely Functional Linux Distribution. Journal of Functional Programming, 20(5–6):577–615, 2010. Google Scholar
Digital Library
- Jeffery Fischer, Rupak Majumdar, and Shahram Esmaeilsabzali. Engage: A Deployment Management System. ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2012. Google Scholar
Digital Library
- Philippa Gardner, Gian Ntzik, and Adam Wright. Local Reasoning about POSIX File Systems. European Symposium on Programming (ESOP), 2014.Google Scholar
- Arjun Guha, Rian Shambaugh, and Aaron Weiss. Rehearsal. Retrieved Apr 15, 2016 from http://plasma.cs.umass. edu/rehearsal.Google Scholar
- Bent Hagemark and Kenneth Zadeck. Site: A Language and System for Configuring Many Computers as One Computing Site. USENIX Large Installation System Administration Conference (LISA), 1989.Google Scholar
- Peng Huang, William J. Bolosky, and Abhishek Singh Yuanyuan Zhou. ConfValley: A Systematic Configuation Validation Framework for Cloud Services. European Conference on Computer Systems (EuroSys), 2015. Google Scholar
Digital Library
- Waldemar Hummer, Florian Rosenberg, Fábio Oliveira, and Tamar Eilam. Testing Idempotence and Convergence for Infrastructure as Code. ACM/IFIP/USENIX International Middleware Conference, 2013.Google Scholar
- Ming Kawaguchi, Patrick Rondon, Alexander Bakst, and Ranjit Jhala. Deterministic Parallelism via Liquid Effects. ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2012. Google Scholar
Digital Library
- Puppet Labs. Puppet Features: Idempotency. Retrieved Apr 15, 2016 from http://docs.puppetlabs.com/guides/ introduction.html#idempotency.Google Scholar
- John McCarthy. Towards a Mathematical Science of Computation. IFIP Congress, 1962.Google Scholar
- Carroll Morgan and Bernard Sufrin. Specification of the UNIX Filing System. IEEE Transactions on Software Engineering (TSE), 10(2):128–142, 1984. Google Scholar
Digital Library
- Tom Ridge, David Sheets, Thomas Tuerk, Anil Madhavapeddy, Andrea Giugliano, and Peter Sewell. SibylFS: formal specification and oracle-based testing for POSIX and real-world file systems. ACM Symposium on Operating Systems Principles (SOSP), 2015. Google Scholar
Digital Library
- Caitlin Sadowski, Stephen N. Freund, and Cormac Flanagan. SingleTrack: A dynamic determinism checker for multithreaded programs. European Symposium on Programming (ESOP), 2009. Google Scholar
Digital Library
- Eric Shamow. Inside Puppet: About Determinism. Retrieved Apr 15, 2016 from http://puppetlabs.com/blog/ inside-puppet-about-determinism.Google Scholar
- Alex Sherman, Philip A. Lisiecki, Andy Berkheimer, and Joel Wein. ACMS: The Akamai Configuration Management System. USENIX Symposium on Networked System Design and Implementation (NSDI), 2005. Google Scholar
Digital Library
- Chunqiang Tang, Thawan Kooburat, Pradeep Venkatachalam, Akshay Chandler, Zhe Wen, Aravind Narayanan, Patrick Dowell, and Robert Karl. Holistic Configuration Management at Facebook. ACM Symposium on Operating Systems Principles (SOSP), 2015. Google Scholar
Digital Library
- Chris Tucker, David Shuffleton, Ranjit Jhala, and Sorin Lerner. OPIUM: Optimal Package Install/Uninstall Manager. International Conference on Software Engineering (ICSE), 2007. Google Scholar
Digital Library
- David B. Tucker and Shriram Krishnamurthi. Programming Languages for Software Configuration. International Workshop on Software Configuration Management (SCM), 2001.Google Scholar
- Ubuntu. Details of package golang-go in trusty. Retrieved Apr 15, 2016 from http://packages.ubuntu.com/ trusty/devel/golang-go.Google Scholar
- Martin Vechev, Eran Yahav, Raghavan Raman, and Vivek Sarkar. Automatic Verification of Determinism for Structured Parallel Programs. International Static Analysis Symposium (SAS), 2010. Google Scholar
Digital Library
Index Terms
Rehearsal: a configuration verification tool for puppet
Recommendations
Rehearsal: a configuration verification tool for puppet
PLDI '16: Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and ImplementationLarge-scale data centers and cloud computing have turned system configuration into a challenging problem. Several widely-publicized outages have been blamed not on software bugs, but on configuration bugs. To cope, thousands of organizations use system ...
You as a Puppet: Evaluation of Telepresence User Interface for Puppetry
UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and TechnologyWe propose an immersive telepresence system for puppetry that transmits a human performer's body and facial movements into a puppet with audiovisual feedback to the performer. The cameras carried in place of puppet's eyes stream live video to the HMD ...
Scaling Puppet and Foreman for HPC
PEARC '18: Proceedings of the Practice and Experience on Advanced Research ComputingThe Ohio Supercomputer Center has deployed a Puppet configuration management and Foreman provisioning environment that scales to almost one thousand servers that are a mix of HPC cluster compute and service nodes as well as storage, web, and ...







Comments