Abstract
The POSIX shell is a widely deployed, powerful tool for managing computer systems. The shell is the expert’s control panel, a necessary tool for configuring, compiling, installing, maintaining, and deploying systems. Even though it is powerful, critical infrastructure, the POSIX shell is maligned and misunderstood. Its power and its subtlety are a dangerous combination.
We define a formal, mechanized, executable small-step semantics for the POSIX shell, which we call Smoosh. We compared Smoosh against seven other shells that aim for some measure of POSIX compliance (bash, dash, zsh, OSH, mksh, ksh93, and yash). Using three test suites—the POSIX test suite, the Modernish test suite and shell diagnostic, and a test suite of our own device—we found Smoosh’s semantics to be the most conformant to the POSIX standard. Modernish judges Smoosh to have the fewest bugs (just one, from using dash’s parser) and no quirks. To show that our semantics is useful beyond yielding a conformant, executable shell, we also implemented a symbolic stepper to illuminate the subtle behavior of the shell.
Smoosh will serve as a foundation for formal study of the POSIX shell, supporting research on and development of new shells, new tooling for shells, and new shell designs.
Supplemental Material
- The Austin Group. 2018. POSIX.1 2017: The Open Group Base Specifications Issue 7 (IEEE Std 1003.1-2008).Google Scholar
- Sandrine Blazy and Xavier Leroy. 2009. Mechanized Semantics for the Clight Subset of the C Language. Journal of Automated Reasoning 43, 3 (01 Oct 2009), 263–288. Google Scholar
Cross Ref
- Andy Chu. 2019. Oil Shell. https://www.oilshell.org/Google Scholar
- Nick Collins, Alex McLean, Julian Rohrhuber, and Adrian Ward. 2003. Live coding in laptop performance. Organised Sound 8, 3 (2003), 321–330.Google Scholar
Digital Library
- Loris D’Antoni, Rishabh Singh, and Michael Vaughn. 2016. NoFAQ: Synthesizing Command Repairs from Examples. CoRR abs/1608.08219 (2016). http://arxiv.org/abs/1608.08219Google Scholar
- Martijn Dekker. 2019. Modernish. https://github.com/modernish/modernishGoogle Scholar
- Jack B. Dennis and Earl C. Van Horn. 1966. Programming Semantics for Multiprogrammed Computations. Commun. ACM 9, 3 (March 1966), 143–155. Google Scholar
Digital Library
- Chucky Ellison and Grigore Rosu. 2012. An Executable Formal Semantics of C with Applications. In Principles of Programming Languages (POPL). ACM, New York, NY, USA, 533–544. Google Scholar
Digital Library
- Kathleen Fisher, Nate Foster, David Walker, and Kenny Q. Zhu. 2011. Forest: A Language and Toolkit for Programming with Filestores. In International Conference on Functional Programming (ICFP). ACM, New York, NY, USA, 292–306. Google Scholar
Digital Library
- Simson Garfinkel, Daniel Weise (Ed.), and Steven Strassman (Ed.). 1994. The UNIX Hater’s Handbook. IDG Books Worldwide, Inc., San Mateo, CA, USA.Google Scholar
- Gabriel Gonzalez. 2018. Turtle: shell programming, Haskell style. https://github.com/Gabriel439/Haskell- Turtle- LibraryGoogle Scholar
- Michael Greenberg. 2018a. The POSIX shell is an interactive DSL for concurrency. DSLDI.Google Scholar
- Michael Greenberg. 2018b. Word expansion supports POSIX shell interactivity. In Programming Companion (presented at Programming eXperience (PX)). ACM. Google Scholar
Digital Library
- Arjun Guha, Claudiu Saftoiu, and Shriram Krishnamurthi. 2010. The Essence of JavaScript. In ECOOP, Theo D’Hondt (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 126–150.Google Scholar
- William Gallard Hatch and Matthew Flatt. 2018. Rash: from reckless interactions to reliable programs. In Generative Programming: Concepts and Experiences (GPCE), Eric Van Wyk and Tiark Rompf (Eds.). ACM, 28–39. Google Scholar
Digital Library
- Alec Heller and Jesse A. Tov. 2008. Caml-Shcaml: an ocaml library for unix shell programming. In ML. ACM, 79–90.Google Scholar
- Nicolas Jeannerod. 2017. Le coquillage dans le CoLiS-mateur. In JFLA 2017 - Vingt-huitième Journées Francophones des Langages Applicatifs. Gourette, France. https://hal.archives- ouvertes.fr/hal- 01432034Google Scholar
- Nicolas Jeannerod, Claude Marché, and Ralf Treinen. 2017a. A Formally Verified Interpreter for a Shell-like Programming Language. In Verified Software: Theories, Tools, and Experiments (VSTTE) (Lecture Notes in Computer Science), Vol. 10712. Heidelberg, Germany. https://hal.archives- ouvertes.fr/hal- 01534747Google Scholar
- Nicolas Jeannerod, Yann Régis-Gianas, and Ralf Treinen. 2017b. Having Fun With 31.521 Shell Scripts. (April 2017). https://hal.archives- ouvertes.fr/hal- 01513750 working paper or preprint.Google Scholar
- Ralf Jung, Jacques-Henri Jourdan, Robbert Krebbers, and Derek Dreyer. 2017. RustBelt: Securing the Foundations of the Rust Programming Language. Proc. ACM Program. Lang. 2, POPL, Article 66 (Dec. 2017), 34 pages. Google Scholar
Digital Library
- Idan Kamara. 2016. explainshell. http://explainshell.com/Google Scholar
- Jeehoon Kang, Chung-Kil Hur, William Mansky, Dmitri Garbuzov, Steve Zdancewic, and Viktor Vafeiadis. 2015. A Formal C Memory Model Supporting Integer-pointer Casts. In Programming Language Design and Implementation (PLDI). ACM, New York, NY, USA, 326–335. Google Scholar
Digital Library
- Robbert Krebbers, Xavier Leroy, and Freek Wiedijk. 2014. Formal C Semantics: CompCert and the C Standard. In Interactive Theorem Proving, Gerwin Klein and Ruben Gamboa (Eds.). Springer International Publishing, Cham, 543–548.Google Scholar
- Sergio Maffeis, John C. Mitchell, and Ankur Taly. 2008. An Operational Semantics for JavaScript. In Asian Symposium on Programming Languages and Systems (APLAS). Springer-Verlag, Berlin, Heidelberg, 307–325. Google Scholar
Digital Library
- Karl Mazurak and Steve Zdancewic. 2007. ABASH: Finding Bugs in Bash Scripts. In PLAS. 105–114. Google Scholar
Digital Library
- Kayvan Memarian, Justus Matthiesen, James Lingard, Kyndylan Nienhuis, David Chisnall, Robert N. M. Watson, and Peter Sewell. 2016. Into the Depths of C: Elaborating the De Facto Standards. In Programming Language Design and Implementation (PLDI). ACM, New York, NY, USA, 1–15. Google Scholar
Digital Library
- Scott Moore, Christos Dimoulas, Dan King, and Stephen Chong. 2014. Shill: A Secure Shell Scripting Language. In USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX.Google Scholar
- Dominic P. Mulligan, Scott Owens, Kathryn E. Gray, Tom Ridge, and Peter Sewell. 2014. Lem: Reusable Engineering of Real-world Semantics. In International Conference on Functional Programming (ICFP). ACM, New York, NY, USA, 175–188. Google Scholar
Digital Library
- Marius Nita, Dan Grossman, and Craig Chambers. 2008. A theory of platform-dependent low-level software. In Principles of Programming Languages (POPL), George C. Necula and Philip Wadler (Eds.). ACM, 209–220. Google Scholar
Digital Library
- Gian Ntzik. 2017. Reasoning About POSIX File Systems. Ph.D. Dissertation. Imperial College London.Google Scholar
- Gian Ntzik, Pedro da Rocha Pinto, Julian Sutherland, and Philippa Gardner. 2018. A Concurrent Specification of POSIX File Systems. In ECOOP. Google Scholar
Cross Ref
- Joe Gibbs Politz, Matthew J. Carroll, Benjamin S. Lerner, Justin Pombrio, and Shriram Krishnamurthi. 2012. A Tested Semantics for Getters, Setters, and Eval in JavaScript. In Dynamic Languages Symposium (DLS). ACM, New York, NY, USA, 1–16. Google Scholar
Digital Library
- Yann Régis-Gianas, Nicolas Jeannerod, and Ralf Treinen. 2018. Morbig: A Static Parser for POSIX Shell. In Software Language Engineering (SLE). Boston, United States. Google Scholar
Digital Library
- Gregor Richards, Christian Hammer, Brian Burg, and Jan Vitek. 2011. The Eval That Men Do. In ECOOP, Mira Mezini (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 52–78.Google Scholar
- Tom Ridge, David Sheets, Thomas Tuerk, Andrea Giugliano, Anil Madhavapeddy, and Peter Sewell. 2015. SibylFS: Formal Specification and Oracle-based Testing for POSIX and Real-world File Systems. In Symposium on Operating Systems Principles (SOSP). ACM, New York, NY, USA, 38–53. Google Scholar
Digital Library
- Grigore Roşu and Traian Florin Şerbănuţă. 2010. An Overview of the K Semantic Framework. Journal of Logic and Algebraic Programming 79, 6 (2010), 397–434. Google Scholar
Cross Ref
- Henry Schreiner, Tomer Filiba, et al. 2018. Plumbum: shell combinators. http://plumbum.readthedocs.io/en/latest/Google Scholar
- Anthony Scopatz et al. 2019. Xonsh. https://xon.sh/Google Scholar
- Olin Shivers. 2006. SCSH manual 0.6.7. https://scsh.net/docu/html/man.htmlGoogle Scholar
- Philipp Emanuel Weidmann. 2016. maybe. https://github.com/p- e- w/maybeGoogle Scholar
- Aaron Weiss, Daniel Patterson, and Amal Ahmed. 2018. Rust Distilled: An Expressive Tower of Languages. CoRR abs/1806.02693 (2018). arXiv: 1806.02693 http://arxiv.org/abs/1806.02693Google Scholar
- Aaron Weiss, Daniel Patterson, Nicholas D. Matsakis, and Amal Ahmed. 2019. Oxide: The Essence of Rust. CoRR abs/1903.00982 (2019). arXiv: 1903.00982 http://arxiv.org/abs/1903.00982Google Scholar
Index Terms
Executable formal semantics for the POSIX shell
Recommendations
Deriving Pretty-Big-Step Semantics from Small-Step Semantics
Proceedings of the 23rd European Symposium on Programming Languages and Systems - Volume 8410Big-step semantics for languages with abrupt termination and/or divergence suffer from a serious duplication problem, addressed by the novel 'pretty-big-step' style presented by Charguéraud at ESOP'13. Such rules are less concise than corresponding ...
A formal executable semantics of Verilog
MEMOCODE '10: Proceedings of the Eighth ACM/IEEE International Conference on Formal Methods and Models for CodesignThis paper describes a formal executable semantics for the Verilog hardware description language. The goal of our formalization is to provide a concise and mathematically rigorous reference augmenting the prose of the official language standard, and ...
Porting Multimedia Applications to the Open System Environment
To migrate DOS-based courseware to the Open System Environment (OSE), process creation and communication in Posix, a portable operating-system interface, were merged with a multilevel client-server architecture. This helped identify some problems that ...






Comments