skip to main content
research-article
Open Access
Artifacts Available
Artifacts Evaluated & Reusable

Executable formal semantics for the POSIX shell

Published:20 December 2019Publication History
Skip Abstract Section

Abstract

The POSIX shell is a widely deployed, powerful tool for managing computer systems. The shell is the expert’s control panel, a necessary tool for configuring, compiling, installing, maintaining, and deploying systems. Even though it is powerful, critical infrastructure, the POSIX shell is maligned and misunderstood. Its power and its subtlety are a dangerous combination.

We define a formal, mechanized, executable small-step semantics for the POSIX shell, which we call Smoosh. We compared Smoosh against seven other shells that aim for some measure of POSIX compliance (bash, dash, zsh, OSH, mksh, ksh93, and yash). Using three test suites—the POSIX test suite, the Modernish test suite and shell diagnostic, and a test suite of our own device—we found Smoosh’s semantics to be the most conformant to the POSIX standard. Modernish judges Smoosh to have the fewest bugs (just one, from using dash’s parser) and no quirks. To show that our semantics is useful beyond yielding a conformant, executable shell, we also implemented a symbolic stepper to illuminate the subtle behavior of the shell.

Smoosh will serve as a foundation for formal study of the POSIX shell, supporting research on and development of new shells, new tooling for shells, and new shell designs.

Skip Supplemental Material Section

Supplemental Material

a43-greenberg.webm

References

  1. The Austin Group. 2018. POSIX.1 2017: The Open Group Base Specifications Issue 7 (IEEE Std 1003.1-2008).Google ScholarGoogle Scholar
  2. Sandrine Blazy and Xavier Leroy. 2009. Mechanized Semantics for the Clight Subset of the C Language. Journal of Automated Reasoning 43, 3 (01 Oct 2009), 263–288. Google ScholarGoogle ScholarCross RefCross Ref
  3. Andy Chu. 2019. Oil Shell. https://www.oilshell.org/Google ScholarGoogle Scholar
  4. Nick Collins, Alex McLean, Julian Rohrhuber, and Adrian Ward. 2003. Live coding in laptop performance. Organised Sound 8, 3 (2003), 321–330.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Loris D’Antoni, Rishabh Singh, and Michael Vaughn. 2016. NoFAQ: Synthesizing Command Repairs from Examples. CoRR abs/1608.08219 (2016). http://arxiv.org/abs/1608.08219Google ScholarGoogle Scholar
  6. Martijn Dekker. 2019. Modernish. https://github.com/modernish/modernishGoogle ScholarGoogle Scholar
  7. Jack B. Dennis and Earl C. Van Horn. 1966. Programming Semantics for Multiprogrammed Computations. Commun. ACM 9, 3 (March 1966), 143–155. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Chucky Ellison and Grigore Rosu. 2012. An Executable Formal Semantics of C with Applications. In Principles of Programming Languages (POPL). ACM, New York, NY, USA, 533–544. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Kathleen Fisher, Nate Foster, David Walker, and Kenny Q. Zhu. 2011. Forest: A Language and Toolkit for Programming with Filestores. In International Conference on Functional Programming (ICFP). ACM, New York, NY, USA, 292–306. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Simson Garfinkel, Daniel Weise (Ed.), and Steven Strassman (Ed.). 1994. The UNIX Hater’s Handbook. IDG Books Worldwide, Inc., San Mateo, CA, USA.Google ScholarGoogle Scholar
  11. Gabriel Gonzalez. 2018. Turtle: shell programming, Haskell style. https://github.com/Gabriel439/Haskell- Turtle- LibraryGoogle ScholarGoogle Scholar
  12. Michael Greenberg. 2018a. The POSIX shell is an interactive DSL for concurrency. DSLDI.Google ScholarGoogle Scholar
  13. Michael Greenberg. 2018b. Word expansion supports POSIX shell interactivity. In Programming Companion (presented at Programming eXperience (PX)). ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Arjun Guha, Claudiu Saftoiu, and Shriram Krishnamurthi. 2010. The Essence of JavaScript. In ECOOP, Theo D’Hondt (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 126–150.Google ScholarGoogle Scholar
  15. William Gallard Hatch and Matthew Flatt. 2018. Rash: from reckless interactions to reliable programs. In Generative Programming: Concepts and Experiences (GPCE), Eric Van Wyk and Tiark Rompf (Eds.). ACM, 28–39. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Alec Heller and Jesse A. Tov. 2008. Caml-Shcaml: an ocaml library for unix shell programming. In ML. ACM, 79–90.Google ScholarGoogle Scholar
  17. Nicolas Jeannerod. 2017. Le coquillage dans le CoLiS-mateur. In JFLA 2017 - Vingt-huitième Journées Francophones des Langages Applicatifs. Gourette, France. https://hal.archives- ouvertes.fr/hal- 01432034Google ScholarGoogle Scholar
  18. Nicolas Jeannerod, Claude Marché, and Ralf Treinen. 2017a. A Formally Verified Interpreter for a Shell-like Programming Language. In Verified Software: Theories, Tools, and Experiments (VSTTE) (Lecture Notes in Computer Science), Vol. 10712. Heidelberg, Germany. https://hal.archives- ouvertes.fr/hal- 01534747Google ScholarGoogle Scholar
  19. Nicolas Jeannerod, Yann Régis-Gianas, and Ralf Treinen. 2017b. Having Fun With 31.521 Shell Scripts. (April 2017). https://hal.archives- ouvertes.fr/hal- 01513750 working paper or preprint.Google ScholarGoogle Scholar
  20. Ralf Jung, Jacques-Henri Jourdan, Robbert Krebbers, and Derek Dreyer. 2017. RustBelt: Securing the Foundations of the Rust Programming Language. Proc. ACM Program. Lang. 2, POPL, Article 66 (Dec. 2017), 34 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Idan Kamara. 2016. explainshell. http://explainshell.com/Google ScholarGoogle Scholar
  22. Jeehoon Kang, Chung-Kil Hur, William Mansky, Dmitri Garbuzov, Steve Zdancewic, and Viktor Vafeiadis. 2015. A Formal C Memory Model Supporting Integer-pointer Casts. In Programming Language Design and Implementation (PLDI). ACM, New York, NY, USA, 326–335. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Robbert Krebbers, Xavier Leroy, and Freek Wiedijk. 2014. Formal C Semantics: CompCert and the C Standard. In Interactive Theorem Proving, Gerwin Klein and Ruben Gamboa (Eds.). Springer International Publishing, Cham, 543–548.Google ScholarGoogle Scholar
  24. Sergio Maffeis, John C. Mitchell, and Ankur Taly. 2008. An Operational Semantics for JavaScript. In Asian Symposium on Programming Languages and Systems (APLAS). Springer-Verlag, Berlin, Heidelberg, 307–325. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Karl Mazurak and Steve Zdancewic. 2007. ABASH: Finding Bugs in Bash Scripts. In PLAS. 105–114. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Kayvan Memarian, Justus Matthiesen, James Lingard, Kyndylan Nienhuis, David Chisnall, Robert N. M. Watson, and Peter Sewell. 2016. Into the Depths of C: Elaborating the De Facto Standards. In Programming Language Design and Implementation (PLDI). ACM, New York, NY, USA, 1–15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Scott Moore, Christos Dimoulas, Dan King, and Stephen Chong. 2014. Shill: A Secure Shell Scripting Language. In USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX.Google ScholarGoogle Scholar
  28. Dominic P. Mulligan, Scott Owens, Kathryn E. Gray, Tom Ridge, and Peter Sewell. 2014. Lem: Reusable Engineering of Real-world Semantics. In International Conference on Functional Programming (ICFP). ACM, New York, NY, USA, 175–188. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Marius Nita, Dan Grossman, and Craig Chambers. 2008. A theory of platform-dependent low-level software. In Principles of Programming Languages (POPL), George C. Necula and Philip Wadler (Eds.). ACM, 209–220. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Gian Ntzik. 2017. Reasoning About POSIX File Systems. Ph.D. Dissertation. Imperial College London.Google ScholarGoogle Scholar
  31. Gian Ntzik, Pedro da Rocha Pinto, Julian Sutherland, and Philippa Gardner. 2018. A Concurrent Specification of POSIX File Systems. In ECOOP. Google ScholarGoogle ScholarCross RefCross Ref
  32. Joe Gibbs Politz, Matthew J. Carroll, Benjamin S. Lerner, Justin Pombrio, and Shriram Krishnamurthi. 2012. A Tested Semantics for Getters, Setters, and Eval in JavaScript. In Dynamic Languages Symposium (DLS). ACM, New York, NY, USA, 1–16. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Yann Régis-Gianas, Nicolas Jeannerod, and Ralf Treinen. 2018. Morbig: A Static Parser for POSIX Shell. In Software Language Engineering (SLE). Boston, United States. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Gregor Richards, Christian Hammer, Brian Burg, and Jan Vitek. 2011. The Eval That Men Do. In ECOOP, Mira Mezini (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 52–78.Google ScholarGoogle Scholar
  35. Tom Ridge, David Sheets, Thomas Tuerk, Andrea Giugliano, Anil Madhavapeddy, and Peter Sewell. 2015. SibylFS: Formal Specification and Oracle-based Testing for POSIX and Real-world File Systems. In Symposium on Operating Systems Principles (SOSP). ACM, New York, NY, USA, 38–53. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Grigore Roşu and Traian Florin Şerbănuţă. 2010. An Overview of the K Semantic Framework. Journal of Logic and Algebraic Programming 79, 6 (2010), 397–434. Google ScholarGoogle ScholarCross RefCross Ref
  37. Henry Schreiner, Tomer Filiba, et al. 2018. Plumbum: shell combinators. http://plumbum.readthedocs.io/en/latest/Google ScholarGoogle Scholar
  38. Anthony Scopatz et al. 2019. Xonsh. https://xon.sh/Google ScholarGoogle Scholar
  39. Olin Shivers. 2006. SCSH manual 0.6.7. https://scsh.net/docu/html/man.htmlGoogle ScholarGoogle Scholar
  40. Philipp Emanuel Weidmann. 2016. maybe. https://github.com/p- e- w/maybeGoogle ScholarGoogle Scholar
  41. Aaron Weiss, Daniel Patterson, and Amal Ahmed. 2018. Rust Distilled: An Expressive Tower of Languages. CoRR abs/1806.02693 (2018). arXiv: 1806.02693 http://arxiv.org/abs/1806.02693Google ScholarGoogle Scholar
  42. Aaron Weiss, Daniel Patterson, Nicholas D. Matsakis, and Amal Ahmed. 2019. Oxide: The Essence of Rust. CoRR abs/1903.00982 (2019). arXiv: 1903.00982 http://arxiv.org/abs/1903.00982Google ScholarGoogle Scholar

Index Terms

  1. Executable formal semantics for the POSIX shell

              Recommendations

              Comments

              Login options

              Check if you have access through your login credentials or your institution to get full access on this article.

              Sign in

              Full Access

              PDF Format

              View or Download as a PDF file.

              PDF

              eReader

              View online with eReader.

              eReader
              About Cookies On This Site

              We use cookies to ensure that we give you the best experience on our website.

              Learn more

              Got it!