skip to main content
research-article
Open Access

Build scripts with perfect dependencies

Published:13 November 2020Publication History
Skip Abstract Section

Abstract

Build scripts for most build systems describe the actions to run, and the dependencies between those actions - but often build scripts get those dependencies wrong. Most build scripts have both too few dependencies (leading to incorrect build outputs) and too many dependencies (leading to excessive rebuilds and reduced parallelism). Any programmer who has wondered why a small change led to excess compilation, or who resorted to a clean step, has suffered the ill effects of incorrect dependency specification. We outline a build system where dependencies are not specified, but instead captured by tracing execution. The consequence is that dependencies are always correct by construction and build scripts are easier to write. The simplest implementation of our approach would lose parallelism, but we are able to recover parallelism using speculation.

Skip Supplemental Material Section

Supplemental Material

Auxiliary Presentation Video

Build scripts for most build systems describe the actions to run, and the dependencies between those actions - but often build scripts get those dependencies wrong. Most build scripts have both too few dependencies (leading to incorrect build outputs) and too many dependencies (leading to excessive rebuilds and reduced parallelism). Any programmer who has wondered why a small change led to excess compilation, or who resorted to a clean step, has suffered the ill effects of incorrect dependency specification. We outline a build system where dependencies are not specified, but instead captured by tracing execution. The consequence is that dependencies are always correct by construction and build scripts are easier to write. The simplest implementation of our approach would lose parallelism, but we are able to recover parallelism using speculation.

References

  1. Jorge Acereda. 2019. FSATrace: filesystem access tracer. ( 2019 ). https://github.com/jacereda/fsatrace.Google ScholarGoogle Scholar
  2. Lennart Augustsson, Mikael Rittri, and Dan Synek. 1994. On generating unique names. JFP 4 ( 01 1994 ), 117-123.Google ScholarGoogle Scholar
  3. Cor-Paul Bezemer, Shane McIntosh, Bram Adams, Daniel German, and Ahmed Hassan. 2017. An empirical study of unspecified dependencies in Make-based build systems. Empirical Software Engineering 22, 6 ( 2017 ), 3117-3148.Google ScholarGoogle Scholar
  4. Eelco Dolstra, Merijn De Jonge, Eelco Visser, and others. 2004. Nix: A Safe and Policy-Free System for Software Deployment. In LISA, Vol. 4. 79-92.Google ScholarGoogle Scholar
  5. Alan Donovan and Brian Kernighan. 2015. The Go Programming Language (1st ed.). Addison-Wesley Professional.Google ScholarGoogle Scholar
  6. Sebastian Erdweg, Moritz Lichter, and Manuel Weiel. 2015. A Sound and Optimal Incremental Build System with Dynamic Dependencies. In Proceedings of OOPSLA 2015. ACM, 89-106. DOI:http://dx.doi.org/10.1145/2814270.2814316 Google ScholarGoogle ScholarCross RefCross Ref
  7. Stuart Feldman. 1979. Make-A program for maintaining computer programs. Software: Practice and experience 9, 4 ( 1979 ), 255-265.Google ScholarGoogle Scholar
  8. Google. 2009. Generate Your Projects. ( 2009 ). https://gyp.gsrc.io/.Google ScholarGoogle Scholar
  9. Google. 2016. Bazel. ( 2016 ). http://bazel.io/.Google ScholarGoogle Scholar
  10. Berwyn Hoyt, Bryan Hoyt, and Ben Hoyt. 2009. Fabricate: The better build tool. ( 2009 ). https://github.com/SimonAlfie/ fabricate.Google ScholarGoogle Scholar
  11. Mark Jones. 1995. Functional programming with overloading and higher-order polymorphism. In Advanced Functional Programming, First International Spring School on Advanced Functional Programming Techniques. 97-136.Google ScholarGoogle Scholar
  12. Oleg Kiselyov. 2015. Polyvariadic functions and keyword arguments: pattern-matching on the type of the context. ( 2015 ). http://okmij.org/ftp/Haskell/polyvariadic.html.Google ScholarGoogle Scholar
  13. Gabriël Konat, Sebastian Erdweg, and Eelco Visser. 2018a. Scalable Incremental Building with Dynamic Task Dependencies. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018 ). Association for Computing Machinery, New York, NY, USA, 76âĂŞ86. DOI:http://dx.doi.org/10.1145/3238147.3238196 Google ScholarGoogle ScholarCross RefCross Ref
  14. Gabriël Konat, Michael J. Steindorfer, Sebastian Erdweg, and Eelco Visser. 2018b. PIE: A Domain-Specific Language for Interactive Software Development Pipelines. The Art, Science, and Engineering of Programming 2, 3 (Mar 2018 ). DOI: http://dx.doi.org/10.22152/programming-journal.org/ 2018 /2/9 Google ScholarGoogle ScholarCross RefCross Ref
  15. Nándor Licker and Andrew Rice. 2019. Detecting incorrect build rules. In Proceedings of ICSE 2019. ACM, 1234-1244. DOI: http://dx.doi.org/10.1109/ICSE. 2019.00125 Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Bill McCloskey. 2008. Memoize. ( 2008 ). https://github.com/kgaughan/memoize.py.Google ScholarGoogle Scholar
  17. Shane McIntosh, Bram Adams, Thanh Nguyen, Yasutaka Kamei, and Ahmed Hassan. 2011. An empirical study of build maintenance efort. In Proceedings of ICSE 2011. ACM, 141-150.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Microsoft. 2020. Build Accelerator. ( 2020 ). https://github.com/microsoft/BuildXL.Google ScholarGoogle Scholar
  19. Peter Miller. 1998. Recursive Make considered harmful. Journal of AUUG Inc 19, 1 ( 1998 ), 14-25.Google ScholarGoogle Scholar
  20. Neil Mitchell. 2012. Shake Before Building-Replacing Make with Haskell. In Proceedings of ICFP 2012. ACM, 55-66.Google ScholarGoogle Scholar
  21. Neil Mitchell. 2020. File access tracing. ( 2020 ). https://neilmitchell.blogspot.com/ 2020 /05/file-tracing.html.Google ScholarGoogle Scholar
  22. Andrey Mokhov. 2019. Stroll: an experimental build system. ( 2019 ). https://blogs.ncl.ac.uk/andreymokhov/stroll/.Google ScholarGoogle Scholar
  23. Andrey Mokhov, Neil Mitchell, and Simon Peyton Jones. 2018. Build systems à la carte. Proceedings ACM Programing Languages 2 ( 2018 ), 1-29. DOI:http://dx.doi.org/10.1145/3236774 Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Andrey Mokhov, Neil Mitchell, Simon Peyton Jones, and Simon Marlow. 2016. Non-recursive Make considered harmful: build systems at scale. In Proceedings of Haskell 2016. ACM, 170-181.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Edmund Nightingale, Peter Chen, and Jason Flinn. 2005. Speculative execution in a distributed file system. ACM SIGOPS operating systems review 39, 5 ( 2005 ), 191-205.Google ScholarGoogle Scholar
  26. David Patterson and John Hennessy. 2013. Computer Organization and Design, Fifth Edition : The Hardware/Software Interface (5th ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.Google ScholarGoogle Scholar
  27. David Roundy. 2019a. Bigbro. ( 2019 ). https://github.com/droundy/bigbro.Google ScholarGoogle Scholar
  28. David Roundy. 2019b. Fac build system. ( 2019 ). https://sites.science.oregonstate.edu/~roundyd/fac/.Google ScholarGoogle Scholar
  29. Tim Sheard and Simon Peyton Jones. 2002. Template meta-programming for Haskell. In Proceedings of the 2002 Haskell Workshop, Pittsburgh. 1-16.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. J Gregory Stefan, Christopher Colohan, Antonia Zhai, and Todd Mowry. 2000. A scalable approach to thread-level speculation. ACM SIGARCH Computer Architecture News 28, 2 ( 2000 ), 1-12.Google ScholarGoogle Scholar
  31. J Gregory Stefan and Todd Mowry. 1998. The potential for using thread-level data speculation to facilitate automatic parallelization. In Proceedings 1998 Fourth International Symposium on High-Performance Computer Architecture. IEEE, 2-13.Google ScholarGoogle ScholarCross RefCross Ref
  32. Steven Swanson, Luke McDowell, Michael Swift, Susan Eggers, and Henry Levy. 2003. An Evaluation of Speculative Instruction Execution on Simultaneous Multithreaded Processors. ACM Transactions on Computer Systems 21, 3 ( 2003 ), 314-340. DOI:http://dx.doi.org/10.1145/859716.859720 Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Cristian Tapus. 2006. Distributed speculations: providing fault-tolerance and improving performance. Ph.D. Dissertation. California Institute of Technology.Google ScholarGoogle Scholar
  34. Robert Tomasulo. 1967. An Eficient Algorithm for Exploiting Multiple Arithmetic Units. IBM Journal of Research and Development 11, 1 ( 1967 ), 25-33.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Adam Welc, Suresh Jagannathan, and Antony Hosking. 2005. Safe futures for Java. In Proceedings of OOPSLA 2005. 439-453.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Build scripts with perfect dependencies

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image Proceedings of the ACM on Programming Languages
      Proceedings of the ACM on Programming Languages  Volume 4, Issue OOPSLA
      November 2020
      3108 pages
      EISSN:2475-1421
      DOI:10.1145/3436718
      Issue’s Table of Contents

      Copyright © 2020 Owner/Author

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 13 November 2020
      Published in pacmpl Volume 4, Issue OOPSLA

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!