Abstract
Build scripts for most build systems describe the actions to run, and the dependencies between those actions - but often build scripts get those dependencies wrong. Most build scripts have both too few dependencies (leading to incorrect build outputs) and too many dependencies (leading to excessive rebuilds and reduced parallelism). Any programmer who has wondered why a small change led to excess compilation, or who resorted to a clean step, has suffered the ill effects of incorrect dependency specification. We outline a build system where dependencies are not specified, but instead captured by tracing execution. The consequence is that dependencies are always correct by construction and build scripts are easier to write. The simplest implementation of our approach would lose parallelism, but we are able to recover parallelism using speculation.
Supplemental Material
- Jorge Acereda. 2019. FSATrace: filesystem access tracer. ( 2019 ). https://github.com/jacereda/fsatrace.Google Scholar
- Lennart Augustsson, Mikael Rittri, and Dan Synek. 1994. On generating unique names. JFP 4 ( 01 1994 ), 117-123.Google Scholar
- Cor-Paul Bezemer, Shane McIntosh, Bram Adams, Daniel German, and Ahmed Hassan. 2017. An empirical study of unspecified dependencies in Make-based build systems. Empirical Software Engineering 22, 6 ( 2017 ), 3117-3148.Google Scholar
- Eelco Dolstra, Merijn De Jonge, Eelco Visser, and others. 2004. Nix: A Safe and Policy-Free System for Software Deployment. In LISA, Vol. 4. 79-92.Google Scholar
- Alan Donovan and Brian Kernighan. 2015. The Go Programming Language (1st ed.). Addison-Wesley Professional.Google Scholar
- Sebastian Erdweg, Moritz Lichter, and Manuel Weiel. 2015. A Sound and Optimal Incremental Build System with Dynamic Dependencies. In Proceedings of OOPSLA 2015. ACM, 89-106. DOI:http://dx.doi.org/10.1145/2814270.2814316 Google Scholar
Cross Ref
- Stuart Feldman. 1979. Make-A program for maintaining computer programs. Software: Practice and experience 9, 4 ( 1979 ), 255-265.Google Scholar
- Google. 2009. Generate Your Projects. ( 2009 ). https://gyp.gsrc.io/.Google Scholar
- Google. 2016. Bazel. ( 2016 ). http://bazel.io/.Google Scholar
- Berwyn Hoyt, Bryan Hoyt, and Ben Hoyt. 2009. Fabricate: The better build tool. ( 2009 ). https://github.com/SimonAlfie/ fabricate.Google Scholar
- Mark Jones. 1995. Functional programming with overloading and higher-order polymorphism. In Advanced Functional Programming, First International Spring School on Advanced Functional Programming Techniques. 97-136.Google Scholar
- Oleg Kiselyov. 2015. Polyvariadic functions and keyword arguments: pattern-matching on the type of the context. ( 2015 ). http://okmij.org/ftp/Haskell/polyvariadic.html.Google Scholar
- Gabriël Konat, Sebastian Erdweg, and Eelco Visser. 2018a. Scalable Incremental Building with Dynamic Task Dependencies. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018 ). Association for Computing Machinery, New York, NY, USA, 76âĂŞ86. DOI:http://dx.doi.org/10.1145/3238147.3238196 Google Scholar
Cross Ref
- Gabriël Konat, Michael J. Steindorfer, Sebastian Erdweg, and Eelco Visser. 2018b. PIE: A Domain-Specific Language for Interactive Software Development Pipelines. The Art, Science, and Engineering of Programming 2, 3 (Mar 2018 ). DOI: http://dx.doi.org/10.22152/programming-journal.org/ 2018 /2/9 Google Scholar
Cross Ref
- Nándor Licker and Andrew Rice. 2019. Detecting incorrect build rules. In Proceedings of ICSE 2019. ACM, 1234-1244. DOI: http://dx.doi.org/10.1109/ICSE. 2019.00125 Google Scholar
Digital Library
- Bill McCloskey. 2008. Memoize. ( 2008 ). https://github.com/kgaughan/memoize.py.Google Scholar
- Shane McIntosh, Bram Adams, Thanh Nguyen, Yasutaka Kamei, and Ahmed Hassan. 2011. An empirical study of build maintenance efort. In Proceedings of ICSE 2011. ACM, 141-150.Google Scholar
Digital Library
- Microsoft. 2020. Build Accelerator. ( 2020 ). https://github.com/microsoft/BuildXL.Google Scholar
- Peter Miller. 1998. Recursive Make considered harmful. Journal of AUUG Inc 19, 1 ( 1998 ), 14-25.Google Scholar
- Neil Mitchell. 2012. Shake Before Building-Replacing Make with Haskell. In Proceedings of ICFP 2012. ACM, 55-66.Google Scholar
- Neil Mitchell. 2020. File access tracing. ( 2020 ). https://neilmitchell.blogspot.com/ 2020 /05/file-tracing.html.Google Scholar
- Andrey Mokhov. 2019. Stroll: an experimental build system. ( 2019 ). https://blogs.ncl.ac.uk/andreymokhov/stroll/.Google Scholar
- Andrey Mokhov, Neil Mitchell, and Simon Peyton Jones. 2018. Build systems à la carte. Proceedings ACM Programing Languages 2 ( 2018 ), 1-29. DOI:http://dx.doi.org/10.1145/3236774 Google Scholar
Digital Library
- Andrey Mokhov, Neil Mitchell, Simon Peyton Jones, and Simon Marlow. 2016. Non-recursive Make considered harmful: build systems at scale. In Proceedings of Haskell 2016. ACM, 170-181.Google Scholar
Digital Library
- Edmund Nightingale, Peter Chen, and Jason Flinn. 2005. Speculative execution in a distributed file system. ACM SIGOPS operating systems review 39, 5 ( 2005 ), 191-205.Google Scholar
- David Patterson and John Hennessy. 2013. Computer Organization and Design, Fifth Edition : The Hardware/Software Interface (5th ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.Google Scholar
- David Roundy. 2019a. Bigbro. ( 2019 ). https://github.com/droundy/bigbro.Google Scholar
- David Roundy. 2019b. Fac build system. ( 2019 ). https://sites.science.oregonstate.edu/~roundyd/fac/.Google Scholar
- Tim Sheard and Simon Peyton Jones. 2002. Template meta-programming for Haskell. In Proceedings of the 2002 Haskell Workshop, Pittsburgh. 1-16.Google Scholar
Digital Library
- J Gregory Stefan, Christopher Colohan, Antonia Zhai, and Todd Mowry. 2000. A scalable approach to thread-level speculation. ACM SIGARCH Computer Architecture News 28, 2 ( 2000 ), 1-12.Google Scholar
- J Gregory Stefan and Todd Mowry. 1998. The potential for using thread-level data speculation to facilitate automatic parallelization. In Proceedings 1998 Fourth International Symposium on High-Performance Computer Architecture. IEEE, 2-13.Google Scholar
Cross Ref
- Steven Swanson, Luke McDowell, Michael Swift, Susan Eggers, and Henry Levy. 2003. An Evaluation of Speculative Instruction Execution on Simultaneous Multithreaded Processors. ACM Transactions on Computer Systems 21, 3 ( 2003 ), 314-340. DOI:http://dx.doi.org/10.1145/859716.859720 Google Scholar
Digital Library
- Cristian Tapus. 2006. Distributed speculations: providing fault-tolerance and improving performance. Ph.D. Dissertation. California Institute of Technology.Google Scholar
- Robert Tomasulo. 1967. An Eficient Algorithm for Exploiting Multiple Arithmetic Units. IBM Journal of Research and Development 11, 1 ( 1967 ), 25-33.Google Scholar
Digital Library
- Adam Welc, Suresh Jagannathan, and Antony Hosking. 2005. Safe futures for Java. In Proceedings of OOPSLA 2005. 439-453.Google Scholar
Digital Library
Index Terms
Build scripts with perfect dependencies
Recommendations
Forward build systems, formally
CPP 2022: Proceedings of the 11th ACM SIGPLAN International Conference on Certified Programs and ProofsBuild systems are a fundamental part of software construction, but their correctness has received comparatively little attention, relative to more prominent parts of the toolchain. In this paper, we address the correctness of forward build systems, ...
Build systems à la carte
Build systems are awesome, terrifying -- and unloved. They are used by every developer around the world, but are rarely the object of study. In this paper we offer a systematic, and executable, framework for developing and comparing build systems, ...
An empirical study of unspecified dependencies in make-based build systems
Software developers rely on a build system to compile their source code changes and produce deliverables for testing and deployment. Since the full build of large software systems can take hours, the incremental build is a cornerstone of modern build ...






Comments