Abstract
Testing a software product line such as Linux implies building the source with different configurations. Manual approaches to generate configurations that enable code of interest are doomed to fail due to the high amount of variation points distributed over the feature model, the build system and the source code. Research has proposed various approaches to generate covering configurations, but the algorithms show many drawbacks related to run-time, exhaustiveness and the amount of generated configurations. Hence, analyzing an entire Linux source can yield more than 30 thousand configurations and thereby exceeds the limited budget and resources for build testing.
In this paper, we present an approach to fill the gap between a systematic generation of configurations and the necessity to fully build software in order to test it. By merging previously generated configurations, we reduce the number of necessary builds and enable global variability-aware testing. We reduce the problem of merging configurations to finding maximum cliques in a graph. We evaluate the approach on the Linux kernel, compare the results to common practices in industry, and show that our implementation scales even when facing graphs with millions of edges.
- Julio Sincero et al. “Is The Linux Kernel a Software Product Line?” In: Proceedings of the International Workshop on Open Source Software and Product Lines (SPLC-OSSPL 2007). 2007.Google Scholar
- Michael Kerrisk. Kernel build/boot testing. 2012.Google Scholar
- Henrik Stuart et al. “Towards Easing the Diagnosis of Bugs in OS Code”. In: Proceedings of the 4th Workshop on Programming Languages and Operating Systems. 2007, 2:1–2:5. Google Scholar
Digital Library
- Julia L Lawall et al. “WYSIWIB: A declarative approach to finding API protocols and bugs in Linux code”. In: IEEE/IFIP International Conference on Dependable Systems & Networks. 2009, pp. 43–52.Google Scholar
Cross Ref
- Dave Jones. Trinity: A Linux system call fuzzer.Google Scholar
- Rafael Lotufo et al. “Evolution of the Linux Kernel Variability Model”. In: Proceedings of the 14th International Conference on Software Product Lines: Going Beyond. 2010, pp. 136–150. Google Scholar
Digital Library
- Steven She et al. “Variability Model of the Linux Kernel”. In: Fourth International Workshop on Variability Modeling of Software-intensive Systems (VaMoS 2010).Google Scholar
- Julio Sincero et al. “Facing the Linux 8000 Feature Nightmare”. In: Proceedings of ACM European Conference on Computer Systems (EuroSys 2010), Best Posters and Demos Session.Google Scholar
- Christian Dietrich et al. “A Robust Approach for Variability Extraction from the Linux Build System”. In: Proceedings of the 16th International Software Product Line Conference. Vol. 1. 2012, pp. 21–30. Google Scholar
Digital Library
- Sarah Nadi and Ric Holt. “Mining Kbuild to detect variability anomalies in Linux”. In: 16th European Conference on Software Maintenance and Reengineering (CSMR). 2012, pp. 107–116. Google Scholar
Digital Library
- Reinhard Tartler et al. “Static analysis of variability in system software: The 90,000# ifdefs issue”. In: Proc. USENIX Conf (2014), pp. 421–432. Google Scholar
Digital Library
- Andreas Ziegler, Bernhard Heinloth, and Daniel Lohmann. “Automatic Feature Selection in Large-Scale System-Software Product Lines”. In: Proceedings of the 2014 International Conference on Generative Programming: Concepts and Experiences. 2014, pp. 39– 48. Google Scholar
Digital Library
- Sarah Nadi et al. “Mining Configuration Constraints: Static Analyses and Empirical Results”. In: Proceedings of the 36th International Conference on Software Engineering. 2014, pp. 140–151. Google Scholar
Digital Library
- Iago Abal, Claus Brabrand, and Andrzej Wasowski. “42 variability bugs in the linux kernel: a qualitative analysis”. In: Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. 2014, pp. 421–432. Google Scholar
Digital Library
- Flávio Medeiros et al. “A Comparison of 10 Sampling Algorithms for Configurable Systems”. In: Proceedings of the 38th International Conference on Software Engineering - Volume 2. ICSE ’16. 2016. Google Scholar
Digital Library
- Sascha El-Sharkawy, Adam Krafczyk, and Klaus Schmid. “Analysing the Kconfig Semantics and Its Analysis Tools”. In: Proceedings of the 2015 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences. 2015, pp. 45– 54. Google Scholar
Digital Library
- D. L. Parnas. “On the Criteria to Be Used in Decomposing Systems into Modules”. In: (1972), pp. 1053– 1058. Google Scholar
Digital Library
- Jörg Liebig et al. “An analysis of the variability in forty preprocessor-based software product lines”. In: ACM/IEEE 32nd International Conference on Software Engineering. 2010, pp. 105–114. Google Scholar
Digital Library
- Michael D. Ernst, Greg J. Badros, and David Notkin. “An Empirical Analysis of C Preprocessor Use”. In: IEEE Trans. Softw. Eng. (2002), pp. 1146–1170. Google Scholar
Digital Library
- Alejandra Garrido and Ralph Johnson. “Analyzing multiple configurations of a C program”. In: Proceedings of the 21st IEEE International Conference on Software Maintenance. 2005, pp. 379–388. Google Scholar
Digital Library
- Jonathan Corbet and Greg Kroah-Hartman. Linux Kernel Development Report 2016. 2016.Google Scholar
- Jonathan Corbet. A day in the life of linux-next.Google Scholar
- Norbert Siegmund et al. “Predicting Performance via Automated Feature-interaction Detection”. In: Proceedings of the 34th International Conference on Software Engineering. 2012, pp. 167–177. Google Scholar
Digital Library
- Brady J Garvin and Myra B Cohen. “Feature interaction faults revisited: An exploratory study”. In: 2011 IEEE 22nd International Symposium on Software Reliability Engineering (ISSRE). 2011, pp. 90–99. Google Scholar
Digital Library
- Yoann Padioleau et al. “Documenting and Automating Collateral Evolutions in Linux Device Drivers”. In: Proceedings of the 3rd ACM SIGOPS/EuroSys European Conference on Computer Systems 2008. 2008, pp. 247– 260. Google Scholar
Digital Library
- Reinhard Tartler et al. “Configuration Coverage in the Analysis of Large-Scale System Software”. In: Proceedings of the 6th Workshop on Programming Languages and Operating Systems. 2011, p. 2. Google Scholar
Digital Library
- Jean Melo et al. “A Quantitative Analysis of Variability Warnings in Linux”. In: Proceedings of the Tenth International Workshop on Variability Modelling of Software-intensive Systems. 2016. Google Scholar
Digital Library
- Reinhard Tartler et al. “Feature Consistency in Compile-Time Configurable System Software”. In: Proceedings of the EuroSys 2011 Conference (EuroSys ’11). 2011, pp. 47–60. Google Scholar
Digital Library
- Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. “An Algorithm for Generating T-Wise Covering Arrays from Large Feature Models”. In: Proceedings of the 16th International Software Product Line Conference-Volume 1. 2012, pp. 46–55. Google Scholar
Digital Library
- Jörg Liebig et al. “Scalable analysis of variable software”. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 2013, pp. 81–91. Google Scholar
Digital Library
- Malte Lochau et al. “Model-based pairwise testing for feature interaction coverage in software product line engineering”. In: Software Quality Journal 20 (2011), pp. 567–604. Google Scholar
Digital Library
- Gilles Perrouin et al. “Automated and Scalable TWise Test Case Generation Strategies for Software Product Lines”. In: Third International Conference on Software Testing, Verification and Validation (ICST). 2010, pp. 459–468. Google Scholar
Digital Library
- Sebastian Oster, Florian Markert, and Philipp Ritter. “Automated Incremental Pairwise Testing of Software Product Lines”. In: Software Product Lines: Going Beyond. 2010, pp. 196–210. Google Scholar
Digital Library
- Linux Coding Style Guidelines.Google Scholar
- Flávio Medeiros et al. “An Empirical Study on Configuration-Related Issues: Investigating Undeclared and Unused Identifiers”. In: Proceedings of the 2015 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences. 2015, pp. 35–44. Google Scholar
Digital Library
- Ryan A Rossi et al. “A fast parallel maximum clique algorithm for large sparse graphs and temporal strong components”. In: CoRR, abs/1302.6256 (2013).Google Scholar
- Andreas Ziegler, Valentin Rothberg, and Daniel Lohmann. “Analyzing the Impact of Feature Changes in Linux”. In: Proceedings of the Tenth International Workshop on Variability Modelling of Softwareintensive Systems. 2016, pp. 25–32. Google Scholar
Digital Library
Index Terms
Towards scalable configuration testing in variable software
Recommendations
Test-case prioritization for configuration testing
ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and AnalysisConfiguration changes are among the dominant causes of failures of large-scale software system deployment. Given the velocity of configuration changes, typically at the scale of hundreds to thousands of times daily in modern cloud systems, checking ...
Towards scalable configuration testing in variable software
GPCE 2016: Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and ExperiencesTesting a software product line such as Linux implies building the source with different configurations. Manual approaches to generate configurations that enable code of interest are doomed to fail due to the high amount of variation points distributed ...
Continuous test suite augmentation in software product lines
SPLC '13: Proceedings of the 17th International Software Product Line ConferenceSoftware Product Line (SPL) engineering offers several advantages in the development of families of software products. There is still a need, however, to generate test cases for individual products in product lines more efficiently. In this paper we ...







Comments