skip to main content
article

Towards scalable configuration testing in variable software

Published:20 October 2016Publication History
Skip Abstract Section

Abstract

Testing a software product line such as Linux implies building the source with different configurations. Manual approaches to generate configurations that enable code of interest are doomed to fail due to the high amount of variation points distributed over the feature model, the build system and the source code. Research has proposed various approaches to generate covering configurations, but the algorithms show many drawbacks related to run-time, exhaustiveness and the amount of generated configurations. Hence, analyzing an entire Linux source can yield more than 30 thousand configurations and thereby exceeds the limited budget and resources for build testing.

In this paper, we present an approach to fill the gap between a systematic generation of configurations and the necessity to fully build software in order to test it. By merging previously generated configurations, we reduce the number of necessary builds and enable global variability-aware testing. We reduce the problem of merging configurations to finding maximum cliques in a graph. We evaluate the approach on the Linux kernel, compare the results to common practices in industry, and show that our implementation scales even when facing graphs with millions of edges.

References

  1. Julio Sincero et al. “Is The Linux Kernel a Software Product Line?” In: Proceedings of the International Workshop on Open Source Software and Product Lines (SPLC-OSSPL 2007). 2007.Google ScholarGoogle Scholar
  2. Michael Kerrisk. Kernel build/boot testing. 2012.Google ScholarGoogle Scholar
  3. Henrik Stuart et al. “Towards Easing the Diagnosis of Bugs in OS Code”. In: Proceedings of the 4th Workshop on Programming Languages and Operating Systems. 2007, 2:1–2:5. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Julia L Lawall et al. “WYSIWIB: A declarative approach to finding API protocols and bugs in Linux code”. In: IEEE/IFIP International Conference on Dependable Systems & Networks. 2009, pp. 43–52.Google ScholarGoogle ScholarCross RefCross Ref
  5. Dave Jones. Trinity: A Linux system call fuzzer.Google ScholarGoogle Scholar
  6. Rafael Lotufo et al. “Evolution of the Linux Kernel Variability Model”. In: Proceedings of the 14th International Conference on Software Product Lines: Going Beyond. 2010, pp. 136–150. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Steven She et al. “Variability Model of the Linux Kernel”. In: Fourth International Workshop on Variability Modeling of Software-intensive Systems (VaMoS 2010).Google ScholarGoogle Scholar
  8. Julio Sincero et al. “Facing the Linux 8000 Feature Nightmare”. In: Proceedings of ACM European Conference on Computer Systems (EuroSys 2010), Best Posters and Demos Session.Google ScholarGoogle Scholar
  9. Christian Dietrich et al. “A Robust Approach for Variability Extraction from the Linux Build System”. In: Proceedings of the 16th International Software Product Line Conference. Vol. 1. 2012, pp. 21–30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Sarah Nadi and Ric Holt. “Mining Kbuild to detect variability anomalies in Linux”. In: 16th European Conference on Software Maintenance and Reengineering (CSMR). 2012, pp. 107–116. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Reinhard Tartler et al. “Static analysis of variability in system software: The 90,000# ifdefs issue”. In: Proc. USENIX Conf (2014), pp. 421–432. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Andreas Ziegler, Bernhard Heinloth, and Daniel Lohmann. “Automatic Feature Selection in Large-Scale System-Software Product Lines”. In: Proceedings of the 2014 International Conference on Generative Programming: Concepts and Experiences. 2014, pp. 39– 48. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Sarah Nadi et al. “Mining Configuration Constraints: Static Analyses and Empirical Results”. In: Proceedings of the 36th International Conference on Software Engineering. 2014, pp. 140–151. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Iago Abal, Claus Brabrand, and Andrzej Wasowski. “42 variability bugs in the linux kernel: a qualitative analysis”. In: Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. 2014, pp. 421–432. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Flávio Medeiros et al. “A Comparison of 10 Sampling Algorithms for Configurable Systems”. In: Proceedings of the 38th International Conference on Software Engineering - Volume 2. ICSE ’16. 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Sascha El-Sharkawy, Adam Krafczyk, and Klaus Schmid. “Analysing the Kconfig Semantics and Its Analysis Tools”. In: Proceedings of the 2015 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences. 2015, pp. 45– 54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. D. L. Parnas. “On the Criteria to Be Used in Decomposing Systems into Modules”. In: (1972), pp. 1053– 1058. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Jörg Liebig et al. “An analysis of the variability in forty preprocessor-based software product lines”. In: ACM/IEEE 32nd International Conference on Software Engineering. 2010, pp. 105–114. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Michael D. Ernst, Greg J. Badros, and David Notkin. “An Empirical Analysis of C Preprocessor Use”. In: IEEE Trans. Softw. Eng. (2002), pp. 1146–1170. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Alejandra Garrido and Ralph Johnson. “Analyzing multiple configurations of a C program”. In: Proceedings of the 21st IEEE International Conference on Software Maintenance. 2005, pp. 379–388. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Jonathan Corbet and Greg Kroah-Hartman. Linux Kernel Development Report 2016. 2016.Google ScholarGoogle Scholar
  22. Jonathan Corbet. A day in the life of linux-next.Google ScholarGoogle Scholar
  23. Norbert Siegmund et al. “Predicting Performance via Automated Feature-interaction Detection”. In: Proceedings of the 34th International Conference on Software Engineering. 2012, pp. 167–177. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Brady J Garvin and Myra B Cohen. “Feature interaction faults revisited: An exploratory study”. In: 2011 IEEE 22nd International Symposium on Software Reliability Engineering (ISSRE). 2011, pp. 90–99. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Yoann Padioleau et al. “Documenting and Automating Collateral Evolutions in Linux Device Drivers”. In: Proceedings of the 3rd ACM SIGOPS/EuroSys European Conference on Computer Systems 2008. 2008, pp. 247– 260. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Reinhard Tartler et al. “Configuration Coverage in the Analysis of Large-Scale System Software”. In: Proceedings of the 6th Workshop on Programming Languages and Operating Systems. 2011, p. 2. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Jean Melo et al. “A Quantitative Analysis of Variability Warnings in Linux”. In: Proceedings of the Tenth International Workshop on Variability Modelling of Software-intensive Systems. 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Reinhard Tartler et al. “Feature Consistency in Compile-Time Configurable System Software”. In: Proceedings of the EuroSys 2011 Conference (EuroSys ’11). 2011, pp. 47–60. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. “An Algorithm for Generating T-Wise Covering Arrays from Large Feature Models”. In: Proceedings of the 16th International Software Product Line Conference-Volume 1. 2012, pp. 46–55. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Jörg Liebig et al. “Scalable analysis of variable software”. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 2013, pp. 81–91. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Malte Lochau et al. “Model-based pairwise testing for feature interaction coverage in software product line engineering”. In: Software Quality Journal 20 (2011), pp. 567–604. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Gilles Perrouin et al. “Automated and Scalable TWise Test Case Generation Strategies for Software Product Lines”. In: Third International Conference on Software Testing, Verification and Validation (ICST). 2010, pp. 459–468. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Sebastian Oster, Florian Markert, and Philipp Ritter. “Automated Incremental Pairwise Testing of Software Product Lines”. In: Software Product Lines: Going Beyond. 2010, pp. 196–210. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Linux Coding Style Guidelines.Google ScholarGoogle Scholar
  35. Flávio Medeiros et al. “An Empirical Study on Configuration-Related Issues: Investigating Undeclared and Unused Identifiers”. In: Proceedings of the 2015 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences. 2015, pp. 35–44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Ryan A Rossi et al. “A fast parallel maximum clique algorithm for large sparse graphs and temporal strong components”. In: CoRR, abs/1302.6256 (2013).Google ScholarGoogle Scholar
  37. Andreas Ziegler, Valentin Rothberg, and Daniel Lohmann. “Analyzing the Impact of Feature Changes in Linux”. In: Proceedings of the Tenth International Workshop on Variability Modelling of Softwareintensive Systems. 2016, pp. 25–32. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Towards scalable configuration testing in variable software

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM SIGPLAN Notices
            ACM SIGPLAN Notices  Volume 52, Issue 3
            GPCE '16
            March 2017
            212 pages
            ISSN:0362-1340
            EISSN:1558-1160
            DOI:10.1145/3093335
            Issue’s Table of Contents
            • cover image ACM Conferences
              GPCE 2016: Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences
              October 2016
              212 pages
              ISBN:9781450344463
              DOI:10.1145/2993236

            Copyright © 2016 ACM

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 20 October 2016

            Check for updates

            Qualifiers

            • article

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!