skip to main content
article

Tool demo: testing configurable systems with FeatureIDE

Published:20 October 2016Publication History
Skip Abstract Section

Abstract

Most software systems are designed to provide custom functionality using configuration options. Testing such systems is challenging as running tests of a single configuration is often not sufficient, because defects may appear in other configurations. Ideally, all configurations of a software system should be tested, which is usually not applicable in practice due to the combinatorial explosion with respect to the configuration options. Multiple sampling strategies aim to reduce the set of tested configurations to a feasible amount, such as T-wise sampling, random configurations, and user-defined configurations. However, these strategies are often not applied in practice as they require manual effort or a specialized testing framework. Within our tool FeatureIDE, we integrate all aforementioned strategies and reduce the manual effort by automating the process of generating and testing configurations. Furthermore, we provide support for unit testing to avoid redundant test executions and for variability-aware testing. With this extension of FeatureIDE, we aim to make recent testing techniques for configurable systems applicable in practice.

References

  1. M. Al-Hajjaji, S. Krieter, T. Thüm, M. Lochau, and G. Saake. IncLing: Efficient Product-Line Testing Using Incremental Pairwise Sampling. In GPCE, 2016. To appear. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake. Similarity-Based Prioritization in Software Product-Line Testing. In SPLC, pp. 197–206. ACM, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. S. Apel, D. Batory, C. Kästner, and G. Saake. Feature-Oriented Software Product Lines: Concepts and Implementation. Springer, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. S. Apel, C. Kästner, and C. Lengauer. Language-Independent and Automated Software Composition: The FeatureHouse Experience. TSE, 39(1):63–79, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. S. Apel, A. von Rhein, P. Wendler, A. Größlinger, and D. Beyer. Strategies for Product-Line Verification: Case Studies and Experiments. In ICSE, pp. 482–491. IEEE, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. D. Batory. Feature Models, Grammars, and Propositional Formulas. In SPLC, pp. 7–20. Springer, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. D. Batory, J. N. Sarvela, and A. Rauschmayer. Scaling Step-Wise Refinement. TSE, 30(6):355–371, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. B. Beckert, R. Hähnle, and P. Schmitt. Verification of Object-Oriented Software: The KeY Approach. Springer, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. D. Benavides, S. Segura, and A. Ruiz-Cortés. Automated Analysis of Feature Models 20 Years Later: A Literature Review. Information Systems, 35(6):615–708, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. J. Bürdek, M. Lochau, S. Bauregger, A. Holzer, A. von Rhein, S. Apel, and D. Beyer. Facilitating Reuse in Multi-goal Test-Suite Generation for Software Product Lines. In FASE, pp. 84–99. Springer, 2015.Google ScholarGoogle Scholar
  11. V. Chvatal. A Greedy Heuristic for the Set-Covering Problem. MOR, 4(3):233–235, 1979. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. B. J. Garvin, M. B. Cohen, and M. B. Dwyer. Evaluating Improvements to a Meta-Heuristic Search for Constrained Interaction Testing. EMSE, 16(1):61–102, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. K. Havelund and T. Pressburger. Model Checking Java Programs Using Java PathFinder. STTT, 2(4):366–381, 2000.Google ScholarGoogle Scholar
  14. C. Henard, M. Papadakis, G. Perrouin, J. Klein, and Y. L. Traon. PLEDGE: A Product Line Editor and Test Generation Tool. In SPLC, pp. 126–129. ACM, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. K. J. Hoffman, P. Eugster, and S. Jagannathan. Semanticsaware Trace Analysis. In PLDI, pp. 453–464. ACM, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. M. Jackson and P. Zave. Distributed Feature Composition: A Virtual Architecture for Telecommunications Services. TSE, 24(10):831–847, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. M. F. Johansen, Ø. Haugen, and F. Fleurey. Properties of Realistic Feature Models Make Combinatorial Testing of Product Lines Feasible. In MODELS, pp. 638–652. Springer, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. M. F. Johansen, Ø. Haugen, and F. Fleurey. An Algorithm for Generating T-Wise Covering Arrays from Large Feature Models. In SPLC, pp. 46–55. ACM, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. G. Kiczales, J. Lamping, A. Mendhekar, C. Maeda, C. Lopes, J.-M. Loingtier, and J. Irwin. Aspect-Oriented Programming. In ECOOP, pp. 220–242. Springer, 1997.Google ScholarGoogle Scholar
  20. C. H. P. Kim, D. Marinov, S. Khurshid, D. Batory, S. Souto, P. Barros, and M. d’Amorim. SPLat: Lightweight Dynamic Analysis for Reducing Combinatorics in Testing Configurable Systems. In ESEC/FSE, pp. 257–267. ACM, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. D. R. Kuhn, D. R. Wallace, and A. M. Gallo Jr. Software Fault Interactions and Implications for Software Testing. TSE, 30(6):418–421, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. D. Le Berre and A. Parrain. The sat4j Library, Release 2.2, System Description. JSAT, 7:59–64, 2010.Google ScholarGoogle Scholar
  23. J. McGregor. Testing a Software Product Line. In Testing Techniques in Software Engineering, pp. 104–140. Springer, 2010.Google ScholarGoogle Scholar
  24. F. Medeiros, C. Kästner, M. Ribeiro, S. Nadi, and R. Gheyi. The Love/Hate Relationship with the C Preprocessor: An Interview Study. In ECOOP, pp. 495–518. Schloss Dagstuhl– Leibniz-Zentrum fuer Informatik, 2015.Google ScholarGoogle Scholar
  25. J. Meinicke, T. Thüm, R. Schröter, S. Krieter, F. Benduhn, G. Saake, and T. Leich. FeatureIDE: Taming the Preprocessor Wilderness. In ICSE, pp. 629–632. ACM, 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. J. Meinicke, C.-P. Wong, C. Kästner, T. Thüm, and G. Saake. On Essential Configuration Complexity: Measuring Interactions in Highly-configurable Systems. In ASE, pp. 483–494. ACM, 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. J. A. Pereira, S. Krieter, J. Meinicke, R. Schröter, G. Saake, and T. Leich. FeatureIDE: Scalable Product Configuration of Variable Systems. In ICSR, pp. 397–401. Springer, 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. G. Perrouin, S. Sen, J. Klein, B. Baudry, and Y. Le Traon. Automated and Scalable T-Wise Test Case Generation Strategies for Software Product Lines. In ICST, pp. 459–468. IEEE, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. M. Steffens, S. Oster, M. Lochau, and T. Fogdal. Industrial Evaluation of Pairwise SPL Testing with MoSo-PoLiTe. In VaMoS, pp. 55–62. ACM, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. R. Tartler, D. Lohmann, J. Sincero, and W. Schröder-Preikschat. Feature Consistency in Compile-Time-Configurable System Software: Facing the Linux 10,000 Feature Problem. In EuroSys, pp. 47–60. ACM, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. T. Thüm, S. Apel, C. Kästner, I. Schaefer, and G. Saake. A Classification and Survey of Analysis Strategies for Software Product Lines. CSUR, 47(1):6:1–6:45, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. T. Thüm, C. Kästner, F. Benduhn, J. Meinicke, G. Saake, and T. Leich. FeatureIDE: An Extensible Framework for Feature-Oriented Software Development. SCP, 79(0):70–85, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. T. Thüm, J. Meinicke, F. Benduhn, M. Hentschel, A. von Rhein, and G. Saake. Potential Synergies of Theorem Proving and Model Checking for Software Product Lines. In SPLC, pp. 177–186. ACM, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. A. von Rhein, S. Apel, and F. Raimondi. Introducing Binary Decision Diagrams in the Explicit-State Verification of Java code. In JavaPathfinder Workshop, 2011.Google ScholarGoogle Scholar
  35. B. Xin, W. N. Sumner, and X. Zhang. Efficient Program Execution Indexing. In PLDI, pp. 238–248. ACM, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Tool demo: testing configurable systems with FeatureIDE

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM SIGPLAN Notices
        ACM SIGPLAN Notices  Volume 52, Issue 3
        GPCE '16
        March 2017
        212 pages
        ISSN:0362-1340
        EISSN:1558-1160
        DOI:10.1145/3093335
        Issue’s Table of Contents
        • cover image ACM Conferences
          GPCE 2016: Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences
          October 2016
          212 pages
          ISBN:9781450344463
          DOI:10.1145/2993236

        Copyright © 2016 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 20 October 2016

        Check for updates

        Qualifiers

        • article

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!