skip to main content
article

IncLing: efficient product-line testing using incremental pairwise sampling

Published:20 October 2016Publication History
Skip Abstract Section

Abstract

A software product line comprises a family of software products that share a common set of features. It enables customers to compose software systems from a managed set of features. Testing every product of a product line individually is often infeasible due to the exponential number of possible products in the number of features. Several approaches have been proposed to restrict the number of products to be tested by sampling a subset of products achieving sufficient combinatorial interaction coverage. However, existing sampling algorithms do not scale well to large product lines, as they require a considerable amount of time to generate the samples. Moreover, samples are not available until a sampling algorithm completely terminates. As testing time is usually limited, we propose an incremental approach of product sampling for pairwise interaction testing (called IncLing), which enables developers to generate samples on demand in a step-wise manner. Furthermore, IncLing uses heuristics to efficiently achieve pairwise interaction coverage with a reasonable number of products. We evaluated IncLing by comparing it against existing sampling algorithms using feature models of different sizes. The results of our approach indicate efficiency improvements for product-line testing.

References

  1. M. Al-Hajjaji. Scalable Sampling and Prioritization for Product-Line Testing. In Proc. Software Engineering (SE), pages 295–298. Gesellschaft für Informatik (GI), 2015.Google ScholarGoogle Scholar
  2. M. Al-Hajjaji, J. Meinicke, S. Krieter, R. Schröter, T. Thüm, T. Leich, and G. Saake. Tool Demo: Testing Configurable Systems with FeatureIDE. In Proc. Int’l Conf. Generative Programming and Component Engineering (GPCE), 2016. To appear. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake. Similarity-Based Prioritization in Software Product-Line Testing. In Proc. Int’l Software Product Line Conf. (SPLC), pages 197–206. ACM, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. S. Apel, D. Batory, C. Kästner, and G. Saake. Feature-Oriented Software Product Lines: Concepts and Implementation. Springer, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. D. Batory, J. N. Sarvela, and A. Rauschmayer. Scaling Step-Wise Refinement. IEEE Trans. Software Engineering (TSE), 30(6):355–371, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. D. Benavides, S. Segura, and A. Ruiz-Cortés. Automated Analysis of Feature Models 20 Years Later: A Literature Review. Information Systems, 35(6):615–708, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. I. D. Carmo Machado, J. D. McGregor, Y. a. C. Cavalcanti, and E. S. De Almeida. On Strategies for Testing Software Product Lines: A Systematic Literature Review. J. Information and Software Technology (IST), 56(10):1183–1199, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. V. Chvatal. A Greedy Heuristic for the Set-Covering Problem. Mathematics of operations research, 4(3):233– 235, 1979. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. P. Clements and L. Northrop. Software Product Lines: Practices and Patterns. Addison-Wesley, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. M. B. Cohen, M. B. Dwyer, and J. Shi. Coverage and Adequacy in Software Product Line Testing. In Proc. Int’l Symposium in Software Testing and Analysis (ISSTA), pages 53–63. ACM, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. M. B. Cohen, M. B. Dwyer, and J. Shi. Interaction Testing of Highly-Configurable Systems in the Presence of Constraints. In Proc. Int’l Symposium in Software Testing and Analysis (ISSTA), pages 129–139. ACM, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. S. A. Cook. The Complexity of Theorem-proving Procedures. In Proc. of ACM Symposium on Theory of Computing (STOC), pages 151–158. ACM, 1971. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. K. Czarnecki and U. Eisenecker. Generative Programming: Methods, Tools, and Applications. ACM/Addison-Wesley, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. L. Engebretsen. The Nonapproximability of Non-Boolean Predicates. SIAM Journal on Discrete Mathematics (JDM), 18(1):114–129, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. B. J. Garvin, M. B. Cohen, and M. B. Dwyer. Evaluating Improvements to a Meta-Heuristic Search for Constrained Interaction Testing. Empirical Software Engineering (EMSE), 16(1):61–102, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. C. Henard, M. Papadakis, G. Perrouin, J. Klein, P. Heymans, and Y. L. Le Traon. Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines. IEEE Trans. Software Engineering (TSE), 40(7):650– 670, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. M. Jackson and P. Zave. Distributed Feature Composition: A Virtual Architecture for Telecommunications Services. IEEE Trans. Software Engineering (TSE), 24(10):831–847, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. M. F. Johansen, Ø. Haugen, and F. Fleurey. Properties of Realistic Feature Models Make Combinatorial Testing of Product Lines Feasible. In Proc. Int’l Conf. Model Driven Engineering Languages and Systems (MODELS), pages 638–652. Springer, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. M. F. Johansen, Ø. Haugen, and F. Fleurey. An Algorithm for Generating T-Wise Covering Arrays from Large Feature Models. In Proc. Int’l Software Product Line Conf. (SPLC), pages 46–55. ACM, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak, and A. S. Peterson. Feature-Oriented Domain Analysis (FODA) Feasibility Study. Technical Report CMU/SEI- 90-TR-21, Software Engineering Institute, 1990.Google ScholarGoogle ScholarCross RefCross Ref
  21. C. H. P. Kim, D. Batory, and S. Khurshid. Reducing Combinatorics in Testing Product Lines. In Proc. Int’l Conf. Aspect-Oriented Software Development (AOSD), pages 57—68. ACM, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. M. Kowal, S. Schulze, and I. Schaefer. Towards Efficient SPL Testing by Variant Reduction. In Proce. Int’l workshop on Variability & composition (VariComp), pages 1–6. ACM, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. D. R. Kuhn, D. R. Wallace, and A. M. Gallo Jr. Software Fault Interactions and Implications for Software Testing. IEEE Trans. Software Engineering (TSE), 30(6):418– 421, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Y. Lei, R. N. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence. IPOG: A General Strategy for T-Way Software Testing. In Proc. Int’l Conf. Engineering of Computer-Based Systems (ECBS), pages 549–556. IEEE, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. J. Liebig, A. von Rhein, C. Kästner, S. Apel, J. Dörre, and C. Lengauer. Scalable Analysis of Variable Software. In Proc. Europ. Software Engineering Conf./Foundations of Software Engineering (ESEC/FSE), pages 81–91. ACM, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. M. Lochau, I. Schaefer, J. Kamischke, and S. Lity. Incremental Model-Based Testing of Delta-oriented Software Product Lines. In Proc. of Int’l Conf. on Tests and Proofs (TAP), pages 67–82. Springer, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. F. Medeiros, C. Kästner, M. Ribeiro, R. Gheyi, and S. Apel. A Comparison of 10 Sampling Algorithms for Configurable Systems. In Proc. Int’l Conf. Software Engineering (ICSE), ICSE ’16, pages 643–654. ACM, 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. J. Melo, E. Flesborg, C. Brabrand, and A. Wasowski. A Quantitative Analysis of Variability Warnings in Linux. In Proc. Int’l Workshop Variability Modelling of Software-Intensive Systems (VaMoS), VaMoS ’16, pages 3–8. ACM, 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. S. Oster, F. Markert, and P. Ritter. Automated Incremental Pairwise Testing of Software Product Lines. In Proc. Int’l Software Product Line Conf. (SPLC), pages 196–210. Springer, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. S. Oster, I. Zorcic, F. Markert, and M. Lochau. MoSo-PoLiTe - Tool Support for Pairwise and Model-Based Software Product Line Testing. In Proc. Int’l Workshop Variability Modelling of Software-Intensive Systems (VaMoS), pages 79–82. ACM, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. G. Perrouin, S. Oster, S. Sen, J. Klein, B. Baudry, and Y. Le Traon. Pairwise Testing for Software Product Lines: Comparison of Two Approaches. Software Quality Journal (SQJ), 20(3-4):605–643, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. G. Perrouin, S. Sen, J. Klein, B. Baudry, and Y. Le Traon. Automated and Scalable T-Wise Test Case Generation Strategies for Software Product Lines. In Proc. Int’l Conf. Software Testing, Verification and Validation (ICST), pages 459–468. IEEE, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. K. Pohl, G. Böckle, and F. J. van der Linden. Software Product Line Engineering: Foundations, Principles and Techniques. Springer, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. J. Shi, M. B. Cohen, and M. B. Dwyer. Integration Testing of Software Product Lines Using Compositional Symbolic Execution. In Proc. Int’l Conf. Fundamental Approaches to Software Engineering (FASE), pages 270– 284. Springer, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. T. Thüm, C. Kästner, F. Benduhn, J. Meinicke, G. Saake, and T. Leich. FeatureIDE: An Extensible Framework for Feature-Oriented Software Development. Science of Computer Programming (SCP), 79(0):70–85, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. D. M. Weiss. The Product Line Hall of Fame. In Proc. Int’l Software Product Line Conf. (SPLC), page 395. IEEE, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. IncLing: efficient product-line testing using incremental pairwise sampling

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!