Learning-based prioritization of test cases in continuous integration of highly-configurable software

ABSTRACT

Continuous Integration (CI) is a practice widely adopted in the industry to allow frequent integration of code changes. During the CI process, many test cases are executed multiple times a day, subject to time constraints. In this scenario, a learning-based approach, named COLEMAN, has been successfully applied. COLEMAN allows earlier execution of the most promising test cases to reveal faults. This approach considers CI particularities such as time budget and volatility of test cases, related to the fact that test cases can be added/removed along the CI cycles. In the CI of Highly Configuration System (HCS), many product variants must be tested, each one with different configuration options, but having test cases that are common to or reused from other variants. In this context, we found, by analogy, another particularity, the volatility of variants, that is, some variants can be included/discontinued along CI cycles. Considering this context, this work introduces two strategies for the application of COLEMAN in the CI of HCS: the Variant Test Set Strategy (VTS) that relies on the test set specific for each variant, and the Whole Test Set Strategy (WST) that prioritizes the test set composed by the union of the test cases of all variants. Both strategies are evaluated in a real-world HCS, considering three test budgets. The results show that the proposed strategies are applicable regarding the time spent for prioritization. They perform similarly regarding early fault detection, but WTS better mitigates the problem of beginning without knowledge, and is more suitable when a new variant to be tested is added.

References

1. Mustafa Al-Hajjaji, Sascha Lity, Remo Lachmann, Thomas Thüm, Ina Schaefer, and Gunter Saake. 2017. Delta-oriented product prioritization for similarity-based product-line testing. In 2017 IEEE/ACM 2nd International Workshop on Variability and Complexity in Software Design (VACE). IEEE, 34--40.
2. Andrea Arcuri and Lionel Briand. 2011. A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering. In 33rd International Conference on Software Engineering (ICSE'11). ACM, New York, NY, USA, 1--10.
3. Antonia Bertolino, Antonio Guerriero, Breno Miranda, Roberto Pietrantuono, and Stefano Russo. 2020. Learning-to-Rank vs Ranking-to-Learn: Strategies for Regression Testing in Continuous Integration. In 42nd International Conference on Software Engineering (ICSE'20). ACM, New York, NY, USA.
4. Thelma Elita Colanzi, Wesley Klewerton Guez Assunção, Paulo Roberto Farah, Silvia Regina Vergilio, and Giovani Guizzo. 2019. A Review of Ten Years of the Symposium on Search-Based Software Engineering. In Symposium on Search-Based Software Engineering. Springer, Cham, 42--57.
5. Hyunsook Do, Siavash Mirarab, Ladan Tahvildari, and Gregg Rothermel. 2008. An Empirical Study of the Effect of Time Constraints on the Cost-Benefits of Regression Testing. In 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, New York, NY, USA, 71--82.
6. Paul M Duvall, Steve Matyas, and Andrew Glover. 2007. Continuous integration: improving software quality and reducing risk. Pearson Education.
7. Sebastian Elbaum, Gregg Rothermel, and John Penix. 2014. Techniques for Improving Regression Testing in Continuous Integration Development Environments. In 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, New York, NY, USA, 235--245.
8. Emelie Engström. 2010. Regression Test Selection and Product Line System Testing. In 3rd International Conference on Software Testing, Verification and Validation. IEEE, 512--515.
9. Alireza Ensan, Ebrahim Bagheri, Mohse Asadi, Dragan Gasevic, and Yevgen Biletskiy. 2011. Goal-Oriented Test Case Selection and Prioritization for Product Line Feature Models. In 8th International Conference on Information Technology: New Generations. IEEE, 291--298.
10. Angelo Gargantini, Justyna Petke, Marco Radavelli, and Paolo Vavassori. 2016. Validation of Constraints Among Configuration Parameters Using Search-Based Combinatorial Interaction Testing. In 8th International Symposium on Search Based Software Engineering. Springer, Cham, 49--63.
11. Satendra Kumar and Rajkumar. 2016. Test case prioritization techniques for software product line: A survey. In International Conference on Computing, Communication and Automation (ICCCA). IEEE, 884--889.
12. Remo Lachmann, Simon Beddig, Sascha Lity, Sandro Schulze, and Ina Schaefer. 2017. Risk-based integration testing of software product lines. In 11th International Workshop on Variability Modelling of Software-intensive Systems. 52--59.
13. Remo Lachmann, Sascha Lity, Sabrina Lischke, Simon Beddig, Sandro Schulze, and Ina Schaefer. 2015. Delta-oriented test case prioritization for integration testing of software product lines. In 19th International Conference on Software Product Line. 81--90.
14. K. Li, A. Fialho, S. Kwong, and Q. Zhang. 2014. Adaptive operator selection with bandits for a multiobjective evolutionary algorithm based on decomposition. Evolutionary Computation, IEEE Transactions on 18, 1 (2014), 114--130.
15. Frank J. van der Linden, Klaus Schmid, and Eelco Rommes. 2007. Software Product Lines in Action: The Best Industrial Practice in Product Line Engineering. Springer-Verlag, Berlin, Heidelberg.
16. Sascha Lity, Manuel Nieke, Thomas Thüm, and Ina Schaefer. 2019. Retest test selection for product-line regression testing of variants and versions of variants. Journal of Systems and Software 147 (2019), 46--63.
17. Malte Lochau, Ina Schaefer, Jochen Kamischke, and Sascha Lity. 2012. Incremental Model-Based Testing of Delta-Oriented Software Product Lines. In Tests and Proofs. Springer, Berlin, Heidelberg, 67--82.
18. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics 18 (1947), 50--60.
19. Dusica Marijan, Arnaud Gotlieb, and Marius Liaaen. 2019. A learning algorithm for optimizing continuous integration development and testing practice. Software: Practice and Experience 49, 2 (2019), 192--213.
20. Dusica Marijan, Arnaud Gotlieb, and Sagar Sen. 2013. Test Case Prioritization for Continuous Regression Testing: An Industrial Case Study. In IEEE International Conference on Software Maintenance (ICMS). IEEE, 540--543.
21. Dusica Marijan and Marius Liaaen. 2017. Test Prioritization with Optimally Balanced Configuration Coverage. In IEEE 18th International Symposium on High Assurance Systems Engineering (HASE). IEEE, 100--103.
22. Dusica Marijan, Marius Liaaen, Arnaud Gotlieb, Sagar Sen, and Carlo Ieva. 2017. TITAN: Test Suite Optimization for Highly Configurable Software. In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, 524--531.
23. Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A Comparison of 10 Sampling Algorithms for Configurable Systems. In 38th International Conference on Software Engineering (ICSE '16). ACM, New York, NY, USA, 643--54.
24. Flávio Medeiros, Márcio Ribeiro, Rohit Gheyi, Sven Apel, Christian Kästner, Bruno Ferreira, Luiz Carvalho, and Baldoino Fonseca. 2018. Discipline Matters: Refactoring of Preprocessor Directives in the #ifdef Hell. IEEE Transactions on Software Engineering 44, 5 (2018), 453--469.
25. Mukelabai Mukelabai, Damir Nešiundefined, Salome Maro, Thorsten Berger, and Jan-Philipp Steghöfer. 2018. Tackling Combinatorial Explosion: A Study of Industrial Needs and Practices for Analyzing Highly Configurable Systems. In 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, New York, USA, 155--166.
26. Raiza Oliveira, Bruno Cafeo, and Andre Hora. 2019. On the Evolution of Feature Dependencies: An Exploratory Study of Preprocessor-Based Systems. In 13th International Workshop on Variability Modelling of Software-Intensive Systems. ACM, New York, NY, USA, 1--9.
27. Jackson A. Prado Lima, Willian D. F. Mendonça, Wesley K. G. Assunção, and Silvia R. Vergilio. 2020. Supplementary Material - Learning-based Prioritization of Test Cases in Continuous Integration of Highly-Configurable Software.
28. Jackson A. Prado Lima and Silvia R. Vergilio. 2020. A Multi-Armed Bandit Approach for Test Case Prioritization in Continuous Integration Environments. IEEE Transactions on Software Engineering (2020), 12.
29. Jackson A. Prado Lima and Silvia R. Vergilio. 2020. Multi-Armed Bandit Test Case Prioritization in Continuous Integration Environments: A Trade-off Analysis. In 5th Brazilian Symposium on Systematic and Automated Software Testing (SAST '20). ACM.
30. Jackson A. Prado Lima and Silvia R. Vergilio. 2020. Test Case Prioritization in Continuous Integration environments: A systematic mapping study. Information and Software Technology 121 (2020), 106268.
31. Xiao Qu, Myra B. Cohen, and Katherine M. Woolf. 2007. Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization. In IEEE International Conference on Software Maintenance. IEEE, 255--264.
32. Herbert Robbins. 1985. Some aspects of the sequential design of experiments. In Herbert Robbins Selected Papers. Springer, 169--177.
33. Gregg Rothermel. 2018. Improving Regression Testing in Continuous Integration Development Environments (Keynote). In 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation (A-TEST 2018). ACM, New York, NY, USA, 1.
34. Gregg Rothermel, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. 1999. Test Case Prioritization: An Empirical Study. In IEEE International Conference on Software Maintenance (ICSM '99). IEEE Computer Society, 179.
35. Per Runeson and Emelie Engström. 2012. Chapter 7 - Regression Testing in Software Product Line Engineering. In Advances in Computers, Ali Hurson and Atif Memon (Eds.). Vol. 86. Elsevier, 223--263.
36. M. Shahin, M. Ali Babar, and L. Zhu. 2017. Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices. IEEE Access 5 (2017), 3909--3943.
37. Helge Spieker, Arnaud Gotlieb, Dusica Marijan, and Morten Mossige. 2017. Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration. In 26th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2017). ACM, New York, NY, USA, 12--22.
38. Andras Vargha and Harold D. Delaney. 2000. A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25, 2 (Jan. 2000), 101--132.
39. Alexander von Rhein, Alexander Grebhahn, Sven Apel, Norbert Siegmund, Dirk Beyer, and Thorsten Berger. 2015. Presence-condition Simplification in Highly Configurable Systems. In 37th International Conference on Software Engineering - Volume 1 (ICSE 2015). IEEE, New York, USA, 178--188.
40. Shuai Wang, Shaukat Ali, and Arnaud Gotlieb. 2015. Cost-effective test suite minimization in product lines using search techniques. Journal of Systems and Software 103 (2015), 370--391.
41. Shuai Wang, Shaukat Ali, Arnaud Gotlieb, and Marius Liaaen. 2016. A systematic test case selection methodology for product lines: results and insights from an industrial case study. Empirical Software Engineering 21 (2016), 1586--1622.
42. Zhihong Xu, Myra B. Cohen, Wayne Motycka, and Gregg Rothermel. 2013. Continuous Test Suite Augmentation in Software Product Lines. In 17th International Software Product Line Conference (SPLC '13). Association for Computing Machinery, New York, NY, USA, 52--61.
43. Shin Yoo and Mark Harman. 2012. Regression Testing Minimization, Selection and Prioritization: A Survey. Software: Testing, Verification, and Reliability 22, 2 (March 2012), 67--120.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Experimentation

• Software and its engineering

  • Software creation and management

    • Software development techniques

      • Reusability

        • Software product lines

    • Software verification and validation

      • Software defect analysis

        • Software testing and debugging