Abstract
This paper presents a novel methodology for localizing faults in code as it evolves. Our insight is that the essence of failure-inducing edits made by the developer can be captured using mechanical program transformations (e.g., mutation changes). Based on the insight, we present the FIFL framework, which uses both the spectrum information of edits (obtained using the existing FaultTracer approach) as well as the potential impacts of edits (simulated by mutation changes) to achieve more accurate fault localization. We evaluate FIFL on real-world repositories of nine Java projects ranging from 5.7KLoC to 88.8KLoC. The experimental results show that FIFL is able to outperform the state-of-the-art FaultTracer technique for localizing failure-inducing program edits significantly. For example, all 19 FIFL strategies that use both the spectrum information and simulated impact information for each edit outperform the existing FaultTracer approach statistically at the significance level of 0.01. In addition, FIFL with its default settings outperforms FaultTracer by 2.33% to 86.26% on 16 of the 26 studied version pairs, and is only inferior than FaultTracer on one version pair.
- R. Abreu, P. Zoeteweij, and A. J. Van Gemund. On the accuracy of spectrum-based fault localization. In Testing: Academic and Industrial Conference Practice and Research Techniques-MUTATION, pages 89--98, 2007. Google Scholar
Digital Library
- E. Alves, M. Gligoric, V. Jagannath, and M. d'Amorim. Fault-localization using dynamic slicing and change impact analysis. In Proc. of ASE, pages 520--523, 2011. Google Scholar
Digital Library
- P. Ammann and J. Offutt. Introduction to software testing. Cambridge University Press, 2008. Google Scholar
Digital Library
- T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Theoretical and empirical studies on using program mutation to test the functional correctness of programs. In Proc. of POPL, pages 220--233, 1980. Google Scholar
Digital Library
- O. Chesley, X. Ren, B. Ryder, and F. Tip. Crisp - A Fault Localization Tool for Java Programs. In Proc. of ICSE, pages 775--779, 2007. Google Scholar
Digital Library
- B. Daniel, V. Jagannath, D. Dig, and D. Marinov. Reassert: Suggesting repairs for broken unit tests. In Proc. of ASE, pages 433--444, 2009. Google Scholar
Digital Library
- B. Daniel, T. Gvero, and D. Marinov. On test repair using symbolic execution. In Proc. of ISSTA, pages 207--218, 2010. Google Scholar
Digital Library
- R. DeMillo and A. Offutt. Constraint-based automatic test data generation. IEEE Transactions on Software Engineering, 17(9):900--910, 1991. Google Scholar
Digital Library
- R. DeMillo, R. Lipton, and F. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 11 (4):34--41, 1978. Google Scholar
Digital Library
- H. Do, S. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering, 10 (4):405--435, 2005. ISSN 1382-3256. Google Scholar
Digital Library
- P. G. Frankl, S. N. Weiss, and C. Hu. All-uses vs mutation testing: an experimental comparison of effectiveness. Journal of Systems and Software, 38(3):235--253, 1997. Google Scholar
Digital Library
- G. Fraser and A. Zeller. Mutation-driven generation of unit tests and oracles. In Proc. of ISSTA, pages 147--158, 2010. Google Scholar
Digital Library
- M. Gligoric, L. Zhang, C. Pereira, and G. Pokam. Selective mutation testing for concurrent code. In Proc. of ISSTA, pages 224--234, 2013. Google Scholar
Digital Library
- R. Hamlet. Testing programs with the aid of a compiler. IEEE Transactions on Software Engineering, (4):279--290, 1977. Google Scholar
Digital Library
- D. Hao, L. Zhang, Y. Pan, H. Mei, and J. Sun. On similarity-awareness in testing-based fault localization. Automated Software Engineering, 15(2):207--249, 2008. Google Scholar
Digital Library
- D. Hao, L. Zhang, T. Xie, H. Mei, and J.-S. Sun. Interactive fault localization using test information. Journal of Computer Science and Technology, 24(5):962--974, 2009. Google Scholar
Digital Library
- D. Hao, T. Xie, L. Zhang, X. Wang, J. Sun, and H. Mei. Test input reduction for result inspection to facilitate fault localization. Automated Software Engineering, 17(1):5--31, 2010. Google Scholar
Digital Library
- M. Harman, Y. Jia, and W. Langdon. Strong higher order mutation-based test data generation. In Proc. of FSE, pages 212--222, 2011. Google Scholar
Digital Library
- M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. A. Spoon, and A. Gujarathi. Regression test selection for Java software. In Proc. of OOPSLA, pages 312--326, 2001. Google Scholar
Digital Library
- W. Howden. Weak mutation testing and completeness of test sets. IEEE Transactions on Software Engineering, (4):371--379, 1982. Google Scholar
Digital Library
- R. Ihaka and R. Gentleman. R: A language for data analysis and graphics. Journal of computational and graphical statistics, 5(3):299--314, 1996.Google Scholar
- D. Jeffrey, N. Gupta, and R. Gupta. Fault localization using value replacement. In Proc. of ISSTA, pages 167--178, 2008. Google Scholar
Digital Library
- J. Jones, M. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In Proc. of ICSE, page 477, 2002. Google Scholar
Digital Library
- S. Kim, E. J. Whitehead, and Y. Zhang. Classifying software changes: Clean or buggy? IEEE TSE, 34(2):181--196, 2008. Google Scholar
Digital Library
- A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung. An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE TSE, 32(12):971--987, 2006. Google Scholar
Digital Library
- E. Krauser, A. Mathur, and V. Rego. High performance software testing on simd machines. IEEE Transactions on Software Engineering, 17(5):403--423, 1991. Google Scholar
Digital Library
- C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In Proc. of ICSE, pages 3--13, 2012. Google Scholar
Digital Library
- B. Liblit, M. Naik, A. Zheng, A. Aiken, and M. Jordan. Scalable statistical bug isolation. In Proc. of PLDI, pages 15--26, 2005. ISBN 1595930566. Google Scholar
Digital Library
- R. Lowry. Concepts and applications of inferential statistics. R. Lowry, 1998.Google Scholar
- C. Mayer, S. Hanenberg, R. Robbes, É. Tanter, and A. Stefik. An empirical study of the influence of static type systems on the usability of undocumented software. In Proc. of OOPSLA, pages 683--702, 2012. Google Scholar
Digital Library
- H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, and G. Rothermel. A static approach to prioritizing junit test cases. TSE, 38 (6):1258--1275, 2012. Google Scholar
Digital Library
- G. Misherghi and Z. Su. Hdd: hierarchical delta debugging. In Proc. of ICSE, pages 142--151, 2006. Google Scholar
Digital Library
- E. Murphy-Hill, T. Zimmermann, C. Bird, and N. Nagappan. The design of bug fixes. In Proc. of ICSE, pages 332--341, 2013. Google Scholar
Digital Library
- A. Offutt, A. Lee, G. Rothermel, R. Untch, and C. Zapf. An experimental determination of sufficient mutant operators. ACM Transactions on Software Engineering and Methodology (TOSEM), 5(2):99--118, 1996. Google Scholar
Digital Library
- A. Offutt, Z. Jin, and J. Pan. The dynamic domain reduction procedure for test data generation. Software-Practice and Experience, 29(2):167--194, 1999. Google Scholar
Digital Library
- M. Papadakis and Y. L. Traon. Using mutants to locate unknown faults. In Proc. of ICST Workshop on Mutation Analysis, pages 691--700, 2012. Google Scholar
Digital Library
- C. Parnin and A. Orso. Are automated debugging techniques actually helping programmers? In Proc. of ISSTA, pages 199--209, 2011. Google Scholar
Digital Library
- Y. Qi, X.Mao, Y. Lei, and C.Wang. Using automated program repair for evaluating the effectiveness of fault localization techniques. In Proc. of ISSTA, pages 191--201, 2013. Google Scholar
Digital Library
- X. Ren and B. Ryder. Heuristic ranking of Java program edits for fault localization. In Proc. of ISSTA, pages 239--249, 2007. Google Scholar
Digital Library
- X. Ren, F. Shah, F. Tip, B. Ryder, and O. Chesley. Chianti: A tool for change impact analysis of Java programs. In Proc. of OOPSLA, 2004. Google Scholar
Digital Library
- G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering, 27(10):929--948, 2001. Google Scholar
Digital Library
- D. Schuler and A. Zeller. Javalanche: Efficient mutation testing for Java. In Proc. of FSE, pages 297--298, 2009. Google Scholar
Digital Library
- D. Schuler, V. Dallmeier, and A. Zeller. Efficient mutation testing by checking invariant violations. In Proc. of ISSTA, pages 69--80, 2009. Google Scholar
Digital Library
- S. S. Shapiro and M. B.Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591--611, 1965.Google Scholar
Cross Ref
- M. Stoerzer, B. Ryder, X. Ren, and F. Tip. Finding failure-inducing changes in Java programs using change classification. In Proc. of FSE, pages 57--68, 2006. Google Scholar
Digital Library
- R. Tzoref, S. Ur, and E. Yom-Tov. Instrumenting where it hurts: an automatic concurrent debugging technique. In Proc. of ISSTA, pages 27--38, 2007. Google Scholar
Digital Library
- R. Untch, A. Offutt, and M. Harrold. Mutation analysis using mutant schemata. In ACM SIGSOFT Software Engineering Notes, volume 18, pages 139--148, 1993. Google Scholar
Digital Library
- W.Weimer, T. Nguyen, C. Le Goues, and S. Forrest. Automatically finding patches using genetic programming. In Proc. of ICSE, pages 364--374, 2009. Google Scholar
Digital Library
- F. Wilcoxon. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80--83, 1945.Google Scholar
Cross Ref
- G. Xu and A. Rountev. Regression test selection for aspectj software. In Proc. of ICSE, pages 65--74, 2007. Google Scholar
Digital Library
- K. Yu, M. Lin, J. Chen, and X. Zhang. Practical isolation of failure-inducing changes for debugging regression faults. In Proc. of ASE, pages 20--29, 2012. Google Scholar
Digital Library
- Y. Yu, J. Jones, and M. Harrold. An empirical study of the effects of test-suite reduction on fault localization. In Proc. of ICSE, pages 201--210, 2008. Google Scholar
Digital Library
- A. Zeller. Yesterday, my program worked. today, it does not. why? In Proc. of FSE, pages 253--267, 1999. Google Scholar
Digital Library
- A. Zeller. Automated debugging: Are we close? Computer, 34(11):26--31, 2001. Google Scholar
Digital Library
- L. Zhang, S.-S. Hou, J.-J. Hu, T. Xie, and H. Mei. Is operator-based mutant selection superior to random mutant selection? In Proc. of ICSE, pages 435--444, 2010. Google Scholar
Digital Library
- L. Zhang, T. Xie, L. Zhang, N. Tillmann, J. De Halleux, and H. Mei. Test generation via dynamic symbolic execution for mutation testing. In Proc. of ICSM, pages 1--10, 2010. Google Scholar
Digital Library
- L. Zhang, M. Kim, and S. Khurshid. Localizing failure-inducing program edits based on spectrum information. In Proc. of ICSM, pages 23--32, 2011. Google Scholar
Digital Library
- L. Zhang, D.Marinov, L. Zhang, and S. Khurshid. Regression mutation testing. In Proc. of ISSTA, pages 331--341, 2012. Google Scholar
Digital Library
- L. Zhang, D. Hao, L. Zhang, G. Rothermel, and H. Mei. Bridging the gap between the total and additional test-case prioritization strategies. In Proc. ICSE, pages 192--201, 2013. Google Scholar
Digital Library
- L. Zhang, D. Marinov, and S. Khurshid. Faster mutation testing inspired by test prioritization and reduction. In Proc. of ISSTA, pages 235--245, 2013. Google Scholar
Digital Library
- S. Zhang. Practical semantic test simplification. In Proc. of ICSE, pages 1173--1176, 2013. Google Scholar
Digital Library
- S. Zhang, Y. Lin, Z. Gu, and J. Zhao. Effective identification of failure-inducing changes: a hybrid approach. In Proc. of PASTE, pages 77--83, 2008. Google Scholar
Digital Library
- S. Zhang, C. Zhang, and M. D. Ernst. Automated documentation inference to explain failed tests. In Proc. of ASE, pages 63--72, 2011. Google Scholar
Digital Library
- X. Zhang, N. Gupta, and R. Gupta. Locating faults through automated predicate switching. In Proc. of ICSE, pages 272--281, 2006. Google Scholar
Digital Library
- X. Zhang, N. Gupta, and R. Gupta. Pruning dynamic slices with confidence. In Proc. of PLDI, pages 169--180, 2006. Google Scholar
Digital Library
- T. Zimmermann, N. Nagappan, P. J. Guo, and B. Murphy. Characterizing and predicting which bugs get reopened. In Proc. of ICSE, pages 1074--1083. IEEE, 2012. Google Scholar
Digital Library
Index Terms
Injecting mechanical faults to localize developer faults for evolving software
Recommendations
Injecting mechanical faults to localize developer faults for evolving software
OOPSLA '13: Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applicationsThis paper presents a novel methodology for localizing faults in code as it evolves. Our insight is that the essence of failure-inducing edits made by the developer can be captured using mechanical program transformations (e.g., mutation changes). Based ...
Injecting software faults in Python applications: The OpenStack case study
AbstractSoftware fault injection techniques have been largely used as means for evaluating the dependability of systems in presence of certain types of faults. Despite the large diversity of tools offering the possibility of emulating the presence of ...
Are faults localizable?
MSR '12: Proceedings of the 9th IEEE Working Conference on Mining Software RepositoriesMany fault localization techniques have been proposed to facilitate debugging activities. Most of them attempt to pinpoint the location of faults (i.e., localize faults) based on a set of failing and correct executions and expect debuggers to ...







Comments