skip to main content
research-article

To be precise: regression aware debugging

Published:19 October 2016Publication History
Skip Abstract Section

Abstract

Bounded model checking based debugging solutions search for mutations of program expressions that produce the expected output for a currently failing test. However, the current localization tools are not regression aware: they do not use information from the passing tests in their localization formula. On the other hand, the current repair tools attempt to guarantee regression freedom: when provided with a set of passing tests, they guarantee that none of these tests can break due to the suggested repair patch, thereby constructing a large repair formula.

In this paper, we propose regression awareness as a means to improve the quality of localization and to scale repair. To enable regression awareness, we summarize the proof of correctness of each passing test by computing Craig Interpolants over a symbolic encoding of the passing execution, and use these summaries as additional soft constraints while synthesizing altered executions corresponding to failing tests. Intuitively, these additional constraints act as roadblocks, thereby discouraging executions that may damage the proof of a passing test. We use a partial MAXSAT solver to relax the proofs in a systematic way, and use a ranking function that penalizes mutations that damage the existing proofs.

We have implemented our algorithms into a tool, TINTIN, that enables regression aware localization and repair. For localizations, our strategy is effective in extracting a superior ranking of suspicious locations: on a set of 52 different versions across 12 different programs spanning three benchmark suites, TINTIN achieves a saving of developer effort by almost 45% (in terms of the locations that must be examined by a developer to reach the ground-truth repair) in the worst case and 27% in the average case over existing techniques. For automated repairs, on our set of benchmarks, TINTIN achieves a 2.3X speedup over existing techniques without sacrificing much on the ranking of the repair patches: the ground-truth repair appears as the topmost suggestion in more than 70% of our benchmarks.

References

  1. Software-artifact infrastructure repository (SIR). http://sir.unl.edu/portal/index.php.Google ScholarGoogle Scholar
  2. Aws Albarghouthi, Arie Gurfinkel, Yi Li, Sagar Chaki, and Marsha Chechik. UFO: Verification with Interpolants and Abstract Interpretation. In Proceedings of the 19th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS’13, pages 637–640, Berlin, Heidelberg, 2013. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Aws Albarghouthi, Yi Li, Arie Gurfinkel, and Marsha Chechik. UFO: A Framework for Abstractionand Interpolation-based Software Verification. In Proceedings of the 24th International Conference on Computer Aided Verification, CAV’12, pages 672–678, Berlin, Heidelberg, 2012. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Aws Albarghouthi and Kenneth L. McMillan. Beautiful Interpolants. In Proceedings of the 25th International Conference on Computer Aided Verification, CAV’13, pages 313– 329, Berlin, Heidelberg, 2013. Springer-Verlag.Google ScholarGoogle Scholar
  5. Thomas Ball, Rupak Majumdar, Todd Millstein, and Sriram K. Rajamani. Automatic Predicate Abstraction of C Programs. In Proceedings of the ACM SIGPLAN 2001 Conference on Programming Language Design and Implementation, PLDI ’01, pages 203–213, New York, NY, USA, 2001. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Thomas Ball, Mayur Naik, and Sriram K. Rajamani. From Symptom to Cause: Localizing Errors in Counterexample Traces. In Proceedings of the 30th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’03, pages 97–105, New York, NY, USA, 2003. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Dirk Beyer. Software Verification and Verifiable Witnesses - (Report on SV-COMP 2016), 2016. (accessed in Jan 2016) https://github.com/sosy-lab/sv-benchmarks.Google ScholarGoogle Scholar
  8. Nikolaj Bjørner, Arie Gurfinkel, Ken McMillan, and Andrey Rybalchenko. Horn clause solvers for program verification. In Fields of Logic and Computation II, pages 24–51. Springer, 2015.Google ScholarGoogle Scholar
  9. Cristian Cadar, Daniel Dunbar, and Dawson Engler. KLEE: Unassisted and Automatic Generation of High-coverage Tests for Complex Systems Programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI’08, pages 209–224, Berkeley, CA, USA, 2008. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Satish Chandra, Emina Torlak, Shaon Barman, and Rastislav Bodik. Angelic Debugging. In Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, pages 121–130, New York, NY, USA, 2011. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Alessandro Cimatti, Alberto Griggio, Bastiaan Joost Schaafsma, and Roberto Sebastiani. The MathSAT5 SMT Solver. In Proceedings of the 19th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS’13, pages 93–107, Berlin, Heidelberg, 2013. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Edmund Clarke, Daniel Kroening, and Flavio Lerda. A tool for checking ANSI-C programs. In Tools and Algorithms for the Construction and Analysis of Systems, pages 168–176. Springer, 2004.Google ScholarGoogle Scholar
  13. Patrick Cousot and Radhia Cousot. Abstract Interpretation: A Unified Lattice Model for Static Analysis of Programs by Construction or Approximation of Fixpoints. In Proceedings of the 4th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages, POPL ’77, pages 238–252, New York, NY, USA, 1977. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. William Craig. Three uses of the Herbrand-Gentzen theorem in relating model theory and proof theory. The Journal of Symbolic Logic, 22(03):269–285, 1957.Google ScholarGoogle ScholarCross RefCross Ref
  15. Hyunsook Do, Sebastian Elbaum, and Gregg Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Softw. Engg., 10(4):405–435, October 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Zhaohui Fu and Sharad Malik. On Solving the Partial MAXSAT Problem. In Proceedings of the 9th International Conference on Theory and Applications of Satisfiability Testing, SAT’06, pages 252–265, Berlin, Heidelberg, 2006. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Patrice Godefroid, Nils Klarlund, and Koushik Sen. DART: Directed Automated Random Testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’05, pages 213–223, New York, NY, USA, 2005. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Alex Groce, Sagar Chaki, Daniel Kroening, and Ofer Strichman. Error Explanation with Distance Metrics. Int. J. Softw. Tools Technol. Transf., 8(3):229–247, June 2006.Google ScholarGoogle ScholarCross RefCross Ref
  19. Joxan Jaffar, Vijayaraghavan Murali, and Jorge A. Navas. Boosting Concolic Testing via Interpolation. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, pages 48–58, New York, NY, USA, 2013. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Joxan Jaffar, Vijayaraghavan Murali, Jorge A. Navas, and Andrew E. Santosa. TRACER: A Symbolic Execution Tool for Verification. In Proceedings of the 24th International Conference on Computer Aided Verification, CAV’12, pages 758–766, Berlin, Heidelberg, 2012. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Joxan Jaffar, Jorge A. Navas, and Andrew E. Santosa. Unbounded Symbolic Execution for Program Verification. In Proceedings of the Second International Conference on Runtime Verification, RV’11, pages 396–411, Berlin, Heidelberg, 2012. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Susmit Jha, Sumit Gulwani, Sanjit A. Seshia, and Ashish Tiwari. Oracle-guided Component-based Program Synthesis. In Proceedings of the 32Nd ACM/IEEE International Conference on Software Engineering - Volume 1, ICSE ’10, pages 215–224, New York, NY, USA, 2010. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Manu Jose and Rupak Majumdar. Cause Clue Clauses: Error Localization Using Maximum Satisfiability. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’11, pages 437– 446, New York, NY, USA, 2011. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. GenProg: A Generic Method for Automatic Software Repair. IEEE Trans. Softw. Eng., 38(1):54–72, January 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Chu Min Li and Felip Manya. MaxSAT, Hard and Soft Constraints. Handbook of satisfiability, 185:613–631, 2009.Google ScholarGoogle Scholar
  26. Ben Liblit, Mayur Naik, Alice X. Zheng, Alex Aiken, and Michael I. Jordan. Scalable Statistical Bug Isolation. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’05, pages 15–26, New York, NY, USA, 2005. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Chao Liu, Xifeng Yan, Long Fei, Jiawei Han, and Samuel P. Midkiff. SOBER: Statistical Model-based Bug Localization. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ESEC/FSE-13, pages 286–295, New York, NY, USA, 2005. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Yongmei Liu and Bing Li. Automated Program Debugging via Multiple Predicate Switching. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI’10, pages 327–332. AAAI Press, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Fan Long and Martin Rinard. Staged Program Repair with Condition Synthesis. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pages 166–178, New York, NY, USA, 2015. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Joao Marques-Silva. Minimal Unsatisfiability: Models, Algorithms and Applications (Invited Paper). In Proceedings of the 2010 40th IEEE International Symposium on Multiple-Valued Logic, ISMVL ’10, pages 9–14, Washington, DC, USA, 2010. IEEE Computer Society. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Joao Marques-Silva and Jordi Planes. Algorithms for Maximum Satisfiability Using Unsatisfiable Cores. In Proceedings of the Conference on Design, Automation and Test in Europe, DATE ’08, pages 408–413, New York, NY, USA, 2008. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. K. L. McMillan. Applications of Craig Interpolants in Model Checking. In Proceedings of the 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS’05, pages 1–12, Berlin, Heidelberg, 2005. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Kenneth L. McMillan. Lazy Abstraction with Interpolants. In Proceedings of the 18th International Conference on Computer Aided Verification, CAV’06, pages 123–136, Berlin, Heidelberg, 2006. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Kenneth L. McMillan. Lazy Annotation for Program Testing and Verification. In Proceedings of the 22Nd International Conference on Computer Aided Verification, CAV’10, pages 104–118, Berlin, Heidelberg, 2010. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Kenneth L. Mcmillan. Lazy Annotation Revisited. In Proceedings of the 16th International Conference on Computer Aided Verification - Volume 8559, pages 243–259, New York, NY, USA, 2014. Springer-Verlag New York, Inc. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. DirectFix: Looking for Simple Program Repairs. In Proceedings of the 37th International Conference on Software Engineering - Volume 1, ICSE ’15, pages 448–458, Piscataway, NJ, USA, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. Angelix: Scalable multiline program patch synthesis via symbolic analysis. In Proceedings of the 38th International Conference on Software Engineering, ICSE ’16, pages 691–701, New York, NY, USA, 2016. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Varun Modi, Subhajit Roy, and Sanjeev K. Aggarwal. Exploring Program Phases for Statistical Bug Localization. In Proceedings of the 11th ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering, PASTE ’13, pages 33–40, New York, NY, USA, 2013. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Martin Monperrus. A Critical Review of ”Automatic Patch Generation Learned from Human-written Patches”: Essay on the Problem Statement and the Evaluation of Automatic Software Repair. In Proceedings of the 36th International Conference on Software Engineering, ICSE 2014, pages 234–242, New York, NY, USA, 2014. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Hoang Duong Thien Nguyen, Dawei Qi, Abhik Roychoudhury, and Satish Chandra. SemFix: Program Repair via Semantic Analysis. In Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pages 772– 781, Piscataway, NJ, USA, 2013. IEEE Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. ThanhVu Nguyen, Westley Weimer, Claire Le Goues, and Stephanie Forrest. Using execution paths to evolve software patches. In Software Testing, Verification and Validation Workshops, 2009. ICSTW’09. International Conference on, pages 152–153. IEEE, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Manos Renieres and Steven P Reiss. Fault localization with nearest neighbor queries. In Automated Software Engineering, 2003. Proceedings. 18th IEEE International Conference on, pages 30–39. IEEE, 2003.Google ScholarGoogle Scholar
  43. Hesam Samimi, Max Schäfer, Shay Artzi, Todd D. Millstein, Frank Tip, and Laurie J. Hendren. Automated repair of HTML generation errors in PHP applications using string constraint solving. In 34th International Conference on Software Engineering, ICSE 2012, June 2-9, 2012, Zurich, Switzerland, pages 277–287, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Ondrej Sery, Grigory Fedyukovich, and Natasha Sharygina. FunFrog: Bounded Model Checking with Interpolation-Based Function Summarization, pages 203–207. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Frank Tip. A survey of program slicing techniques. J. Prog. Lang., 3(3), 1995.Google ScholarGoogle Scholar
  46. Wei Wang and Clark Barrett. Cascade 2016 - (Competition Contribution), 2016.Google ScholarGoogle Scholar
  47. (accessed in Feb 2016) http://cascade.cims.nyu.edu/vmcai.html.Google ScholarGoogle Scholar
  48. Westley Weimer. Patches As Better Bug Reports. In Proceedings of the 5th International Conference on Generative Programming and Component Engineering, GPCE ’06, pages 181–190, New York, NY, USA, 2006. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Westley Weimer, ThanhVu Nguyen, Claire Le Goues, and Stephanie Forrest. Automatically Finding Patches Using Genetic Programming. In Proceedings of the 31st International Conference on Software Engineering, ICSE ’09, pages 364– 374, Washington, DC, USA, 2009. IEEE Computer Society. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Andreas Zeller and Ralf Hildebrandt. Simplifying and Isolating Failure-Inducing Input. IEEE Trans. Softw. Eng., 28(2):183–200, February 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. To be precise: regression aware debugging

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM SIGPLAN Notices
          ACM SIGPLAN Notices  Volume 51, Issue 10
          OOPSLA '16
          October 2016
          915 pages
          ISSN:0362-1340
          EISSN:1558-1160
          DOI:10.1145/3022671
          Issue’s Table of Contents
          • cover image ACM Conferences
            OOPSLA 2016: Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications
            October 2016
            915 pages
            ISBN:9781450344449
            DOI:10.1145/2983990

          Copyright © 2016 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 19 October 2016

          Check for updates

          Qualifiers

          • research-article

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!