skip to main content
research-article
Open Access
Artifacts Available
Artifacts Evaluated & Functional

Test generation for higher-order functions in dynamic languages

Published:24 October 2018Publication History
Skip Abstract Section

Abstract

Test generation has proven to provide an effective way of identifying programming errors. Unfortunately, current test generation techniques are challenged by higher-order functions in dynamic languages, such as JavaScript functions that receive callbacks. In particular, existing test generators suffer from the unavailability of statically known type signatures, do not provide functions or provide only trivial functions as inputs, and ignore callbacks triggered by the code under test. This paper presents LambdaTester, a novel test generator that addresses the specific problems posed by higher-order functions in dynamic languages. The approach automatically infers at what argument position a method under test expects a callback, generates and iteratively improves callback functions given as input to this method, and uses novel test oracles that check whether and how callback functions are invoked. We apply LambdaTester to test 43 higher-order functions taken from 13 popular JavaScript libraries. The approach detects unexpected behavior in 12 of the 13 libraries, many of which are missed by a state-of-the-art test generator.

Skip Supplemental Material Section

Supplemental Material

a161-selakovic.webm

References

  1. Saba Alimadadi, Sheldon Sequeira, Ali Mesbah, and Karthik Pattabiraman. 2014. Understanding JavaScript event-based interactions. In 36th International Conference on Software Engineering, ICSE ’14, Hyderabad, India - May 31 - June 07, 2014. 367–377. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Esben Andreasen, Liang Gong, Anders Møller, Michael Pradel, Marija Selakovic, Koushik Sen, and Cristian-Alexandru Staicu. 2017. A Survey of Dynamic Analysis and Test Generation for JavaScript. Comput. Surveys (2017). Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Andrea Arcuri and Lionel Briand. 2011. Adaptive Random Testing: An Illusion of Effectiveness?. In Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA ’11). ACM, New York, NY, USA, 265–275. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Shay Artzi, Julian Dolby, Simon Holm Jensen, Anders Møller, and Frank Tip. 2011. A framework for automated testing of JavaScript web applications. In ICSE. 571–580. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: automated testing based on Java predicates. In International Symposium on Software Testing and Analysis (ISSTA). 123–133. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Jacob Burnim, Sudeep Juvekar, and Koushik Sen. 2009. WISE: Automated test generation for worst-case complexity. In ICSE. IEEE, 463–473. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Cristian Cadar, Daniel Dunbar, and Dawson R. Engler. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In Symposium on Operating Systems Design and Implementation (OSDI). USENIX, 209–224. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Tsong Yueh Chen, Fei-Ching Kuo, Robert G. Merkel, and T. H. Tse. 2010. Adaptive Random Testing: The ART of test case diversity. Journal of Systems and Software 83, 1 (2010), 60–66. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Tsong Yueh Chen, Hing Leung, and IK Mak. 2004. Adaptive random testing. In Annual Asian Computing Science Conference. Springer, 320–329.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Ravai Chugh, Jeffrey A. Meister, Ranjit Jhala, and Sorin Lerner. 2009. Staged Information Flow for JavaScript. In Conference on Programming Language Design and Implementation (PLDI). ACM, 50–62. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Ilinca Ciupa, Andreas Leitner, Manuel Oriol, and Bertrand Meyer. 2008. ARTOO: adaptive random testing for object-oriented software. In International Conference on Software Engineering (ICSE). ACM, 71–80. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. I. Ciupa, A. Pretschner, M. Oriol, A. Leitner, and B. Meyer. 2011. On the number and nature of faults found by random testing. Software Testing, Verification and Reliability 21, 1 (2011), 3–28. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Koen Claessen and John Hughes. 2011. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. SIGPLAN Not. 46, 4 (May 2011), 53–64. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Christoph Csallner and Yannis Smaragdakis. 2004. JCrasher: an automatic robustness tester for Java. Software Prac. Experience 34, 11 (2004), 1025–1050. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Joe W. Duran and Simeon C. Ntafos. 1984. An Evaluation of Random Testing. IEEE Trans. Softw. Eng. 10, 4 (July 1984), 438–444. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Markus Ermuth and Michael Pradel. 2016. Monkey See, Monkey Do: Effective Generation of GUI Tests with Inferred Macro Events. In International Symposium on Software Testing and Analysis (ISSTA). 82–93. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Asger Feldthaus and Anders Møller. 2014. Checking correctness of TypeScript interfaces for JavaScript libraries. In Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA). ACM, 1–16. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In SIGSOFT-/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-19) and ESEC’11: 13th European Software Engineering Conference (ESEC-13), Szeged, Hungary, September 5-9, 2011. 416–419. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing. SIGPLAN Not. 40, 6 (June 2005), 213–223. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Patrice Godefroid, Michael Y. Levin, and David A. Molnar. 2008. Automated Whitebox Fuzz Testing. In Network and Distributed System Security Symposium (NDSS).Google ScholarGoogle Scholar
  21. Liang Gong, Michael Pradel, and Koushik Sen. 2015a. JITProf: Pinpointing JIT-unfriendly JavaScript Code. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 357–368. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Liang Gong, Michael Pradel, Manu Sridharan, and Koushik Sen. 2015b. DLint: Dynamically Checking Bad Coding Practices in JavaScript. In International Symposium on Software Testing and Analysis (ISSTA). 94–105. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Alex Groce, Gerard Holzmann, and Rajeev Joshi. 2007. Randomized Differential Testing As a Prelude to Formal Verification. In Proceedings of the 29th International Conference on Software Engineering (ICSE ’07). IEEE Computer Society, Washington, DC, USA, 621–631. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Daniel Hedin, Arnar Birgisson, Luciano Bello, and Andrei Sabelfeld. 2014. JSFlow: tracking information flow in JavaScript and its APIs. In SAC. 1663–1671. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Stefan Heule, Manu Sridharan, and Satish Chandra. 2015. Mimic: computing models for opaque code. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015. 710–720. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Simon Holm Jensen, Anders Møller, and Peter Thiemann. 2009. Type Analysis for JavaScript. In Symposium on Static Analysis (SAS). Springer, 238–255. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Simon Holm Jensen, Manu Sridharan, Koushik Sen, and Satish Chandra. 2015. MemInsight: platform-independent memory debugging for JavaScript. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015. 345–356. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. J. C. King. 1976. Symbolic Execution and Program Testing. Commun. ACM 19, 7 (1976), 385–394. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Casey Klein, Matthew Flatt, and Robert Bruce Findler. 2010. Random testing for higher-order, stateful programs. In Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). ACM, 555–566. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Pieter Koopman and Rinus Plasmeijer. 2006. Automatic Testing of Higher Order Functions. In Asian Symposium on Programming Languages and Systems (APLAS). 148–164. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Erik Krogh Kristensen and Anders Møller. 2017. Type test scripts for TypeScript testing. PACMPL 1, OOPSLA (2017), 90:1–90:25. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Magnus Madsen, Benjamin Livshits, and Michael Fanning. 2013. Practical static analysis of JavaScript applications in the presence of frameworks and libraries. In ESEC/SIGSOFT FSE. 499–509. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Magnus Madsen, Frank Tip, and Ondrej Lhoták. 2015. Static analysis of event-driven Node.js JavaScript applications. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015. 505–519. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Alessandro Marchetto, Paolo Tonella, and Filippo Ricca. 2008. State-Based Testing of Ajax Web Applications. In ICST. IEEE Computer Society, 121–130. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. William M. McKeeman. 1998. Differential Testing for Software. Digital Technical Journal 10, 1 (1998), 100–107.Google ScholarGoogle Scholar
  36. Atif M. Memon. 2007. An event-flow model of GUI-based applications for testing. Softw. Test., Verif. Reliab. (2007), 137–157. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Ali Mesbah, Engin Bozdag, and Arie van Deursen. 2008. Crawling Ajax by Inferring User Interface State Changes. In International Conference on Web Engineering (ICWE). 122–134. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Ali Mesbah and Arie van Deursen. 2009. Invariant-based automatic testing of Ajax user interfaces. In ICSE. 210–220. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Erdal Mutlu, Serdar Tasiran, and Benjamin Livshits. 2015. Detecting JavaScript Races that Matter. In European Software Engineering Conference and International Symposium on Foundations of Software Engineering (ESEC/FSE). Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Phuc C. Nguyen and David Van Horn. 2015. Relatively complete counterexamples for higher-order programs. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, Portland, OR, USA, June 15-17, 2015. 446–456. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. S. C. Ntafos. 2001. On comparisons of random, partition, and proportional partition testing. IEEE Transactions on Software Engineering 27, 10 (Oct 2001), 949–960. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Carlos Pacheco and Michael D. Ernst. 2007. Randoop: Feedback-directed Random Testing for Java. In Companion to the 22Nd ACM SIGPLAN Conference on Object-oriented Programming Systems and Applications Companion (OOPSLA ’07). ACM, New York, NY, USA, 815–816. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Carlos Pacheco, Shuvendu K. Lahiri, and Thomas Ball. 2008. Finding Errors in .Net with Feedback-directed Random Testing. In Proceedings of the 2008 International Symposium on Software Testing and Analysis (ISSTA ’08). ACM, New York, NY, USA, 87–96. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Michal H. Palka, Koen Claessen, Alejandro Russo, and John Hughes. 2011. Testing an Optimising Compiler by Generating Random Lambda Terms. In Proceedings of the 6th International Workshop on Automation of Software Test (AST ’11). ACM, New York, NY, USA, 91–97. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Jibesh Patra, Pooja N. Dixit, and Michael Pradel. 2018. ConflictJS: Finding and Understanding Conflicts Between JavaScript Libraries. In ICSE. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Jacques A. Pienaar and Robert Hundt. 2013. JSWhiz: Static analysis for JavaScript memory leaks. In Proceedings of the 2013 IEEE/ACM International Symposium on Code Generation and Optimization, CGO 2013, Shenzhen, China, February 23-27, 2013. 11:1–11:11. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Michael Pradel and Thomas R. Gross. 2012. Fully Automatic and Precise Detection of Thread Safety Violations. In Conference on Programming Language Design and Implementation (PLDI). 521–530. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Michael Pradel, Parker Schuh, George Necula, and Koushik Sen. 2014. EventBreak: Analyzing the Responsiveness of User Interfaces through Performance-Guided Test Generation. In Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). 33–47. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Michael Pradel, Parker Schuh, and Koushik Sen. 2015. TypeDevil: Dynamic Type Inconsistency Analysis for JavaScript. In International Conference on Software Engineering (ICSE). Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Veselin Raychev, Pavol Bielik, Martin T. Vechev, and Andreas Krause. 2016. Learning programs from noisy data. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20 - 22, 2016. 761–774. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Malavika Samak and Murali Krishna Ramanathan. 2014. Multithreaded Test Synthesis for Deadlock Detection. In Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA). 473–489. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Marija Selakovic, Thomas Glaser, and Michael Pradel. 2017. An Actionable Performance Profiler for Optimizing the Order of Evaluations. In International Symposium on Software Testing and Analysis (ISSTA). 170–180. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Koushik Sen, Swaroop Kalasapur, Tasneem Brutch, and Simon Gibbs. 2013. Jalangi: A Selective Record-replay and Dynamic Analysis Framework for JavaScript. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013). ACM, New York, NY, USA, 488–498. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A Concolic Unit Testing Engine for C. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-13). ACM, New York, NY, USA, 263–272. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Manu Sridharan, Julian Dolby, Satish Chandra, Max Schäfer, and Frank Tip. 2012. Correlation Tracking for Points-To Analysis of JavaScript. In ECOOP 2012 - Object-Oriented Programming - 26th European Conference, Beijing, China, June 11-16, 2012. Proceedings. 435–458. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Peter Thiemann. 2005. Towards a Type System for Analyzing JavaScript Programs. In European Symposium on Programming (ESOP). 408–422. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. 2009. MSeqGen: Objectoriented unit-test generation via mining source code. In European Software Engineering Conference and International Symposium on Foundations of Software Engineering (ESEC/FSE). ACM, 193–202. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Zhendong Su. 2011. Synthesizing method sequences for high-coverage testing.. In OOPSLA. 189–206. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Willem Visser, Corina S. Pasareanu, and Sarfraz Khurshid. 2004. Test input generation with Java PathFinder. In International Symposium on Software Testing and Analysis (ISSTA). ACM, 97–107. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. 2005. Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution. In Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS). Springer, 365–381. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Wujie Zheng, Qirun Zhang, Michael Lyu, and Tao Xie. 2010. Random Unit-test Generation with MUT-aware Sequence Recommendation. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE ’10). ACM, New York, NY, USA, 293–296. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Test generation for higher-order functions in dynamic languages

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!