Abstract
Test generation has proven to provide an effective way of identifying programming errors. Unfortunately, current test generation techniques are challenged by higher-order functions in dynamic languages, such as JavaScript functions that receive callbacks. In particular, existing test generators suffer from the unavailability of statically known type signatures, do not provide functions or provide only trivial functions as inputs, and ignore callbacks triggered by the code under test. This paper presents LambdaTester, a novel test generator that addresses the specific problems posed by higher-order functions in dynamic languages. The approach automatically infers at what argument position a method under test expects a callback, generates and iteratively improves callback functions given as input to this method, and uses novel test oracles that check whether and how callback functions are invoked. We apply LambdaTester to test 43 higher-order functions taken from 13 popular JavaScript libraries. The approach detects unexpected behavior in 12 of the 13 libraries, many of which are missed by a state-of-the-art test generator.
Supplemental Material
- Saba Alimadadi, Sheldon Sequeira, Ali Mesbah, and Karthik Pattabiraman. 2014. Understanding JavaScript event-based interactions. In 36th International Conference on Software Engineering, ICSE ’14, Hyderabad, India - May 31 - June 07, 2014. 367–377. Google Scholar
Digital Library
- Esben Andreasen, Liang Gong, Anders Møller, Michael Pradel, Marija Selakovic, Koushik Sen, and Cristian-Alexandru Staicu. 2017. A Survey of Dynamic Analysis and Test Generation for JavaScript. Comput. Surveys (2017). Google Scholar
Digital Library
- Andrea Arcuri and Lionel Briand. 2011. Adaptive Random Testing: An Illusion of Effectiveness?. In Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA ’11). ACM, New York, NY, USA, 265–275. Google Scholar
Digital Library
- Shay Artzi, Julian Dolby, Simon Holm Jensen, Anders Møller, and Frank Tip. 2011. A framework for automated testing of JavaScript web applications. In ICSE. 571–580. Google Scholar
Digital Library
- Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: automated testing based on Java predicates. In International Symposium on Software Testing and Analysis (ISSTA). 123–133. Google Scholar
Digital Library
- Jacob Burnim, Sudeep Juvekar, and Koushik Sen. 2009. WISE: Automated test generation for worst-case complexity. In ICSE. IEEE, 463–473. Google Scholar
Digital Library
- Cristian Cadar, Daniel Dunbar, and Dawson R. Engler. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In Symposium on Operating Systems Design and Implementation (OSDI). USENIX, 209–224. Google Scholar
Digital Library
- Tsong Yueh Chen, Fei-Ching Kuo, Robert G. Merkel, and T. H. Tse. 2010. Adaptive Random Testing: The ART of test case diversity. Journal of Systems and Software 83, 1 (2010), 60–66. Google Scholar
Digital Library
- Tsong Yueh Chen, Hing Leung, and IK Mak. 2004. Adaptive random testing. In Annual Asian Computing Science Conference. Springer, 320–329.Google Scholar
Digital Library
- Ravai Chugh, Jeffrey A. Meister, Ranjit Jhala, and Sorin Lerner. 2009. Staged Information Flow for JavaScript. In Conference on Programming Language Design and Implementation (PLDI). ACM, 50–62. Google Scholar
Digital Library
- Ilinca Ciupa, Andreas Leitner, Manuel Oriol, and Bertrand Meyer. 2008. ARTOO: adaptive random testing for object-oriented software. In International Conference on Software Engineering (ICSE). ACM, 71–80. Google Scholar
Digital Library
- I. Ciupa, A. Pretschner, M. Oriol, A. Leitner, and B. Meyer. 2011. On the number and nature of faults found by random testing. Software Testing, Verification and Reliability 21, 1 (2011), 3–28. Google Scholar
Digital Library
- Koen Claessen and John Hughes. 2011. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. SIGPLAN Not. 46, 4 (May 2011), 53–64. Google Scholar
Digital Library
- Christoph Csallner and Yannis Smaragdakis. 2004. JCrasher: an automatic robustness tester for Java. Software Prac. Experience 34, 11 (2004), 1025–1050. Google Scholar
Digital Library
- Joe W. Duran and Simeon C. Ntafos. 1984. An Evaluation of Random Testing. IEEE Trans. Softw. Eng. 10, 4 (July 1984), 438–444. Google Scholar
Digital Library
- Markus Ermuth and Michael Pradel. 2016. Monkey See, Monkey Do: Effective Generation of GUI Tests with Inferred Macro Events. In International Symposium on Software Testing and Analysis (ISSTA). 82–93. Google Scholar
Digital Library
- Asger Feldthaus and Anders Møller. 2014. Checking correctness of TypeScript interfaces for JavaScript libraries. In Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA). ACM, 1–16. Google Scholar
Digital Library
- Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In SIGSOFT-/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-19) and ESEC’11: 13th European Software Engineering Conference (ESEC-13), Szeged, Hungary, September 5-9, 2011. 416–419. Google Scholar
Digital Library
- Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing. SIGPLAN Not. 40, 6 (June 2005), 213–223. Google Scholar
Digital Library
- Patrice Godefroid, Michael Y. Levin, and David A. Molnar. 2008. Automated Whitebox Fuzz Testing. In Network and Distributed System Security Symposium (NDSS).Google Scholar
- Liang Gong, Michael Pradel, and Koushik Sen. 2015a. JITProf: Pinpointing JIT-unfriendly JavaScript Code. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 357–368. Google Scholar
Digital Library
- Liang Gong, Michael Pradel, Manu Sridharan, and Koushik Sen. 2015b. DLint: Dynamically Checking Bad Coding Practices in JavaScript. In International Symposium on Software Testing and Analysis (ISSTA). 94–105. Google Scholar
Digital Library
- Alex Groce, Gerard Holzmann, and Rajeev Joshi. 2007. Randomized Differential Testing As a Prelude to Formal Verification. In Proceedings of the 29th International Conference on Software Engineering (ICSE ’07). IEEE Computer Society, Washington, DC, USA, 621–631. Google Scholar
Digital Library
- Daniel Hedin, Arnar Birgisson, Luciano Bello, and Andrei Sabelfeld. 2014. JSFlow: tracking information flow in JavaScript and its APIs. In SAC. 1663–1671. Google Scholar
Digital Library
- Stefan Heule, Manu Sridharan, and Satish Chandra. 2015. Mimic: computing models for opaque code. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015. 710–720. Google Scholar
Digital Library
- Simon Holm Jensen, Anders Møller, and Peter Thiemann. 2009. Type Analysis for JavaScript. In Symposium on Static Analysis (SAS). Springer, 238–255. Google Scholar
Digital Library
- Simon Holm Jensen, Manu Sridharan, Koushik Sen, and Satish Chandra. 2015. MemInsight: platform-independent memory debugging for JavaScript. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015. 345–356. Google Scholar
Digital Library
- J. C. King. 1976. Symbolic Execution and Program Testing. Commun. ACM 19, 7 (1976), 385–394. Google Scholar
Digital Library
- Casey Klein, Matthew Flatt, and Robert Bruce Findler. 2010. Random testing for higher-order, stateful programs. In Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). ACM, 555–566. Google Scholar
Digital Library
- Pieter Koopman and Rinus Plasmeijer. 2006. Automatic Testing of Higher Order Functions. In Asian Symposium on Programming Languages and Systems (APLAS). 148–164. Google Scholar
Digital Library
- Erik Krogh Kristensen and Anders Møller. 2017. Type test scripts for TypeScript testing. PACMPL 1, OOPSLA (2017), 90:1–90:25. Google Scholar
Digital Library
- Magnus Madsen, Benjamin Livshits, and Michael Fanning. 2013. Practical static analysis of JavaScript applications in the presence of frameworks and libraries. In ESEC/SIGSOFT FSE. 499–509. Google Scholar
Digital Library
- Magnus Madsen, Frank Tip, and Ondrej Lhoták. 2015. Static analysis of event-driven Node.js JavaScript applications. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015. 505–519. Google Scholar
Digital Library
- Alessandro Marchetto, Paolo Tonella, and Filippo Ricca. 2008. State-Based Testing of Ajax Web Applications. In ICST. IEEE Computer Society, 121–130. Google Scholar
Digital Library
- William M. McKeeman. 1998. Differential Testing for Software. Digital Technical Journal 10, 1 (1998), 100–107.Google Scholar
- Atif M. Memon. 2007. An event-flow model of GUI-based applications for testing. Softw. Test., Verif. Reliab. (2007), 137–157. Google Scholar
Digital Library
- Ali Mesbah, Engin Bozdag, and Arie van Deursen. 2008. Crawling Ajax by Inferring User Interface State Changes. In International Conference on Web Engineering (ICWE). 122–134. Google Scholar
Digital Library
- Ali Mesbah and Arie van Deursen. 2009. Invariant-based automatic testing of Ajax user interfaces. In ICSE. 210–220. Google Scholar
Digital Library
- Erdal Mutlu, Serdar Tasiran, and Benjamin Livshits. 2015. Detecting JavaScript Races that Matter. In European Software Engineering Conference and International Symposium on Foundations of Software Engineering (ESEC/FSE). Google Scholar
Digital Library
- Phuc C. Nguyen and David Van Horn. 2015. Relatively complete counterexamples for higher-order programs. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, Portland, OR, USA, June 15-17, 2015. 446–456. Google Scholar
Digital Library
- S. C. Ntafos. 2001. On comparisons of random, partition, and proportional partition testing. IEEE Transactions on Software Engineering 27, 10 (Oct 2001), 949–960. Google Scholar
Digital Library
- Carlos Pacheco and Michael D. Ernst. 2007. Randoop: Feedback-directed Random Testing for Java. In Companion to the 22Nd ACM SIGPLAN Conference on Object-oriented Programming Systems and Applications Companion (OOPSLA ’07). ACM, New York, NY, USA, 815–816. Google Scholar
Digital Library
- Carlos Pacheco, Shuvendu K. Lahiri, and Thomas Ball. 2008. Finding Errors in .Net with Feedback-directed Random Testing. In Proceedings of the 2008 International Symposium on Software Testing and Analysis (ISSTA ’08). ACM, New York, NY, USA, 87–96. Google Scholar
Digital Library
- Michal H. Palka, Koen Claessen, Alejandro Russo, and John Hughes. 2011. Testing an Optimising Compiler by Generating Random Lambda Terms. In Proceedings of the 6th International Workshop on Automation of Software Test (AST ’11). ACM, New York, NY, USA, 91–97. Google Scholar
Digital Library
- Jibesh Patra, Pooja N. Dixit, and Michael Pradel. 2018. ConflictJS: Finding and Understanding Conflicts Between JavaScript Libraries. In ICSE. Google Scholar
Digital Library
- Jacques A. Pienaar and Robert Hundt. 2013. JSWhiz: Static analysis for JavaScript memory leaks. In Proceedings of the 2013 IEEE/ACM International Symposium on Code Generation and Optimization, CGO 2013, Shenzhen, China, February 23-27, 2013. 11:1–11:11. Google Scholar
Digital Library
- Michael Pradel and Thomas R. Gross. 2012. Fully Automatic and Precise Detection of Thread Safety Violations. In Conference on Programming Language Design and Implementation (PLDI). 521–530. Google Scholar
Digital Library
- Michael Pradel, Parker Schuh, George Necula, and Koushik Sen. 2014. EventBreak: Analyzing the Responsiveness of User Interfaces through Performance-Guided Test Generation. In Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). 33–47. Google Scholar
Digital Library
- Michael Pradel, Parker Schuh, and Koushik Sen. 2015. TypeDevil: Dynamic Type Inconsistency Analysis for JavaScript. In International Conference on Software Engineering (ICSE). Google Scholar
Digital Library
- Veselin Raychev, Pavol Bielik, Martin T. Vechev, and Andreas Krause. 2016. Learning programs from noisy data. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20 - 22, 2016. 761–774. Google Scholar
Digital Library
- Malavika Samak and Murali Krishna Ramanathan. 2014. Multithreaded Test Synthesis for Deadlock Detection. In Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA). 473–489. Google Scholar
Digital Library
- Marija Selakovic, Thomas Glaser, and Michael Pradel. 2017. An Actionable Performance Profiler for Optimizing the Order of Evaluations. In International Symposium on Software Testing and Analysis (ISSTA). 170–180. Google Scholar
Digital Library
- Koushik Sen, Swaroop Kalasapur, Tasneem Brutch, and Simon Gibbs. 2013. Jalangi: A Selective Record-replay and Dynamic Analysis Framework for JavaScript. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013). ACM, New York, NY, USA, 488–498. Google Scholar
Digital Library
- Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A Concolic Unit Testing Engine for C. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-13). ACM, New York, NY, USA, 263–272. Google Scholar
Digital Library
- Manu Sridharan, Julian Dolby, Satish Chandra, Max Schäfer, and Frank Tip. 2012. Correlation Tracking for Points-To Analysis of JavaScript. In ECOOP 2012 - Object-Oriented Programming - 26th European Conference, Beijing, China, June 11-16, 2012. Proceedings. 435–458. Google Scholar
Digital Library
- Peter Thiemann. 2005. Towards a Type System for Analyzing JavaScript Programs. In European Symposium on Programming (ESOP). 408–422. Google Scholar
Digital Library
- Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. 2009. MSeqGen: Objectoriented unit-test generation via mining source code. In European Software Engineering Conference and International Symposium on Foundations of Software Engineering (ESEC/FSE). ACM, 193–202. Google Scholar
Digital Library
- Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Zhendong Su. 2011. Synthesizing method sequences for high-coverage testing.. In OOPSLA. 189–206. Google Scholar
Digital Library
- Willem Visser, Corina S. Pasareanu, and Sarfraz Khurshid. 2004. Test input generation with Java PathFinder. In International Symposium on Software Testing and Analysis (ISSTA). ACM, 97–107. Google Scholar
Digital Library
- Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. 2005. Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution. In Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS). Springer, 365–381. Google Scholar
Digital Library
- Wujie Zheng, Qirun Zhang, Michael Lyu, and Tao Xie. 2010. Random Unit-test Generation with MUT-aware Sequence Recommendation. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE ’10). ACM, New York, NY, USA, 293–296. Google Scholar
Digital Library
Index Terms
Test generation for higher-order functions in dynamic languages
Recommendations
Writing Tests for This Higher-Order Function First: Automatically Identifying Future Callings to Assist Testers
Internetware '19: Proceedings of the 11th Asia-Pacific Symposium on InternetwareIn functional programming languages, such as Scala and Haskell, a higher-order function is a function that takes one or more functions as parameters or returns a function. Using higher-order functions in programs can increase the generality and reduce ...
Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection
TAIC-PART '09: Proceedings of the 2009 Testing: Academic and Industrial Conference - Practice and Research TechniquesWhere no independent specification is available, object-oriented unit testing is limited to exercising all interleaved method paths, seeking unexpected failures.A recent trend in unit testing, that interleaves dynamic analysis between each test cycle, ...
Automated conformance testing for JavaScript engines via deep compiler fuzzing
PLDI 2021: Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and ImplementationJavaScript (JS) is a popular, platform-independent programming language. To ensure the interoperability of JS programs across different platforms, the implementation of a JS engine should conform to the ECMAScript standard. However, doing so is ...






Comments