Abstract
Web applications, such as collaborative editors that allow multiple clients to concurrently interact on a shared resource, are difficult to implement correctly. Existing techniques for analyzing concurrent software do not scale to such complex systems or do not consider multiple interacting clients. This paper presents Simian, the first fully automated technique for systematically analyzing multi-client web applications.
Naively exploring all possible interactions between a set of clients of such applications is practically infeasible. Simian obtains scalability for real-world applications by using a two-phase black-box approach. The application code remains unknown to the analysis and is first explored systematically using a single client to infer potential conflicts between client events triggered in a specific context. The second phase synthesizes multi-client interactions targeted at triggering misbehavior that may result from the potential conflicts, and reports an inconsistency if the clients do not converge to a consistent state.
We evaluate the analysis on three widely used systems, Google Docs, Firepad, and ownCloud Documents, where it reports a variety of inconsistencies, such as incorrect formatting and misplaced text fragments. Moreover, we find that the two-phase approach runs 10x faster compared to exhaustive exploration, making systematic analysis practically applicable.
Supplemental Material
Available for Download
This artifact contains a virtual machine with the source code and running instructions for the Simian application of our paper Systematic Black-Box Analysis of Collaborative Web Applications. The artifact also contains an installation of test software that was used in the evaluation, where it was available for download. This allows you to run your own experiments.
- C. Q. Adamsen, G. Mezzetti, and A. Møller. Systematic execution of Android test suites in adverse conditions. In nternational Symposium on Software Testing and Analysis (ISSTA), pages 83–93, 2015. Google Scholar
Digital Library
- C. Artho, K. Havelund, and A. Biere. High-level data races. Software Testing, Verification and Reliability, 13(4):207–227, 2003.Google Scholar
Cross Ref
- S. Artzi, J. Dolby, S. H. Jensen, A. Møller, and F. Tip. A framework for automated testing of JavaScript web applications. In International Conference on Software Engineering (ICSE), pages 571–580, 2011. Google Scholar
Digital Library
- H. Attiya, S. Burckhardt, A. Gotsman, A. Morrison, H. Yang, and M. Zawirski. Specification and complexity of collaborative text editing. In Symposium on Principles of Distributed Computing (PODC), pages 259–268, 2016. Google Scholar
Digital Library
- M. D. Bond, K. E. Coons, and K. S. McKinley. PACER: proportional detection of data races. In Conference on Programming Language Design and Implementation (PLDI), pages 255–268, 2010. Google Scholar
Digital Library
- P. A. Brooks and A. M. Memon. Automated GUI testing guided by usage profiles. In International Conference on Automated Software Engineering (ASE), pages 333–342, 2007. Google Scholar
Digital Library
- L. Brutschy, D. Dimitrov, P. Müller, and M. Vechev. Serializability for eventual consistency: criterion, analysis, and applications. In Symposium on Principles of Programming Languages (POPL), pages 458–472, 2017. Google Scholar
Digital Library
- S. Burckhardt, P. Kothari, M. Musuvathi, and S. Nagarakatte. A randomized scheduler with probabilistic guarantees of finding bugs. In Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 167–178, 2010. Google Scholar
Digital Library
- W. Choi, G. Necula, and K. Sen. Guided GUI testing of Android apps with minimal restart and approximate learning. In Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), pages 623–640, 2013. Google Scholar
Digital Library
- A. Choudhary, S. Lu, and M. Pradel. Efficient detection of thread safety violations via coverage-guided generation of concurrent tests. In International Conference on Software Engineering (ICSE), 2017. Google Scholar
Digital Library
- K. E. Coons, S. Burckhardt, and M. Musuvathi. GAMBIT: effective unit testing for concurrency libraries. In Symposium on Principles and Practice of Parallel Programming, (PPOPP), pages 15–24, 2010. Google Scholar
Digital Library
- L. Effinger-Dean, B. Lucia, L. Ceze, D. Grossman, and H.-J. Boehm. IFRit: Interference-free regions for dynamic data-race detection. In Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), pages 467–484, 2012. Google Scholar
Digital Library
- C. A. Ellis and S. J. Gibbs. Concurrency control in groupware systems. In International Conference on Management of Data (MOD), pages 399–407, 1989. Google Scholar
Digital Library
- M. Ermuth and M. Pradel. Monkey see, monkey do: Effective generation of GUI tests with inferred macro events. In International Symposium on Software Testing and Analysis (ISSTA), pages 82–93, 2016. Google Scholar
Digital Library
- C. Flanagan and S. N. Freund. Atomizer: a dynamic atomicity checker for multithreaded programs. In Symposium on Principles of Programming Languages (POPL), pages 256–267, 2004. Google Scholar
Digital Library
- C. Flanagan and P. Godefroid. Dynamic partial-order reduction for model checking software. In Symposium on Principles of Programming Languages (POPL), pages 110–121, 2005. Google Scholar
Digital Library
- C. Flanagan and S. Qadeer. A type and effect system for atomicity. In Conference on Programming Language Design and Implementation (PLDI), pages 338–349, 2003. Google Scholar
Digital Library
- C. Hsiao, C. Pereira, J. Yu, G. Pokam, S. Narayanasamy, P. M. Chen, Z. Kong, and J. Flinn. Race detection for eventdriven mobile applications. In Conference on Programming Language Design and Implementation (PLDI), pages 326– 336, 2014. Google Scholar
Digital Library
- C. S. Jensen, M. R. Prasad, and A. Møller. Automated testing with targeted event sequence generation. In International Symposium on Software Testing and Analysis (ISSTA), pages 67–77, 2013. Google Scholar
Digital Library
- C. S. Jensen, A. Møller, V. Raychev, and M. Vechev. Stateless model checking of event-driven applications. In Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA), pages 57–73, 2015. Google Scholar
Digital Library
- S. Lu, S. Park, E. Seo, and Y. Zhou. Learning from mistakes: a comprehensive study on real world concurrency bug characteristics. In Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 329–339, 2008. Google Scholar
Digital Library
- P. Maiya, A. Kanade, and R. Majumdar. Race detection for Android applications. In Conference on Programming Language Design and Implementation (PLDI), pages 316– 325, 2014. Google Scholar
Digital Library
- A. Marchetto, P. Tonella, and F. Ricca. State-based testing of Ajax web applications. In International Conference on Software Testing, Verification, and Validation (ICST), pages 121–130, 2008. Google Scholar
Digital Library
- D. Marino, M. Musuvathi, and S. Narayanasamy. LiteRace: effective sampling for lightweight data-race detection. In Conference on Programming Language Design and Implementation (PLDI), pages 134–143, 2009. Google Scholar
Digital Library
- A. M. Memon. An event-flow model of GUI-based applications for testing. Software Testing, Verification and Reliability, 17(3):137–157, 2007. Google Scholar
Digital Library
- M. Musuvathi, S. Qadeer, T. Ball, G. Basler, P. A. Nainar, and I. Neamtiu. Finding and reproducing Heisenbugs in concurrent programs. In Symposium on Operating Systems Design and Implementation (OSDI), pages 267–280, 2008. Google Scholar
Digital Library
- E. Mutlu, S. Tasiran, and B. Livshits. Detecting JavaScript races that matter. In European Software Engineering Conference and International Symposium on Foundations of Software Engineering (ESEC/FSE), 2015. Google Scholar
Digital Library
- M. Naik, C.-S. Park, K. Sen, and D. Gay. Effective static deadlock detection. In International Conference on Software Engineering (ICSE), pages 386–396, 2009. Google Scholar
Digital Library
- B. Petrov, M. Vechev, M. Sridharan, and J. Dolby. Race detection for web applications. In Conference on Programming Language Design and Implementation (PLDI), pages 251– 262, 2012. Google Scholar
Digital Library
- M. Pradel and T. R. Gross. Fully automatic and precise detection of thread safety violations. In Conference on Programming Language Design and Implementation (PLDI), pages 521–530, 2012. Google Scholar
Digital Library
- M. Pradel, M. Huggler, and T. R. Gross. Performance regression testing of concurrent classes. In International Symposium on Software Testing and Analysis (ISSTA), pages 13–25, 2014. Google Scholar
Digital Library
- M. Pradel, P. Schuh, G. Necula, and K. Sen. EventBreak: Analyzing the responsiveness of user interfaces through performance-guided test generation. In Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), pages 33–47, 2014. Google Scholar
Digital Library
- V. Raychev, M. Vechev, and M. Sridharan. Effective race detection for event-driven programs. In Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), pages 151–166, 2013. Google Scholar
Digital Library
- M. Samak and M. K. Ramanathan. Multithreaded test synthesis for deadlock detection. In Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA), pages 473–489, 2014. Google Scholar
Digital Library
- M. Samak and M. K. Ramanathan. Synthesizing tests for detecting atomicity violations. In European Software Engineering Conference and International Symposium on Foundations of Software Engineering (ESEC/FSE), pages 131–142, 2015. Google Scholar
Digital Library
- M. Samak, M. K. Ramanathan, and S. Jagannathan. Synthesizing racy tests. In Conference on Programming Language Design and Implementation (PLDI), pages 175–185, 2015. Google Scholar
Digital Library
- M. Samak, O. Tripp, and M. K. Ramanathan. Directed synthesis of failing concurrent executions. In International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), pages 430–446, 2016. Google Scholar
Digital Library
- K. Sen. Effective random testing of concurrent programs. In International Conference on Automated Software Engineering (ASE), pages 323–332, 2007. Google Scholar
Digital Library
- K. Sen. Race directed random testing of concurrent programs. In Conference on Programming Language Design and Implementation (PLDI), pages 11–21, 2008. Google Scholar
Digital Library
- O. Shacham, N. Bronson, A. Aiken, M. Sagiv, M. Vechev, and E. Yahav. Testing atomicity of composed concurrent operations. In Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA), pages 51– 64, 2011. Google Scholar
Digital Library
- S. Tasharofi, M. Pradel, Y. Lin, and R. Johnson. Bita: Coverage-guided, automatic testing of actor programs. In Conference on Automated Software Engineering (ASE), 2013. Google Scholar
Digital Library
- V. Terragni and S.-C. Cheung. Coverage-driven test code generation for concurrent classes. In International Conference on Software Engineernig (ICSE), pages 1121–1132, 2016. Google Scholar
Digital Library
- S. Thummalapenta, K. V. Lakshmi, S. Sinha, N. Sinha, and S. Chandra. Guided test generation for web applications. In International Conference on Software Engineering (ICSE), pages 162–171, 2013. Google Scholar
Digital Library
- W. Visser, K. Havelund, G. P. Brat, S. Park, and F. Lerda. Model checking programs. International Conference on Automated Software Engineering (ASE), pages 203–232, 2003. Google Scholar
Digital Library
- W. Vogels. Eventually consistent. Communications of the ACM, 52(1):40–44, 2009. Google Scholar
Digital Library
- C. Wang, M. Said, and A. Gupta. Coverage guided systematic concurrency testing. In International Conference on Software Engineering (ICSE), pages 221–230, 2011. Google Scholar
Digital Library
- A. Williams, W. Thies, and M. D. Ernst. Static deadlock detection for Java libraries. In European Conference on Object-Oriented Programming (ECOOP), pages 602–629, 2005. Google Scholar
Digital Library
- T. Yu and M. Pradel. Syncprof: Detecting, localizing, and optimizing synchronization bottlenecks. In International Symposium on Software Testing and Analysis (ISSTA), pages 389– 400, 2016. Google Scholar
Digital Library
Index Terms
Systematic black-box analysis of collaborative web applications
Recommendations
Systematic black-box analysis of collaborative web applications
PLDI 2017: Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and ImplementationWeb applications, such as collaborative editors that allow multiple clients to concurrently interact on a shared resource, are difficult to implement correctly. Existing techniques for analyzing concurrent software do not scale to such complex systems ...
A black-box approach for web application SLA
SAC '06: Proceedings of the 2006 ACM symposium on Applied computingWeb servers nowadays have to cope with unprecedented amounts of workload, due to increasing popularity and complexity; in particular, dynamically generated content becomes the standard, hence the term Web application. Providing enough resources to ...
Practical AJAX race detection for JavaScript web applications
ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringAsynchronous client-server communication is a common source of errors in JavaScript web applications. Such errors are difficult to detect using ordinary testing because of the nondeterministic scheduling of AJAX events. Existing automated event race ...






Comments