Abstract
Modern applications, such as social networking systems and e-commerce platforms are centered around using large-scale storage systems for storing and retrieving data. In the presence of concurrent accesses, these storage systems trade off isolation for performance. The weaker the isolation level, the more behaviors a storage system is allowed to exhibit and it is up to the developer to ensure that their application can tolerate those behaviors. However, these weak behaviors only occur rarely in practice and outside the control of the application, making it difficult for developers to test the robustness of their code against weak isolation levels.
This paper presents MonkeyDB, a mock storage system for testing storage-backed applications. MonkeyDB supports a key-value interface as well as SQL queries under multiple isolation levels. It uses a logical specification of the isolation level to compute, on a read operation, the set of all possible return values. MonkeyDB then returns a value randomly from this set. We show that MonkeyDB provides good coverage of weak behaviors, which is complete in the limit. We test a variety of applications for assertions that fail only under weak isolation. MonkeyDB is able to break each of those assertions in a small number of attempts.
Supplemental Material
- Parosh Aziz Abdulla, Stavros Aronis, Mohamed Faouzi Atig, Bengt Jonsson, Carl Leonardsson, and Konstantinos Sagonas. 2015. Stateless Model Checking for TSO and PSO. In Tools and Algorithms for the Construction and Analysis of Systems - 21st International Conference, TACAS 2015, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2015, London, UK, April 11-18, 2015. Proceedings, Christel Baier and Cesare Tinelli (Eds.) (Lecture Notes in Computer Science, Vol. 9035). Springer, 353–367. https://doi.org/10.1007/978-3-662-46681-0_28 Google Scholar
Digital Library
- A. Adya. 1999. Weak Consistency: A Generalized Theory and Optimistic Implementations for Distributed Transactions. USA. Google Scholar
Digital Library
- Atul Adya, Barbara Liskov, and Patrick E. O’Neil. 2000. Generalized Isolation Level Definitions. In Proceedings of the 16th International Conference on Data Engineering, San Diego, California, USA, February 28 - March 3, 2000, David B. Lomet and Gerhard Weikum (Eds.). IEEE Computer Society, 67–78. https://doi.org/10.1109/ICDE.2000.839388 Google Scholar
Digital Library
- Deepthi Devaki Akkoorath and Annette Bieniusa. 2016. Antidote: the highly-available geo-replicated database with strongest guarantees. https://pages.lip6.fr/syncfree/attachments/article/59/antidote-white-paper.pdfGoogle Scholar
- Valter Balegas, Sérgio Duarte, Carla Ferreira, Rodrigo Rodrigues, Nuno M. Preguiça, Mahsa Najafzadeh, and Marc Shapiro. 2015. Putting consistency back into eventual consistency. In Proceedings of the Tenth European Conference on Computer Systems, EuroSys 2015, Bordeaux, France, April 21-24, 2015, Laurent Réveillère, Tim Harris, and Maurice Herlihy (Eds.). ACM, 6:1–6:16. https://doi.org/10.1145/2741948.2741972 Google Scholar
Digital Library
- Sidi Mohamed Beillahi, Ahmed Bouajjani, and Constantin Enea. 2019. Checking Robustness Against Snapshot Isolation. In Computer Aided Verification - 31st International Conference, CAV 2019, New York City, NY, USA, July 15-18, 2019, Proceedings, Part II, Isil Dillig and Serdar Tasiran (Eds.) (Lecture Notes in Computer Science, Vol. 11562). Springer, 286–304. https://doi.org/10.1007/978-3-030-25543-5_17 Google Scholar
- Sidi Mohamed Beillahi, Ahmed Bouajjani, and Constantin Enea. 2019. Robustness Against Transactional Causal Consistency. In 30th International Conference on Concurrency Theory, CONCUR 2019, August 27-30, 2019, Amsterdam, the Netherlands, Wan J. Fokkink and Rob van Glabbeek (Eds.) (LIPIcs, Vol. 140). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 30:1–30:18. https://doi.org/10.4230/LIPIcs.CONCUR.2019.30 Google Scholar
Cross Ref
- Hal Berenson, Philip A. Bernstein, Jim Gray, Jim Melton, Elizabeth J. O’Neil, and Patrick E. O’Neil. 1995. A Critique of ANSI SQL Isolation Levels. In Proceedings of the 1995 ACM SIGMOD International Conference on Management of Data, San Jose, California, USA, May 22-25, 1995, Michael J. Carey and Donovan A. Schneider (Eds.). ACM Press, 1–10. https://doi.org/10.1145/223784.223785 Google Scholar
Digital Library
- Giovanni Bernardi and Alexey Gotsman. 2016. Robustness against Consistency Models with Atomic Visibility. In 27th International Conference on Concurrency Theory, CONCUR 2016, August 23-26, 2016, Québec City, Canada, Josée Desharnais and Radha Jagadeesan (Eds.) (LIPIcs, Vol. 59). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 7:1–7:15. https://doi.org/10.4230/LIPIcs.CONCUR.2016.7 Google Scholar
- Ranadeep Biswas and Constantin Enea. 2019. On the complexity of checking transactional consistency. Proc. ACM Program. Lang., 3, OOPSLA (2019), 165:1–165:28. https://doi.org/10.1145/3360591 Google Scholar
Digital Library
- Ranadeep Biswas and Constantin Enea. 2019. On the Complexity of Checking Transactional Consistency. CoRR, abs/1908.04509 (2019), arxiv:1908.04509. arxiv:1908.04509Google Scholar
- Ranadeep Biswas, Diptanshu Kakwani, Jyothi Vedurada, Constantin Enea, and Akash Lal. 2021. MonkeyDB: Effectively Testing Correctness against Weak Isolation Levels. CoRR, abs/2103.02830 (2021), arXiv:2103.02830. arxiv:2103.02830Google Scholar
- Ranadeep Biswas, Diptanshu Kakwani, Jyothi Vedurada, Constantin Enea, and Akash Lal. 2021. MonkeyDB: Effectively Testing Correctness under Weak Isolation Levels. https://doi.org/10.5281/zenodo.5504052 Google Scholar
Cross Ref
- Nathan Bronson, Zach Amsden, George Cabrera, Prasad Chakka, Peter Dimov, Hui Ding, Jack Ferris, Anthony Giardullo, Sachin Kulkarni, Harry C. Li, Mark Marchukov, Dmitri Petrov, Lovro Puzar, Yee Jiun Song, and Venkateshwaran Venkataramani. 2013. TAO: Facebook’s Distributed Data Store for the Social Graph. In 2013 USENIX Annual Technical Conference, San Jose, CA, USA, June 26-28, 2013, Andrew Birrell and Emin Gün Sirer (Eds.). USENIX Association, 49–60. https://www.usenix.org/conference/atc13/technical-sessions/presentation/bronson Google Scholar
Digital Library
- Lucas Brutschy, Dimitar I. Dimitrov, Peter Müller, and Martin T. Vechev. 2017. Serializability for eventual consistency: criterion, analysis, and applications. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages, POPL 2017, Paris, France, January 18-20, 2017, Giuseppe Castagna and Andrew D. Gordon (Eds.). ACM, 458–472. http://dl.acm.org/citation.cfm?id=3009895 Google Scholar
Digital Library
- Lucas Brutschy, Dimitar I. Dimitrov, Peter Müller, and Martin T. Vechev. 2018. Static serializability analysis for causal consistency. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018, Jeffrey S. Foster and Dan Grossman (Eds.). ACM, 90–104. https://doi.org/10.1145/3192366.3192415 Google Scholar
Digital Library
- Andrea Cerone, Giovanni Bernardi, and Alexey Gotsman. 2015. A Framework for Transactional Consistency Models with Atomic Visibility. In 26th International Conference on Concurrency Theory, CONCUR 2015, Madrid, Spain, September 1.4, 2015, Luca Aceto and David de Frutos-Escrig (Eds.) (LIPIcs, Vol. 42). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 58–71. https://doi.org/10.4230/LIPIcs.CONCUR.2015.58 Google Scholar
- Andrea Cerone and Alexey Gotsman. 2018. Analysing Snapshot Isolation. J. ACM, 65, 2 (2018), 11:1–11:41. https://doi.org/10.1145/3152396 Google Scholar
Digital Library
- Transaction Processing Performance Council. 2020. TPC-C Benchmark Specification. http://tpc.org/tpc_documents_current_versions/pdf/tpc-c_v5.11.0.pdfGoogle Scholar
- Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007. Dynamo: amazon’s highly available key-value store. In Proceedings of the 21st ACM Symposium on Operating Systems Principles 2007, SOSP 2007, Stevenson, Washington, USA, October 14-17, 2007, Thomas C. Bressoud and M. Frans Kaashoek (Eds.). ACM, 205–220. https://doi.org/10.1145/1294261.1294281 Google Scholar
Digital Library
- Djellel Eddine Difallah, Andrew Pavlo, Carlo Curino, and Philippe Cudré-Mauroux. 2013. OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases. Proc. VLDB Endow., 7, 4 (2013), 277–288. https://doi.org/10.14778/2732240.2732246 Google Scholar
Digital Library
- Alan D. Fekete, Dimitrios Liarokapis, Elizabeth J. O’Neil, Patrick E. O’Neil, and Dennis E. Shasha. 2005. Making snapshot isolation serializable. ACM Trans. Database Syst., 30, 2 (2005), 492–528. https://doi.org/10.1145/1071610.1071615 Google Scholar
Digital Library
- MariaDB Foundation. 2020. MariaDB: An Open Source Relational Database. https://mariadb.org/Google Scholar
- Galera. 2020. Galera Cluster for MySQL. https://galeracluster.com/Google Scholar
- Yifan Gan, Xueyuan Ren, Drew Ripberger, Spyros Blanas, and Yang Wang. 2020. IsoDiff: Debugging Anomalies Caused by Weak Isolation. Proc. VLDB Endow., 13, 12 (2020), July, 27732786. issn:2150-8097 https://doi.org/10.14778/3407790.3407860 Google Scholar
Digital Library
- Alexey Gotsman, Hongseok Yang, Carla Ferreira, Mahsa Najafzadeh, and Marc Shapiro. 2016. ’Cause I’m strong enough: reasoning about consistency choices in distributed systems. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20 - 22, 2016, Rastislav Bodík and Rupak Majumdar (Eds.). ACM, 371–384. https://doi.org/10.1145/2837614.2837625 Google Scholar
Digital Library
- Java Platform. [n. d.]. JDBC: Java Database Connectivity API. https://en.wikipedia.org/wiki/Java_Database_ConnectivityGoogle Scholar
- Jepsen. 2020. Distributed Systems Testing. https://jepsen.io/Google Scholar
- Jepsen Nemesis. 2021. Tutorial on Jepsen nemesis variants.. https://jepsen-io.github.io/jepsen/jepsen.nemesis.htmlGoogle Scholar
- Sudhir Jorwekar, Alan D. Fekete, Krithi Ramamritham, and S. Sudarshan. 2007. Automating the Detection of Snapshot Isolation Anomalies. In Proceedings of the 33rd International Conference on Very Large Data Bases, University of Vienna, Austria, September 23-27, 2007, Christoph Koch, Johannes Gehrke, Minos N. Garofalakis, Divesh Srivastava, Karl Aberer, Anand Deshpande, Daniela Florescu, Chee Yong Chan, Venkatesh Ganti, Carl-Christian Kanne, Wolfgang Klas, and Erich J. Neuhold (Eds.). ACM, 1263–1274. http://www.vldb.org/conf/2007/papers/industrial/p1263-jorwekar.pdf Google Scholar
Digital Library
- Gowtham Kaki, Kapil Earanky, K. C. Sivaramakrishnan, and Suresh Jagannathan. 2018. Safe replication through bounded concurrency verification. Proc. ACM Program. Lang., 2, OOPSLA (2018), 164:1–164:27. https://doi.org/10.1145/3276534 Google Scholar
Digital Library
- Michalis Kokologiannakis, Ori Lahav, Konstantinos Sagonas, and Viktor Vafeiadis. 2018. Effective stateless model checking for C/C++ concurrency. Proc. ACM Program. Lang., 2, POPL (2018), 17:1–17:32. https://doi.org/10.1145/3158105 Google Scholar
Digital Library
- Leslie Lamport. 1978. Time, Clocks, and the Ordering of Events in a Distributed System. Commun. ACM, 21, 7 (1978), 558–565. https://doi.org/10.1145/359545.359563 Google Scholar
Digital Library
- Cheng Li, João Leitão, Allen Clement, Nuno M. Preguiça, Rodrigo Rodrigues, and Viktor Vafeiadis. 2014. Automating the Choice of Consistency Levels in Replicated Systems. In 2014 USENIX Annual Technical Conference, USENIX ATC ’14, Philadelphia, PA, USA, June 19-20, 2014, Garth Gibson and Nickolai Zeldovich (Eds.). USENIX Association, 281–292. https://www.usenix.org/conference/atc14/technical-sessions/presentation/li_cheng_2 Google Scholar
Digital Library
- Christopher Lidbury and Alastair F. Donaldson. 2017. Dynamic race detection for C++11. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages, POPL 2017, Paris, France, January 18-20, 2017, Giuseppe Castagna and Andrew D. Gordon (Eds.). ACM, 443–457. https://doi.org/10.1145/3009837.3009857 Google Scholar
Digital Library
- Wyatt Lloyd, Michael J. Freedman, Michael Kaminsky, and David G. Andersen. 2011. Don’t settle for eventual: scalable causal consistency for wide-area storage with COPS. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles 2011, SOSP 2011, Cascais, Portugal, October 23-26, 2011, Ted Wobber and Peter Druschel (Eds.). ACM, 401–416. https://doi.org/10.1145/2043556.2043593 Google Scholar
Digital Library
- Haonan Lu, Kaushik Veeraraghavan, Philippe Ajoux, Jim Hunt, Yee Jiun Song, Wendy Tobagus, Sanjeev Kumar, and Wyatt Lloyd. 2015. Existential Consistency: Measuring and Understanding Consistency at Facebook. In Proceedings of the 25th Symposium on Operating Systems Principles (SOSP ’15). Association for Computing Machinery, New York, NY, USA. 295310. isbn:9781450338349 https://doi.org/10.1145/2815400.2815426 Google Scholar
Digital Library
- Microsoft. 2020. Azure Cosmos DB Local Emulator. https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulatorGoogle Scholar
- Microsoft Coyote. 2019. Fearless coding for reliable asynchronous software. https://github.com/microsoft/coyoteGoogle Scholar
- Madanlal Musuvathi and Shaz Qadeer. 2008. Fair stateless model checking. In Proceedings of the ACM SIGPLAN 2008 Conference on Programming Language Design and Implementation, Tucson, AZ, USA, June 7-13, 2008, Rajiv Gupta and Saman P. Amarasinghe (Eds.). ACM, 362–371. https://doi.org/10.1145/1375581.1375625 Google Scholar
Digital Library
- Kartik Nagar and Suresh Jagannathan. 2018. Automated Detection of Serializability Violations Under Weak Consistency. In 29th International Conference on Concurrency Theory, CONCUR 2018, September 4-7, 2018, Beijing, China, Sven Schewe and Lijun Zhang (Eds.) (LIPIcs, Vol. 118). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 41:1–41:18. https://doi.org/10.4230/LIPIcs.CONCUR.2018.41 Google Scholar
- Kartik Nagar, Prasita Mukherjee, and Suresh Jagannathan. 2020. Semantics, Specification, and Bounded Verification of Concurrent Libraries in Replicated Systems. In Computer Aided Verification - 32nd International Conference, CAV 2020, Los Angeles, CA, USA, July 21-24, 2020, Proceedings, Part I, Shuvendu K. Lahiri and Chao Wang (Eds.) (Lecture Notes in Computer Science, Vol. 12224). Springer, 251–274. https://doi.org/10.1007/978-3-030-53288-8_13 Google Scholar
Digital Library
- Sreeja S. Nair, Gustavo Petri, and Marc Shapiro. 2020. Proving the Safety of Highly-Available Distributed Objects. In Programming Languages and Systems - 29th European Symposium on Programming, ESOP 2020, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2020, Dublin, Ireland, April 25-30, 2020, Proceedings, Peter Müller (Ed.) (Lecture Notes in Computer Science, Vol. 12075). Springer, 544–571. https://doi.org/10.1007/978-3-030-44914-8_20 Google Scholar
Digital Library
- Brian Norris and Brian Demsky. 2013. CDSchecker: checking concurrent data structures written with C/C++ atomics. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA 2013, part of SPLASH 2013, Indianapolis, IN, USA, October 26-31, 2013, Antony L. Hosking, Patrick Th. Eugster, and Cristina V. Lopes (Eds.). ACM, 131–150. https://doi.org/10.1145/2509136.2509514 Google Scholar
Digital Library
- Burcu Kulahcioglu Ozkan. 2020. Verifying Weakly Consistent Transactional Programs Using Symbolic Execution. In Networked Systems - 8th International Conference, NETYS 2020, Marrakech, Morocco, June 3-5, 2020, Proceedings, Chryssis Georgiou and Rupak Majumdar (Eds.) (Lecture Notes in Computer Science, Vol. 12129). Springer, 261–278. https://doi.org/10.1007/978-3-030-67087-0_17 Google Scholar
Digital Library
- Christos H. Papadimitriou. 1979. The serializability of concurrent database updates. J. ACM, 26, 4 (1979), 631–653. https://doi.org/10.1145/322154.322158 Google Scholar
Digital Library
- Andrew Pavlo. 2017. What Are We Doing With Our Lives? Nobody Cares About Our Concurrency Control Research. In Proceedings of the 2017 ACM International Conference on Management of Data (SIGMOD ’17). Association for Computing Machinery, New York, NY, USA. 3. isbn:9781450341974 https://doi.org/10.1145/3035918.3056096 Google Scholar
Digital Library
- Jos Rolando Guay Paz. 2018. Microsoft Azure Cosmos DB Revealed: A Multi-Modal Database Designed for the Cloud (1st ed.). Apress, USA. isbn:1484233506 Google Scholar
Digital Library
- Kia Rahmani. 2018. Lifting Jepsen Tests to the Application Level. https://kiarahmani.github.io/posts/2018/12/jepsen/Google Scholar
- Kia Rahmani, Kartik Nagar, Benjamin Delaware, and Suresh Jagannathan. 2019. CLOTHO: Directed Test Generation for Weakly Consistent Database Systems. Proc. ACM Program. Lang., 3, OOPSLA (2019), Article 117, Oct., 28 pages. https://doi.org/10.1145/3360543 Google Scholar
Digital Library
- K. C. Sivaramakrishnan, Gowtham Kaki, and Suresh Jagannathan. 2015. Declarative programming over eventually consistent data stores. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, Portland, OR, USA, June 15-17, 2015, David Grove and Stephen M. Blackburn (Eds.). ACM, 413–424. https://doi.org/10.1145/2737924.2737981 Google Scholar
Digital Library
- Twissandra. Accessed November 1, 2020. Twitter clone on Cassandra. https://github.com/twissandra/twissandraGoogle Scholar
- Todd Warszawski and Peter Bailis. 2017. ACIDRain: Concurrency-Related Attacks on Database-Backed Web Applications. In Proceedings of the 2017 ACM International Conference on Management of Data (SIGMOD ’17). Association for Computing Machinery, New York, NY, USA. 520. isbn:9781450341974 https://doi.org/10.1145/3035918.3064037 Google Scholar
Digital Library
- ANSI X3. 1992. 135-1992. American National Standard for Information Systems-Database Language-SQL.Google Scholar
Index Terms
MonkeyDB: effectively testing correctness under weak isolation levels
Recommendations
On the complexity of checking transactional consistency
Transactions simplify concurrent programming by enabling computations on shared data that are isolated from other concurrent computations and are resilient to failures. Modern databases provide different consistency models for transactions corresponding ...
Dynamic Partial Order Reduction for Checking Correctness against Transaction Isolation Levels
Modern applications, such as social networking systems and e-commerce platforms are centered around using large-scale databases for storing and retrieving data. Accesses to the database are typically enclosed in transactions that allow computations on ...
Weak Mutation Testing and Completeness of Test Sets
Different approaches to the generation of test data are described. Error-based approaches depend on the definition of classes of commonly occurring program errors. They generate tests which are specifically designed to determine if particular classes of ...






Comments