Abstract
Operation-based Conflict-free Replicated Data Types (op-based CRDTs) are a family of distributed data structures where all operations are designed to commute, so that replica states eventually converge. Additionally, op-based CRDTs require that operations be propagated between replicas in causal order. This paper presents a framework for verifying safety properties of CRDT implementations using separation logic. The framework consists of two libraries. One implements a Reliable Causal Broadcast (RCB) protocol so that replicas can exchange messages in causal order. A second “OpLib” library then uses RCB to simplify the creation and correctness proofs of op-based CRDTs. OpLib allows clients to implement new CRDTs as purely-functional data structures, without having to reason about network operations, concurrency control and mutable state, and without having to each time re-implement causal broadcast. Using OpLib, we have implemented 12 example CRDTs from the literature, including multiple versions of replicated registers and sets, two CRDT combinators for products and maps, and two example use cases of the map combinator. Our proofs are conducted in the Aneris distributed separation logic and are formalized in Coq. Our technique is the first work on verification of op-based CRDTs that satisfies both of the following properties: it is modular and targets executable implementations, as opposed to high-level protocols.
- Mustaque Ahamad, Gil Neiger, James E. Burns, Prince Kohli, and Phillip W. Hutto. 1995. Causal Memory: Definitions, Implementation, and Programming. Distributed Comput., 9, 1 (1995), 37–49. https://doi.org/10.1007/BF01784241
Google Scholar
Digital Library
- Peter Bailis, Ali Ghodsi, Joseph M. Hellerstein, and Ion Stoica. 2013. Bolt-on causal consistency. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2013, New York, NY, USA, June 22-27, 2013. 761–772. https://doi.org/10.1145/2463676.2465279
Google Scholar
Digital Library
- Carlos Baquero, Paulo Sérgio Almeida, and Ali Shoker. 2014. Making operation-based CRDTs operation-based. In Proceedings of the First Workshop on the Principles and Practice of Eventual Consistency, [email protected] 2014, April 13, 2014, Amsterdam, The Netherlands, Marc Shapiro (Ed.). ACM, 7:1–7:2. https://doi.org/10.1145/2596631.2596632
Google Scholar
Digital Library
- Lars Birkedal and Aleš Bizjak. 2017. Lecture Notes on Iris: Higher-Order Concurrent Separation Log. http://iris-project.org/tutorial-pdfs/iris-lecture-notes.pdf
Google Scholar
- Kenneth Birman, Andre Schiper, and Pat Stephenson. 1991. Lightweight Causal and Atomic Group Multicast. ACM Transactions on Computer Systems (TOCS), 9, 3 (1991), 272–314. https://doi.org/10.1145/128738.128742
Google Scholar
Digital Library
- Sebastian Burckhardt, Alexey Gotsman, Hongseok Yang, and Marek Zawirski. 2014. Replicated Data Types: Specification, Verification, Optimality. In 41st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL 2014). ACM, 271–284. https://doi.org/10.1145/2535838.2535848
Google Scholar
Digital Library
- Christian Cachin, Rachid Guerraoui, and Luís Rodrigues. 2011. Introduction to Reliable and Secure Distributed Programming. Springer Science & Business Media.
Google Scholar
- Tej Chajed, Joseph Tassarotti, M. Frans Kaashoek, and Nickolai Zeldovich. 2019. Verifying concurrent, crash-safe systems with Perennial. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, SOSP 2019, Huntsville, ON, Canada, October 27-30, 2019, Tim Brecht and Carey Williamson (Eds.). ACM, 243–258. https://doi.org/10.1145/3341301.3359632
Google Scholar
Digital Library
- Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Michael Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber. 2008. Bigtable: A Distributed Storage System for Structured Data. ACM Trans. Comput. Syst., 26, 2 (2008), 4:1–4:26. https://doi.org/10.1145/1365815.1365816
Google Scholar
Digital Library
- Kristina Chodorow and Michael Dirolf. 2010. MongoDB - The Definitive Guide: Powerful and Scalable Data Storage. O’Reilly.
Google Scholar
- Colin J Fidge. 1987. Timestamps in Message-Passing Systems That Preserve the Partial Ordering.
Google Scholar
- Seth Gilbert and Nancy A. Lynch. 2002. Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News, 33, 2 (2002), 51–59. https://doi.org/10.1145/564585.564601
Google Scholar
Digital Library
- Victor B. F. Gomes, Martin Kleppmann, Dominic P. Mulligan, and Alastair R. Beresford. 2017. Verifying Strong Eventual Consistency in Distributed Systems. Proc. ACM Program. Lang., 1, OOPSLA (2017), 109:1–109:28. https://doi.org/10.1145/3133933
Google Scholar
Digital Library
- Léon Gondelman, Simon Oddershede Gregersen, Abel Nieto, Amin Timany, and Lars Birkedal. 2021. Distributed Causal Memory: Modular Specification and Verification in Higher-Order Distributed Separation Logic. Proc. ACM Program. Lang., 5, POPL (2021), 1–29. https://doi.org/10.1145/3434323
Google Scholar
Digital Library
- Maurice Herlihy and Jeannette M. Wing. 1990. Linearizability: A Correctness Condition for Concurrent Objects. ACM Trans. Program. Lang. Syst., 12, 3 (1990), 463–492. https://doi.org/10.1145/78969.78972
Google Scholar
Digital Library
- Ralf Jung, Robbert Krebbers, Jacques-Henri Jourdan, Ales Bizjak, Lars Birkedal, and Derek Dreyer. 2018. Iris from the ground up: A modular foundation for higher-order concurrent separation logic. J. Funct. Program., 28 (2018), e20. https://doi.org/10.1017/S0956796818000151
Google Scholar
Cross Ref
- Ralf Jung, David Swasey, Filip Sieczkowski, Kasper Svendsen, Aaron Turon, Lars Birkedal, and Derek Dreyer. 2015. Iris: Monoids and Invariants as an Orthogonal Basis for Concurrent Reasoning. In Proceedings of the 42nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2015, Mumbai, India, January 15-17, 2015. 637–650. https://doi.org/10.1145/2676726.2676980
Google Scholar
Digital Library
- Morten Krogh-Jespersen, Amin Timany, Marit Edna Ohlenbusch, Simon Oddershede Gregersen, and Lars Birkedal. 2020. Aneris: A Mechanised Logic for Modular Reasoning about Distributed Systems. In Programming Languages and Systems - 29th European Symposium on Programming, ESOP 2020, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2020, Dublin, Ireland, April 25-30, 2020, Proceedings. 336–365. https://doi.org/10.1007/978-3-030-44914-8_13
Google Scholar
Digital Library
- Leslie Lamport. 1978. Time, Clocks, and the Ordering of Events in a Distributed System. Commun. ACM, 21, 7 (1978), 558–565. https://doi.org/10.1145/359545.359563
Google Scholar
Digital Library
- Adriaan Leijnse, Paulo Sérgio Almeida, and Carlos Baquero. 2019. Higher-Order Patterns in Replicated Data Types. In [email protected]. ACM, 5:1–5:6. https://doi.org/10.1145/3301419.3323971
Google Scholar
Digital Library
- Hongjin Liang and Xinyu Feng. 2021. Abstraction for Conflict-Free Replicated Data Types. In PLDI ’21: 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, Virtual Event, Canada, June 20-25, 20211, Stephen N. Freund and Eran Yahav (Eds.). ACM, 636–650. https://doi.org/10.1145/3453483.3454067
Google Scholar
Digital Library
- Yiyun Liu, James Parker, Patrick Redmond, Lindsey Kuper, Michael Hicks, and Niki Vazou. 2020. Verifying Replicated Data Types with Typeclass Refinements in Liquid Haskell. Proc. ACM Program. Lang., 4, OOPSLA (2020), 216:1–216:30. https://doi.org/10.1145/3428284
Google Scholar
Digital Library
- Wyatt Lloyd, Michael J. Freedman, Michael Kaminsky, and David G. Andersen. 2011. Don’t settle for eventual: scalable causal consistency for wide-area storage with COPS. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles 2011, SOSP 2011, Cascais, Portugal, October 23-26, 2011. 401–416. https://doi.org/10.1145/2043556.2043593
Google Scholar
Digital Library
- Friedemann Mattern. 1988. Virtual Time and Global States of Distributed Systems. Univ., Department of Computer Science.
Google Scholar
- Kartik Nagar and Suresh Jagannathan. 2019. Automated Parameterized Verification of CRDTs. In CAV (2) (Lecture Notes in Computer Science, Vol. 11562). Springer, 459–477.
Google Scholar
- Sreeja S. Nair, Gustavo Petri, and Marc Shapiro. 2020. Proving the Safety of Highly-Available Distributed Objects. In Programming Languages and Systems - 29th European Symposium on Programming, ESOP 2020, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2020, Dublin, Ireland, April 25-30, 2020, Proceedings, Peter Müller (Ed.) (Lecture Notes in Computer Science, Vol. 12075). Springer, 544–571. https://doi.org/10.1007/978-3-030-44914-8_20
Google Scholar
Digital Library
- Abel Nieto, Léon Gondelman, Alban Reynaud, Amin Timany, and Lars Birkedal. 2022. Modular Verification of Op-Based CRDTs in Separation Logic (Proof Artifact). https://doi.org/10.5281/zenodo.7055010
Google Scholar
Digital Library
- Patrick Redmond, Gan Shen, Niki Vazou, and Lindsey Kuper. 2022. Verified Causal Broadcast with Liquid Haskell. arXiv preprint arXiv:2206.14767, https://doi.org/10.48550/arXiv.2206.14767
Google Scholar
- A. W. Roscoe. 1996. Intensional Specifications of Security Protocols. In CSFW. IEEE Computer Society, 28–38.
Google Scholar
- Marc Shapiro, Nuno Preguiça, Carlos Baquero, and Marek Zawirski. 2011. A comprehensive study of Convergent and Commutative Replicated Data Types. INRIA. http://hal.inria.fr/inria-00555588/
Google Scholar
- Marc Shapiro, Nuno M. Preguiça, Carlos Baquero, and Marek Zawirski. 2011. Conflict-Free Replicated Data Types. In Stabilization, Safety, and Security of Distributed Systems - 13th International Symposium, SSS 2011, Grenoble, France, October 10-12, 2011. Proceedings, Xavier Défago, Franck Petit, and Vincent Villain (Eds.) (Lecture Notes in Computer Science, Vol. 6976). Springer, 386–400. https://doi.org/10.1007/978-3-642-24550-3_29
Google Scholar
Cross Ref
- Swaminathan Sivasubramanian. 2012. Amazon dynamoDB: a seamlessly scalable non-relational database service. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2012, Scottsdale, AZ, USA, May 20-24, 2012. 729–730. https://doi.org/10.1145/2213836.2213945
Google Scholar
Digital Library
- Andrew S. Tanenbaum and Maarten van Steen. 2007. Distributed systems - principles and paradigms, 2nd Edition. Pearson Education. isbn:978-0-13-239227-3
Google Scholar
Digital Library
- Amin Timany and Lars Birkedal. 2021. Reasoning about Monotonicity in Separation Logic. In CPP. ACM, 91–104. https://doi.org/10.1145/3437992.3439931
Google Scholar
Digital Library
- Amin Timany, Simon Oddershede Gregersen, Léo Stefanesco, Léon Gondelman, Abel Nieto, and Lars Birkedal. 2021. Trillium: Unifying refinement and higher-order distributed separation logic. arXiv preprint arXiv:2109.07863, https://doi.org/10.48550/arXiv.2109.07863
Google Scholar
- Misha Tyulenev, Andy Schwerin, Asya Kamsky, Randolph Tan, Alyson Cabral, and Jack Mulrow. 2019. Implementation of Cluster-wide Logical Clock and Causal Consistency in MongoDB. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD Conference 2019, Amsterdam, The Netherlands, June 30 - July 5, 2019. 636–650. https://doi.org/10.1145/3299869.3314049
Google Scholar
Digital Library
- Niki Vazou, Eric L. Seidel, Ranjit Jhala, Dimitrios Vytiniotis, and Simon L. Peyton Jones. 2014. Refinement Types for Haskell. In ICFP. ACM, 269–282. https://doi.org/10.1145/2628136.2628161
Google Scholar
Digital Library
- Peter Zeller, Annette Bieniusa, and Arnd Poetzsch-Heffter. 2014. Formal Specification and Verification of CRDTs. In FORTE (Lecture Notes in Computer Science, Vol. 8461). Springer, 33–48. https://doi.org/10.1007/978-3-662-43613-4_3
Google Scholar
Cross Ref
Index Terms
Modular verification of op-based CRDTs in separation logic
Recommendations
Strong eventual consistency of the collaborative editing framework WOOT
AbstractCommutative Replicated Data Types (CRDTs) are a promising new class of data structures for large-scale shared mutable content in applications that only require eventual consistency. The WithOut Operational Transforms (WOOT) framework is the first ...
Distributed causal memory: modular specification and verification in higher-order distributed separation logic
We present the first specification and verification of an implementation of a causally-consistent distributed database that supports modular verification of full functional correctness properties of clients and servers. We specify and reason about the ...
Integration Challenges of Pure Operation-based CRDTs in Redis
PMLDC '16: First Workshop on Programming Models and Languages for Distributed ComputingPure operation-based (op-based) Conflict-free Replicated Data Types (CRDTs) are generic and very efficient as they allow for compact solutions in both sent messages and state size. Although the pure op-based model looks promising, it is still not fully ...






Comments