skip to main content

ECROs: building global scale systems from sequential code

Published:15 October 2021Publication History
Skip Abstract Section

Abstract

To ease the development of geo-distributed applications, replicated data types (RDTs) offer a familiar programming interface while ensuring state convergence, low latency, and high availability. However, RDTs are still designed exclusively by experts using ad-hoc solutions that are error-prone and result in brittle systems. Recent works statically detect conflicting operations on existing data types and coordinate those at runtime to guarantee convergence and preserve application invariants. However, these approaches are too conservative, imposing coordination on a large number of operations. In this work, we propose a principled approach to design and implement efficient RDTs taking into account application invariants. Developers extend sequential data types with a distributed specification, which together form an RDT. We statically analyze the specification to detect conflicts and unravel their cause. This information is then used at runtime to serialize concurrent operations safely and efficiently. Our approach derives a correct RDT from any sequential data type without changes to the data type's implementation and with minimal coordination. We implement our approach in Scala and develop an extensive portfolio of RDTs. The evaluation shows that our approach provides performance similar to conflict-free replicated data types for commutative operations, and considerably improves the performance of non-commutative operations, compared to existing solutions.

Skip Supplemental Material Section

Supplemental Material

Auxiliary Presentation Video

This is a presentation of our OOPSLA 2021 paper "ECROs: Building Global Scale Systems from Sequential Code”. We propose a principled approach to design and implement efficient replicated data types (RDTs) taking into account application invariants. Developers extend sequential data types with a distributed specification, which together form an RDT. We implement our approach in Scala and develop an extensive portfolio of RDTs. The evaluation shows that our approach provides performance similar to conflict-free replicated data types for commutative operations, and considerably improves the performance of non-commutative operations, compared to existing solutions.

References

  1. Paulo Sérgio Almeida, Ali Shoker, and Carlos Baquero. 2015. Efficient State-Based CRDTs by Delta-Mutation. In Networked Systems - Third International Conference, NETYS 2015, Agadir, Morocco, May 13-15, 2015, Revised Selected Papers (Lecture Notes in Computer Science, Vol. 9466). Springer, 62–76. https://doi.org/10.1007/978-3-319-26850-7_5 Google ScholarGoogle ScholarCross RefCross Ref
  2. Peter Bailis, Alan D. Fekete, Michael J. Franklin, Ali Ghodsi, Joseph M. Hellerstein, and Ion Stoica. 2014. Coordination Avoidance in Database Systems. Proc. VLDB Endow., 8, 3 (2014), 185–196. https://doi.org/10.14778/2735508.2735509 Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Valter Balegas, Sérgio Duarte, Carla Ferreira, Rodrigo Rodrigues, and Nuno M. Preguiça. 2018. IPA: Invariant-preserving Applications for Weakly consistent Replicated Databases. Proc. VLDB Endow., 12, 4 (2018), 404–418. https://doi.org/10.14778/3297753.3297760 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Valter Balegas, Sérgio Duarte, Carla Ferreira, Rodrigo Rodrigues, Nuno M. Preguiça, Mahsa Najafzadeh, and Marc Shapiro. 2015. Putting consistency back into eventual consistency. In Proceedings of the Tenth European Conference on Computer Systems, EuroSys 2015, Bordeaux, France, April 21-24, 2015. ACM, 6:1–6:16. https://doi.org/10.1145/2741948.2741972 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Carlos Baquero, Paulo S. Almeida, and Ali Shoker. 2017. Pure Operation-Based Replicated Data Types. CoRR, abs/1710.04469 (2017), arxiv:1710.04469.Google ScholarGoogle Scholar
  6. Eric Brewer. 2012. CAP Twelve years later: How the “Rules” have Changed. Computer, 45 (2012), 02, 23–29. https://doi.org/10.1109/MC.2012.37 Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Eric A. Brewer. 2000. Towards robust distributed systems (abstract). In Proceedings of the Nineteenth Annual ACM Symposium on Principles of Distributed Computing, July 16-19, 2000, Portland, Oregon, USA. ACM, 7. https://doi.org/10.1145/343477.343502 Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Sebastian Burckhardt, Manuel Fähndrich, Daan Leijen, and Benjamin P. Wood. 2012. Cloud Types for Eventual Consistency. In ECOOP 2012 - Object-Oriented Programming - 26th European Conference, Beijing, China, June 11-16, 2012. Proceedings (Lecture Notes in Computer Science, Vol. 7313). Springer, 283–307. https://doi.org/10.1007/978-3-642-31057-7_14 Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Emmanuel Cecchet and Julie Marguerite. 2009. RUBiS: Rice University Bidding System. http://rubis.ow2.org/Google ScholarGoogle Scholar
  10. Kevin De Porre, Carla Ferreira, Nuno Preguiça, and Elisa Gonzalez Boix. 2021. ECROs: Building Global Scale Systems from Sequential Code (Appendix). http://soft.vub.ac.be/Publications/2021/vub-tr-soft-21-09-appendix.pdfGoogle ScholarGoogle Scholar
  11. Kevin De Porre and Elisa Gonzalez Boix. 2019. Squirrel: an extensible distributed key-value store. In Proceedings of the 4th ACM SIGPLAN International Workshop on Meta-Programming Techniques and Reflection, [email protected] 2019, Athens, Greece, October 20, 2019. ACM, 21–30. https://doi.org/10.1145/3358502.3361271 Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Kevin De Porre, Florian Myter, Christophe De Troyer, Christophe Scholliers, Wolfgang De Meuter, and Elisa Gonzalez Boix. 2019. Putting Order in Strong Eventual Consistency. In Distributed Applications and Interoperable Systems - 19th IFIP WG 6.1 International Conference, DAIS 2019, Held as Part of the 14th International Federated Conference on Distributed Computing Techniques, DisCoTec 2019, Kongens Lyngby, Denmark, June 17-21, 2019, Proceedings (Lecture Notes in Computer Science, Vol. 11534). Springer, 36–56. https://doi.org/10.1007/978-3-030-22496-7_3 Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Kevin De Porre, Florian Myter, Christophe Scholliers, and Elisa Gonzalez Boix. 2020. CScript: A distributed programming language for building mixed-consistency applications. J. Parallel Distributed Comput., 144 (2020), 109–123. https://doi.org/10.1016/j.jpdc.2020.05.010 Google ScholarGoogle ScholarCross RefCross Ref
  14. Dimitar I. Dimitrov, Veselin Raychev, Martin T. Vechev, and Eric Koskinen. 2014. Commutativity race detection. In ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’14, Edinburgh, United Kingdom - June 09 - 11, 2014. ACM, 305–315. https://doi.org/10.1145/2594291.2594322 Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Alexey Gotsman, Hongseok Yang, Carla Ferreira, Mahsa Najafzadeh, and Marc Shapiro. 2016. ’Cause I’m strong enough: reasoning about consistency choices in distributed systems. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20 - 22, 2016. ACM, 371–384. https://doi.org/10.1145/2837614.2837625 Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Rachid Guerraoui, Matej Pavlovic, and Dragos-Adrian Seredinschi. 2016. Incremental Consistency Guarantees for Replicated Objects. In 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4, 2016. USENIX Association, 169–184. isbn:978-1-931971-33-1 https://www.usenix.org/conference/osdi16/technical-sessions/presentation/guerraouiGoogle ScholarGoogle Scholar
  17. Brandon Holt, James Bornholt, Irene Zhang, Dan R. K. Ports, Mark Oskin, and Luis Ceze. 2016. Disciplined Inconsistency with Consistency Types. In Proceedings of the Seventh ACM Symposium on Cloud Computing, Santa Clara, CA, USA, October 5-7, 2016. ACM, 279–293. https://doi.org/10.1145/2987550.2987559 Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Farzin Houshmand and Mohsen Lesani. 2019. Hamsaz: replication coordination analysis and synthesis. Proc. ACM Program. Lang., 3, POPL (2019), 74:1–74:32. https://doi.org/10.1145/3290387 Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Gowtham Kaki, Kapil Earanky, K. C. Sivaramakrishnan, and Suresh Jagannathan. 2018. Safe replication through bounded concurrency verification. Proc. ACM Program. Lang., 2, OOPSLA (2018), 164:1–164:27. https://doi.org/10.1145/3276534 Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Gowtham Kaki, Swarn Priya, K. C. Sivaramakrishnan, and Suresh Jagannathan. 2019. Mergeable replicated data types. Proc. ACM Program. Lang., 3, OOPSLA (2019), 154:1–154:29. https://doi.org/10.1145/3360580 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Anne-Marie Kermarrec, Antony I. T. Rowstron, Marc Shapiro, and Peter Druschel. 2001. The IceCube approach to the reconciliation of divergent replicas. In Proceedings of the Twentieth Annual ACM Symposium on Principles of Distributed Computing, PODC 2001, Newport, Rhode Island, USA, August 26-29, 2001. ACM, 210–218. https://doi.org/10.1145/383962.384020 Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Martin Kleppmann. 2015. A Critique of the CAP Theorem. ArXiv, abs/1509.05393 (2015).Google ScholarGoogle Scholar
  23. Martin Kleppmann and Alastair R. Beresford. 2017. A Conflict-Free Replicated JSON Datatype. IEEE Trans. Parallel Distributed Syst., 28, 10 (2017), 2733–2746. https://doi.org/10.1109/TPDS.2017.2697382 Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Mirko Köhler, Nafise Eskandani, Pascal Weisenburger, Alessandro Margara, and Guido Salvaneschi. 2020. Rethinking safe consistency in distributed object-oriented programming. Proc. ACM Program. Lang., 4, OOPSLA (2020), 188:1–188:30. https://doi.org/10.1145/3428256 Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Milind Kulkarni, Donald Nguyen, Dimitrios Prountzos, Xin Sui, and Keshav Pingali. 2011. Exploiting the commutativity lattice. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2011, San Jose, CA, USA, June 4-8, 2011. ACM, 542–555. https://doi.org/10.1145/1993498.1993562 Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Leslie Lamport. 1978. Time, Clocks, and the Ordering of Events in a Distributed System. Commun. ACM, 21, 7 (1978), 558–565. https://doi.org/10.1145/359545.359563 Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Cheng Li, João Leitão, Allen Clement, Nuno M. Preguiça, Rodrigo Rodrigues, and Viktor Vafeiadis. 2014. Automating the Choice of Consistency Levels in Replicated Systems. In 2014 USENIX Annual Technical Conference, USENIX ATC ’14, Philadelphia, PA, USA, June 19-20, 2014. USENIX Association, 281–292. isbn:978-1-931971-10-2 https://www.usenix.org/conference/atc14/technical-sessions/presentation/li_cheng_2Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Cheng Li, Daniel Porto, Allen Clement, Johannes Gehrke, Nuno M. Preguiça, and Rodrigo Rodrigues. 2012. Making Geo-Replicated Systems Fast as Possible, Consistent when Necessary. In 10th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2012, Hollywood, CA, USA, October 8-10, 2012. USENIX Association, 265–278. isbn:978-1-931971-96-6 https://www.usenix.org/conference/osdi12/technical-sessions/presentation/liGoogle ScholarGoogle Scholar
  29. Cheng Li, Nuno M. Preguiça, and Rodrigo Rodrigues. 2018. Fine-grained consistency for geo-replicated systems. In 2018 USENIX Annual Technical Conference, USENIX ATC 2018, Boston, MA, USA, July 11-13, 2018. USENIX Association, 359–372. https://www.usenix.org/conference/atc18/presentation/li-chengGoogle ScholarGoogle Scholar
  30. Xiao Li, Farzin Houshmand, and Mohsen Lesani. 2020. Hampa: Solver-Aided Recency-Aware Replication. In Computer Aided Verification - 32nd International Conference, CAV 2020, Los Angeles, CA, USA, July 21-24, 2020, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 12224). Springer, 324–349. https://doi.org/10.1007/978-3-030-53288-8_16 Google ScholarGoogle ScholarCross RefCross Ref
  31. Christopher Meiklejohn and Peter Van Roy. 2015. Lasp: a language for distributed, coordination-free programming. In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, Siena, Italy, July 14-16, 2015. ACM, 184–195. https://doi.org/10.1145/2790449.2790525 Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Matthew Milano and Andrew C. Myers. 2018. MixT: a language for mixing consistency in geodistributed transactions. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018. ACM, 226–241. https://doi.org/10.1145/3192366.3192375 Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Matthew Milano, Rolph Recto, Tom Magrino, and Andrew C. Myers. 2019. A Tour of Gallifrey, a Language for Geodistributed Programming. In 3rd Summit on Advances in Programming Languages, SNAPL 2019, May 16-17, 2019, Providence, RI, USA (LIPIcs, Vol. 136). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 11:1–11:19. https://doi.org/10.4230/LIPIcs.SNAPL.2019.11 Google ScholarGoogle ScholarCross RefCross Ref
  34. Florian Myter, Christophe Scholliers, and Wolfgang De Meuter. 2018. A CAPable distributed programming model. In Proceedings of the 2018 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2018, Boston, MA, USA, November 7-8, 2018. ACM, 88–98. https://doi.org/10.1145/3276954.3276957 Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Sreeja S. Nair, Gustavo Petri, and Marc Shapiro. 2020. Proving the Safety of Highly-Available Distributed Objects. In Programming Languages and Systems - 29th European Symposium on Programming, ESOP 2020, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2020, Dublin, Ireland, April 25-30, 2020, Proceedings (Lecture Notes in Computer Science, Vol. 12075). Springer, 544–571. https://doi.org/10.1007/978-3-030-44914-8_20 Google ScholarGoogle ScholarCross RefCross Ref
  36. OpenJDK. [n. d.]. JMH - OpenJDK. https://openjdk.java.net/projects/code-tools/jmh/ Accessed: 13-05-2020.Google ScholarGoogle Scholar
  37. David J. Pearce and Paul H. J. Kelly. 2006. A dynamic topological sort algorithm for directed acyclic graphs. ACM J. Exp. Algorithmics, 11 (2006), https://doi.org/10.1145/1187436.1210590 Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Julien Ponge. July 2014. Avoiding Benchmarking Pitfalls on the JVM. https://www.oracle.com/technical-resources/articles/java/architect-benchmarking.html Accessed: 13-05-2020.Google ScholarGoogle Scholar
  39. Marc Shapiro, Nuno Preguiça, Carlos Baquero, and Marek Zawirski. 2011. A comprehensive study of Convergent and Commutative Replicated Data Types. Inria – Centre Paris-Rocquencourt ; INRIA, 50.Google ScholarGoogle Scholar
  40. Marc Shapiro, Nuno M. Preguiça, Carlos Baquero, and Marek Zawirski. 2011. Conflict-Free Replicated Data Types. In Stabilization, Safety, and Security of Distributed Systems - 13th International Symposium, SSS 2011, Grenoble, France, October 10-12, 2011. Proceedings (Lecture Notes in Computer Science, Vol. 6976). Springer, 386–400. https://doi.org/10.1007/978-3-642-24550-3_29 Google ScholarGoogle ScholarCross RefCross Ref
  41. K. C. Sivaramakrishnan, Gowtham Kaki, and Suresh Jagannathan. 2015. Declarative programming over eventually consistent data stores. 413–424. https://doi.org/10.1145/2737924.2737981 Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Tim Soethout, Tijs van der Storm, and Jurgen J. Vinju. 2019. Static local coordination avoidance for distributed objects. In Proceedings of the 9th ACM SIGPLAN International Workshop on Programming Based on Actors, Agents, and Decentralized Control, [email protected] 2019, Athens, Greece, October 22, 2019. ACM, 21–30. https://doi.org/10.1145/3358499.3361222 Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Douglas B. Terry, Marvin Theimer, Karin Petersen, Alan J. Demers, Mike Spreitzer, and Carl Hauser. 1995. Managing Update Conflicts in Bayou, a Weakly Connected Replicated Storage System. In Proceedings of the Fifteenth ACM Symposium on Operating System Principles, SOSP 1995, Copper Mountain Resort, Colorado, USA, December 3-6, 1995. ACM, 172–183. https://doi.org/10.1145/224056.224070 Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Werner Vogels. 2009. Eventually Consistent. Commun. ACM, 52, 1 (2009), 40–44. https://doi.org/10.1145/1435417.1435432 Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Chao Wang, Constantin Enea, Suha Orhun Mutluergil, and Gustavo Petri. 2019. Replication-aware linearizability. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2019, Phoenix, AZ, USA, June 22-26, 2019. ACM, 980–993. https://doi.org/10.1145/3314221.3314617 Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Matthew Weidner, Heather Miller, and Christopher Meiklejohn. 2020. Composing and decomposing op-based CRDTs with semidirect products. Proc. ACM Program. Lang., 4, ICFP (2020), 94:1–94:27. https://doi.org/10.1145/3408976 Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Nosheen Zaza and Nathaniel Nystrom. 2016. Data-centric Consistency Policies: A Programming Model for Distributed Applications with Tunable Consistency. In First Workshop on Programming Models and Languages for Distributed Computing, [email protected] 2016, Rome, Italy, July 17, 2016. ACM, 3. https://doi.org/10.1145/2957319.2957377 Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Xin Zhao and Philipp Haller. 2018. Observable atomic consistency for CvRDTs. In Proceedings of the 8th ACM SIGPLAN International Workshop on Programming Based on Actors, Agents, and Decentralized Control, [email protected] 2018, Boston, MA, USA, November 5, 2018. ACM, 23–32. https://doi.org/10.1145/3281366.3281372 Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Xin Zhao and Philipp Haller. 2020. Replicated data types that unify eventual consistency and observable atomic consistency. J. Log. Algebraic Methods Program., 114, 100561. https://doi.org/10.1016/j.jlamp.2020.100561 Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. ECROs: building global scale systems from sequential code

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!