Abstract
Besides well-known benefits, commodity cloud storage also raises concerns that include security, reliability, and consistency. We present Hybris key-value store, the first robust hybrid cloud storage system, aiming at addressing these concerns leveraging both private and public cloud resources.
Hybris robustly replicates metadata on trusted private premises (private cloud), separately from data, which are dispersed (using replication or erasure coding) across multiple untrusted public clouds. Hybris maintains metadata stored on private premises at the order of few dozens of bytes per key, avoiding the scalability bottleneck at the private cloud. In turn, the hybrid design allows Hybris to efficiently and robustly tolerate cloud outages but also potential malice in clouds without overhead. Namely, to tolerate up to f malicious clouds, in the common case of the Hybris variant with data replication, writes replicate data across f+1 clouds, whereas reads involve a single cloud. In the worst case, only up to f additional clouds are used. This is considerably better than earlier multi-cloud storage systems that required costly 3f+1 clouds to mask f potentially malicious clouds. Finally, Hybris leverages strong metadata consistency to guarantee to Hybris applications strong data consistency without any modifications to the eventually consistent public clouds.
We implemented Hybris in Java and evaluated it using a series of micro and macro-benchmarks. Our results show that Hybris significantly outperforms comparable multi-cloud storage systems and approaches the performance of bare-bone commodity public cloud storage.
- Ittai Abraham, Gregory Chockler, Idit Keidar, and Dahlia Malkhi. 2006. Byzantine disk paxos: Optimal resilience with byzantine shared memory. Distrib. Comput. 18, 5 (2006), 387--408. Google Scholar
Digital Library
- Hussam Abu-Libdeh, Lonnie Princehouse, and Hakim Weatherspoon. 2010. RACS: A case for cloud storage diversity. In Proceedings of the ACM Symposium on Cloud Computing (SoCC’10). 229--240. http://doi.acm.org/10.1145/1807128.1807165Google Scholar
Digital Library
- Atul Adya, William J. Bolosky, Miguel Castro, Gerald Cermak, Ronnie Chaiken, John R. Douceur, Jon Howell, Jacob R. Lorch, Marvin Theimer, and Roger Wattenhofer. 2002. FARSITE: Federated, available, and reliable storage for an incompletely trusted environment. In Proceedings of the Symposium on Operating System Design and Implementation (OSDI’02). http://www.usenix.org/events/osdi02/tech/adya.html.Google Scholar
Cross Ref
- Amazon. 2016. Amazon DynamoDB Pricing. (2016). Retrieved from https://aws.amazon.com/dynamodb/pricing/.Google Scholar
- Elli Androulaki, Christian Cachin, Dan Dobre, and Marko Vukolic. 2014. Erasure-coded byzantine storage with separate metadata. In Proceedings of the International Conference on Principles of Distributed Systems (OPODIS’14). 76--90. DOI:http://dx.doi.org/10.1007/978-3-319-14472-6_6 Google Scholar
Cross Ref
- Apache. 2016. Apache JClouds. Retrieved from http://jclouds.apache.org/.Google Scholar
- Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy H. Katz, Andy Konwinski, Gunho Lee, David A. Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. 2010. A view of cloud computing. Commun. ACM 53, 4 (2010), 50--58. DOI:http://dx.doi.org/10.1145/1721654.1721672 Google Scholar
Digital Library
- Peter Bailis and Ali Ghodsi. 2013. Eventual consistency today: Limitations, extensions, and beyond. Commun. ACM 56, 5 (2013), 55--63. http://doi.acm.org/10.1145/2447976.2447992 Google Scholar
Digital Library
- Cristina Basescu, Christian Cachin, Ittay Eyal, Robert Haas, Alessandro Sorniotti, Marko Vukolić, and Ido Zachevsky. 2012. Robust data sharing with key-value stores. In Proceedings of the IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’12). 1--12. DOI:http://dx.doi.org/10.1109/DSN.2012.6263920 Google Scholar
Cross Ref
- David Bermbach and Stefan Tai. 2014. Benchmarking eventual consistency: Lessons learned from long-term experimental studies. In Proceedings of the IEEE International Conference on Cloud Engineering (IC2E’14). 47--56. DOI:http://dx.doi.org/10.1109/IC2E.2014.37 Google Scholar
Digital Library
- Alysson Bessani, Ricardo Mendes, Tiago Oliveira, Nuno Neves, Miguel Correia, Marcelo Pasin, and Paulo Verissimo. 2014. SCFS: A shared cloud-backed file system. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’14).Google Scholar
- Alysson Neves Bessani, Miguel Correia, Bruno Quaresma, Fernando André, and Paulo Sousa. 2013. DepSky: Dependable and secure storage in a cloud-of-clouds. ACM Trans. Stor. 9, 4 (2013), 12. http://doi.acm.org/10.1145/2535929Google Scholar
Digital Library
- Carlos Eduardo Benevides Bezerra, Fernando Pedone, and Robbert van Renesse. 2014. Scalable state-machine replication. In Proceedings of the IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’14). 331--342. DOI:http://dx.doi.org/10.1109/DSN.2014.41 Google Scholar
Digital Library
- Kevin D. Bowers, Ari Juels, and Alina Oprea. 2009. HAIL: A high-availability and integrity layer for cloud storage. In Proceedings of the ACM Conference on Computer and Communications Security (CCS’09). 187--198. http://doi.acm.org/10.1145/1653662.1653686Google Scholar
Digital Library
- Eric A. Brewer. 2012. Pushing the CAP: Strategies for consistency and availability. IEEE Comput. 45, 2 (2012), 23--29. http://doi.ieeecomputersociety.org/10.1109/MC.2012.37Google Scholar
Digital Library
- Christian Cachin, Dan Dobre, and Marko Vukolic. 2014. Separating data and control: Asynchronous BFT storage with 2t + 1 data replicas. In Proceedings of the Conference on Stabilization, Safety, and Security of Distributed Systems (SSS’14). 1--17. DOI:http://dx.doi.org/10.1007/978-3-319-11764-5_1 Google Scholar
Cross Ref
- Ignacio Cano, Srinivas Aiyar, and Arvind Krishnamurthy. 2016. Characterizing private clouds: A large-scale empirical analysis of enterprise clusters. In Proceedings of the ACM Symposium on Cloud Computing (SoCC’16). 29--41. DOI:http://dx.doi.org/10.1145/2987550.2987584 Google Scholar
Digital Library
- Houssem-Eddine Chihoub, Shadi Ibrahim, Gabriel Antoniu, and María S. Pérez-Hernández. 2012. Harmony: Towards automated self-adaptive consistency in cloud storage. In Proceedings of the IEEE International Conference on Cluster Computing (CLUSTER’12). 293--301. DOI:http://dx.doi.org/10.1109/CLUSTER.2012.56 Google Scholar
Digital Library
- Gregory Chockler, Dan Dobre, and Alexander Shraer. 2013. Brief announcement: Consistency and complexity tradeoffs for highly-available multi-cloud store. In Proceedings of the International Symposium on Distributed Computing (DISC’13).Google Scholar
- Byung-Gon Chun, Petros Maniatis, Scott Shenker, and John Kubiatowicz. 2007. Attested append-only memory: Making adversaries stick to their word. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP’07). 189--204. http://doi.acm.org/10.1145/1294261.1294280Google Scholar
Digital Library
- CloudSpaces. 2015. CloudSpaces EU FP7 project. Retrieved from http://cloudspaces.eu/.Google Scholar
- Consul. 2016a. Consul—Consensus protocol. Retrieved from https://www.consul.io/docs/internals/consensus.html.Google Scholar
- Consul. 2016b. Consul —Distributed service discovery and configuration. Retrieved from https://www.consul.io/.Google Scholar
- Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking cloud serving systems with YCSB. In Proceedings of the ACM Symposium on Cloud Computing (SoCC’10). 143--154. http://doi.acm.org/10.1145/1807128.1807152Google Scholar
Digital Library
- James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, J. J. Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, Wilson C. Hsieh, Sebastian Kanthak, Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura, David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Yasushi Saito, Michal Szymaniak, Christopher Taylor, Ruth Wang, and Dale Woodford. 2013. Spanner: Google’s globally distributed database. ACM Trans. Comput. Syst. 31, 3 (2013), 8. http://doi.acm.org/10.1145/2491245Google Scholar
Cross Ref
- Miguel Correia, Daniel Gómez Ferro, Flavio Paiva Junqueira, and Marco Serafini. 2012. Practical hardening of crash-tolerant systems. In Proceedings of the USENIX Annual Technical Conference.Google Scholar
- Miguel Correia, Nuno Ferreira Neves, and Paulo Veríssimo. 2004. How to tolerate half less one byzantine nodes in practical distributed systems. In Proceedings of the Symposium on Reliable Distributed Systems (SRDS’04). 174--183. http://doi.ieeecomputersociety.org/10.1109/RELDIS.2004.1353018Google Scholar
Cross Ref
- Idilio Drago, Enrico Bocchi, Marco Mellia, Herman Slatman, and Aiko Pras. 2013. Benchmarking personal cloud storage. In Proceedings of the ACM SIGCOMM Internet Measurement Conference (IMC’13). 205--212. DOI:http://dx.doi.org/10.1145/2504730.2504762 Google Scholar
Digital Library
- Idilio Drago, Marco Mellia, Maurizio M. Munafò, Anna Sperotto, Ramin Sadre, and Aiko Pras. 2012. Inside dropbox: Understanding personal cloud storage services. In Proceedings of the ACM SIGCOMM Internet Measurement Conference (IMC’12). 481--494. DOI:http://dx.doi.org/10.1145/2398776.2398827 Google Scholar
Digital Library
- Cynthia Dwork, Nancy Lynch, and Larry Stockmeyer. 1988. Consensus in the presence of partial synchrony. J. ACM 35, 2 (Apr.1988), 288--323. DOI:http://dx.doi.org/10.1145/42282.42283 Google Scholar
Digital Library
- Eurecom. 2016. Hybris—Robust hybrid cloud storage library. Retrieved from https://github.com/pviotti/hybris.Google Scholar
- Rui Fan and Nancy Lynch. 2003. Efficient replication of large data objects. In Proceedings of the Diocesan Information Systems Conference (DISC’03). 75--91. Google Scholar
Cross Ref
- Aishwarya Ganesan, Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2017. Redundancy does not imply fault tolerance: Analysis of distributed storage reactions to single errors and corruptions. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST’17).Google Scholar
- Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. 2003. The google file system. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP’03). 29--43. http://doi.acm.org/10.1145/945445.945450Google Scholar
Digital Library
- Garth A. Gibson, David Nagle, Khalil Amiri, Jeff Butler, Fay W. Chang, Howard Gobioff, Charles Hardin, Erik Riedel, David Rochberg, and Jim Zelenka. 1998. A cost-effective, high-bandwidth storage architecture. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems. 92--103. DOI:http://dx.doi.org/10.1145/291069.291029 Google Scholar
Digital Library
- Seth Gilbert and Nancy A. Lynch. 2002. Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News 33, 2 (2002), 51--59. http://doi.acm.org/10.1145/564585.564601Google Scholar
Digital Library
- Wojciech M. Golab, Xiaozhou Li, and Mehul A. Shah. 2011. Analyzing consistency properties for fun and profit. In Proceedings of the ACM Symposium on Principles of Distributed Computing (PODC’11), 2011. 197--206. Google Scholar
Digital Library
- Haryadi S. Gunawi, Mingzhe Hao, Riza O. Suminto, Agung Laksono, Anang D. Satria, Jeffry Adityatama, and Kurnia J. Eliazar. 2016. Why does the cloud stop computing?: Lessons from hundreds of service outages. In Proceedings of the ACM Symposium on Cloud Computing (SoCC’16). 1--16. DOI:http://dx.doi.org/10.1145/2987550.2987583 Google Scholar
Digital Library
- Raluca Halalai, Pierre Sutra, Etienne Riviere, and Pascal Felber. 2014. ZooFence: Principled service partitioning and application to the zookeeper coordination service. In Proceedings of the IEEE International Symposium on Reliable Distributed Systems. 67--78. DOI:http://dx.doi.org/10.1109/SRDS.2014.41 Google Scholar
Digital Library
- James Hamilton. 2009. The cost of latency. Retreived from http://perspectives.mvdirona.com/2009/10/the-cost-of-latency/.Google Scholar
- Maurice Herlihy. 1991. Wait-free synchronization. ACM Trans. Program. Lang. Syst. 13, 1 (1991). Google Scholar
Digital Library
- Maurice P. Herlihy and Jeannette M. Wing. 1990. Linearizability: A correctness condition for concurrent objects. ACM Trans. Program. Lang. Syst. 12, 3 (1990). Google Scholar
Digital Library
- Patrick Hunt, Mahadev Konar, Flavio P. Junqueira, and Benjamin Reed. 2010. ZooKeeper: Wait-free coordination for internet-scale systems. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’10). 11--11.Google Scholar
- Flavio Paiva Junqueira, Ivan Kelly, and Benjamin Reed. 2013. Durability with bookkeeper. Operat. Syst. Rev. 47, 1 (2013), 9--15. http://doi.acm.org/10.1145/2433140.2433144Google Scholar
Digital Library
- Rüdiger Kapitza, Johannes Behl, Christian Cachin, Tobias Distler, Simon Kuhnle, Seyed Vahid Mohammadi, Wolfgang Schröder-Preikschat, and Klaus Stengel. 2012. CheapBFT: Resource-efficient byzantine fault tolerance. In Proceedings of the European Professional Society on Computer Systems (EuroSys’12). 295--308. http://doi.acm.org/10.1145/2168836.2168866Google Scholar
Digital Library
- Ramakrishna Kotla, Lorenzo Alvisi, and Michael Dahlin. 2007. SafeStore: A durable and practical storage system. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’07). 129--142.Google Scholar
- Leslie Lamport. 1979. How to make a multiprocessor computer that correctly executes multiprocess programs. IEEE Trans. Comput. 28, 9 (1979), 690--691. Google Scholar
Digital Library
- Shengyun Liu, Paolo Viotti, Christian Cachin, Vivien Quéma, and Marko Vukolic. 2016. XFT: Practical fault tolerance beyond crashes. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI’16). https://www.usenix.org/conference/osdi16/technical-sessions/presentation/liu.Google Scholar
- Pedro Garcia Lopez, Sergi Toda, Cristian Cotes, Marc Sanchez-Artigas, and John Lenton. 2014. StackSync: Bringing elasticity to dropbox-like file synchronization. In Proceedings of the ACM/IFIP/USENIX Middleware Conference. Google Scholar
Digital Library
- Nancy A. Lynch and Alexander A. Shvartsman. 2002. RAMBO: A reconfigurable atomic memory service for dynamic networks. In Proceedings of the Diocesan Information Systems Conference (DISC’02). 173--190. Google Scholar
Cross Ref
- Prince Mahajan, Srinath T. V. Setty, Sangmin Lee, Allen Clement, Lorenzo Alvisi, Michael Dahlin, and Michael Walfish. 2011. Depot: Cloud storage with minimal trust. ACM Trans. Comput. Syst. 29, 4 (2011), 12. Google Scholar
Digital Library
- Memcached. 2016. Memcached. (2016). Available online at http://memcached.org/; visited September 2016.Google Scholar
- Diego Ongaro and John K. Ousterhout. 2014. In search of an understandable consensus algorithm. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’14). 305--319.Google Scholar
- M. Pease, R. Shostak, and L. Lamport. 1980. Reaching agreement in the presence of faults. J. ACM 27, 2 (1980). Google Scholar
Digital Library
- J. S. Plank, S. Simmerman, and C. D. Schuman. 2008. Jerasure: A Library in C/C++ Facilitating Erasure Coding for Storage Applications—Version 1.2. Technical Report CS-08-627. University of Tennessee.Google Scholar
- Benjamin Reed and Flavio P. Junqueira. 2008. A simple totally ordered broadcast protocol. In Proceedings of the Workshop on Large-Scale Distributed Systems and Middleware (LADIS’08). 2:1--2:6. DOI:http://dx.doi.org/10.1145/1529974.1529978 Google Scholar
Digital Library
- Rodrigo Rodrigues and Barbara Liskov. 2005. High availability in DHTs: Erasure coding vs. replication. In Proceedings of the International Workshop on Peer-to-Peer Systems (IPTPS’05). 226--239. http://dx.doi.org/10.1007/11558989_21 Google Scholar
Digital Library
- K. C. Sivaramakrishnan, Gowtham Kaki, and Suresh Jagannathan. 2015. Declarative programming over eventually consistent data stores. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. 413--424. DOI:http://dx.doi.org/10.1145/2737924.2737981 Google Scholar
Digital Library
- SoftLayer. 2016. IBM SoftLayer. Retrieved from http://www.softlayer.com/.Google Scholar
- Emil Stefanov, Marten van Dijk, Elaine Shi, Christopher W. Fletcher, Ling Ren, Xiangyao Yu, and Srinivas Devadas. 2013. Path ORAM: An extremely simple oblivious RAM protocol. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS’13). http://doi.acm.org/10.1145/2508859.2516660Google Scholar
Digital Library
- Syncany. 2016. Secure file synchronization software for arbitrary storage backends. Retrieved from https://www.syncany.org/.Google Scholar
- Douglas B. Terry, Alan J. Demers, Karin Petersen, Mike Spreitzer, Marvin Theimer, and Brent B. Welch. 1994. Session guarantees for weakly consistent replicated data. In Proceedings of the Parallel and Distributed Information Systems (PDIS’94). 140--149. Google Scholar
Cross Ref
- Douglas B. Terry, Vijayan Prabhakaran, Ramakrishna Kotla, Mahesh Balakrishnan, Marcos K. Aguilera, and Hussam Abu-Libdeh. 2013. Consistency-based service level agreements for cloud storage. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles (SOSP’13). 309--324. DOI:http://dx.doi.org/10.1145/2517349.2522731 Google Scholar
Digital Library
- Francisco J. Torres-Rojas, Mustaque Ahamad, and Michel Raynal. 1999. Timed consistency for shared distributed objects. In Proceedings of the ACM Symposium on Principles of Distributed Computing (PODC’99). 163--172. Google Scholar
Digital Library
- Giuliana Santos Veronese, Miguel Correia, Alysson Neves Bessani, Lau Cheuk Lung, and Paulo Veríssimo. 2013. Efficient byzantine fault-tolerance. IEEE Trans. Comput. 62, 1 (2013), 16--30.Google Scholar
Digital Library
- Paolo Viotti and Marko Vukolic. 2016. Consistency in non-transactional distributed storage systems. ACM Comput. Surv. 49, 1 (2016), 19. DOI:http://dx.doi.org/10.1145/2926965 Google Scholar
Digital Library
- VMware. 2013. The Snowden Leak: A Windfall for Hybrid Cloud? (2013). Retrieved from http://blogs.vmware.com/consulting/2013/09/the-snowden-leak-a-windfall-for-hybrid-cloud.html.Google Scholar
- Werner Vogels. 2009. Eventually consistent. Commun. ACM 52, 1 (2009), 40--44. http://doi.acm.org/10.1145/1435417.1435432 Google Scholar
Digital Library
- Marko Vukolić. 2010. The byzantine empire in the intercloud. SIGACT News 41, 3 (2010), 105--111. DOI:http://dx.doi.org/10.1145/1855118.1855137 Google Scholar
Digital Library
- Yang Wang, Lorenzo Alvisi, and Mike Dahlin. 2012. Gnothi: Separating data and metadata for efficient and available storage replication. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’12). 38--38.Google Scholar
- Hakim Weatherspoon and John Kubiatowicz. 2002. Erasure coding vs. replication: A quantitative comparison. In Proceedings of the International Workshop on Peer-to-Peer Systems (IPTPS’02). 328--338.Google Scholar
Cross Ref
- Sage A. Weil, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, and Carlos Maltzahn. 2006. Ceph: A scalable, high-performance distributed file system. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI’06). 307--320.Google Scholar
- Zhe Wu, Michael Butkiewicz, Dorian Perkins, Ethan Katz-Bassett, and Harsha V. Madhyastha. 2013. SPANStore: Cost-effective geo-replicated storage spanning multiple cloud services. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP’13).Google Scholar
- Jian Yin, Jean-Philippe Martin, Arun Venkataramani, Lorenzo Alvisi, and Michael Dahlin. 2003. Separating agreement from execution for byzantine fault tolerant services. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP’03). 253--267. http://doi.acm.org/10.1145/945445.945470Google Scholar
Digital Library
- Hassan Zeineddine and Wael Bazzi. 2011. Rationing data updates with consistency considerations in distributed systems. In Proceedings of the IEEE International Conference on Networks (ICON’11). 165--170. DOI:http://dx.doi.org/10.1109/ICON.2011.6168469 Google Scholar
Digital Library
- Heng Zhang, Mingkai Dong, and Haibo Chen. 2016. Efficient and available in-memory KV-store with hybrid erasure coding and replication. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST’16).Google Scholar
Digital Library
- Observers ZooKeeper. 2009. Observers—Making ZooKeeper scale even further. Retrieved from https://blog.cloudera.com/blog/2009/12/observers-making-zookeeper-scale-even-further/.Google Scholar
Index Terms
Hybris: Robust Hybrid Cloud Storage
Recommendations
Hybris: Robust Hybrid Cloud Storage
SOCC '14: Proceedings of the ACM Symposium on Cloud ComputingBesides well-known benefits, commodity cloud storage also raises concerns that include security, reliability, and consistency. We present Hybris key-value store, the first robust hybrid cloud storage system, aiming at addressing these concerns ...
A Trust Management Solution in the Context of Hybrid Clouds
WETICE '14: Proceedings of the 2014 IEEE 23rd International WETICE ConferenceCloud computing is a revolutionary paradigm which enables on-demand provisioning of computing resources. Resources are delivered to cloud consumers in the form of infrastructure, platform and software services. These resources are deployed on three ...
Cloud Storage as the Infrastructure of Cloud Computing
ICICCI '10: Proceedings of the 2010 International Conference on Intelligent Computing and Cognitive InformaticsAs an emerging technology and business paradigm, Cloud Computing has taken commercial computing by storm. Cloud computing platforms provide easy access to a company’s high-performance computing and storage infrastructure through web services. With cloud ...






Comments