Abstract
This article describes the design, implementation, and evaluation of Depot, a cloud storage system that minimizes trust assumptions. Depot tolerates buggy or malicious behavior by any number of clients or servers, yet it provides safety and liveness guarantees to correct clients. Depot provides these guarantees using a two-layer architecture. First, Depot ensures that the updates observed by correct nodes are consistently ordered under Fork-Join-Causal consistency (FJC). FJC is a slight weakening of causal consistency that can be both safe and live despite faulty nodes. Second, Depot implements protocols that use this consistent ordering of updates to provide other desirable consistency, staleness, durability, and recovery properties. Our evaluation suggests that the costs of these guarantees are modest and that Depot can tolerate faults and maintain good availability, latency, overhead, and staleness even when significant faults occur.
- Abu-Libdeh, H., Princehouse, L., and Weatherspoon, H. 2010. RACS: A case for cloud storage diversity. In Proceedings of the ACM Symposium on Cloud Computing (SOCC). Google Scholar
Digital Library
- Ahamad, M., Neiger, G., Burns, J. E., Kohli, P., and Hutto, P. 1995. Causal memory: Definitions, implementation and programming. Distrib. Comput. 9, 1, 37--49.Google Scholar
Digital Library
- Amazon. 2011. Summary of the Amazon EC2 and Amazon RDS service disruption in the US East region. http://aws.amazon.com/message/65648.Google Scholar
- Amazon S3 Team. 2008. Amazon S3 Team. Amazon S3 availability event: July 20, 2008. http://status.aws.amazon.com/s3-20080720.html.Google Scholar
- Beckmann, C. 2009. Google App Engine: Information regarding 2 July 2009 outage. http://groups.google.com/group/google-appengine/browse_thread/thread/e9237fc7b0aa7df5/ba95ded980c8c179.Google Scholar
- Belaramani, N., Dahlin, M., Gao, L., Nayate, A., Venkataramani, A., Yalagandula, P., and Zheng, J. 2006. PRACTI replication. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI). Google Scholar
Digital Library
- Birrell, A., Levin, R., Needham, R., and Schroeder, M. 1982. Grapevine: An exercise in distributed computing. Comm. ACM 25, 4. Google Scholar
Digital Library
- Cachin, C., Shelat, A., and Shraer, A. 2007. Efficient fork-linearizable access to untrusted shared memory. In Proceedings of the ACM Symposium on the Principles of Distributed Computing (PODC). Google Scholar
Digital Library
- Cachin, C., Keidar, I., and Shraer, A. 2009. Fail-aware untrusted storage. In Proceedings of the International Conference on Dependable Systems and Networks (DSN).Google Scholar
- Calore, M. 2009. Magnolia suffers major data loss, site taken offline. Wired Mag.Google Scholar
- Castro, M. and Liskov, B. 2002. Practical Byzantine fault tolerance and proactive recovery. ACM Trans. Comput. Syst. 20, 4. Google Scholar
Digital Library
- Chang, F., Dean, J., Ghemawat, S., Hsieh, W., Wallach, D., Burrows, M., Chandra, T., Fikes, A., and Gruber, R. 2006. Bigtable: A distributed storage system for structured data. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). Google Scholar
Digital Library
- Chun, B.-G., Maniatis, P., Shenker, S., and Kubiatowicz, J. 2007. Attested append-only memory: Making adversaries stick to their word. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP). Google Scholar
Digital Library
- CircleID. 2009. Survey: Cloud computing ‘no hype’, but fear of security and control slowing adoption. http://www.circleid.com/posts/20090226_cloud_computing_hype_security/.Google Scholar
- Clement, A., Kapritsos, M., Lee, S., Wang, Y., Alvisi, L., Dahlin, M., and Riché, T. 2009. UpRight cluster services. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP). Google Scholar
Digital Library
- CNet News. 2011. Google probing lost Gmail messages, contacts. http://news.cnet.com/8301-1023 3-20037019-93.html.Google Scholar
- Cook, B. 2009. Seattle data center fire knocks out Bing Travel, other web sites. http://www.techflash.com/seattle/2009/07/Seattle_data_center_fire_knocks_out_Bing_Travel_other_Web_sites_49876777.html.Google Scholar
- Cooper, B., Ramakrishnan, R., Srivastava, U., Silberstein, A., Bohannon, P., Jacobsen, H., Puz, N., Weaver, D., and Yerneni, R. 2008. PNUTS: Yahoo!’s hosted data serving platform. In Proceedings of the International Conference on Very Large Data Bases (VLDB).Google Scholar
- DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., and Vogels, W. 2007. Dynamo: Amazon’s highly available key-value store. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP). Google Scholar
Digital Library
- Demmer, M., Du, B., and Brewer, E. 2008. TierStore: A distributed filesystem for challenged networks in developing regions. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST). Google Scholar
Digital Library
- Feldman, A. J., Zeller, W. P., Freedman, M. J., and Felten, E. W. 2010. SPORC: Group collaboration using untrusted cloud resources. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). Google Scholar
Digital Library
- Frigo, M. and Luchangco, V. 1998. Computation-centric memory models. In Proceedings of the ACM Symposium on Parallelism in Algorithms and Architectures (SPAA). Google Scholar
Digital Library
- Fu, K., Kaashoek, M. F., and Mazières, D. 2002. Fast and secure distributed read-only file system. ACM Trans. Comput. Syst. 20, 1, 1--24. Google Scholar
Digital Library
- Gilbert, S. and Lynch, N. 2002. Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News 33, 51--59. Google Scholar
Digital Library
- Goh, E.-J., Shacham, H., Modadugu, N., and Boneh, D. 2003. SiRiUS: Securing remote untrusted storage. In Proceedings of the Annual Network & Distributed System Security Symposium (NDSS).Google Scholar
- Gray, J. and Shenoy, P. 2000. Rules of thumb in data engineering. In Data Engineering. 3--12. Google Scholar
Digital Library
- Guerraoui, R., Knezevic, N., Quema, V., and Vukolic, M. 2010. The next 700 BFT protocols. In Proceedings of the European Conference on Computer Systems (EuroSys). Google Scholar
Digital Library
- Haeberlen, A., Mislove, A., and Druschel, P. 2005. Glacier: Highly durable, decentralized storage despite massive correlated failures. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI). Google Scholar
Digital Library
- Haeberlen, A., Kouznetsov, P., and Druschel, P. 2007. PeerReview: Practical accountability for distributed systems. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP). Google Scholar
Digital Library
- Hendricks, J., Ganger, G. R., and Reiter, M. K. 2007. Low-overhead Byzantine fault-tolerant storage. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP). Google Scholar
Digital Library
- Herlihy, M. P. and Wing, J. M. 1990. Linearizability: A correctness condition for concurrent objects. ACM Trans. Program. Lang. Syst. 12, 3. Google Scholar
Digital Library
- Kallahalla, M., Riedel, E., Swaminathan, R., Wang, Q., and Fu, K. 2003. Plutus: Scalable secure file sharing on untrusted storage. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST). Google Scholar
Digital Library
- Kang, B. 2004. S2D2: A framework for scalable and secure optimistic replication. Ph.D. thesis, University of California Berkeley. Google Scholar
Digital Library
- Karger, D., Lehman, E., Leighton, T., Panigrahy, R., Levine, M., and Lewin, D. 1997. Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the world wide web. In Proceedings of the ACM Symposium on Theory of Computing (STOC). Google Scholar
Digital Library
- Kistler, J. and Satyanarayanan, M. 1992. Disconnected operation in the coda file system. ACM Trans. Comput. Syst. 10, 1, 3--5. Google Scholar
Digital Library
- Kotla, R., Alvisi, L., and Dahlin, M. 2007. SafeStore: A durable and practical storage system. In Proceedings of the USENIX Annual Technical Conference (ATC). Google Scholar
Digital Library
- Kubiatowicz, J., Bindel, D., Chen, Y., Czerwinski, S., Eaton, P., Geels, D., Gummadi, R., Rhea, S., Weatherspoon, H., Weimer, W., Wells, C., and Zhao, B. 2000. OceanStore: An architecture for global-scale persistent storage. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Google Scholar
Digital Library
- Lamport, L. 1978. Time, clocks, and the ordering of events in a distributed system. Comm. ACM 21, 7. Google Scholar
Digital Library
- Lamport, L. 1979. How to make a multiprocessor computer that correctly executes multiprocess programs. IEEE Trans. Comput. 28, 9, 690--691. Google Scholar
Digital Library
- Lamport, L., Shostak, R., and Pease, M. 1982. The Byzantine generals problem. ACM Trans. Program. Lang. Syst. 4, 3, 382--401. Google Scholar
Digital Library
- Levin, D., Douceur, J. R., Lorch, J. R., and Moscibroda, T. 2009. TrInc: Small trusted hardware for large distributed systems. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI). Google Scholar
Digital Library
- Li, J. and Mazières, D. 2007. Beyond one-third faulty replicas in Byzantine fault tolerant systems. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI). Google Scholar
Digital Library
- Li, J., Krohn, M., Mazières, D., and Shasha, D. 2004. Secure untrusted data repository (SUNDR). In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). Google Scholar
Digital Library
- Malkhi, D. and Reiter, M. 1998. Byzantine quorum systems. Distrib. Comput. 11, 4, 203--213. Google Scholar
Digital Library
- Maniatis, P. 2003. Historic integrity in distributed systems. Ph.D. thesis, Stanford. Google Scholar
Digital Library
- Mazières, D., Kaminsky, M., Kaashoek, M. F., and Witchel, E. 1999. Separating key management from file system security. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP). Google Scholar
Digital Library
- Miller, R. 2009. FBI siezes servers at Dallas data center. http://www.datacenterknowledge.com/archives/2009/04/03/fbi-seizes-servers-at-dallas-data-center/.Google Scholar
- Nath, S., Yu, H., Gibbons, P. B., and Seshan, S. 2006. Subtleties in tolerating correlated failures in wide-area storage systems. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI). Google Scholar
Digital Library
- News, C. 2002. Victims of lost files out of luck. http://news.cnet.com/Victims-of-lost-files-out-of-luck/2100-1023_3-887849.html.Google Scholar
- Nightingale, E., Veeraraghavan, K., Chen, P., and Flinn, J. 2008. Rethink the sync. ACM Trans. Comput. Syst. 26, 3. Google Scholar
Digital Library
- Oppenheimer, D., Ganapathi, A., and Patterson, D. A. 2003. Why do Internet services fail, and what can be done about it? In Proceedings of the USENIX Symposium on Internet Technologies and Systems (USITS). Google Scholar
Digital Library
- Oprea, A. and Reiter, M. 2006. On consistency of encrypted files. In Proceedings of the International Symposium on Distributed Computing (DISC). Google Scholar
Digital Library
- Parker, D. S., Popek, G. J., Rudisin, G., Stoughton, A., Walker, B. J., Walton, E., Chow, J. M., Kiser, S., Edwards, D., and Kline, C. 1983. Detection of mutual inconsistency in distributed systems. IEEE Trans. Softw. Engin. 9, 3, 240--247. Google Scholar
Digital Library
- Petersen, K., Spreitzer, M. J., Terry, D. B., Theimer, M. M., and Demers, A. J. 1997. Flexible update propagation for weakly consistent replication. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP). Google Scholar
Digital Library
- Pinheiro, E., Weber, W., and Barroso, L. 2007. Failure trends in a large disk drive population. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST). Google Scholar
Digital Library
- Popa, R. A., Lorch, J. R., Molnar, D., Wang, H. J., and Zhuang, L. 2011. Enabling security in cloud storage SLAs with CloudProof. In Proceedings of the USENIX Annual Technical Conference (ATC). Google Scholar
Digital Library
- Prabhakaran, V., Bairavasundaram, L., Agrawal, N., Gunawi, H., Arpaci-Dusseau, A., and Arpaci-Dusseau, R. 2005. IRON file systems. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP). Google Scholar
Digital Library
- Ramasubramanian, V., Rodeheffer, T., Terry, D. B., Walraed-Sullivan, M., Wobber, T., Marshall, C. C., and Vahdat, A. 2009. Cimbiosys: A platform for content-based partial replication. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI). Google Scholar
Digital Library
- Reiher, P., Heidemann, J., Ratner, D., Skinner, G., and Popek, G. 1994. Resolving file conflicts in the Ficus file system. In Proceedings of the USENIX Summer Technical Conference. Google Scholar
Digital Library
- S3 App Catalog. 2011. AWS forum: Customer app catalog. http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=66.Google Scholar
- Schiper, A., Birman, K., and Stephenson, P. 1991. Lightweight causal and atomic group multicast. ACM Trans. Comput. Syst. 9, 3. Google Scholar
Digital Library
- Shah, M., Baker, M., Mogul, J., and Swaminathan, R. 2007. Auditing to keep online storage services honest. In Proceedings of the Workshop on Hot Topics in Operating Systems (HotOS). Google Scholar
Digital Library
- Shraer, A., Cachin, C., Cidon, A., Keidar, I., Michalevsky, Y., and Shaket, D. 2010. Venus: Verification for untrusted cloud storage. In Proceedings of the ACM Cloud Computing Security Workshop (CCSW). Google Scholar
Digital Library
- Singh, A., Fonseca, P., Kuznetsov, P., Rodrigues, R., and Maniatis, P. 2009. Zeno: Eventually consistent Byzantine fault tolerance. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI). Google Scholar
Digital Library
- Strunk, J., Goodson, G., Scheinholtz, M., Soules, C., and Ganger, G. 2000. Self-Securing storage: Protecting data in compromised systems. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). Google Scholar
Digital Library
- Terry, D. B., Demers, A. J., Petersen, K., Spreitzer, M., Theimer, M., and Welch, B. W. 1994. Session guarantees for weakly consistent replicated data. In Proceedings of the International Conference on Parallel and Distributed Information Systems (PDIS). Google Scholar
Digital Library
- Terry, D. B., Theimer, M. M., Petersen, K., Demers, A. J., Spreitzer, M. J., and Hauser, C. H. 1995. Managing update conflicts in Bayou, a weakly connected replicated storage system. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP). Google Scholar
Digital Library
- US. 2005. US Secret Service report on insider attacks. http://www.sei.cmu.edu/about/press/insider-2005.html.Google Scholar
- Vogels, W. 2006. Life is not a state-machine: The long road from research to production. In Proceedings of the ACM Symposium on the Principles of Distributed Computing (PODC). Google Scholar
Digital Library
- White, B., Lepreau, J., Stoller, L., Ricci, R., Guruprasad, S., Newbold, M., Hibler, M., Barb, C., and Joglekar, A. 2002. An integrated experimental environment for distributed systems and networks. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). Google Scholar
Digital Library
- Wobber, T., Rodeheffer, T. L., and Terry, D. B. 2010. Policy-Based access control for weakly consistent replication. In Proceedings of the European Conference on Computer Systems (EuroSys). Google Scholar
Digital Library
- Yahoo. 2011. Yahoo’s offloading of delicious a reminder of cloud risks. http://www.infoworld.com/t/cloud-computing/yahoos-offloading-delicious-reminder-cloud-risks-735.Google Scholar
- Yang, J., Sar, C., and Engler, D. 2006. EXPLODE: A lightweight, general system for finding serious storage system errors. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). Google Scholar
Digital Library
- Yip, A., Chen, B., and Morris, R. 2006. Pastwatch: A distributed version control system. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI). Google Scholar
Digital Library
- Yumerefendi, A. and Chase, J. 2007. Strong accountability for network storage. ACM Trans. Storage 3, 3. Google Scholar
Digital Library
Index Terms
Depot: Cloud Storage with Minimal Trust
Recommendations
Making CRDTs Byzantine fault tolerant
PaPoC '22: Proceedings of the 9th Workshop on Principles and Practice of Consistency for Distributed DataIt is often claimed that Conflict-free Replicated Data Types (CRDTs) ensure consistency of replicated data in peer-to-peer systems. However, peer-to-peer systems usually consist of untrusted nodes that may deviate from the specified protocol (i.e. ...
GRADE: Graceful Degradation in Byzantine Quorum Systems
SRDS '12: Proceedings of the 2012 IEEE 31st Symposium on Reliable Distributed SystemsDistributed storage systems are expected to provide correct services in the presence of Byzantine failures, which do not have any assumptions about the behavior of faulty servers and clients. In designing such systems, we often encounter the paradox of ...
Anticipating Requests to Improve Performance and Reduce Costs in Cloud Storage
Clouds are a suitable place to store data with scalability and financial flexibility. However, it is difficult to ensure the reliability of the data stored in a cloud. Byzantine fault tolerance can improve reliability, but at a high cost. This paper ...








Comments