skip to main content
research-article

SecondSite: disaster tolerance as a service

Published:03 March 2012Publication History
Skip Abstract Section

Abstract

This paper describes the design and implementation of SecondSite, a cloud-based service for disaster tolerance. SecondSite extends the Remus virtualization-based high availability system by allowing groups of virtual machines to be replicated across data centers over wide-area Internet links. The goal of the system is to commodify the property of availability, exposing it as a simple tick box when configuring a new virtual machine. To achieve this in the wide area, we have had to tackle the related issues of replication traffic bandwidth, reliable failure detection across geographic regions and traffic redirection over a wide-area network without compromising on transparency and consistency.

References

  1. M. K. Aguilera, W. Chen, and S. Toueg. Heartbeat: A timeout-free failure detector for quiescent reliable communication. Technical report, Ithaca, NY, USA, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. M. K. Aguilera, W. Chen, and S. Toueg. Using the heartbeat failure detector for quiescent reliable communication and consensus in partitionable networks. Theor. Comput. Sci., 220: 3--30, June 1999. ISSN 0304--3975.% http://dx.doi.org/10.1016/S0304--3975(98)00235--7. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. G. Altekar and I. Stoica. ODR: output-deterministic replay for multicore debugging. In SOSP '09: Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, pages 193--206, New York, NY, USA, 2009. ACM. ISBN 978--1--60558--752--3.% http://doi.acm.org/10.1145/1629575.1629594. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In SOSP '03: Proceedings of the nineteenth ACM symposium on Operating systems principles, pages 164--177, New York, NY, USA, 2003. ACM Press. ISBN 1--58113--757--5.% http://doi.acm.org/10.1145/945445.945462. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. rg}wanmigrationR. Bradford, E. Kotsovinos, A. Feldmann, and H. Schiöberg. Live wide-area migration of virtual machines including local persistent state. In VEE '07: Proceedings of the 3rd international conference on Virtual execution environments, pages 169--179, New York, NY, USA, 2007. ACM Press. ISBN 978--1--59593--630--1.% http://doi.acm.org/10.1145/1254810.1254834. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. T. C. Bressoud and F. B. Schneider. Hypervisor-based fault-tolerance. In Proceedings of the Fifteenth ACM Symposium on Operating System Principles, pages 1--11, December 1995. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. C. Brooks. Heroku learns the hard way from amazon ec2 outage. SearchCloudComputing.com, January 2010.Google ScholarGoogle Scholar
  8. T. D. Chandra and S. Toueg. Unreliable failure detectors for reliable distributed systems. J. ACM, 43: 225--267, March 1996. ISSN 0004--5411. http://doi.acm.org/10.1145/226643.226647. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live migration of virtual machines. In Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, Berkeley, CA, USA, 2005. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. A. Cockroft, C. Hicks, and G. Orzell. Lessons Netflix Learned from the AWS Outage. http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.%html, April 2011.Google ScholarGoogle Scholar
  11. B. Cully, G. Lefebvre, D. Meyer, M. Feeley, N. Hutchinson, and A. Warfield. Remus: high availability via asynchronous virtual machine replication. In NSDI'08: Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, pages 161--174, Berkeley, CA, USA, 2008. USENIX Association. ISBN 111--999--5555--22--1. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. D. Dolev, C. Dwork, and L. Stockmeyer. On the minimal synchronism needed for distributed consensus. J. ACM, 34: 77--97, January 1987. ISSN 0004--5411.% http://doi.acm.org/10.1145/7531.7533. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. G. W. Dunlap, S. T. King, S. Cinar, M. A. Basrai, and P. M. Chen. Revirt: Enabling intrusion analysis through virtual-machine logging and replay. In Proceedings of the 5th Symposium on Operating Systems Design & Implementation (OSDI 2002), 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. G. W. Dunlap, D. G. Lucchetti, M. A. Fetterman, and P. M. Chen. Execution replay of multiprocessor virtual machines. In VEE '08: Proceedings of the fourth ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, pages 121--130, New York, NY, USA, 2008. ACM. ISBN 978--1--59593--796--4.% http://doi.acm.org/10.1145/1346256.1346273. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. C. Dwork, N. Lynch, and L. Stockmeyer. Consensus in the presence of partial synchrony. J. ACM, 35: 288--323, April 1988. ISSN 0004--5411.% http://doi.acm.org/10.1145/42282.42283. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. C. Fetzer, M. Raynal, and F. Tronel. An adaptive failure detection protocol. In Proceedings of the 2001 Pacific Rim International Symposium on Dependable Computing, PRDC '01, pages 146--, Washington, DC, USA, 2001. IEEE Computer Society. ISBN 0--7695--1414--6.% URL http://portal.acm.org/citation.cfm?id=882475.883554. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. A. Ganguly, A. Agrawal, P. Boykin, and R. Figueiredo. WOW: Self-Organizing Wide Area Overlay Networks of Virtual Workstations. High-Performance Distributed Computing, International Symposium on, 0: 30--42, 2006.% http://doi.ieeecomputersociety.org/10.1109/HPDC.2006.1652133.Google ScholarGoogle Scholar
  18. D. K. Gifford. Weighted voting for replicated data. In Proceedings of the seventh ACM symposium on Operating systems principles, SOSP '79, pages 150--162, New York, NY, USA, 1979. ACM. ISBN 0--89791-009--5.% http://doi.acm.org/10.1145/800215.806583. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. E. Harney, S. Goasguen, J. Martin, M. Murphy, and M. Westall. The efficacy of live virtual machine migrations over the internet. In Proceedings of the 2nd international workshop on Virtualization technology in distributed computing, VTDC '07, pages 8:1--8:7, New York, NY, USA, 2007. ACM. ISBN 978--1--59593--897--8.% http://doi.acm.org/10.1145/1408654.1408662. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. T. Hirofuchi, H. Nakada, H. Ogawa, S. Itoh, and S. Sekiguchi. A live storage migration mechanism over wan and its performance evaluation. In Proceedings of the 3rd international workshop on Virtualization technologies in distributed computing, VTDC '09, pages 67--74, New York, NY, USA, 2009. ACM. ISBN 978--1--60558--580--2.% http://doi.acm.org/10.1145/1555336.1555348. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. X. Jiang and D. Xu. VIOLIN: Virtual Internetworking on Overlay Infrastructure. In ISPA, pages 937--946, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. C. Labovitz, A. Ahuja, A. Bose, and F. Jahanian. Delayed internet routing convergence. In in Proc. ACM SIGCOMM, pages 175--187, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. D. Lee, B. Wester, K. Veeraraghavan, S. Narayanasamy, P. M. Chen, and J. Flinn. Respec: efficient online multiprocessor replayvia speculation and external determinism. In ASPLOS '10: Proceedings of the fifteenth edition of ASPLOS on Architectural support for programming languages and operating systems, pages 77--90, New York, NY, USA, 2010. ACM. ISBN 978--1--60558--839--1.% http://doi.acm.org/10.1145/1736020.1736031. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. U. F. Minhas, S. Rajagopalan, B. Cully, A. Aboulnaga, K. Salem, and A. Warfield. Remusdb: Transparent high availability for database systems. PVLDB, 4 (11): 738--748, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. C. C. T. A. P. Outage. R. miller. datacenterknowledge.com, May 2010.Google ScholarGoogle Scholar
  26. R. H. Patterson, S. Manley, M. Federwisch, D. Hitz, S. Kleiman, and S. Owara. SnapMirror: File-System-Based Asynchronous Mirroring for Disaster Recovery. In FAST '02: Proceedings of the 1st USENIX Conference on File and Storage Technologies, page 9, Berkeley, CA, USA, 2002. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. P. Reisner and L. Ellenberg. Drbd v8 -- replicated storage with shared disk semantics. In Proceedings of the 12th International Linux System Technology Conference, October 2005.Google ScholarGoogle Scholar
  28. D. J. Scales, M. Nelson, and G. Venkitachalam. The design and evaluation of a practical system for fault-tolerant virtual machines. Technical Report VMWare-RT-2010-001, VMWare, Inc., Palo Alto, CA 94304, May 2010.Google ScholarGoogle Scholar
  29. R. Strom and S. Yemini. Optimistic recovery in distributed systems. ACM Trans. Comput. Syst., 3 (3), 1985.% http://doi.acm.org/10.1145/3959.3962. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. et al.(2011)Svärd, Hudzia, Tordsson, and Elmroth}deltacompressP. Svärd, B. Hudzia, J. Tordsson, and E. Elmroth. Evaluation of delta compression techniques for efficient live migration of large virtual machines. In Proceedings of the 7th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, VEE '11, pages 111--120, New York, NY, USA, 2011. ACM. ISBN 978-1-4503-0687-4. http://doi.acm.org/10.1145/1952682.1952698. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. F. Travostino, P. Daspit, L. Gommans, C. Jog, C. de Laat, J. Mambretti, I. Monga, B. van Oudenaarde, S. Raghunath, and P. Y. Wang. Seamless live migration of virtual machines over the MAN/WAN. Future Gener. Comput. Syst., 22: 901--907, October 2006. ISSN 0167--739X. 10.1016/j.future.2006.03.007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. R. van Renesse, Y. Minsky, and M. Hayden. A gossip-style failure detection service. In Proceedings of the IFIP International Conference on Distributed Systems Platforms and Open Distributed Processing, Middleware '98, pages 55--70, London, UK, 1998. Springer-Verlag. ISBN 1--85233-088-0.% URL http://portal.acm.org/citation.cfm?id=1659232.1659238. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. T. Wood, H. A. Lagar-Cavilla, K. K. Ramakrishnan, P. Shenoy, and J. Van der Merwe. Pipecloud: using causality to overcome speed-of-light delays in cloud-based disaster recovery. In Proceedings of the 2nd ACM Symposium on Cloud Computing, SOCC '11, pages 17:1--17:13, New York, NY, USA, 2011. ACM. ISBN 978--1--4503-0976--9.% http://doi.acm.org/10.1145/2038916.2038933. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. T. Wood, K. K. Ramakrishnan, P. Shenoy, and J. van der Merwe. CloudNet: dynamic pooling of cloud resources by live WAN migration of virtual machines. In Proceedings of the 7th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, VEE '11, pages 121--132, New York, NY, USA, 2011. ACM. ISBN 978--1--4503-0687--4.% http://doi.acm.org/10.1145/1952682.1952699. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. M. Xu, R. Bodik, and M. D. Hill. A "flight data recorder" for enabling full-system multiprocessor deterministic replay. SIGARCH Comput. Archit. News, 31 (2): 122--135, 2003. ISSN 0163--5964.% http://doi.acm.org/10.1145/871656.859633. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. }blktap2 Xen Blktap2 Driver. http://wiki.xensource.com/xenwiki/blktap2 \natexlaba.Google ScholarGoogle Scholar
  37. }ebs-outage Summary of the Amazon EC2 and Amazon RDS Service Disruption in the US East Region. http://aws.amazon.com/message/65648/ \natexlabb.Google ScholarGoogle Scholar
  38. }everrunDR Marathon Technologies: everRun DR. http://www.marathontechnologies.com/ \natexlabc.Google ScholarGoogle Scholar
  39. }gae Google app engine. http://code.google.com/appengine/ \natexlabd.Google ScholarGoogle Scholar
  40. }spot-instances Amazon EC2 Spot Instances. http://aws.amazon.com/ec2/spot-instances/ \natexlabf.Google ScholarGoogle Scholar
  41. }vmware-ping VMware KB: Configuring Split-Brain Avoidance in a WAN. http://kb.vmware.com/kb/1008606 \natexlabg.Google ScholarGoogle Scholar
  42. }dvdstore Dell DVD Store Database Test Suite. http://www.delltechcenter.com/page/DVDGoogle ScholarGoogle Scholar
  43. Store \natexlabh.Google ScholarGoogle Scholar
  44. }specweb05 SPECweb2005. http://www.spec.org/web2005/ \natexlabi.Google ScholarGoogle Scholar
  45. }xentop Xentop. http://linux.die.net/man/1/xentop \natexlabj.Google ScholarGoogle Scholar

Index Terms

  1. SecondSite: disaster tolerance as a service

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM SIGPLAN Notices
        ACM SIGPLAN Notices  Volume 47, Issue 7
        VEE '12
        July 2012
        229 pages
        ISSN:0362-1340
        EISSN:1558-1160
        DOI:10.1145/2365864
        Issue’s Table of Contents
        • cover image ACM Conferences
          VEE '12: Proceedings of the 8th ACM SIGPLAN/SIGOPS conference on Virtual Execution Environments
          March 2012
          248 pages
          ISBN:9781450311762
          DOI:10.1145/2151024

        Copyright © 2012 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 3 March 2012

        Check for updates

        Qualifiers

        • research-article

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!