skip to main content
research-article

Reducing Repair Traffic in P2P Backup Systems: Exact Regenerating Codes on Hierarchical Codes

Published:01 October 2011Publication History
Skip Abstract Section

Abstract

Peer to peer backup systems store data on “unreliable” peers that can leave the system at any moment. In this case, the only way to assure durability of the data is to add redundancy using either replication or erasure codes. Erasure codes are able to provide the same reliability as replication requiring much less storage space. Erasure coding breaks the data into blocks that are encoded and then stored on different nodes. However, when storage nodes permanently abandon the system, new redundant blocks must be created, which is referred to as repair. For “classical” erasure codes, generating a new block requires the transmission of k blocks over the network, resulting in a high repair traffic. Recently, two new classes of erasure codes, Regenerating Codes and Hierarchical Codes, have been proposed that significantly reduce the repair traffic. Regenerating Codes reduce the amount of data uploaded by each peer involved in the repair, while Hierarchical Codes reduce the number of nodes participating in the repair. In this article we propose to combine these two codes to devise a new class of erasure codes called ER-Hierarchical Codes that combine the advantages of both.

References

  1. Adya, A., Bolosky, W., Castro, M., Cermak, G., Chaiken, R., Douceur, J., Howell, J., Lorch, J., Theimer, M., and Wattenhofer, R. 2002. Farsite: Federated, available and reliable storage for an incompletely trusted environment. In Proceedings of the 5th USENIX Symposium on Operating Systems Design and Implementation. 1--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Blake, C. and Rodrigues, R. 2003. High availability, scalable storage, dynamic peer networks: Pick two. In Proceedings of 9th the Usenix Workshop on Hot Topics in Operating Systems. Lihue, Hawaii, 1--6. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Chun, B.-G., Dabek, F., Haeberlen, A., Sit, E., Weatherspoon, H., Kaashoek, M., and Kubiatowicz, J. 2006a. Efficient replica maintenance for distributed storage systems. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Chun, B.-G., Dabek, F., Haeberlen, A., Sit, E., Weatherspoon, H., M. Frans Kaashoek, J. K., and Morris, R. 2006b. Proactive replication for data durability. In Proceedings of the USENIX International Workshop on Peer-to-Peer Systems.Google ScholarGoogle Scholar
  5. Chun, B.-G., Ratnasamy, S., and Kohler, E. 2008. Netcomplex: A complexity metric for networked system designs. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation. 393--406. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Dabek, F., Kaashoek, M. F., Karger, D. R., Morris, R., and Stoica, I. 2001. Wide-area cooperative storage with CFS. In Proceedings of the ACM Symposium on Operating Systems Principles. 202--215. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Dimakis, A. G., Godfrey, P. B., Wu, Y., Wainwright, M. J., and Ramchandran, K. 2010a. Network coding for distributed storage systems. IEEE Trans. Inform. Theory 56, 9, 4539--4551. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Dimakis, A. G., Ramchandran, K., Wu, Y., and Suh, C. 2010b. A survey on network codes for distributed storage. CoRR. http://arxiv.org/abs/1004.4438.Google ScholarGoogle Scholar
  9. Dischinger, M., Gummadi, K. P., Haeberlen, A., and Saroiu, S. 2007. Characterizing residential broadband networks. In Proceedings of the ACM Internet Measurement Conference (IMC). Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Druschel, P. and Rowstron, A. 2001. PAST: A large-scale, persistent peer-to-peer storage utility. In Proceedings of the USENIX Workshop on Hot Topics in Operating Systems. 75. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Duminuco, A. 2009. Data redundancy and maintenance for peer-to-peer file backup systems. Ph.D. thesis, Telecom ParisTech.Google ScholarGoogle Scholar
  12. Duminuco, A. and Biersack, E. 2009a. A practical study of regenerating codes for peer-to-peer backup systems. In Proceedings of the IEEE International Conference on Distributed Computing Systems. 376--384. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Duminuco, A. and Biersack, E. W. 2009b. Hierarchical codes: A flexible trade-off for erasure codes. J. Peer-to-Peer Netw. Appl. 2, 52--66.Google ScholarGoogle Scholar
  14. Duminuco, A., Biersack, E., and En-Najjary, T. 2007. Proactive replication in distributed storage systems using machine availability estimation. In Proceedings of the ACM Conference on emerging Networking Experiments and Technologies. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Godfrey, B. 2006. Repository of Availability Traces. http://www.cs.berkeley.edu/ pbg/availability/.Google ScholarGoogle Scholar
  16. Haeberlen, A., Mislove, A., and Druschel, P. 2005. Glacier: Highly durable, decentralized storage despite massive correlated failures. In Proceedings of the 2nd USENIX Symposium on Networked Systems Design and Implementation. 143--158. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. HardDriver. 2010. Cost of hard drive storage space. http://ns1758.ca/winch/winchest.html.Google ScholarGoogle Scholar
  18. Kondo, D., Javadi, B., Iosup, A., and Epema, D. 2010. The failure trace archive: Enabling comparative analysis of failures in diverse distributed systems. Proceedings of the IEEE/ACM International Conference on Cluster, Cloud and Grid Computing. 398--407. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Kubiatowicz, J., Bindel, D., Chen, Y., Czerwinski, S., Eaton, P., Geels, D., Gummadi, R., Rhea, S., Weatherspoon, H., Weimer, W., Wells, C., and Zhao, B. 2000. Oceanstore: An architecture for global-scale persistent storage. In Proceedings of the 9th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 190--201. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Lin, W. K., Chiu, D. M., and Lee, Y. B. 2004. Erasure code replication revisited. In Proceedings of the IEEE International Conference on Peer-to-Peer Computing. 90--97. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Mitzenmacher, M. 2004. Digital fountains: A survey and look forward. In Proceedings of the IEEE Information Theory Workshop. 271--276.Google ScholarGoogle ScholarCross RefCross Ref
  22. Pamies-Juarez, L. and Biersack, E. 2011. Cost analysis of redundancy schemes for distributed storage systems. CoRR. http://arxiv.org/abs/1103.2662.Google ScholarGoogle Scholar
  23. Pamies-Juarez, L., Garcia-Lopez, P., and Sanchez-Artigas, M. 2010. Availability and Redundancy in Harmony: Measuring Retrieval Times in P2P Storage Systems. In Proceedings of the IEEE International Conference on Peer-to-Peer Computing. 1--10.Google ScholarGoogle Scholar
  24. Plank, J. S. 2005. Erasure codes for storage applications. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST’05).Google ScholarGoogle Scholar
  25. Plank, J. S. 2008. A new MDS erasure code for RAID-6. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST’08).Google ScholarGoogle Scholar
  26. Rashmi, K. V., Shah, N. B., Kumar, P. V., and Ramchandran, K. 2009. Explicit construction of optimal exact regenerating codes for distributed storage. In Proceedings of the Annual Allerton Conference on Communication, Control, and Computing. 1243--1249. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Rashmi, K. V., Shah, N. B., and Kumar, P. V. 2010. Optimal exact-regenerating codes for distributed storage at the msr and mbr points via a product-matrix construction. CoRR abs/1005.4178. http://arxiv.org/abs/1005.4178. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Richardson, T. and Urbanke, R. 2008. Modern Coding Theory. Cambridge University Press, Cambridge, UK. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Rodrigues, R. and Liskov, B. 2005. High availability in DHTs: Erasure coding vs. replication. In Proceedings of the USENIX International Workshop on Peer-To-Peer Systems. New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Shah, N. B., V., R. K., Kumar, P. V., and Ramachandran, K. 2010. Interference alignment in regenerating codes for distributed storage: Necessity and code constructions. CoRR (submitted to the IEEE Trans. Inform. Theory). http://arxiv.org/abs/1005.1634.Google ScholarGoogle Scholar
  31. Shojania, H. and Li, B. 2009. Pushing the envelope: Extreme network coding on the GPU. In Proceedings of the 29th International Conference on Distributed Computing Systems (ICDCS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Suh, C. and Ramchandran, K. 2010. Exact regeneration codes for distributed storage repair using interference alignment. http://arxiv.org/abs/1001.0107.Google ScholarGoogle Scholar
  33. Weatherspoon, H. and Kubiatowicz, J. D. 2002. Erasure coding vs. replication: A quantitative comparison. In Proceedings of the USENIX International Workshop on Peer-To-Peer Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Wu, Y. and Dimakis, A. G. 2009. Reducing repair traffic for erasure coding-based storage via interference alignment. In Proceedings of the IEEE International Symposium on Information Theory. 2276--2280. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Wuala. 2010. Official website. http://www.wuala.com.Google ScholarGoogle Scholar
  36. Zhang, Z., Deshpande, A., Ma, X., Thereska, E., and Narayanan, D. 2010. Does erasure coding have a role to play in my data center? Microsoft Research Tech. rep. MSR-TR-2010-52.Google ScholarGoogle Scholar

Index Terms

  1. Reducing Repair Traffic in P2P Backup Systems: Exact Regenerating Codes on Hierarchical Codes

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!