skip to main content
research-article

High-bandwidth data dissemination for large-scale distributed systems

Published:10 March 2008Publication History
Skip Abstract Section

Abstract

This article focuses on the multireceiver data dissemination problem. Initially, IP multicast formed the basis for efficiently supporting such distribution. More recently, overlay networks have emerged to support point-to-multipoint communication. Both techniques focus on constructing trees rooted at the source to distribute content among all interested receivers. We argue, however, that trees have two fundamental limitations for data dissemination. First, since all data comes from a single parent, participants must often continuously probe in search of a parent with an acceptable level of bandwidth. Second, due to packet losses and failures, available bandwidth is monotonically decreasing down the tree.

To address these limitations, we present Bullet, a data dissemination mesh that takes advantage of the computational and storage capabilities of end hosts to create a distribution structure where a node receives data in parallel from multiple peers. For the mesh to deliver improved bandwidth and reliability, we need to solve several key problems: (i) disseminating disjoint data over the mesh, (ii) locating missing content, (iii) finding who to peer with (peering strategy), (iv) retrieving data at the right rate from all peers (flow control), and (v) recovering from failures and adapting to dynamically changing network conditions. Additionally, the system should be self-adjusting and should have few user-adjustable parameter settings. We describe our approach to addressing all of these problems in a working, deployed system across the Internet. Bullet outperforms state-of-the-art systems, including BitTorrent, by 25-70% and exhibits strong performance and reliability in a range of deployment settings. In addition, we find that, relative to tree-based solutions, Bullet reduces the need to perform expensive bandwidth probing.

References

  1. Ahlswede, R., Cai, N., Li, S.-Y. R., and Yeung, R. W. 2000. Network information flow. IEEE Trans. Inform. Theor. 46, 4 (Jul.), 1204--1216. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Albrecht, J., Tuttle, C., Snoeren, A. C., and Vahdat, A. 2006. PlanetLab application management using Plush. ACM SIGOPS Operat. Syst. Rev. 40 1 (Jan.), 33--40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Andersen, D. G., Balakrishnan, H., Kaashoek, M. F., and Morris, R. 2001. Resilient overlay networks. In Proceedings of the Symposium on Operating Systems Principles. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Banerjee, S., Bhattacharjee, B., and Kommareddy, C. 2002. Scalable application layer multicast. In Proceedings of ACM SIGCOMM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Bharambe, A., Herley, C., and Padmanabhan, V. 2006. Analyzing and improving a BitTorrent network's performance mechanisms. In Proceedings of IEEE INFOCOM.Google ScholarGoogle Scholar
  6. Bickson, D. and Malkhi, D. 2005. The Julia content distribution network. In Proceedings of the Second Workshop on Real, Large Distributed Systems (WORLDS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Birman, K., Hayden, M., Ozkasap, O., Xiao, Z., Budiu, M., and Minsky, Y. 1999. Bimodal multicast. ACM Trans. Comput. Syst. 17, 2 (May), 41--88. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Bloom, B. 1970. Space/time trade-offs in hash coding with allowable errors. Commun. ACM 13, 7 (Jul.), 422--426. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Broder, A. 1997. On the resemblance and containment of documents. In Proceedings of the Conference on Compression and Complexity of Sequences (SEQUENCES'97). Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Byers, J. W., Considine, J., Mitzenmacher, M., and Rost, S. 2002. Informed content delivery across adaptive overlay networks. In Proceedings of ACM SIGCOMM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Byers, J. W., Luby, M., Mitzenmacher, M., and Rege, A. 1998. A digital fountain approach to reliable distribution of bulk data. In Proceedings of ACM SIGCOMM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Calvert, K., Doar, M., and Zegura, E. W. 1997. Modeling Internet topology. IEEE Commun. Mag. 35, 6 (Jun.), 160--163. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Castro, M., Druschel, P., Kermarrec, A.-M., Nandi, A., Rowstron, A., and Singh, A. 2003. Splitstream: High-bandwidth content distribution in cooperative environments. In Proceedings of the 19th ACM Symposium on Operating System Principles. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Chang, H., Govindan, R., Jamin, S., Shenker, S., and Willinger, W. 2002. Towards capturing representative AS-level Internet topologies. In Proceedings of ACM SIGMETRICS. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Cherkasova, L. and Lee, J. 2003. FastReplica: Efficient large file distribution within content delivery networks. In Proceedings of the 4th USENIX Symposium on Internet Technologies and Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Chou, P. A., Wu, Y., and Jain, K. 2003. Practical network coding. In Proceedings of the Allerton Conference on Communication, Control, and Computing.Google ScholarGoogle Scholar
  17. Cohen, B. 2003. Incentives build robustness in bittorrent. In Proceedings of the First Workshop on the Economics of Peer-to-Peer Systems.Google ScholarGoogle Scholar
  18. Cohen, R. and Kaempfer, G. 2001. A unicast-based approach for streaming multicast. In Proceeding of INFOCOM. 440--448.Google ScholarGoogle Scholar
  19. Dabek, F., Li, J., Sit, E., Kaashoek, F., Morris, R., and Blake, C. 2004. Designing a DHT for low latency and high throughput. In Proceedings of the USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI, San Francisco, CA). Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Dalal, Y. K. and Metcalfe, R. M. 1978. Reverse path forwarding of broadcast packets. Commun. ACM 21, 12, 1040--1048. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Deering, S. E. 1991. Multicast routing in a datagram internetwork. Ph.D. dissertation. Stanford University, Stanford, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Demers, A., Greene, D., Hauser, C., Irish, W., Larson, J., Shenker, S., Sturgis, H., Swinehart, D., and Terry, D. 1987. Epidemic algorithms for replicated database maintenance. In Proceedings of the Sixth Symposium on Principles of Distributed Computing. 1--12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Eugster, P., Guerraoui, R., Handurukande, S., Kouznetsov, P., and Kermarrec, A.-M. 2003. Lightweight probabilistic broadcast. ACM Trans. Comput. Syst. 21, 4, 341--374. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Eugster, P., Handurukande, S., Guerraoui, R., Kermarrec, A.-M., and Kouznetsov, P. 2001. Lightweight probabilistic broadcast. In Proceedings of The International Conference on Dependable Systems and Networks (DSN). Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Floyd, S., Handley, M., Padhye, J., and Widmer, J. 2000. Equation-based congestion control for unicast applications. In Proceedings of ACM SIGCOMM (Stockholm, Sweden). 43-- 56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Floyd, S., Jacobson, V., Liu, C.-G., McCanne, S., and Zhang, L. 1997. A reliable multicast framework for light-weight sessions and application level framing. IEEE / ACM Trans. Network. 5, 6, 784--803. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Gkantsidis, C., Miller, J., and Rodriguez, P. 2006. Anatomy of a p2p content distribution system with network coding. In Proceedings of the Second International Peer to Peer Symposium (IPTPS).Google ScholarGoogle Scholar
  28. Gkantsidis, C. and Rodriguez, P. R. 2005. Network coding for large scale content distribution. In Proceedings of IEEE INFOCOM.Google ScholarGoogle Scholar
  29. Goyal, V. K. 2001. Multiple description coding: Compression meets the network. IEEE Signal Process. Mag. 18, 5, 74--93.Google ScholarGoogle ScholarCross RefCross Ref
  30. Hu, N. and Steenkiste, P. 2003. Evaluation and characterization of available bandwidth probing techniques. IEEE J. Select are Commun. 21, 6, 879--895. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Hua Chu, Y., Ganjam, A., Ng, T. S. E., Rao, S. G., Sripanidkulchai, K., Zhan, J., and Zhang, H. 2004. Early experience with an Internet broadcast system based on overlay multicast. In Proceedings of the USENIX 2004 Annual Technical Conference. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Hua Chu, Y., Rao, S., and Zhang, H. 2000. A case for end system multicast. In Proceedings of the ACM Sigmetrics 2000 International Conference on Measurement and Modeling of Computer Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Hua Chu, Y., Rao, S. G., Seshan, S., and Zhang, H. 2001. Enabling Conferencing Applications on the Internet using an overlay multicast architecture. In Proceedings of ACM SIGCOMM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. InformationWeek. 2004. Go online to http://informationweek.com/story/showArticle.jhtml? articleID=50900297.Google ScholarGoogle Scholar
  35. Jain, K., Lovasz, L., and Chou, P. A. 2005. Building scalable and robust peer-to-peer overlay networks for broadcasting using network coding. In Proceedings of the ACM Conference on Principles Of Distributed Computing (PODC). Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Jain, K., Mahdian, M., and Salavatipour, M. R. 2003. Packing Steiner trees. In Proceeding of the 14th ACMSIAM Symposium on Discrete Algorithms. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Jain, M. and Dovrolis, C. 2002. End-to-end available bandwidth: Measurement methodology, dynamics, and relation with TCP Throughput. In Proceedings of ACM SIGCOMM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Jannotti, J., Gifford, D. K., Johnson, K. L., Kaashoek, M. F., and James W. O'Toole, J. 2000. Overcast: Reliable multicasting with an overlay network. In Proceedings of the Conference on Operating Systems Design and Implementation (OSDI). Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Katabi, D., Handley, M., and Rohrs, C. 2002. Internet congestion control for high bandwidth-delay product networks. In Proceedings of ACM SIGCOMM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Kim, M. S., Lam, S. S., and Lee, D.-Y. 2002. Optimal distribution tree for Internet streaming media. Tech. rep. TR-02-48. Department of Computer Sciences, University of Texas at Austin, Austin, TX.Google ScholarGoogle Scholar
  41. Kostić, D., Braud, R., Killian, C., Vandekieft, E., Anderson, J. W., Snoeren, A. C., and Vahdat, A. 2005. Maintaining high bandwidth under dynamic network conditions. In Proceedings of the USENIX 2005 Annual Technical Conference. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Kostić, D., Rodriguez, A., Albrecht, J., Bhirud, A., and Vahdat, A. 2003a. Using random subsets to build scalable network services. In Proceedings of the USENIX Symposium on Internet Technologies and Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Kostić, D., Rodriguez, A., Albrecht, J., and Vahdat, A. 2003b. Bullet: High bandwidth data dissemination using an overlay mesh. In Proceedings of the 19th ACM Symposium on Operating System Principles. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Krohn, M. N., Freedman, M. J., and Mazieres, D. 2004. On-the-fly verification of rateless erasure codes for efficient content distribution. In Proceedings of the IEEE Symposium on Security and Privacy (Oakland, CA).Google ScholarGoogle Scholar
  45. Legout, A., Urvoy-Keller, G., and Michiardi, P. 2006. Rarest first and choke algorithms are enough. In IMC'06: Proceedings of the 6th ACM SIGCOMM Conference on Internet Measurement. 203--216. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Luby, M. 2002. LT codes. In Proceedings of 43rd Annual IEEE Symposium on Foundations of Computer Science. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Luby, M. G., Mitzenmacher, M., Shokrollahi, M. A., Spielman, D. A., and Stemann, V. 1997. Practical loss-resilient codes. In Proceedings of the 29th Annual ACM Symposium on the Theory of Computing (STOC '97). Ass. Comp. Mach. Press, New York, 150-- 159. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Massoulie, L. and Vojnovic, M. 2005. Coupon replication systems. In Proceedings of ACM SIGMETRICS. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Maymounkov, P. and Mazieres, D. 2003. Rateless codes and big downloads. In Proceedings of the Second International Peer to Peer Symposium (IPTPS).Google ScholarGoogle Scholar
  50. Padhye, J., Firoiu, V., Towsley, D., and Krusoe, J. 1998. Modeling TCP throughput: A simple model and its empirical validation. In Proceedings of ACM SIGCOMM (Vancouver, Canada). 303--314. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Padmanabhan, V. N., Qiu, L., and Wang, H. J. 2003. Server-based inference of Internet link lossiness. In Proceedings of the IEEE INFOCOM. (San Francisco, CA).Google ScholarGoogle Scholar
  52. Padmanabhan, V. N., Wang, H. J., and Chou, P. A. 2003b. Resilient peer-to-peer streaming. In Proceedings of the 11th ICNP (Atlanta, GA). Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Padmanabhan, V. N., Wang, H. J., Chou, P. A., and Sripanidkulchai, K. 2002. Distributing streaming media content using cooperative networking. In Proceedings of ACM/IEEE NOSSDAV. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Pai, V., Kumar, K., Tamilmani, K., Sambamurthy, V., and Mohr, A. E. 2005. Chainsaw: Eliminating trees from overlay multicast. In Proceedings of I PT PS. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Park, K. and Pai, V. S. 2006. Scale and performance in the CoBlitz large-file distribution service. In Proceedings of the 3rd USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI). Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Peterson, L., Anderson, T., Culler, D., and Roscoe, T. 2002. A blueprint for introducing disruptive technology into the Internet. In Proceedings of ACM HotNets-I.Google ScholarGoogle Scholar
  57. Piatek, M., Isdal, T., Anderson, T., Krishnamurthy, A., and Venkataramani, A. 2007. Do incentives build robustness in BitTorrent? In Proceedings of the 4th USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI). Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Prim, R. C. 1957. Shortest connection networks and some generalizations. In Bell Syst. Tech. J. 36, 6, 1389--1401.Google ScholarGoogle ScholarCross RefCross Ref
  59. Qiu, D. and Srikant, R. 2004. Modeling and performance analysis of Bittorrentlike peer-to-peer networks. In Proceedings of ACM SIGCOMM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Rodriguez, A., Killian, C., Bhat, S., Kostić, D., and Vahdat, A. 2004a. MACEDON: Methodology for automatically creating, evaluating, and designing overlay networks. In Proceedings of the USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI, San Francisco, CA). Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Rodriguez, A., Kostić, D., and Vahdat, A. 2004b. Scalability in adaptive multi-metric overlays. In Proceedings the 24th International Conference on Distributed Computing Systems (ICDCS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Rowstron, A., Kermarrec, A.-M., Castro, M., and Druschel, P. 2001. SCRIBE: The design of a large-scale event notification infrastructure. In Proceedings of the Third International Workshop on Networked Group Communication. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Saroiu, S., Gummadi, K. P., Dunn, R. J., Gribble, S. D., and Levy, H. M. 2002. An analysis of Internet content delivery systems. In Proceedings of 5th Symposium on Operating Systems Design and Implementation (OSDI). Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Savage, S., Collins, A., Hoffman, E., Snell, J., and Anderson, T. 1999. The end-to-end effects of Internet path selection. In Proceedings of ACM SIGCOMM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Sherwood, R., Braud, R., and Bhattacharjee, B. 2004. Slurpie: A cooperative bulk data transfer protocol. In Proceedings of IEEE INFOCOM.Google ScholarGoogle Scholar
  66. Shokrollahi, A. 2003. Raptor codes. Tech. rep. DF2003-06-001. Digital Fountain, Inc.Google ScholarGoogle Scholar
  67. Snoeren, A. C., Conley, K., and Gifford, D. K. 2001. Mesh-based content routing using XML. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP). Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. Sung, Y.-W., Bishop, M., and Rao, S. 2006. Enabling contribution awareness in an overlay broadcasting system. In Proceedings of ACM SIGCOMM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. Vahdat, A., Yocum, K., Walsh, K., Mahadevan, P., Kostić, D., Chase, J., and Becker, D. 2002. Scalability and accuracy in a large-scale network emulator. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI). Google ScholarGoogle ScholarDigital LibraryDigital Library
  70. Wang, L., Park, K., Pang, R., Pai, V. S., and Peterson, L. 2004. Reliability and Security in the CoDeeN content distribution network. In Proceedings of the USENIX 2004 Annual Technical Conference. Google ScholarGoogle ScholarDigital LibraryDigital Library
  71. Young, A., Chen, J., Ma, Z., Krishnamurthy, A., Peterson, L., and Wang, R. Y. 2004. Overlay mesh construction using interleaved spanning trees. In Proceedings of IEEE INFOCOM.Google ScholarGoogle Scholar
  72. Zhang, X., Liuy, J., Liz, B., and Yum, T.-S. P. 2005. CoolStreaming/DONet: A data-driven overlay network for efficient live media streaming. In Proceedings of IEEE INFOCOM.Google ScholarGoogle Scholar

Index Terms

  1. High-bandwidth data dissemination for large-scale distributed systems

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Computer Systems
          ACM Transactions on Computer Systems  Volume 26, Issue 1
          February 2008
          153 pages
          ISSN:0734-2071
          EISSN:1557-7333
          DOI:10.1145/1328671
          Issue’s Table of Contents

          Copyright © 2008 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 10 March 2008
          • Accepted: 1 November 2007
          • Revised: 1 June 2007
          • Received: 1 March 2006
          Published in tocs Volume 26, Issue 1

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!