skip to main content
research-article

The XtreemOS Resource Selection Service

Published:01 December 2012Publication History
Skip Abstract Section

Abstract

Many large-scale utility computing infrastructures comprise heterogeneous hardware and software resources. This raises the need for scalable resource selection services that identify resources that match application requirements. Such a service must provide an efficient lookup in spite of changing resource attributes such as disk size, changing application requirements such as installed software libraries, and changing system composition as resources join or leave. We present a fully decentralized, self-managing Resource Selection Service (RSS) algorithm by which resources autonomously select themselves when their attributes match a query. An application specifies what it expects from a resource by means of a conjunction of (attribute,value-range) pairs, which are matched against the attribute values of resources. The set of search attributes can also be updated online to reflect new requirements. We show that our solution scales in the number of resources and in the number of attributes, while being relatively insensitive to churn and other membership changes like node failures. Our RSS continuously self-adapts its routing structure in response to variations in the distribution of node attributes and queries. We show that this autonomous optimization maintains performance and availability in a long-lived service even when the set of application requirements used to select resources changes.

References

  1. Aberer, K., Datta, A., and Hauswirth, M. 2005. Multifaceted simultaneous load balancing in DHT-based P2P systems: A new game with old balls and bins. In Self-Star Properties in Complex Information Systems. Lecture Notes in Computer Science, vol. 3460. Springer, 373--391. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Albrecht, J., Oppenheimer, D., Vahdat, A., and Patterson, D. A. 2008. Design and implementation trade-offs for wide-area resource discovery. ACM Trans. Interet Technol. 8, 4. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Anderson, D. P. and Reed, K. 2009. Celebrating diversity in volunteer computing. In Proceedings of the Hawaii International Conference on System Sciences. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Anderson, T. and Roscoe, T. 2006. Learning from PlanetLab. In Proceedings of the Workshop on Real, Large Distributed Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Beaumont, O., Kermarrec, A.-M., Marchal, L., and Rivière, E. 2007. VoroNet: A scalable object network based on Voronoi tessellations. In Proceedings of the International Parallel and Distributed Processing Symposium (IPDPS).Google ScholarGoogle Scholar
  6. Bharambe, A. R., Agrawal, M., and Seshan, S. 2004. Mercury: Supporting scalable multi-attribute range queries. In Proceedings of ACM SIGCOMM. 353--366. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Cai, M., Frank, M., Chen, J., and Szekely, P. 2003. MAAN: A multi-attribute addressable network for grid information services. In Proceedings of the International Workshop on Grid Computing. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Coppola, M., Jégou, Y., Matthews, B., Morin, C., Prieto, L. P., Sánchez, O. D., Yang, E., and Yu, H. 2008. Virtual organization support within a grid-wide operating system. IEEE Internet Comput. 12, 2. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Costa, P., Napper, J., Pierre, G., and van Steen, M. 2009. Autonomous resource selection for decentralized utility computing. In Proceedings of the IEEE International Conference on Distributed Computing Systems (ICDCS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. DAS-3. http://www.cs.vu.nl/das3/.Google ScholarGoogle Scholar
  11. Drost, N., van Nieuwpoort, R. V., and Bal, H. 2006. Simple locality-aware co-allocation in peer-to-peer supercomputing. In Proceedings of the Workshop on Global and Peer-2-Peer Computing.Google ScholarGoogle Scholar
  12. Fiscato, M., Costa, P., and Pierre, G. 2008. On the feasibility of decentralized grid scheduling. In Proceedings of the International Workshop on Decentralized Self Management for Grids, P2P, and User Communities (Selfman’08). Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Foster, I. and Iamnitchi, A. 2003. On death, taxes, and the convergence of peer-to-peer and grid computing. In Proceedings of the International Workshop on Peer-to-Peer Systems.Google ScholarGoogle Scholar
  14. Ganesan, P., Yang, B., and Garcia-Molina, H. 2004. One torus to rule them all: Multi-dimensional queries in P2P systems. In Proceedings of the International Workshop on the Web and Databases. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Gupta, A., Sahin, O. D., Agrawal, D., and Abbadi, A. E. 2004. Meghdoot: Content-based publish/subscribe over P2P networks. In Proceedings of the International Middleware Conference. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Iosup, A., Jan, M., Sonmez, O., and Epema, D. H. J. 2007. On the dynamic resource availability in grids. In Proceedings of the International Conference on Grid Computing. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Iosup, A., Li, H., Jan, M., Anoep, S., Dumitrescu, C., Wolters, L., and Epema, D. H. J. 2008. The grid workloads archive. Future Gen. Comput. Syst. 24, 7, 672--686. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Jelasity, M. and Kermarrec, A.-M. 2006. Ordered slicing of very large-scale overlay networks. In Proceedings of P2P. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Jelasity, M., Voulgaris, S., Guerraoui, R., Kermarrec, A.-M., and van Steen, M. 2007. Gossip-based peer sampling. ACM Trans. Comput. Syst. 25, 3. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Jelasity, M., Montresor, A., Jesi, G. P., and Voulgaris, S. The Peersim simulator. http://peersim.sf.net.Google ScholarGoogle Scholar
  21. Ranjan, R., Harwood, A., and Buyya, R. 2008. Peer-to-peer based resource discovery in global grids: A tutorial. IEEE Comm. Surveys Tutorials 10, 2. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Ratnasamy, S., et al. 2001. A scalable content addressable network. In Proceedings of SIGCOMM. 161--172. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Rhea, S., Geels, D., Roscoe, T., and Kubiatowicz, J. 2004. Handling churn in a DHT. In Proceedings of the USENIX Technical Conference. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Sacha, J., Napper, J., Stratan, C., and Pierre, G. 2010. Adam2: Reliable distribution estimation in decentralised environments. In Proceedings of the IEEE International Conference on Distributed Computing Systems (ICDCS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Sanchez-Artigas, M., Garcia-Lopez, P., and Skarmeta, A. F. G. 2008. On the feasibility of dynamic superpeer ratio maintenance. In Proceedings of the Conference on P2P, 333--342. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Saroiu, S., Gummadi, K. P., and Gribble, S. D. 2003. Measuring and analyzing the characteristics of Napster and Gnutella hosts. Multimed. Syst. 9, 2, 170--184. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Schmidt, C. and Parashar, M. 2003. Flexible information discovery in decentralized distributed systems. In Proceedings of the 12th International Symposium on High-Performance Distributed Computing (HPDC). Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Scholtes, I., Botev, J., Hohfeld, A., Schloss, H., and Esch, M. 2008. Awareness-driven phase transitions in very large scale distributed systems. In Proceedings of the 2nd IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO). 25--34. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Schütt, T., et al. 2007. A structured overlay for multi-dimensional range queries. In Proceedings of the International Euro-Par Conference. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Sharma, B., Chudnovsky, V., Hellerstein, J. L., Rifaat, R., and Das, C. R. 2011. Modeling and synthesizing task placement constraints in google compute clusters. In Proceedings of the 2nd ACM Symposium on Cloud Computing (SOCC). Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Snyder, P. L., Greenstadt, R., and Valetto, G. 2009. Myconet: A fungi-inspired model for superpeer-based peer-to-peer overlay topologies. In Proceedings of the 3rd IEEE International Conference on Self-Adaptive and Organizing Systems (SASO). 40--50. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Spence, D. and Harris, T. 2003. Distributed resource discovery in the XenoServer open platform. In Proceedings of the 12th International Symposium on High-Performance Distributed Computing (HPDC). Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Steele, T., Vishnumurthy, V., and Francis, P. 2008. A parameter-free load balancing mechanism for p2p networks. In Proceedings of the International Workshop on Peer-to-Peer Systems (IPTPS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Tanin, E., Harwood, A., and Samet, H. 2007. Using a distributed quadtree in peer-to-peer networks. VLDB J. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. van Renesse, R., Birman, K. P., and Vogels, W. 2003. Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining. ACM Trans. Comput. Syst. 21, 2, 164--206. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. van Steen, M. and Ballintijn, G. 2002. Achieving scalability in hierarchical location services. In Proceedings of the International Computer Software and Applications Conference. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Voulgaris, S. and van Steen, M. 2005. Epidemic-style management of semantic overlays for content-based searching. In Proceedings of the International Euro-Par Conference on Parallel Processing. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. XtremLab Project. http://xw01.lri.fr:4320/.Google ScholarGoogle Scholar
  39. Yalagandula, P. and Dahlin, M. 2004. A scalable distributed information management system. In Proceedings of the ACM SIGCOMM. 379--390. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Zanikolas, S. and Sakellariou, R. 2005. A taxonomy of grid monitoring systems. Future Gen. Comput. Syst. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. The XtreemOS Resource Selection Service

            Recommendations

            Reviews

            Lamine Aouad

            This paper presents a resource selection approach in large-scale distributed systems. The approach is decentralized and self-adapting, and aims to address scalability and resiliency in the face of changes in the number of attributes (representing the characteristics of nodes and the requirements of users and applications). The authors also address changes in the overall system composition (nodes opting in and out of the system). As noted by the authors, the paper is an extended version of a 2009 conference paper [1]. The paper is well written and nicely structured. The authors introduce challenges related to the resource selection problem in terms of scalability, and how to dynamically adapt to changes over time in the composition of the system or in the requirements of applications. They then present the system model, describing how they characterize the nodes and the overall system, and describe base resource discovery (without self-adaptation), including the network topology used, and query routing. The paper includes interesting discussions on dealing with updates and the reconfiguration of the system and providing maintenance in the presence of dynamic changes (churn, failure, and the set of attributes). Self-adaptation can impact or improve the system (in terms of workload, delivery, and so on). The presented approach is based on gossip-based protocols, which are also explained in this paper. Every section includes an evaluation section with simulations or emulated setups (including some that use PeerSim). There are, however, no experiments involving actual deployments, as claimed in the conclusion. Some of the simulation setups are based on data from actual deployment traces from the Berkeley open infrastructure for network computing (BOINC) platform, but these are two completely different things. This is a drawback of the paper, as some of the simulation results are actually confusing and difficult to interpret. Confusing details include, for example, quite random routing overhead versus number of attributes in the system, and inexplicable and quite random drops in the delivery rates versus churn (rates even drop to zero in figure 9(a)). Real-world deployment and experimentation would greatly enhance the value of the work presented in this paper. Online Computing Reviews Service

            Access critical reviews of Computing literature here

            Become a reviewer for Computing Reviews.

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            • Published in

              cover image ACM Transactions on Autonomous and Adaptive Systems
              ACM Transactions on Autonomous and Adaptive Systems  Volume 7, Issue 4
              Special Section: Extended Version of SASO 2011 Best Paper
              December 2012
              167 pages
              ISSN:1556-4665
              EISSN:1556-4703
              DOI:10.1145/2382570
              Issue’s Table of Contents

              Copyright © 2012 ACM

              Publisher

              Association for Computing Machinery

              New York, NY, United States

              Publication History

              • Published: 1 December 2012
              • Accepted: 1 March 2012
              • Revised: 1 November 2011
              • Received: 1 May 2011
              Published in taas Volume 7, Issue 4

              Permissions

              Request permissions about this article.

              Request Permissions

              Check for updates

              Qualifiers

              • research-article
              • Research
              • Refereed

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!