skip to main content
research-article

A Game-Theoretic Approach for Elastic Distributed Data Stream Processing

Published:06 June 2016Publication History
Skip Abstract Section

Abstract

Distributed data stream processing applications are structured as graphs of interconnected modules able to ingest high-speed data and to transform them in order to generate results of interest. Elasticity is one of the most appealing features of stream processing applications. It makes it possible to scale up/down the allocated computing resources on demand in response to fluctuations of the workload. On clouds, this represents a necessary feature to keep the operating cost at affordable levels while accommodating user-defined QoS requirements. In this article, we study this problem from a game-theoretic perspective. The control logic driving elasticity is distributed among local control agents capable of choosing the right amount of resources to use by each module. In a first step, we model the problem as a noncooperative game in which agents pursue their self-interest. We identify the Nash equilibria and we design a distributed procedure to reach the best equilibrium in the Pareto sense. As a second step, we extend the noncooperative formulation with a decentralized incentive-based mechanism in order to promote cooperation by moving the agreement point closer to the system optimum. Simulations confirm the results of our theoretical analysis and the quality of our strategies.

References

  1. Joao Gama and Mohamed Medhat Gaber. 2007. Learning from Data Streams: Processing Techniques in Sensor Networks (1 ed.). Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. 2015. FastFlow (FF). Retrieved from http://http://calvados.di.unipi.it/fastflow/.Google ScholarGoogle Scholar
  3. M. Akdere, C. Ç. Bilgin, O. Gerdaneri, I. Korpeoglu, Ö. Ulusoy, and U. Çetintemel. 2006. A comparison of epidemic algorithms in wireless sensor networks. Comput. Commun. 29, 13--14 (Aug. 2006), 2450--2457. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. T. Alpcan and L. Pavel. 2009. Nash equilibrium design and optimization. In Proceedings of the International Conference on Game Theory for Networks, 2009 (GameNets’09). 164--170. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. H. Andrade, B. Gedik, and D. Turaga. 2014. Fundamentals of Stream Processing. Cambridge University Press. Cambridge Books Online. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. H. Andrade, B. Gedik, K. L. Wu, and P. S. Yu. 2011. Processing high data rate streams in system S. J. Parallel Distrib. Comput. 71, 2 (Feb. 2011), 145--156. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. D. Ardagna, B. Panicucci, and M. Passacantando. 2013. Generalized Nash equilibria for the service provisioning problem in cloud systems. IEEE Trans. Services Comput. 6, 4 (Oct. 2013), 429--442. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. S. Babu and J. Widom. 2001. Continuous queries over data streams. SIGMOD Rec. 30, 3 (Sept. 2001), 109--120. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. C. Bertolli, G. Mencagli, and M. Vanneschi. 2009. Adaptivity in risk and emergency management applications on pervasive grids. In Proceedings of the 2009 10th International Symposium on Pervasive Systems, Algorithms, and Networks (ISPAN). 550--555. DOI:http://dx.doi.org/10.1109/I-SPAN.2009.92 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. C. Bertolli, G. Mencagli, and M. Vanneschi. 2010. A cost model for autonomic reconfigurations in high-performance pervasive applications. In Proceedings of the 4th ACM International Workshop on Context-Awareness for Self-Managing Systems (CASEMANS’10). ACM, New York, NY, Article 3, 10 pages. DOI:http://dx.doi.org/10.1145/1858367.1858370 Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. A. Buades, B. Coll, and J.-M. Morel. 2005. A non-local algorithm for image denoising. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) - Volume 2. IEEE Computer Society, Washington, DC, 60--65. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. R. C. Fernandez, M. Migliavacca, E. Kalyvianaki, and P. Pietzuch. 2013. Integrating scale out and fault tolerance in stream processing using operator state management. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data (SIGMOD’13). ACM, New York, NY, 725--736. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. S. Chaisiri, B.-S. Lee, and D. Niyato. 2012. Optimization of resource provisioning cost in cloud computing. IEEE Trans. Serv. Comput. 5, 2 (April 2012), 164--177. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. G. Chalkiadakis, E. Elkind, and M. Wooldridge. 2012. Cooperative game theory: Basic concepts and computational challenges. IEEE Intelligent Syst. 27, 3 (2012), 86--90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. G. Cugola and A. Margara. 2012. Processing flows of information: From data stream to complex event processing. ACM Comput. Surv. 44, 3, Article 15 (June 2012), 62 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. P. Dubey. 1986. Inefficiency of Nash equilibria. Math. Oper. Res. 11, 1 (1986), 1--8. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. B. Espinasse, G. Picolet, and E. Chouraqui. 1997. Negotiation support systems: A multi-criteria and multi-agent approach. Eur. J. Oper. Res. 103, 2 (1997), 389--409.Google ScholarGoogle ScholarCross RefCross Ref
  18. R. Gayathri and R. S. Sabeenian. 2013. A performance analysis of efficient schemes and algorithms in image denoising procedures. In Proceedings of the 2013 International Conference on Computer Communication and Informatics (ICCCI’13). 1--5.Google ScholarGoogle Scholar
  19. B. Gedik, S. Schneider, M. Hirzel, and K.-L. Wu. 2014. Elastic scaling for data stream processing. IEEE Parallel Distr. Syst. 25, 6 (June 2014), 1447--1463. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. J. Geibig and D. Bradler. 2010. Self-organized aggregation in irregular wireless networks. In Proceedings of the 2010 IFIP Wireless Days (WD’10), 1--7.Google ScholarGoogle Scholar
  21. A. Gohad, N. C. Narendra, and P. Ramachandran. 2013. Cloud pricing models: A survey and position paper. In 2013 IEEE International Conference on Cloud Computing in Emerging Markets (CCEM’13). 1--8.Google ScholarGoogle Scholar
  22. E. R. Gomes, Q. B. Vo, and R. Kowalczyk. 2012. Pure exchange markets for resource sharing in federated clouds. Concurrency Comput. Pract. Exper. 24, 9 (2012), 977--991. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. H. González-Vélez and M. Leyton. 2010. A survey of algorithmic skeleton frameworks: High-level structured parallel programming enablers. Softw. Pract. Exper. 40, 12 (Nov. 2010), 1135--1160. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. V. Gulisano, R. Jimenez-Peris, M. Patino-Martinez, C. Soriente, and P. Valduriez. 2012. StreamCloud: An elastic and scalable data streaming system. IEEE Trans. Parallel Distrib. Syst. 23, 12 (Dec. 2012), 2351--2365. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. T. Heinze, Z. Jerzak, G. Hackenbroich, and C. Fetzer. 2014. Latency-aware elastic scaling for distributed data stream processing systems. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems (DEBS’14). ACM, New York, NY, 13--22. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. W. Hummer, B. Satzger, and S. Dustdar. 2013. Elastic stream processing in the cloud. Wiley Interdisc. Rev. Data Mining Knowl. Discov. 3, 5 (2013), 333--345.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Amazon Inc. 2008. Amazon Elastic Compute Cloud (Amazon EC2). Amazon Inc. Retrieved from http://aws.amazon.com/ec2/#pricing. http://aws.amazon.com/ec2/#pricing.Google ScholarGoogle Scholar
  28. D. Kempe, A. Dobra, and J. Gehrke. 2003. Gossip-based computation of aggregate information. In Proceedings of the 44th Annual IEEE Symposium on Foundations of Computer Science (FOCS’03). IEEE Computer Society, 482. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Y.-K. Kwok, K. Hwang, and S. Song. 2007. Selfish grids: Game-theoretic modeling and NAS/PSA benchmark evaluation. IEEE Trans. Parallel Distrib. Syst. 18, 5 (May 2007), 621--636. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. A. M. Law and D. M. Kelton. 1999. Simulation Modeling and Analysis (3rd ed.). McGraw-Hill Higher Education. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. C. Leopold. 2001. Parallel and Distributed Computing: A Survey of Models, Paradigms and Approaches. John Wiley & Sons, New York, NY. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. H. Li, C. Wu, Z. Li, and F. C. M. Lau. 2013. Profit-maximizing virtual machine trading in a federation of selfish clouds. In Proceedings of the 2013 IEEE INFOCOM, 25--29.Google ScholarGoogle ScholarCross RefCross Ref
  33. M. Lin, A. Wierman, L. L. H. Andrew, and E. Thereska. 2013. Dynamic right-sizing for power-proportional data centers. IEEE/ACM Trans. Netw. 21, 5 (Oct. 2013), 1378--1391. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. B. Lohrmann, P. Janacik, and O. Kao. 2015. Elastic stream processing with latency guarantees. In Proceedings of the 2015 IEEE 35th International Conference on Distributed Computing Systems (ICDCS). Columbus, OH, 399--410. DOI:10.1109/ICDCS.2015.48Google ScholarGoogle ScholarCross RefCross Ref
  35. M. Maggio, H. Hoffmann, A. V. Papadopoulos, J. Panerati, M. D. Santambrogio, A. Agarwal, and A. Leva. 2012. Comparison of decision-making strategies for self-optimization in autonomic computing systems. ACM Trans. Auton. Adapt. Syst. 7, 4, Article 36 (Dec. 2012), 32 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. R. Makhloufi, G. Bonnet, G. Doyen, and D. Gaïti. 2009. Decentralized aggregation protocols in peer-to-peer networks: A survey. In Modelling Autonomic Communications Environments, J. C. Strassner and Y. M. Ghamri-Doudane (Eds.). Lecture Notes in Computer Science, Vol. 5844. Springer, Berlin, 111--116. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. R. Makhloufi, G. Doyen, G. Bonnet, and D. Gaïti. 2014. A survey and performance evaluation of decentralized aggregation schemes for autonomic management. Int. J. Netw. Manage. 24, 6 (2014), 469--498. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. D. Meilander, S. Kottinger, and S. Gorlatch. 2013. A scalability model for distributed resource management in real-time online applications. In Proceedings of the 2013 42nd International Conference on Parallel Processing (ICPP’13). 763--772. DOI:http://dx.doi.org/10.1109/ICPP.2013.90 Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. G. Mencagli. 2012. A Control-Theoretic Methodology for Controlling Adaptive Structured Parallel Computations. Ph.D Thesis, Department of Computer Science, University of Pisa, Italy.Google ScholarGoogle Scholar
  40. G. Mencagli and M. Vanneschi. 2011. QoS-control of structured parallel computations: A predictive control approach. In Proceedings of the 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science (CloudCom’11). 296--303. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. G. Mencagli, M. Vanneschi, and E. Vespa. 2013a. Control-theoretic adaptation strategies for autonomic reconfigurable parallel applications on cloud environments. In 2013 International Conference on High Performance Computing and Simulation (HPCS’13). 11--18.Google ScholarGoogle Scholar
  42. G. Mencagli, M. Vanneschi, and E. Vespa. 2013b. Reconfiguration stability of adaptive distributed parallel applications through a cooperative predictive control approach. In Proceedings of the 19th International Conference on Parallel Processing (Euro-Par’13). Springer-Verlag, Berlin, 329--340. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. G. Mencagli, M. Vanneschi, and E. Vespa. 2014. A cooperative predictive control approach to improve the reconfiguration stability of adaptive distributed parallel applications. ACM Trans. Auton. Adapt. Syst. 9, 1, Article 2 (March 2014), 27 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. J. Nash. 1951. Non-cooperative games. Ann. Math. 54, 2 (Sept. 1951), 286--295.Google ScholarGoogle ScholarCross RefCross Ref
  45. D. Niyato, K. Zhu, and P. Wang. 2011. Cooperative virtual machine management for multi-organization cloud computing environment. In Proceedings of the 5th International ICST Conference on Performance Evaluation Methodologies and Tools (VALUETOOLS’11). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, Belgium, 528--537. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. A. Núñez, J. L. Vázquez-Poletti, A. C. Caminero, G. G. Castañé, J. Carretero, and I. M. Llorente. 2012. iCanCloud: A flexible and scalable cloud infrastructure simulator. J. Grid Comput. 10, 1 (March 2012), 185--209. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. J. Park and M. van der Schaar. 2010. A game theoretic analysis of incentives in content production and sharing over peer-to-peer networks. IEEE J. Selected Topics Signal Process. 4, 4 (Aug. 2010), 704--717.Google ScholarGoogle Scholar
  48. A. G. Prieto and R. Stadler. 2007. A-GAP: An adaptive protocol for continuous network monitoring with accuracy objectives. IEEE Trans. Netw. Serv. Manag. 4, 1 (June 2007), 2--12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. T. Reuter and P. Cimiano. 2012. Event-based classification of social media streams. In Proceedings of the 2nd ACM International Conference on Multimedia Retrieval (ICMR’12). ACM, New York, NY, Article 22, 8 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. W. Rogerson. 1994. A theory of incentives in procurement and regulation by Jean-Jacques Laffont; Jean Tirole. J. Political Econ. 102, 2 (1994), 397--402.Google ScholarGoogle ScholarCross RefCross Ref
  51. C. U. Saraydar, N. B. Mandayam, and D. Goodman. 2002. Efficient power control via pricing in wireless data networks. IEEE Trans. Commun. 50, 2 (Feb. 2002), 291--303.Google ScholarGoogle ScholarCross RefCross Ref
  52. R. Scattolini. 2009. Architectures for distributed and hierarchical model predictive control - a review. J. Process Control 19, 5 (2009), 723--731.Google ScholarGoogle ScholarCross RefCross Ref
  53. Y. Tang and B. Gedik. 2013. Autopipelining for data stream processing. IEEE Trans. Parallel Distrib. Syst. 24, 12 (Dec. 2013), 2344--2354. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. N. Vorobjov. 1994. Foundations of Game Theory - Noncooperative Games. Birkhäuser. I--VI, 1--496 pages.Google ScholarGoogle Scholar
  55. Y. Wu and K.-L. Tan. 2015. ChronoStream: Elastic stateful stream computation in the cloud. In Proceedings of the 2015 IEEE 31st International Conference on Data Engineering (ICDE’15). 723--734.Google ScholarGoogle ScholarCross RefCross Ref
  56. L. Yang, J. Cao, Y. Yuan, T. Li, A. Han, and A. Chan. 2013. A framework for partitioning and execution of data stream applications in mobile cloud computing. SIGMETRICS Perform. Eval. Rev. 40, 4 (April 2013), 23--32. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. D. Ye and J. Chen. 2013. Non-cooperative games on multidimensional resource allocation. Future Gener. Comput. Syst. 29, 6 (Aug. 2013), 1345--1352. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Q. Yuan, Z. Liu, J. Peng, X. Wu, J. Li, F. Han, Q. Li, W. Zhang, X. Fan, and S. Kong. 2011. A leasing instances based billing model for cloud computing. In Proceedings of the 6th International Conference on Advances in Grid and Pervasive Computing (GPC’11). Springer-Verlag, Berlin, 33--41. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. T. Zhang and P. Xiao. 2014. A novel resource pricing mechanism based on multi-player gaming model in cloud environments. J. Softw. 9, 6 (2014), 1574--1580.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. A Game-Theoretic Approach for Elastic Distributed Data Stream Processing

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Autonomous and Adaptive Systems
        ACM Transactions on Autonomous and Adaptive Systems  Volume 11, Issue 2
        Special Section on Best Papers from SASO 2014 and Regular Articles
        July 2016
        267 pages
        ISSN:1556-4665
        EISSN:1556-4703
        DOI:10.1145/2952298
        Issue’s Table of Contents

        Copyright © 2016 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 6 June 2016
        • Accepted: 1 March 2016
        • Revised: 1 January 2016
        • Received: 1 July 2015
        Published in taas Volume 11, Issue 2

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!