skip to main content
research-article
Open Access

Mutual Influence-aware Runtime Learning of Self-adaptation Behavior

Published:12 September 2019Publication History
Skip Abstract Section

Abstract

Self-adaptation has been proposed as a mechanism to counter complexity in control problems of technical systems. A major driver behind self-adaptation is the idea to transfer traditional design-time decisions to runtime and into the responsibility of systems themselves. To deal with unforeseen events and conditions, systems need creativity—typically realized by means of machine learning capabilities. Such learning mechanisms are based on different sources of knowledge. Feedback from the environment used for reinforcement purposes is probably the most prominent one within the self-adapting and self-organizing (SASO) systems community. However, the impact of other (sub-)systems on the success of the individual system’s learning performance has mostly been neglected in this context.

In this article, we propose a novel methodology to identify effects of actions performed by other systems in a shared environment on the utility achievement of an autonomous system. Consider smart cameras (SC) as illustrating example: For goals such as 3D reconstruction of objects, the most promising configuration of one SC in terms of pan/tilt/zoom parameters depends largely on the configuration of other SCs in the vicinity. Since such mutual influences cannot be pre-defined for dynamic systems, they have to be learned at runtime. Furthermore, they have to be taken into consideration when self-improving their own configuration decisions based on a feedback loop concept, e.g., known from the SASO domain or the Autonomic and Organic Computing initiatives.

We define a methodology to detect such influences at runtime, present an approach to consider this information in a reinforcement learning technique, and analyze the behavior in artificial as well as real-world SASO system settings.

References

  1. J. O. Kephart and D. M. Chess. 2003. The vision of autonomic computing. Computer 36, 1 (Jan. 2003), 41--50. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Christian Müller-Schloer and Sven Tomforde. 2018. Organic Computing—Technical Systems for Survival in the Real World. Springer International Publishing.Google ScholarGoogle Scholar
  3. David Tennenhouse. 2000. Proactive computing. CACM 43, 5 (May 2000), 43--50. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Michael Wooldridge. An Introduction to Multiagent Systems. John Wiley 8 Sons. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. S. Tomforde, J. Hähner, and B. Sick. 2014. Interwoven systems. Inform.-Spekt. 37, 5 (2014), 483--487.Google ScholarGoogle ScholarCross RefCross Ref
  6. Serge Kernbach, Thomas Schmickl, and Jonathan Timmis. Collective adaptive systems: Challenges beyond evolvability. Arxiv abs/1108.5643 ({n.d.}).Google ScholarGoogle Scholar
  7. Danny Weyns, Bradley Schmerl, Vincenzo Grassi, Sam Malek, Raffaela Mirandola, Christian Prehofer, Jochen Wuttke, Jesper Andersson, Holger Giese, and Karl M. Göschka. 2013. On patterns for decentralized control in self-adaptive systems. In Software Engineering for Self-Adaptive Systems II. Springer, 76--107.Google ScholarGoogle Scholar
  8. Sven Tomforde and Christian Müller-Schloer. 2014. Incremental design of adaptive systems. Journal of Ambient Intelligence and Smart Environments 6, 2 (2014), 179--198. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Christian Krupitzer, Felix Maximilian Roth, Sebastian VanSyckel, Gregor Schiele, and Christian Becker. 2015. A survey on engineering approaches for self-adaptive systems. Pervas. Mob. Comput. 17 (2015), 184--206. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. M. W. Maier. 1998. Architecting principles for systems-of-systems. Syst. Eng. 1, 4 (1998), 267--284.Google ScholarGoogle ScholarCross RefCross Ref
  11. Ada Diaconescu, Sylvain Frey, Christian Müller-Schloer, Jeremy Pitt, and Sven Tomforde. 2016. Goal-oriented holonics for complex system (self-)integration: Concepts and case studies. In Proceedings of the 10th IEEE International Conference on Self-Adaptive and Self-Organising Systems. IEEE, 100--109.Google ScholarGoogle ScholarCross RefCross Ref
  12. S. Tomforde, J. Hähner, H. Seebach, W. Reif, B. Sick, A. Wacker, and I. Scholtes. Engineering and mastering interwoven systems. In Proceedings of the 27th International Conference on Architecture of Computing Systems, Workshop (ARCS’14). 1--8. Retrieved from: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6775093.Google ScholarGoogle Scholar
  13. Sven Tomforde, Stefan Rudolph, Kirstie L. Bellman, and Rolf P. Würtz. An organic computing perspective on self-improving system interweaving at runtime. In Proceedings of the IEEE International Conference on Autonomic Computing (ICAC’16). 276--284.Google ScholarGoogle Scholar
  14. Stefan Rudolph, Sarah Edenhofer, Sven Tomforde, and Jörg Hähner. Reinforcement learning for coverage optimization through PTZ camera alignment in highly dynamic environments. In Proceedings of the International Conference on Distributed Smart Cameras (ICDSC’14). 19:1--19:6. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Claudio Piciarelli, Lukas Esterle, Asif Khan, Bernhard Rinner, and Gian Luca Foresti. 2016. Dynamic reconfiguration in camera networks: A short survey. IEEE Trans. Circ. Syst. Vid. Technol. 26, 5 (2016), 965--977.Google ScholarGoogle ScholarCross RefCross Ref
  16. Lukas Esterle. 2017. Centralised, decentralised, and self-organised coverage maximisation in smart camera networks. In Proceedings of the IEEE 11th International Conference on Self-Adaptive and Self-Organizing Systems (SASO’17). IEEE, 1--10.Google ScholarGoogle ScholarCross RefCross Ref
  17. Kirstie L. Bellman, Jean Botev, Ada Diaconescu, Lukas Esterle, Christian Gruhl, Christopher Landauer, Peter R. Lewis, Anthony Stein, Sven Tomforde, and Rolf P. Würtz. 2018. Self-improving system integration—Status and challenges after five years of SISSY. In Proceedings of the IEEE 3rd International Workshops on Foundations and Applications of Self<sup>*</sup> Systems (FAS<sup>*</sup>W’18). 160--167.Google ScholarGoogle Scholar
  18. Kirstie L. Bellman, Christian Gruhl, Christopher Landauer, and Sven Tomforde. 2019. Self-improving system integration—On a definition and characteristics of the challenges. In Proceedings of the IEEE 4th International Workshops on Foundations and Applications of Self<sup>*</sup> Systems (FAS<sup>*</sup>W’19). 1--6.Google ScholarGoogle ScholarCross RefCross Ref
  19. Stefan Rudolph, Rainer Hihn, Sven Tomforde, and Jörg Hähner. Comparison of dependency measures for the detection of mutual influences in organic computing systems. In Proceedings of the 29th International Conference on Architecture of Computing Systems (ARCS’16). 334--347. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Stefan Rudolph, Sven Tomforde, and Jörg Hähner. A mutual influence-based learning algorithm. In Proceedings of the 8th International Conference on Agents and Artificial Intelligence (ICAART’16), Volume 1. 181--189. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Stefan Rudolph, Sven Tomforde, Bernhard Sick, and Jörg Hähner. A mutual influence detection algorithm for systems with local performance measurement. In Proceedings of the IEEE 9th International Conference on Self-Adaptive and Self-Organizing Systems. 144--149. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. S. Tomforde, B. Hurling, and J. Hähner. 2010. Dynamic control of mobile ad-hoc networks—Network protocol parameter adaptation using organic network control. In Proceedings of the 7th International Conference on Informatics in Control, Automation, and Robotics (ICINCO’10). INSTICC, Setubal, 28--35.Google ScholarGoogle Scholar
  23. S. Tomforde, M. Steffen, J. Hähner, and C. Müller-Schloer. 2009. Towards an organic network control system. In Proceedings of the 6th International Conference on Autonomic and Trusted Computing (ATC’09). Springer Verlag, 2--16. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning (1st ed.). The MIT Press, Cambridge, MA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. A. van Lamsweerde. Goal-oriented requirements engineering: A guided tour. In Proceedings of the 5th IEEE International Symposium on Requirements Engineering. 249--262. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Axel van Lamsweerde. Requirements Engineering: From System Goals to UML Models to Software Specifications (1st ed.). Wiley Publishing. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Erik M. Fredericks, Byron DeVries, and Betty H. C. Cheng. Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty. In Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’14). ACM, New York, NY, 17--26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. S. Tomforde and J. Hähner. 2011. Biologically Inspired Networking and Sensing: Algorithms and Architectures. IGI, Chapter: Organic Network Control—Turning Standard Protocols into Evolving Systems, 11--35.Google ScholarGoogle Scholar
  29. L. Busoniu, R. Babuska, and B. De Schutter. 2008. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38, 2 (2008), 156--172. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Marco Wiering and Martijn van Otterlo. Reinforcement Learning: State-of-the-Art. Springer Publishing Company, Incorporated. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Christopher J. C. H. Watkins and Peter Dayan. Technical note Q-learning. 8 ({n.d.}), 279--292. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. S. Rudolph, S. Tomforde, B. Sick, H. Heck, A. Wacker, and J. Hähner. An online influence detection algorithm for organic computing systems. In Proceedings of the 28th International Conference on Architecture of Computing Systems (ARCS'15). 1--8.Google ScholarGoogle Scholar
  33. David Keil and Dina Q. Goldin. Modeling indirect interaction in open computational systems. In Proceedings of the 12th IEEE International Workshops on Enabling Technologies (WETICE’03). 371--376. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Robert Logie, Jon G. Hall, and Kevin G. Waugh. Towards mining for influence in a multi agent environment. In Proceedings of the IADIS European Conference on Data Mining, Ajith Abraham (Ed.). IADIS, 97--101.Google ScholarGoogle Scholar
  35. Robert Logie, Jon G. Hall, and Kevin G. Waugh. 2010. Investigating agent influence and nested other-agent behaviour. Int. J. Adv. Intell. Syst. 2, 4 (2010).Google ScholarGoogle Scholar
  36. Jan M. Broersen. CTL.STIT: Enhancing ATL to express important multi-agent system verification properties. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’10), Volume 1--3. 683--690. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Peter Stone and Manuela Veloso. Multiagent systems: A survey from a machine learning perspective. 8, 3 ({n.d.}), 345--383. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Jelle R. Kok, Matthijs T. J. Spaan, and Nikos Vlassis. 2003. Multi-robot decision making using coordination graphs. In Proceedings of the International Conference on Advanced Robotics (ICAR’03), A. T. de Almeida and U. Nunes (Eds.). 1124--1129.Google ScholarGoogle Scholar
  39. Jelle R. Kok, Pieter Jan ’t Hoen, Bram Bakker, and Nikos Vlassis. 2005. Utile coordination: Learning interdependencies among cooperative agents. In Proceedings of the IEEE Symposium on Computational Intelligence and Games (CIG’05). 29--36.Google ScholarGoogle Scholar
  40. Yann-Michaël De Hauwere, Peter Vrancx, and Ann Nowé. 2009. Learning what to observe in multi-agent systems. In Proceedings of the 20th Belgian-Netherlands Conference on Artificial Intelligence. 83--90.Google ScholarGoogle Scholar
  41. Yann-Michael De Hauwere, Peter Vrancx, and Ann Nowé. 2010. Learning multi-agent state space representations. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 715--722. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Yann-Michaël De Hauwere, Peter Vrancx, and Ann Nowé. 2011. Solving delayed coordination problems in MAS. In Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’11), Volume 3. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1115--1116. Retrieved from: http://dl.acm.org/citation.cfm?id&equals;2034396.2034445. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. 2017. A unified game-theoretic approach to multiagent reinforcement learning. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 4190--4203. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy. 27, 8 ({n.d.}), 1226--1238. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Amir Massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesvári, and Shie Mannor. Regularized policy iteration. In Proceedings of the 22nd Conference on Advances in Neural Information Processing Systems. 441--448. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. J. Zico Kolter and Andrew Y. Ng. 2009. Regularization and feature selection in least-squares temporal difference learning. In Proceedings of the 26th International Conference on Machine Learning (ICML’09). ACM, New York, NY, 521--528. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. De-Rong Liu, Hong-Liang Li, and Ding Wang. Feature selection and feature learning for high-dimensional batch reinforcement learning: A survey. 12, 3 ({n.d.}), 229--242. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Jérémy Boes and Frédéric Migeon. 2017. Self-organizing multi-agent systems for the control of complex systems. J. Syst. Softw. 134 (2017), 12--28. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Stefan Rudolph, Rainer Hihn, Sven Tomforde, and Jörg Hähner. Towards discovering delayed mutual influences in organic computing systems. In Proceedings of the 30th GI/ITG International Conference on Architecture of Computing Systems (ARCS’17). 39--46.Google ScholarGoogle Scholar
  50. Karl Pearson. Note on regression and inheritance in the case of two parents. 58, 347--352 ({n.d.}), 240--242.Google ScholarGoogle Scholar
  51. Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika 30, 1/2 (1938), 81--93.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Gábor J. Székely, Maria L. Rizzo, and Nail K. Bakirov. 2007. Measuring and testing dependence by correlation of distances. The Annals of Statistics. 35, 6 (2007), 2769--2794.Google ScholarGoogle ScholarCross RefCross Ref
  53. Claude Shannon and Warren Weaver. The Mathematical Theory of Communication. University of Illinois Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. 2004. Estimating mutual information. Phys. Rev. E. 69 (2004), 066138. Issue 6.Google ScholarGoogle ScholarCross RefCross Ref
  55. David N. Reshef, Yakir A. Reshef, Hilary K. Finucane, Sharon R. Grossman, Gilean McVean, Peter J. Turnbaugh, Eric S. Lander, Michael Mitzenmacher, and Pardis C. Sabeti. 2011. Detecting novel associations in large data sets. Science 334, 6062 (2011), 1518--1524.Google ScholarGoogle ScholarCross RefCross Ref
  56. Sean Luke, Claudio Cioffi-Revilla, Liviu Panait, Keith Sullivan, and Gabriel Balan. 2005. Mason: A multiagent simulation environment. Simul.: Trans. Soc. Model. Simul. Int. 82, 7 (2005), 517--527. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Matthew E. Taylor and Peter Stone. 2009. Transfer learning for reinforcement learning domains: A survey. J. Machine Learn. Res. 10 (July 2009), 1633--1685. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. S. W. Wilson. Classifier fitness based on accuracy. 3, 2 ({n.d.}), 149--175. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Stewart W. Wilson. 2000. Get real! XCS with continuous-valued inputs. In Learning Classifier Systems, Pier Luca Lanzi, Wolfgang Stolzmann, and Stewart W. Wilson (Eds.). Springer Berlin, 209--219. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Xianneng Li and Guangfei Yang. 2016. Transferable XCS. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’16). ACM, New York, NY, 453--460. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Bartosz Krawczyk, Leandro L. Minku, João Gama, Jerzy Stefanowski, and Michał Woźniak. 2017. Ensemble learning for data stream analysis: A survey. Inform. Fus. 37 (2017), 132--156. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Wolfgang Härdle and Léopold Simar. 2007. Applied Multivariate Statistical Analysis. Vol. 22007. Springer.Google ScholarGoogle Scholar

Index Terms

  1. Mutual Influence-aware Runtime Learning of Self-adaptation Behavior

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!