Abstract
Self-adaptation has been proposed as a mechanism to counter complexity in control problems of technical systems. A major driver behind self-adaptation is the idea to transfer traditional design-time decisions to runtime and into the responsibility of systems themselves. To deal with unforeseen events and conditions, systems need creativity—typically realized by means of machine learning capabilities. Such learning mechanisms are based on different sources of knowledge. Feedback from the environment used for reinforcement purposes is probably the most prominent one within the self-adapting and self-organizing (SASO) systems community. However, the impact of other (sub-)systems on the success of the individual system’s learning performance has mostly been neglected in this context.
In this article, we propose a novel methodology to identify effects of actions performed by other systems in a shared environment on the utility achievement of an autonomous system. Consider smart cameras (SC) as illustrating example: For goals such as 3D reconstruction of objects, the most promising configuration of one SC in terms of pan/tilt/zoom parameters depends largely on the configuration of other SCs in the vicinity. Since such mutual influences cannot be pre-defined for dynamic systems, they have to be learned at runtime. Furthermore, they have to be taken into consideration when self-improving their own configuration decisions based on a feedback loop concept, e.g., known from the SASO domain or the Autonomic and Organic Computing initiatives.
We define a methodology to detect such influences at runtime, present an approach to consider this information in a reinforcement learning technique, and analyze the behavior in artificial as well as real-world SASO system settings.
- J. O. Kephart and D. M. Chess. 2003. The vision of autonomic computing. Computer 36, 1 (Jan. 2003), 41--50. Google Scholar
Digital Library
- Christian Müller-Schloer and Sven Tomforde. 2018. Organic Computing—Technical Systems for Survival in the Real World. Springer International Publishing.Google Scholar
- David Tennenhouse. 2000. Proactive computing. CACM 43, 5 (May 2000), 43--50. Google Scholar
Digital Library
- Michael Wooldridge. An Introduction to Multiagent Systems. John Wiley 8 Sons. Google Scholar
Digital Library
- S. Tomforde, J. Hähner, and B. Sick. 2014. Interwoven systems. Inform.-Spekt. 37, 5 (2014), 483--487.Google Scholar
Cross Ref
- Serge Kernbach, Thomas Schmickl, and Jonathan Timmis. Collective adaptive systems: Challenges beyond evolvability. Arxiv abs/1108.5643 ({n.d.}).Google Scholar
- Danny Weyns, Bradley Schmerl, Vincenzo Grassi, Sam Malek, Raffaela Mirandola, Christian Prehofer, Jochen Wuttke, Jesper Andersson, Holger Giese, and Karl M. Göschka. 2013. On patterns for decentralized control in self-adaptive systems. In Software Engineering for Self-Adaptive Systems II. Springer, 76--107.Google Scholar
- Sven Tomforde and Christian Müller-Schloer. 2014. Incremental design of adaptive systems. Journal of Ambient Intelligence and Smart Environments 6, 2 (2014), 179--198. Google Scholar
Digital Library
- Christian Krupitzer, Felix Maximilian Roth, Sebastian VanSyckel, Gregor Schiele, and Christian Becker. 2015. A survey on engineering approaches for self-adaptive systems. Pervas. Mob. Comput. 17 (2015), 184--206. Google Scholar
Digital Library
- M. W. Maier. 1998. Architecting principles for systems-of-systems. Syst. Eng. 1, 4 (1998), 267--284.Google Scholar
Cross Ref
- Ada Diaconescu, Sylvain Frey, Christian Müller-Schloer, Jeremy Pitt, and Sven Tomforde. 2016. Goal-oriented holonics for complex system (self-)integration: Concepts and case studies. In Proceedings of the 10th IEEE International Conference on Self-Adaptive and Self-Organising Systems. IEEE, 100--109.Google Scholar
Cross Ref
- S. Tomforde, J. Hähner, H. Seebach, W. Reif, B. Sick, A. Wacker, and I. Scholtes. Engineering and mastering interwoven systems. In Proceedings of the 27th International Conference on Architecture of Computing Systems, Workshop (ARCS’14). 1--8. Retrieved from: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6775093.Google Scholar
- Sven Tomforde, Stefan Rudolph, Kirstie L. Bellman, and Rolf P. Würtz. An organic computing perspective on self-improving system interweaving at runtime. In Proceedings of the IEEE International Conference on Autonomic Computing (ICAC’16). 276--284.Google Scholar
- Stefan Rudolph, Sarah Edenhofer, Sven Tomforde, and Jörg Hähner. Reinforcement learning for coverage optimization through PTZ camera alignment in highly dynamic environments. In Proceedings of the International Conference on Distributed Smart Cameras (ICDSC’14). 19:1--19:6. Google Scholar
Digital Library
- Claudio Piciarelli, Lukas Esterle, Asif Khan, Bernhard Rinner, and Gian Luca Foresti. 2016. Dynamic reconfiguration in camera networks: A short survey. IEEE Trans. Circ. Syst. Vid. Technol. 26, 5 (2016), 965--977.Google Scholar
Cross Ref
- Lukas Esterle. 2017. Centralised, decentralised, and self-organised coverage maximisation in smart camera networks. In Proceedings of the IEEE 11th International Conference on Self-Adaptive and Self-Organizing Systems (SASO’17). IEEE, 1--10.Google Scholar
Cross Ref
- Kirstie L. Bellman, Jean Botev, Ada Diaconescu, Lukas Esterle, Christian Gruhl, Christopher Landauer, Peter R. Lewis, Anthony Stein, Sven Tomforde, and Rolf P. Würtz. 2018. Self-improving system integration—Status and challenges after five years of SISSY. In Proceedings of the IEEE 3rd International Workshops on Foundations and Applications of Self<sup>*</sup> Systems (FAS<sup>*</sup>W’18). 160--167.Google Scholar
- Kirstie L. Bellman, Christian Gruhl, Christopher Landauer, and Sven Tomforde. 2019. Self-improving system integration—On a definition and characteristics of the challenges. In Proceedings of the IEEE 4th International Workshops on Foundations and Applications of Self<sup>*</sup> Systems (FAS<sup>*</sup>W’19). 1--6.Google Scholar
Cross Ref
- Stefan Rudolph, Rainer Hihn, Sven Tomforde, and Jörg Hähner. Comparison of dependency measures for the detection of mutual influences in organic computing systems. In Proceedings of the 29th International Conference on Architecture of Computing Systems (ARCS’16). 334--347. Google Scholar
Digital Library
- Stefan Rudolph, Sven Tomforde, and Jörg Hähner. A mutual influence-based learning algorithm. In Proceedings of the 8th International Conference on Agents and Artificial Intelligence (ICAART’16), Volume 1. 181--189. Google Scholar
Digital Library
- Stefan Rudolph, Sven Tomforde, Bernhard Sick, and Jörg Hähner. A mutual influence detection algorithm for systems with local performance measurement. In Proceedings of the IEEE 9th International Conference on Self-Adaptive and Self-Organizing Systems. 144--149. Google Scholar
Digital Library
- S. Tomforde, B. Hurling, and J. Hähner. 2010. Dynamic control of mobile ad-hoc networks—Network protocol parameter adaptation using organic network control. In Proceedings of the 7th International Conference on Informatics in Control, Automation, and Robotics (ICINCO’10). INSTICC, Setubal, 28--35.Google Scholar
- S. Tomforde, M. Steffen, J. Hähner, and C. Müller-Schloer. 2009. Towards an organic network control system. In Proceedings of the 6th International Conference on Autonomic and Trusted Computing (ATC’09). Springer Verlag, 2--16. Google Scholar
Digital Library
- Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning (1st ed.). The MIT Press, Cambridge, MA. Google Scholar
Digital Library
- A. van Lamsweerde. Goal-oriented requirements engineering: A guided tour. In Proceedings of the 5th IEEE International Symposium on Requirements Engineering. 249--262. Google Scholar
Digital Library
- Axel van Lamsweerde. Requirements Engineering: From System Goals to UML Models to Software Specifications (1st ed.). Wiley Publishing. Google Scholar
Digital Library
- Erik M. Fredericks, Byron DeVries, and Betty H. C. Cheng. Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty. In Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’14). ACM, New York, NY, 17--26. Google Scholar
Digital Library
- S. Tomforde and J. Hähner. 2011. Biologically Inspired Networking and Sensing: Algorithms and Architectures. IGI, Chapter: Organic Network Control—Turning Standard Protocols into Evolving Systems, 11--35.Google Scholar
- L. Busoniu, R. Babuska, and B. De Schutter. 2008. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38, 2 (2008), 156--172. Google Scholar
Digital Library
- Marco Wiering and Martijn van Otterlo. Reinforcement Learning: State-of-the-Art. Springer Publishing Company, Incorporated. Google Scholar
Digital Library
- Christopher J. C. H. Watkins and Peter Dayan. Technical note Q-learning. 8 ({n.d.}), 279--292. Google Scholar
Digital Library
- S. Rudolph, S. Tomforde, B. Sick, H. Heck, A. Wacker, and J. Hähner. An online influence detection algorithm for organic computing systems. In Proceedings of the 28th International Conference on Architecture of Computing Systems (ARCS'15). 1--8.Google Scholar
- David Keil and Dina Q. Goldin. Modeling indirect interaction in open computational systems. In Proceedings of the 12th IEEE International Workshops on Enabling Technologies (WETICE’03). 371--376. Google Scholar
Digital Library
- Robert Logie, Jon G. Hall, and Kevin G. Waugh. Towards mining for influence in a multi agent environment. In Proceedings of the IADIS European Conference on Data Mining, Ajith Abraham (Ed.). IADIS, 97--101.Google Scholar
- Robert Logie, Jon G. Hall, and Kevin G. Waugh. 2010. Investigating agent influence and nested other-agent behaviour. Int. J. Adv. Intell. Syst. 2, 4 (2010).Google Scholar
- Jan M. Broersen. CTL.STIT: Enhancing ATL to express important multi-agent system verification properties. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’10), Volume 1--3. 683--690. Google Scholar
Digital Library
- Peter Stone and Manuela Veloso. Multiagent systems: A survey from a machine learning perspective. 8, 3 ({n.d.}), 345--383. Google Scholar
Digital Library
- Jelle R. Kok, Matthijs T. J. Spaan, and Nikos Vlassis. 2003. Multi-robot decision making using coordination graphs. In Proceedings of the International Conference on Advanced Robotics (ICAR’03), A. T. de Almeida and U. Nunes (Eds.). 1124--1129.Google Scholar
- Jelle R. Kok, Pieter Jan ’t Hoen, Bram Bakker, and Nikos Vlassis. 2005. Utile coordination: Learning interdependencies among cooperative agents. In Proceedings of the IEEE Symposium on Computational Intelligence and Games (CIG’05). 29--36.Google Scholar
- Yann-Michaël De Hauwere, Peter Vrancx, and Ann Nowé. 2009. Learning what to observe in multi-agent systems. In Proceedings of the 20th Belgian-Netherlands Conference on Artificial Intelligence. 83--90.Google Scholar
- Yann-Michael De Hauwere, Peter Vrancx, and Ann Nowé. 2010. Learning multi-agent state space representations. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 715--722. Google Scholar
Digital Library
- Yann-Michaël De Hauwere, Peter Vrancx, and Ann Nowé. 2011. Solving delayed coordination problems in MAS. In Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’11), Volume 3. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1115--1116. Retrieved from: http://dl.acm.org/citation.cfm?id=2034396.2034445. Google Scholar
Digital Library
- Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. 2017. A unified game-theoretic approach to multiagent reinforcement learning. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 4190--4203. Google Scholar
Digital Library
- Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy. 27, 8 ({n.d.}), 1226--1238. Google Scholar
Digital Library
- Amir Massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesvári, and Shie Mannor. Regularized policy iteration. In Proceedings of the 22nd Conference on Advances in Neural Information Processing Systems. 441--448. Google Scholar
Digital Library
- J. Zico Kolter and Andrew Y. Ng. 2009. Regularization and feature selection in least-squares temporal difference learning. In Proceedings of the 26th International Conference on Machine Learning (ICML’09). ACM, New York, NY, 521--528. Google Scholar
Digital Library
- De-Rong Liu, Hong-Liang Li, and Ding Wang. Feature selection and feature learning for high-dimensional batch reinforcement learning: A survey. 12, 3 ({n.d.}), 229--242. Google Scholar
Digital Library
- Jérémy Boes and Frédéric Migeon. 2017. Self-organizing multi-agent systems for the control of complex systems. J. Syst. Softw. 134 (2017), 12--28. Google Scholar
Digital Library
- Stefan Rudolph, Rainer Hihn, Sven Tomforde, and Jörg Hähner. Towards discovering delayed mutual influences in organic computing systems. In Proceedings of the 30th GI/ITG International Conference on Architecture of Computing Systems (ARCS’17). 39--46.Google Scholar
- Karl Pearson. Note on regression and inheritance in the case of two parents. 58, 347--352 ({n.d.}), 240--242.Google Scholar
- Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika 30, 1/2 (1938), 81--93.Google Scholar
Digital Library
- Gábor J. Székely, Maria L. Rizzo, and Nail K. Bakirov. 2007. Measuring and testing dependence by correlation of distances. The Annals of Statistics. 35, 6 (2007), 2769--2794.Google Scholar
Cross Ref
- Claude Shannon and Warren Weaver. The Mathematical Theory of Communication. University of Illinois Press. Google Scholar
Digital Library
- Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. 2004. Estimating mutual information. Phys. Rev. E. 69 (2004), 066138. Issue 6.Google Scholar
Cross Ref
- David N. Reshef, Yakir A. Reshef, Hilary K. Finucane, Sharon R. Grossman, Gilean McVean, Peter J. Turnbaugh, Eric S. Lander, Michael Mitzenmacher, and Pardis C. Sabeti. 2011. Detecting novel associations in large data sets. Science 334, 6062 (2011), 1518--1524.Google Scholar
Cross Ref
- Sean Luke, Claudio Cioffi-Revilla, Liviu Panait, Keith Sullivan, and Gabriel Balan. 2005. Mason: A multiagent simulation environment. Simul.: Trans. Soc. Model. Simul. Int. 82, 7 (2005), 517--527. Google Scholar
Digital Library
- Matthew E. Taylor and Peter Stone. 2009. Transfer learning for reinforcement learning domains: A survey. J. Machine Learn. Res. 10 (July 2009), 1633--1685. Google Scholar
Digital Library
- S. W. Wilson. Classifier fitness based on accuracy. 3, 2 ({n.d.}), 149--175. Google Scholar
Digital Library
- Stewart W. Wilson. 2000. Get real! XCS with continuous-valued inputs. In Learning Classifier Systems, Pier Luca Lanzi, Wolfgang Stolzmann, and Stewart W. Wilson (Eds.). Springer Berlin, 209--219. Google Scholar
Digital Library
- Xianneng Li and Guangfei Yang. 2016. Transferable XCS. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’16). ACM, New York, NY, 453--460. Google Scholar
Digital Library
- Bartosz Krawczyk, Leandro L. Minku, João Gama, Jerzy Stefanowski, and Michał Woźniak. 2017. Ensemble learning for data stream analysis: A survey. Inform. Fus. 37 (2017), 132--156. Google Scholar
Digital Library
- Wolfgang Härdle and Léopold Simar. 2007. Applied Multivariate Statistical Analysis. Vol. 22007. Springer.Google Scholar
Index Terms
Mutual Influence-aware Runtime Learning of Self-adaptation Behavior
Recommendations
Lifelong self-adaptation: self-adaptation meets lifelong machine learning
SEAMS '22: Proceedings of the 17th Symposium on Software Engineering for Adaptive and Self-Managing SystemsIn the past years, machine learning (ML) has become a popular approach to support self-adaptation. While ML techniques enable dealing with several problems in self-adaptation, such as scalable decision-making, they are also subject to inherent ...
Analyzing Latency-Aware Self-Adaptation Using Stochastic Games and Simulations
Special Section on Best Papers from SEAMS 2014 and Regular ArticlesSelf-adaptive systems must decide which adaptations to apply and when. In reactive approaches, adaptations are chosen and executed after some issue in the system has been detected (e.g., unforeseen attacks or failures). In proactive approaches, ...
Reducing large adaptation spaces in self-adaptive systems using classical machine learning
AbstractModern software systems often have to cope with uncertain operation conditions, such as changing workloads or fluctuating interference in a wireless network. To ensure that these systems meet their goals these uncertainties have to be ...
Highlights- Real self-adaptive systems require efficient analysis of the adaptation space.
- ...






Comments