Abstract
A long-standing goal in artificial intelligence and robotics is synthesising agents that can effectively learn and adapt throughout their lifetime. One open-ended approach to behaviour learning in autonomous robots is online evolution, which is part of the evolutionary robotics field of research. In online evolution approaches, an evolutionary algorithm is executed on the robots during task execution, which enables continuous optimisation and adaptation of behaviour. Despite the potential for automatic behaviour learning, online evolution has not been widely adopted because it often requires several hours or days to synthesise solutions to a given task. In this respect, research in the field has failed to develop a prevalent algorithm able to effectively synthesise solutions to a large number of different tasks in a timely manner. Rather than focusing on a single algorithm, we argue for more general mechanisms that can combine the benefits of different algorithms to increase the performance of online evolution of robot controllers. We conduct a comprehensive assessment of a novel approach called online hyper-evolution (OHE). Robots executing OHE use the different sources of feedback information traditionally associated with controller evaluation to find effective evolutionary algorithms during task execution. First, we study two approaches: OHE-fitness, which uses the fitness score of controllers as the criterion to select promising algorithms over time, and OHE-diversity, which relies on the behavioural diversity of controllers for algorithm selection. We then propose a novel class of techniques called OHE-hybrid, which combine diversity and fitness to search for suitable algorithms. In addition to their effectiveness at selecting suitable algorithms, the different OHE approaches are evaluated for their ability to construct algorithms by controlling which algorithmic components should be employed for controller generation (e.g., mutation, crossover, among others), an unprecedented approach in evolutionary robotics. Results show that OHE (i) facilitates the evolution of controllers with high performance, (ii) can increase effectiveness at different stages of evolution by combining the benefits of multiple algorithms over time, and (iii) can be effectively applied to construct new algorithms during task execution. Overall, our study shows that OHE is a powerful new paradigm that allows robots to improve their learning process as they operate in the task environment.
- H. Asmuni, E. Burke, J. Garibaldi, and B. McCollum. 2004. Fuzzy multiple heuristic orderings for examination timetabling. In Proceedings of the International Conference on the Practice and Theory of Automated Timetabling. Springer, Berlin, 334--353 Google Scholar
Digital Library
- R. Bianco and S. Nolfi. 2004. Toward open-ended evolutionary robotics: Evolving elementary robotic units able to self-assemble and self-reproduce. Connection Science 16, 4, 227--248.Google Scholar
Cross Ref
- N. Bredeche, J. M. Montanier, W. Liu, and A. F. T. Winfield. 2012. Environment-driven distributed evolutionary adaptation in a population of autonomous robotic agents. Mathematical and Computer Modelling of Dynamical Systems 18, 1, 101--129.Google Scholar
Cross Ref
- E. Burke, M. Gendreau, M. Hyde, G. Kendall, G. Ochoa, E. Özcan, and R. Qu. 2013. Hyper-heuristics: A survey of the state of the art. Journal of the Operational Research Society 64, 12, 1695--1724.Google Scholar
Cross Ref
- E. Burke, B. McCollum, A. Meisels, S. Petrovic, and R. Qu. 2007. A graph-based hyper-heuristic for educational timetabling problems. European Journal of Operational Research 176, 1, 177--192.Google Scholar
Cross Ref
- P. Cowling, G. Kendall, and E. Soubeiga. 2000. A hyperheuristic approach to scheduling a sales summit. In Proceedings of the International Conference on the Practice and Theory of Automated Timetabling. Springer, Berlin, 176--190. Google Scholar
Digital Library
- G. Cuccu and F. Gomez. 2011. When novelty is not enough. In Proceedings of the European Conference on the Applications of Evolutionary Computation. Springer, Berlin, 234--243. Google Scholar
Digital Library
- K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation 6, 2, 182--197. Google Scholar
Digital Library
- S. Doncieux and J.-B. Mouret. 2014. Beyond black-box optimization: A review of selective pressures for evolutionary robotics. Evolutionary Intelligence 7, 2, 71--93.Google Scholar
Cross Ref
- M. Duarte, F. Silva, T. Rodrigues, S. M. Oliveira, and A. L. Christensen. 2014. JBotEvolver: A versatile simulation platform for evolutionary robotics. In Proceedings of the International Conference on the Synthesis and Simulation of Living Systems. MIT Press, Cambridge, MA, 210--211.Google Scholar
- A. Eiben, Z. Michalewicz, M. Schoenauer, and J. Smith. 2007. Parameter control in evolutionary algorithms. In Parameter Setting in Evolutionary Algorithms. Studies in Computational Intelligence, Vol. 54. Springer, Berlin, 19--46.Google Scholar
- D. Floreano and F. Mondada. 1994. Automatic creation of an autonomous agent: Genetic evolution of a neural-network driven robot. In Proceedings of the International Conference on Simulation of Adaptive Behavior. MIT Press, Cambridge, MA, 421--430. Google Scholar
Digital Library
- D. Floreano and F. Mondada. 1996. Evolution of homing navigation in a real mobile robot. IEEE Transactions on Systems, Man, and Cybernetics 26, 3, 396--407. Google Scholar
Digital Library
- D. Fogel. 2006. Nils Barricelli -- Artificial life, coevolution, self-adaptation. IEEE Computational Intelligence Magazine 1, 1, 41--45. Google Scholar
Digital Library
- P. Garcia-Sánchez, A. E. Eiben, Evert Haasdijk, B. Weel, and J. J. Merelo. 2012. Testing diversity-enhancing migration policies for hybrid on-line evolution of robot controllers. In Proceedings of the European Conference on the Applications of Evolutionary Computation. Springer, Berlin, 52--62. Google Scholar
Digital Library
- D. Goldberg, K. Deb, and B. Korb. 1989. Messy genetic algorithms: Motivation, analysis, and first results. Complex Systems 3, 493--530.Google Scholar
- J. Gomes, P. Urbano, and A. L. Christensen. 2012. Progressive minimal criteria novelty search. In Proceedings of the Ibero-American Conference on Artificial Intelligence. Springer, Berlin, 281--290.Google Scholar
- J. Grobler, A. Engelbrecht, G. Kendall, and V. Yadavalli. 2012. Investigating the use of local search for improving meta-hyper-heuristic performance. In Proceedings of the IEEE Congress on Evolutionary Computation. IEEE Press, Piscataway, NJ, 1--8.Google Scholar
- G. Hommel. 1988. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75, 2, 383--386.Google Scholar
Cross Ref
- W. Kantschik, P. Dittrich, M. Brameier, and W. Banzhaf. 1999. Empirical analysis of different levels of meta-evolution. In Proceedings of the IEEE Congress on Evolutionary Computation. IEEE Press, Piscataway, NJ, 2086--2093.Google Scholar
- J. Lehman and K. O. Stanley. 2010. Revising the evolutionary computation abstraction: Minimal criteria novelty search. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM Press, New York, NY, 103--110. Google Scholar
Digital Library
- J. Lehman and K. O Stanley. 2011. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation 19, 2, 189--223. Google Scholar
Digital Library
- H. Lipson and J. Pollack. 2000. Automatic design and manufacture of robotic lifeforms. Nature 406, 974--978.Google Scholar
Cross Ref
- F. Mondada, M. Bonani, X. Raemy, J. Pugh, C. Cianci, A. Klaptocz, S. Magnenat, J. Zufferey, D. Floreano, and A. Martinoli. 2009. The e-puck, a robot designed for education in engineering. In Proceedings of the Conference on Autonomous Robot Systems and Competitions. IPCB, Castelo Branco, Portugal, 59--65.Google Scholar
- J.-B. Mouret and S. Doncieux. 2012. Encouraging behavioral diversity in evolutionary robotics: An empirical study. Evolutionary Computation 20, 1, 91--133. Google Scholar
Digital Library
- N. Pillay and W. Banzhaf. 2007. A genetic programming approach to the generation of hyper-heuristics for the uncapacitated examination timetabling problem. In Proceedings of the Portuguese Conference on Artificial Intelligence. Springer, Berlin, 223--234. Google Scholar
Digital Library
- P. Ross and J. Martin-Blazquez. 2005. Constructive hyper-heuristics in class timetabling. In Proceedings of the IEEE Congress on Evolutionary Computation. IEEE Press, Piscataway, NJ, 1493--1500.Google Scholar
- S. Russell and P. Norvig. 1995. Artificial Intelligence: A Modern Approach. Prentice Hall, Upper Saddle River, NJ. Google Scholar
Digital Library
- P. Ryser-Welch and J. Miller. 2014. A review of hyper-heuristic frameworks. In Proceedings of the Annual Convention of the Society for the Study of Artificial Intelligence and the Simulation of Behaviour. Retrieved August 8, 2017 from http://doc.gold.ac.uk/aisb50/.Google Scholar
- J. W. Sammon Jr.1969. A nonlinear mapping for data structure analysis. IEEE Transactions on Computers C-18, 5, 401--409. Google Scholar
Digital Library
- F. Silva, L. Correia, and A. L. Christensen. 2015b. A case study on the scalability of online evolution of robotic controllers. In Proceedings of the Portuguese Conference on Artificial Intelligence. Springer, Berlin, 189--200.Google Scholar
- F. Silva, L. Correia, and A. L. Christensen. 2016a. Evolutionary robotics. Scholarpedia 11, 7, 33333.Google Scholar
Cross Ref
- F. Silva, L. Correia, and A. L. Christensen. 2016b. Leveraging online racing and population cloning in evolutionary multirobot systems. In Proceedings of the European Conference on the Applications of Evolutionary Computation. Springer International Publishing, Switzerland, 165--180.Google Scholar
- F. Silva, L. Correia, and A. L. Christensen. 2016c. Online hyper-evolution of controllers in multirobot systems. In Proceedings of the IEEE International Conference on Self-Adaptive and Self-Organizing Systems. IEEE Computer Society, Los Alamitos, CA, 11--20.Google Scholar
- F. Silva, L. Correia, and A. L. Christensen. 2017. Evolutionary online behaviour learning and adaptation in real robots. Royal Society Open Science 4, 160938.Google Scholar
Cross Ref
- F. Silva, M. Duarte, L. Correia, S. M. Oliveira, and A. L. Christensen. 2016. Open issues in evolutionary robotics. Evolutionary Computation 24, 2, 205--236. Google Scholar
Digital Library
- F. Silva, P. Urbano, L. Correia, and A. L. Christensen. 2015a. odNEAT: An algorithm for decentralised online evolution of robotic controllers. Evolutionary Computation 23, 3, 421--449. Google Scholar
Digital Library
- R. A. Watson, S. G. Ficici, and J. B. Pollack. 2002. Embodied evolution: Distributing an evolutionary algorithm in a population of robots. Robotics and Autonomous Systems 39, 1, 1--18.Google Scholar
- S. Wischmann, K. Stamm, and F. Wörgötter. 2007. Embodied evolution and learning: The neglected timing of maturation. In Proceedings of the European Conference on Artificial Life. Springer, Berlin, 284--293. Google Scholar
Digital Library
Index Terms
Hyper-Learning Algorithms for Online Evolution of Robot Controllers
Recommendations
Engineering Online Evolution of Robot Behaviour: (Doctoral Consortium)
AAMAS '15: Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent SystemsEvolutionary computation techniques have been widely studied to automate the synthesis of behavioural control for robots. In online evolution, an evolutionary algorithm is executed on the robots themselves during task execution so as to continuously ...
Online Evolution of Adaptive Robot Behaviour
The authors propose and evaluate a novel approach to the online synthesis of neural controllers for autonomous robots. The authors combine online evolution of weights and network topology with neuromodulated learning. The authors demonstrate our method ...
Odneat: An algorithm for decentralised online evolution of robotic controllers
Online evolution gives robots the capacity to learn new tasks and to adapt to changing environmental conditions during task execution. Previous approaches to online evolution of neural controllers are typically limited to the optimisation of weights in ...






Comments