skip to main content
research-article

Hyper-Learning Algorithms for Online Evolution of Robot Controllers

Authors Info & Claims
Published:20 September 2017Publication History
Skip Abstract Section

Abstract

A long-standing goal in artificial intelligence and robotics is synthesising agents that can effectively learn and adapt throughout their lifetime. One open-ended approach to behaviour learning in autonomous robots is online evolution, which is part of the evolutionary robotics field of research. In online evolution approaches, an evolutionary algorithm is executed on the robots during task execution, which enables continuous optimisation and adaptation of behaviour. Despite the potential for automatic behaviour learning, online evolution has not been widely adopted because it often requires several hours or days to synthesise solutions to a given task. In this respect, research in the field has failed to develop a prevalent algorithm able to effectively synthesise solutions to a large number of different tasks in a timely manner. Rather than focusing on a single algorithm, we argue for more general mechanisms that can combine the benefits of different algorithms to increase the performance of online evolution of robot controllers. We conduct a comprehensive assessment of a novel approach called online hyper-evolution (OHE). Robots executing OHE use the different sources of feedback information traditionally associated with controller evaluation to find effective evolutionary algorithms during task execution. First, we study two approaches: OHE-fitness, which uses the fitness score of controllers as the criterion to select promising algorithms over time, and OHE-diversity, which relies on the behavioural diversity of controllers for algorithm selection. We then propose a novel class of techniques called OHE-hybrid, which combine diversity and fitness to search for suitable algorithms. In addition to their effectiveness at selecting suitable algorithms, the different OHE approaches are evaluated for their ability to construct algorithms by controlling which algorithmic components should be employed for controller generation (e.g., mutation, crossover, among others), an unprecedented approach in evolutionary robotics. Results show that OHE (i) facilitates the evolution of controllers with high performance, (ii) can increase effectiveness at different stages of evolution by combining the benefits of multiple algorithms over time, and (iii) can be effectively applied to construct new algorithms during task execution. Overall, our study shows that OHE is a powerful new paradigm that allows robots to improve their learning process as they operate in the task environment.

References

  1. H. Asmuni, E. Burke, J. Garibaldi, and B. McCollum. 2004. Fuzzy multiple heuristic orderings for examination timetabling. In Proceedings of the International Conference on the Practice and Theory of Automated Timetabling. Springer, Berlin, 334--353 Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. R. Bianco and S. Nolfi. 2004. Toward open-ended evolutionary robotics: Evolving elementary robotic units able to self-assemble and self-reproduce. Connection Science 16, 4, 227--248.Google ScholarGoogle ScholarCross RefCross Ref
  3. N. Bredeche, J. M. Montanier, W. Liu, and A. F. T. Winfield. 2012. Environment-driven distributed evolutionary adaptation in a population of autonomous robotic agents. Mathematical and Computer Modelling of Dynamical Systems 18, 1, 101--129.Google ScholarGoogle ScholarCross RefCross Ref
  4. E. Burke, M. Gendreau, M. Hyde, G. Kendall, G. Ochoa, E. Özcan, and R. Qu. 2013. Hyper-heuristics: A survey of the state of the art. Journal of the Operational Research Society 64, 12, 1695--1724.Google ScholarGoogle ScholarCross RefCross Ref
  5. E. Burke, B. McCollum, A. Meisels, S. Petrovic, and R. Qu. 2007. A graph-based hyper-heuristic for educational timetabling problems. European Journal of Operational Research 176, 1, 177--192.Google ScholarGoogle ScholarCross RefCross Ref
  6. P. Cowling, G. Kendall, and E. Soubeiga. 2000. A hyperheuristic approach to scheduling a sales summit. In Proceedings of the International Conference on the Practice and Theory of Automated Timetabling. Springer, Berlin, 176--190. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. G. Cuccu and F. Gomez. 2011. When novelty is not enough. In Proceedings of the European Conference on the Applications of Evolutionary Computation. Springer, Berlin, 234--243. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation 6, 2, 182--197. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. S. Doncieux and J.-B. Mouret. 2014. Beyond black-box optimization: A review of selective pressures for evolutionary robotics. Evolutionary Intelligence 7, 2, 71--93.Google ScholarGoogle ScholarCross RefCross Ref
  10. M. Duarte, F. Silva, T. Rodrigues, S. M. Oliveira, and A. L. Christensen. 2014. JBotEvolver: A versatile simulation platform for evolutionary robotics. In Proceedings of the International Conference on the Synthesis and Simulation of Living Systems. MIT Press, Cambridge, MA, 210--211.Google ScholarGoogle Scholar
  11. A. Eiben, Z. Michalewicz, M. Schoenauer, and J. Smith. 2007. Parameter control in evolutionary algorithms. In Parameter Setting in Evolutionary Algorithms. Studies in Computational Intelligence, Vol. 54. Springer, Berlin, 19--46.Google ScholarGoogle Scholar
  12. D. Floreano and F. Mondada. 1994. Automatic creation of an autonomous agent: Genetic evolution of a neural-network driven robot. In Proceedings of the International Conference on Simulation of Adaptive Behavior. MIT Press, Cambridge, MA, 421--430. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. D. Floreano and F. Mondada. 1996. Evolution of homing navigation in a real mobile robot. IEEE Transactions on Systems, Man, and Cybernetics 26, 3, 396--407. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. D. Fogel. 2006. Nils Barricelli -- Artificial life, coevolution, self-adaptation. IEEE Computational Intelligence Magazine 1, 1, 41--45. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. P. Garcia-Sánchez, A. E. Eiben, Evert Haasdijk, B. Weel, and J. J. Merelo. 2012. Testing diversity-enhancing migration policies for hybrid on-line evolution of robot controllers. In Proceedings of the European Conference on the Applications of Evolutionary Computation. Springer, Berlin, 52--62. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. D. Goldberg, K. Deb, and B. Korb. 1989. Messy genetic algorithms: Motivation, analysis, and first results. Complex Systems 3, 493--530.Google ScholarGoogle Scholar
  17. J. Gomes, P. Urbano, and A. L. Christensen. 2012. Progressive minimal criteria novelty search. In Proceedings of the Ibero-American Conference on Artificial Intelligence. Springer, Berlin, 281--290.Google ScholarGoogle Scholar
  18. J. Grobler, A. Engelbrecht, G. Kendall, and V. Yadavalli. 2012. Investigating the use of local search for improving meta-hyper-heuristic performance. In Proceedings of the IEEE Congress on Evolutionary Computation. IEEE Press, Piscataway, NJ, 1--8.Google ScholarGoogle Scholar
  19. G. Hommel. 1988. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75, 2, 383--386.Google ScholarGoogle ScholarCross RefCross Ref
  20. W. Kantschik, P. Dittrich, M. Brameier, and W. Banzhaf. 1999. Empirical analysis of different levels of meta-evolution. In Proceedings of the IEEE Congress on Evolutionary Computation. IEEE Press, Piscataway, NJ, 2086--2093.Google ScholarGoogle Scholar
  21. J. Lehman and K. O. Stanley. 2010. Revising the evolutionary computation abstraction: Minimal criteria novelty search. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM Press, New York, NY, 103--110. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. J. Lehman and K. O Stanley. 2011. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation 19, 2, 189--223. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. H. Lipson and J. Pollack. 2000. Automatic design and manufacture of robotic lifeforms. Nature 406, 974--978.Google ScholarGoogle ScholarCross RefCross Ref
  24. F. Mondada, M. Bonani, X. Raemy, J. Pugh, C. Cianci, A. Klaptocz, S. Magnenat, J. Zufferey, D. Floreano, and A. Martinoli. 2009. The e-puck, a robot designed for education in engineering. In Proceedings of the Conference on Autonomous Robot Systems and Competitions. IPCB, Castelo Branco, Portugal, 59--65.Google ScholarGoogle Scholar
  25. J.-B. Mouret and S. Doncieux. 2012. Encouraging behavioral diversity in evolutionary robotics: An empirical study. Evolutionary Computation 20, 1, 91--133. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. N. Pillay and W. Banzhaf. 2007. A genetic programming approach to the generation of hyper-heuristics for the uncapacitated examination timetabling problem. In Proceedings of the Portuguese Conference on Artificial Intelligence. Springer, Berlin, 223--234. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. P. Ross and J. Martin-Blazquez. 2005. Constructive hyper-heuristics in class timetabling. In Proceedings of the IEEE Congress on Evolutionary Computation. IEEE Press, Piscataway, NJ, 1493--1500.Google ScholarGoogle Scholar
  28. S. Russell and P. Norvig. 1995. Artificial Intelligence: A Modern Approach. Prentice Hall, Upper Saddle River, NJ. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. P. Ryser-Welch and J. Miller. 2014. A review of hyper-heuristic frameworks. In Proceedings of the Annual Convention of the Society for the Study of Artificial Intelligence and the Simulation of Behaviour. Retrieved August 8, 2017 from http://doc.gold.ac.uk/aisb50/.Google ScholarGoogle Scholar
  30. J. W. Sammon Jr.1969. A nonlinear mapping for data structure analysis. IEEE Transactions on Computers C-18, 5, 401--409. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. F. Silva, L. Correia, and A. L. Christensen. 2015b. A case study on the scalability of online evolution of robotic controllers. In Proceedings of the Portuguese Conference on Artificial Intelligence. Springer, Berlin, 189--200.Google ScholarGoogle Scholar
  32. F. Silva, L. Correia, and A. L. Christensen. 2016a. Evolutionary robotics. Scholarpedia 11, 7, 33333.Google ScholarGoogle ScholarCross RefCross Ref
  33. F. Silva, L. Correia, and A. L. Christensen. 2016b. Leveraging online racing and population cloning in evolutionary multirobot systems. In Proceedings of the European Conference on the Applications of Evolutionary Computation. Springer International Publishing, Switzerland, 165--180.Google ScholarGoogle Scholar
  34. F. Silva, L. Correia, and A. L. Christensen. 2016c. Online hyper-evolution of controllers in multirobot systems. In Proceedings of the IEEE International Conference on Self-Adaptive and Self-Organizing Systems. IEEE Computer Society, Los Alamitos, CA, 11--20.Google ScholarGoogle Scholar
  35. F. Silva, L. Correia, and A. L. Christensen. 2017. Evolutionary online behaviour learning and adaptation in real robots. Royal Society Open Science 4, 160938.Google ScholarGoogle ScholarCross RefCross Ref
  36. F. Silva, M. Duarte, L. Correia, S. M. Oliveira, and A. L. Christensen. 2016. Open issues in evolutionary robotics. Evolutionary Computation 24, 2, 205--236. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. F. Silva, P. Urbano, L. Correia, and A. L. Christensen. 2015a. odNEAT: An algorithm for decentralised online evolution of robotic controllers. Evolutionary Computation 23, 3, 421--449. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. R. A. Watson, S. G. Ficici, and J. B. Pollack. 2002. Embodied evolution: Distributing an evolutionary algorithm in a population of robots. Robotics and Autonomous Systems 39, 1, 1--18.Google ScholarGoogle Scholar
  39. S. Wischmann, K. Stamm, and F. Wörgötter. 2007. Embodied evolution and learning: The neglected timing of maturation. In Proceedings of the European Conference on Artificial Life. Springer, Berlin, 284--293. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Hyper-Learning Algorithms for Online Evolution of Robot Controllers

              Recommendations

              Comments

              Login options

              Check if you have access through your login credentials or your institution to get full access on this article.

              Sign in

              Full Access

              • Article Metrics

                • Downloads (Last 12 months)14
                • Downloads (Last 6 weeks)3

                Other Metrics

              PDF Format

              View or Download as a PDF file.

              PDF

              eReader

              View online with eReader.

              eReader
              About Cookies On This Site

              We use cookies to ensure that we give you the best experience on our website.

              Learn more

              Got it!