skip to main content
research-article

Towards adaptive programming: integrating reinforcement learning into a programming language

Published:19 October 2008Publication History
Skip Abstract Section

Abstract

Current programming languages and software engineering paradigms are proving insufficient for building intelligent multi-agent systems--such as interactive games and narratives--where developers are called upon to write increasingly complex behavior for agents in dynamic environments. A promising solution is to build adaptive systems; that is, to develop software written specifically to adapt to its environment by changing its behavior in response to what it observes in the world. In this paper we describe a new programming language, An Adaptive Behavior Language (A2BL), that implements adaptive programming primitives to support partial programming, a paradigm in which a programmer need only specify the details of behavior known at code-writing time, leaving the run-time system to learn the rest. Partial programming enables programmers to more easily encode software agents that are difficult to write in existing languages that do not offer language-level support for adaptivity. We motivate the use of partial programming with an example agent coded in a cutting-edge, but non-adaptive agent programming language (ABL), and show how A2BL can encode the same agent much more naturally.

References

  1. David Andre and Stuart Russell. Programmable reinforcement learning agents. In Advances in Neural Information Processing Systems, volume 13, 2001.Google ScholarGoogle Scholar
  2. David Andre and Stuart Russell. State abstraction for programmable reinforcement learning agents. In AAAI-02, Edmonton, Alberta, 2002. AAAI Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Sooraj Bhat, Charles Isbell, and Michael Mateas. On the difficulty of modular reinforcement learning for real-world partial programming. In Proceedings of the Twenty-First National Conference on Artificial Intelligence (AAAI-06), Boston, MA, USA, July 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Thomas G. Dietterich. The MAXQ method for hierarchical reinforcement learning. In Proc. 15th International Conf. on Machine Learning, pages 118--126. Morgan Kaufmann, San Francisco, CA, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Leslie Pack Kaelbling, Michael L. Littman, and Andrew P. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research}, 237--285, 1996. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. A. B. Loyall and J. Bates. Hap: A reactive adaptive architecture for agents. Technical Report CMU-CS-91-147, 1991.Google ScholarGoogle Scholar
  7. Michael Mateas and Andrew Stern. Facade: An experiment in building a fully-realized interactive drama. In Game Developers Conference: Game Design Track, San Jose, CA, March 2003.Google ScholarGoogle Scholar
  8. Michael Mateas and Andrew Stern. Life-like Characters. Tools, Affective Functions and Applications, chapter A Behavior Language: Joint Action and Behavioral Idioms. Springer, 2004.Google ScholarGoogle Scholar
  9. Tom Mitchell. Machine Learning. McGraw-Hill, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Peter Norvig. Decision theory: The language of adaptive agent software. Presentation, March 1998. http://www.norvig.com/adaptive/index.htmGoogle ScholarGoogle Scholar
  11. Peter Norvig and David Cohn. Adaptive software, 1998. http://norvig.com/adapaper-pcai.html Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. In Michael I. Jordan, Michael J. Kearns, and Sara A. Solla, editors, Advances in Neural Information Processing Systems, volume 10. The MIT Press, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prenticce Hall, Upper Saddle River, NJ, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. R.S. Sutton and A.G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Sprague, N., and Ballard, D. Multiple-Goal Reinforcement Learning with Modular Sarsa(0). In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, 2003. Workshop paper. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Towards adaptive programming: integrating reinforcement learning into a programming language

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!