Abstract
In multiagent systems, social optimality is a desirable goal to achieve in terms of maximizing the global efficiency of the system. We study the problem of coordinating on socially optimal outcomes among a population of agents, in which each agent randomly interacts with another agent from the population each round. Previous work [Hales and Edmonds 2003; Matlock and Sen 2007, 2009] mainly resorts to modifying the interaction protocol from random interaction to tag-based interactions and only focus on the case of symmetric games. Besides, in previous work the agents’ decision making processes are usually based on evolutionary learning, which usually results in high communication cost and high deviation on the coordination rate. To solve these problems, we propose an alternative social learning framework with two major contributions as follows. First, we introduce the observation mechanism to reduce the amount of communication required among agents. Second, we propose that the agents’ learning strategies should be based on reinforcement learning technique instead of evolutionary learning. Each agent explicitly keeps the record of its current state in its learning strategy, and learn its optimal policy for each state independently. In this way, the learning performance is much more stable and also it is suitable for both symmetric and asymmetric games. The performance of this social learning framework is extensively evaluated under the testbed of two-player general-sum games comparing with previous work [Hao and Leung 2011; Matlock and Sen 2007]. The influences of different factors on the learning performance of the social learning framework are investigated as well.
- Allison, P. D. 1992. The cultural evolution of beneficent norms. Social Forces. 279--301.Google Scholar
- Bowling, M. H. and Veloso, M. M. 2003. Multiagent learning using a variable learning rate. Artif. Intell. 136, 215--250. Google Scholar
Digital Library
- Brafman, R. I. and Tennenholtz, M. 2004. Efficient learning equilibrium. Artif. Intell. 159, 27--47. Google Scholar
Digital Library
- Chao, I., Ardaiz, O., and Sanguesa, R. 2008. Tag mechanisms evaluated for coordination in open multi-agent systems. In Proceedings of the 8th International Workshop on Engineering Societies in the Agents World. 254--269. Google Scholar
Digital Library
- Claus, C. and Boutilier, C. 1998. The dynamics of reinforcement learning in cooperative multiagent systems. In Proceedings of AAAI’98. 746--752. Google Scholar
Digital Library
- Conitzer, V. and Sandholm, T. 2006. AWESOME: A general multiagent learning algorithm that converges in self-play and learns a best response against stationary opponents. In Proceedings of ICML’06. 83--90.Google Scholar
- Crandall, J. W. and Goodrich, M. A. 2005. Learning to teach and follow in repeated games. In Proceedings of the AAAI Workshop on Multiagent Learning.Google Scholar
- Greenwald, A. and Hall, K. 2003. Correlated Q-Learning. In Proceedings of ICML’03. 242--249.Google Scholar
- Hales, D. 2000. Cooperation without space or memory-tag, groups and the Prisoner’s Dilemma. In Multi-Agent-Based Simulation, Lecture Notes in Artificial Intelligence. Google Scholar
Digital Library
- Hales, D. and Edmonds, B. 2003. Evolving social rationality for MAS using “Tags”. In Proceedings of AA-MAS’03. 497--503. Google Scholar
Digital Library
- Hao, J. Y. and Leung, H. F. 2010. Strategy and fairness in repeated two-agent interaction. In Proceedings of ICTAI’10. 3--6. Google Scholar
Digital Library
- Hao, J. Y. and Leung, H. F. 2011. Learning to achieve social rationality using tag mechanism in repeated interactions. In Proceedings of ICTAI’11. 148--155. Google Scholar
Digital Library
- Hao, J. Y. and Leung, H. F. 2012. Learning to achieve socially optimal solutions in general-sum games. In Proceedings of the PRICAI’12. 88--99. Google Scholar
Digital Library
- Hoen, P. J., Tuyls, K.l, Panait, L., Luke, S., and Poutr, J. A. L. 2005. An overview of cooperative and competitive multiagent learning. In Proceedings of the 1st International Workshop on Learning and Adaption in Multi-Agent Systems. 1--46. Google Scholar
Digital Library
- Hogg, L. M. and Jennings, N. R. 1997. Socially rational agents. In Proceedings of the AAAI Fall Symposium on Socially Intelligent Agents. 61--63.Google Scholar
- Hogg, L. M. J. and Jennings, N. R. 2001. Socially intelligent reasoning for autonomous agents. IEEE Trans. Syst. Man. Cybernetics, Part A: Syst. Humans. 381--393. Google Scholar
Digital Library
- Holland, J. H., Holyoak, K., Nisbett, R., and Thagard, P. 1986. Induction: Processes of Inferences, Learning, and Discovery. MIT Press, Cambridge, MA. Google Scholar
Digital Library
- Howley, E. and O’Riordan, C. 2005. The emergence of cooperation among agents using simple fixed bias tagging. In Proceedings of the IEEE Congress on Evolutionary Computation.Google Scholar
Cross Ref
- Hu, J. and Wellman, M. 1998. Multiagent reinforcement learning: theoretical framework and an algorithm. In Proceedings of the ICML’98. Google Scholar
Digital Library
- Kapetanakis, S. and Kudenko, D. 2002. Reinforcement learning of coordination in cooperative multi-agent systems. In Proceedings of the AAAI’02. 326--331. Google Scholar
Digital Library
- Littman, M. 1994. Markov games as a framework for multi-agent reinforcement learning. In Proceedings of ICML’94. 322--328.Google Scholar
Cross Ref
- Matlock, M. and Sen, S. 2005. The success and failure of tag-mediated evolution of cooperation. In Proceedings of the 1st International Workshop on Learning and Adaption in Multi-Agent Systems. Springer, 155--164. Google Scholar
Digital Library
- Matlock, M. and Sen, S. 2007. Effective tag mechanisms for evolving coordination. In Proceedings of AAMAS’07. 1340--1347. Google Scholar
Digital Library
- Matlock, M. and Sen, S. 2009. Effective tag mechanisms for evolving coperation. In Proceedings of AAMAS’09. 489--496. Google Scholar
Digital Library
- Maynard, J. S. 1982. Evolution and The Theory of Games. Cambridge University Press, Cambridge, UK.Google Scholar
- Nowak, M. and Sigmund, K. 1993. A strategy of winstay, lose-shit that outperforms tit-for-tat in the prisoner’s dilemma game. Nature, 56--58.Google Scholar
- Osborne, M. J. and Rubinstein, A. 1994. A Course in Game Theory. MIT Press, Cambridge, MA.Google Scholar
- Panait, L. and Luke, S. 2005. Cooperative multi-agent learning: The state of the art. In Proceedings of AAMAS. 387--434.Google Scholar
- Pitt, J., Schaumeier, J., Busquets, D., and Macbeth, S. 2012. Self-organising common-pool resource allocation and canons of distributive justice. In Proceedings of SASO’12. 119--128. Google Scholar
Digital Library
- Riol, R. and Cohen, M. D. 2001. Cooperation withour reciprocity. Nature, 441--443.Google Scholar
- Sen, S. and Airiau, S. 2007. Emergence of norms through social learning. In Proceedings of IJCAI’07. 1507--1512. Google Scholar
Digital Library
- Verbeeck, K., Nowé, A., Parent, J., and Tuyls, K. 2006. Exploring selfish reinforcement learning in repeated games with stochastic rewards. In Proceedings of AAMAS’06. 239--269.Google Scholar
- Villatoro, D., Sabater-Mir, J., and Sen, S. 2011. Social instruments for robust convention emergence. In Proceedings of IJCAI’11. 420--425. Google Scholar
Digital Library
- Wakano, J. Y. and Yamamura, N. 2001. A simple learning strategy that realizes robust cooperation better than Pavlov in iterated Prisoner’s Dilemma. J. Ethol. 19, 9--15.Google Scholar
Cross Ref
- Watkins, C. J. C. H. and Dayan, P. D. 1992. Q-learning. Mach. Learn. 279--292. Google Scholar
Digital Library
Index Terms
Achieving Socially Optimal Outcomes in Multiagent Systems with Reinforcement Social Learning
Recommendations
Multiagent Reinforcement Social Learning toward Coordination in Cooperative Multiagent Systems
Most previous works on coordination in cooperative multiagent systems study the problem of how two (or more) players can coordinate on Pareto-optimal Nash equilibrium(s) through fixed and repeated interactions in the context of cooperative games. ...
The dynamics of reinforcement social learning in cooperative multiagent systems
IJCAI '13: Proceedings of the Twenty-Third international joint conference on Artificial IntelligenceCoordination in cooperative multiagent systems is an important problem in multiagent learning literature. In practical complex environments, the interactions between agents can be sparse, and each agent's interacting partners may change frequently and ...
Reinforcement social learning of coordination in cooperative multiagent systems
AAMAS '13: Proceedings of the 2013 international conference on Autonomous agents and multi-agent systemsCoordination in cooperative multiagent systems is an important problem and has received a lot of attention in multiagent learning literature. Most of previous works study the problem of how two (or more) players can coordinate on Pareto-optimal Nash ...






Comments