Abstract
On the Web, there is always a need to aggregate opinions from the crowd (as in posts, social networks, forums, etc.). Different mechanisms have been implemented to capture these opinions such as Like in Facebook, Favorite in Twitter, thumbs-up/-down, flagging, and so on. However, in more contested domains (e.g., Wikipedia, political discussion, and climate change discussion), these mechanisms are not sufficient, since they only deal with each issue independently without considering the relationships between different claims. We can view a set of conflicting arguments as a graph in which the nodes represent arguments and the arcs between these nodes represent the defeat relation. A group of people can then collectively evaluate such graphs. To do this, the group must use a rule to aggregate their individual opinions about the entire argument graph. Here we present the first experimental evaluation of different principles commonly employed by aggregation rules presented in the literature. We use randomized controlled experiments to investigate which principles people consider better at aggregating opinions under different conditions. Our analysis reveals a number of factors, not captured by traditional formal models, that play an important role in determining the efficacy of aggregation. These results help bring formal models of argumentation closer to real-world application.
- Leila Amgoud, Jean-François Bonnefon, and Henri Prade. 2005. An argumentation-based approach to multiple criteria decision. In L. Godo (Ed.), Symbolic and Quantitative Approaches to Reasoning with Uncertainty. ECSQARU 2005. Lecture Notes in Computer Science, vol. 3571. Springer, Berlin, Heidelberg. Google Scholar
Digital Library
- Gordana Apic, Matthew Betts, and Robert Russell. 2011. Content disputes in Wikipedia reflect geopolitical instability. PloS ONE 6, 6 (2011), e20902.Google Scholar
Cross Ref
- Kenneth J. Arrow. 1951. Social Choice and Individual Values. Wiley, New York, NY.Google Scholar
- Kenneth J. Arrow, A. K. Sen, and K. Suzumura (Eds.). 2002. Handbook of Social Choice and Welfare. Vol. 1. Elsevier Science Publishers (North-Holland).Google Scholar
- Edmond Awad, Richard Booth, Fernando Tohmé, and Iyad Rahwan. 2017. Judgement aggregation in multi-agent argumentation. J. Logic Comput. 27, 1 (2017), 227--259.Google Scholar
Cross Ref
- Anna Bassi. 2015. Voting systems and strategic manipulation: An experimental study. J. Theor. Pol. 27, 1 (2015), 58--85.Google Scholar
Cross Ref
- Salem Benferhat, Jean-François Bonnefon, and Rui da Silva Neves. 2005. An overview of possibilistic handling of default reasoning, with experimental studies. Synthese 146 (2005), 53--70.Google Scholar
Cross Ref
- Jean-François Bonnefon. 2007. How do individuals solve the doctrinal paradox in collective decisions? An empirical investigation. Psychol. Sci. 18 (2007), 753--755.Google Scholar
Cross Ref
- Jean-François Bonnefon. 2010. Behavioral evidence for framing effects in the resolution of the doctrinal paradox. Soc. Choice Welfare 34, 4 (2010), 631--641.Google Scholar
Cross Ref
- Jean-François Bonnefon, Didier Dubois, Hélène Fargier, and Sylvie Leblois. 2008. Qualitative heuristics for balancing the pros and cons. Theory Decision 65 (2008), 71--95.Google Scholar
Cross Ref
- Richard Booth, Edmond Awad, and Iyad Rahwan. 2014. Interval methods for judgment aggregation in argumentation. In Proceedings of the 14th International Conference on Principles of Knowledge Representation and Reasoning (KR’14). 594--597. Google Scholar
Digital Library
- Luc Bovens and Wlodek Rabinowicz. 2006. Democratic answers to complex questions--an epistemic perspective. Synthese 150, 1 (2006), 131--153.Google Scholar
Cross Ref
- Simon Buckingham Shum. 2008. Cohere: Towards Web 2.0 argumentation. In Proceedings of the 2nd International Conference on Computational Models of Argument (COMMA’08), Philippe Besnard, Sylvie Doutre, and Anothony Hunter (Eds.). IOS Press, Amsterdam, The Netherlands, 97--108. Google Scholar
Digital Library
- Martin Caminada. 2006. On the issue of reinstatement in argumentation. In Proceedings of the 10th European Conference on Logics in Artificial Intelligence (JELIA’06), Michael Fisher, Wiebe van der Hoek, Boris Konev, and Alexei Lisitsa (Eds.). Lecture Notes in Computer Science, Vol. 4160. Springer, 111--123. Google Scholar
Digital Library
- Martin Caminada and Dov M. Gabbay. 2009. A logical account of formal argumentation. Studia Logica 93, 2--3 (2009), 109--145.Google Scholar
Cross Ref
- Martin Caminada and Gabriella Pigozzi. 2011. On judgment aggregation in abstract argumentation. Auton. Agents Multi-Agent Syst. 22, 1 (2011), 64--102. Google Scholar
Digital Library
- Elad Dokow and Ron Holzman. 2010a. Aggregation of binary evaluations with abstentions. J. Econ. Theory 145, 2 (2010), 544--561.Google Scholar
Cross Ref
- Elad Dokow and Ron Holzman. 2010b. Aggregation of non-binary evaluations. Adv. Appl. Math. 45, 4 (2010), 487--504. Google Scholar
Digital Library
- Didier Dubois, Hélène Fargier, and Jean-François Bonnefon. 2008. On the qualitative comparison of decisions having positive and negative features. J. Artif. Intell. Res. 32 (2008), 385--417. Google Scholar
Cross Ref
- Phan M. Dung. 1995. On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artif. Intell. 77, 2 (1995), 321--358. Google Scholar
Digital Library
- Olive Jean Dunn. 1961. Multiple comparisons among means. J. Am. Statist. Assoc. 56, 293 (1961), 52--64.Google Scholar
Cross Ref
- Ulle Endriss and Raquel Fernández. 2013. Collective annotation of linguistic resources: Basic principles and a formal model. In BNAIC 2013: Proceedings of the 25th Benelux Conference on Artificial Intelligence, Delft, The Netherlands, November 7-8, 2013. Delft University of Technology (TU Delft); under the auspices of the Benelux Association for Artificial Intelligence (BNVKI) and the Dutch Research School for Information and Knowledge Systems (SIKS).Google Scholar
- Morris P. Fiorina and Charles R. Plott. 1978. Committee decisions under majority rule: An experimental study. Am. Pol. Sci. Rev. 72, 2 (1978), 575--598.Google Scholar
Cross Ref
- Robert Forsythe, Thomas Rietz, Roger Myerson, and Robert Weber. 1996. An experimental study of voting rules and polls in three-candidate elections. Int. J. Game Theory 25, 3 (1996), 355--383. Google Scholar
Digital Library
- Davide Grossi and Gabriella Pigozzi. 2014. Judgment Aggregation: A Primer. Morgan 8 Claypool. Google Scholar
Digital Library
- Serena Guarnaschelli, Richard D. McKelvey, and Thomas R. Palfrey. 2000. An experimental study of jury decision rules. Am. Pol. Sci. Rev. 94, 2 (2000), 407--423.Google Scholar
Cross Ref
- Joel M. Guttman. 1998. Unanimity and majority rule: The calculus of consent reconsidered*. Eur. J. Pol. Econ. 14, 2 (1998), 189--207.Google Scholar
Cross Ref
- Joshua Introne, Robert Laubacher, Gary Olson, and Thomas W. Malone. 2011. The Climate CoLab: Large scale model-based collaborative planning. In Collaboration Technologies and Systems (CTS), 2011 International Conference on. IEEE, 40--47.Google Scholar
- Mark Klein and Luca Iandoli. 2008. Supporting collaborative deliberation using a large-scale argumentation system: The MIT collaboratorium. Available at SSRN 1099082 4691-08 (2008).Google Scholar
- Tehila Kogut and Ilana Ritov. 2006. The “identified victim” effect: an identified group, or just a single individual? J. Behav. Dec. Making 18 (2006), 157--167.Google Scholar
- Jinxin Lin and Alberto O. Mendelzon. 1999. Knowledge base merging by majority. In Dynamic Worlds. Springer, 195--218.Google Scholar
- Christian List. 2006. The discursive dilemma and public reason. Ethics 18 (2006), 362--402.Google Scholar
Cross Ref
- Christian List. 2012. The theory of judgment aggregation: An introductory review. Synthese 187, 1 (2012), 179--207.Google Scholar
Cross Ref
- Christian List and Ben Polak. 2010. Introduction to judgment aggregation. J. Econ. Theory 145, 2 (2010), 441--466.Google Scholar
Cross Ref
- Christian List and Clemens Puppe. 2009. Judgment aggregation: A survey. In The Handbook of Rational and Social Choice, Paul Anand, Clemens Puppe, and Prasanta Pattanaik (Eds.). Oxford University Press, Oxford, UK.Google Scholar
- Seth Marvel, Jon Kleinberg, Robert Kleinberg, and Steven Strogatz. 2011. Continuous-time model of structural balance. Proc. Natl. Acad. Sci. U.S.A. 108, 5 (2011), 1771--1776.Google Scholar
Cross Ref
- Iain McLean, Arnold B. Urken, and Fiona Hewitt. 1995. Classics of Social Choice. University of Michigan Press.Google Scholar
- Luis Miller and Christoph Vanberg. 2015. Group size and decision rules in legislative bargaining. Eur. J. Pol. Econ. 37 (2015), 288--302.Google Scholar
Cross Ref
- Mike Oaksford and Ulrike Hahn. 2004. A bayesian approach to the argument from ignorance. Can. J. Exp. Psychol. 58, 2 (2004), 75.Google Scholar
Cross Ref
- Philip Pettit. 2001. Deliberative democracy and the discursive dilemma. Noûs 35, s1 (2001), 268--299.Google Scholar
Cross Ref
- Gabriella Pigozzi. 2006. Belief merging and the discursive dilemma: An argument-based account to paradoxes of judgment aggregation. Synthese 152, 2 (2006), 285--298.Google Scholar
Cross Ref
- Antonio Quesada. 2011. Parallel axiomatizations of majority and unanimity. Econ. Lett. 111, 2 (2011), 151--154.Google Scholar
Cross Ref
- Iyad Rahwan, Mohammed I. Madakkatel, Jean-François Bonnefon, Ruqiyabi N. Awan, and Sherief Abdallah. 2010. Behavioral experiments for assessing the abstract argumentation semantics of reinstatement. Cogn. Sci. 34, 8 (2010), 1483--1502.Google Scholar
Cross Ref
- Iyad Rahwan and Fernando Tohmé. 2010. Collective argument evaluation as judgement aggregation. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Volume 1-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 417--424. Google Scholar
Digital Library
- Iyad Rahwan, Fouad Zablith, and Chris Reed. 2007. Laying the foundations for a world wide argument web. Artif. Intell. 171, 10 (2007), 897--921. Google Scholar
Digital Library
- David Rönnegard. 2015. The Fallacy of Corporate Moral Agency. Springer.Google Scholar
- Keith Stenning and Richard Cox. 1995. Attitudes to logical independence: Traits in quantifier interpretation. In Proceedings of the 17th Annual Conference of the Cognitive Science Society, J. D. Moore and J. Fain Lehman (Eds.). Hillsdale, NJ, 742--747. Lawrence Erlbaum Associates, Inc.Google Scholar
- Karine Van der Straeten, Jean-François Laslier, Nicolas Sauger, and André Blais. 2010. Strategic, sincere, and heuristic voting under four election rules: An experimental study. Soc. Choice Welfare 35, 3 (2010), 435--472.Google Scholar
Cross Ref
Index Terms
Experimental Assessment of Aggregation Principles in Argumentation-Enabled Collective Intelligence
Recommendations
Computing Arguments and Attacks in Assumption-Based Argumentation
CaSAPI (Credulous and Skeptical Argumentation: Prolog Implementation) 3.0 determines the acceptability of claims, using the general-purpose framework of assumption-based argumentation, under the semantics of admissible extensions. This framework reduces ...
A structured argumentation system with backing and undercutting
This work introduces Extended Defeasible Logic Programming (E-DeLP), a structured argumentation system enabling the expression of reasons for and against using defeasible rules. E-DeLPextends the formalism of Defeasible Logic Programming (DeLP) by ...
On judgment aggregation in abstract argumentation
Judgment aggregation is a field in which individuals are required to vote for or against a certain decision (the conclusion) while providing reasons for their choice. The reasons and the conclusion are logically connected propositions. The problem is ...






Comments