skip to main content
research-article

Experimental Assessment of Aggregation Principles in Argumentation-Enabled Collective Intelligence

Authors Info & Claims
Published:12 June 2017Publication History
Skip Abstract Section

Abstract

On the Web, there is always a need to aggregate opinions from the crowd (as in posts, social networks, forums, etc.). Different mechanisms have been implemented to capture these opinions such as Like in Facebook, Favorite in Twitter, thumbs-up/-down, flagging, and so on. However, in more contested domains (e.g., Wikipedia, political discussion, and climate change discussion), these mechanisms are not sufficient, since they only deal with each issue independently without considering the relationships between different claims. We can view a set of conflicting arguments as a graph in which the nodes represent arguments and the arcs between these nodes represent the defeat relation. A group of people can then collectively evaluate such graphs. To do this, the group must use a rule to aggregate their individual opinions about the entire argument graph. Here we present the first experimental evaluation of different principles commonly employed by aggregation rules presented in the literature. We use randomized controlled experiments to investigate which principles people consider better at aggregating opinions under different conditions. Our analysis reveals a number of factors, not captured by traditional formal models, that play an important role in determining the efficacy of aggregation. These results help bring formal models of argumentation closer to real-world application.

References

  1. Leila Amgoud, Jean-François Bonnefon, and Henri Prade. 2005. An argumentation-based approach to multiple criteria decision. In L. Godo (Ed.), Symbolic and Quantitative Approaches to Reasoning with Uncertainty. ECSQARU 2005. Lecture Notes in Computer Science, vol. 3571. Springer, Berlin, Heidelberg. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Gordana Apic, Matthew Betts, and Robert Russell. 2011. Content disputes in Wikipedia reflect geopolitical instability. PloS ONE 6, 6 (2011), e20902.Google ScholarGoogle ScholarCross RefCross Ref
  3. Kenneth J. Arrow. 1951. Social Choice and Individual Values. Wiley, New York, NY.Google ScholarGoogle Scholar
  4. Kenneth J. Arrow, A. K. Sen, and K. Suzumura (Eds.). 2002. Handbook of Social Choice and Welfare. Vol. 1. Elsevier Science Publishers (North-Holland).Google ScholarGoogle Scholar
  5. Edmond Awad, Richard Booth, Fernando Tohmé, and Iyad Rahwan. 2017. Judgement aggregation in multi-agent argumentation. J. Logic Comput. 27, 1 (2017), 227--259.Google ScholarGoogle ScholarCross RefCross Ref
  6. Anna Bassi. 2015. Voting systems and strategic manipulation: An experimental study. J. Theor. Pol. 27, 1 (2015), 58--85.Google ScholarGoogle ScholarCross RefCross Ref
  7. Salem Benferhat, Jean-François Bonnefon, and Rui da Silva Neves. 2005. An overview of possibilistic handling of default reasoning, with experimental studies. Synthese 146 (2005), 53--70.Google ScholarGoogle ScholarCross RefCross Ref
  8. Jean-François Bonnefon. 2007. How do individuals solve the doctrinal paradox in collective decisions? An empirical investigation. Psychol. Sci. 18 (2007), 753--755.Google ScholarGoogle ScholarCross RefCross Ref
  9. Jean-François Bonnefon. 2010. Behavioral evidence for framing effects in the resolution of the doctrinal paradox. Soc. Choice Welfare 34, 4 (2010), 631--641.Google ScholarGoogle ScholarCross RefCross Ref
  10. Jean-François Bonnefon, Didier Dubois, Hélène Fargier, and Sylvie Leblois. 2008. Qualitative heuristics for balancing the pros and cons. Theory Decision 65 (2008), 71--95.Google ScholarGoogle ScholarCross RefCross Ref
  11. Richard Booth, Edmond Awad, and Iyad Rahwan. 2014. Interval methods for judgment aggregation in argumentation. In Proceedings of the 14th International Conference on Principles of Knowledge Representation and Reasoning (KR’14). 594--597. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Luc Bovens and Wlodek Rabinowicz. 2006. Democratic answers to complex questions--an epistemic perspective. Synthese 150, 1 (2006), 131--153.Google ScholarGoogle ScholarCross RefCross Ref
  13. Simon Buckingham Shum. 2008. Cohere: Towards Web 2.0 argumentation. In Proceedings of the 2nd International Conference on Computational Models of Argument (COMMA’08), Philippe Besnard, Sylvie Doutre, and Anothony Hunter (Eds.). IOS Press, Amsterdam, The Netherlands, 97--108. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Martin Caminada. 2006. On the issue of reinstatement in argumentation. In Proceedings of the 10th European Conference on Logics in Artificial Intelligence (JELIA’06), Michael Fisher, Wiebe van der Hoek, Boris Konev, and Alexei Lisitsa (Eds.). Lecture Notes in Computer Science, Vol. 4160. Springer, 111--123. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Martin Caminada and Dov M. Gabbay. 2009. A logical account of formal argumentation. Studia Logica 93, 2--3 (2009), 109--145.Google ScholarGoogle ScholarCross RefCross Ref
  16. Martin Caminada and Gabriella Pigozzi. 2011. On judgment aggregation in abstract argumentation. Auton. Agents Multi-Agent Syst. 22, 1 (2011), 64--102. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Elad Dokow and Ron Holzman. 2010a. Aggregation of binary evaluations with abstentions. J. Econ. Theory 145, 2 (2010), 544--561.Google ScholarGoogle ScholarCross RefCross Ref
  18. Elad Dokow and Ron Holzman. 2010b. Aggregation of non-binary evaluations. Adv. Appl. Math. 45, 4 (2010), 487--504. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Didier Dubois, Hélène Fargier, and Jean-François Bonnefon. 2008. On the qualitative comparison of decisions having positive and negative features. J. Artif. Intell. Res. 32 (2008), 385--417. Google ScholarGoogle ScholarCross RefCross Ref
  20. Phan M. Dung. 1995. On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artif. Intell. 77, 2 (1995), 321--358. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Olive Jean Dunn. 1961. Multiple comparisons among means. J. Am. Statist. Assoc. 56, 293 (1961), 52--64.Google ScholarGoogle ScholarCross RefCross Ref
  22. Ulle Endriss and Raquel Fernández. 2013. Collective annotation of linguistic resources: Basic principles and a formal model. In BNAIC 2013: Proceedings of the 25th Benelux Conference on Artificial Intelligence, Delft, The Netherlands, November 7-8, 2013. Delft University of Technology (TU Delft); under the auspices of the Benelux Association for Artificial Intelligence (BNVKI) and the Dutch Research School for Information and Knowledge Systems (SIKS).Google ScholarGoogle Scholar
  23. Morris P. Fiorina and Charles R. Plott. 1978. Committee decisions under majority rule: An experimental study. Am. Pol. Sci. Rev. 72, 2 (1978), 575--598.Google ScholarGoogle ScholarCross RefCross Ref
  24. Robert Forsythe, Thomas Rietz, Roger Myerson, and Robert Weber. 1996. An experimental study of voting rules and polls in three-candidate elections. Int. J. Game Theory 25, 3 (1996), 355--383. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Davide Grossi and Gabriella Pigozzi. 2014. Judgment Aggregation: A Primer. Morgan 8 Claypool. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Serena Guarnaschelli, Richard D. McKelvey, and Thomas R. Palfrey. 2000. An experimental study of jury decision rules. Am. Pol. Sci. Rev. 94, 2 (2000), 407--423.Google ScholarGoogle ScholarCross RefCross Ref
  27. Joel M. Guttman. 1998. Unanimity and majority rule: The calculus of consent reconsidered*. Eur. J. Pol. Econ. 14, 2 (1998), 189--207.Google ScholarGoogle ScholarCross RefCross Ref
  28. Joshua Introne, Robert Laubacher, Gary Olson, and Thomas W. Malone. 2011. The Climate CoLab: Large scale model-based collaborative planning. In Collaboration Technologies and Systems (CTS), 2011 International Conference on. IEEE, 40--47.Google ScholarGoogle Scholar
  29. Mark Klein and Luca Iandoli. 2008. Supporting collaborative deliberation using a large-scale argumentation system: The MIT collaboratorium. Available at SSRN 1099082 4691-08 (2008).Google ScholarGoogle Scholar
  30. Tehila Kogut and Ilana Ritov. 2006. The “identified victim” effect: an identified group, or just a single individual? J. Behav. Dec. Making 18 (2006), 157--167.Google ScholarGoogle Scholar
  31. Jinxin Lin and Alberto O. Mendelzon. 1999. Knowledge base merging by majority. In Dynamic Worlds. Springer, 195--218.Google ScholarGoogle Scholar
  32. Christian List. 2006. The discursive dilemma and public reason. Ethics 18 (2006), 362--402.Google ScholarGoogle ScholarCross RefCross Ref
  33. Christian List. 2012. The theory of judgment aggregation: An introductory review. Synthese 187, 1 (2012), 179--207.Google ScholarGoogle ScholarCross RefCross Ref
  34. Christian List and Ben Polak. 2010. Introduction to judgment aggregation. J. Econ. Theory 145, 2 (2010), 441--466.Google ScholarGoogle ScholarCross RefCross Ref
  35. Christian List and Clemens Puppe. 2009. Judgment aggregation: A survey. In The Handbook of Rational and Social Choice, Paul Anand, Clemens Puppe, and Prasanta Pattanaik (Eds.). Oxford University Press, Oxford, UK.Google ScholarGoogle Scholar
  36. Seth Marvel, Jon Kleinberg, Robert Kleinberg, and Steven Strogatz. 2011. Continuous-time model of structural balance. Proc. Natl. Acad. Sci. U.S.A. 108, 5 (2011), 1771--1776.Google ScholarGoogle ScholarCross RefCross Ref
  37. Iain McLean, Arnold B. Urken, and Fiona Hewitt. 1995. Classics of Social Choice. University of Michigan Press.Google ScholarGoogle Scholar
  38. Luis Miller and Christoph Vanberg. 2015. Group size and decision rules in legislative bargaining. Eur. J. Pol. Econ. 37 (2015), 288--302.Google ScholarGoogle ScholarCross RefCross Ref
  39. Mike Oaksford and Ulrike Hahn. 2004. A bayesian approach to the argument from ignorance. Can. J. Exp. Psychol. 58, 2 (2004), 75.Google ScholarGoogle ScholarCross RefCross Ref
  40. Philip Pettit. 2001. Deliberative democracy and the discursive dilemma. Noûs 35, s1 (2001), 268--299.Google ScholarGoogle ScholarCross RefCross Ref
  41. Gabriella Pigozzi. 2006. Belief merging and the discursive dilemma: An argument-based account to paradoxes of judgment aggregation. Synthese 152, 2 (2006), 285--298.Google ScholarGoogle ScholarCross RefCross Ref
  42. Antonio Quesada. 2011. Parallel axiomatizations of majority and unanimity. Econ. Lett. 111, 2 (2011), 151--154.Google ScholarGoogle ScholarCross RefCross Ref
  43. Iyad Rahwan, Mohammed I. Madakkatel, Jean-François Bonnefon, Ruqiyabi N. Awan, and Sherief Abdallah. 2010. Behavioral experiments for assessing the abstract argumentation semantics of reinstatement. Cogn. Sci. 34, 8 (2010), 1483--1502.Google ScholarGoogle ScholarCross RefCross Ref
  44. Iyad Rahwan and Fernando Tohmé. 2010. Collective argument evaluation as judgement aggregation. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Volume 1-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 417--424. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Iyad Rahwan, Fouad Zablith, and Chris Reed. 2007. Laying the foundations for a world wide argument web. Artif. Intell. 171, 10 (2007), 897--921. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. David Rönnegard. 2015. The Fallacy of Corporate Moral Agency. Springer.Google ScholarGoogle Scholar
  47. Keith Stenning and Richard Cox. 1995. Attitudes to logical independence: Traits in quantifier interpretation. In Proceedings of the 17th Annual Conference of the Cognitive Science Society, J. D. Moore and J. Fain Lehman (Eds.). Hillsdale, NJ, 742--747. Lawrence Erlbaum Associates, Inc.Google ScholarGoogle Scholar
  48. Karine Van der Straeten, Jean-François Laslier, Nicolas Sauger, and André Blais. 2010. Strategic, sincere, and heuristic voting under four election rules: An experimental study. Soc. Choice Welfare 35, 3 (2010), 435--472.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Experimental Assessment of Aggregation Principles in Argumentation-Enabled Collective Intelligence

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Internet Technology
          ACM Transactions on Internet Technology  Volume 17, Issue 3
          Special Issue on Argumentation in Social Media and Regular Papers
          August 2017
          201 pages
          ISSN:1533-5399
          EISSN:1557-6051
          DOI:10.1145/3106680
          • Editor:
          • Munindar P. Singh
          Issue’s Table of Contents

          Copyright © 2017 Owner/Author

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 12 June 2017
          • Accepted: 1 January 2017
          • Received: 1 November 2016
          Published in toit Volume 17, Issue 3

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!