skip to main content
research-article
Open Access

Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision

Authors Info & Claims
Published:15 October 2020Publication History
Skip Abstract Section

Abstract

The interpretation of data is fundamental to machine learning. This paper investigates practices of image data annotation as performed in industrial contexts. We define data annotation as a sense-making practice, where annotators assign meaning to data through the use of labels. Previous human-centered investigations have largely focused on annotators? subjectivity as a major cause of biased labels. We propose a wider view on this issue: guided by constructivist grounded theory, we conducted several weeks of fieldwork at two annotation companies. We analyzed which structures, power relations, and naturalized impositions shape the interpretation of data. Our results show that the work of annotators is profoundly informed by the interests, values, and priorities of other actors above their station. Arbitrary classifications are vertically imposed on annotators, and through them, on data. This imposition is largely naturalized. Assigning meaning to data is often presented as a technical matter. This paper shows it is, in fact, an exercise of power with multiple implications for individuals and society.

References

  1. Muhammad Ali, Piotr Sapiezynski, Miranda Bogen, Aleksandra Korolova, Alan Mislove, and Aaron Rieke. 2019. Discrimination Through Optimization : How Facebook 's Ad Delivery Can Lead to Biased Outcomes. Proc. ACM Hum.-Comput. Interact., Vol. 3, CSCW (Nov. 2019), 199:1--199:30. https://doi.org/10.1145/3359301Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Ali Alkhatib and Michael Bernstein. 2019. Street-Level Algorithms : A Theory at the Gaps Between Policy and Decisions. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, 530:1--530:13. https://doi.org/10.1145/3290605.3300760Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Luis Araujo and Martin Spring. 2006. Services, Products, and the Institutional Structure of Production. Industrial Marketing Management, Vol. 35, 7 (Oct. 2006), 797--805. https://doi.org/10.1016/j.indmarman.2006.05.013Google ScholarGoogle ScholarCross RefCross Ref
  4. Paul Baker and Amanda Potts. 2013. `Why Do White People Have Thin Lips?' Google and the Perpetuation of Stereotypes via Auto-Complete Search Forms. Critical Discourse Studies, Vol. 10, 2 (May 2013), 187--204. https://doi.org/10.1080/17405904.2012.744320Google ScholarGoogle ScholarCross RefCross Ref
  5. Chelsea Barabas, Colin Doyle, JB Rubinovitz, and Karthik Dinakar. 2020. Studying up: Reorienting the Study of Algorithmic Fairness around Issues of Power. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT * '20). Association for Computing Machinery, Barcelona, Spain, 167--176. https://doi.org/10.1145/3351095.3372859Google ScholarGoogle Scholar
  6. Solon Barocas and Andrew D. Selbst. 2016. Big Data 's Disparate Impact. California Law Review, Vol. 104, 3 (2016), 671--732. https://doi.org/10.15779/Z38BG31Google ScholarGoogle Scholar
  7. Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing : Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics, Vol. 6 (2018), 587--604. https://doi.org/10.1162/tacl_a_00041Google ScholarGoogle ScholarCross RefCross Ref
  8. Jeanette Blomberg and Chuck Darrah. 2015. Toward an Anthropology of Services. The Design Journal, Vol. 18, 2 (2015), 171--192. https://doi.org/10.2752/175630615X14212498964196Google ScholarGoogle ScholarCross RefCross Ref
  9. Pierre Bourdieu. 1977. Outline of a Theory of Practice .Cambridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511812507Google ScholarGoogle Scholar
  10. Pierre Bourdieu. 1985. The Social Space and the Genesis of Groups. Theory and Society, Vol. 14, 6 (1985), 723--744. https://doi.org/10.1007/BF00174048Google ScholarGoogle ScholarCross RefCross Ref
  11. Pierre Bourdieu. 1989. Social Space and Symbolic Power. Sociological Theory, Vol. 7, 1 (1989), 14--25. https://doi.org/10.2307/202060Google ScholarGoogle ScholarCross RefCross Ref
  12. Pierre Bourdieu. 1990. The logic of practice reprinted ed.). Polity Press, Cambridge.Google ScholarGoogle Scholar
  13. Pierre Bourdieu. 1992. Language and Symbolic Power new ed.). Blackwell Publishers, Cambridge.Google ScholarGoogle Scholar
  14. Pierre Bourdieu. 2000. Pascalian Meditations .Stanford University Press, Stanford, Calif.Google ScholarGoogle Scholar
  15. Geoffrey C. Bowker. 2000. Biodiversity Datadiversity. Social Studies of Science, Vol. 30, 5 (Oct. 2000), 643--683. https://doi.org/10.1177/030631200030005001Google ScholarGoogle ScholarCross RefCross Ref
  16. Geoffrey C. Bowker and Susan Leigh Star. 1999. Sorting Things out: Classification and Its Consequences .MIT Press, Cambridge, Mass. BD175 .B68 1999Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. danah boyd and Kate Crawford. 2012. Critical Questions for Big Data : Provocations for a Cultural, Technological, and Scholarly Phenomenon. Information, Communication & Society, Vol. 15, 5 (June 2012), 662--679. https://doi.org/10.1080/1369118X.2012.678878Google ScholarGoogle ScholarCross RefCross Ref
  18. C. E. Brodley and M. A. Friedl. 1999. Identifying Mislabeled Training Data. Journal of Artificial Intelligence Research, Vol. 11 (Aug. 1999), 131--167. https://doi.org/10.1613/jair.606Google ScholarGoogle ScholarCross RefCross Ref
  19. Joy Buolamwini and Timnit Gebru. 2018. Gender Shades : Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, Vol. 81. PMLR, 77--91.Google ScholarGoogle Scholar
  20. Ryan Burns. 2019. New Frontiers of Philanthro -capitalism: Digital Technologies and Humanitarianism. Antipode, Vol. 51, 4 (April 2019), 1101--1122. https://doi.org/10.1111/anti.12534Google ScholarGoogle Scholar
  21. Kathy Charmaz. 2006. Constructing Grounded Theory : A Practical Guide through Qualitative Analysis .Sage Publications, London ; Thousand Oaks, Calif. H61.24 .C45 2006Google ScholarGoogle Scholar
  22. Justin Cheng and Dan Cosley. 2013. How Annotation Styles Influence Content and Preferences. In Proceedings of the 24th ACM Conference on Hypertext and Social Media - HT '13. Association for Computing Machinery, Paris, France, 214--218. https://doi.org/10.1145/2481492.2481519Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Angèle Christin. 2016. From Daguerreotypes to Algorithms : Machines, Expertise, and Three Forms of Objectivity. SIGCAS Computers and Society, Vol. 46, 1 (2016), 27--32. https://doi.org/10.1145/2908216.2908220Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Danielle Keats Citron and Frank Pasquale. 2014. The Scored Society : Due Process for Automated Predictions. Washington Law Review, Vol. 89, 1 (2014).Google ScholarGoogle Scholar
  25. Juliet M. Corbin and Anselm L. Strauss. 2015. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory fourth edition ed.). SAGE, Los Angeles. HA29 .C7724 2015Google ScholarGoogle Scholar
  26. Nick Couldry and Ulises A. Mejias. 2019. Data Colonialism : Rethinking Big Data 's Relation to the Contemporary Subject. Television & New Media, Vol. 20, 4 (May 2019), 336--349. https://doi.org/10.1177/1527476418796632Google ScholarGoogle ScholarCross RefCross Ref
  27. Kate Crawford and Trevor Paglen. 2019. Excavating AI. https://www.excavating.ai.Google ScholarGoogle Scholar
  28. Ciaran Cronin. 1996. Bourdieu and Foucault on Power and Modernity. Philosophy & Social Criticism, Vol. 22, 6 (Nov. 1996), 55--85. https://doi.org/10.1177/019145379602200603Google ScholarGoogle ScholarCross RefCross Ref
  29. Hannah Davis. 2020. A Dataset Is a Worldview. https://towardsdatascience.com/a-dataset-is-a-worldview-5328216dd44d.Google ScholarGoogle Scholar
  30. Catherine D'Ignazio and Lauren F. Klein. 2020. Data Feminism .The MIT Press, Cambridge, Massachusetts. HQ1190 .D574 2020Google ScholarGoogle Scholar
  31. Ravit Dotan and Smitha Milli. 2020. Value-Laden Disciplinary Shifts in Machine Learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT * '20). Association for Computing Machinery, Barcelona, Spain, 294. https://doi.org/10.1145/3351095.3373157Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Emile Durkheim and Marcel Mauss. 1963. Primitive Classification. University of Chicago Press.Google ScholarGoogle Scholar
  33. M. C. Elish and danah boyd. 2018. Situating Methods in the Magic of Big Data and AI. Communication Monographs, Vol. 85, 1 (Jan. 2018), 57--80. https://doi.org/10.1080/03637751.2017.1375130Google ScholarGoogle Scholar
  34. Virginia Eubanks. 2018. Automating Inequality : How High -Tech Tools Profile, Police, and Punish the Poor .St. Martin's Press, New York.Google ScholarGoogle Scholar
  35. Melanie Feinberg. 2017. A Design Perspective on Data. In CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). Association for Computing Machinery, Denver, Colorado, USA, 2952--2963. https://doi.org/10.1145/3025453.3025837Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating Named Entities in Twitter Data with Crowdsourcing. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon 's Mechanical Turk (CSLDAMT '10). Association for Computational Linguistics, Los Angeles, California, 80--88. https://doi.org/10.5555/1866696.1866709Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Marion Fourcade and Kieran Healy. 2013. Classification Situations: Life -Chances in the Neoliberal Era. Accounting, Organizations and Society, Vol. 38, 8 (Nov. 2013), 559--572. https://doi.org/10.1016/j.aos.2013.11.002Google ScholarGoogle ScholarCross RefCross Ref
  38. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumeé III, and Kate Crawford. 2018. Datasheets for Datasets. arXiv:1803.09010 (March 2018). arxiv: 1803.09010Google ScholarGoogle Scholar
  39. R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, and Jenny Huang. 2020. Garbage in, Garbage out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT * '20). Association for Computing Machinery, Barcelona, Spain, 325--336. https://doi.org/10.1145/3351095.3372862Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Bhavya Ghai, Q. Vera Liao, Yunfeng Zhang, and Klaus Mueller. 2020. Measuring Social Biases of Crowd Workers Using Counterfactual Queries. In Workshop on Fair & Responsible AI at ACM CHI Conference on Human Factors in Computing Systems. Honolulu, HI, USA.Google ScholarGoogle Scholar
  41. Tarleton Gillespie and Tarleton Gillespie. 2014. The Relevance of Algorithms. In Media Technologies : Essays on Communication, Materiality, and Society,, Pablo J. Boczkowski and Kirsten A. Foot (Eds.). The MIT Press, 167--194. https://doi.org/10.7551/mitpress/9780262525374.003.0009Google ScholarGoogle Scholar
  42. Lisa Gitelman (Ed.). 2013. "Raw Data" Is an Oxymoron .The MIT Press, Cambridge, Massachusetts ; London, England. Q355 .R385 2013Google ScholarGoogle Scholar
  43. Barney G. Glaser and Anselm L. Strauss. 1998. Grounded theory: Strategien qualitativer Forschung. Huber, Bern.Google ScholarGoogle Scholar
  44. Mary L. Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt, Boston.Google ScholarGoogle Scholar
  45. Foad Hamidi, Morgan Klaus Scheuerman, and Stacy M. Branham. 2018. Gender Recognition or Gender Reductionism - The Social Implications of Embedded Gender Recognition Systems.. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). Association for Computing Machinery, New York, 1--13. https://doi.org/10.1145/3173574.3173582Google ScholarGoogle Scholar
  46. Alex Hanna, Emily Denton, Andrew Smart, and Jamila Smith-Loud. 2020. Towards a Critical Race Methodology in Algorithmic Fairness. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT * '20). Association for Computing Machinery, Barcelona, Spain, 501--512. https://doi.org/10.1145/3351095.3372826Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018. The Dataset Nutrition Label : A Framework To Drive Higher Data Quality Standards. arXiv:1805.03677 (2018).Google ScholarGoogle Scholar
  48. Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300637Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Gunay Kazimzade and Milagros Miceli. 2020. Biased Priorities, Biased Outcomes : Three Recommendations for Ethics -Oriented Data Annotation Practices. In Proceedings of the AAAI /ACM Conference on Artificial Intelligence, Ethics, and Society. (AIES '20). Association for Computing Machinery, New York, NY, USA, 1--7. https://doi.org/10.1145/3375627.3375809Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Lucy Kimbell and Jeanette Blomberg. 2017. The Object of Service Design. In Designing for Service : Key Issues and New Directions. Bloomsbury Publishing, 81--94.Google ScholarGoogle Scholar
  51. Rob Kitchin. 2017. Thinking Critically about and Researching Algorithms. Information, Communication & Society, Vol. 20, 1 (Jan. 2017), 14--29. https://doi.org/10.1080/1369118X.2016.1154087Google ScholarGoogle Scholar
  52. Gary Klein, Jennifer K. Phillips, Erica L. Rall, and Deborah A. Peluso. 2007. A Data-Frame Theory of Sensemaking. In Expertise out of Context: Proceedings of the Sixth International Conference on Naturalistic Decision Making. Lawrence Erlbaum Associates Publishers, Mahwah, NJ, US, 113--155.Google ScholarGoogle Scholar
  53. Ulrike Klinger and Jakob Svensson. 2018. The End of Media Logics? On Algorithms and Agency. New Media & Society, Vol. 20, 12 (Dec. 2018), 4653--4670. https://doi.org/10.1177/1461444818779750Google ScholarGoogle ScholarCross RefCross Ref
  54. Natalia M Libakova and Ekaterina A Sertakova. 2015. The Method of Expert Interview as an Effective Research Procedure of Studying the Indigenous Peoples of the North. Journal of Siberian Federal University. Humanities & Social Sciences, Vol. 8, 1 (2015), 114--129. https://doi.org/10.17516/1997--1370--2015--8--1--114--129Google ScholarGoogle Scholar
  55. Michael A. Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna Wallach. 2020. Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, Honolulu, HI, USA, 1--14. https://doi.org/10.1145/3313831.3376445Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Astrid Mager. 2012. Algorithmic Ideology : How Capitalist Society Shapes Search Engines. Information, Communication & Society, Vol. 15, 5 (June 2012), 769--787. https://doi.org/10.1080/1369118X.2012.676056Google ScholarGoogle ScholarCross RefCross Ref
  57. Steffen Mau. 2019. The Metric Society: On the Quantification of the Social .Polity, Cambridge ; Medford, MA. MR 2800 M447 M5Google ScholarGoogle Scholar
  58. Frauke Mörike. 2019. Ethnography for Human Factors Researchers. Collecting and Interweaving Threads of HCI.Google ScholarGoogle Scholar
  59. Michael Muller. 2014. Curiosity, Creativity, and Surprise as Analytic Tools : Grounded Theory Method. In Ways of Knowing in HCI,, Judith S. Olson and Wendy A. Kellogg (Eds.). Springer, New York, NY, 25--48. https://doi.org/10.1007/978--1--4939-0378--8_2Google ScholarGoogle Scholar
  60. Michael Muller, Shion Guha, Eric P.S. Baumer, David Mimno, and N. Sadat Shami. 2016. Machine Learning and Grounded Theory Method : Convergence, Divergence, and Combination. In Proceedings of the 19th International Conference on Supporting Group Work (GROUP '16). Association for Computing Machinery, Sanibel Island, Florida, USA, 3--8. https://doi.org/10.1145/2957276.2957280Google ScholarGoogle Scholar
  61. Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q. Vera Liao, Casey Dugan, and Thomas Erickson. 2019. How Data Science Workers Work with Data : Discovery, Capture, Curation, Design, Creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, Glasgow, Scotland Uk, 1--15. https://doi.org/10.1145/3290605.3300356Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Safiya Umoja Noble. 2018. Algorithms of Oppression : How Search Engines Reinforce Racism .NYU Press, New York.Google ScholarGoogle Scholar
  63. Cathy O'Neil. 2017. Weapons of Math Destruction : How Big Data Increases Inequality and Threatens Democracy .PENGUIN BOOKS, London.Google ScholarGoogle Scholar
  64. Juho P"a"akkönen, Matti Nelimarkka, Jesse Haapoja, and Airi Lampinen. 2020. Bureaucracy as a Lens for Analyzing and Designing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, Honolulu, HI, USA., 1--14. https://doi.org/10.1145/3313831.3376780Google ScholarGoogle Scholar
  65. Samir Passi and Solon Barocas. 2019. Problem Formulation and Fairness. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT * '19). Association for Computing Machinery, Atlanta, GA, USA, 39--48. https://doi.org/10.1145/3287560.3287567Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. Samir Passi and Steven Jackson. 2017. Data Vision : Learning to See Through Algorithmic Abstraction. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). Association for Computing Machinery, Portland, Oregon, USA, 2436--2447. https://doi.org/10.1145/2998181.2998331Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. Samir Passi and Steven J. Jackson. 2018. Trust in Data Science : Collaboration, Translation, and Accountability in Corporate Data Science Projects. Proc. ACM Hum.-Comput. Interact., Vol. 2, CSCW (Nov. 2018), 1--28. https://doi.org/10.1145/3274405Google ScholarGoogle Scholar
  68. Kathleen H. Pine and Max Liboiron. 2015. The Politics of Measurement and Action. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). Association for Computing Machinery, New York, NY, USA, 3147--3156. https://doi.org/10.1145/2702123.2702298Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché-Buc, Emily Fox, and Hugo Larochelle. 2020. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program ). arXiv:2003.12206 (April 2020). arxiv: 2003.12206Google ScholarGoogle Scholar
  70. Alex Rosenblat, Tamara Kneese, and Danah Boyd. 2014. Networked Employment Discrimination. SSRN Electronic Journal (2014). https://doi.org/10.2139/ssrn.2543507Google ScholarGoogle Scholar
  71. Morgan Klaus Scheuerman, Jacob M. Paul, and Jed R. Brubaker. 2019. How Computers See Gender : An Evaluation of Gender Classification in Commercial Facial Analysis Services. Proc. ACM Hum.-Comput. Interact., Vol. 3, CSCW (Nov. 2019). https://doi.org/10.1145/3359246Google ScholarGoogle Scholar
  72. Morgan Klaus Scheuerman, Kandrea Wade, Caitlin Lustig, and Jed R Brubaker. 2020. How We 've Taught Algorithms to See Identity : Constructing Race and Gender in Image Databases for Facial Analysis. Proc. ACM Hum.-Comput. Interact., Vol. 4, CSCW1 (2020). https://doi.org/10.1145/3392866Google ScholarGoogle Scholar
  73. Nick Seaver. 2019. Knowing Algorithms. In digitalSTS : A Field Guide for Science & Technology Studies. Princeton University Press, PRINCETON; OXFORD, 412--422.Google ScholarGoogle Scholar
  74. Ismaïla Seck, Khouloud Dahmane, Pierre Duthon, and Gaëlle Loosli. 2018. Baselines and a Datasheet for the Cerema AWP Dataset. In Conférence d'Apprentissage CAp (Conférence d'Apprentissage Francophone 2018). Rouen, France. https://doi.org/10.13140/RG.2.2.36360.93448Google ScholarGoogle Scholar
  75. Susan Leigh Star and Anselm Strauss. 1999. Layers of Silence, Arenas of Voice : The Ecology of Visible and Invisible Work. Computer Supported Cooperative Work, Vol. 8, 1--2 (March 1999), 9--30. https://doi.org/10.1023/A:1008651105359Google ScholarGoogle ScholarDigital LibraryDigital Library
  76. Robert Thornberg. 2012. Informed Grounded Theory. Scandinavian Journal of Educational Research, Vol. 56, 3 (June 2012), 243--259. https://doi.org/10.1080/00313831.2011.581686Google ScholarGoogle Scholar
  77. Fabian L. Wauthier and Michael I. Jordan. 2011. Bayesian Bias Mitigation for Crowdsourcing. In Proceedings of the 24th International Conference on Neural Information Processing Systems (NIPS '11). Curran Associates Inc., Granada, Spain, 1800--1808.Google ScholarGoogle Scholar
  78. Jennifer Wortman Vaughan and Hanna Wallach. 2020. A Human -Centered Agenda for Intelligible Machine Learning. In Machines We Trust : Getting Along with Artificial Intelligence.Google ScholarGoogle Scholar
  79. Eviatar Zerubavel. 1993. The Fine Line : Making Distinctions in Everyday Life. 2nd ed. ed.). University of Chicago Press.Google ScholarGoogle Scholar
  80. Honglei Zhuang and Joel Young. 2015. Leveraging In -Batch Annotation Bias for Crowdsourced Active Learning. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining (WSDM '15). Association for Computing Machinery, Shanghai, China, 243--252. https://doi.org/10.1145/2684822.2685301Google ScholarGoogle ScholarDigital LibraryDigital Library
  81. Shoshana Zuboff. 2019. The Age of Surveillance Capitalism: The Fight for the Future at the New Frontier of Power .Profile Books, London.Google ScholarGoogle Scholar

Index Terms

  1. Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!