ACM Digital Library home
ACM home
Advanced Search
10.1145/3514094.3534189acmconferencesArticle/Chapter ViewAbstractPublication PagesaiesConference Proceedingsconference-collections
research-article
Open Access

Resume Format, LinkedIn URLs and Other Unexpected Influences on AI Personality Prediction in Hiring: Results of an Audit

AIES '22: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and SocietyJuly 2022Pages 572–587https://doi.org/10.1145/3514094.3534189
Published:27 July 2022Publication History
  • 0citation
  • 846
  • Downloads
Metrics
Total Citations0
Total Downloads846
Last 12 Months846
Last 6 weeks266

AIES '22: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society

Resume Format, LinkedIn URLs and Other Unexpected Influences on AI Personality Prediction in Hiring: Results of an Audit
Pages 572–587
PreviousChapterNextChapter

ABSTRACT

Automated hiring systems are among the fastest-developing of all high-stakes AI systems. Among these are algorithmic personality tests that use insights from psychometric testing, and promise to surface personality traits indicative of future success based on job seekers' resumes or social media profiles. We interrogate the reliability of such systems using stability of the outputs they produce, noting that reliability is a necessary, but not a sufficient, condition for validity. We develop a methodology for an external audit of stability of algorithmic personality tests, and instantiate this methodology in an audit of two systems, Humantic AI and Crystal. Rather than challenging or affirming the assumptions made in psychometric testing -- that personality traits are meaningful and measurable constructs, and that they are indicative of future success on the job -- we frame our methodology around testing the underlying assumptions made by the vendors of the algorithmic personality tests themselves.

In our audit of Humantic AI and Crystal, we find that both systems show substantial instability on key facets of measurement, and so cannot be considered valid testing instruments. For example, Crystal frequently computes different personality scores if the same resume is given in PDF vs. in raw text, violating the assumption that the output of an algorithmic personality test is stable across job-irrelevant input variations. Among other notable findings is evidence of persistent --- and often incorrect --- data linkage by Humantic AI.

An open-source implementation of our auditing methodology, and of the audits of Humantic AI and Crystal, is available at https://github.com/DataResponsibly/hiring-stability-audit.

Skip Supplemental Material Section

Supplemental Material

AIES22-p71.mp4

Presentation video for "Resume Format, LinkedIn URLs and Other Unexpected Influences on AI Personality Prediction in Hiring: Results of an Audit." Algorithmic personality tests are in broad use, but do they work? We seek to answer this question by interrogating the validity of algorithmic personality tests that claim to estimate a job seeker's personality based on their resume or social media profile. We develop a methodology for auditing the stability of predictions made by these tests. Crucially, we frame our methodology around testing the assumptions made by the vendors of these tools. We use this methodology to conduct an external audit of two commercial systems, Humantic AI and Crystal, over a dataset of job applicant profiles collected through an IRB-approved study. The key take-away is that both systems show instability on key facets of measurement, and so cannot be considered valid testing instruments for pre-hire assessment.

mp4

28.7 MB

Play streamDownload

References

  1. Ifeoma Ajunwa. 2021. An Auditing Imperative for Automated Hiring Systems. Harvard Journal of Law & Technology 34, 2 (2021), 80. https://doi.org/10.2139/ssrn.3437631Google ScholarGoogle Scholar
  2. Luisa Araujo, Andrea Saltelli, and Sylke V. Schnepf. 2017. Do PISA data justify PISA-based education policy? International Journal of Comparative Education and Development 19, 1 (Jan. 2017), 20--34. https://doi.org/10.1108/IJCED-12--2016-0023 Publisher: Emerald Publishing Limited.Google ScholarGoogle ScholarCross RefCross Ref
  3. Jack Bandy. 2021. Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits. Proceedings of the ACM on Human-Computer Interaction 5 (2021). Issue CSCW1. https://doi.org/10.1145/3449148Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Solon Barocas and Andrew D. Selbst. 2016. Big Data's Disparate Impact. California Law Review 671, 104 (2016). https://doi.org/10.2139/ssrn.2477899Google ScholarGoogle Scholar
  5. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B (Methodological) 57, 1 (1995), 289--300. https://doi.org/10.1111/j.2517--6161.1995.tb02031.x arXiv:https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2517--6161.1995.tb02031.xGoogle ScholarGoogle Scholar
  6. Miranda Bogen and Aaron Rieke. 2018. Help Wanted: An Exploration of Hiring Algorithms, Equity and Bias. Technical Report. Upturn. 75 pages. https://www.upturn.org/static/reports/2018/hiring-algorithms/files/Upturn%20--%20Help%20Wanted%20-%20An%20Exploration%20of%20Hiring%20Algorithms,%20Equity%20and%20Bias.pdfGoogle ScholarGoogle Scholar
  7. Shea Brown, Jovana Davidovic, and Ali Hasan. 2021. The algorithm audit: Scoring the algorithms that score us. Big Data & Society 8, 1 (Jan. 2021), 2053951720983865. https://doi.org/10.1177/2053951720983865 Publisher: SAGE Publications Ltd.Google ScholarGoogle Scholar
  8. Jean Cardinet, Yvan Tourneur, and Linda Allal. 1976. The Symmetry of Generalizability Theory: Applications to Educational Measurement. Journal of Educational Measurement 13, 2 (1976), 119--135. https://www.jstor.org/stable/1434233 Publisher: National Council on Measurement in Education, Wiley.Google ScholarGoogle ScholarCross RefCross Ref
  9. Edward Carmines and Richard Zeller. 1979. Reliability and Validity Assessment. SAGE Publications, Inc., 2455 Teller Road, Thousand Oaks California 91320 United States of America. https://doi.org/10.4135/9781412985642Google ScholarGoogle Scholar
  10. Le Chen, Ruijun Ma, Anikó Hannák, and Christo Wilson. 2018. Investigating the Impact of Gender on Rank in Resume Search Engines. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1--14. https://doi.org/10.1145/3173574.3174225Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. "The European Commission". 2021. Better regulation toolbox. https://ec.europa.eu/info/law/law-making-process/planning-and-proposing-law/better-regulation-why-and-how/better-regulation-guidelines-and-toolbox/better-regulation-toolbox_en.Google ScholarGoogle Scholar
  12. Jeffrey Dastin. 2018. Amazon scraps secret AI recruiting tool that showed bias against women. Reuters (Oct. 2018). https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08GGoogle ScholarGoogle Scholar
  13. Amit Datta, Michael Carl Tschantz, and Anupam Datta. 2015. Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination. Proceedings on Privacy Enhancing Technologies 2015, 1 (April 2015), 92--112. https://doi.org/10.1515/popets-2015-0007Google ScholarGoogle Scholar
  14. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, Atlanta GA USA, 120--128. https://doi.org/10.1145/3287560.3287572Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Merve Emre. 2018. The Personality Brokers: The Strange History of Myers-Briggs and the Birth of Personality Testing (first ed.). Doubleday, New York.Google ScholarGoogle Scholar
  16. Equal Employment Opportunity Commission (EEOC), Civil Service Commission, Department of Labor, and Department of Justice. 1978. Uniform guidelines on employee selection procedures. Federal Register (Aug. 1978). https://www.ojp.gov/pdffiles1/Digitization/58846NCJRS.pdfGoogle ScholarGoogle Scholar
  17. Jessica Fjeld, Nele Achten, Hannah Hilligoss, Adam Nagy, and Madhulika Srikumar. 2020. Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-Based Approaches to Principles for AI. SSRN Electronic Journal (2020). https://doi.org/10.2139/ssrn.3518482Google ScholarGoogle Scholar
  18. Batya Friedman and Helen Nissenbaum. 1996. Bias in computer systems. ACM Transactions on Information Systems 14, 3 (July 1996), 330--347. https://doi.org/10.1145/230538.230561Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Alessandro Galli, Mario Giampietro, Steve Goldfinger, Elias Lazarus, David Lin, Andrea Saltelli, Mathis Wackernagel, and Felix Müller. 2016. Questioning the Ecological Footprint. Ecological Indicators 69 (Oct. 2016), 224--232. https://doi.org/10.1016/j.ecolind.2016.04.014Google ScholarGoogle Scholar
  20. Leonard D Goodstein and Richard I Lanyon. 1999. Applications of Personality Assessment to the Workplace: A Review. Journal of Business and Psychology 13, 3 (1999), 32.Google ScholarGoogle ScholarCross RefCross Ref
  21. Robert M. Guion and Richard F. Gottier. 1965. Validity Of Personality Measures In Personnel Selection. Personnel Psychology 18, 2 (1965), 135--164. https://doi.org/10.1111/j.1744--6570.1965.tb00273.x _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1744--6570.1965.tb00273.x.Google ScholarGoogle ScholarCross RefCross Ref
  22. Richard A. Guzzo, Alexis A. Fink, Eden King, Scott Tonidandel, and Ronald S. Landis. 2015. Big Data Recommendations for Industrial-Organizational Psychology. Industrial and Organizational Psychology 8, 4 (Dec. 2015), 491--508. https://doi.org/10.1017/iop.2015.40 Num Pages: 18 Place: Bowling Green, United Kingdom Publisher: Cambridge University Press.Google ScholarGoogle ScholarCross RefCross Ref
  23. Thilo Hagendorff. 2020. The Ethics of AI Ethics: An Evaluation of Guidelines. Minds and Machines 30, 1 (March 2020), 99--120. https://doi.org/10.1007/s11023-020-09517--8Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Anikó Hannák, Claudia Wagner, David Garcia, Alan Mislove, Markus Strohmaier, and Christo Wilson. 2017. Bias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, Portland Oregon USA, 1914--1933. https://doi.org/10.1145/2998181.2998327Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Leaetta M. Hough, Newell K. Eaton, Marvin D. Dunnette, John D. Kamp, and Rodney A. McCloy. 1990. Criterion-related validities of personality constructs and the effect of response distortion on those validities. Journal of Applied Psychology 75, 5 (1990), 581--595. https://doi.org/10.1037/0021--9010.75.5.581 Place: US Publisher: American Psychological Association.Google ScholarGoogle ScholarCross RefCross Ref
  26. Gregory M. Hurtz and John J. Donovan. 2000. Personality and job performance: The Big Five revisited. Journal of Applied Psychology 85, 6 (2000), 869--879. https://doi.org/10.1037/0021--9010.85.6.869 Place: US Publisher: American Psychological Association.Google ScholarGoogle ScholarCross RefCross Ref
  27. Aislinn Kelly-Lyth. 2020. Challenging Biased Hiring Algorithms. SSRN Scholarly Paper ID 3744248. Social Science Research Network, Rochester, NY. https://papers.ssrn.com/abstract=3744248Google ScholarGoogle Scholar
  28. Pauline T Kim. 2017. Data-Driven Discrimination at Work. William & Mary Law 58 (2017), 81.Google ScholarGoogle Scholar
  29. Adriano Koshiyama, Emre Kazim, Philip Treleaven, Pete Rai, Lukasz Szpruch, Giles Pavey, Ghazi Ahamat, Franziska Leutner, Randy Goebel, Andrew Knight, Janet Adams, Christina Hitrova, Jeremy Barnett, Parashkev Nachev, David Barber, Tomas Chamorro-Premuzic, Konstantin Klemmer, Miro Gregorovic, Shakeel Khan, and Elizabeth Lomas. 2021. Towards Algorithm Auditing: A Survey on Managing Legal, Ethical and Technological Risks of AI, ML and Associated Algorithms. Technical Report. https://www.ssrn.com/abstract=3778998Google ScholarGoogle Scholar
  30. Max Langenkamp, Allan Costa, and Chris Cheung. 2020. Hiring Fairly in the Age of Algorithms. arXiv:2004.07132 [cs] (April 2020). http://arxiv.org/abs/2004.07132 arXiv: 2004.07132.Google ScholarGoogle Scholar
  31. Samuele Lo Piano and Marguerite Robinson. 2019. Nutrition and public health economic evaluations under the lenses of post normal science | Elsevier Enhanced Reader. Futures 112 (2019). https://doi.org/10.1016/j.futures.2019.06.008Google ScholarGoogle Scholar
  32. Kira Lussier. 2018. Temperamental Workers: Psychology, Business, and the Humm-Wadsworth Temperament Scale in Interwar America. History of Psychology 21, 2 (2018), 79. https://doi.org/10.1037/hop0000081 Publisher: US: American Psychological Association.Google ScholarGoogle ScholarCross RefCross Ref
  33. Dori Meinert. 2015. What Do Personality Tests Really Reveal? https://www.shrm.org/hr-today/news/hr-magazine/pages/0615-personality-tests.aspxGoogle ScholarGoogle Scholar
  34. Jacob Metcalf, Emanuel Moss, Elizabeth Anne Watkins, Ranjit Singh, and Madeleine Clare Elish. 2021. Algorithmic Impact Assessments and Accountability: The Co-construction of Impacts. https://papers.ssrn.com/abstract=3736261Google ScholarGoogle Scholar
  35. Frederick P. Morgeson, Michael A. Campion, Robert L. Dipboye, John R. Hollenbeck, Kevin Murphy, and Neal Schmitt. 2007. Reconsidering the use of personality tests in personnel selection contexts. Personnel Psychology 60, 3 (2007), 683--729. https://doi.org/10.1111/j.1744--6570.2007.00089.x Place: United Kingdom Publisher: Blackwell Publishing.Google ScholarGoogle ScholarCross RefCross Ref
  36. James R. Morrow and Allen w. Jackson. 1993. How "Significant" is Your Reliability? Research Quarterly for Exercise and Sport 64, 3 (Sept. 1993), 352--355. https://doi.org/10.1080/02701367.1993.10608821Google ScholarGoogle ScholarCross RefCross Ref
  37. Ralph O. Mueller and Thomas R. Knapp. 2018. Reliability and Validity. In The Reviewer's Guide to Quantitative Methods in the Social Sciences (2 ed.). Routledge. Num Pages: 5.Google ScholarGoogle Scholar
  38. Jakob Mökander, Jessica Morley, Mariarosaria Taddeo, and Luciano Floridi. 2021. Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations. Science and Engineering Ethics 27, 4 (Aug. 2021), 44. https://doi.org/10.1007/s11948-021-00319--4Google ScholarGoogle Scholar
  39. Jum C. Nunnally and Ira H. Bernstein. 1994. Psychometric Theory (3 ed.). McGraw-Hill, Inc.Google ScholarGoogle Scholar
  40. Luis Oala, Jana Fehr, Luca Gilli, Pradeep Balachandran, Alixandro Werneck Leite, Saul Calderon-Ramirez, Danny Xie Li, Gabriel Nobis, Erick Alejandro Muñoz Alvarado, Giovanna Jaramillo-Gutierrez, Christian Matek, Arun Shroff, Ferath Kherif, Bruno Sanguinetti, and Thomas Wiegand. 2020. ML4H Auditing: From Paper to Practice. In Machine Learning for Health. PMLR, 280--317. https://proceedings.mlr.press/v136/oala20a.html ISSN: 2640--3498.Google ScholarGoogle Scholar
  41. O'Neil Risk Consulting and Algorithmic Auditing (ORCAA). 2020. Description of Algorithmic Audit: Pre-built Assessments. Technical Report. https://techinquiry.org/HireVue-ORCAA.pdfGoogle ScholarGoogle Scholar
  42. Manish Raghavan, Solon Barocas, Jon Kleinberg, and Karen Levy. 2020. Mitigating bias in algorithmic hiring: evaluating claims and practices. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. ACM, Barcelona Spain, 469--481. https://doi.org/10.1145/3351095.3372828Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. ACM, Barcelona, Spain, 12. https://doi.org/10.1145/3351095.3372873Google ScholarGoogle Scholar
  44. Saman Razavi, Anthony Jakeman, Andrea Saltelli, Clémentine Prieur, Bertrand Iooss, Emanuele Borgonovo, Elmar Plischke, Samuele Lo Piano, Takuya Iwanaga, William Becker, Stefano Tarantola, Joseph H. A. Guillaume, John Jakeman, Hoshin Gupta, Nicola Melillo, Giovanni Rabitti, Vincent Chabridon, Qingyun Duan, Xifu Sun, Stefán Smith, Razi Sheikholeslami, Nasim Hosseini, Masoud Asadzadeh, Arnald Puy, Sergei Kucherenko, and Holger R. Maier. 2021. The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support. Environmental Modelling & Software 137 (March 2021), 104954. https://doi.org/10.1016/j.envsoft.2020.104954Google ScholarGoogle Scholar
  45. Alene K. Rhea, Kelsey Markey, Lauren D'Arinzo, Hilke Schellmann, Mona Sloane, Paul Squires, Falaah Arif Khan, and Julia Stoyanovich. 2022. An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring. CoRR abs/2201.09151 (2022). arXiv:2201.09151 https://arxiv.org/abs/2201.09151Google ScholarGoogle Scholar
  46. Ronald E. Robertson, Shan Jiang, Kenneth Joseph, Lisa Friedland, David Lazer, and Christo Wilson. 2018. Auditing Partisan Audience Bias within Google Search. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (Nov. 2018), 1--22. https://doi.org/10.1145/3274417Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Andrea Saltelli, Gabriele Bammer, Isabelle Bruno, Erica Charters, Monica Di Fiore, Emmanuel Didier, Wendy Nelson Espeland, John Kay, Samuele Lo Piano, Deborah Mayo, Roger Pielke Jr, Tommaso Portaluri, Theodore M. Porter, Arnald Puy, Ismael Rafols, Jerome R. Ravetz, Erik Reinert, Daniel Sarewitz, Philip B. Stark, Andrew Stirling, Jeroen van der Sluijs, and Paolo Vineis. 2020. Five ways to ensure that models serve society: a manifesto. Nature 582, 7813 (June 2020), 482--484. https://doi.org/10.1038/d41586-020-01812--9 Bandiera_abtest: a Cg_type: Comment Number: 7813 Publisher: Nature Publishing Group Subject_term: Communication, Policy, Epidemiology.Google ScholarGoogle ScholarCross RefCross Ref
  48. Andrea Saltelli and Samuele Lo Piano. 2017. Problematic Quantifications: a Critical Appraisal of Scenario Making for a Global ?Sustainable' Food Production. Food Ethics 1, 2 (Aug. 2017), 173--179. https://doi.org/10.1007/s41055-017-0020--6Google ScholarGoogle Scholar
  49. Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. In Data and Discrimination: Converting Critical Concerns into Productive Inquiry. Seattle, WA, USA, 23.Google ScholarGoogle Scholar
  50. Hilke Schellmann, Jennifer Strong, Ian, and Siegel. 2021. Hired by an algorithm. https://podcasts.apple.com/us/podcast/hired-by-an-algorithm/id1523584878?i=1000526571833. In Machines We Trust podcast series, MIT Technology Review (2021).Google ScholarGoogle Scholar
  51. Hilke Schellmann, Jennifer Strong, Ian, and Siegel. 2021. Want a job? The AI will see you now. https://podcasts.apple.com/us/podcast/want-a-job-the-ai-will-see-you-now/id1523584878?i=1000528104144. In Machines We Trust podcast series, MIT Technology Review (2021).Google ScholarGoogle Scholar
  52. Frank L. Schmidt, Huy Le, and Remus Ilies. 2003. Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual-differences constructs. Psychological Methods 8, 2 (2003), 206. https://doi.org/10.1037/1082--989X.8.2.206 Publisher: US: American Psychological Association.Google ScholarGoogle ScholarCross RefCross Ref
  53. Neal Schmitt, Richard Z. Gooding, Raymond A. Noe, and Michael Kirsch. 1984. Metaanalyses of Validity Studies Published Between 1964 and 1982 and the Investigation of Study Characteristics. Personnel Psychology 37, 3 (1984), 407--422. https://doi.org/10.1111/j.1744--6570.1984.tb00519.x _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1744--6570.1984.tb00519.x.Google ScholarGoogle ScholarCross RefCross Ref
  54. Science Advice for Policy by European Academies (SAPEA). 2019. Making sense of science for policy under conditions of complexity and uncertainty. Science Advice for Policy by European Academies, DE. https://doi.org/10.26356/masosGoogle ScholarGoogle Scholar
  55. Wesley A. Scroggins, Steven L. Thomas, and Jerry A. Morris. 2008. Psychological Testing in Personnel Selection, Part I: A Century of Psychological Testing. Public Personnel Management 37, 1 (March 2008), 99--109. https://doi.org/10.1177/009102600803700107 Publisher: SAGE Publications Inc.Google ScholarGoogle Scholar
  56. Shubham Sharma, Jette Henderson, and Joydeep Ghosh. 2020. CERTIFAI: A Common Framework to Provide Explanations and Analyse the Fairness and Robustness of Black-box Models. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES '20). Association for Computing Machinery, New York, NY, USA, 166--172. https://doi.org/10.1145/3375627.3375812Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Ben Shneiderman. 2020. Bridging the Gap Between Ethics and Practice: Guidelines for Reliable, Safe, and Trustworthy Human-centered AI Systems. ACM Transactions on Interactive Intelligent Systems 10, 4 (Dec. 2020), 1--31. https://doi.org/10.1145/3419764Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Mona Sloane. 2021. The Algorithmic Auditing Trap. https://onezero.medium.com/the-algorithmic-auditing-trap-9a6f2d4d461dGoogle ScholarGoogle Scholar
  59. Mona Sloane, Emanuel Moss, and Rumman Chowdhury. 2022. A Silicon Valley Love Triangle: Hiring Algorithms, Pseudo-Science, and the Quest for Auditability. Patterns 3, 2 (2022). https://doi.org/10.1016/j.patter.2021.100425Google ScholarGoogle Scholar
  60. Luke Stark and Jevan Hutson. 2021. Physiognomic Artificial Intelligence. SSRN Electronic Journal (2021). https://doi.org/10.2139/ssrn.3927300Google ScholarGoogle Scholar
  61. Supreme Audit Institutions of Finland, Germany, the Netherlands, Norway and the UK. 2020. Auditing machine learning algorithms. Technical Report. 57 pages. https://www.auditingalgorithms.net/Google ScholarGoogle Scholar
  62. Tom Sühr, Sophie Hilgard, and Himabindu Lakkaraju. 2021. Does Fair Ranking Improve Minority Outcomes? Understanding the Interplay of Human and Algorithmic Biases in Online Hiring. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. ACM, Virtual Event USA, 989--999. https://doi.org/10.1145/3461702.3462602Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Peter Turney. 1995. Technical note: Bias and the quantification of stability. Machine Learning 20, 1--2 (1995), 23--33. https://doi.org/10.1007/BF00993473Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Tyler J VanderWeele and Maya B Mathur. 2019. SOME DESIRABLE PROPERTIES OF THE BONFERRONI CORRECTION: IS THE BONFERRONI CORRECTION REALLY SO BAD? American Journal of Epidemiology 188, 3 (March 2019), 617--618. https://doi.org/10.1093/aje/kwy250Google ScholarGoogle ScholarCross RefCross Ref
  65. Giridhari Venkatadri, Athanasios Andreou, Yabing Liu, Alan Mislove, Krishna P. Gummadi, Patrick Loiseau, and Oana Goga. 2018. Privacy Risks with Facebook's PII-Based Targeting: Auditing a Data Broker's Advertising Interface. In 2018 IEEE Symposium on Security and Privacy (SP). 89--107. https://doi.org/10.1109/SP.2018.00014 ISSN: 2375--1207.Google ScholarGoogle ScholarCross RefCross Ref
  66. Christo Wilson, Avijit Ghosh, Shan Jiang, Alan Mislove, Lewis Baker, Janelle Szary, Kelly Trindel, and Frida Polli. 2021. Building and Auditing Fair Algorithms: A Case Study in Candidate Screening. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. ACM, Virtual Event Canada, 666--677. https://doi.org/10.1145/3442188.3445928Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. Songkai Xue, Mikhail Yurochkin, and Yuekai Sun. 2020. Auditing ML Models for Individual Bias and Unfairness. In International Conference on Artificial Intelligence and Statistics. PMLR, 4552--4562. https://proceedings.mlr.press/v108/xue20a.html ISSN: 2640--3498.Google ScholarGoogle Scholar

Index Terms

  1. Resume Format, LinkedIn URLs and Other Unexpected Influences on AI Personality Prediction in Hiring: Results of an Audit

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          Get this Publication

          • Article Metrics

            • Downloads (Last 12 months)846
            • Downloads (Last 6 weeks)266

            Other Metrics

            View Author Metrics

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          View Table of Contents
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!