How to Train a (Bad) Algorithmic Caseworker: A Quantitative Deconstruction of Risk Assessments in Child Welfare

ABSTRACT

Child welfare (CW) agencies use risk assessment tools as a means to achieve evidence-based, consistent, and unbiased decision-making. These risk assessments act as data collection mechanisms and have been further developed into algorithmic systems in recent years. Moreover, several of these algorithms have reinforced biased theoretical constructs and predictors because of the easy availability of structured assessment data. In this study, we critically examine the Washington Assessment of Risk Model (WARM), a prominent risk assessment tool that has been adopted by over 30 states in the United States and has been repurposed into more complex algorithmic systems. We compared WARM against the narrative coding of casenotes written by caseworkers who used WARM. We found significant discrepancies between the casenotes and WARM data where WARM scores did not not mirror caseworkers’ notes about family risk. We provide the SIGCHI community with some initial findings from the quantitative de-construction of a child-welfare risk assessment algorithm.

References

1. Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. 2018. A reductions approach to fair classification. In International Conference on Machine Learning. PMLR, 60–69.
2. Asbjørn Ammitzbøll Flügge, Thomas Hildebrandt, and Naja Holten Møller. 2021. Street-level algorithms and AI in bureaucratic decision-making: A caseworker perspective. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1(2021), 1–23.
3. Anna Brown, Alexandra Chouldechova, Emily Putnam-Hornstein, Andrew Tobin, and Rhema Vaithianathan. 2019. Toward Algorithmic Accountability in Public Services: A Qualitative Study of Affected Community Perspectives on Algorithmic Decision-making in Child Welfare Services. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 41.
4. Hao-Fei Cheng, Logan Stapleton, Ruiqi Wang, Paige Bullock, Alexandra Chouldechova, Zhiwei Steven Steven Wu, and Haiyi Zhu. 2021. Soliciting Stakeholders’ Fairness Notions in Child Maltreatment Predictive Systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–17.
5. Hao-Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O’Connell, Terrance Gray, F Maxwell Harper, and Haiyi Zhu. 2019. Explaining decision-making algorithms through UI: Strategies to help non-expert stakeholders. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–12.
6. Kevin Clancy, Joseph Chudzik, Aleksandra J Snowden, and Shion Guha. 2022. Reconciling data-driven crime analysis with human-centered algorithms. Cities 124(2022), 103604.
7. Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020. A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12.
8. Alan J Dettlaff, Stephanie L Rivaux, Donald J Baumann, John D Fluke, Joan R Rycraft, and Joyce James. 2011. Disentangling substantiation: The influence of race, income, and risk on the substantiation decision in child welfare. Children and Youth Services Review 33, 9 (2011), 1630–1637.
9. Howard J Doueck, Diana J English, Diane DePanfilis, and Gerald T Moote. 1993. Decision-making in child protective services: a comparison of selected risk-assessment systems.Child Welfare 72, 5 (1993), 441–452.
10. Ellen A Drost. 2011. Validity and reliability in social science research. Education Research and perspectives 38, 1 (2011), 105–123.
11. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference. 214–226.
12. Frank Edwards and Christopher Wildeman. 2018. Characteristics of the front-line child welfare workforce. Children and Youth Services Review 89 (2018), 13–26.
13. Diana J English, Sherry C Brummel, J Christopher Graham, and Laura K Coghlan. 2002. Factors that influence the decision not to substantiate a CPS referral phase II: Mail and telephone surveys of child protective services social workers. Management Services Division, Office of Children’s Administration Research (2002).
14. Guy Enosh and Tali Bayer-Topilsky. 2015. Reasoning and bias: Heuristics in safety assessment and placement decisions for children at risk. The British Journal of Social Work 45, 6 (2015), 1771–1787.
15. David Finkelhor. 2018. Screening for Adverse Childhood Experiences (ACEs): Cautions and Suggestions. Child abuse & neglect 85 (2018), 174–179.
16. Eileen Gambrill and Aron Shlonsky. 2001. The Need for Comprehensive Risk Management Systems in Child Welfare. Children and Youth Services Review 23, 1 (2001), 79–107.
17. James Gladstone, Gary Dumbrill, Bruce Leslie, Andrew Koster, Michelle Young, and Afisi Ismaila. 2014. Understanding worker–parent engagement in child protection casework. Children and Youth Services Review 44 (2014), 56–64.
18. James P Gleeson. 1987. Implementing Structured Decision-Making Procedures at Child Welfare Intake.Child Welfare 66, 2 (1987).
19. Martin T Hall, Jeanelle Sears, and Matthew T Walton. 2020. Motivational interviewing in child welfare services: a systematic review. Child maltreatment 25, 3 (2020), 263–276.
20. Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. arXiv preprint arXiv:1610.02413(2016).
21. Naja Holten Møller, Irina Shklovski, and Thomas T Hildebrandt. 2020. Shifting Concepts of Value: Designing Algorithmic Decision-support Systems for Public Services. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society. 1–12.
22. Jakko Kemper and Daan Kolkman. 2019. Transparent to whom? No algorithmic accountability without a critical audience. Information, Communication & Society 22, 14 (2019), 2081–2096.
23. René F Kizilcec. 2016. How much information? Effects of transparency on trust in an algorithmic interface. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2390–2395.
24. Todd Kulesza, Simone Stumpf, Margaret Burnett, Sherry Yang, Irwin Kwan, and Weng-Keen Wong. 2013. Too much, too little, or just right? Ways explanations impact end users’ mental models. In 2013 IEEE Symposium on visual languages and human centric computing. IEEE, 3–10.
25. Camilla Lauritzen, Svein A Vis, and Sturla Fossum. 2018. Factors that determine decision making in child protection investigations: A review of the literature. Child & Family Social Work 23, 4 (2018), 743–756.
26. Stephen Magura, Beth Silverman Moses, and Mary Ann Jones. 1987. Assessing risk and measuring change in families: The Family Risk Scales. Vol. 51. Child Welfare League of Amer.
27. Juho Pääkkönen, Matti Nelimarkka, Jesse Haapoja, and Airi Lampinen. 2020. Bureaucracy as a Lens for Analyzing and Designing Algorithmic Systems. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.
28. Diane Boyd Rauber. 2009. From the courthouse to the statehouse: Parents as partners in child welfare. Child Law Practice 28, 10 (2009), 149–156.
29. Mary A Rawlings and Emily R Blackmer. 2019. Assessing engagement skills in public child welfare using OSCE: A pilot study. Journal of Public Child Welfare 13, 4 (2019), 441–461.
30. Joanna Redden, Lina Dencik, and Harry Warne. 2020. Datafied child welfare services: unpacking politics, economics and power. Policy Studies (2020), 1–20.
31. Cheryl Regehr, Marion Bogo, Aron Shlonsky, and Vicki LeBlanc. 2010. Confidence and professional judgment in assessing children’s risk of abuse. Research on social work practice 20, 6 (2010), 621–628.
32. Samantha Robertson, Tonya Nguyen, and Niloufar Salehi. 2021. Modeling Assumptions Clash with the Real World: Transparency, Equity, and Community Challenges for Student Assignment Algorithms. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
33. Devansh Saxena, Karla Badillo-Urquiola, Pamela Wisniewski, and Shion Guha. 2020. Child Welfare System: Interaction of Policy, Practice and Algorithms. In Companion of the 2020 ACM International Conference on Supporting Group Work. 119–122.
34. Devansh Saxena, Karla Badillo-Urquiola, Pamela J Wisniewski, and Shion Guha. 2020. A Human-Centered Review of Algorithms used within the US Child Welfare System. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–15.
35. Devansh Saxena, Karla Badillo-Urquiola, Pamela J Wisniewski, and Shion Guha. 2021. A framework of high-stakes algorithmic decision-making for the public sector developed through a case study of child-welfare. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2(2021), 1–41.
36. Devansh Saxena, Erina Seh-young Moon, Dahlia Shehata, and Shion Guha. 2022. Unpacking Invisible Work Practices, Constraints, and Latent Power Relationships in Child Welfare through Casenote Analysis. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems.
37. Craig S Schwalbe. 2008. Strengthening the integration of actuarial risk assessment with clinical judgment in an evidence based practice framework. Children and Youth Services Review 30, 12 (2008), 1458–1464.
38. Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the conference on fairness, accountability, and transparency. 59–68.
39. Aron Shlonsky and Dennis Wagner. 2005. The next step: Integrating actuarial risk assessment and clinical judgment into an evidence-based practice framework in CPS case management. Children and Youth Services Review 27, 4 (2005), 409–427.
40. Brenda D Smith. 2008. Child welfare service plan compliance: Perceptions of parents and caseworkers. Families in Society 89, 4 (2008), 521–532.
41. Paul A Sunseri. 2020. Hidden figures: Is improving family functioning a key to better treatment outcomes for seriously mentally ill children?Residential Treatment for Children & Youth 37, 1 (2020), 46–64.
42. Karmen Toros, Diana Maria DiNitto, and Anne Tiko. 2018. Family engagement in the child welfare system: A scoping review. Children and Youth Services Review 88 (2018), 598–607.
43. Maranke Wieringa. 2020. What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 1–18.

Index Terms

• Applied computing

  • Law, social and behavioral sciences

  • Life and medical sciences

• Human-centered computing

• Social and professional topics

  • Professional topics