Teaching Ethics in Computing: A Systematic Literature Review of ACM Computer Science Education Publications

The computing education research community now has at least 40 years of published research on teaching ethics in higher education. To examine the state of our field, we present a systematic literature review of papers in the Association for Computing Machinery computing education venues that describe teaching ethics in higher-education computing courses. Our review spans all papers published to SIGCSE, ICER, ITiCSE, CompEd, Koli Calling, and TOCE venues through 2022, with 100 papers fulfilling our inclusion criteria. Overall, we found a wide variety in content, teaching strategies, challenges, and recommendations. The majority of the papers did not articulate a conception of “ethics,” and those that did used many different conceptions, from broadly applicable ethical theories to social impact to specific computing application areas (e.g., data privacy and hacking). Instructors used many different pedagogical strategies (e.g., discussions, lectures, assignments) and formats (e.g., stand-alone courses, incorporated within a technical course). Many papers identified measuring student knowledge as a particular challenge, and 59% of papers included mention of assessments or grading. Of the 69% of papers that evaluated their ethics instruction, most used student self-report surveys, course evaluations, and instructor reflections. While many papers included calls for more ethics content in computing, specific recommendations were rarely broadly applicable, preventing a synthesis of guidelines. To continue building on the last 40 years of research and move toward a set of best practices for teaching ethics in computing, our community should delineate our varied conceptions of ethics, examine which teaching strategies are best suited for each, and explore how to measure student learning.

curriculum and faced similar challenges in terms of strategies, assessment, and instructor buyin [ 89 , 90 , 96 , 133 ].To address some of these challenges, engineering researchers have called for the inclusion of both micro-ethical [ 18 , 42 ] and macro-ethical [ 31 ] engineering instruction [ 132 , 138 , 154 ], arguing that a comprehensive ethics education in engineering should incorporate both perspectives to develop a holistic ethical mindset [ 167 ].Building upon these insights from engineering ethics education, our research aims to provide a comprehensive analysis of the existing computing ethics education research for higher education published in ACM computer science education venues.By synthesizing the various approaches, we can contribute to the ongoing dialogue and guide future developments for computing ethics education.

METHODS
To examine how the computing education research community has discussed ethics, we conducted a systematic literature review.Following guidelines by Kitchenham et al. [ 116 ], we articulate the need for this review, formulation of research questions, data sources and search strategy, data extraction with a codebook, and data synthesis.

Need for Review: To Synthesize Knowledge about Ethics in Post-secondary
Computing Education As stated in Section 1 , this literature review responds to the growing interest in integrating ethics instruction into post-secondary computing education.While researchers have conducted related reviews on ethics education in more specific domains (e.g., machine learning [ 169 ]) or related fields (e.g., engineering [ 99 ]), no review has brought together decades of combined knowledge about computing ethics education.This review attempts to synthesize knowledge that can serve as a foundation for research and teaching in post-secondary computing ethics education.

Formulating Research Questions: Considering Plurality of Ethics, Practitioner, and
Researcher Inquiries To formulate our research questions defined in Section 1 , we first considered the general research inquiry of understanding what we know about how ethics is taught within post-secondary computing education.However, ethics has no singular definition.In discussions with 10 philosophers, political scientists, and ethicists, we concluded that there was a plurality in the conception of ethics, requiring us to first explore how papers conceived of ethics and how experts from beyond computing were involved (RQ1).Prior work, such as a survey of 138 higher education computing instructors [ 185 ], substantiated the importance of identifying effective pedagogical approaches for computing ethics education to practitioners.Therefore, we sought to understand how ethics was taught and assessed and to whom (RQ2).The effectiveness of pedagogy often relies on rigorous evaluation to understand impacts on students' learning experiences, so we then sought to explore how researchers evaluated ethics interventions (RQ3).Finally, we wanted to understand what recommendations and challenges prior work identified so future work can build off of these discoveries (RQ4).Combined, these research questions seek to synthesize research on post-secondary computing ethics education to recognize the breadth in conceptions of ethics, pedagogical approaches, evaluations, and implications to both researchers and practitioners.

Data Sources and Search Strategy: Post-secondary ACM CS Education Papers That
Include "Ethic*" To focus our review on computing education research, we limited our search to publications within all ACM Special Interest Group on Computer Science Education venues, the ACM Transactions on Computing Education Journal (TOCE) journal, and Koli Calling: Using the ACM Digital Library, we identified all of the papers published within these venues, from each venue's inception through August 2022 (note that CompEd was only held once-in 2019).Because practitioners often publish in a variety of formats, we did not restrict our review to full papers.Instead, "papers" can refer to full papers, short papers, posters, and panel discussions.

Excluded Venues.
We only considered venues focused on computing education research, excluding other discipline-specific education research venues (e.g., Journal of Engineering Education ), education research venues that did not focus on the teaching and learning of computing (e.g., Frontiers in Education Conference, Learning at Scale), and computing venues that focused on ethics and society but not education (e.g., CHI, FAccT, Ethicomp).
Within computing education research venues, we excluded those that were not focused on postsecondary computing education, not published to ACM, and were more regional in focus.We did not include venues that focus on primar y/elementar y, secondar y/high school, or any learning experiences oriented towards minors, such as the WiPSCE Conference on Primary and Secondary Computing Education Research.We made this scoping decision because differences in learning contexts (e.g., between primar y, secondar y, and post-secondary education) and developmental stages in life (e.g., children, teenagers, and adults) were too important to aggregate in a single review.
Additionally, we did not employ a snowballing approach [ 210 ] to add papers to our analysis, as this approach could potentially lead to papers published outside of the ACM.We focused on papers published to the ACM for consistency in search mechanism as well as restrictions to access.For example, we attempted to include the Taylor & Francis Journal of Computer Science Education but could not get access to its entire proceedings despite having access to libraries from multiple R1 institutions.This also meant that we excluded venues such as the IEEE Conference on Research in Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT).We also excluded venues with more regional emphasis, such as the Australasian Computing Education (ACE) Conference and Consortium for Computing Sciences in Colleges (CCSC) regional conferences.These exclusions were not a reflection of the quality and importance of publications to these venues but rather due to our capacity and resource constraints.Nevertheless, the set of venues we included are consistent with most prior computing education research literature reviews (Table 1 in Reference [ 92 ]).
With the sheer volume and diversity of works published on the topic of ethics in computing, setting boundaries was essential to ensure the feasibility of our study.Limiting our scope to ACM computing education venues was a strategic decision to balance comprehensiveness with manageability.Nevertheless, we understand that this decision omits some important work from our analysis (e.g., References [ 81 , 130 ]).We encourage researchers to view our study as a starting point for future work, potentially employing a snowballing approach to achieve a more exhaustive review.

Search Strategy.
After identifying papers from our venues of interest, we performed a full-text keyword search of "ethic*" (to include instances of all variations of this word, including "ethic," "ethics," and "ethical"), creating a set of all papers with those terms.We selected "ethics" as a keyword, because it is often used within the computing education research community (e.g., References [ 3 , 114 ]) and computing accreditation boards (e.g., References [ 43 , 104 ]) to refer to standards of conduct and moral decisions of computing professionals.While this provided us with a robust dataset due to the common use of this word in computing education research, we also acknowledge that this scoping decision may have excluded valuable papers related to our topic that discuss more specific concepts of ethics using different terms (e.g., responsible computing or socially just computing).Since these terms refer to aspects of ethical computing, we expected papers that use them to also include the term "ethic*."Therefore, these papers would still be eligible for inclusion in our dataset (e.g., "responsible computing" in Reference [ 41 ] and "justice-centered" computing in Reference [ 124 ]).To verify the impact of this decision, we ran a full-text keyword search of papers published within the ACM that included the term "responsible computing" but not "ethics" (or any variation).The two resulting papers [ 127 , 168 ] within our chosen venues did not address responsible computing in a manner consistent with the criteria for our analysis.Therefore, we deemed our search strategy to provide a dataset that sufficiently captured the broader discourse on ethics in computing education.However, we did not search for other terms, and checking this against a comprehensive list of all related terms used throughout the field would be challenging.Therefore, readers need to interpret our findings through the lens of this scoping decision, and we encourage future research to consider reviewing the literature on related terms individually to ensure their perspectives are justly highlighted.This is discussed further in Section 5.1 .Further, since our research questions are grounded in practical descriptions of ethics education, we limited our paper selection to those that provided specific examples of teaching ethics.To examine this community's empirical work on teaching ethics in higher-education computing courses, we additionally used the following exclusion criteria:  [ 1 , 30 , 55 , 93 , 101 , 106 , 113 , 115 , 117 , 120 , 123 , 124 , 149 , 150 , 156 , 194 , 197 , 204 ] Fig. 1.Included papers by publication year.
Three researchers independently applied the exclusion criteria to all papers in the initial set by reading the paragraphs in which the word "ethic" (or any variation) appeared.Based on the criteria, each researcher independently marked each paper for inclusion, exclusion, or discussion.The three researchers then resolved disagreements and uncertainties through discussion, until all papers were unanimously marked for inclusion or exclusion (see Table 1 ).Our final dataset included 100 papers published over 39 years (Table 2 ).Nearly half of our dataset was published in the past five years, and over 30% published in the past two years, as shown in Figure 1 .

Data Extraction: Iterative Development of a Qualitative Codebook
In our initial analysis, we approached our research questions as close ended, to be answered by categorizing the papers in our dataset into discrete groups.We used codebook analysis to maintain focus and to support consistency, starting with the iterative development of a qualitative codebook [ 116 , 155 ].To create the initial codebook, two researchers chose 10 papers from the dataset, selected for diversity in topic, publication year, and paper format.Both researchers independently read the 10 papers and performed open coding, resulting in an initial set of 21 codes.Those researchers, with Fiesler, refined the codebook to better match the research questions.They then handed off the codebook to Brown and Xie, who further refined it and then independently coded five randomly selected papers from the dataset, reading each fully and assigning at least one code per code group to each paper.They then reviewed their coding differences and refined the codebook in response.After repeating this process for five more randomly selected papers, they agreed the codebook was stable.The 12 code groups, with 34 individual codes, are in Tables 3 , 4 , 5 , and 6 , organized by their associated research question.
Brown and Xie used the final codebook to code all papers in the dataset, including re-coding the papers used to generate earlier versions of the codebook.They both read and coded each paper independently, working in batches of 10 randomly selected papers and then resolving disagreements through discussion.To apply the codes, they used ATLAS.ti1(software for qualitative data analysis).Each author uploaded a pdf of each paper into ATLAS.tiand then associated each applied code with a specific sentence or paragraph.The coders were in agreement if they applied the same codes to the paper, even if the codes were applied to different sentences.However, associating text segments with each code facilitated discussion when the coders disagreed.Brown was a co-author on one included paper in our dataset, but Xie was not.Other authors who were co-authors of included papers were not directly involved in the coding or analysis.Given that we report numeric results of our qualitative analysis, we used a consensus-based approach to ensure a consistent representation of the data [ 140 ].Ultimately, Brown and Xie reached 100% agreement for all papers.
Four code groups consisted of non-exclusive codes, such that multiple codes from the group could apply to one paper (Codes of Ethics, Student Level, Ethics Integration, and Pedagogical Strategy).For example, Turk and Wiley [ 196 ] included multiple pedagogical strategies: 'exposure' ("read a relevant article"), 'submitted work' ("write a short paper"), and 'participation' ("discuss the response").The other eight code groups were exclusive, noting presence or absence.For example, Kirdani-Ryan and Ko [ 115 ] was coded as 'assessment: included,' since it explicitly mentioned that the ethics activity was assessed: "We evaluated students along four axes [...] using a 3 point standards-based grading scale."A paper that did not mention whether the ethics activity was Teaching Ethics in Computing: A Systematic Literature Review of ACM CSEd 6:9 assessed was coded as 'assessment: no mention.'To make sure we did not overlook instructors' efforts, we erred on the side of generosity in coding; if a strategy or approach was only briefly or vaguely mentioned, we still applied that code.For example, a paper that mentioned a graded homework assignment consisting of both technical and ethical questions would be coded as 'assessment: included' even if the assessment strategy specific to the ethics component of the homework was not explicit.Similarly, a paper with a discussion of data from student course evaluations would be coded as 'evaluated: included' even if the data were not rigorously or formally evaluated.We acknowledge that many papers may not explicitly mention every code in our codebook, and we do not criticize such papers for their absence; a lack of a code does not necessarily indicate that the strategy was not employed or considered by the instructor or researcher.Additionally, we 6:10 N. Brown et al. recognize that researchers may have various reasons for not reporting on all items in our codebook, such as limitations in page or word counts or their focus and area of interest.

Data Synthesis: Code Counts and Narrative Description of Consistencies and Inconsistencies
After coding was complete, we sought to answer our research questions by synthesizing the data through quantitative summary and qualitative description [ 116 ].We showed frequency of codes to quantitatively summarize the distribution of codes within a group.These findings are shown in Figures 2 , 3 , 4 , and 5 .
We also qualitatively investigated coding results to discuss broader trends and identify notable inconsistencies.To do so, we exported text segments with associated codes from ATLAS.ti and uploaded them to Miro, 2 where Brown and Xie iteratively grouped the data to create broader themes within each code group (this post hoc analysis is presented in Section 4 ) [ 26 ].

Sensitivity Analysis: A Majority of Papers are from SIGCSE Technical Symposium.
Sensitivity analysis is used in literature reviews to consider how systematic differences of studies may impact synthesis of findings [ 116 ].Therefore, we note how most of the included papers (62%) are from the SIGCSE Technical Symposium (referred to as SIGCSE throughout the paper), as shown in Table 1 .The SIGCSE symposium is massive, having published twice as many papers as the second largest venue we included (ITiCSE) and having more papers published than all other included venues combined.As a result, SIGCSE papers have represented a large proportion of previous computing education literature reviews [ 92 ], with some oversampling from smaller or newer venues to compensate [ 152 ].
Teaching Ethics in Computing: A Systematic Literature Review of ACM CSEd 6:11 SIGCSE papers are shorter (typically limited to six pages or less) than those in other venues and more often authored or co-authored by instructors or teaching faculty with more practitioner experience.This page limit may result in authors not providing certain information or detail that they deem less relevant.However, practitioner perspectives from SIGCSE are crucial to this literature review, especially for RQ2 (Pedagogy).We compensated for this by coding for the existence of codes with our codebooks without judgment of rigor or quality.For example, we coded for inclusion of course assessment (Table 4 ) and research evaluation (Table 5 ) without judgment on the quality of either.By doing so, we tried to limit systemic bias that could come from a strict page limit resulting in authors omitting study details.While the extent to which SIGCSE authors still had to omit information and details is unknown, the practitioner expertise that comes with many SIGCSE papers was important to include in this literature review.

Positionality Statement
This research required analysis of written publications, which are situated within cultural and societal norms.Therefore, we present our standpoints as they reflect our assumptions and values, following critical approaches to quantifying qualitative data [ 87 ].We acknowledge the bias that comes from all authors and much of the computing education research (CER) community being situated in Western, educated, industrialized, rich, and democratic societies that do not reflect most of the global population [ 95 ].We are all members of the CER community, and we also see ourselves specifically as members of the ethics community within it.Brown and Wiese collaborate with CS faculty to design ethics-integrated instruction, Xie is an Embedded Ethics fellow, Sarder is a former computer science undergraduate student with an interest in computing ethics, and Fiesler advocates for ethics in computing (while incorporating it into her own courses).As computing ethics instructors and researchers, we set out to find best practices to guide our own work, knowing first-hand the difficulties of teaching and assessing ethics in higher education.We therefore write this article with a goal of constructively advancing knowledge of and for the CER community, with a shared commitment to broadening participation and cultural competency in computing.Further limitations are stated in Section 5.1 .

RESULTS
Overall, papers engaged with ethics in a variety of ways.Some papers focused primarily on computing ethics education, while others only briefly mentioned ethics as a portion of a larger class or project.In this section, we organize our results by research question, describing frequency of qualitative codes (Figures 2 , 3 , 4 , and 5 ) along with qualitative results generated through post hoc thematic analysis for codes related to each research question.All numbers are proportions of our total dataset of 100 papers.Where appropriate, we identify salient themes in italics and illustrate our results with specific cited ideas from papers in our dataset.However, in a few cases where our findings might be interpreted as critical of certain papers, we choose to withhold relevant citations to avoid singling out individual researchers (as has been done in prior literature reviews [ 152 ]).This approach reflects our commitment to practicing critical generosity [ 52 ]; it is not our intention to criticize individual researchers or papers for perceived shortcomings but rather to provide an overall description of this work.

RQ1 Results: What Conception of Ethics Was Taught? How Were People with
Ethics-related Expertise Involved?As ethics encompasses a wide range of concepts such as responsibility and impact, and has roots in various fields such as philosophy, business, and law, we wanted to explore what instructors were teaching as ethics and how disciplines beyond computing were included in this process.Figure 2 presents coding results for codes related to this research question.

Conceptions of Ethics: Relative to Philosophical Frameworks, Computing Applications, and
Social Impacts of Computing.Describing conceptions of ethics is crucial for ensuring a shared understanding between authors and readers, given the variation in framings of ethics.Forty-one percent of papers provided an explicit conception of ethics (41/100), as shown in conception of ethics in Figure 2 .We found that conceptions of ethics tended to mention multiple normative ethical theories or situate ethics relative to computing concepts, social impact, and professionalism.
Five papers mentioned multiple ethical theories from Western philosophy [ 46 , 47 , 93 , 169 , 202 ], with all of them mentioning consequentialism (consequence-based ethics, including utilitarinism) and deontology (duty-based ethics) and two of them also mentioning virtue-based ethics.Saltz et al. [ 169 ] incorporated consequentialism, deontology, and virtue theory into their work as they were previously described in an introductory textbook on ethics and science [ 27 ].Hedayati-Mehdiabadi [ 93 ] conducted a research study within a computing professional ethics course that taught three ethical theories (consequence-based, duty-based, and virtue-based ethics) to understand factors that influenced ethical decision-making processes of computing students.Three other papers [ 3 , 36 , 65 ] conceived of ethics as "deal[ing] with questions about people doing the right thing" [ 3 ].Decision-making processes were also provided as conceptions of ethics, with one paper proposing a framework consisting of "brainstorming, analysis, and decision-making" phases to help students "rationally analyze an ethically or legally ambiguous scenario" [ 36 ].
Other papers situated ethics relative to computing concepts and application areas (e.g., data privacy, robotics).Papers considered the ethics of data privacy and security from legal (intellectual property [ 63 ], copyright [ 211 ]), security [ 136 ], civil liberties [ 160 ], and psychological perspectives [ 182 ].For example, a theater project focused on data privacy from policy, psychological, and personal perspectives to support learning about computing ethics [ 182 ].A paper on teaching 6:13 robotics considered moral responsibility for decisions made by autonomous systems, including their use in the battlefield [ 11 ].Other computing topics included software development [ 173 ], the internet [ 12 , 125 ], and hacking [ 63 ].
Several papers conceived of ethics relative to the social impacts of computing , emphasizing the relationship between designing technology and its impact on society [ 114 , 117 ].In addition, Tseng et al. [ 194 ] considered ethical considerations in AI that focused on accessibility, an often overlooked aspect of identity in computing education [ 152 ].
Finally, some papers approached ethics as a situated phenomena that requires consideration of cultural and contextual factors.Quinn [ 158 ] proposed "a bottom up methodology for ethical reasoning [...] a useful alternative to top-down methods, such as Kantianism, utilitarianism, rightsbased theories, and virtue ethics" that they called "case-based analysis."They described this as "the process of determining whether an action is right or wrong by comparing the action with unambiguous paradigm cases or closely related cases that have already been analyzed" [ 158 ].To critique dominant norms in computing, Kirdani-Ryan and Ko [ 115 ] developed the "House of Computing" metaphor, which portrayed the computing discipline as an old house that has been remodeled and expanded over time.Other examples included a service-learning course that situated ethics relative to creativity and empathy [ 201 ] and a computing ethics course that considered "information cultures" of organizations and individuals [ 46 ].

Codes of Ethics: The ACM Code of Ethics Was the Most Common Ethical Code Referenced.
To further explore how papers framed ethics and whether this framing was influenced by an existing code of ethics, we coded papers for explicit mention of ethical codes ( codes of ethics in Figure 2 ).A majority of the papers (76) did not reference a code of ethics, but 26 papers cited a professional code of ethics (e.g., ACM Code of Ethics and Professional Conduct [ 69 ]) and four referred to a code outlined by their academic institution.
Our analysis found that the vast majority of the papers that referenced a code of ethics cited the ACM Code of Ethics and Professional Conduct [ 69 ].Other codes mentioned the IEEE Code of Ethics [ 114 , 196 ], the General Data Protection Regulation [ 110 ] or other country codes of ethics [ 126 ], and Asimov's three laws [ 30 ]. Petelka et al. [ 156 ] provided students with three codes of ethics: the Menlo Report [ 13 ], the Feminist Manifest-No [ 40 ], and Nussbaum's capabilities framework [ 148 ]. de Freitas and Weingart [ 54 ] described an approach to have students develop their own codes of ethics and then compare these to codes developed by academia [ 20 ], industry (e.g., Google's principles [ 79 ]), and government [ 19 ].

Expertise: Ethics Expertise Manifested through Guest Lectures, Co-instruction, and Curriculum Development.
Because ethics is a multidisciplinary field, often with roots in philosophy, business, law, and social sciences, we sought to understand how courses engaged with ethics expertise beyond the computing domain.We found that 23% (23/100) of papers mentioned involving experts from fields outside of computing.We found that these papers mentioned engagement with experts from the domains of philosophy (e.g., References [ 60 , 101 , 165 , 193 ]), non-profit/advocacy work [ 175 , 201 ], education [ 65 , 145 ], political science [ 160 ], history [ 65 ], social sciences [ 135 ], law [ 181 , 191 ], and public policy [ 55 ].
Of the papers that mentioned interdisciplinary engagement, most (14/23) did so by bringing in experts as invited/guest speakers to their courses.Guest lecturers provided real-world context to course topics, such as one who shared examples of AI systems in assistive technologies as part of a course that integrated accessibility into AI [ 194 ].Guest lectures appeared at the beginning of the course, such as a member of a non-profit providing an overview of a service learning course [ 201 ], and at the end, such as in the form of a panel of ethicists to address questions students generated throughout a computer security course [ 156 ].Dean and Nourbakhsh [ 55 ] exemplified an interdisciplinary approach by featuring five guest lectures from experts in economics, ethics, and philosophy.Many of these visits were preceded by related readings, leading to more engaging discussions and better-informed questions.
Co-teaching was another strategy for interdisciplinary engagement.In an intermediate-level CS2 level course, two embedded ethics modules were co-taught by a "philosopher with an expertise in ethics" [ 101 ].A computing ethics course was co-taught by "three faculty instructors, from philosophy, political science, and [computer science], each bring[ing] their respective lens to four related course modules" [ 160 ].A previous computing ethics course at the same institution was co-taught by a philosophy professor and a computer science professor [ 165 ].
Other papers engaged with people with expertise beyond computing in the development of instructional materials .One paper engaged computer scientists, a computer ethicist, and an assessment specialist to develop a computer ethics rubric [ 145 ].In another example, faculty from social and computer sciences collaborated to integrate a computing project into an undergraduate social sciences course [ 135 ], a rare paper about integrating computing ethics into a non-computing course.

RQ2 Results: How Was Ethics Taught and Assessed? To whom?
To understand how instructors were teaching ethics and how they were assessing student knowledge, we explored integration and pedagogical strategies, assessment techniques, and student demographics.Figure 3 displays the coding outcomes related to our second research question.

Ethics Integration: Ethics Was Taught in Both Stand-alone Ethics Courses and Technical
Computing Courses.Stand-alone computing-ethics courses (32) and ethics integration in multiple modules across a technical computing class (35) were the most commonly mentioned modalities for ethics education ( ethics integration in Figure 3 ).Twenty-six other papers integrated ethics in a single module in a technical computing course, while 17 mentioned teaching ethics in multiple courses.Only 5 papers did not specify a particular strategy used for ethics integration.Notably, none of the papers discussed teaching ethics in a stand-alone general-ethics course (i.e., not specific to computing), likely due to our search criteria, which limited publications to computing-specific venues.

Pedagogical
Strategy: Discussions, Lectures, Reading, and Writing Assignments Were Commonly Described Pedagogical Strategies for Teaching Ethics.We classified three categories of pedagogical strategies for teaching ethics: 'participation' (e.g., discussions and active learning), 'exposure' (e.g., lectures and reading), and 'submitted work' (e.g., assignments and projects), shown in Figure 3 , pedagogical strategy .'Submitted work' was the most common (76 mentions), followed by 'participation' (71) and 'exposure' (59).Some papers used multiple strategies (e.g., a lecture followed by an assignment was coded with both 'exposure' and 'submitted work').Only 10 papers did not mention a strategy.Table 7 shows detailed data for these codes, including references for papers that used various strategies.These data serve as a resource for instructors seeking different ways to teach ethics.
Our analysis found that commonly described pedagogical strategies included discussions , typically whole-class or small group, followed by readings , lectures or guest lectures, and writing assignments such as essays, journal entries, or blogs.Ethics was also incorporated into larger projects or assignments .Assignments related to ethics were either technical coding assignments (e.g., graph search in an ethical context [ 30 ], describing ethical issues in machine learning projects [ 169 ]) or ethics-focused assignments (e.g., applying ethical theories to case studies [ 93 ], defending personal opinions [ 39 ]).Less commonly mentioned pedagogical techniques included videos [ 9 , 78 , 142 , 196 ], role-playing [ 91 , 146 , 175 ], mock-trials or debates [ 12 , 36 , 102 , 165 ], games or competitions [ 5 , Fig. 3. Coding results for codes reflecting how ethics was taught ( ethics integration and pedagogical strategy ), how ethics was assessed ( assessment ), and who received the instruction ( student level and diversity ).

]
, engaging with local communities [ 25 , 157 ], question and answer sessions with a panel of experts [ 114 , 156 ], and student presentations [ 121 , 122 , 146 ].Table 7 shows that most papers mentioned a combination of pedagogical approaches to teach ethics.For example, OConnor [ 149 ] supplemented lectures on ethical theories with a practical lab.Several papers mentioned classroom discussions about assigned readings (e.g., References [ 47 , 48 , 68 , 114 , 212 ]).Some combined strategies to teach ethics while teaching technical material, such as Fiesler et al. [ 66 ], who used guest lectures, instructor lectures, discussions, and contextualized assignments in an introductory programming course.Similarly, Wang et al. [ 204 ] integrated ethics into a coding assignment followed by a discussion, and Reich et al. [ 160 ] assigned readings, coding assignments with ethical components, philosophical essays, and group work to emphasize multidisciplinary integration.

Assessment: Papers Described Various Grading Methods, Including Assessing Class Participation and Peer Evaluations.
Of the documents, 59 specified that the ethics component was assessed, while 41 did not ( assessment in Figure 3 ).Among those that provided details on assessment, many specified grade percentages (e.g., 25% for ethics essays [ 125 ] or one-third of final grade for a project [ 194 ]), while others mentioned the overall approach (e.g., specifications-based grading [ 124 ] and grading writing quality [ 51 , 78 , 126 ]).
One study assigned a low-stakes ethical reflection writing activity worth a minimal weight in the overall grade (1%) following mini-lectures and active-learning exercises [ 101 ].Some papers awarded points to students for classroom participation [ 28 , 47 , 157 , 170 ].Others described their assessment techniques in detail, such as valuing student input in grading [ 120 ], using studentgenerated exam questions [ 3 ], peer assessment [ 146 ], or self-evaluation with self-defined ethical guidelines [ 54 ].
Some papers offered tangible guidance for assessing ethical components.Canosa and Lucas [ 36 ] and Moskal et al. [ 145 ] shared rubrics for evaluating students' mastery of course outcomes and essays on computer ethics, respectively.Wahl [ 202 ] provided point values for grading paper components, and Kirdani-Ryan and Ko [ 115 ] outlined four axes used to assess a creative assignment.These details may help instructors overcome the challenge of assessing ethical topics [ 41 , 72 ].
Of the remaining articles, 12 papers did not specify the student level, while 9 papers discussed courses with both undergraduate and graduate students.Six papers included courses with only graduate students [ 41 , 110 , 123 , 169 , 175 , 196 ], although many of these papers included courses for other student levels in addition to graduate-only courses (e.g., Shapiro et al. [ 175 ] described their activity for teaching ethics to "a large first year seminar course, a senior-level computing and society class, and three different online graduate level courses").Mixed courses covered computing ethics [ 3 , 55 , 114 ], a capstone course [ 7 ], and other courses open to both undergraduate and graduate students [ 2 , 113 , 120 , 182 ].According to Townsend [ 193 ], opening the course to graduate students can benefit undergraduates by providing mentorship.Notably, only one article described education for adult students outside of academia [ 22 ].This paper described a series on AI intended for a public audience through massive, open online courses where "societal, ethical and philosophical implications are addressed throughout the courses" [ 22 ].

Diversity: Mentioned in Terms of Students, in Computing Communities, in Society, and as a
Goal.We examined both descriptive approaches of diverse communities and engagement strategies that address the broader impact of computing as it relates to diversity, equity, and inclusion.Fifty-nine percent of papers (59/100) mentioned diversity in terms of students in the class, computing more broadly, or social justice related goals ( diversity in Figure 4 ).
While most papers mentioned demographics as students for describing the course, some considered student diversity in the design of their ethics modules and interpretation of results.Two papers connected diversity of students with the design of ethics modules: King and Nolen [ 114 ] considered diversity of majors as contributing to "lively and challenging discussions/debates," and Hazzan and Har-Shai [ 91 ] considered the diversity of student teams as an asset that "enhances and promotes the use of soft skills."Two other papers considered lack of student diversity as limitations to their implementation of ethics modules [ 68 , 212 ].
A notable trend was the use of "aggregate terms " that refer to broad demographic groups, often ambiguously [ 152 ].Multiple papers referred to each of the following terms without further clarification: "(underrepresented) minority," "homogenous," "diversity," and "culture."For example, one paper described the "homogeneous economic, cultural, and ethnic backgrounds" of students in their course as a limitation to considerations of diversity.Other papers used aggregate terms to refer to student majors (e.g., "non-computer science majors" and "non-technical major").Prior work has shown that aggregate terms rely on dominant cultural assumptions (e.g., who is underrepresented) and can be harmful, especially without further clarification [ 152 , 208 ].
We also note that two papers discussed gender in such a way that could potentially perpetuate gender stereotypes [ 189 , 203 ].One paper, published over 20 years ago, suggested that "students with particular learning styles, especially women, are often turned off to computer science."This statement may align with feminist perspectives (e.g., Reference [ 75 ]) and educational psychology research [ 161 ] of the time.Another paper asked students to "analyze the design of the male and female home pages belonging to their class or some specific group of students."These framings of gender could imply broad commonalities based on biological gender without adequate consideration of social and contextual factors ( gender essentialism [ 184 ]).
In addition to discussing students' majors and demographics, several papers explored the concept of diversity in the broader context of computing and its influence on society .These papers focused on how computing education can contribute to the creation of a more diverse and inclusive technological landscape, highlighting the need to address issues such as bias and accessibility.Six papers mentioned gender bias in relation to computing [ 83 , 202 ], computing education [ 76 ], search engines [ 110 ], "electronic gameplaying" [ 193 ], and hiring decisions [ 39 ].In a computing course titled "Culture and Coding," an emphasis was placed on highlighting "women's contributions to the field" [ 125 ].Additionally, two papers focused on accessibility in the context of AI [ 194 ] and the equity of self-driving cars [ 175 ].Another paper engaged with race by considering how data on Asian college applicants showed an incomplete picture [ 204 ].Two other papers considered stakeholders who would be impacted by technology [ 101 , 169 ].Finally, two other papers emphasized having students consider "other cultures outside of their own culture" [ 105 ] and consider "cross-cultural perspectives, within collaborative, interdisciplinary teams" [ 64 ].
Some papers saw diversity as crucial for broadening participation in computing and promoting social justice .For instance, six papers justified new pedagogical approaches to "increase diversity in computing" through service learning [ 113 ], engagement with the local community [ 157 ], bringing learning opportunities to "the least developed countries" [ 22 ], humanitarian free and open source software [ 25 ], introducing robotics [ 11 ], and curriculum overhauls more broadly [ 197 ].Five other papers described diversity in the context of social and historical injustice [ 41 , 65 , 115 , 117 , 124 ].For example, one paper used counternarratives as a pedagogical strategy to "establish links between individual problems in society and their societal manifestations" [ 115 ].Multiple papers considered power relationships in society [ 65 , 117 ], such as how instructors considered "power imbalances" in society when reflecting on an ethical speculation exercise [ 117 ].Finally, one paper challenged the framing of "equity as inclusion," questioning the relationships between diversity in computing and "continued profitability of capitalist start-ups and technology giants" [ 124 ].

RQ3 Results: How Did Researchers Evaluate Ethics Interventions?
We were interested in understanding how researchers evaluated their ethics interventions.This is distinct from assessing student knowledge (as discussed in Section 4.2.3 ); this research question explores formal evaluations of research hypotheses or methods.Figure 4 displays our coding results for RQ3.

Pedagogical Context: Most Papers Investigated How Ethics Was Taught in the Classroom.
In our dataset, 95% (95/100) of papers focused on ethics interventions within a classroom setting ( participant context in Figure 4 ).However, five papers engaged participants outside of the classroom [ 62 , 83 , 93 , 112 , 182 ], which also provide valuable insights into ethics in computing education.These studies included a lab study [ 83 ] and a grounded theory analysis [ 93 ] investigating computing students' ethical decision-making.Skirpan et al. [ 182 ] presented an interdisciplinary theater show that highlighted potential ethical harms of technology, and Elsherbiny and Edwards [ 62 ] and Kiesler [ 112 ] worked directly with instructors to gather their insights.

Evaluation: Most Papers had Evaluations Involving Instructors Analyzing Student Feedback.
Sixty-nine percent of papers (69/100) mentioned an evaluation of their described approach, while the remaining 31 did not mention whether their ethics intervention included a formal research evaluation ( evaluation in Figure 4 ).Papers typically evaluated students enrolled in a course with some ethics component, with some evaluating multiple stakeholders (e.g., students, teaching assistants, and faculty [ 41 ]).
Papers evaluated student satisfaction, interest in ethics, and awareness of social issues.Papers measured student satisfaction both quantitatively (e.g., Likert-type scale [ 11 ]) and qualitatively (e.g., written evaluation of utility of each assignment [ 165 ]).Papers also evaluated change in topic interest including service learning [ 201 ] and social and ethical subjects more broadly [ 102 ].Other papers evaluated change in awareness of social issues, such as structural inequity [ 115 ] and accessibility [ 194 ].Evaluations of demonstrated ethics knowledge were rare , although one paper evaluated demonstrated practices and dispositions relating to information literacy and social issues [ 94 ].

Evaluation Instruments Asked Students to Answer
Likert-scale Questions.While rare, we found four studies that described instruments to measure student motivations, attitudes, selfefficacy, and perceptions, as well as course effectiveness.All four evaluation instruments involved students answering Likert-scale questions.Horton et al. [ 101 ] developed the Ethics Attitudes and Self-Efficacy instrument to measure student attitudes and self-efficacy regarding ethics and technology.Similarly, Mäses et al. [ 136 ] combined parts of existing instruments to develop a questionnaire that measured students' attitudes and ethical views before and after three homework assignments containing ethical aspects and one lecture on the ethics of permissions and cybersecurity.Kilkenny et al. [ 113 ] designed the Computer and Information Science Student Service Learning Survey to measure course-taking motivations, sense of civic duty, and perceptions of service learning.To measure the impact of a redesigned CS0 course that had "ethics and society" as a topic, Cramer and Toll [ 48 ] developed 17 Likert-scale questions to administer at the end of each semester during their study period.

RQ4 Results: What Were the Challenges and Recommendations for Incorporating
Ethics in Computing Courses?To support instructors and explore opportunities for future development, we coded for specific challenges and recommendations related to ethics integration.Figure 5 depicts these coding results.

Challenges: Papers Described Integration, Training, Instructional Design, and Assessment as
Challenges.Our analysis supports a common assertion [ 185 ]: integrating ethics within computing courses poses many challenges.Among the papers in our dataset, 57% (57/100) identified at least one challenge associated with teaching ethics ( challenges in Figure 5 ).
Many challenges highlighted the difficulty of integrating ethical and technical material in computing courses.Instructors may acknowledge the importance of ethics, but they face difficulty in fitting it into a technical curriculum [ 193 , 201 ].This could be due to the perceived lack of relevance of ethics to the technical content or it being too complex [ 157 ] or time-consuming to include in technical courses [ 55 , 123 , 160 ].Moreover, some instructors already have an overloaded curriculum, making it challenging to prioritize teaching ethics [ 9 , 54 , 114 , 151 ].Other papers mentioned that students must have a strong technical foundation to engage with ethical material, complicating ethical integration in complex technical courses [ 66 , 182 ].
Some papers noted that computing instructors often lack a formal background or training in ethics [ 60 , 93 , 117 , 145 , 170 , 196 ].This can make it challenging for instructors to feel qualified to teach ethics, even if they believe it is important.Finding experts in ethics to collaborate with may also be difficult.Students may also lack formal instruction on the topic and feel unqualified to engage with ethical material [ 41 , 110 ].Additionally, students may struggle to see the connection between ethics and the technical content they are learning in class [ 41 , 55 , 93 , 135 , 181 , 212 ], or may be more motivated to learn technical topics rather than ethics [ 9 , 35 ].
Several papers mentioned instructional design challenges, including a lack of existing resources suitable for in-class use [ 68 , 123 , 194 , 196 ] and the need for significant changes to adapt existing materials to another class [ 66 ].Additionally, creating material that incorporates ethics can be difficult due to challenges in framing problems [ 66 ] and estimating student workload [ 120 ].
Section 4.2.3 highlights the challenge of assessing ethical knowledge in a class primarily focused on technical topics.Many papers mentioned the difficulties involved with evaluating ethics [ 21 , 30 , 32 , 41 , 170 , 204 ], sometimes due to the open-ended nature of ethics [ 146 , 156 ].Remaining challenges related to practicality and complexity : Teaching some ethical topics can be emotionally taxing [ 115 ] and a classroom setting may not be able to adequately prepare students for the complexities involved in real-world ethical dilemmas [ 83 , 149 ].

Recommendations:
For Pedagogy, Content, and Instructional Design.While 62% of papers (62/100) included recommendations for integrating ethics in computing courses ( recommendations in Figure 5 ), specific recommendations varied.Skirpan et al. [ 181 ] suggested integrating ethics into existing technical courses, Petelka et al. [ 156 ] called for in-depth instruction on ethics, and Cohen et al. [ 41 ] recommended using both integrated and stand-alone course approaches.Some emphasized involving ethics experts in the instruction process [ 41 , 60 , 114 , 193 ] or involving the community [ 201 ].
Several papers provided pedagogical recommendations , such as increasing student engagement [ 54 , 122 ], using a variety of strategies [ 5 , 157 , 194 , 202 ], or allowing students to choose their own project topic [ 125 ].Some papers also offered recommendations for managing the complexity of the field [ 1 , 83 ] by starting small [ 196 ] and explicitly linking technical topics to ethics [ 194 ].
The most common content recommendation was to increase the relevance of ethical topics [ 201 ] by including real-world examples [ 35 , 93 ], news articles [ 47 , 196 ], or personal stories [ 115 ].Hedayati-Mehdiabadi [ 93 ] also stressed the importance of developing students' critical thinking skills in addition to teaching ethical frameworks, since "using an ethical framework or ethical standard by itself will not guarantee an ethical decision."The paper also recommended discussing with students the "ways one might use these frameworks as an excuse to justify one's stance" and suggested instructors "help students feel more comfortable with ambiguities of ethical problems and encourage them to consider different aspects of a situation" [ 93 ].
To overcome instructional design challenges, some recommended careful framing of questions or projects [ 25 , 66 , 103 , 160 ].Others suggested following a formal design process [ 16 ] or engaging with an instructional design expert [ 41 ].Several papers also recommended considering how a project or assignment would be evaluated to help with instructional design or to better understand students' understanding of the topics [ 21 , 121 , 145 , 170 ].Turk [ 195 ] stressed documentation as a resource for other instructors, such as how Saltz et al. [ 169 ] included guiding questions instructional designers can apply to their assignments to integrate ethics within technical courses.

DISCUSSION
This literature review synthesizes a strong and growing interest in teaching ethics in highereducation computing courses.The increasing rate of computing ethics publications provided evidence for this growing interest: We analyzed papers that span nearly 40 years (1983-2022), yet almost half of these papers (48%) were published in only the past 5 years (Figure 1 ).We qualitatively coded for elements that pertained to our research questions.While many papers in our dataset did not explicitly include all of the elements we coded for (e.g., only 23% of papers included mention of engaging with an ethics expert, Figure 2 ), we do not propose that all future papers must do so.However, we do recommend that all papers explicitly describe their conceptions of what "ethics" means for a given instructional context (see Section 5.2.1 ).Further, because sample materials are particularly useful for other instructors [ 185 ], we encourage researchers to provide details of their instructional activities and assessments and to explain their design decisions in light of their learning goals.Our dataset shows that many computing education instructors are incorporating ethics into computing courses [ 82 ].We propose that our next steps as a community are to explore best practices for different goals and contexts and to focus our research on known challenges (such as assessment).We organize our discussion by considering different ways to interpret our findings, starting with limitations and then considering each research question (mirroring our formatting from Section 4 ).As this study examined numerous dimensions of computing ethics education, we underscore our most significant takeaways and promising avenues for future research by emphasizing them with italicized text.

Limitations
The most important limitations in our review stem from our inclusion criteria, as described in Section 3.3 .We excluded papers outside of specific ACM venues and papers that did not include the word "ethics" or its variations (elaborated on in Section 3.3.1 ).While these venues are likely places for computing researchers to publish and to look for prior work, and "ethics" is a common term within this community, valuable perspectives from other venues are not included in our work, and we did not comprehensively search the literature on all related terms (e.g., cultural competency [ 205 ] or socially just computing).While this review describes how publications in ACM education venues discuss ethics, it does not provide a comprehensive synthesis of how ethics in computing is conceptualized and studied.Future work can consider a more expansive review of other fields (e.g., Ethics in Engineering Education), inclusion of additional search terms (e.g., "moral," "justice," and "professionalism") and/or analysis of course syllabi.
Additionally, we excluded argument papers advocating for the inclusion of ethics education or papers with criticisms regarding its current state (e.g., References [ 118 , 159 , 183 ]).We did this to ground our study in practice, but this presents an opportunity for future work to explore how researchers are discussing ethics education from beyond teaching experiences.Finally, there are inherent biases in what kind of work gets published, and insights from blog posts, repositories of instructional materials, professional development guides, or live conversations from panels and workshops are not represented.
Despite these limitations, this literature review still provides a synthesis of 100 computing education papers across nearly 40 years.This synthesis provides a foundation and makes space for practitioners and researchers to build upon.However, when interpreting our findings, it is important to consider the limited scope of our analysis and the diversity of objectives and boundaries within each study included in our review.We encourage readers to garner a more comprehensive perspective on the subject by searching the literature outside of the ACM and on related terms.By doing so, readers can better understand the nuances of our current findings in relationship with broader discussions and conceptualizations of ethics.We, therefore, view this analysis as both an informative piece and a catalyst for future inquiry.

RQ1: Conceptualize Ethics When Possible; Align Ethics Expertise with Objectives
and Student Interests Given the variety of conceptions and application areas of ethics, we were interested in exploring what students learned about ethics, and whether courses engaged with experts from beyond computing.We found varying conceptions of ethics and few cross-disciplinary collaborations.Together, our findings present room for clarification and collaboration to enhance students' ethics education.

Conceptualizing Ethics Provides More Clarity, and Could Extend beyond Dominant Western
Philosophical Traditions.Our analysis identified that most papers did not describe a conceptualization of ethics (Section 4.1.1 ).Those that did provide conceptions referred to theories from Western philosophy (e.g., consequentialism, deontology), computing application areas (e.g., data privacy), social concepts (e.g., social justice), or a decision-making framework.We interpret these conceptions of ethics to span disciplines including philosophy, computing, and social sciences.Within even philosophy, ethics is a plurality.
We interpret these conceptions of ethics relative to prior work in engineering education research that categorizes ethics as "micro-ethics" or "macro-ethics" to differentiate individual and collective social responsibilities [ 97 ].This framing can also connect ethics, often considered from individualistic lenses in dominant Western philosophy, and social justice, which is often considered at collective and systemic levels within social sciences.Unifying frameworks such as this one can support bridging across disciplines, as "ethics" is a term often used within philosophy domains and "social justice" is one often used within social science domains.Indeed, both micro-ethics and macro-ethics holds significance and contributes to a more comprehensive ethical education; nevertheless, it is impractical to anticipate that a single instructor or course can sufficiently cover such a spectrum.
Given the cross-disciplinary conceptions of ethics we have identified, we find it constructive for authors to provide a conceptualization of what "ethics" entails when possible.This recommendation echoes that of Hess and Fore [ 99 ], whose systematic literature review on engineering ethics education described the importance of conceptualizing ethics to understand whether an intervention met the intended learning goals.They similarly emphasized, "educators must clearly define what they mean by ethics" [ 99 ].Considering the broad conceptions of ethics, a clear conceptualization can help researchers better contextualize new findings in relation to previous studies.Further, this clarity can assist computing instructors in integrating the facet of ethics that aligns with their teaching objectives.
If an instructor cannot provide a clear conceptualization of ethics, then we recommend still articulating clear learning objectives and connections to computing concepts and social impact.Computing instructors rarely have formal philosophy or humanities training, thus limiting conceptions of ethics they are familiar with.Cross-disciplinary collaborations can help bridge such knowledge gaps, but these are not always possible.To support a plurality of conceptions of ethics, we suggest still articulating clear ethics learning objectives (e.g., through Backwards Design [ 206 ]), relevant computing concepts (e.g., AI, software engineering), and social impact of focus (e.g., data privacy, surveillance).That is, we support design explorations of computing ethics pedagogy that extend beyond dominant conceptions of ethics and consider a broader plurality.
Some papers used alternative terms to frame their conceptions of ethics.For example, Lin [ 124 ] used the term "justice-centered approaches," which "frame [computer science] learning as a means for advancing peace, antiracism, and social justice rather than war, empire, and corporations."Similarly, Cohen et al. [ 41 ] clarified their use of the term "socially responsible computing," which "include[s] a combination of understanding power and technology, making ethical design decisions, building accessible systems, and testing systems for (un)desirable impacts on various stakeholders."We encourage future work to synthesize the various terms used across the field in papers outside of this analysis, demonstrating how precise terminology might be used to foster a common understanding of conceptualizations of ethics.We might consider adopting shared terminology when discussing specific ethical dimensions, ensuring clearer communication and collective understanding among researchers and practitioners.
Additionally, future work should consider integrating conceptions of ethics from beyond dominant Western philosophical traditions.For example, we did not find explicit implementation of non-dominant ethical theories, such as feminist/care ethics and non-Western ethics.Care ethics is a contextual, relational approach to morality [ 10 , 74 , 147 ].While Hedayati-Mehdiabadi [ 93 ] called for introducing care ethics in computing education, the mention of care ethics was scarce in this literature review.Similarly, Hess and Fore [ 99 ] did not find any instances of care ethics in engineering courses.However, prior work has applied it to culturally responsive teaching [ 177 ] and to technology design [ 200 ].Looking beyond Western philosophies may also provide promise.For example, Mhlambi [ 141 ] has used the relational Sub-Saharan African philosophy to reconcile the ethical implications of AI.Given the plurality in conceptions of ethics (micro-ethics with individual responsibilities, macro-ethics with collective and social responsibilities, non-Western philosophies), future work can align appropriate conceptions of ethics with different objectives (organizational and course goals), populations (computing majors, courses for non-computing majors), and institutional contexts [ 199 ].

Interdisciplinary Expertise
Can Support Student Learning.We found that less than a quarter of papers engaged with ethics experts from beyond computing as guest speakers, co-instructors, and curriculum designers (Section 4.1.3).Interdisciplinary collaboration is valuable in computing ethics education, because ethical issues in computing often involve complex social, political, philosophical, and technical considerations.Drawing upon a wide range of disciplinary knowledge and perspectives can enhance the quality and relevance of computing ethics education [ 159 ].Moreover, interdisciplinary collaboration can help foster a more holistic and integrated approach to computing ethics education, in which ethical considerations are integrated throughout the curriculum and not seen as an afterthought or add-on [ 77 ].Future work can explore further mutually beneficial partnerships with ethics experts that align with course objectives, pedagogical strategies, and student interests.

RQ2: Ethics Can Be Taught in Stand-alone Courses or Integrated with Open
Questions about Assessment and DEI.Our analysis revealed that there are a variety of approaches being used to teach and assess ethics.The range of strategies is likely a reflection of the range of learning goals [ 131 ], which are tied to the varied conceptions of ethics in our dataset.Future work could include design studies to examine how to create and improve instruction for specific learning goals, and comparison studies to uncover causal links between instructional design choices and learning.Furthermore, rigorous measures of students' learning (e.g., comparing pre-and post-tests) can support both of these research approaches.

Examine Pedagogical Strategies and Formats in Relation to Learning Goals.
A common debate in our field is whether to teach ethics as a separate course or integrate it into existing computing courses.Our analysis supports the idea that the field is split on this issue (Section 4.2.1 ), and we suggest future research examine the affordances and tradeoffs of each approach.However, even when ethics is included in technical courses, there are varying levels of integration, ranging from teaching ethical frameworks separate from technical implementation to combined technical and ethical knowledge in problem-solving.However, our results from RQ1 suggest there is a more productive framing for this discussion than "level of integration" by itself: Rather, our field should explore what kind of integration is best suited for different conceptions of ethics, learning goals, and contexts.For instance, teaching formal ethical frameworks or theories may be best suited to a stand-alone course that has more time to cover these topics in depth, while professional ethics and issue-spotting within specific technical implementations may be best suited to technical courses where students are supported in learning the relevant technical details.Likewise, design studies and experiments can identify which aspects of the various pedagogical approaches in our dataset (e.g., discussions, essays, assignments, and lectures) can support specific learning goals.Within the computing education community, "ethics" encompasses many types of knowledge and skills, from understanding the societal-level impact of technology to writing individual lines of code that shape the behavior of a program.Therefore, in addition to investigating how to teach a particular kind of ethics, we encourage further research to explore how students make connections between these different levels of ethical issue-spotting or decision-making.In all cases, we recommend that researchers contextualize their findings in terms of their ethics learning goals, acknowledging the varied conceptions of ethics in our field.

Delineating Challenges in Assessment May Support the Development of More Robust
Assessment Strategies.While some researchers have provided useful resources for evaluating ethics assignments (e.g., References [ 145 , 171 ]), many papers in our dataset identified assessment as a challenge.Grading in computing courses is typically based on objective criteria (e.g., Reference [ 170 ]) and often performed automatically, particularly for large courses (e.g., Reference [ 204 ]).Uncertainty over how to grade subjective work, or the time required to do so, can lead instructors to grade based on participation (e.g., Reference [ 41 ]).As a step toward developing assessments that are rigorous and practical, we encourage further reflection on the nature of these challenges.For example, if students write programs that enact their ethical decisions, then the functionality of those programs will be varied, posing an obstacle to autograding with a pre-set suite of test cases.Design studies could examine the feasibility of different approaches for supported automated grading while allowing for different functionality choices (perhaps by 6:25 co-creating test cases with students).As engineering ethics educators have called for, researchers could also explore the development of standardized instruments and scoring rubrics [ 133 ] to help overcome assessment challenges.
From an instructional design perspective, some assessment challenges may be avoided by careful consideration of learning goals.For example, students can examine different programs, determine how their functionalities differ, and match those differences to their ethical implications through multiple-choice formats.Other types of assignments are inherently subjective and cannot be transformed into auto-gradable formats.Here, the computing education community can particularly benefit from the expertise of instructors in the humanities, and reflective collaborations can generate insights for other computing instructors who are not able to engage in those collaborations themselves.
At a high level, we hypothesize some challenges with assessment may be a reflection of the difficulty in creating learning goals for ethics in computing.Clear learning goals define what students should be able to do with their knowledge and skills, and serve as guideposts for assessment design [ 207 ].Instructors may want to include ethics in their courses but not know how to translate that desire into action-based learning goals.We present this hypothesis cautiously, because the development of learning goals was not part of our analysis.Still, we encourage further work on the relationship between defining learning goals and developing aligned assessments for ethics in computing.

Opportunities for Connections between Ethics and Diversity, Equity, and Inclusion (DEI).
We found that 59% of papers mentioned diversity as it relates to student demographics, in computing communities, in society, and as a goal (Section 4.2.5 ).We interpreted these findings as reflecting a breadth of engagement between ethics and DEI approaches but not without challenges.On one hand, including ethics and social relevance in computing can improve participation and retention of students from historically underrepresented groups [ 15 ] and ethics topics often relate to developing cultural competency [ 66 , 205 ].However, computing instructors and students may take an exclusionary perspective to ethics pedagogy, deeming it too "soft" or "social" for computing [ 159 ].

Connecting ethics to discussions of social justice and cultural competency can support DEI efforts, but this requires teachers and students challenging exclusionar y disciplinar y and organizational norms.
As stated by Raji et al. [ 159 ]: A way to address this structural divide operative in the ways ethics is taught in computer science, we need to develop pedagogies able to forge a new ground for the relation between epistemology and ethics, truth and the good, individual and collective responsibility.
While it is important to engage in discourse on diversity, risks of harm from stereotyping exist.For example, we identified a few papers that framed women as an outsider group in computing (Section 4.2.5 ).Taking an intersectionality lens [ 49 ], gender is one of many factors that define people's lived experiences.Furthermore, a deficit framing of non-dominant genders can evoke stereotype threat.A more constructive approach to engaging with gender and other factors of identity is to develop cultural competence [ 205 ] and instead problematize exclusionary norms within computing [ 152 , 159 ].

RQ3: Evaluating Ethics Education with Rigor, Contexts beyond Classrooms, and
Instruments Most papers in our dataset engaged with participants in class (Section 4.3.1 ).While we believe this setting is appropriate for evaluating ethics pedagogy in computing courses, future work could explore alternative approaches (e.g., external evaluators and out-of-class evaluations) that avoid the limitations of classroom studies, such as the potential for coercion of students and bias in feedback.More out-of-class evaluations, like rigorous lab studies, could provide alternative and valuable perspectives on computing ethics education, since they allow for precise control over specific variables.Furthermore, our dataset was lacking in studies on ethics education in informal learning environments (e.g., internships, research experiences, online communities, extracurricular activities).Further research is necessary to gain a deeper understanding of the potential impact of informal learning environments on computing ethics education.For example, it is unclear how computing ethics education aligns with ethical dilemmas faced in professional contexts (e.g., in internships).By identifying additional strategies for fostering ethical development in students, we may be able to better equip future computing professionals to navigate complex ethical issues.
Additionally, most papers included some form of evaluation of student satisfaction, interest, and awareness of ethics and social issues (Section 4.3.2 ).Data collection methods included student-created content, course evaluations, and self-reported surveys.We interpret these findings cautiously, as student content and self-reported data may not reflect the process of learning about ethics.Future work could involve collecting data that reflect the learning process as well as developing validity evidence for measurement instruments such as the Ethics Attitudes and Self-Efficacy instrument [ 101 ].Rigorous data collection and analysis can deepen our understanding of how students learn ethics in computing courses.
Ethics is often open ended, so future work could also explore more open-ended evaluation techniques to understand contextual alignment and impact .Many evaluations we analyzed focused on students' responses to Likert-scale questions or course-related content.More open-ended evaluation could explore alignment between provided ethics training, students' values, and demands for future careers.For example, while rare, Hess and Fore [ 99 ] found instances of qualitative analyses of in-and out-of-class obser vations, inter views or focus groups, and homework analyses as methods to evaluate engineering ethics interventions.Future work could also consider computing ethics education in broader higher education contexts, understanding how institutional norms and resources impact computing instructors' and students' interest in ethics.Doing so can help the computing education research community understand how computing departments make cultural and programmatic changes to teach ethics.

RQ4: Challenges with Positioning Ethics Relative to Computing,
Recommendations to Align Ethics with Instructional Context Our analysis (Sections 4.4.1 and 4.4.2 ) revealed challenges and recommendations for incorporating ethics in computing courses.Many challenges identified align with the results of a prior survey of higher education computing instructors [ 185 ], including prioritizing technical content, lacking incentives or knowledge to integrate ethics, and delegating ethics to other departments.The challenges we identified may indicate assumptions that instructors and researchers have about what constitutes ethics in computing.Most computing faculty are not experts in philosophical frameworks, law, or broader social contexts that technology operates within.These conceptions of ethics are also likely to be far removed from the technical learning goals of a computing course, requiring instructors to remove technical content to make space for ethics.However, computing faculty are experts in professional computing ethics [ 53 , 109 ], whether they realize it or not.This type of ethics is deeply tied to technical expertise: determining if the assumptions of an algorithm are met, communicating the limits of a validation approach for a certain context, and testing for potential bugs and vulnerabilities are just a few examples.Because professional computing ethics focuses on the duties of a computing professional given their technical expertise, it is a natural complement to technical instruction.Highlighting the human judgement required to select an appropriate technical approach and identify that approach's limitations may even help instructors identify new technical learning goals [ 30 ].
Given the range of ethics conceptions, it is possible that some of the identified challenges resulted from a mismatch between the ethical learning goals and the format or context of the instruction.Future work should propose recommendations to help instructors determine what kinds of ethics and what learning goals are most appropriate for their courses and their own expertise, and also what challenges to anticipate even when those elements are aligned.The computing education community recognizes many forms of ethics and the many layers of ethical decisions involved in technology creation, development, and use.As researchers, respecting this breadth in conception of ethics requires us to contextualize challenges and recommendations within specific conceptions of ethics, and to consider what kinds of guidelines generalize across learning goals and instructional formats.

CONCLUSION
To gain insight into the state of ethics education in computing, we conducted a systematic literature review of 100 papers published in ACM computing education venues.Our analysis revealed a wide range of approaches to integrating ethics into computing courses.This study increases awareness of the current state of ethics education in computing and can guide teaching practices as ethics is increasingly integrated across the curriculum.It also highlights the need for more focused research directions in computing ethics education, including clarifying expectations for computing students' understanding of ethics and identifying effective teaching and assessment methods.Going forward, the computing ethics education community can continue exploring and implementing ethics pedagogy, building upon the last 40 years of research.

6 : 5 -
ACM Special Interest Group on Computer Science Education Technical Symposium (SIGCSE) -ACM International Computing Education Research Conference (ICER) -ACM Innovation and Technology in Computer Science Education Conference (ITiSCE) -ACM Global Computing Education Conference (CompEd) -ACM Koli Calling International Conference on Computing Education Research (Koli Calling) -ACM TOCE

Fig. 2 .
Fig.2.Coding results for codes reflecting how ethics was conceptualized ( conception of ethics and codes of ethics ) and how people with expertise beyond computing were involved ( expertise ).

Fig. 4 .
Fig. 4. Coding results for what contexts publications investigated ( participant context ), and whether publications included evaluations of ethics interventions ( evaluation ).

Fig. 5 .
Fig. 5. Coding results for codes reflecting the challenges ( challenges ) and recommendations ( recommendations ) for incorporating ethics in computing courses.

Table 1 .
Dataset Collection Results, Including the Number of Total Articles (and Their Associated Publication Venues and Years) Returned by Our Search and the Number of Articles We Included in Our Analysis after Applying Our Inclusion/Exclusion Criteria Note: at the time of this analysis, CompEd has only been held once (in 2019).

Table 2 .
Our Full Dataset of 100 Included Papers, by Decade until 2019 and by Year Afterwards, with Publication Venues for Each Group

Table 7 .
Papers That Used Combinations of Various Pedagogical Strategies to Teach Ethics