A Quantitative Methodological Review of Research on Broadening Participation in Computing, 2005-2022

Given the persistent underrepresentation of women, Black, Latinx, and other marginalized groups in computing fields, scholars from a variety of disciplines have generated a breadth of empirical findings on factors that are salient in broadening participation in undergraduate computing (BPC). While such literature yields insights into the computing experiences and outcomes of underrepresented groups, there is limited understanding of how BPC-related research is conducted. Our paper focuses on the methods used in BPC-focused journal articles and conference proceedings published between 2005-2022, centering exclusively on 129 studies that use quantitative methodologies. The study follows Moher et al.'s (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for reporting methodological review processes and findings. Results of the paper highlight important directions for future BPC research; in particular, studies should (a) include more longitudinal and multi-institutional approaches; (b) incorporate more sophisticated and multivariate techniques; and (c) yield insights on smaller racial/ethnic groups, community college students, and students with disabilities, while also accounting for students' intersecting identities.


INTRODUCTION AND BACKGROUND
Despite the national computing enrollment boom that took hold in the early 2010s [7,22], there remains a persistent underrepresentation of women, Black, Latinx, and other marginalized groups across all academic and professional levels of computing and technology [26].Such equity gaps have driven agencies such as the National Science Foundation (NSF) to commit significant funding to support initiatives focused on broadening participation in computing (BPC) and to enable research on BPC efforts.As such, scholars from various disciplines have generated a breadth of empirical findings on factors that are salient in BPC [20].Indeed, generating empirical evidence that practitioners and policymakers can use for program and policy implementation is critical given the need to understand what works (and what does not) in diversifying undergraduate computing.
The extant research has provided key insights on BPC efforts in undergraduate education [20], including research that focuses specifically on factors contributing to the gender gap in computing [e.g., 9] or on how equity gaps are heightened for Women of Color [e.g., 16,21].Such scholarship provides critical insights, yet there is a limited understanding of how the recent wave of BPC-related research has been conducted.
One way to advance scholarship in this area is to conduct a systematic methodological review of current undergraduate BPC research.The most recent methodological reviews on computing research were published in the mid-to-late 2000s [27][28][29][30], prior to the computing enrollment boom and proliferation of BPC initiatives over the past decade.Given current imperatives to develop and fund evidence-based programs and practices, assessing the methodological approaches and rigor [2,14,35] of more recent undergraduate BPC research is critical.
To enable us to make clearer comparisons across the methodologies used in BPC-related research, the present study focuses exclusively on research using quantitative methodologies 1and is thus deemed a quantitative methodological review (QMR).Unlike a meta-analysis focusing on synthesizing research outcomes [12,19], QMRs rely on a content analysis approach to contextualize research methods and designs [27].This enables researchers to investigate how studies are conducted (e.g., types of data sources, breadth, and strength of analyses) within a body of work that shares common goals (e.g., BPC, computing interventions).
While QMR studies are common in the health fields [e.g., 12,19] and have been published in computing, the field of computing lacks an updated methodological review of literature from the last 15+ years that spans both journal articles and conference proceedings-an outlet valued by the field [17].Among QMR research in computing, Valentine [33] focused on SIGCSE technical conference proceedings from 1974-2004 to understand first-year CS courses.Further, Randolph alone [27,28] and with colleagues [29,30] produced a series of methodological reviews on articles published in major computer science education forums from 2000-2005.Across these computing methodological reviews, we can glean that (a) published research in computing grew rapidly after the computing enrollment boom in the 1980s; (b) computing research had shifted away from mere program descriptions and towards the use of more robust methodologies; and (c) there were no substantive differences in the methodological sophistication of published conference proceedings versus journal articles.

ANALYTIC FRAMEWORK
Our study is guided by Moher and colleagues' Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [19], a commonly used analytic tool for reporting methodological review processes and findings.The statement suggests that authors conducting such reviews provide their readers with clear information on the approaches taken, along with a flow diagram outlining the identification, screening, eligibility, and inclusion criteria of studies pertaining to the review at hand.By using the PRISMA systematic review research tool, we ensure transparency in how we selected and analyzed relevant scholarship for this QMR.

The Current Study
The extant methodological reviews in computing education research were published nearly two decades ago, and the recent shifts in computing and associated BPC-related efforts require a current understanding of this research.Hence, this study is guided by the PRISMA reporting format in our examination of the methods used in BPC-focused peer-reviewed quantitative journal articles and conference proceedings published between 2005-2022.We examine the following questions centered around quantitative BPC scholarship: (1) What institutions are examined?(2) Which students are included?(3) How are data analyzed?(4) How are gender and race/ethnicity considered?

METHODS
This QMR draws from the Undergraduate BPC Literature Database (UBLD), an NSF-funded research tool developed by Momentum at UCLA and shared on the Computing Research Association's (CRA) BPCnet.org[20].The UBLD compiles peerreviewed scholarly papers published since 2005 on undergraduate BPC research conducted in the United States.Guided by PRISMA (e.g., information sources, eligibility criteria) [19], the following section describes the selection processes for the UBLD and for the final papers used in this QMR (see Figure 1).postsecondary institutions (our analysis includes database updates through March 2022).These studies must have examined the impact of BPC interventions, explored factors related to inequitable outcomes in computing, and/or advanced theoretical perspectives and frameworks for at least one systemically minoritized social group identified via the NSF's BPC initiative [5] (i.e., "women, persons with disabilities, Blacks and African Americans, Hispanics and Latinos, American Indians, Alaska Natives, Native Hawaiians, and other Pacific Islanders").The UBLD also considered studies focused on students from other systemically minoritized social identities2 in computing (e.g., trans*/genderqueer students, LGBQ+ students, first-generation college students, and students from economically disadvantaged backgrounds) [20].
3.1.2Information Sources & Key Search Terms.With the search parameters described in the prior section and in consultation with an institutional librarian, the Momentum team searched 12 databases across relevant disciplines (e.g., Education Source, Academic Search Complete, ACM Digital Library), using a combination of Boolean operators and search terms (e.g., computer science education, universit*, wom*, black*, equit*, recruit*) that would systematically and optimally capture scholarship on undergraduate BPC.

Selection for UBLD.
Once all articles were downloaded from the databases (n = 4,477), 891 duplicates were removed, resulting in 3,586 articles for title and abstract screening to exclude papers that were beyond the scope of computing, focused on K-12 computer science education, or conducted about institutions outside the U.S.This screening resulted in a total of 232 articles that were included in the UBLD.The UBLD technical report (which describes the full list of databases, key terms/phrases, and screening procedures) and the UBLD can be found at https://bpcnet.org/bpc-literature-database/.

3.2.1
Study Selection for QMR.For this paper, our team screened the abstracts and full texts of the 232 articles within the UBLD to identify research studies and experience reports 3 with quantitative components focused on undergraduate students.Articles that were purely qualitative, purely program descriptions, or were purely conceptual/theoretical were excluded.Further, given the purpose of conducting this systematic methodological review, quantitative papers that did not have clearly defined methodologies were excluded.This screening resulted in 129 studies 4 for content analysis.Of these studies, 104 were purely quantitative, 18 were mostly quantitative with some qualitative elements, and seven were evenly quantitative and qualitative.A total of 63 studies were published in conference proceedings (most often the SIGCSE Technical Symposium), and 66 were published in journals (most often in ACM Transactions on Computing Education and Computer Science Education.)

Protocol, Data Collection Process, & Data Items.
To conduct a content analysis of each article for the QMR, the team developed a review instrument comprised of several dozen items that assessed and coded the content of each study centering on how it was conducted, specifically around BPC (e.g., sample sizes, statistical analyses, and how gender and race/ethnicity were considered).The review instrument was developed over six cycles, where each cycle included the application of the instrument to five randomly selected papers from the final list of QMR articles by four research team members.Coding memos were recorded by each team member and brought to the group for further development of the review instrument.
The four reviewers engaged in content analysis, with two assigned to each study.After every 20 studies, each pair of reviewers met to discuss and finalize the coding for each of their assigned studies.Subsequently, all four reviewers met to discuss their respective articles and reconcile any coding discrepancies.Once the coding of all 129 studies was completed, data for each paper was compiled and prepared for analysis in SPSS.Descriptive statistics, mainly frequency distributions and crosstabulations, were used to analyze the data.

Limitations
This systematic QMR has several limitations.First, while the UBLD included literature on students from a wide range of systemically minoritized social identities within computing, this QMR sought to identify studies that focused on BPC groups recognized by NSF CISE [25] (based on gender, race/ethnicity, and disability status).(Of note, however, few papers focused on students with disabilities.)Further, our study is restricted to articles that appear in the UBLD database and does not include BPC studies that were not selected for inclusion due to the nature of the screening process.In addition, this QMR focused on papers with quantitative components and thus does not reflect the approaches used in equally important qualitative BPC studies.

RESULTS
The results presented here reflect the approaches used in 129 studies published between 2005 and 2022 that used quantitative methods to examine BPC research in U.S. undergraduate education.

What Institutions Are Examined?
Table 1 shows that half of BPC studies included students from public universities (50.4%), with relatively fewer studies examining students from private universities (16.3%), private fouryear colleges (5.4%), public four-year colleges (4.7%), and community colleges (3.1%).It is important to point out that a significant proportion of studies (16.3%) did not provide much specificity on the institution type (e.g., indicates "university" but not public or private), and a full 38.8% of studies did not specify institution type.The 65 studies that focused only on students at a single institution offered more clarity on institutional type, with can offer similarly-advanced quantitative methods as research studies.As such, both formats are considered for this QMR. 4 Some papers included multiple studies drawing from different data sources.For these papers, each study was considered and reviewed separately; thus, there were a total of 129 studies across 121 papers.more than two-thirds (69.2%) conducted on students attending public universities.
It is also important to know whether BPC studies addressed whether students attended minority-serving institutions (MSIs).Among all studies, 13.2% mentioned that the student sample attended historically Black colleges and universities (HBCUs), 8.5% described a focus on students attending Hispanic-serving institutions (HSIs), 3.9% centered on students at tribal colleges and universities (TCUs), and none mentioned whether students attended institutions serving the AANAPI population (Asian American Native American Pacific Islander-serving institutions).It is likely that more studies did include institutions that have federal designation as MSIs, however we were interested in the number of studies in which MSI status was specifically named.

Which Students are Included?
All 129 studies collected data directly from or about undergraduate students.In most cases, data were collected directly from students (91.5%), with relatively fewer papers incorporating student data from the institution (19.4%) (e.g., course grades, degree attainment records).Only 2.3% of studies incorporated data from federal sources into their analysis.Unsurprisingly, many papers restricted their sample to students taking computing courses, whether introductory computing courses (38.8%) or other undergraduate computing courses (17.1%).Most studies did not include restrictions based on students' major, though 18.6% were restricted to computer science majors, 9.3% were restricted to other specific computing majors (e.g., informatics), and 10.1% limited their samples to students majoring in computing broadly defined.Relatively few studies restricted samples based on students' class standing, though a focus on first-year students was most prevalent (9.3%).
Across the 129 studies, student samples ranged from small (0-50, 4.7%) to very large (3000+, 14.0%), though as shown in Figure 2, there is a bimodal tendency in sample size.About half of all studies included 300 or fewer students, with the most common range of 100-200, and about half of the studies involved quite large samples, most commonly in the 3000+ range.
Our team also examined whether information on response rates was reported.Among the 107 papers for which response rates could have reported (i.e., excluding studies that relied exclusively on federal data, registrar's data, etc.), only 43.0% provided clear information on response rates, even if that meant to say that response rates could not be calculated.

How are Data Analyzed?
As shown in Table 2, BPC quantitative research tends toward descriptive analysis, typically involving comparing percentages or means across groups.The most commonly used methods are the Chi-square (37.2%), t-test (28.7%), and ANOVA (28.7%).Simple correlational analysis is also among the more prevalent approaches (21.7%).Fewer BPC studies rely on regression-based approaches, such as linear or logistic regression, structural equation modeling, or hierarchical linear modeling.Fewer than half of studies considered time as an element of analysis (44.2%).Among the 57 studies that did, most tracked individual students longitudinally (75.4%), whereas additional studies addressed time in other ways (e.g., comparing students from different cohorts).

How are Gender and Race/Ethnicity
Considered?
Given our focus on BPC research, it is unsurprising that most studies accounted for students' gender and/or race/ethnicity.However, studies were more likely to report on gender than race/ethnicity, both in terms of describing the sample and in the actual analysis (see Table 3).Further, relatively few studies considered intersections of gender and race/ethnicity.For papers that analyzed data in terms of gender, race/ethnicity, or intersections of the two, Table 4 reports on how such characteristics were considered.In the 111 studies that analyzed in terms of gender, most commonly gender was used to report group results (e.g., reporting percentages or means for women or men) (91.9%), with few studies using gender as an independent variable in a multivariate analysis (26.1%), and even fewer studies restricting only to students of a particular gender (4.5%).A similar trend is reported for the 71 studies that considered race/ethnicity in their analysis and the 20 studies that considered intersections of gender and race/ethnicity.An important takeaway from Table 4 is that most studies do not consider gender, race/ethnicity, or their intersections in a way that distinguishes how such characteristics operate relative to other relevant variables.
We also consider how gender and race/ethnicity are defined across BPC studies.Of the 111 studies that considered gender, 96.4% defined gender in terms of a binary (e.g., man/woman, male/female), with only 3.6% incorporating non-binary gender categories.Table 5 reports on how race/ethnicity was considered in analysis.The first column reports on the 37 studies that analyzed data in terms of distinct racial/ethnic categories, showing that the vast majority of such studies reported separately on the following groups: White (97.3%),Black (91.9%),Asian (89.2%), and Latinx (86.5%).Far fewer studies reported results specifically for students who are American Indian/Alaskan Native (40.5%),Native Hawaiian/Pacific Islander (16.2%), or Middle Eastern (8.1%).Of studies that considered race/ethnicity in their analyses, three considered these categories separately and as URG/URM categories.

DISCUSSION AND RECOMMENDATIONS
In a time of heightened investment in BPC initiatives and related scholarship, this QMR sheds light on the current state of quantitative BPC research by systematically reporting how scholars have conducted their studies.In this discussion, we highlight selected articles that may serve as methodological exemplars, noting that the full UBLD contains many other rigorous studies.

Going Beyond Descriptive Analyses
In general, BPC research tends to rely on descriptive analytical approaches, with fewer studies utilizing multivariate or regression-based techniques.While descriptive statistics offer foundational tools for examining data, multivariate techniques allow for an even deeper investigation of BPC factors that may be most salient for computing outcomes.For example, Blaney and Wofford's [4] use of logistic regression illuminated distinct factors that are salient for predicting Ph.D. aspirations.Alternatively, Alvarado et al. [1] used propensity score matching as a way of minimizing bias in comparing undergraduate CS research program participants with similar non-participants.Further, while experimental design is less common in education research (given the typically non-random assignment of students to courses or programs), an experimental design was used by Bowman et al. [6], who applied cluster randomization as a way of testing the effects of pair programming.Cheryan et al. [8] also used an experimental design to control for stereotypical and nonstereotypical CS environments, focusing on how these environments affect women's interest in CS.To be clear, we do not suggest that more sophisticated methods replace descriptive analysis, but instead underscore that using (a combination of) different methods may offer more robust findings that inform tailored implications for research and practice.

Expanding Focus to Diverse Institutions and Time Elements
Most studies focused on public university students, with relatively few studies yielding insights into other institutions that enroll diverse populations (e.g., community colleges, MSIs) [18,24].This is especially important considering that community colleges enrolled nearly 41% of all undergraduate students in 2020-2021 [10], and MSIs are a critical resource for developing diverse STEM talent [23].As such, we encourage researchers to examine a broader range of institutions in BPC studies.
Given that BPC efforts occur over time, it is also important for research to consider collecting and analyzing data across more than one time point.Since 2005, only a minority of BPC studies tracked students longitudinally.While cross-sectional studies are typically less costly and offer important insights into BPC efforts, longitudinal studies offer tools for assessing change and controlling for confounding effects.For example, Settle et al. [32] examined mean scores on a range of measures before and after a computing course.Similarly, Blaney [3] accounted for the biasing effects of a pre-test in regression analysis.In addition, Denner et al.'s [11] application of multilevel regression on a longitudinal and multi-institution sample of community college students offered insights into changes in motivation while also accounting for class-and school-level differences.We encourage the use of longitudinal data, enabling methodological techniques that assess change over time and reduce biases that may mask the true effects of factors salient in computing.

Providing Important Contexts
We also found that fewer than half of studies that could have reported information on response rates did so and only about half include a limitations section or paragraph.Response rates offer important insight into data collection efforts that precede data analysis and yield information readers can use to assess a study's generalizability [13].Additionally, stating limitations allows readers to better consider the applicability of findings to their institutions and programs, and offers researchers with tailored future directions.

Gender and Race in Future BPC Research
Examining how BPC categories were addressed in computing research since 2005 yielded particularly important findings.While all studies focused on BPC in some way, studies were more likely to report on gender than race/ethnicity, both in terms of describing the sample and in the actual analysis, and relatively few studies considered intersections of gender and race/ethnicity.Further, while research should continue to use gender in describing the sample and in analysis (given persistent gender disparities in computing) [27], gender is too often considered in a binary, which eclipses what can be learned about gender nonbinary populations.
Future research should also be more intentional in incorporating race/ethnicity into their studies, especially for groups supported by BPC initiatives.Further, smaller racial/ethnic groups (e.g., American Indian/Alaskan Native, Native Hawaiian/Pacific Islander) are rarely considered as unique categories for analysis; such groups are typically included in definitions that combine underrepresented groups into a larger umbrella category (e.g., URM).While there may be conceptual and/or statistical rationales for aggregating or disaggregating racial/ethnic groups, we encourage future BPC research to problematize these decisions.At a minimum, researchers should provide conceptual framing and theoretically-driven rationales for methodological decisions regarding how gender and race/ethnicity are treated.For example, considering racial/ethnic groups separately in analysis provides more distinct insights on each group relative to aggregating findings by URM/URG; it is even more preferable to incorporate students' intersecting identities [15,16].

Conclusion
This QMR illuminates the current landscape of methodologies used in quantitative BPC research.While we acknowledge the significant value and methodological range of existing BPC scholarship, we encourage researchers to address BPC groups that are persistently minoritized in the field and have also been understudied over the past two decades, including populations defined beyond gender and race/ethnicity (e.g., students with disabilities, transfer students, students from economically disadvantaged backgrounds).We also recommend that researchers employ more robust research designs that enable greater certainty in applying research to practice.For example, while descriptive statistics generate important foundations for understanding BPC groups, layering in multivariate and/or predictive analysis may provide more precise guidance for program planning and implementation.Finally, we underscore the importance of providing context in BPC research (e.g., reporting responses rates, describing limitations).Doing so stands to improve the BPC community's ability to assess the value and applicability of research findings to the specific groups they serve.

Figure 1 .
Figure 1.Flowchart for selection of final studies (guided by Moher et al., 2009) 3.1 Undergraduate BPC Literature Database 3.1.1Search Parameters and Eligibility Criteria.Search parameters were created to ensure that the scholarship included in this database was published in 2005 or later with a BPC focus on U.S. postsecondary institutions (our analysis includes database updates through March 2022).These studies must have examined the impact of BPC interventions, explored factors related to inequitable outcomes in computing, and/or advanced theoretical perspectives and frameworks for at least one systemically minoritized social group identified via the NSF's BPC initiative [5] (i.e., "women, persons with disabilities, Blacks and African

Table 1 .
Institutional Characteristics of Undergraduate BPC Literature, 2005-2022 1Percentages add up to more than 100 since studies may have included more than one institution type.

Table 3 .
Proportion of Studies That Included Demographic Information on Their Study Sample or Considered These Demographics in Their Analyses (n = 129)

Table 4 .
How Selected BPC Catergories 1 Were Used in Analyses of Undergraduate BPC Literature, 2005-2022

Table 5 .
Racial/Ethnic Breakdown in Studies That Considered These Characteristics in Their Analyses1