What Factors Influence Persistence in Project-based Programming Courses at Community Colleges?

The rapid adoption of emergent technologies is creating significant shortfall in the CS/IT workforce. With not enough students in the educational pipeline to meet the forthcoming demand over the next decade, community colleges are making the effort to train confident, knowledgeable, and self-driven workers in this field. Project-based learning (PBL) has been shown to be effective for these ends, but it poses distinct challenges in resource-limited community college contexts since it may require more time, preparation, and motivation than other teaching modalities, from both the student and the instructor. We studied fifteen sections of an introductory project-based Python course taught at six community colleges, investigating several features of PBL theorized to be particular barriers to student persistence, particularly among women and other identities traditionally underrepresented in technical fields. We describe successes and challenges faced by students in these areas and suggest implications for project-based learning curriculum and platform design.


INTRODUCTION
With the rapid adoption of emerging technologies, there is an associated growth in demand for talent in emerging technology areas.
With not enough students in the educational pipeline to meet the forthcoming demand over the next decade, community colleges offer a great opportunity to address the labor or talent gap.These institutions are making the effort to train confident, knowledgeable, and self-driven workers in this field.We set out to partner with community colleges to co-design, evaluate, and improve the efficacy of project-based learning (PBL) to meet these challenges.
PBL has been shown to be effective for these ends.PBL refers to an inquiry-based instructional method in which individuals or teams acquire knowledge and develop skills by creating end products to solve authentic real-world problems [7,15,17,25,26].
Evidence is emerging of PBL's positive impact on underrepresented groups.Exploring Computer Science (ECS) was purposefully designed to address the persistent underrepresentation of women and individuals from diverse backgrounds in the field of computing, where students engage in project-based activities to develop computational skills that connect computing with their daily lives, which bolstered student interest and engagement in computer science [13,14].Another example is a study introducing e-textiles in K-12 CS education, which heightened students' confidence and interest in CS across different demographics [22].
However, PBL poses distinct challenges in resource-limited community college contexts since it may require more time, preparation, and motivation than other teaching modalities from both the student and the instructor.In contrast with lecture-based instruction, PBL is an instructional approach that requires more guidance [17].Allowing learners to navigate their own path without timely expert guidance and instructor feedback can lead to persistent misconceptions, repeated unsuccessful attempts, becoming "stuck, " or directing inquiry efforts towards unproductive directions [8].Properly-designed scaffolding becomes essential to furnish learners with manageable and captivating challenges within PBL [17].Poorly-designed PBL risks discouraging students, particularly those starting out with low self-efficacy.
Providing well-tailored scaffolding requires an instructor to have domain expertise, an understanding of students' learning paths, and the skills to implement and deliver such scaffolding.Institutions with limited resources may find it impractical to independently design, develop, and teach PBL.The expenses associated with technology and maintenance compound these difficulties.
To identify good strategies for adapting PBL to introductory computer science in community colleges, we partnered with 6 colleges and offered 15 instances of an introductory computing course with 231 students using a variety of modalities (in person, remote sync, remote async).Colleges selected a subset of the training modules and taught the courses their way.
We provided a curriculum and platform to deliver an introductory Python course through 7 community colleges.We used surveys, demographic data, student reflections, and course performance data to characterize student experiences relating to barriers identified in previous literature, especially associated with gender-linked differences.
We investigated several features of PBL theorized to be particular barriers to student persistence, particularly among women and minorities traditionally underrepresented in technical fields.We describe challenges faced by students in these areas, benefits evident to students who overcame them, and suggest improvements to instruction and curriculum design for PBL to foster greater persistence.
Our goal in this research is to evaluate a technique for making PBL feasible in the community college environment, and determine how and to what extent PBL's theoretical benefits played out in practice.
Our specific research questions are: RQ1 What factors and interventions helped students persist through the end of the course?RQ2 What effects did completing the projects have on student self-confidence and STEM identity?RQ3 What were the differences in the impact of the course on students from underrepresented groups?
We conclude by proposing implications for the design and implementation of project-based CS courses at community colleges.

RELATED WORK
The expectancy-value theory (EVT) of motivation proposes that people's choices and persistence for achievement-based tasks are guided by their beliefs regarding how well they will perform on an activity (their expectancy for success) and the value they assign to the activity [44].The expectancy-value theory becomes relevant when considering Vincent Tinto's theoretical framework on student persistence in higher education [39,40].Tinto theorizes that a student's persistence is influenced by their academic and social integration within the educational environment.If students perceive a sense of belonging and find value in their educational journey, they are more likely to persist.This perspective gains additional support from Bers and Smith, which emphasizes that the interplay of academic and social integration, educational goals, and employment status is a significant determining factor in the persistence of community college students [5].
Moving towards the concept of self-efficacy as developed by Bandura [4], it is important to acknowledge its role in student persistence.In a study investigating college student persistence and performance predictors, Lotkowski et al. [29] found that academic self-efficacy is a significant non-academic factor predicting student retention rates.Further research on dropout rates within online courses augmented this finding, noting an association between students' persistence in online learning environments and their self-efficacy levels [18].
The guiding principle in strengthening persistence in projectbased programming courses is creating a supportive learning environment, as warranted by intertwining constructs of EVT theory, Tinto's framework on academic and social integration, and Bandura's self-efficacy theory.A fundamental part of this supportive learning environment is social integration, as it plays a pivotal role in establishing a sense of belonging, hence fostering a meaningful learning experience and improved student persistence.With online learning gaining momentum, research has pivoted to examining how online social interactions impact learning outcomes.Empirical evidence supports the positive influence of online engagement through collaborative projects [3,27], discussions [3], and peer feedback [28,33,41] on learning and overall engagement.This cardinal notion of collaborative learning has been conceptualized in the Community of Inquiry (CoI) model proposed by Garrison et al.According to them, learning is a shared process conducted within a group, driven by the combination of three essential components: cognitive presence, social presence, and teaching presence [12].They further argue that computer-mediated communication can effectively nurture an educational CoI during online and blended educational experiences.
DuBow et al. described factors that fuel persistence for women in computing domains [11].They emphasize the cultivation of a "computing identity" as a driving force behind their continued engagement in the field.The findings drawn from interviews with women considering computing careers assert that such identity formation is influenced by rewarding computing experiences that boost self-efficacy, and the support and encouragement from friends, family members, and instructors toward choosing computing as a career path.Their observations further strengthen the notion of the supportive environmental and social integration defined previously, providing evidence of its impact in real-world contexts, especially among historically underrepresented demographics.

PLATFORM AND TEACHING METHODOLOGY
We created a Python course offering online conceptual content with summative quizzes, challenging projects with inline formative quizzes, an extensive autograding infrastructure allowing incremental, constructive feedback on partially-complete projects, and a forum where students are required to reflect on and discuss their approaches after each project.The course was offered through a platform originally developed at our university for delivery of project-based learning courses.We adapted the platform to link to most institutions' Learning Management Systems (LMS), and we have formalized training and support processes for instructors.
In the following subsections, we describe our effort to apply evidence-based instructional methods to enable the implementation of PBL in an online environment with emphasis on enhancing students' self-efficacy and persistence.

Authentic Real-World Scenarios
To deliver an immersive and meaningful learning experience, we designed the hands-on projects to mirror authentic scenarios that encapsulate real-world processes, constraints, tasks, and industrystandard quality measures [32].By simulating actual professional practice, we aimed to cultivate learners' understanding of industry standards and norms (including practices such as unit tests, code refactoring, and troubleshooting) in addition to the acquisition of skills and knowledge [10].

Challenging Yet Manageable Problems
Effective PBL offers complex problem-solving scenarios encouraging iterative learning and inquiry [32].To steer learners successfully through these complex problems, we incorporated a variety of scaffolding techniques: • Primers: tutorials that equip learners with fundamental concepts and offer practical exposure through worked examples[2, p.106], teaching skills and knowledge they can apply to tackle more advanced hands-on tasks.• Starter code: helps learners gain a primary understanding of the problem without being overwhelmed by its complexity, thereby making tasks more attainable.[35] • Autograders: software that evaluates learners' work and provides contextualized and actionable feedback for each subproblem, allowing learners to accumulate scores as they solve each subproblem.The intent is to foster self-monitoring among learners by provideing regular, tangible acknowledgment of learners' incremental accomplishments.For example, one project, teaching basic string manipulation, asks the learner to format the output of a simulation to make it readable.The scenario casts the learner in a role (a novice developer at a company) for whom a task like this might be appropriate; it also set expectations about code quality, not just correct functionality.
A primer tells them how to set up a programming environment; a project writeup sets up the scenario and describes the approach needed to succeed.Learners can submit repeatedly to an autograder, whose feedback indicates how to improve each component of their solution, and with which learning objective the advice is associated.

Reflective Learning and Discussion
In each PBL project, subsequent to the completion of all handson tasks, the platform facilitates reflective learning by requiring learners to craft a reflection post encompassing their entire learning journey throughout the project.Guidelines are provided to aid students in this process, focusing on two key objectives: to selfassess their attainment of the learning objectives, and to share their problem-solving experience and takeaways.Following the project deadline, all students' reflection posts become accessible to the entire class for review, and learners are required to engage with their peers' posts by providing feedback and discussion.This social and interactive element aspires to nurture communication among peers along with social integration, thereby allowing learners to connect and contribute to a community of mutual inquiry and progressive learning.

Professional Development
Despite the integration of various methods designed to mitigate the challenges inherent to PBL and to alleviate the heavy time commitments from instructors, the role of instructors remains integral to student success in PBL.To ensure a successful transition to the PBL model, we organized a week-long preparatory workshop for instructors.This workshop prepared instructors to support students and identify potentially at-risk students using a learning dashboard, enabling timely communication and intervention.

RESEARCH METHODOLOGY
We used surveys, demographics and course performance to characterize student experiences relating to barriers identified in previous literature, especially associated with gender-linked differences.

Course delivery
Each participating instructor decided how they would employ the curriculum at their own institution, and which of the curriculum's eight units to use.One section required just 4 projects, seven required 6 projects, two required 7, and five required all eight.Three of the sections were offered in a physical classroom; these required 4, 6, and 6 projects, respectively.Sections requiring all eight projects were all offered online as self-paced courses.For this study, we omit data from one college, which offered a single self-paced section in which the surveys were not administered at appropriate times.
From the remaining schools, 231 students consented to research.Figure 1 describes some characteristics of the students: 20.8% of participants who answered the question identified as women, on par with US national averages for women graduating in computer science.Out of 15 class sections in the Fall of 2022 and Spring of 2023, 231 students consented to research; of those, 194 (84%) took the background survey.152 identified as men, 32 as women, 2 as non-binary, and eight preferred not to answer.

Informed consent and data collection
The study design and instruments were approved by our university's IRB.Students and instructors were presented with an informed consent form when they first logged into the course; those who did not consent could still take the course, but their data was withheld from researchers.
To understand how and when students engaged with the course platform and curriculum, we gathered student assignment submissions, and assignment grades.

Survey instruments
To understand our PBL's effects on student confidence and STEM identity and the role of support and belongingness in their interaction with the course, we administered a 28-question "frame of mind" survey asking about self-efficacy, belongingness, and STEMidentity.Both surveys were administered at the beginning of the term; the frame-of-mind survey was administered twice more: at mid-term and at the end of the term.The frame of mind survey measured several aspects of selfefficacy, using a five-point Likert scale (1=Not at all confident, and 5=Extremely confident).We asked students about: • Course-SE: Their confidence in their ability to succeed in the course.Questions were derived from Pintrich's Motivated Strategies for Learning Questionnaire (MSLQ).[34] • CourseLO-SE: Their confidence in their ability to accomplish specific learning objectives in the course.These were designed to be similar to questions used by Steinhorst [38] • STEM-SE: Their confidence in their ability to succeed in a STEM career; adapted from the Longitudinal Assessment of Engineering Self-Efficacy (LAESE) questionnaire [20].• BELONG: The sense that the student was respected and their growing expertise acknowledged by other students.We adapted questions from Knekta et al. [24], adapting a subset of questions from both the "valued competence" and "social acceptance" subscales.• STEM-ID: STEM Identity, the extent to which a student identified as a STEM professional; we used a single-item scale validated by McDonald et al. [30].
To understand how the approach interacted with students' background and situation, we also administered a background survey, asking eleven questions about student characteristics such as ethnicity, gender, age, and working hours and similar questions not analyzed here; this survey was administered just once at the start of the term.

Survey analysis
We mapped students' Likert responses to an integer, and averaged within each metric described above, resulting in a 1-7 scale for STEM-ID and a 1-5 scale for Course-SE, CourseLO-SE, STEM-SE, and BELONG.
Student compliance with end-of-term surveys was significantly lower than start-of-term surveys; about 25% of students who completed the first survey also completed the last survey.The 25% was not a fair sample of the course, since students who dropped out or stopped participating in the course did not complete the final survey.When comparing start and end of semester survey results, we will clarify whether we are comparing the full population or just students who completed both surveys; we also take into account the sampling bias in our interpretation.

Why did students persist?
Out of 231 students who consented to research, 88 (38%) persisted to the end and attempted the last project they were assigned.The largest factor we identified influencing persistence was the modality of delivery.
Our study engaged with students across three distinct learning modalities: face-to-face synchronous (13%), in which instructors set deadlines and met with students regularly in a classroom; online synchronous (48%) in which instructors set deadlines but only met with students remotely, and online asynchronous (39%) which were self-paced online courses, in which instructors were available to answer questions and provide support.(Figure 1).
Persistence rates varied significantly across these three modalities.The face-to-face synchronous mode, while constituting only a minority of the student population, exhibited notably higher performance (p=0.0001) and persistence.In Project 4, the most advanced project that was assigned by instructors in all three modalities, 77.8% of face-to-face students tried the project, 60.4% of onlinesynchronous students, and only 43.3% of online-asynchronous students (Figure 2).
We were unable to compile clearly comparable information about face-to-face retention rates in the participating schools, but what clues we could find suggested that our retention rates were lower than their non-PBL classes.PBL's complexity and high demands on instructors and students of might contribute to lower retention.
On the other hand, the persistence rates in online, asynchronous delivery exceed what one might anticipate from a MOOC.Generally, MOOCs are reported to have retention rates falling below 15% [21,23,42,47].
These modalities offer varying degrees of direct human interaction, which can foster different levels of perceived social support.
As we shall see in Section 5.4, persistence did not differ significantly by race or gender, however students' perceptions and beliefs about themselves and their abilities did.

Self-efficacy
As described in Subsection 4.3, we measured three types of selfefficacy on a five-point scale (1=low, 5=high): for success in the course (COURSE-SE), for success at a sampling of course learning objectives (CourseLO-SE), and for success in a STEM career (STEM-SE).Table 1 summarizes the results.
The first row shows that at the start of the term, students' sense of belonging, STEM-identity, and self-efficacy for the course and for a STEM career was above the midpoint of the scale.They had less self-efficacy for the particular learning objectives of the course; the wording asked if they could write, debug, or explain simple programs "at this time", so it is to be expected that they would not yet be confident about these tasks.
By the end of the term, students' CourseLO-SE increased more than a full Likert scale point, on average, from 2.3 to 3.4.The endof-class survey had poor compliance (only about 25% took it, partly because of high dropout of the course); thus the difference may be partly due to selection bias, since people who left the course early might likely have low self-efficacy for course learning objectives.
Factoring out such students by considering only students who took both exams shows only a slightly smaller increase.Those students who persisted had slightly higher starting self-efficacy for these LOs (2.5 vs. 2.3).
Course-SE started out fairly high for most students at 3.8 (just below 4, "Very" confident) and decreased significantly to 3.3 (p=0.02).The contrast between these measures suggests that students perceived the course as more difficult than expected, even for those who stuck with it to the end.

Belongingness, support, and STEM identity
Students' sense of belonging was also fairly high (3.71) to start with and decreased to 3.55 by the end, a small but statistically significant drop (p=0.02).This drop is due mostly to online classes.Belongingness did not change significantly in face-to-face classes.
Students' average STEM identity scale started at 3.79 (just above the midpoint on a 7-point scale), and did not change significantly; this is unsurprising as STEM-identity is theorized to be a relatively stable trait.In a focus group session, one instructor commented on students who seemed resistant to changing the way they saw themselves: e.g.students from other departments who did not see themselves as future computer scientists.Sometimes such students seemed unwilling to make the "emotional investment" to wrestle with a problem requiring exploration and reasoning.
Related but different from the STEM-ID measure was STEM career self-efficacy (STEM-SE); although it appears in the table to decrease, the difference was not significant since the standard deviation was quite wide for this measure (1.79 Likert points for the start-of-term question vs. 0.8 for BELONG at start of term).

Demographic Differences
Women were severely underrepresented coming into the class (Figure 1, upper left pie chart); 16.5% of students answering the startof-term background survey identified themselves as women, while 78.4% identified as men.While this is a typical ratio for enrollment in postsecondary computer science education classes, the scarcity of women highlights the importance of retention.
Women performed as well as men in both project scores and in persistence.Women averaged 39% of the possible points on the projects; men averaged 46%; the difference was not significant (p=0.21).71% of men attempted the last project, compared with 52% of women; this difference was also not significant (p=0.58).
However, their perceptions were different.Women started out with lower average self-efficacy for course learning objectives than men did (CourseLO-SE: Women:1.9Men:2.3, t-test, p=0.27).While men's efficacy on this measure increased by the end of the course (CourseLO-SE 2.3-3.5, difference of 1.2, t-test p=0.0001), women's did not (CourseLO-SE 1.9-2.4,difference of 0.5, t-test p=0.38)This echoes prior work showing that women's self-efficacy in postsecondary computer science education tends to be lower than men's despite similar grades [19].Men and women did not differ in other measures of self-efficacy.
The racial distribution was about the same as the US population.There were no significant differences in the attitudinal measures by race (p > 0.05 according to ANOVA for each question) with one exception, change in STEM-SE, STEM career self efficacy.Native American students became significantly more confident in their likelihood to succeed in a STEM career (4 Likert scale points, however there were only 4 Native students out of 231) and Black students became less confident (2 Likert scale points (24/231)); White students (83/231) stayed about the same.

DISCUSSION
6.1 Implications for Teaching 6.1.1Metacognitive skills to enable persistence.Project-based learning requires more intentional engagement with the material, but when students first approach the class they may lack the selfefficacy or the metacognitive skills to do so.Our results suggest that a divide-and-conquer strategy should be an explicit learning objective of a course, for example by breaking early projects down into tasks for students, explaining the thought process, and giving opportunities for positive feedback along the way; or incorporating diagrams or instructional videos that guide students through task decomposition.
6.1.2Providing Social Support in PBL.Our results suggest that face-to-face delivery of the course is bound to be more successful than online delivery; PBL necessarily demands some resilience and curiosity of students.An instructor who regular sees students in person can encourage students they see struggling, and model cognitive strategies, such as curiosity and iteration in the face of failure, carefully attending to feedback, and the practicalities of using testing and debugging tools.
However, our experience with community colleges suggests that face-to-face delivery will not always be realistic.Online courses allow resource-strapped institutions to provide instruction to far  1: Survey results for self-efficacy, belongingness, and STEM-identity."All responses" averages all surveys taken; "Paired responses" counts only surveys from students who took both start-and end-of-term surveys.All measures scale from 1-5, except STEM-ID, which is 1-7.* = significant change at p<0.05 more students than they could afford to face-to-face, and asynchronous courses similarly extend a school's reach by enabling working students with many responsibilities to learn at their own pace.
Student persistence rates in the self-paced sections was high compared to typical MOOCs; it may be that MOOC-like courses offering project-based learning could benefit from using support features such as interactive autograders, reflection forum discussions, and opportunities for interaction with the instructor.
6.1.3Gender and ethnic equity.It is not enough that project scores and persistence were equitable for different gender and race groups; if students leave the course without the confidence and intention to pursue further education and careers in the field, then we will not succeed as well at growing and diversifying the workforce.The fact that women left the course with doubts about their ability to perform course learning objectives, and Black students had doubts about their likely career success, needs to be addressed.

Threats to Validity
6.2.1 Generalizability of the PBL approach.This research adopted a particular approach to project-based learning, but the effects we saw may have been particular to our implementation and not to project-based learning in general.In particular, some definitions of PBL emphasize collaborative group work.

Sampling. Several sources of bias complicated our analysis:
Selection bias Our samples of community colleges and students were not randomized; colleges chose to adopt our curriculum, and students chose to take the course.Thus, we cannot easily generalize to other community colleges.Furthermore, our sample was small enough that breaking down results by ethnicity and gender resulted in very small numbers of students.Future experiments with larger cohorts may yield more fine-grained information about the effects of PBL on different populations.Larger sample size will also provide statistical power, given the high variability of our courses, in geography, modality, instructor, and subset of modules taught.
Response bias We saw relatively low compliance with the final survey; some students opted out, and some did not complete the course.We did not offer incentives for taking the survey, since some instructors were not comfortable with this, and we wanted uniformity across the sections.In later semesters, we are reminding instructors to mention the research and prompt students to take the surveys.However, the response bias caused by students dropping the course before the second survey can't easily be corrected; the bias must be considered in the interpretation of the results.
Social desirability bias Like most research on psychological measures like self-efficacy and belongingness, there is a risk that participants may be motivated to answer in the way they want to be perceived, rather than candidly.

CONCLUSION
We recognize the potential for project-based learning in community colleges to address the problem of training the CS/IT workforce.PBL not only imparts content knowledge but also fosters skills such as critical thinking, problem-solving, and metacognition.Moreover, PBL exhibits promise in addressing the underrepresentation of women and minority groups in technical domains.
However, implementing PBL in resource-limited community college settings presents unique challenges.Our results underscore the role of social support and engagement methods in enhancing student outcomes, and the need to find ways to build such support when face-to-face instruction is not practical.
As community colleges strive to make PBL a viable mode of instruction, we suggest the need for tailored strategies to address barriers, encourage self-efficacy, and build a sense of belonging among students.The findings emphasize the significance of both pedagogical design and instructor training to optimize the benefits of PBL, particularly in diverse and resource-constrained learning environments.The study's insights provide a foundation for future developments in project-based learning and its adaptation to meet the demands of an evolving CS/IT workforce.

Figure 1 :
Figure 1: Breakdowns of students by gender, ethnicity, and course modality.Students roughly matched US ethnic breakdown, but gender skewed heavily male.Most students took the class in an online modality, some synchronous (i.e. with shared deadlines) and other asynchronous (i.e.self-paced).

Figure 2 :
Figure 2: Students stopped attempting projects as the course went on, especially in online or asynchronous courses, and thus achieved lower scores on average.(Left) X-axis is projects; Y-axis is number of students who attempted the project.(Right) Y-axis is distribution of scores achieved.