Applying CS0/CS1 Student Success Factors and Outcomes to Biggs' 3P Educational Model

Over the past decades, computer science education (CSEd) research has studied the multitude of factors that may impact student success in introductory programming courses (CS0/CS1). The lack of foundational structure behind how these factors interrelate has made it difficult to gain a thorough understanding of this area of CSEd literature. Gaining a deeper understanding and applying structure to these factors would allow CSEd to adopt better teaching practices, study habits, learning environments, course materials, etc. and to better understand the student experience to better foster success among a broader population of students. Our systematic literature review used search criteria for factors that predicted student success in CS0/CS1, which yielded 311 research articles. We then mapped this body of work under the Biggs' 3P (Presage, Process, Product) educational model, which provides a comprehensive framework for how students engage with learning opportunities. We discovered that although many studies focused on the Presage and Product phases of the model, fewer studies mapped to the Process phase, which describes the students' active learning processes. Our study shows there is a potential gap in the literature and future studies should focus more specifically on how students choose to engage with learning opportunities and what factors may be hindering that engagement throughout a learning period.


INTRODUCTION
For many students, introductory computer science (CS0/CS1) is the course that introduces them to the world of computing.For many others, it is notoriously the course that deters them from considering computer science as an academic or professional pursuit.In an attempt to understand and address student experience and performance, researchers have sought to study the impacts of learner characteristics, interventions, and pedagogies on student learning and experiences in CS0/CS1.The factors themselves can range across many categories such as sense of belonging [16], self-efficacy [15,29], instructional practices [28,38,41], peer learning [31,42], cognitive ability [2,20], and achievement goals [49,50].Of these factors, some have consistent and replicated outcomes, such as prior experience being positively correlated with higher performance or women generally entering CS0/CS1 with less prior experience than men [4,11,45,46].However, other factors are understudied or produce inconsistent results, limiting our knowledge of how to best support students.To our knowledge, there is no taxonomy that attempts to survey and organize the extensive amount of CS0/CS1 literature in a way that organizes factors into categories, highlights what has been heavily studied, explores the relationships between factors, and identifies what gaps exist.
For this study, we used Biggs' 3P model because of its dynamic qualities and relationships between three phases: presage, process, and product.At its core, this model describes the learning process as an interaction of many factors from different sources of origin and quality.We use this model as a framework to organize the current CS0/CS1 literature by mapping papers in our review to the different components of the 3P model in order to highlight what aspects of the student experience have been studied.Our results of this study are as follows: (1) The research landscape does not sufficiently take a student's learning process and engagement into account.While outcomes are heavily studied, the actual learning process a student has with the material is seldom evaluated.(2) Many studies do not focus on understanding how student factors are evolving or how those factors are specifically impacting how students engage with materials or how that engagement evolves and changes.

PREVIOUS WORK 2.1 Understanding CS0/CS1 Success and Struggle
A popular area of CS1 research has focused on understanding what characteristics of learners, educators, and environments are associated with student success in these courses.Studies have discovered a variety of mutable and immutable characteristics and experiences that correlate with or predict student CS1 success.For example, one of the most consistent predictive factors of CS1 success is prior programming experience; many studies conclude that students entering with prior programming or computing experience tend to outperform their peers who enter with little to no experience [4,11,[45][46][47].Additional factors of success that have been identified through research span across cognitive and spatial skills [2,3,6,7,9,13,20,22,23,30], behavioral [10,18,26], and mindset [17,27,33,34,40].Within these large areas, research has explored the effect of specific factors.For example, certain cognitive abilities, such as mental models and performance on specific cognitive tasks (e.g., paper folding), have been associated with success in CS1 [3,6,9,13,22,23,30].Other studies have shown how a student's sense of belonging, selfefficacy, and confidence is associated with their performance and enjoyment of CS1 [1,15,16,19,25].Additional research has focused on how pedagogy can improve student learning in CS1 [1,32,38,42].Overall, success characteristics in CS1 cover a broad range of topics, and it is difficult to determine how these characteristics interact and whether some are more significant than others.For example, while sense of belonging and self-efficacy are both related to success, some research suggests that sense of belonging is an outcome of self-efficacy rather than a causal predictor of success [19,21].A nuanced understanding of how these factors interact allows researchers and educators to focus on factors that affect success (i.e., self-efficacy).
There have been some studies focusing on the interactions between different factors and how they influence CS1 student outcomes [20,36,43,44].A study conducted by Lishinski et al. looked at differences in student programming performance based on their gender, motivation, goals, and levels of self-efficacy [21].They discovered goal orientation and metacognitive exercises impact student self-efficacy, which predicts students' CS1 outcomes.Another study by Rountree et al. used decision tree classifiers to predict success or failure in a CS1 course, concluding that groupings of factors such as age, prior background, and desired grades were predictive of success and failure in the course [36].Early work conducted by Wilson and Shrock studied twelve factors that contributed towards success in a CS1 course [47].Their findings indicated that factors such as comfort level, math background, and prior computing experience were positively associated with performance while attributing success and failure to luck was negatively associated with performance.
Aside from understanding what predicts success, other studies have shifted the focus from factors of success to sources of student struggle.A recent study conducted by Salguero et al. looked at student success factors through a more holistic lens.Their study claims that student struggles in introductory computing courses possibly interact with one another and are not purely academic or personal [37].They found that the lowest performing students reported high stress levels across more than a single source of struggle, indicating that when students struggle heavily in intro programming, they appear to struggle with many issues.
Many studies have also looked at the experiences of certain demographic groups in CS1.Research has identified that women and members of certain demographic groups, such as Black and LatinX students, significantly struggle to succeed and be retained in CS programs [5,14,38].This work has led researchers to explore what reasons are causing CS attrition amongst these students.It has pointed to factors such as low self-efficacy, lack of prior programming experience, and low interest in the field as reasons for these negative outcomes, which echoes a recent conclusion that students who struggle typically struggle with multiple risk factors [8,24,35,37].The range of factors students appear to be impacted by has led us to search for a framework to begin organizing these research results.

Biggs' 3P Model
Biggs' 3P (Presage-Process-Product) model is an educational model that structures the student learning experience as relationships between potentially interacting factors across three phases: presage, process, and product.
The presage phase refers to factors related to students prior to engaging in learning.This phase includes both student-related factors and instructor/teaching-related factors.Some examples of student-centered presage factors include prior experience, mindset, demographics, and their preferred learning approaches.Because Biggs' model focuses primarily on the student in the learning process phase, non-student factors that set up a learning experience from the instructor's perspective are also considered presage.Some examples include teaching philosophy, departmental policies and norms, and the types of assignments and tasks the instructor designs.
The process phase refers to factors related to the actual act of student learning.Biggs defines these factors in terms of a student's learning approach, which they classify as being either a deep or surface learning approach.When a student is presented with a task (assignment, reading, etc.), they decide how to go about completing the task.Based on the technique or process chosen, it likely can be classified as either a deep or surface learning approach.A deep approach refers to actively seeking to understand the material or subject while surface learning is more passive, in which a student focuses on simply reproducing the material without necessarily fully understanding it.Prior work in CSEd has shown the benefits of a deep learning approach over a surface learning approach when measuring programming performance [12,39,48].In short, the process phase describes the techniques (active or surface) that a student decides to use when engaged with a task.
The product phase is the final phase of Biggs' 3P model and is defined as the resulting experiences and outcomes a student achieves after the learning experience.These outcomes can represent different aspects of student success and experiences, such as performance (e.g., grade), affect (e.g., emotional responses), and preparedness (e.g., skills).
Prior CS1 studies have focused on many different types of factors and their interactions in relation to student outcomes.The Biggs' 3P

STUDY DESIGN 3.1 Research Questions
Our study addresses the following through the lens of the 3P model: • How do CS1 research papers map onto the Biggs' 3P model?Which areas have high concentrations of papers?Which areas have low concentration of papers?

Paper Selection
The goal of this project is to explore a sample of the CS0/CS1 literature and map papers to the 3P model.To do this, we searched for relevant literature related to success or struggles in CS0/CS1.Key terms were selected based on desired student outcomes (retention/performance), student characteristics (self-efficacy and BLNPI/URM), and pedagogy.Pedagogy was selected because the research team sees this as a research area that actively aims to improve student outcomes.Synonyms and phrases to represent these key terms were then brainstormed amongst the research team.
We created a Python script to query Google Scholar using the following search phrase, "Factors influencing [X] in [Y]".We decided to use Google Scholar as it acts as an aggregator of papers across databases.Although we acknowledge that not all papers will be selected, our intention is to get a representative sample, not all existing papers.The possible values of X are key terms related to popular areas or outcomes from CSEd literature.The possible values of Y are context terms that describe introductory CS which include introductory computer science course, CS1, CS0, and introductory programming.Refer to Table 1 to see the key terms for X that were searched.An example of a query built from the terms is, "Factors influencing drop outs in CS1", where "drop outs" is the X value in the interest of retention, and "CS1" is the Y value.
After running the script, we collected 2090 papers in our initial dataset.The research team then read the titles and abstracts of the papers and removed any that had no clear connection to CS0/CS1 or were duplicates.This reduced the number of papers to 947.
A key aspect of this work is that we wanted to examine traditional university/post-secondary CS0/CS1 contexts.CS0/CS1 has gained popularity and has expanded into the K-12 and online space throughout the years.To remain within scope, there were certain criteria used for inclusion of papers.

Academically
Published.The paper must have been published in an academic conference or journal as an article.We omitted graduate thesis, dissertation, working group, poster presentation, abstract-only, and doctoral symposium papers.We believe work found in these articles will be available in other research articles by the same author.Textbooks were also not considered as we believe the information found in them would be based on relevant literature that our search would include.

3.2.2
Post-Secondary CS0/CS1.Because we want this work to focus primarily on traditional post-secondary introductory CS offerings, we omitted any research paper that reported on other types of offerings.Papers on offerings of CS0/CS1 in the K-12 or MOOC space were removed from our analysis.We believe the settings of these offerings create new challenges that are unique to that setting and not typical in traditional offerings.Additionally, we only included papers that included samples from CS0/CS1 exclusively.If other courses were studied in the same paper, we only included the paper in our analysis if the results for CS0/CS1 participants were independent of the other course samples.

Experience Reports.
We did not include experience reports if they did not provide sufficient statistical analysis or did not provide sufficient exploratory/explanatory power in their results sections.This criterion also excludes studies that report on a learning tool's or intervention's development and mentions future plans for implementation.

3.2.4
Qualitative, Quantitative, and Mixed Methods.We included papers that used qualitative, quantitative, or mixed-method approaches.However, we required that qualitative studies must include more analysis aside from simply stating feedback from participants.Some level of qualitative analysis, such as coding responses or phenomenography must be used to be included.Quantitative studies must provide some form of inferential statistical or empirical analysis to be included, such as significance testing, regression coefficients, or effect sizes.
Once these criteria were applied, our final sample included 311 papers.Figure 1 illustrates how many papers we had at each phase of our literature search.The full list of papers can be found at https://github.com/adsalgue/SIGCSE2024_3PModelPaper.git.

Analysis Process
The following sections describe the process of analyzing the 311 papers in our final dataset.We describe the content we gathered from each paper and how papers were grouped and analyzed.Figure 2: Labeled Biggs' 3P model.I-IV are the nodes and aspects that make up each phase.A-F denote the relationships between these groupings.Since Biggs' model is dynamic, each of these factors across the phases can potentially interact and influence each other.

Data Gathered Per Paper.
For each paper we recorded the relationships, outcomes, and conclusions made by the paper.Relationships between factors and outcomes of the paper were the most important pieces of information we recorded and are at the core of our results.Although many papers report on many relationships, we only considered statistically significant outcomes in quantitative studies.In qualitative studies, due to the high likelihood of there being no statistical testing, we decided to include the relationships reported while marking results being qualitative in nature.Once again, qualitative studies were required to have some level of qualitative analysis, such as coding responses.Other information such as sample size, sample characteristics, data collection methods, and data analysis methods were recorded but not part of our mapping to the 3P model.

Analyzing Papers
3.4.1 Mapping to Biggs' 3P Model.For our mapping process, we read and recorded information from 311 research papers.Of those studies, 297 reported significant findings.We only consider papers with significant findings in our 3P model mapping.For example, if a study looked at gender and pedagogy in relation to student CS0/CS1 performance, but only gender was statistically significant, we only include gender in our mapping and not pedagogy.After recording all the necessary information from our papers, we mapped the significant outcomes and their respective factors onto the 3P model as seen in Figure 2. We will use graph notation of nodes and edges to discuss our results.For the mapping we define the nodes (I-IV) as the following: The edges (A-F) are meant to represent associations or relationships between the types of factors above.These edges do not indicate that the relationship is causal, rather that simply one exists.These edges will highlight the majority of outcomes reported in the papers of our literature review.
Factors were mapped onto a phase and outcomes onto one of the arrows in the 3P model.For example, if a paper focused on measuring the relationship between sense of belonging prior to the class and their final exam results, this would map to the Presage and Product phases along with the arrow connecting the two.Once we recorded all the necessary information, we used negotiated agreement in the mapping process.The first author made an initial pass through each of the study outcomes and mapped each factor to one of the nodes seen in Biggs' 3P model (i.e.Student Factors, Instructor Factors, Learning Process, and Outcomes).After the first author mapped all the factors, the second author reviewed those mappings.At this time, authors one and two discussed the mappings of each paper's outcomes based on the node definitions seen above.Initial agreement on factor mapping is labeled as a raw agreement.When author two disagreed with the mapping done by author one, a discussion was held to reach a consensus on which mapping would be correct.This involved discussing what nodes best define the factors in the paper's outcomes.Once again, heavy use was made of the node definitions.When a consensus was reached between authors one and two, this was labeled as a negotiated agreement.In the cases where no consensus was reached despite much discussion, these were labeled as disagreements.
The same process was conducted for mapping each paper's outcomes to the edges of the 3P model.After completing the negotiated agreement for the nodes, the first author created mappings of edges for each paper on their own.The second author was then brought in to discuss edge mappings.Table 2 and 3 lists the results of our negotiated agreement process of mapping papers to the nodes and edges of the 3P model respectively.research is to examine how students perform at the end on some sort of performance or qualitative metric.Finally, less than half of the papers had findings related to the student learning process in CS0/CS1.This is an important finding because it shows not many papers explored how a student reacts and learns during a course, only the final overall outcome.This is an important aspect of the student experience that future studies should focus on, primarily which factors impact students in their learning process and how certain factors cause them to directly engage (or disengage) with CS0/CS1 material.

Edge Analysis
Table 5 shows the count and proportion of edges that were mapped using significant relationships and outcomes reported in our sample of papers.Note that each paper often mapped more to one edge, which is why the proportions do not sum to 100%.Our mapping of the edges to Biggs' 3P model indicates large gaps in the types of significant relationships and outcomes being reported on in the CS0/CS1 literature.
The majority of papers mapped to the edges connected Student Factors to Outcomes and Instructor Factors to Outcomes.These connections are understandable as many papers in CS0/CS1 focus on how student factors (sense of belonging, CS interest, race/ethnicity, and prior programming experience) relate to their performance or other metrics in a CS0/CS1 course.Similarly, with correlations between Instructor Factors and Outcomes, pedagogical interventions and improvements are a major part of CS0/CS1 research, which would fall under the instructor factor node.The research primarily focuses on how pedagogy can impact student outcomes such as pass rates, retention, and interest.Therefore, a high proportion of papers exploring such relationships was expected.
However, when we begin exploring the edges related to the student learning process node (edges D-F), we see a large drop in the number of mapped papers.This indicates that CS0/CS1 research is not focusing on identifying significant correlations between student learning processes and other factors.Many papers are missing observations and data about how specifically certain factors are influencing student engagement and how they choose to approach material.Is a certain pedagogy or student personal factor causing them to engage more with the material and build interest, in turn allowing them to employ deeper learning techniques?Although assumptions can be made, additional observations and data gathering/analysis will be needed.This is crucial information that requires more research as it is important to understand how factors impact the learning process during the course and if they are truly responsible for the observed outcomes.An updated Biggs' 3P model showing our mapping results can be found in Figure 3.

DISCUSSION
Our results highlight areas that have been studied in CS0/CS1 research and opportunities to further our knowledge in future studies.

Presage and Outcomes
Our findings indicate that many studies focus on understanding the relationship between a student's presage factors and outcomes.Researchers appear to use these factors as a prediction tool to see how students will perform on performance metrics such as exams or grades.Although this information is valuable, we believe it will be invaluable to the community to understand the interaction among these presage factors and how those directly impact the student learning process.Although papers do not outright ignore the learning process of the student, data gathering and analysis within this phase is small compared to the Presage and Product phases.

Importance of Learning Process
The processes students employ when learning is a key aspect of the student experience.Their level of engagement along with the techniques they employ merits closer investigation as it can provide a clearer picture of their experience in a CS0/CS1 course.Although certain presage factors appear to be correlated with student success, the research community needs to explore how these factors influence student decisions during the course rather than only identifying correlations to outcomes.For example, one study can focus on understanding how prior knowledge impacts coping mechanisms during the course and how those mechanisms are evolving during specific milestones in the course.

Commonly Appearing Factors
During this project, the research team recognized many common factors with similar outcomes.We noticed heavy evidence showing factors such as prior experience and active learning to be associated with positive outcomes.Meanwhile, factors such as low sense of belonging, low interest, and lack of prior experience were associated with negative outcomes.Given the abundance of studies on these topics, our findings suggest further work on these topics should examine the relationship between these factors and the student learning process to better refine our understanding of the role these factors play.

Call for Theory
The application of this work to Biggs' 3P model aims to highlight the pressing need for a CS-specific theory to explain the CS0/CS1 student experience.Currently, there are some popular theories being used in the field such as Bandura's theory of self-efficacy, achievement goal theory, and constructivism, to name a few.This overarching analysis shows that CS0/CS1 research literature is fractured into small pockets of interrelated studies.We encourage future work to focus on consolidating this current body of research into a common theoretical framework to support a broader understanding of how various factors relate to student learning and subsequent outcomes.

CONCLUSION
In this study, we mapped 297 out of our 311 original CS1 papers onto Biggs' 3P educational model that describes student learning.The majority of papers focused on instructor and student factors and their direct relationship with outcomes and did not include the actual process of a student engaging and learning the material.We found that a lower concentration of papers focused on the actual learning processes and techniques a student employs in learning CS during a course.
Overall, our study indicated several areas for future work in this field.We have seen in prior work these factors can have interacting effects on students so now it is the time to begin formulating and organizing all this research into cohesive frameworks that consider the interactions among different types of factors and how they affect learning throughout the learning process.In the future this could perhaps lead to the development of a CS0/CS1-specific theory to understand the impact factors and their interactions have on student learning in these courses.

Figure 1 :
Figure 1: Number of papers after each phase of our sampling procedure.

•
I -Student Factors: These are factors related to the qualities students possess across a variety of different areas such as technical skills, psychological, cognitive, and self-perceived.• II -Instructor Factors: These factors are related to the conditions of the course that instructors employ and have some level of control over such as assignments, pedagogy, learning environment, and evaluation metrics for student performance.• III -Learning Process: These factors relate to things involved in the learning of materials during the course.These include factors related to how students engage with the learning process and material such as learning approaches and various in-class behaviors such as help seeking.• IV -Outcomes: These are the different types of outcomes students can achieve after a learning experience such as grades or emotional responses.

Figure 3 :
Figure 3: Updated Biggs' 3P model with our results from our node and edge analysis in the format [n, %] where n is the raw number of papers mapped and % is the percentage in relation to the entire sample.

Table 1 :
Set of key terms that were used to scrape research papers online.These values were for X in the query, "Factors influencing X in Y".
model can capture this dynamic interaction and interrelationship that appears to exist during a student's CS1 experience.The model defines learning as an ever-evolving process where a student approaches a learning situation with some mindset based on existing qualities, experiences something within a learning context, and emerges with some change in qualities or skills that can influence the next learning experience.We believe that mapping current CS1 research to this framework can provide important insights into the status of this research area.

Table 2 :
Results of negotiated agreement process for 3P nodes.

Table 3 :
Results of negotiated agreement process for 3P edges.

Table 4 :
Number of papers mapped to each node in Biggs' 3P model.

Table 5 :
Number of papers mapped to each edge in Biggs' 3P model.

Table 4
shows where the counts of papers that had significant findings fall in each of the nodes of the 3P model.We see that over half the papers reported conclusions involving Student Factors, Instructor Factors, and Outcomes.Papers that fell into student factors focused on areas such as gender, race/ethnicity, sense of belonging, prior experience, attitudes, and cognitive abilities.It is surprising that just over half of our papers had significant findings on Student Factors.Considering how CS0/CS1 research is focused heavily on the student experience, we would have expected more papers to fall under this category.Papers that fell into the Instructor Factors category were primarily focused on student responses to different pedagogy, curriculum, interventions, or types of assignments.Papers that fell into Outcomes focused on some sort of assessment (exam, programming assignments, test score, etc.) or on a student's change in ability or attitude.It is understandable that a large proportion of papers map onto Outcomes as a shared goal of CS0/CS1