The First Five Years of a Dual Track Programming Series

The varying skill levels of students in entry-level university programming courses complicates an already challenging task for instructors. Combining students of differing skill levels into a single course can have negative impact on the classroom experience making some students feel inadequate compared to their peers. Prior to 2018, our large four-year R1 research university in the United States had a single lower-division programming sequence. Since only a single sequence existed all students (both majors and non-majors) were grouped together in the same courses. In this paper we analyze the impact of developing a dual lower division programming course sequence and deploying a placement exam. Our analysis tracks student performance in upper division courses over the first five years of the dual track programming course sequence, and we present the performance difference based upon the particular track the students take.


INTRODUCTION
The varying skill levels of students in entry-level university programming courses complicates an already challenging task for instructors.Combining students of differing skill levels into a single course can have negative impact on the classroom experience making some students feel inadequate compared to their peers [5].Prior to 2018, our large four-year R1 research university in the United States had a single lower-division programming sequence.
The first two years at our institution are lower-division -that is, mainly preparatory with the latter two years being upper-division focused on subject depth.Since only a single sequence existed, all students (both majors and non-majors) were grouped together in the same courses.Recognizing the issues of having all students grouped together in the single sequence, and with the rise of other majors requiring programming, we developed a second programming sequence designed for non-Computer Science/non-Computer Engineering majors.Additionally, to aid students in deciding in which course to begin, we developed a placement exam that students must pass to start in the first major track course.The remainder of this paper is organized as follows: in Section 2 we provide the background about our previous programming course sequence followed by related work in Section 3. In Section 4 we describe the dual lower-division programming course sequence and the placement exam, and in Section 5 we provide the results of our analysis.A discussion of the results is in Section 6, and limitations of the work are provided in Section 7. We conclude and provide a path for future work in Section 8.

BACKGROUND
The Department of Computer Science at UC Davis had a single programming sequence of Engineering Computer Science (ECS) courses ECS 10, ECS 30, ECS 40, and ECS 60.Each course in the sequence was 10 weeks with 40 contact hours (30 hours of lecture, and 10 hours of discussion).The ECS 10 course taught students Python and was akin to a CS0/CS1 in that it assumed no prior programming experience.The ECS 30 course was also an introduction to programming course and taught students C and Linux.While ECS 30 was akin to a CS1 course, we advised students with no prior experience to start with ECS 10.Unfortunately, students do not always take advice from faculty and staff, and hence the need for a placement exam.Students learned introduction to software and object-oriented programming in C++ in ECS 40, and data-structures in C++ in ECS 60.The majority of our Computer Science (LCSI) and Computer Science and Engineering (ECSE) majors started in ECS 30.Our instructors had concerns over the years that our ECS 30, ECS 40, and ECS 60 courses had continued to be "watered down" to meet the needs of the large number of students most of which were non-majors.Anecdotally, the instructors felt that we needed to separate out the majors from non-majors, and that we were not serving either population well.Our instructors did not believe that non-majors necessarily needed C++ and felt that there wasn't another course for students that wished to learn more but in an approachable language like Python.At that time, we also had a newer Cognitive Science major with a computational track and a developing Data Science major that would need programming courses.Considering all needs we decided to develop a programming sequence for non-majors.

RELATED WORK
The most related prior work that we have found has been around the development and assessment of placement exams.We are unaware of analysis of dual lower-division programming sequences and their impact on the upper-division course performance.Several prior works have discussed the deployment of a placement exam to guide students to either one course or another [1,2,6,9], or for placement in a normal or accelerated/advanced track of same course [4,8].Archer, et al. [1] describes using a placement exam that is offered at the same time as the mathematics placement exams to place students in CS1121 or CS 1131.The offering of the placement exam is timed similar to ours, and the use of a survey portion to determine prior experience is also similar to our work.Miller and Reese [6] describe the use of a placement exam for the CSII course at Mississippi State University; the placement exam proved to show very strong correlation for success in a course that assumed prior object-oriented software development in C++.The authors state that the placement exam was being used to advise students if they should take CSI before taking CSII.Our placement exam differs in that it is not language specific and has many fewer questions.Soh et al. [9] designed an hours-long placement exam for CS1 based upon ACM CS Curricula 2001 that contained 50 questions in ten knowledge areas.In addition to assigning students to a pre-CS1 or CS1 course, the placement exam was also used for pre/post learning assessment.Again, our placement exam was much shorter and focused only on programming.More recently Bockmon and Bourke [2] developed a Placement Skills Inventory CS0/CS1 placement exam that consisted of 24 multiple choice questions to be completed within 90 minutes.The first 12 questions were taken from the Computational Thinking Concepts and Skills Test [7], with the second 12 question taken from the Second Computer Science 1 Exam Revised version 2 [3].Our placement exam is shorter in duration, and we only advise students to take one of two courses, where Bockmon and Bourke's work may advise students to take CS2.Podlipnig [8] utilized a placement exam over 4 years for placing students into two tracks of a CS1 course.The students who passed the placement exam started in the middle of the semester.Our work separates the students into two 10-week courses, so is different than Podlipnig's approach, but also somewhat similar as it is not too far off in duration of halfway through a 15-week semester.Decker et al. [4] used a placement survey with eleven questions, three of them being programming questions to separate between an accelerated and normal speed CS1 course.A major advantage of the cohorting of the students was an improved classroom environment.Our work is similar to Decker et al. [4] in that we surveyed the students for prior programming experience, but our placement exam requires no writing of code and has more programming questions, though our questions are somewhat sub-questions of four main groups.

DUAL TRACK AND PLACEMENT EXAM
This section describes the dual programming track and placement exam.

Dual Track
We started the development of the dual track by renumbering ECS 30, ECS 40, and ECS 60 as ECS 36A, ECS 36B, and ECS 36C to highlight the sequential nature of the courses.The ECS 10 course was renumbered to ECS 32A, and ECS 32B was developed as a follow-on course for non-majors who wanted to learn more about programming.The ECS 32B course is a data structures course but is taught in Python.Considering students might start taking the nonmajor track and then decide they want to major in LCSI or ECSE we developed ECS 32C and ECS 34.The ECS 32C teaches students to implement data structures in C, and the ECS 34 course covers C++ and graph algorithms.The ECS 34 course serves two purposes, first as a bridge for non-majors into the major and second as additional preparation for Community College graduates transferring into the LCSI or ECSE major.

Placement Exam
When designing the placement exam, we had several goals in mind.First, we wanted an exam that students with prior programming experience could pass, regardless of the specific programming language they learned.This is not unlike the approach that Tew and Guzdial [10] used for the design of the FCS1, though we did not validate our exam, and the purpose of our exam was for placement.Second, we wanted an exam that was easy to administer and grade since there would be many students taking it.Last, the exam needed to be available to students prior to their arrival at our university as they needed to enroll prior to the beginning of the term.We opted for a very simple language design that looks like Python and tests the students on tracing through several program examples to determine output.The language description is shown in Figure 6 in the appendix.Only two control structures are defined in the language: an if-then with optional else and a while-do loop.The exam is composed of four main programming snippets: a set of nested ifs, a simple while loop, a while with an if, and nested while loops.For each programming snippet there are multiple questions testing the output based upon different initial conditions.There are six questions on the exam that test nested ifs, three questions tested simple while loop, one question tested a while with an if, and three with nested while loops.An example of the simple while loop question is provided in Figure 7.The weighting of each problem is not equal; the more challenging problems are weighted lower in point value than are some of the simpler if problems.Currently 15 out of a maximum score of 18 is required to pass the exam.This represents the student being able to correctly answer all but the nested loops problems (or possibly missing a question through simple mistake).More importantly, our data shows that students who scored 15 or higher were able to succeed in ECS 36A without first taking ECS 32A.As part of the exam the students report their prior programming experience in months, maximum lines of code, and languages in which they have programmed.The exam is administered through Google Forms and is designed not to take longer than one hour (though no actual time limit is set).The students are required to use their institutional credentials to take the exam, but beyond that there is no actual proctoring.Since the students do not receive credit for the exam but are just allowed to take ECS 36A without first taking ECS 32A, we are not concerned about potential academic misconduct.This is a rare situation where students would truly be cheating themselves out of an opportunity to take the correct preparatory course.

Dual Track and Placement Exam Research Questions
After having run the dual track long enough for students to have taken the placement exam, completed one of the two tracks, and then have graduated (even those that take a fifth year), we had several questions we wanted to answer; in particular, we were concerned about performance in the upper-division ECS courses.We chose to compare difference in prior experience of six months (separate those of single semester vs. academic year or more), and largest program of 400 lines of code (separate typical ECS 36A assignment size vs. smaller).For performance in upper-division ECS courses we calculated the Grade Point Average (GPA) using a 0.0 to 4.3 scale for letter grades of "F" to "A+": • Is there a statistically significant correlation in placement exam score based upon self-reported prior experience in months?• Is there a statistically significant correlation in upper-division ECS performance for students based upon placement exam score?• Is there a statistically significant difference in upper-division ECS performance for students based upon placement exam score of 14 and below vs 15 and above?• Is there a statistically significant difference in upper-division ECS performance for students who did not pass the placement exam and took ECS 32A vs. those who ignored advice?• Is there a statistically significant correlation in upper-division ECS performance for students based self-reported prior experience in months?

RESULTS
This section presents the results of our analysis for the placement exam, the dual tracks, and the student upper-division ECS course performance.The placement exam was administered to 1,687 undergraduate students over five years.An analysis of the students upper-division ECS course performance was correlated back to their placement exam performance and the lower-division programming track that they completed.In each of the subsections we restate the questions from the previous section before answering them.

Placement Exam Efficacy
• Is there a statistically significant correlation in placement exam score based upon self-reported prior experience in months?We were first interested in determining if there was a correlation with self-reported prior experience in months and the performance on the placement exam.We found that there was a Pearson Correlation Coefficient (PCC) of 0.442 meaning there is a medium to large effect size.Given that the placement exam scores were well correlated with self-reported prior experience we are confident in continuing to use it as a tool to place students in ECS 32A or ECS 36A.Additionally, in the next subsection we show the results for students who followed our advice to take ECS 32A vs. those who did not.

Placement Exam Score and Upper-Division ECS Performance
• Is there a statistically significant correlation in upper-division ECS performance for students based upon placement exam score?• Is there a statistically significant difference in upper-division ECS performance for students based upon placement exam score of 14 and below vs. 15 and above?Given that there was strong correlation with placement exam score and prior experience, we wanted to know if there was a correlation with placement exam score and upper-division ECS course performance.We found that there was a PCC of 0.129 meaning there is a small effect size.We also compared the difference between those who scored 14 or below vs those who scored 15 and above by using a t-test and found that there is a statistically significant difference in performance (p-value ≪ 0.01).We found that the upper-division ECS GPA of students who did not pass the placement exam was  1 shows the upper-division ECS grade distribution for students who scored 14 or below vs those who scored 15 and above.The boxes show quartiles with the error bars going to the min/max and the X shows the mean.The numbers in parenthesis underneath the score are the number of samples in each group.
• Is there a statistically significant difference in upper-division ECS performance for students who did not pass the placement exam and took ECS 32A vs. those who ignored advice?Though the vast majority of students followed the advice based upon placement exam performance, not all students who scored low on the placement exam took ECS 32A.We used a t-test to determine if there was a difference in upper-division ECS GPA for students who followed the advice of taking ECS 32A vs. those who did not follow the advice.We found that there is a statistically significant difference in performance (p-value ≪ 0.01) with the those who took ECS 32A having an upper-division ECS GPA of 3.32 vs. 2.42 for those who did not.

Prior Experience and Upper-Division ECS Performance
• Is there a statistically significant correlation in upper-division ECS performance for students based self-reported prior experience in months?• Is there a statistically significant difference in upper-division ECS performance for students based upon self-reported prior experience of six months or less vs. more than six months?• Is there a statistically significant correlation in upper-division ECS performance for students based upon self-reported largest program?• Is there a statistically significant difference in upper-division ECS performance for students based upon self-reported largest program of less than 400 lines of code vs. more than 400 lines of code?The results of correlating prior experience with placement exam score, along with correlation of placement exam score with upperdivision ECS GPA led us to investigate if there was a statistically significant correlation with prior experience in months or prior largest program written with upper-division ECS GPA.There was a statistically significant correlation between prior experience with both months of experience and largest program written with both  Both have a small effect size but were statistically significant.We compared the difference between six months or less vs. more than six of prior experience and also between largest program of less than 400 lines of code vs. more than 400 lines of code by using a t-test.Not surprisingly we found that both were statistically significant (p-value ≪ 0.01).We found that the upper-division ECS GPA of students who reported having less than six months of experience was 3.42 vs. 3.54 for those who reported six months or more (the 99% CI was [0.065, 0.183]).As for the students reporting having written less than 400 lines of code their upper-division ECS GPA was 3.44 vs. 3.62 for those who reported having written 400 or more lines of code in their largest program (the 99% CI was [0.120, 0.228]).Table 1 summarizes the upper-division ECS GPA and t-test results.Figure 2 and Figure 3 show the upper-division ECS grade distributions for students based upon prior experience and largest prior program written respectively.

Lower-Division Track and Upper-Division ECS Performance
• Is there a statistically significant difference in upper-division ECS performance for students who took ECS 32 + ECS 34 path vs. ECS 36 path?One of our major concerns was that we did not want students taking the ECS 32 + ECS 34 path to be less prepared for upper-division ECS courses than those who took the ECS 36 path.We used a t-test to determine if there was a statistically significant difference between the two tracks.We found that there was a statistically significant difference (p-value ≪ 0.01) with students in the ECS 32 + ECS 34 path having an upper-division ECS GPA of 3.30 vs. 3.47 for those who took the ECS 36 path; the 99% CI was [0.099, 0.231].Figure 4 shows the upper-division ECS grade distribution by the terminal lower-division course (ECS 34 or ECS 36C).
• Is there a statistically significant difference in upper-division ECS performance for students who took ECS 32 + ECS 34 path vs. ECS 36 path who all started in ECS 32A?While this was concerning at first, we decided to compare students who all had the same starting point.Since many of the students in the ECS 36 path may have passed the placement exam and started in ECS 36A we wanted to separate them from those who started in ECS 32A.We used a t-test to determine if there was a statistical significance and found that there was not (p-value = 0.96); both the ECS 32 + ECS 34 and ECS 36 paths for students who started in ECS 32A resulted in an upper-division ECS GPA of 3.32.
• Is there a statistically significant difference in upper-division ECS performance for students who change into either the LCSI or ECSE major and took ECS 32 + ECS 34 path vs. ECS 36 path and started in ECS 32A?The ECS 32 + ECS 34 track was designed for students to change into either the ECSE or LCSI major, so we wanted to determine if there was a statistically significant difference between the two paths for change of major students.We used a t-test and found that there was not a difference (p-value = 0.79); the upper-division ECS GPA was 3.35 and 3.34 for the ECS 32 + ECS 34 and ECS 36 paths respectively.
• Is there a statistically significant difference in upper-division ECS performance for students who took ECS 32A and those who did not?Given that students performed the same regardless of lowerdivision track if they started in ECS 32A, we wanted to see if the difference was mainly driven by the start course which in turn is driven by the prior programming experience.We used a t-test to determine if there was a statistically significant difference between those who started in ECS 32A and those who did not.We found that there was a statistically significant difference (p-value ≪ 0.01) with students who started in ECS 32A with an upper-division ECS GPA of 3.24 vs. 3.38 for those who did not start in ECS 32A; the 99%

DISCUSSION
The design of the non-major track with the ECS 32 courses has successfully separated the ECSE and LCSI majors from the non-majors.
The ECS 34 course provides a mechanism for the non-majors to change into the ECSE or LCSI major.Our initial concern that the non-major track may lead to underprepared students in our upperdivision ECS courses does not appear to have come to fruition.While students who complete the ECS 36 path perform better than those who took the ECS 32 + ECS 34 path, this is due to the difference in prior programming experience and not inherent in the structure of the courses.Given that we did not observe a performance difference in upper-division ECS courses for those who started in ECS 32A, either track of courses may be a valid solution.
A sequence of two courses in Python followed by further in-depth implementation courses in C, C++, and Linux appears to equally prepare students for upper-division ECS courses as a single introductory course in Python followed by three in-depth courses on C, C++, and Linux.The placement exam was easy to implement and administer, and has served to advise students as to which of the two courses to start in (ECS 32A or ECS 36A).The placement exam scores are correlated with self-reported prior programming experience which appears to have a statistically significant small effect on upper-division ECS course performance.Our results appear to be in line with Decker et al. [4] who showed that they were able to place students in a normal or accelerated course sequence using a primarily self-reported survey.Fortunately, the majority of students took ECS 32A when advised to; however, our results show that those who were advised to and did not take ECS 32A have an upper-division ECS GPA of nearly a letter grade lower (0.9 GPA delta) than those who followed the advice.Furthermore, our results indicate that placing students by initial ability into different courses has the advantage of preparing those without prior experience while not delaying those with prior experience in their path toward graduation.

LIMITATIONS
There are several limitations to this work.Primarily the development and deployment of the dual track with placement exam was only conducted at a single institution with 10-week terms instead of more common 15-week semesters.Additionally, in the middle of the study we were forced online due to the COVID-19 pandemic.Finally, a direct comparison of pre and post dual track is unfortunately not possible.We did not deploy the placement exam for placement in ECS 10 and ECS 30 (predecessors to ECS 32A and ECS 36A), nor do we have data about the advice students received prior to enrollment.Furthermore, having analyzed 30 years of our university's course grades, the trends have shown consistent inflation, so any analysis of upper-division grades of a pre/post event would likely show improvement in performance.While each institution will always have its own idiosyncrasies, the difference in term length 10-week vs. 15-week may be a limitation of this approach from being directly implemented at other institutions.The logistics of reorganizing the courses when they are 15-weeks in length may be infeasible.The most concerning limitation of this work is the sudden transition to online instruction due to COVID-19.We saw a dramatic increase in course grades during online instruction which had the potential to hide differences between the different groups; however, this work covers five years where four of the normal 10-week terms were instructed online, nine terms were instructed in-person, and two terms were instructed hybrid (majority in-person with online).

CONCLUSIONS
Over the past five years we have administered a placement exam to advise students to take one of two introductory programming courses (ECS 32A or ECS 36A).Additionally, we have developed dual lower-division programming tracks: one for ECSE and LCSI majors, and another for non-majors.Our work shows that a simple non-programming language specific placement exam can effectively guide students into the correct course.The performance on the exam can also serve as a weak predictor of upper-division ECS course performance; however, it is most likely that the placement exam score is a proxy for prior programming experience which appears to have an impact even throughout the upper-division courses.We also found that students who started at the same entry level programming course (ECS 32A) performed the same in upperdivision ECS courses regardless of lower-division programming track taken.If this result holds at other institutions, it may be strong evidence that more courses at a higher-level programming language like Python may not be detrimental to students' performance later.
Additionally, it appears that a single additional 10-week introductory programming course (ECS 32A) is not sufficient to bridge the gap of multiple months of prior programming experience.Considering that the COVID-19 pandemic shifted the courses to online, we would like to continue collecting data for several more years of in-person instruction.Additionally, our department has instituted measures to require students changing into ECSE or LCSI majors to take the ECS 32 + ECS 34 path.We plan to see if differences arise between the change of majors vs. those admitted into the major.

ACKNOWLEDGMENTS
We would like to thank Adrienne Decker for her valuable feedback, and to thank the hundreds of unnamed students that have come through our programs.
A APPENDIX Available Operators (in order from highest precedence to lowest.) • Scope: () (expressions within parentheses must be evaluated before anything else in a statement.)

Figure 1 :
Figure 1: Upper-Division ECS Grade Distribution by Placement Exam Score

Figure 2 :
Figure 2: Upper-Division ECS Grade Distribution by Prior Programming Experience

Figure 3 :
Figure 3: Upper-Division ECS Grade Distribution by Largest Program

Figure 4 :
Figure 4: Upper-Division ECS Grade Distribution by Terminal Lower-Division Course CourseUD ECS Grade Distribution by Start Course

Figure 5 :
Figure 5: Upper-Division ECS Grade Distribution by Start Course

Figure 6 :
Figure 6: Placement Exam Language Description

Figure 7 :
Figure 7: Placement Exam Example Problem

Table 1 :
Upper-Division ECS GPA and t-test Results < (less than), > (greater than), = (equal to), <= (not equal to) >= (greater than or equal to), <= (less than or equal to) • Boolean Logic: and (logical and.Both sides must be true for the expression to be true.)or (logical or.The expression is true if either or both sides are true.)• Assignment: := (sets the value of a variable on the left to the value on the right).• Print(expression): display expression to the screen Control Structures • if Boolean_expression then Statement or group of statements • if Boolean_expression then Statement or group of statements else Statement or group of statements • while Boolean_expression do Statement or group of statements