Small Steps, Big Progress: Analyzing District Led Goals to Advance CS Education

The demand to provide high-quality computer science (CS) education to K-12 students across the United States continues to grow due to societal transformations driven by AI and cybersecurity. However, the impact of state initiatives and mandates on district leaders' decision making remains an under-explored area in the literature. In 2022, CSforALL began work in Tennessee, a state poised to enact CS education policy, as part of a Research Practice Partnership (RPP). This study investigates the first eight school districts who participated in theStrategic CSforALL Resource and Implementation Planning Tool (SCRIPT) workshops in 2022 and 2023, setting goals based on the SCRIPT rubric. The study takes a general qualitative approach underpinned by the Capacity, Access, Participation, and Experience (CAPE) Framework to develop a coding scheme analyzing the districts' related rubric scores and goals, and to investigate the impacts on equity indicators. The districts participated in three SCRIPT workshops held in 2022 and 2023, and this study dives deeply into the initial goals as well as analyzing the ways the SCRIPT rubric aligned to the CAPE Framework to investigate how district leaders make decisions which impact teacher and student outcomes which lead to equitable high-quality CS education.


INTRODUCTION
There is a growing recognition that teacher and student outcomes in computer science (CS) are dependent upon the policy and implementation environments which they inhabit.State level policy can have large implications for broadening participation in CS [2], especially for smaller or rural school systems who are often lagging behind in the implementation of state policies related to CS [27].State level policy is experienced at the district and school levels, and the local education agencies (LEAs) responsible for the translation of state policy into local policy and implementation can have large impacts on the equitable day to day experience of students.In 2018, Google supported a survey report on K12 CS education [4].The report specifically highlights "implementation considerations" and "systemic obstacles" as key components of equitable implementation at scale.
In this paper we describe work from the SCRIPT Crew Tennessee Research Practice Partnership (RPP) focused on supporting almost half of the school districts in Tennessee as district leaders work to adjust to new regulations regarding CS education.The RPP is focused on providing supports to district leaders while exploring a research question evaluating how goals and decisions set at the SCRIPT workshop lead to action taken and expanded equitable opportunities to capacity, access, participation, and experience in computer science education.In this paper, we present an analysis of the three-months goals set by eight public districts in the state who received support through SCRIPT in the fall of 2022 and early 2023.We analyzed the three-month goals as indicators of the decisions made during the workshops, and compared the sub-components on the SCRIPT rubric in which goals were set to the areas of selfperceived strength or weakness as indicated by the rubric scores LEAs self-assigned.We also aligned the goals set to the CAPE framework.This contributes to understanding how school leaders make the kinds of decisions for computer science education that they do.

LITERATURE REVIEW
Computer science (CS) education exists within institutional contexts, and this project focuses on the decisions made by a leadership team in order to enact institutional change that enables the offering of CS classes in K-12 environments, and supports equity enabling practices within those courses.The project uses a framework named CAPE [14] in order to disaggregate different components of assessing and evaluating equity within educational institutions.
The acronym stands for Capacity, Access, Participation, and Experience, and the framework was originally developed to understand how CS education was distributed within a state context.[13] This framework calls out the foundational nature of capacity-building that education leaders must plan for and undertake, before the supports and context are in place for students to have equitable access to CS learning, so student participation in CS learning can equitably reflect the diversity in an LEA, and so that ALL students engaged in CS learning can have a high quality experience.The many specific kinds of capacity that a school district or other LEA would need to have to carry out equitable CS plans for all their students have been explored further by CS education researchers [19].Prior work also calls attention to the way in which states respond to standard implementation while developing equitable pathways for computing competencies across grade levels [5].
Many of these explicit kinds of capacity building extend beyond teacher professional development.Within the rubric used in SCRIPT there are many facets of necessary capacity for CS education in an LEA that are made explicit, these include readiness and involvement of district-level administrators, connections and communication with families, training and involvement of school counselors and other instructional staff, and readiness of physical assets such as devices, in school internet connectivity, availability of internet connectivity in homes, and software, just to name a few.[9] Prior work has shown the importance of working with administrators to see the bigger picture of capacity and ensure reinforcing structures for equity-aligned practices [12].Similarly, a study conducted by Fancsali et al. [11] emphasized the importance of teachers partnering with administrators to prioritize CS course offerings, PD opportunities for practitioners, and how to scale the implementation of a large-district initiative to avoid inequities.Internationally, studies have also recognized the importance of considering the multifaceted system and policy makers and deciders who are responsible for implementing CS education.[15] CS education implementation requires decision making at a variety of levels, as highlighted by previous research [15], however those decision makers are not always aligned in their understanding of policy intent or how to implement [23].While many studies and technical reports focus on the policy choices made at the state level [2] and its impact on student access to and participation in CS [6], little work has been done to study the priorities of administrators when making decisions about how to build capacity within their schools or districts.
The need to engage administrators and decision makers has been identified in many surveys of educators, including CSTA's teacher survey most recently in 2022 [8].This echos decades of education research on curriculum reform from math and ELA [3], as well as science education [16].In this work we explore the types of decisions made by teams of administrators and teachers after engaging with a scaffolded process to consider creating coherent plans for CS education.

METHODOLOGY
This section describes the methods in which we recruited participants, the profile of participating local education agencies (LEAs) and their team representatives, the collection of data from the teams, and the analysis of the data.

Research Practice Partnership
At the onset of the project in 2021, Tennessee was poised to enact CS education implementation policies.In April 2020, the state legislature approved a plan for CS education implementation.The state also approved significant funds for CS education including teacher professional development starting in fiscal year 2021.In 2019-2020, only 50% of high schools in Tennessee offered CS courses, and these schools were much more likely to be in suburban communities, while city, town, and rural communities lagged behind in access.Data from the state showed that even in the places where CS is offered, women are much less likely to take CS, especially Advanced Placement CS, than their male peers [6].All of these factors point to a state ready to begin a push to implement CS.In partnership with researchers at CSforALL, local practitioners in the state determined a core problem of practice: School districts in state need support to prepare for and increase capacity to achieve equitable and high quality CS education, which is standards-complete and culturally relevant.
In May of 2022, the governor of Tennessee signed Chapter 979 of the Public Acts of 2022[1], setting in place a policy environment that would require all high school students graduating in the class of 2028 to earn a Computer Science course credit as a graduation requirement.Additionally, starting in the school year 2024-2025, all middle school students would be required to take at least one marking period of computer science at some point during middle school, and all elementary school students would receive gradeappropriate CS content embedded into the instruction they receive.
This moment marked a monumental shift in state educational policy and the decision-making environment of school and district leaders in Tennessee [23].The research practice partnership (RPP) began reaching out to school districts in the first half of 2022 to offer a supported strategic planning workshop, alongside a small implementation stipend for the district ($500).The RPP is comprised of a state-based nonprofit organization, Code Crew, committed to mentoring historically marginalized youth to become tech innovators and leaders through practical, hands-on computer science education programs, as well as two national non-profits focused on K12 CS education, CSforALL and CSedresearch.org.
Through the advocacy efforts of multiple partners, the state policy lays the groundwork for universal access and participation in CS education, through required course offerings.This policy creates an environment where local decision makers, school district leaders, will need to make changes to courses offered and who is enrolled to comply with the new requirements.In order to support the local change, the partnership identified planning tools that could be used by school leaders to consider how to implement CS education.The partnership leveraged the Strategic CSforALL Resource and Implementation Planning Tool (SCRIPT) workshops supporting teams of administrators and teachers from districts and area education agencies.This program was designed by CSforALL and is available through a national network of trained facilitators since 2017 [9], reaching more than 750 school districts and other LEAs across the United States.The SCRIPT program creates opportunities for team visioning, allowing local education leaders to identify values for teaching their community youth computer science [24].SCRIPT then introduces explanatory rubrics to encourage district-team selfreflection and provide guideposts for the many forms of capacity a district would need to equitably implement a K-12 CS Pathway of learning for all of their students [9,10].Finally, SCRIPT requires district teams to set three month, six month, and long-term goals to bring their CS Vision to life in each of six capacity areas mentioned within the Rubric: 1. Materials and Curriculum, 2. Leadership, 3. Teacher Capacity and Development, 4. Partners, 5. Community, and 6.Technology Infrastructure [28].

Participants
Code Crew used the SCRIPT [9] framework to advertise, host, and evaluate all workshops referenced in this study.Code Crew attended CSforALL hosted SCRIPT facilitator training sessions, and engaged with the CSforALL SCRIPT program team throughout the process in order to ensure adherence to SCRIPT norms and delivery expectations.Districts were recruited beginning in November 2021 throughout May 2022 by the local non-profit through calling contacts and forming relationships.The strategy on the ground was to target STEM-innovation hubs to disseminate information and recruit participants.A trusted voice within this network was able to recruit the first six-districts and subsequently two more districts joined after meeting with an experienced superintendent with a strong commitment to CS education.
The first six school districts attended a workshop in November of 2022 where they set goals.Two additional districts attended workshops in February and May of 2023 and set goals.The goals from all eight districts are analyzed in this paper.Districts also participated in follow up with their facilitator at 3 and 6 month post-intervals to review goal progress and set new intentions.For this paper we analyzed the initial goals set by district teams.Each district was asked to identify participants based on guidance from the SCRIPT team at CSforALL and Table 1 shows the relative distribution of district team members for the workshop.
Table 2 represents the features of each district who participated in the SCRIPT workshops.The data was retrieved from the National Center for Educational Statistics (NCES) School District Search System.As noted in Table 2, the eight districts were primarily small, rural areas, with only two districts with more than ten schools.The demographics of all eight districts suggest they are predominantly white (all above 50%), and even in the smallest school district, with only one school (LEA 06), 64.9% of homes have access to the internet.

Rubric and Goal Setting Process
As a part of the workshop, the teams were facilitated through a process of collaboratively self-rating against each of the SCRIPT Rubrics, and setting short term goals for improving their rubric score.Each team was provided a brief introduction to the section, and then up to an hour to work as a district team to read the rubric and select a rating for each subcomponent.Teams were also asked to set 3-month, 6-month, and 9-month goals for each rubric area.The workshop facilitators encouraged the district team members to construct (SMARTIE) specific, measurable, action-oriented, relevant, time-bound, inclusive, and equitable goals [17].Participants were shown the SMARTIE framework as well as several examples of SMARTIE goals as a part of the introduction to goal setting in order to encourage the use of the framework in team discourse.For example, the teacher capacity and development rubric has 4 subcomponents.For each subcomponent, teams would rate themselves as novice (1), emerging (2), developing (3), or highly developed (4).A team would read and self-rate on each of the subcomponents, and then create at least 1 goal for each time period (3, 6, and 9 months) for the area of teacher capacity and development.Some teams complete the full rubric before writing goals, while others go back and forth between setting goals and self rating.
For this paper, we reviewed the 3-month goals set by the districts as an important look at short term approaches to CS education advancement.A general qualitative approach [20,21,25] informed the mode of analysis as our primary research question for this data set focused on evaluating how strategic planning using SCRIPT materials, routines, and data tools impact buy-in, decision making, and actions taken at the school district level in the state.

Codebook Development
After extracting the 3-month goals submitted from the eight participating school districts, two raters engaged in a general qualitative approach to data analysis [7].The raters constructed a codebook using a collaborative spreadsheet with all district information de-identified.The spreadsheet contained four main columns: randomized District identifications, notations, excerpts for each of the 3-month goals, and the related rubric score the district scored themselves.The top six horizontal rows, however, served as both a theoretical and categorical framework for coding the data: Rows 1-4 aligned to the high-level explanations of CAPE framework [14] as follows: (C) Capacity, A systemic change or change in resources available that provided support for or implementation of CS education; (A) Access, A framing by which administration and leaders can evaluate if courses, professional development (PD) experiences, materials, tools, are offered in low-income schools at the same rate at other schools; (P) Participation, An equitable way to analyze student enrollment in CS coursework.District teams focus on which subgroups are underrepresented in CS courses and to what extent; as well as number of students and families engaged in school-based extracurricular CS activities; (E) Experiences, A focus on authentic and interest driven CS learning which creates a feeling of belonging across student subgroups.Rows 5-6 contained the SCRIPT Rubric categories (6) and sub-components (30) respectively.

Codebook Scoring
The total number of goals placed in the codebook for analysis was n = 67.Two goals, however, were omitted from the final analysis due to the quality of goal.An example of an excluded goal is as follows: "Create an awareness of CS expectations."Both raters agreed the goal was too general and did not provide a clear indication of rubric category or sub-component.The district team members could have intended to create an awareness for Teacher Capacity, Leadership, or Community.Therefore, the final number of goals analyzed was n = 65.The process of analyzing the data consisted of simultaneous coding [20]; [22] in which the raters determined whether each of the district goals aligned to the CAPE framework as well as which of the 30 sub-components the goal corresponded to.The raters assigned a corresponding letter indicating whether the goal either intended to build capacity, provide equitable access, ensure increasing participation of students and families, or high quality experiences driving increased engagement in CS content leading to multiple pathways in CS education were available.An example goal, related rubric score, and rubric sub-component categorization is as follows: "Raise awareness of CS education requirements across the district at all grade levels.Begin to design plans with school level input.(Emerging) (Planning)."The raters then assigned this goal a "C" as the distinction of raising awareness indicated the district's desire to build capacity within their leadership and plan for requirements.One rater noted this goal aligned with the need to comply with state mandates.

ANALYSIS
The approach in which the researchers operationalized the data analysis protocol included considerations for rigor, sincerity, credibility, significant contribution, and meaningful coherence [26].As mentioned previously, the data gathered is underpinned by the theoretical framework of CAPE [14].The CSforALL researchers participated in bi-weekly meetings with the larger Research Practice Partnership to discuss reflections, challenges, successes, and plans from 8 districts' SCRIPT workshops; indicating reflexivity and transparency throughout the data collection and analysis processes.
In reviewing the codebook and data for this study, the two raters sought to establish inter-rater reliability, a crucial component of credibility and triangulation.When establishing inter-rater reliability, 14 goals were chosen randomly and scored together with both raters for norming.The remaining goals were then categorized and reviewed independently.Cohen's  was calculated to determine if there was agreement between the two raters.There was statistically significant agreement between the two raters,  = .63.
In a previous goal analysis [28] of the first 1,023 goals school districts set at SCRIPT workshops, future implications suggested the need to examine "which rubric areas do districts tackle, are those rubric areas strengths or weaknesses, and do goals increase rubric scores" (p.489).This study expanded upon the aforementioned contribution by looking closely at the type of goals set at baseline (3-month) during the initial SCRIPT workshop, how do the districts rate themselves in a given rubric sub-component, and if the goal-to-sub-component categorization aligns to equitable capacity, increased access, increased participation, and experience driven computer science education outcomes for students.Figure 1 below

CAPE Score Frequency and Emerging Themes
The findings from this analysis provided a data story in which district teams prioritized specific elements in their goal and strategic planning to increase capacity for positive CS education implementation.Coupled with the new state-level mandate, the leadership teams highly prioritized building capacity to raise awareness about new CS initiatives to students and families in order to achieve buyin.An additional significant finding are the teams' prioritization of high-level planning within the leadership team, accounting for teacher professional development (PD) needs, and building local partnerships.Previous research [27] indicated barriers to scaling and increasing access and participation to equitable CS education in rural areas, similar in demographics to the districts in this study, lack resources to offer PD.While early in the implementation stage, the emphasis on teacher preparation, PD, and endorsement, increases the likelihood more CS coursework will be offered in these rural districts.
As displayed in the Heat Map in Figure 1, the five most frequently scored Rubric subcomponents were Planning ( n = 7), Initial Teacher PD (n = 6), Local Partners (n = 4), Families ( n = 12), and Area Schools & Educational Institutions (n =4 ).Overwhelmingly, the most commonly used CAPE rating for each goal in these areas was "C" or Capacity.
Seven of the eight district teams who participated in the SCRIPT workshops are classified as either rural distant, rural remote, or rural fringe.One district, however, is classified as a small city.These distinctions, as determined by the National Center for Educational Statistics, are important to note and present a brief analysis of the perceived differences in how themes emerged and goals were set.The greatest difference amongst these districts as noted in Table 2 resides in the population of students, teachers, and number of students.Yet, by-and-large, the districts set goals corresponding to the same subcomponents as noted above.This alignment may reflect the early nature of the work, as the average rubric score for any of the district teams was less than 2. A rubric score of 1 (or Novice) indicates little happening in the district, and therefore

Related Rubric Scores and Goal Correlation
Each district set between five and 10 3-month goals related to advancing computer science education during their SCRIPT workshops.The raters evaluated the goals in relation to the Rubric sub-component areas and the LEA self-assessed scores.The raters then calculated an overall mean score as provided by the districts on a the rubric, a mean for goals set, as well as a mean for subcomponents where no goal was set to understand how the LEAs might perceive themselves across Rubric areas.Table 3 displays these averages by district.A trend based on the scores in LEA01 (M= 1.62, SD= 0.61), LEA06 (M= 1.49, SD= 0.66), and LEA07 (M= 1.79, SD= 0.61) suggests the districts are setting goals in areas of perceived relative weakness.In contrast, LEA02 (M= 1.83, SD= 0.85), and LEA05 (M= 1.96, SD= 0.76) set goals in areas of perceived relative strength.Further research is needed to determine if there are particular benefits to districts if they focus on leveraging assets (strength based approaches) or building capacity focused on areas of need (perceived weakness) for long term implementation progress.

LIMITATIONS
While there is a precedent for analyzing the goals set during SCRIPT workshops [28], the recent initiative, coupled with mandates in the state analyzed in this study, created a rich environment for developing understanding of how strategic planning using SCRIPT materials, routines, and data tools impact buy-in, decision making, and actions taken at the school district level.As part of an RPP, this analysis is only a small subset of data collected from November 2022, February 2023, and May 2023 SCRIPT workshops.Additionally, district teams are only required to report their related rubric scores which correspond to goals set at their initial SCRIPT workshop.

IMPLICATIONS
The national CS education community increasingly impacts efforts at encouraging adoption of state-level CS graduation and certification mandates, as well as provision of important supports and materials for teachers and schools, and for good reason [23].The findings of this study indicate an understudied crossroads in which the demands and resources placed in front of education decisionmakers are translated into an achievable CS plan that will be felt by educators and students.The CS education research community needs to better understand the ways education leaders prioritize limited local resources of attention, human resources, and good will to set strategic first steps for realizing mandates and utilizing available supports.Most importantly, a trend emerged from this study in Tennessee that the utilization of strategic planning using SCRIPT materials, routines, and tools impacted the the way in which decision-makers prioritized family and student engagement as a means to build capacity to develop high-quality CS education implementation [18].

FUTURE WORK
The RPP team continues to analyze the goals from the November 2022, February 2023, and May 2023 SCRIPT workshops in Tennessee.Important next steps include a comparison of the 3-6-9 month goals set using the same methodology described in this paper: alignment of the district goals to related rubric scores to Rubric sub-component area and scored as C, A, P, or E to indicate equitable capacity, access, participation, or experiences to improve outcomes for students in CS education.Additionally, districts complete delayed post-workshop surveys.An in depth review of LEA actions taken based on the survey responses in comparison to state data from the NCES would lend to future support in implementation efforts.Finally, a deeper analysis linking CAPE to the SCRIPT rubric and workshop can provide a rich narrative for understanding how goals are written and achieved as well as add to the story of equitable high quality K-12 CS education.

Figure 1 :
Figure 1: Distribution of Goals Mapped to CAPE

Table 1 :
District Participant Profiles District ID # of District # of Teacher # of Principal # of Counselor # of CS Instructional # Total Team

Table 2 :
Districts Participating in Workshop Demographics

Table 3 :
Mean Scores Rubric & Goal Correlation focus on early capacity is important.A rubric score of 2 (emerging) requires teachers to already by doing some instruction and so capacity building may shift to other areas.