Investigating Demographics and Motivation in Engineering Education Using Radio and Phone-Based Educational Technologies

Despite the best intentions to support equity with educational technologies, they often lead to a “rich get richer” effect, in which communities of more advantaged learners gain greater benefit from these solutions. Effective design of these technologies necessitates a deeper understanding of learners in understudied contexts and their motivations to pursue an education. Consequently, we studied a 15-week remote course launched in 2021 with 17,896 learners that provided engineering education through a radio and phone-based system aimed for use in rural settings within Northern Uganda. We address shifts in learners’ motivations for course participation and investigate the impact of demographic features and motivations of students on persistence and performance. We found significant increases in student motivation to learn more about and pursue STEM. Importantly, the course was most successful for learners in demographics who typically experience fewer educational opportunities, showing promise for such technologies to close opportunity gaps.


INTRODUCTION
Education is an axiomatic right for all individuals.Nevertheless, inequality in educational opportunities still prevents millions of people from receiving the education they deserve.While online educational systems, such as MOOCs, aim to encourage social mobility in learning, studies have discovered signifcant barriers that prevent underserved learners from participating in these systems [16].Consequently, there is still a need to employ more accessible and feasible learning experiences for these communities of learners, particularly those in low and middle-income countries [42].
Educational inequality arises due to numerous obstacles within rural living contexts that many learners from low and middleincome countries are routinely confronted with, preventing them from receiving even basic educational experiences [38,49].Some of these barriers include long and hazardous traveling distances to the nearest school, lack of transportation, and hefty school tuition fees [25].Learners may experience educational disparities, which include the shortage of skilled instructors, school infrastructure, and overall educational support, which reduce the opportunities available to them [26,49].In addition, as subsistence farming is a primary source of income for many families in rural regions of lowincome countries, there is an unavoidable need for children's labor at their homes, which can deprioritize school attendance [13,38,49].Furthermore, pressure on women and girls to stay at home to fulfll household duties can lead to disproportionate female attendance at schools [30,49].
To address these various impediments, recent work in educational technologies has sought solutions in mobile education for rural learners in low-infrastructure communities.Mobile education or "m-learning" provides any individual with educational experiences through electronic and technological means not fxed in location or time [9,45,50].Due to the increasing use and dissemination of mobile learning technologies, which can vary in platform from phones to radio to tablets, there is an increased focus on understanding the impact of such approaches to helping learners advance their education [45].Despite their decades of deployment in low infrastructure contexts, we have yet to interpret the applicability of mobile learning within underserved communities where basic education and technological infrastructure are exceedingly scarce, especially in understanding how these technologies increase education access for various learners in specifc learning contexts [5].
To address these limitations, we study an existing virtual course launched in 2021 that delivers practical education on engineering through basic interactive technologies (radio broadcasts and basic keypad phones) for learners within remote and rural contexts in Northern Uganda.Radio and basic phones are two of the most commonly used and possessed technologies in Ugandan households [37] and do not require internet connectivity or advanced technical skills to learn from a virtual course.As this remote engineering educational approach minimizes physical and fnancial barriers, we aim to understand better how to provide equitable access to high-quality engineering education for all Ugandan learners.
In this work, we analyzed log interaction data from an existing course.We frst investigated the experience in this virtual course across all participating learners.In particular, we examine the association between the course and changes in learners' • Motivations to pursue occupations and learning opportunities in STEM • Engineering-based mindset, and • Income mobility.
Subsequently, we performed an in-depth study into the features that afect these changes, further discussing learner outcomes: • Course completion/persistence • Performance on the fnal exam in relation to the following three factors: • Initial motivation for partaking in the course, • Various demographic features of the learners, and • Access to technology Through this study, we found signifcant changes in student motivation and their engineering-based mindset throughout the course.There was a general movement in student motivation to learn more about science and technology and positive perspectives on the values of an engineering-based mindset.Students were also more likely to say they would pursue STEM-related careers and educational opportunities by the end of the course.We also found a rise in income mobility for students during the course, in which there was a signifcant shift in receiving higher weekly incomes by the end of the course.We found gender, family, urbanicity, educational experience, and access to the internet to be infuential features on course completion, with the course being most successful for learners in demographics that typically experience less educational opportunity.Moreover, we identifed prior educational experience, employment, and student motivations in taking this course as impactful features on student course performance.

RELATED WORK
Due to growing recognition of the need to support education in low-infrastructure communities, there has been a surge in research directed towards understanding how mobile devices can ofer new learning opportunities.This section frst discusses successful deployments of diverse, innovative mobile learning technologies across low infrastructure communities that inform our study.To understand how distinctive student demographic groups and diferent learning goals are mobilized and motivated in distance education, we also review studies conducted on the efect of Massive Open Online Courses (MOOCs) on these students' learning.While our work centered on a virtual interactive course that employs radioand phone-based platforms, previous studies on MOOCs show the efects of general distance education on the learning of students with diverse backgrounds and motivations.

Successful Deployments of Basic Mobile Educational Technologies
Previous studies deployed various mobile educational technologies to provide diverse educational opportunities for instructors and in-school students in low infrastructure settings, which include remote co-teaching and instructor-oriented technological interventions [21,52].Hence, many of these applications are created for students to learn in varied settings, particularly within school environments and remote settings in their homes.While tablet-based educational applications are a common theme [41,49], other studies present potential educational technology alternatives for virtual or distance learning in low-infrastructure communities [50].For instance, there have been studies conducted on the afordances of mobile learning in other learning contexts within the global south [28,31,35,55].As teacher strikes, political unrest, natural crises, and overall lack of basic resources have continually prevented school attendance in some contexts, it has been hypothesized that phone-based education can supplement both formal and informal schooling [28,46].For instance, Kizilcec et al. [28] studied the impacts of a text message-based application that delivers educational lesson content and quizzes to Kenyan learners and a voice-based system that provides students with voice-recorded lessons, quizzes, and feedback to improve literacy skills of learners in Côte d'Ivoire.They found that learners showed perseverance in using these phone-based technologies to supplement their formal learning in moments of disruption due to two factors: afordability of text and voice-based messaging and trustworthiness in using familiar messaging-based platforms in Kenya and Côte d'Ivoire [28].Hence, phone-based learning applications show promise to supply important opportunities for students to continue learning through various disruptions [28,50].These fndings demonstrate that adaptable mobile education has some potential to support autonomous learning regardless of location and time.Prior work also suggests that learners are more likely to adapt to learning from lower educational technologies rather than complex technological advances.This is implied in another study by Poon et al. [40], where they deployed a quiz-based intervention through SMS and WhatsApp to help Cameroonian learners improve in exam practice.
The older technology of radios as a learning tool has been extensively studied as a low-cost educational technology with a lengthy historical background in positively impacting the educational experiences of low-infrastructure communities [7].This positive impact is evidenced by the infuence of Interactive Radio Instruction (IRI), a distance education system that merges radio broadcasts and active learning to ameliorate the quality of education and instructional practices [47].The impact of IRI can be seen in numerous low infrastructure settings, including regions in Nigeria and Bangladesh where radio educational systems have fourished and shown positive impacts on learning outcomes [27,36].While IRI is typically used as a support tool for teachers who provide the "interactive" component, innovative approaches in IRI have delivered open and distance learning to pastoral nomads who may not have access to formal schooling [36].Due to prior successful employment of phone-based and radio-based educational systems, our work expected the integration of these two educational technologies to be a viable learning opportunity for Ugandan learners in remote settings.
Finally, interactive audio on basic phones has also been studied as a modality for learning.Moloo et al. [33] described an "audio MOOC, " promoting distance education through lower-end mobile phones to low-literate populations who face digital inequalities due to lack of internet access, connectivity, cultural diferences, etc.This is an interactive course in which audio course content was delivered to mobile phones, and students responded to related activities by phoning in their responses through a system-embedded framework using voice over internet protocol (VoIP) and interactive voice response (IVR) [33].Similar VoIP-based and IVR-based approaches, including AlloAlphabet [28], Learning through Interactive Voice Educational Systems (LIVES), and Capacity Plus, a mobile learning system on family planning implemented in Senegal [17,53].These systems generally work through phone calls, recorded lesson questions, and related context delivered to the user, in which they answer these questions with their mobile phone and can hear feedback on their correct or incorrect answers [17,53].While these systems only use mobile phones, their combined text message-and audio-based course delivery and interaction are similar to the multi-platform learning approach of our studied course.However, lengthy phone calls remain prohibitively expensive for many communities, so IVR for delivering course content must be either philanthropically sponsored or remain a research tool.
Importantly, prior research utilizing these low-tech approaches has been typically limited to efcacy studies, demonstrating that platforms such as radio can improve student learning outcomes.One reason is that student interaction data in radio education is explicitly challenging to log, leading to potential gaps in data on learner demographics, course activity, and student perspectives on these technologies.More recent combinations of radio education with phone-based technologies could allow for synchronous logging of student responses on learner demographics, course activity, and student motivations from surveys.These innovations open up new opportunities to better understand the who, what, where, and why behind these generally favorable outcomes.

Efect of MOOCs on Learning in Student Demographic and Motivational Groups
While interactive radio education has yet to experience a data revolution, our work can take inspiration from the plethora of studies on an alternative form of distance education: MOOCs.MOOCs and IRI both intend to reach learners at scale, particularly learners with low access to traditional educational opportunities (although MOOCs favor higher education while IRI tends to be at a basic level).Like IRI, many MOOCs deliver instruction at a gated yet mostly asynchronous pace.They both ofer opportunities for individual learning, yet formal and informal communities have formed around their deployment for learners to study together.Therefore, we believe that prior work on MOOCs ofers relevant insights to the current study.Over the past decade, a number of studies that focus on MOOCbased learning have investigated the multiple demographic and motivational factors that afect student enrollment and learning performance in an online course [14,23,51].This issue was of central importance due to the consistent fnding that the number of learners who begin a MOOC course tends to be dramatically larger than the number of students who complete the course (similar to earlier work on distance education in general) [11,29].Through demographic studies, researchers have made progress in determining for whom such courses will likely be successful.It has also helped researchers better understand how learner motivations impact their learning behaviors, including reframing the idea of "dropout" to include alternative defnitions of success [20].The most common set of learner demographic features prior work has investigated includes: • language profciency, and • language preference Many studies found gender to have no signifcant impact on online course completion and achievement [8,34,58].On the other hand, prior educational and online experience does have a signifcant impact; participants with higher educational and digital profciency tend to have a higher course completion rate and learning performance across many studies [34,58].One study by Guo et al. [20], who analyzed student interaction data from four distinct MOOCs conducted in 196 countries, found that learners in countries with higher student-to-teacher ratios (e.g., India, Kenya) linearly proceeded through course content.Hence, it was implied that these students were using these online courses to supplement their formal education [20].However, most students who received a certifcate of completion were from countries with lower studentto-teacher ratios (e.g., United States, European countries) [20].This diference in course completion may have occurred because students from countries with higher student-to-teacher ratios are more accustomed to an instructor-driven education [20].While educational technologies, including MOOCs, aim to be globally inclusive, disparities in learning between students in diferent geographic locations are still apparent.
A less understood demographic feature is occupation status [34,51].Some studies reported that unemployed students have a higher course completion rate [34].In contrast, other studies noted that MOOC learners typically consist of students whose job employment requires higher education, and therefore, employment is practically a prerequisite [51].These conficting results are likely the result of a diference in goals and content across MOOCs, e.g., whether they are targeted toward advanced employment-related skills or basic knowledge.However, across studies, most MOOC participants are from a higher socioeconomic status [23,39].
Language profciency and preferences for instruction are also complex features.Previous MOOC research noted diferences in the interaction with online videos between native and non-native English speakers, if not in learning outcomes [43,48].Students whose primary language is not the course language may be required to simultaneously learn course content and the language of instruction, which can widen the learning gap between native and non-native speakers [43].While language profciency and preference are known to afect course performance and engagement in MOOCs, this is still an active area of research.
Additionally, motivation for taking MOOCs matters.Student motivation ranges between extrinsic (e.g., certifcate acquisition and advancements in personal careers) and intrinsic (e.g., innate interest to learn) [4,12,24,44].MOOCs may consist of students with either or both types of motivations [54].Since students voluntarily engage in MOOCs, motivation can infuence course completion rate and learning performances of learners with difering goals and educational background [14,15,19,44].For instance, student motivation in relation to the fnancial ability to aford formal education had no signifcant infuence on course completion [14].Other studies found social motivations, such as personal recommendations from others to take a MOOC and certifcate acquisition, to be important features in predicting course performance and engagement [56,58].Previous research also found a positive correlation between students' intrinsic motivations and course participation, in which increased interaction and time spent on a MOOC is associated with higher course completion [2,22,44,57].
There is a general consensus amongst these previous studies that while MOOCs intended to reduce educational inequality by making quality educational resources available worldwide, more advantaged learners are more likely to complete and beneft from these online courses [51].While our work centralizes on a distance learning system using mobile technologies, our study investigated similar learner demographic features and student motivation and their efect on course completion and learning performance to understand how they might support more marginalized or outof-school learners.Knowing the impact of student demographics and motivational features on learning outcomes is important to understand how a distance interactive educational system can best support learning for diverse students in rural or low-infrastructure settings.

METHODOLOGY
This section discusses the course content and delivery, student recruitment, measures analyzed for data collection, and analyses used to investigate learner outcomes.We did not participate in the course design and implementation because we were given data on an existing course.

Course Content and Delivery
The content of the course we study in this work followed the steps of the engineering design process and ended with students creating an innovative product relevant to their context [32]; in this iteration of the course, the fnal product students created was a solar cell, a device that converts sunlight into electricity.The instruction encourages students to perform hands-on technological and scientifc experiments at their homes using locally available materials, and was loosely based on curricula from the Museum of Science in Boston.At the start of each week of the curriculum, students typically take a phone-based assessment on the previous week's lesson content.Students are expected to have prepared the materials and related assignments necessary for that week's lesson content.Lesson instruction and content are then delivered on specifc days of the week through radio broadcasts.On these days, students turn on their radio to the channel that relays the lesson content that an instructor of the course broadcasts.While listening to the radio content, students learn from the lesson material and follow instructions related to creating the fnal product of the course.Students are able to actively engage with the radio broadcast by calling into the station, where instructors provide questions and exercises for students to attempt.Students can complete these activities via unstructured supplementary service data (USSD) through their mobile phones, messaging their responses to the course system, or by physically calling in their responses during the radio broadcast.This allows students the option to access the activities entirely asynchronously, with the exception of the radio broadcast.The course was launched on June 28th, 2021, and ofcially ended on October 12th, 2021, but to allow learners to have time to complete all assessments, USSD responses were collected until January 5th, 2022, at which point the system was cut of for this course.We received an existing dataset from this course deployment in which all personally identifable information was removed; the IRB classifed this work as an exempt study.

Recruitment of Participants
The course was deployed in the northern region of Uganda.The radio sessions for the course were broadcast from two diferent radio stations in the region to enable greater coverage.However, the phone-based questions could be accessed by anyone who registered, thus allowing students to engage with the interactive content without listening to the radio.Learners were recruited through advertisements announced through a radio broadcast.Learners did not pay to access the course and received no compensation for participating, but they did receive a certifcate if they met certain criteria for course completion.
Students interested in registering for the course received questions via USSD to create a student profle.The registration questions asked students about their personal demographic information and other basic information on their educational background and access to technology.Any student who completed the registration was enrolled in the course, received SMS messages, and could access USSD activities throughout the duration of the course.17,950 participants were initially registered for the course of study.After fltering out test accounts and other non-student profles, we identifed 17,896 ofcial student participants who completed the registration process.

Measures
In our work, we used three distinct measures to analyze learner outcomes: baseline and endline surveys, the fnal exam of the course, and course completion.This section addresses the implementation of each measure in the course.

Baseline and Endline Surveys.
Before registered students ofcially began the course, they were required to fll out a baseline survey asking about their motivation to take a STEM education course and initial perspectives on engineering-based learning values.Students were given an isomorphic endline survey as a refection of their learning experiences in taking the course.The majority of questions on these two surveys were required to move forward in the course, with the exception of two questions about income, which could include sensitive information.Students accessed these surveys over USSD.

Final Exam.
After completing the last step of the engineering design process, students took a 10-question cumulative assessment on the engineering course content.This assessment was also delivered and completed through USSD.

Course Completion.
To determine persistence throughout the course, we defned a 'course completion' variable which required the completion of both the baseline survey and the endline survey, which was the fnal activity delivered to learners' phones.Taking the fnal exam was not required to receive access to the endline survey, even though the endline survey was delivered after the exam.Students who skipped the fnal exam but completed the endline survey were still fagged as 'completed.'

Analyses
We conducted standard Chi-square tests of independence to investigate whether the proportions of responses in the baseline survey were statistically distinct from the endline survey.Subsequently, to investigate precisely how learners changed their responses for each question within the baseline to endline surveys, we use the McNemar-Bowker symmetry test, a statistical test used for nominal symmetric data.While a Chi-Square test measures the independent relationship between two variables, the McNemar test solely observes the constancy of responses between paired categorical or nominal variables over a duration of time, typically for pre to post-test design [6,18].The McNemar test specifcally analyzes the change in responses by directly comparing the count data across two diferent nominal variables [6,18].Signifcant movement in responses is usually shown by an increased number of participants selecting one response over another response [18].Participants who persisted with the same response across paired nominal variables are not considered in the McNemar test.Rather, it is more relevant to observe if changes from one response to a diferent response are signifcant or random by observing the adjusted pvalue for each variable comparison [18].Using this analysis, we measure the degree of signifcance of the numerical change from one response to another response for each corresponding question to observe changes in student motivation, engineering-based mindset, income mobility, and outlook on pursuing STEM education or related careers in the future.
To measure the impact of student demographic and motivation features on course completion, we conducted standard Chi-square tests of independence to determine whether the proportions of students in a certain feature group who completed the baseline surveys were statistically contrasted from the endline survey.
To fnd predominant demographic and motivational features that impact student learning, we calculated student performance on the course's fnal assessment for each identifed student group related to these features.We applied a linear regression model on student response data from the fnal assessment to measure the impact of each identifed demographic and motivational feature on student performance on the fnal assessment.

FINDINGS AND RESULTS
In this section, we show the statistical signifcance of changes in student motivation, mindset, and income throughout the course and infuential demographic features on course completion and fnal exam performance.

Participation in Course Components
As in other forms of massive open courseware, a signifcant portion of learners who initially registered for the course did not complete

Change in Learners' Motivations, Mindset, and Income over Time
To understand how students generally changed over the course's duration, the system asked identical questions within the baseline and endline surveys.The curriculum had a focus on promoting positive engineering mindsets in its instruction, including the concepts of using engineering to help your community and valuing creativity and exploration.It was also intended to encourage learning of and employment in science and engineering, with the potential for learners to consider earning income from the products they create.We explore how learners changed in these mindsets over time, restricting the data frame to only students who completed the course, that is, submitted both baseline and endline responses.This included 4,119 learners unless otherwise noted.

Change in Student Motivation.
Motivations for taking this course, as listed on the baseline and endline survey, include making money, learning about science and technology, helping people, and passing an exam."None of the above" was also included to capture participants who did not agree with any of these motivations.Table 2 shows the percentage of learners who reported each motivation for taking the course at baseline and endline.A Chi-square test showed that the responses in the baseline signifcantly difered from the responses in the endline, 2 (16, N = 4,119) = 859.99,p < 0.0001.
We then conducted a McNemar-Bowker test, shown in Table 3, to investigate exactly how responses changed throughout the course.In particular, we observed a 7% increase in the number of responses for the answer choice, "Learn science/tech", from the baseline to the endline survey, which is the largest identifed increase in the proportion of responses across these surveys.Two signifcant groups of students moved to a Learn motivation: 467 students (about 46% the original Make Money cell) shifted their response to "Learn science/tech" (p < 0.0001), and 237 students shifted their response from Exam to Learn (p < 0.0001).These students are almost 40% of the learners who initially chose "Pass exam".
We also see a signifcant shift in responses from "Pass exam" to "Make money".85 students who initially chose "Pass exam", about 14% of those who chose this response in the baseline survey, shifted their response to the latter (p < 0.05) in the endline survey.
We also identifed three groups of students who shifted their response to "None of above": 32 students (approximately 3% of the initial Make Money cell) moved their response from "Make Money" to "None of above" (p < 0.05), 67 students (about 3% of the original number of responses for the Learn motivation) shifted their response from Learn to "None of the above" (p < 0.01), and 36 students changed their response from "Pass Exam" to "None of above" (p < 0.01).These students are about 6% of learners who chose "Pass exam" in the baseline survey.

Change in Engineering-based Mindsets.
Next, we explored the course's association with changes in the engineering-based mindset of students.In particular, we analyzed student responses from two questions within the baseline and endline surveys that target diferent course goals of an engineeringbased mindset: • "Do you help solve problems in your community?" • "If you try something, and it doesn't work, what do you do?"A Chi-square test showed that the responses for the question, "Do you help solve problems in your community?" in the baseline signifcantly difered from the responses in the endline, 2 (4, N = 4,119) = 807.82,p < 0.0001.In Table 2, we observe increases in the number of responses for answer choices "Occasionally" and "Often, " a >3% increase for each response from the baseline to endline survey.From the McNemar test analysis shown in Table 4, the signifcant efects are driven by students who originally chose "Never" and by the end of the course, shifted this response to either "Occasionally" (303 students, a third of those who initially chose "Never"; p < 0.0001), or "Often" (278 students, another third of those who originally said "Never"; p < 0.0001).
A Chi-square test showed that the responses in Table 2 for the question, "If you try something, and it doesn't work, what do you do?", in the baseline signifcantly difered from the responses in the endline 2 (4, N = 4,119) = 674.53,p < 0.0001.In Table 2, we frst see a notable increase in the number of responses for the answer choice of "Try a diferent solution to achieve your goal," which is about a 2.7% increase from the baseline to endline survey.From the McNemar test analysis shown in Table 5, 604 students who initially chose "Try again" shifted their response to "Try a Diferent Solution" (p < 0.0001).These students are approximately 46% of students who chose the former response in the baseline survey.
There is also a slight increase in the number of responses for the answer choice, "Give up," representing a minimal actual number of students.Specifcally, 81 students who originally chose "Try a Diferent Solution" (3% of students who initially chose this response) switched their response to "Give up" (p < 0.05), and 68 students shifted their response from "Try again" to "Give up" (p < 0.01).These students are about 5% of students who chose the former response in the baseline survey.

Potential for Future STEM Education and Related Careers.
We then investigated changes in student motivation to pursue STEM education and STEM-related careers in the future by analyzing Nemar test analysis shown in Table 6: showed a signifcant shift for 416 students who initially chose "Never" (about 48% of the number of students who initially chose this response) and changed their response to "Defnitely" (p < 0.0001) and 645 students who changed their response from "Not sure" to "Defnitely" (p < 0.0001).These students are over half of those who originally chose "Not sure" in the baseline survey.A Chi-square test showed that the responses for the question, "How likely are you to pursue a job in STEM?", in the baseline signifcantly difered from the responses in the endline, 2 (4, N = 4,119) = 350.71,p < 0.0001.We observed a similar shift in responses to the previous question, in which there was a signifcant increase in the number of responses for the answer choice "Defnitely" (almost a 5% increase), shown in Table 2.More precisely, from the McNemar test analysis shown in Table 7, 535 students who originally responded "Not sure" shifted their response to "Defnitely" in the endline survey (p < 0.0001).These students are over 40% of students who initially were not sure in the baseline survey.Finally, we examined the course's association with changes in student income and economic status throughout the duration of the course.To do so, we analyzed student responses from three questions included in the baseline and endline surveys, starting with whether students earned any income.Only those learners who answered 'Yes' then received two optional follow-up questions, and accordingly, only a subset of responses was recorded for these questions on the baseline and the endline.Table 8 lists each question with its corresponding total number of student responses for those who answered the question on both the baseline and endline (thus allowing us to conduct a comparison to investigate change).

Kwon, et al.
A Chi-square test showed that the responses in Table 8 for the question "Do you earn any income now?" in the baseline signifcantly difered from the responses in the endline, 2 (1, N = 4,119) = 570.81,p < 0.0001.
Though a majority of students who answered this question responded that they did not earn income in both the baseline and endline surveys, there is an increase of approximately 8% in responses for the answer choice, "Yes".Specifcally, in conducting the McNemar test shown in Table 9, 682 students who originally chose "No" shifted their response to "Yes" in the endline (p < 0.0001).These students are about 21% of those who initially answered "No" at the start of the course.
For those learners who said they earned income, a follow-up question asked in what occupation they made their income.A Chisquare test showed that the responses in Table 8 in the baseline signifcantly difered from the responses in the endline, 2 (9, N = 635) = 163.63,p < 0.0001.
All students who initially chose "Other" in the baseline shifted their response to the other answer choices, while no learners chose this response in the endline.Hence, "Other" as a response was removed from the McNemar test as it can produce unreliable conclusions when the summative count data for a response is 0.
The largest notable shift in the number of responses from the baseline to endline surveys was a 11% increase for the answer choice, "Through a science project."In particular, the McNemar test analysis in Table 10 shows 45 students who initially chose "Through farming" shifted their response to the Science Project cell (p < 0.0001).
A second follow-up question to learners who reported income asked how much they earned on a weekly basis.Answer choices for this question were initially categorized into fve intervals of weekly income: under 25,000 ugx, 26,000-50,000 ugx, 51,000-75,000 ugx, 75,000-100,000 ugx, and over 100,000 ugx (ugx represents the Ugandan currency).However, given the low number of responses on both baseline and endline for the three highest income intervals, we combined these intervals into one category, "51,000 ugx or above, " to provide a more robust analysis of student responses for this question.
The largest increase in the number of responses from the baseline to endline survey is for the answer choice, "51,000 ugx or above", which had a 6.42% increase shown in Table 8 and specifcally, in the McNemar test analysis shown in Table 11, 61 students who originally chose "under 25,000 ugx" in the baseline shifted their response to "51,000 ugx or above" in the endline survey (p < 0.05).These students are almost 16% of students who originally chose the former response.

Impact of Demographic Features on Course Completion
Section 4.2 investigated whether there were changes across time in mindset, motivation, and income for learners who completed the course.However, not all learners completed the course, and it is very likely that this attrition was not randomly distributed across the population.Therefore, to better understand how these motivational changes were distributed, we investigated what, if any, demographic factors were associated with course completion, including gender, urbanicity, family, employment, language, and educational status.For these analyses, our data frame included all students who completed at least the baseline survey.
A Chi-square test investigating gender as a factor showed that while fewer students who identifed as female initially enrolled in the course, shown in Table 12, those who enrolled completed the course at a higher rate than male students, 2 (1, N = 17,896) = 19.35,p < 0.00001.
Caring for children may also greatly impact available time and resources for studying, as shown in Table 12.A Chi-square test showed that students without children had a higher course completion rate 2 (1, N = 17,896) = 10.207,p = 0.0014.
Next, we investigated how completion was associated with prior educational experience in formal schooling, as shown in Table 12, specifcally if they had previously been or were concurrently enrolled in schools during this course.A greater number of students enrolled in school started the course compared to unenrolled students, and a Chi-square test showed that they also then completed the course at a higher rate than unenrolled students 2 (1, N = 17,896) = 6.5816, p = 0.0103.
Moving beyond a binary school enrollment, we next looked at how the level of completion in school was associated with completion of the course.Learners were asked to self-report the highest level they completed in the formal schooling system.The Ugandan schooling system is based on a British academic model, and so the seven classifcations included students who completed some or all primary school (P7 and below), some or all secondary school (S1; S2; S3; or S4), and some or all of the post-secondary courses known as A-levels (S5 and above; Completed A Level) [1].Table 13 shows the classifcations along with their expected equivalence in student age, although it should be noted that the actual age of enrollment in Ugandan schools varies widely, and these expectations are often not met.
A Chi-square test demonstrated a signifcant diference between the responses given in the baseline and endline surveys, 2 (6, N = 17,896) = 39.826,p < 0.0001.Specifcally, a Chi-square post hoc test showed a signifcant drop in the number of students in the "Completed A Level" category from the baseline to endline survey (p < 0.05), shown in Table 13.
While the target population for the course was residents in Northern Uganda, living environments and contexts heavily difered from student to student.Students may have been located in (in order of increasing urbanicity) refugee camps, villages, town centers, or cities.A Chi-square test showed that the urbanicity of students was associated with difering levels of course completion 2 (3, N = 17,896) = 8.0367, p = 0.045.Specifcally, a Chi-square post hoc test showed that students residing in villages completed the course at a higher rate than students living in other neighborhood types, as shown in Table 12.
Although the course was taught entirely in English, the ofcial language of Uganda, students were asked to indicate their preferred language to understand what translations might be useful in the future.Choices included English and local languages spoken in various parts of Northern Uganda.These local languages include Langi, Acholi, Luganda, Swahili, Ruyankore, and Lusoga.A Chi-square test demonstrated that there is no signifcant efect of language preference on course completion 2 (6, N = 17,896) = 9.4713, p = 0.1488.
Finally, we investigated the relationship between course completion and access to technology, shown in Table 12.Although the course materials were intended to be delivered by radio broadcast, a Chi-square test showed that there is no signifcant efect of access

Course Performance
Finally, we investigated how the initial motivations that learners claimed from Section 4.2 and the demographic features from Section 4.3 may have afected the opportunity for learners to succeed in the course, as evidenced by their scores on the fnal exam.Motivation for taking a course is a frequent covariate for analyzing MOOC performance.In these analyses, the data frame included all learners who attempted the fnal exam, which had 3,044 students.We applied a linear regression model to measure the impact of each identifed demographic feature on performance on the fnal assessment shown in Table 14.

Initial Motivations Associated with Final Assessment Performance.
The initial motivations we investigated were determined by categorization of student responses from two questions included in the baseline survey: • How likely are you to take a STEM course?
• What is the best reason to take this course?
Students who initially reported "Defnitely" and "Not sure" to the baseline question, "How likely are you to take a STEM course?" scored higher averages on the fnal assessment, at 60%.Students who reported "Never" in the baseline performed signifcantly worse than those who reported "Defnitely" and "Not sure," scoring an average of about 55% (p < 0.0001).
Students who took the course to "Learn science/tech" and to "Make Money" scored an average of 60% on the fnal assessment.These average scores contrast with the scores of students who chose "Pass exam" and "None of above." These students scored an average of 55% (p < 0.001) and 52% (p < 0.01) on the fnal assessment respectively.

Instrumental Demographic Features Associated with Final Assessment Performance.
Gender (male vs. female).Male and female students performed similarly on the fnal assessment, with both student groups scoring an average of 60%.Gender did not signifcantly afect overall student learning performance (p = 0.54).
Family Background.Family background was an infuential feature, as students with children performed an average of 56% (p CHI '24, May 11-16, 2024, Honolulu, HI, USA Enrollment Status in Schools and Educational Level.Both student groups who were and were not enrolled in schools performed similarly on the fnal assessment (p = 0.51).However, students with a more advanced educational background, particularly those with educational experience in S5 and above, scored higher than students with experience S4 and below.Students with educational experience in S5 and above scored an average of 60% on the fnal assessment, while students with S4 experience scored an average of 55% on the fnal assessment (p < 0.05).Students with S2 or S3 experience scored an average of 51% and 52%, respectively, on the fnal assessment (p < 0.0001).Those with S1 educational experience scored an average of 50% (p < 0.0001), while students with experience in Primary 7 and below scored an average of 49% on the fnal assessment (p < 0.0001).
Employment Status.On average, employed students scored 4.5% less than unemployed students (p < 0.0001).Employment was an infuential feature, as students who were employed during the course performed signifcantly lower than those who were unemployed on the course's fnal assessment.
Urbanicity.Students of all neighborhood types scored an average of 60% on the fnal assessment.Hence, type of residence as a demographic feature had no signifcant impact on student performance on the fnal assessment, as shown in Table 14.
Language Preference.Language preference had no signifcant impact on student performance on the fnal assessment, as shown in Table 14-students of diverse language preferences performed similarly on the fnal assessment.

Association of Access to Technology with Final Assessment
Performance.Overall, access to internet connection and radio had little impact on student performance on the fnal assessment, with all students with or without internet or radio access scoring an average of 60% on the fnal assessment.

DISCUSSION
As we drew signifcant insights from prior MOOC literature to the current study, this section discusses our fndings on signifcant changes in student motivation, mindset, and income, as well as infuential demographic features on course completion and performance in context with fndings from previous MOOC studies.

Change in Learners' Motivations, Mindset, and Income over Time
This work is intended to increase knowledge about how a low-tech, low-infrastructure, yet interactive learning platform provides the type of educational opportunity learners lack.It is frst important to understand if learners benefted from participating in the course.We found that, indeed, motivations, mindset, and income changed over the course duration for those who completed the curriculum.Learners were more likely to say that the reason to take the course was to learn science rather than pass an exam or make money.They were more likely to say they often helped solve problems in their community and that they would try diferent solutions to achieve their goal rather than try the same thing over and over again.They were also more likely to say they would defnitely take a STEM course in the future and that they would defnitely pursue a job in STEM.Furthermore, students who were more likely to defnitely take a future STEM course, along with those who were more likely to take the course to learn science and technology, scored higher on the fnal assessment.Altogether, these fndings imply a similar positive association between personal intrinsic motivation and overall course performance to that seen in prior MOOC research.Though some MOOC studies emphasize the importance of intrinsic motivations, our study also regarded income mobility as an important outcome of student learning.As subsistence farming is a major income source for many families, seasonal changes afect farm activities, which can lead to inconsistent income fow.Hence, creating another potential source of income through what students have learned and built throughout the course can elicit a benefcial fnancial change.Learners were, in fact, more likely to say they earned income after the course, and for the quarter of the learners that did earn some amount both before and after taking the course, that income went up and was more likely to be earned from science projects rather than farming.However, while not statistically signifcant, 37% of students shifted their response to not making income at the end of the course.Some of these learners may not have completed building a saleable solar cell.The seasonal nature of farming may also have accounted for income fuctuation.Our analysis also showed that a signifcant number of students shifted their response to say that there is no best reason to take the course, implying that some students found that their learning did not align with their initial goals or became unmotivated to learn.Moreover, our analysis showed that a signifcant number of students shifted their response to giving up when confronted with something that does not initially work.These fndings may suggest that this virtual classroom necessitates a feedback loop that allows students to evaluate the course in alignment with their goals throughout the duration of the course.Implementing regular mid-course surveys may be valuable to understand which parts of the curriculum student motivations and initial survey responses are changing to improve course curriculum and design and personalize learning for these students in relation to their goals.

Impact of Demographic Features on Course Completion and Performance
As so often happens with educational technologies, the learning benefts observed above have the potential to accrue only to the learners who already have opportunity and privilege, increasing rather than decreasing the digital divide.Therefore, we investigated how diferent features of learners' backgrounds and motivations impacted their ability to persist with and achieve in this type of course.
We observed that learners in what are typically lower-status positions were equally or better able to take advantage of the opportunities in this course.For instance, we found that unemployed learners, in fact, had an advantage on the fnal exam, which contrasts with prior MOOC literature that reports that MOOCs typically consist of employed participants with backgrounds in higher education.Despite the course being delivered in English, those with preferences for other languages did not sufer in terms of completion or exam performance.This is an encouraging fnding, as previous MOOC studies emphasized the gap between native and non-native English speakers in using online courses.We also found that fewer female-identifying students initially enrolled in the course, but those who did completed at higher rates than males and scored just as well on the fnal exam.Unfortunately, learners with children were less likely to enroll and less likely to complete.Additionally, those who did complete performed signifcantly lower on the fnal exam.Women are more likely to have children at a younger age and to be the primary caregivers, and adolescent pregnancy is a primary contributor to girls leaving school in the frst place.This is an important advance in the understanding of the impact of gender in MOOCs.
Due to the remote and sometimes inaccessible nature of rural communities in Uganda, urban inhabitants are the typical benefciaries of educational opportunities.A prior study on computer-based learning systems in the United States stated that rural learners show higher levels of engagement using educational technologies than other learners [3].Our study also investigated urbanicity as a potential infuential demographic on student learning performance and course completion.We found that learners in villages were more likely to complete the course than learners from any other locale and scored just as well on the fnal exam.This suggests that it was not necessary to have access to city resources for this platform to reach learners, adding to the literature by Baker et al. [3]; although refugee camp residents did not appear to receive the same beneft, only a very small number of these learners enrolled, which may not have been sufcient to see the impacts.
Another feature of city infrastructure is the greater access to technology.A signifcant diference between this virtual course and typical MOOCs is the (lack of) requirement for internet access.While this course did not require students to use the internet to learn, it is possible that having internet access is correlated with other life advantages.However, we found that learners without access to the internet were more likely to complete the course, and they performed equally well on the fnal assessment as those internet users who did complete the course.Interestingly, we observed that access to radio had no signifcant efect.As radio broadcasts that deliver course content are a central component of the studied course, one would expect that access to radio should be positively correlated with course performance and completion.However, learners were not technically required to listen to the radio broadcast as they received survey and assessment questions via SMS.Additionally, community centers where learners can share technologies are common features in low-infrastructure communities.Hence, many students may have reported not owning a personal radio when they actually listened to a communal radio.Also, it is possible that students with higher educational experience did not rely on radio broadcasts as they were already knowledgeable about course content.This result is worthy of further study to understand the context of radio use and the relative value of content delivery versus interactive practice opportunities.
On the other hand, there are still remaining disparities in who benefts.In one sense, our fndings align with prior MOOC literature that shows learners with prior educational experience, particularly those with secondary and postsecondary schooling experience, are more likely to show higher course completion rates and learning performance.Each fewer year of completed education was associated with a lower score on the fnal assessment.Unlike prior MOOC literature, though, learners with lower levels of education, even those with only experience in primary school, were equally as likely as those with four years of secondary school to complete the course, and in fact, those with the greatest amount of education, in particular, tended to drop out.The necessity of at least a basic level of education may be an expected yet challenging reality for systems that attempt to reach out-of-school learners.
While MOOCs aim to "democratize education for all, " many studies have revealed that most MOOC participants are advantaged in higher education and socioeconomic status [10].This study provides a global alternative to MOOCs for underrepresented populations as we found that students with no access to the internet, non-English language preferences, lower degrees of urbanicity, and features related to lower-status positions completed the course at higher rates and performed just as well on the fnal assessment as other students.The above fndings may speak to the unique benefts of a platform like this for underserved learners.Alternatively, given that many of the fndings indicate increased completion rates, it may be that learners with more advantages fnd that they have alternative opportunities and leave to pursue those.For instance, learners with internet access may join a MOOC, a multimedia-or video-based learning tool, or one of the many other opportunities they fnd online.Even so, the higher completion rates that are found in MOOCs and the fnding that most underserved groups performed equally well on the fnal indicate that there is value to be found here.

LIMITATIONS
Due to the large-scale nature of the study, the course designers could not collect more details about learners beyond their responses to the phone-based surveys and questions.In fact, even these responses were limited by the afordances of low-cost technologies, which include a small character limit for messages and no multimedia.Data collection in such contexts involves critical tradeofs between survey length, complexity, and outcomes.Surveys were kept short as every question added leads to greater student dropout.Mobile phones also have smaller screen space, restricting the number of characters for each survey question and their corresponding responses.
Specifcally, the course only asked two survey questions on students' engineering mindsets.While these questions do not rigorously cover this topic, they provide an initial investigation into students' engineering mindset while accounting for a low-literacy population.Consequently, an important next step is to conduct a qualitative investigation with students during future courses to better capture their subjective views on both engineering mindset and motivations for taking the course, whether through interviews or by delivering a longer paper-based survey to a subset of learners.
Additionally, while investigating these types of changes over time is a common approach in large-scale educational interventions, it is important to remember that the fndings were not derived from a randomized controlled study, so the results are correlational rather than causal.Other interpretations may be valid for the results that we have presented here.For instance, it is possible that the baseline survey but not the endline survey coincided with an agricultural harvest season, hence limiting the opportunities for learners to make money from farming at the endline as they would at other times of the year.

CONCLUSION AND FUTURE WORK
Previous lower-tech approaches, such as IRI and phone-based interventions, have already been deemed successful in delivering educational opportunities for learners in remote communities worldwide.However, these interventions are nonreciprocal, in which students learn from delivered questions and lesson content.Prior mobile phone-based research has shown that learners in similar low-infrastructure contexts struggled with using an automated messaging system as they were more familiar with texting with a human peer.While radio instruction provides an interactive aspect of learning as human instructors broadcast educational content, interaction data is difcult to collect, preventing an in-depth understanding of infuential features that may afect student learning.Additionally, many phone-based studies employed quiz-based interventions, which may not be sufcient for a full educational experience.Our work shows a combined interactive system's potential to address these shortcomings.The remote course "closes the loop" with human instructors broadcasting course content, survey questions, and activities in which students interact by sending their responses via USSD.Students can learn from a well-rounded course curriculum with test questions, real-time lesson content, and creative activities for building an innovative product.This learning system also permits collecting student demographic, motivation, and course interaction data as student responses are received via USSD, allowing researchers to draw connections between infuential student demographics and motivation and course learning outcomes.
Our work has explored a dataset from learners engaging with an interactive radio instructional platform focused on innovative engineering education for low infrastructure settings.We investigated the course's association with changes in student motivation, income mobility, and engineering-based mindset, and the association of various demographic and motivation features on student completion rate and performance in the studied distance learning course.
Overall, we found that students were more inclined to learn more about science and technology after this course, including a signifcant number of students reporting that they would like to pursue STEM-related job opportunities beyond the course.Furthermore, we detected overall student development in attaining an engineering-based mindset.We also observed changes in the income mobility of students in alignment with the course, specifcally, signifcant shifts in students earning money through science projects.
Our work determined gender, family background, school enrollment, prior educational experience, urbanicity, and access to the internet to be infuential demographic features associated with course completion.We also found various demographic and motivational features associated with student performance on the fnal assessment, including prior educational experience, job employment, initial motivations in taking this course, and the likelihood of pursuing STEM education in the future.While some of these features aligned with prior work on dropout and performance in MOOCs, in many cases, we observed novel promise in such a platform for retaining and training learners who are denied many opportunities for education.
In our investigation of infuential demographic and motivational features on student learning, we noticed a signifcant number of students from each group who withdrew their participation at different stages of the course, as has previously been found in distance education.While some of these students completely dropped out, others participated intermittently, and some succeeded.In our next steps in this work, we analyze these students' behavioral and engagement patterns, examining when, where, and why students of various demographic and motivation groups change their engagement by investigating their detailed learning activity.

Figure 1 :
Figure 1: Visual cycle on how students interact with the course

Figure 1
summarizes student interaction with the course.All responses are recorded and logged within a custom database of the course.Students can review their progress through learner reports and recorded previous lessons that are available on their phones through SMS.At the end of the course, students complete a fnal assessment, which is delivered through the radio broadcast during the last week of the course.Again, all answers are synchronously logged within the same custom database.The radio broadcasts were delivered over a total of 15 weeks, each consecutive week focusing on a step of the course curriculum.The curriculum follows the ordered steps of the engineering design process in creating the fnal product: Introduction Step: Basics of engineering education; Step 1: Identify; Step 2: Investigate; Step 3: Brainstorm; Step 4: Plan; Step 5: Create; Step 6: Test; Step 7: Improve; Step 8: Launch.Steps 2 and 7 were each allocated two weeks of lesson content, while Step 5 was allocated three weeks.These distributions were based on piloting and data from a previous iteration of the course.

Table 1 :
[11]ent Participation in Course ComponentsTable1shows the number of learners who engaged with the various components of the course.4,149out of the 17,896 learners who registered for the course made it to the endline survey, a rate of approximately 23%.This percentage is more than double the under 10% completion rate cited by[11]for MOOCs (although MOOCs have a wide variance depending on type), perhaps indicating that learners treated this type of course diferently or had diferent motivations for engaging in the frst place.

Table 2 :
Percentages of students who selected each response on the baseline and the endline for each survey question Full response choice is "Entrepreneurship/Make money", ** = Full response choice is "Try a diferent solution to achieve your goal"

Table 3 :
Contingency table showing shifts in responses for the question: What is the best reason to take this course?

Table 4 :
Contingency table showing shifts in responses for the

Table 5 :
Contingency table showing shifts in responses for question: Do you help solve problems in your community?
2(4, N = 4,119) = 393.63,p<0.0001.The answer choice "Defnitely" showed an • How likely are you to take a STEM course?approximate 13% increase, shown in Table2.Specifcally, the Mc-• How likely are you to pursue a job in STEM?

Table 6 :
Contingency table showing shifts in responses for the question: How likely are you to take a STEM course?

Table 7 :
Contingency table showing shifts in responses for the question: How likely are you to pursue a job in STEM?

Table 8 :
Percentages of students who selected each response on the baseline and the endline for each survey question on student income and economic status

Table 9 :
Contingency table showing shifts in responses for the question: Do you earn any income now?

Table 10 :
Contingency table showing shifts in responses for the question: How do you earn your income?

Table 11 :
Contingency table showing shifts in responses for the question: How much do you earn weekly?

Table 12 :
Number of students in each demographic group who completed each survey Demographic Features Demographic Groups Baseline Responses Endline Responses

Table 13 :
Contingency table of the number of students in each educational level group who completed each survey Education Level ∼Age Baseline Endline

Table 14 :
This table shows the results of the applied linear regression model on scores of students who took the fnal assessment *< 0.0001), performing signifcantly lower than students without children on the course's fnal assessment.