Integrating Canvas and GitLab to Enrich Learning Processes

Version Control Systems (VCS) are increasingly used for software education. However they are often used independently of general-purpose Learning Management Systems (LMS), leading to poor user experiences and data fragmentation. This paper presents a high level programming framework that integrates the Canvas LMS and the GitLab VCS. Eight use case applications are implemented with it, motivated by related work, educator and student surveys we conducted, and our early experience of using GitLab for software education. The framework semi-automates feedback loops, improves the overall user experience for students and educators and reduces administrative time - saving educators up to 16.7 hours in a 12 week semester. The components of our framework are open source for learning technologists to develop their own use cases.


INTRODUCTION
Programming is a central component of computer science education and its teaching has been the focus of extensive research, methods and tools [17].Using web based Version Control Systems (VCSs) means students experience DevOps, Continuous Integration (CI) and Test Driven Development (TDD) in their software education.The automated code testing and extensibility support of the GitLab VCS allows us to scale software education to thousands of students across our multiple global campuses.
The drawback of using web based VCSs for teaching is the fragmentation that it introduces in course delivery.VCSs are typically used independently of an institution's general purpose LMS.Students have to switch between VCS and LMS systems to access (1) assessment results, (2) enrolment information, (3) learning materials, (4) starter code and ( 5) feedback.Educators have to switch between both to monitor student engagement.This creates a poor user experience and impedes the ability to analyse student activity.
This paper presents a high level programming framework that integrates an LMS (Canvas) and a VCS (GitLab) to overcome these fragmentation issues.It enables composition of API calls to Canvas, GitLab and 3rd party libraries that support software education.
This paper makes the following contributions: • A methodology for gathering LMS/VCS integration requirements using surveys, computer science education literature and our early GitLab experience, and the design of eight use cases to address these requirements (Section 2).• An open-source high level software framework that supports data integration across Canvas and GitLab (Section 3).• The implementation and evaluation of the use cases.They save administrative time for educators, they semi-automate feedback loops and improve the overall user experience for students and educators (Section 4).

MOTIVATIONS 2.1 Literature Requirements
General purpose LMSs provide a range of course management features.However, they lack vital support for computer science education.For example, they do not provide an integrated development environment (IDE) or integrated code versioning.Rößling et al. idealised an LMS for computer science education, identifying nine requirements for a Computing Augmented LMS (CALMS) [24].Our framework aims to meet the following five CALMS requirements: (1) Automatic programming exercises assessment: Combining an LMS and automatic programming assessment so that educators can access the outcomes of automatic assessments while marking within the LMS.(2) Collection of assessment results: Incorporating assessment materials and results into the LMS data base.(3) Specialised tools for programming: Connect an LMS to programming tools (for example, a VCS) to expose students to real world environments, enable comprehensive tracking of student activity and enhance collaboration.(4) Support for drill-and-practice: Provide tools for the first stages of learning, for topics that do not require creativity but only understanding (drill-and-practice exercises).( 5) Support for complex construction problems: Include support for developing comprehensive programs that require student creativity, such as collaborative support for editing, discussion and version control.

Educator and Student Surveys
We surveyed students and educators in the Computer Science department at Heriot-Watt University about their experiences with Canvas and GitLab (our research protocol was reviewed and approved in accordance with the ethical process of our University).
Pre-surveys gathered input on possible features for integration between Canvas and GitLab.Post-surveys were sent to both groups after the implementation to evaluate the framework.[I] make a sheet related to information collected from GitLab.Identifying a student's GitLab project for a coursework on Canvas requires another manual effort.An educator stated: I use links to GitLab submissions within Excel.We also asked educators which metrics they would like to be shared between Canvas and GitLab.5 of 7 educators would like to see the number of git commits and the commit timestamps on Canvas to see the provenance of student progress up to the deadline.4 of 7 educators asked to see the pipeline status of a GitLab project on Canvas, allowing the results of automated testing of student code to be visible on both systems.6 of 7 educators requested an extraction of GitLab log-in data to identify at-risk students.

Student Pre
-Survey.There were 9 responses from students in the Computer Science faculty.5 were in the 4th or 5th year of their BSc or MEng degrees, all of which had used GitLab and Canvas.The other 4 were on MSc programmes, who had no prior GitLab experience.Students were asked to provide feedback on their satisfaction with the current setup of GitLab and Canvas, and share their ideas for potential integrations.The pre-survey asked students to rate statements about Canvas and GitLab (Figure 1) on a Likert Scale.5 of 9 respondents struggle associating coursework on Canvas with their own GitLab project.6 of 9 students agree they often switched between GitLab and Canvas.7 of 9 students would like assessment deadlines to appear in GitLab.

Experience with GitLab
We trialled GitLab in 2018.For online-only teaching during COVID-19, we scaled GitLab's use to 20 of our programming courses [29].When using GitLab at a global scale (thousands of students), the time costs of several administrative tasks emerged including: • Students had to manually add lab helpers to their GitLab projects to enable close-to-code discussion.• Educators had to manually add students to the GitLab group for their course.• Educators needed to periodically check for merge requests to delete them, because merge requests against a starter project exposes code to all students in a GitLab group.This poor user experience supports Glassey's call for a lower entry barrier for git use in teaching and course management [11].We identified two requirements to address these limitations: (1) Remove manual effort of administering GitLab.
(2) Avoid data fragmentation to enhance the user experience when using both GitLab and Canvas in a single course.We address these with a server-side event driven API, which is also used by three use case applications (Section 3.2).

Use Case Design
This section presents our eight use cases.They integrate Canvas and GitLab when data exchange between both systems is required.Three require mechanising only GitLab focusing on student code.
Table 1 gives an overview of the use cases.It identifies the survey(s) that inspired each one, and which CALMS 1-4 requirement they satisfy.CALMS 5 is met by our framework as a whole, as integrating GitLab offers collaborative discussion, and version control.Close-to-code discussion between students and educators on GitLab meets the call for support for complex construction problems.

Student Enhancements.
Deadline Reminders.This use case exchanges assessment deadlines from Canvas, where the educator enters this date and time to a GitLab issue to each student coursework project (Figure 2).This use case was prioritised highly by students and educators in the pre-surveys.7 of 9 of students indicated they would find it useful to see the deadline for assessed work on both Canvas and GitLab. 2 educators identified the import of the deadline information from Canvas into GitLab as a desirable feature of the framework.
Test Driven Feedback.This use case tests a student's code with GitLab's CI when they push git commits.Rößling et al. recommended automatic assessment of programming exercises (CALMS 1).Adopting GitLab as a DevOps platform does this by enabling programming courses to adopt TDD for immediate student feedback, code assessment and automated assessment [30].
We use TDD by mapping unit tests to learning outcomes, where each coding task is associated with multiple tests.For example, our  We also assess students' ability to translate text-based requirements into testable assertions.GitLab tests these when a student pushes their commits.The immediacy of CI reports serves as a formative, automated feedback mechanism satisfying CALMS 1.

Educator Enhancements.
Single-Course Student Analytics.Currently, any time a student interacts with GitLab, the analytics features in Canvas do not detect this interaction.To analyse students' engagement, educators have to analyse activity data across both systems separately, and combine them manually.The CALMS 3 requirement identified this problem.
To identify non-engaging students, 6 of 7 educators use grades on Canvas and the number of git commits on GitLab.They manually browse this on the respective web interfaces.This use case supports educators by reducing the time spent on student engagement analysis.It creates a spreadsheet (Figure 3) combining data from both Canvas and GitLab.It provides fine-grained insights into one course, listing all assignments of this course.
Multi-Course Student Analytics.This use case generates a spreadsheet (Figure 4) with data collected from both Canvas and GitLab across multiple courses.It provides programme directors with a wider picture to identify students whose engagement is not consistent across courses.It obtains the last log-in dates for multiple  Semantic Code Analysis.Some of our courses require students to implement data structures from scratch, without using standard libraries.This use case creates a note on commits if a student illicitly imports disallowed data structures for a Java-based coursework (Figure 5).This use case helps to turn GitLab administration into an automated process (Section 2.3).It also addresses CALMS 1 by partly automating assessment and CALMS 4 by providing the student with immediate feedback on their coding exercises.
Submission Logs.This use case imports GitLab project activity into Canvas.When a student pushes git commits it creates a comment on the student's corresponding Canvas assignment, with the URL of the GitLab project, the number of commits in the push and the latest CI pipeline status if available (Figure 6).The link to the source code for the assignment.GitLab project url, creation time, added members Two responses mentioned that they would also like to see the pipeline status of the GitLab project.
Synchronised User Profiles.To integrate GitLab with Canvas, student information should be up to date on both systems (CALMS 3).This way, educators can more easily recognise students on GitLab without having to go back to the LMS for more information.This simplifies marking students' code.When asked what data educators would like to import into Canvas, one responded: I think it would be useful for ensuring students are easily recognisable on GitLab.This use case obtains a student's Canvas profile and adds campus location, student bio and preferred pronouns to their GitLab profile.
Plagiarism Detection.In [5] 22 of 49 academic staff estimated occurrences of source-code plagiarism in initial programming courses to be 20%-50%.Plagiarised code can be obtained by (1) 3rd party contract cheating, (2) stealing another student's source code or (3) collaborating with student peers contravening submission policy [4].Web based version control systems like GitLab are vulnerable to stealing exploits if access to student projects is unrestricted.
To prevent unauthorised collaboration between students, this use case integrates GitLab with MOSS [27], a plagiarism checker system for measuring code similarity.The use case automates assessment of programming exercises (CALMS requirement 1), returning a MOSS report containing pairwise similarity scores of students' code.
We prevent code stealing by defaulting all GitLab projects to private access.This access restriction is inherited by forked student projects.This does not impede feedback mechanisms because server side automation (Section 3.2) adds educators to private student forks to facilitate online educator/student interactions (Section 4.1.2).

FRAMEWORK ARCHITECTURE
A key design principle of the framework is a ease of use.Our framework is in Haskell, thereby embedding GitLab and Canvas into a single programming language, allowing seamless composition of API calls across both systems.A secondary design principle is extensibility, so that programs can use 3rd party libraries.Embedding our framework into a general purpose language allows use case implementations to benefit from a substantial collection of available open source libraries1 , e.g., to parse student source code for ).This section describes how our framework integrates the Canvas LMS, the GitLab VCS and 3rd party libraries.

API Client Libraries
We have developed Haskell libraries for Canvas and GitLab.They lift their web APIs into statically typed functions and data types.

canvas-haskell.
The OpenAPI framework2 aims to standardise the description of REST web APIs using OpenAPI specification (OAS) syntax.A client generator tool [8] generates our Haskell client from our OAS specification of the Canvas API.

gitlab-haskell.
There is not currently a complete OAS for the GitLab REST API.We therefore manually implemented a Haskell client, which provides complete coverage of the GitLab API.

Pagination Support.
Both libraries support the respective API's pagination.They repeatedly send requests to paginated endpoints under the hood.Paginated data is parsed and included in the request until the last page is received.With our pagination support, every function calling a paginated endpoint automatically aggregates all data instead of just the first fragment.
We integrated custom pagination code to canvas-haskell after the library is generated by [8], because the OAS framework does not allow specifying pagination in a specification file.This OAS feature has been frequently requested, but to date has not been implemented.To our knowledge, one other Canvas client library for Java implements pagination similarly 3 , while others retrieve the next pages on demand 4 .Some libraries require manual specification of endpoint paths 5 , or do not support pagination.Gitbeaker6 , a Node.jslibrary for GitLab, implements pagination support similarly to gitlab-haskell.Some GitLab libraries expose the paginated nature of REST endpoints where the developer must use a generator object, which only handles the next API call when required (python-gitlab 7 , gitlab for Ruby 8 ).Others either do not offer pagination support (GitLabApiClient9 ), or require the developer to manually loop through the responses (go-gitlab10 ).

A Server Side GitLab API
Some use cases in Section 2.4 require a real-time response to student activity on GitLab.For example, to facilitate student/educator feedback discussion and to ensure student code does not become visible to other students if a student creates a merge request against the starter project.Our early use of GitLab required educators to manually delete these.This need for intervention does not scale to hundreds of students, where the possibility of plagiarism from temporarily visible code solutions grows.We use GitLab's officially supported server-side automation to remove manual effort: (1) Continuous Integration (CI): GitLab runs a script within the .gitlab-ci.yml file if present in a project.These scripts can test code, e.g.JUnit tests on Java coursework.(2) System hooks: GitLab provides real-time event information as JSON objects to executables or 3rd party web services.
To lift the system hooks support into a framework to monitor GitLab events, we have abstracted this feature into a high level event driven API within gitlab-haskell.The API (Figure 8) is simple with just two functions, and uses pattern-matching on events.It supports 24 different server-side GitLab events, including the creation of users, groups, projects and merge requests.Rules specified with the match function will always fire, so the GitLab action function will be executed when the event type such as {Push} and {UserCreate} is pattern matched on.Rules specified with the matchIf function will only fire if the predicate function provided returns True for the given event and the correct event is pattern matched.These rules are compiled to an executable file to be deployed on a GitLab server as a file hook.
This API is used in two use cases, Submission Logs (Section 4.2.4) and Synchronised User Profiles (Section 4.2.5).It is also used for three administrative duties on GitLab: (1) Add educators as members to a student GitLab project if it is a fork of a starter project.When the student then messages to @all, the educators receive that question, triggering feedback discussion online.(2) Delete merge requests on starter projects instantly.
(3) Add newly registered GitLab student accounts to relevant course groups on GitLab based on the courses they are enrolled on.This is to improve the GitLab experience for new undergraduate and postgraduate students.
The following example adds a GitLab group called "lab-helpers" to a newly created project if it is a fork of the "lab6" project.This allows @all discussions to be received by members of the "lab helpers" GitLab group.This rule pattern matches on ProjectCreate, which is the Haskell data type that represents the event of project creation.

USE CASE IMPLEMENTATIONS
This section presents the implementation and evaluation of our eight use cases, which were developed using the framework.
The educator post-survey asked about five use cases: Deadline Reminders, Synchronised User Profiles, Single-Course Student Analytics, Multi-Course Student Analytics and Submission Logs.Educators were asked to report on their potential to save time, the usefulness of the use cases and provide ideas for improvement (Table 2).The student post-survey aimed to evaluate the Deadline Reminders use case and provide ideas for improvement.
Canvas and GitLab both authenticate students using OAuth against their university IT account, so usernames on Canvas and GitLab are identical.Student usernames are used for many of the use cases when flowing student data between GitLab and Canvas.
The use cases implemented as server-side deployments are shown in Figure 9.The use cases implemented as client-side applications are shown in Figure 10.All use cases are discussed in detail below.

Student Enhancements
4.1.1Deadline Reminders.Issues are a GitLab mechanism that can be understood as a To-Do item.In a software education context, educators use them to discuss feedback with students [10].The GitLab interface does not support setting a deadline on a GitLab project itself.Therefore, the Deadline Reminders application creates a new issue on the project, and sets a deadline for that issue.This creates a To-Do in a student's GitLab inbox and emails them a notification.If the issue is not closed (the GitLab term for mark as done) before the deadline, the date will turn red on the website.
To demonstrate the conciseness of GitLab/Canvas applications, the Haskell implementation 11 of the Deadline Reminders use case is in Figure 11.The input to the application is the Canvas course ID, assignment name and section ID, the GitLab project name and the text to appear in the issue description that the educator specifies.The application (Figure 10a) retrieves a list of student users, and the assignment's deadline, using the canvas-haskell library.Each obtained Canvas user value contains the student's login_id, which is used to locate GitLab accounts with gitlab-haskell.This provides a list of GitLab users of all students on the given course.The application then searches for a GitLab project with the given name for each of those GitLab student accounts.If a project is found, i.e. the student has forked the coursework starter project, then an issue is created on their project and a deadline is set on that issue.
This application was run in 2022 and 2023 for two pieces of coursework on a first year course, and one piece of coursework on a fourth year course.5 students responded to the post-survey about this use case.4 students had noticed the deadline reminder.3 first noticed it as To-Do pop-up on GitLab's navigation bar, the other student from the email.These 4 students gave this use case an average rating of 3.84 points for usefulness on the Likert Scale.All 5 students agreed they would like to receive more deadline reminder issues in the future.None of them mentioned any negative feedback when asked if there were any aspects they did not like.4 out of 6 of educators surveyed agreed that they could save time using this application with an average of saving 3 hours per semester.
4.1.2Test Driven Feedback.This application (Figure 9a) provides students with immediate feedback on the functional correctness of their code.The drill-and-practise requirement (CALMS 4) is met using CI, where we assess students on their ability to complete partial implementations of data structures and algorithms to retain learned knowledge.Moreover, educators benefit from CI reports to shorten marking times.The example in Figure 9a shows test driven feedback between a student, two educators and GitLab's CI.This demonstrates how a web based VCS can facilitate both lecturers and lab helpers to passively monitor student progress, allowing any member of the teaching team to intervene with formative feedback.
Since moving to test-driven feedback with automated testing, we have observed students gamifying their own learning, motivated by changing failed tests (✗) to passes (✓) in CI reports.This behaviour change is observed with similar approaches [18,28].
Our gitlab-haskell library allows applications to digest unit test results for a project's CI pipeline: This TestReport record has success_count and failed_count fields with test result outcomes.We have implemented other applications, not described in this paper, that use the pipelineCI function to customise weekly feedback messages containing code test results sent as GitLab issues and as emails.

Single-Course Student
Analytics.This use case provides lecturers with a fine grained spreadsheet of student engagement data across both Canvas and GitLab for a single course.The educator can view the grade from Canvas and the number of commits, pipeline status, visibility, and last activity from GitLab for each student's assignment.The application takes a Canvas course ID, and the name of the GitLab group corresponding to the Canvas course.It has a column for each Canvas assignment, with the grade for each student submission.There are multiple columns with GitLab data for each assignment: the number of git commits, which demonstrates engagement and that the work is the student's own; the CI pipeline, which provides a lightweight indication of whether the code is functionally correct if TDD is used; visibility, which shows that projects are private (for individually assessed work) or public (for group work); and the date of the last activity on the GitLab project, which indicates engagement levels.
This application (Figure 10e) obtains student grades from Canvas with the following function.It returns a list of Assignment values for each student, and each assignment on the course: Each Assignment value contains the student's username, name and grade, which are included as three columns of the generated spreadsheet (Figure 3).The application then adds columns for each GitLab metric extracted from projects associated with each student who is a member of the GitLab course group.

Multi-Course Student
Analytics.In addition to fine grained single-course analytics, programme directors may prefer to see reports showing a wider view of student engagement across multiple courses.Given a list of course IDs to report on, the application (Figure 10d) finds all students on those courses and gets their last login date from both Canvas and GitLab.The output is a spreadsheet containing each student's username, name, and their last login dates on GitLab and each Canvas course.For each student, the last log-in field is accessed through the canvas-haskell function: getAllSectionEnrollments :: Text -> Canvas [Enrollment] This application uses gitlab-haskell to lookup a student's most recent activity on their most recently updated project.The date is accessed through the function: userProjects :: User -> ProjectAttrs -> GitLab [Project] updating project_last_activity_at in ProjectAttrs to retrieve a project's list in order of how recently they were modified.
This use case received the highest usefulness rating, with an average of 4.67 from educators.They agreed that it could save them time, which they estimated at an average of 5.33 hours per semester.10b) prevents students from using disallowed data structures.It downloads student source files with gitlab-haskell, then an open source library 12 to parse Java source files into an abstract syntax tree for analysis.Here we benefit from integrating our framework within  a general purpose language (Haskell): there are no interoperability issues with 3rd party software components.All students fork a GitLab starter project, which has an initial directory structure.The homogeneity of file paths in forks of the same starter project allow us to mechanise the checking of all GitLab student projects.This application attaches notes to commits that import disallowed Java classes to ensure all students implement their own hash map data structure.It downloads src/Collection.javafrom student GitLab projects, and parses the Java abstract syntax tree from it.Shown in Figure 5, if the application finds that java.util.HashMap is imported then it posts a commit note warning on the student's fork advising student not to use the HashMap class.

Submission
Logs.This application (Figure 9b) is implemented as a server-side rule to import students' GitLab project information into Canvas.It is implemented with matchIf (Section 3.2).It gets triggered every time a {Push} event happens on the GitLab server, i.e. when a student pushes from their workstation, and the project is associated with a given course.The application extracts the student's username, the GitLab project URL and the number of commits.To obtain the latest automated CI test results for that project, the application uses gitlab-haskell's function: The application pulls the latest pipeline for up to 30 seconds, giving the server time to run the tests.It then locates the Canvas assignment with canvas-haskell and creates a comment for a student's submission for that assignment.The comment includes the GitLab project URL, the number of commits and CI status.
Educators rated this use case as 4.17 in terms of usefulness.The use case enables them to directly access a student's GitLab project from Canvas. 5 of 6 educators agreed that it would help them save time, with an average of 7.4 hours saved per semester.One educator stated that they had a similar, manual set-up in Microsoft Excel, which this use case makes redundant.They also stated that the use case might be helpful to lab helpers: [...] it would save time to the team of markers who currently need to check a url to see if a student has committed.9c) feeds up-to-date Canvas profile data to GitLab.The student's campus information is added to GitLab's organisation field, which is visible on a user's profile page.Both Canvas and GitLab allow students to set their bio and pronouns.If not already set on GitLab, the application will copy this data from Canvas to GitLab.This application runs for one course at a time.The application obtains profile data for all students from Canvas.It then creates a UserAttrs value to send to GitLab to modify the account with the given student integer ID, using gitlab-haskell's function: Educators rated the usefulness of this use case as 3.17 on average.3 out of 6 educators agreed it would save them time, with an average of 1.6 hours saved per semester.One educator said: Student name and ID (possibly programme and campus), pass/fail info on previous exercises in the course.

Detecting
Plagiarism.We use MOSS to detect plagiarism.This application (Figure 10c) detects plagiarism amongst student submissions.The homogeneous directory structure across all student GitLab project forks allows for the application to perform plagiarism detection at scale.Again we benefit from adopting a general purpose language for our software education framework, as there is a Haskell library binding to MOSS 13 .

User Experiences
5.1.1Time Savings for Educators.We focused on usefulness and the potential to save time when evaluating our framework with educators.6 educators responded to the post-survey about the educator enhancements (Section 4.2).During the survey, educators were presented a description and diagram of the use cases.Although listed as a Student Enhancement, we also surveyed educators on the Deadline Reminders as it gives them the ability to effectively notify students, and therefore may reduce administrative overheads.
Table 2 gives an overview of the post-survey results.The Time Saved column shows how much time educators estimated to save throughout a 12 week semester on average.The Usefulness column shows how useful educators rated each use case on average.There is a strong correlation between estimated time saving and usefulness with a Pearson's correlation coefficient of 0.818 [9].The more time a use case saves, the more useful it is to educators.
An educator making use of all four evaluated use cases would save 16.7 hours per semester on average.This demonstrates the potential of our framework.As the amount of use cases increases in the future, so will the amount of time saved for educators.
5.1.2Feedback Summary.The absence of any negative feedback shows that both students and educators welcome the integration framework to automate or streamline their activities.For analysing student activity, educators are given the ability to efficiently generate spreadsheets containing complete student activity data from both systems, which was not possible before.This makes manual combination redundant and saves educators 5.3 hours per semester.More efficient assessment is facilitated through automatically detecting plagiarism and connecting projects across Canvas and GitLab, importing the number and time of commits into Canvas.The latter saves educators 7.4 hours per semester on average.
The framework enhances students' experience by gamifying their learning process through enabling Test Driven Feedback in their programming exercises.When working on assessed projects, the deadline and codebase of the project are no longer split across Canvas and GitLab, but combined in GitLab.Their feedback shows encouraging approval, with students stating that they would like to receive more Deadline Reminders in the future.

Use Case Improvements.
We also surveyed educators on ideas for improving the use cases.To improve the Deadline Reminders, one educator stated that they would like to be able to set multiple reminders.Several educators recommended considering the annoyance of creating too many GitLab issues for students.They stated that the deadline reminder might be more useful when combined with more filtering, such that only non-engaging students receive the reminder issues.Combining these two improvements could create an application which sends multiple reminders, but only to students who have not yet engaged.
When using the Submission Logs use case, an educator suggested that adding the push dates to the comment would be useful.A respondent also stated: An additional feature would be to have a specified branch name for final submission so that the Canvas notification is not on commits, but on creation of such final branch or on commits on that final branch.It would be possible to implement this request by extracting branch information from the push event using the field push_ref of gitlab-haskell.The application would depend on the particular course set-up, i.e., students need to be told to use a final branch.Not every educator structures their assignments in this way.However, the application could be extended to cater to these specific requests by allowing the user to choose when a comment on Canvas should be created and what should be included.

Our Experiences
5.2.1 Automation versus Flexibility.Integrating LMSs and VCSs at scale presents a trade-off between automation and flexibility.Our use of GitLab varies across courses due to multiple factors: how many students, whether it is an introductory or advanced course, whether it is deployed across multiple global campuses, etc.For example, some lecturers use GitLab sub-groups to: • Organise students into different campuses.
• Organise teaching staff into lab helpers and lecturers.
• Separate work-in-progress projects for staff to develop code, from projects shared with students for forking.
Our year 1 courses do not use sub-groups to keep their Git-Lab setup very simple to avoid confusing new students.Whilst some tools make opinionated and rigid choices about automating course deployment, e.g.RepoBee [16], our programming framework provides greater flexibility for automating whichever deployment structures are most appropriate for each course.

Integrated Learning Management
Systems.A lot of potential lies in creating a bridge between an LMS and a VCS.Even with our small set of eight use cases, the integration gives perspectives to significantly increased educators' productivity.Both students and educators appreciate the advantages that the integration offers.

Limitations.
Access restrictions.Depending on a university's set-up, restrictions limit data access for several Canvas and GitLab endpoints.For example, it is not possible to obtain any student's bio or pronouns without an account-level role on Canvas, which is usually only granted to IT sysadmin accounts rather than educator accounts.This creates challenges for testing applications without elevated access, and it limits deployment to only IT staff with necessary elevated credentials on Canvas and GitLab.
Keeping in sync with Canvas.To keep exchanged information up-to-date, GitLab web hooks can trigger the server-side use case applications, ensuring that any change on GitLab is copied to Canvas.However, changes on Canvas go unnoticed by the framework without manual intervention, since there are no webhooks implemented on our Canvas instance at Heriot-Watt University.
Robustness of generated OpenAPI clients.Due to the use of the OpenAPI framework, errors or outdated information in the API documentation are inherited by the OpenAPI specification, which in turn get passed onto the generated Canvas library.The List users in course endpoint is an example of this problem, as its description lists last_login as a field returned in the response.However, this is currently not returned and developers have to use a different endpoint to get the desired data 14 .Our effort to engage with the Canvas community about the OpenAPI specification has yielded no success to date 15 .Engagement with the Canvas development community would be crucial to the further improvement of canvas-haskell (and any other generated Canvas clients), as any changes to the API could be added to the client library with very little effort given an accurate OpenAPI specification.This would greatly support educators who wish to develop their own use cases.
Scalability.Our framework for orchestrating Canvas and GitLab with server and client side applications enabled us to scale 20 programming courses across four global campuses.It was primarily designed to be used by learning technologists to support educators deliver their courses with relative ease.The eight use cases in Section 4 were developed by the authors of this paper.In practice, time constraints of learning technologists may become the limiting scaling factor.Future work could explore usability enhancements to target educators and students as primary users of the framework.

Lessons Learned
Beyond our specific context of a Haskell-built integration of GitLab and Canvas, we share with the software engineering education research community broader lessons we learned: (1) Open API access to education technology systems enables the development of integrated frameworks, enriched with new features.Providing machine-readable definitions of these APIs, such as an OpenAPI specification, means integrated education frameworks can be more complete, maintainable and robust, and are substantially quicker to develop.(2) Integrating LMSs with industry-standard software development tools from the outset of university programmes exposes students to professional practices.This readies student workflows for their more advanced software development courses and dissertation projects.(3) Educators are more likely to deploy software education technology in their teaching practice if a seamless integration with their institute's LMS is available.
These lessons need to be investigated in future work (1) targeting educators as primary users of integrated programmable frameworks (rather than targeting learning technologists), (2) wider scale usage evaluation with more survey respondents, (3) longitudinal evaluation of student skills with LMS and VCS integration in place and (4) benefits to educators beyond time saving efficiencies.

RELATED WORK 6.1 Git for Software Education
In recent years, web based VCS like GitLab and GitHub have been employed for computing education [7,31].Using these systems in a teaching context can increase the learning outcomes and learning experience of students [14].They provide graduate training by exposing students to industry quality development platforms.GitLab can be customised for deploying coding exercises using tools like RepoBee [16] or Submit 50 [15], which automate repetitive tasks like cloning repositories and updating student projects for marking.One GitLab-based approach [23] goes one step further in making GitLab the sole LMS under the "Courseware as Code" mantra.However, GitLab is not designed to be an LMS and does not have feature parity with widely used general purpose LMSs such as Canvas, Blackboard or Moodle.These LMSs enable the distribution of course material, grading, and monitoring student activity through Learning Analytics [1], helping educators identify at-risk students and improve course content [20].

Benefits for Course Organisation
Prior to web based VCSs, students and educators would exchange source code by email or zip files, and formative feedback would happen in cumbersome or undocumented communication steps.One of GitLab's main advantages is the ease of distributing assignments to large rosters of students who are potentially spread over multiple campuses [31].Students appreciate the convenience of bug tracking and git support in modern code editors [12], and web based VCSs allow students to build a software portfolio to future employers [31].It also provides students a facility to manage their code assignments and collaborate on group work.

Benefits for Learning Analytics
Learning Analytics provide insights to improve the student experience and increase performance [20], for example, to identify students with low engagement.With this knowledge, instructors can counteract any problematic behaviour early on [6].Git data can elevate Learning Analytics [3], where every interaction with a GitLab repository is recorded.
This transparency of student activity can be used to make predictions about students' success [11].Git metrics such as the total number of commits and number of commits per day have been used for predictions on student performance [13].Visualisation tools have also been developed to evaluate contributions to git projects [21].The digital footprint of student activity can be useful for metrics to detect students at risk [2].For example Git4School [22] is a monitoring platform for educators that uses student git repositories and temporal contextual information.Other student performance predictors use data mining and Learning Analytics e.g.[19].

CONCLUSION
This paper addresses the data fragmentation and user experience problems when web based VCSs are used independently to general purpose LMSs.We present a high level programming framework that integrates the Canvas LMS with the GitLab VCS as a solution to these issues.Our methodology uses student and educator surveys, related work recommendations and our early GitLab experience to identify eight integration use cases.These use cases increase productivity for educators by an average of 16.7 hours over a 12 week semester.Feedback from students and educators is wholly positive about these use cases.The components of our framework are open source to allow other learning technologists to extend our use cases and develop their own.

2. 2 . 1
Educator Pre-Survey.We received responses from 7 educators who have prior experience of using both systems.The survey asked about their processes for (1) setting up assessed work, (2) marking assessed work, and (3) identifying non-engaging students.The responses highlight the tedious process of recording data about each student's individual activity on GitLab.Respondents stated: I connect to each [GitLab] project.I record data about individual activity (commits, issues, . . .).

Figure 1 :
Figure 1: Likert Chart of Student Pre-Survey

Figure 6 :
Figure 6: Submission Logs on Canvas

Figure 10 :
Figure 10: Client Side Use Case Algorithms

Table 1 :
Use Cases Overview Canvas section is a student group at one campus, allowing this use case to run for one or multiple campuses, providing flexibility if campuses have different deadline dates.
(b) Submission Logs with server-side matchIf API

Table 2 :
Results of the Lecturer Post-Survey (7 responses)