Visualizing Intelligent Tutor Interactions for Responsive Pedagogy

Intelligent tutoring systems leverage AI models of expert learning and student knowledge to deliver personalized tutoring to students. While these intelligent tutors have demonstrated improved student learning outcomes, it is still unclear how teachers might integrate them into curriculum and course planning to support responsive pedagogy. In this paper, we conducted a design study with five teachers who have deployed Apprentice Tutors, an intelligent tutoring platform, in their classes. We characterized their challenges around analyzing student interaction data from intelligent tutoring systems and built VisTA (Visualizations for Tutor Analytics), a visual analytics system that shows detailed provenance data across multiple coordinated views. We evaluated VisTA with the same five teachers, and found that the visualizations helped them better interpret intelligent tutor data, gain insights into student problem-solving provenance, and decide on necessary follow-up actions - such as providing students with further support or reviewing skills in the classroom. Finally, we discuss potential extensions of VisTA into sequence query and detection, as well as the potential for the visualizations to be useful for encouraging self-directed learning in students.


INTRODUCTION
Intelligent tutoring systems [24] are a type of computerized educational technology that provides students with individualized practice problems, correctness feedback, and hints.Leveraging AI models of expert knowledge and student learning, intelligent tutors estimate student knowledge from their interactions with the system to deliver personalized instruction.While they have been widely demonstrated to improve student outcomes [2,23,33], few studies have looked at how teachers might be supported in better integrating intelligent tutoring systems into their curriculum and course planning in ways that enable responsive pedagogical practice [26,32,34,37].
Closely related to intelligent tutoring systems are other online learning technologies such as learning management systems (LMSs) and massive open online courses (MOOCs).In prior studies, researchers have looked at how these online learning technologies might be supported through visualizations [11,13,51].These visualizations, such as learner trajectories, help analyze how students navigate through course content [18,29], and have been advocated as a useful means to both guide student-centered teaching [43,45] and optimize online course design [15].Specific to intelligent tutoring systems, other studies have looked at how visualization dashboards might be developed to help teachers analyze student data [1,3,21,22,39,47].However, the majority of these dashboards were focused on aggregate measures.In contrast, the data collected by intelligent tutors are often fine-grained logs of student interactions and estimated skill mastery as they work through problems in the tutor.There is thus potential for visualizations of problem solving trajectory (i.e. the history or "provenance" [35] of steps taken to solve a problem) to help teachers better understand how students use intelligent tutors.These insights can in turn allow teachers to better integrate tutors into their curriculum, and tailor their responses to individual student needs during classroom instruction.
In this paper, we leverage the fine-grained interaction logs collected by intelligent tutors to address new design opportunities structured around three central questions: 1) How can we visualize the provenance of student problem solving processes and overall performance?2) How can we help teachers understand and operationalize this data?3) How might teachers adjust classroom instruction to respond to individual student needs?
We work with Apprentice Tutors, a web-based intelligent tutoring platform currently deployed in university classroom settings.Apprentice Tutors collects data on how students work through problems, their interactions, estimated skill mastery, correctness, and the time spent solving problems.Collaborating with five teachers at a local state university who have deployed the tutor to their students, we conducted a design study [38] to introduce them to the data collected and understand their user tasks and goals.Based on participant feedback, we identified three user challenges centered around: summarizing student engagement, identifying issues with each problem-solving step, and comparing student performance over time.These challenges then guided our design of the VisTA visual analytics tool.
VisTA has four main views: the Overview, Student view, Problem Type view, and Details view.Collectively, these views enable teachers to see aggregate metrics of student engagement, identify student problem solving strategies, compare problem solving provenance between different problem types and students, and identify skills students commonly struggle with.We conducted a qualitative evaluation of our system with the same five teachers to assess its usability, effectiveness, and the insights generated.From their feedback and suggestions, we found that VisTA helped teachers better understand student problem solving processes and decide on the necessary follow-up actions -such as identifying when students require further support or which skills should be reviewed in the classroom.Finally, we also discuss possible extensions of VisTA into sequence query and detection, as well as the potential for the visualizations to encourage self-directed learning in students.
In summary, our contributions are: i) a characterization of the challenges for teachers interested in using intelligent tutor data in their classroom planning, ii) the VisTA system, designed to visualize student interactions with the Apprentice Tutor intelligent tutoring system, and iii) a user evaluation with five teachers demonstrating how VisTA helps them better interpret intelligent tutor data, and gain insights into student problem-solving provenance.Taken together, these insights from VisTA can help teachers better integrate tutors into their curriculum, and tailor their responses to individual student needs during classroom instruction.

RELATED WORKS 2.1 Visualizing Provenance for Online Learning
Many online learning systems, from Massive Open Online Courses (MOOCs) to learning management software (LMSs), increasingly deliver more sophisticated educational content to learners from diverse backgrounds [31,40].These systems collect rich data about student interactions and performance, which have in turn motivated the development of visual analytics tools that help teachers make sense of this data and gain insight into how students learn (see [11,13,51] for recent surveys of visual analytics in online learning).Of these, visualizations of learner trajectories have been widely adopted for understanding student activity and how students navigate through course content [18,29].Learner trajectories are paths by which learning takes place, and have been advocated as a useful means to guide student-centered teaching [43,45] and optimize online course design [15].However, most visualizations of learner trajectories have focused on when and how students access course content.They do not account for the types of fine-grained interaction data collected by intelligent tutoring systems, which are more concerned with student problem solving provenance and skill mastery.
Prior work have found that provenance visualizations that depict the history of user actions can help us better understand user goals [12,49], evaluate their performance [4,30], and assess their analytic strategies [12,35].In particular, user interactions, defined as "the history of user actions and commands with a system" [35], were a common type of data typically captured for provenance visualization [4,7,16,42,49].Although diverse visualization types have been proposed for provenance visualization in a range of application domains (for recent surveys, see [49] and [4]), there are, to the best of our knowledge, no existing visual analytics systems for exploring student problem solving provenance based on interaction data collected by intelligent tutoring systems.

Visualizations for Intelligent Tutors
Intelligent tutors are computer-based systems with two key models: (1) a knowledge-tracing model that estimates what a student knows, enabling personalized recommendations about which skills need more practice; and (2) an expert model that can provide learners with contextualized hints, worked examples, and correctness feedback [44].Benefits of these tutors have been widely acknowledged by researchers and educators over the years [2,23,33].For instance, an intelligent tutor for algebra problem solving, PAT, was used in the Pittsburgh Urban Math Project (PUMP), and demonstrated one of the largest intelligent tutor deployments within a school system [23].These systems are also well liked by teachers because they provide individualized scaffolding and guidance to learners as they practice problem solving, allowing teachers to allocate time to other pedagogical tasks.
However, only a few intelligent tutors have made efforts to visualize student analytics as dashboards for teachers [1,3,21,47] or students [22].Findings from these prior works demonstrate that dashboards helped teachers develop their knowledge of individual students, and influenced their lesson plans [1,47], suggesting that visualizations of intelligent tutor data can help teachers better understand and respond to students' needs during classroom instruction.However, the majority of these dashboards only visualized aggregate data such as average skill mastery and average time spent on practice.In cases where stepwise data was available, such as in the Tutti dashboard [1], each step could only be viewed as complete interface snapshots or replays, which can make it challenging for teachers to compare across students and summarize common patterns in problem-solving approaches.Similarly, the Many Eyes Word Tree graphic proposed by [39] used stepwise data to depict sequential patterns of student states using a word tree diagram.Although this was helpful for algorithmic pattern discovery and prediction tasks, the graphic was not ultimately presented to students or teachers.
In this paper, we build on existing studies to develop visualizations of student interactions as they work through problems in the Apprentice Tutor platform.We aim to provide insight into detailed student problem solving provenance and skill mastery.In our evaluations, we validate that these visualizations can help teachers better integrate tutors into their curriculum, and tailor their responses to individual student needs during classroom instruction.

APPRENTICE TUTORS
Since many different intelligent tutors have been developed to date, each catering to different topics, courses, and users, we decided to focus our study on a specific intelligent tutor, Apprentice Tutors1 .The system currently provides access to four mathematics tutors: Factoring Polynomials, Exponents, Radicals, and Rational Equations.The system allows learners to engage with the content from any web browser, offering a convenient and flexible learning platform.The Apprentice Tutors uses Bayesian Knowledge Tracing (BKT) [9], a widely used knowledge tracing approach, to estimate students' mastery of each skill.Using these skill estimates, the system adaptively selects problems for the student to practice next that will enhance the student's learning.
Upon entry, students can select one of four tutors.For each tutor, they can either opt for a specific problem type or an adaptive problem approach.For example, the Factoring Polynomials tutor has multiple problem types including: "leading coefficient of 1", "grouping method", "slip and slide method", and "area box method".The four problem types provide students with guidance on how to factor polynomials using various methods.After selecting a problem type, a problem is dynamically generated for the student.
Adaptive problem selection is a core capability of the Apprentice Tutors that personalizes the student's learning experience by targeting skills that the student has yet to master.Skills within the Apprentice Tutors are referred to as knowledge components (KCs), defined as acquired units of cognitive function that can be inferred from performance on a set of related tasks [25].Each step in a problem is associated with a skill or KC label, and all problems of the same problem type have the same steps, which usually occur in the same sequence.When students correctly complete a step associated with a KC, the system updates the associated student mastery estimate.If students choose adaptive problem selection (instead of a single problem type), the tutor adaptively selects problem that exercise unmastered skills until the system estimates that the student has mastered all of the relevant KCs.
Figure 1 (Bottom) illustrates a dynamically generated "leading coefficient 1" problem, accompanied by two rows of captured usage data (Fig. 1 Top) that correspond to two consecutive interactions.In this example, the problem is 2 − 5 + 4. In the first two steps, students must practice skills (or KCs) for determining the  and  values from the polynomial.In this example, the student has correctly input that  = −5, but incorrectly input that  = −4.
The system records the time spent and the correctness for each interaction.If an interaction is incorrect, the student can make multiple attempts until their input for that step is correct.They can also request hints by using the Ask for Hint! button.Incorrect answers and hints decrease a student's estimated skill mastery.The Apprentice tutors offer a three-tiered hint system: the first points to the required input box, the second rephrases the question, and the third provides the answer.Students must provide the correct input for all steps, arrive at the final factored answer of ( − 4) ( − 1), and click the Done button to complete the problem.A problem where all steps are correctly solved is considered "complete", otherwise it is considered "incomplete" (e.g. if the student stops halfway).Hint requests for each step are also recorded.

DESIGN STUDY
To inform the design of the Apprentice Tutors visualizations, we conducted a formative focus group study session to understand how educators currently use intelligent tutors in the classroom, as well as how they would like to use and analyze intelligent tutor data.We recruited five teachers (P1-5, 4F/1M) from a local state university 2 who have deployed the Apprentice Tutors in their classrooms for at least a semester prior to our focus group.Participants were compensated with a $25 gift card for their time.The sharing of anonymous student data and the details of the focus group protocol were approved by our Institutional Review Board (IRB) prior to the study.Participants also provided their documented consent before the session.
The focus group took place virtually, and all participants were present at the same session.Since none of the teachers had accessed the Apprentice Tutor data prior to the focus group, we first introduced them to the data set (Fig. 1).The data was anonymized beforehand, and no identifiable information about students was shared.The session was structured around the following questions: -Do you have any questions about the data?-Is there anything in the data that is more interesting/useful?-Is there anything in the data that is not interesting/useful?-Is there anything else you want to see in the student data?-Outside of Apprentice Tutors, do you typically collect and analyze data about your students?-(If yes:) What are your processes and tools?We asked follow-up questions where appropriate.The formative study lasted for about an hour.

Design Challenges
Based on findings from our focus group, we used an affinity diagramming process to identify main themes in user needs and requirements.This was supplemented by findings from prior works [1] to derive three design challenges that should be addressed in our visual analytics system.C1 Summarizing student engagement.Our participants described the need for an intuitive summary of student engagement in terms of which problems were being attempted and aggregate performance metrics.As P4 described, "Overall, I need to know how many [problems] students answered correctly and how many incorrect."P2 also mentioned that she wanted to know the names of students who use the tutor, and the frequency of their participation.Our interface should allow teachers to quickly identify which of their students are engaging with the intelligent tutors and rapidly evaluate student accuracy for attempted problems.
C2 Identifying specific issues at each problem-solving step.Participants wanted to understand how well their students understood each knowledge component or problem step.For P2, her current practice at the end of each topic was to ask students to "go back and look at it and see where you guys get lost" and provide feedback for places where they need revision.Overall, teachers wanted to identify and focus on common steps their students found challenging.Thus, our visualizations should help teachers understand step-wise problem-solving provenance across all students in order to identify challenging steps that need further attention.
C3 Comparing students' performance over time.Finally, participants also wanted to look at performance data for students in context of time spent.Teachers mentioned that they were currently using activity logs to understand "how hard [students] are working and how much time they're dedicating" (P3).They also found that there was "a correlation between how much time [students] are spending on practicing the homework and their grade" (P3).As such, our system should visualize both total time spent as well as detailed temporal break downs of each step.

THE VisTA SYSTEM
Based on the challenges identified from our focus group session, we designed the VisTA (Visualizations for Tutor Analytics) system.VisTA is an analytics tab that teachers can access through the Apprentice Tutors platform.All visualizations are implemented in JavaScript and D3 [5].In the following sections, we describe the main views in VisTA and walk through example use cases.When VisTA is deployed, teachers using VisTA will be able to see student names.However, we omit names and any other identifiable information in screenshots throughout the paper to preserve anonymity.

System Implementation
5.1.1Overview.When teachers access VisTA, they will first see a horizontal stacked bar chart of attempted problems broken down by problem type (Fig. 2).This histogram provides an overview of student participation (C1), and allows teachers to quickly identify problem types that students have practiced more.Each bar in the histogram is further divided into segments representing the proportion of practice questions that were completed correctly versus  those that were incorrect or partially complete (C1).There is a third group of skipped questions that is deselected by default.Skipped questions are those that were started but which the students did not interact with (i.e. the student did not attempt any steps of the question).teachers can click on the respective labels in the legend to include skipped questions in the overview histogram.From Fig. 2, we can see that problems about exponents (exponent_product_rule and exponent_quotient_rule) were the most attempted problem types and students completed them accurately.Comparatively, students tended to be more inaccurate on problems about radicals.

Student View.
From the drop-down menu on the left, teachers can select a particular student in their class to view all problems attempted by that student (C1) in a single horizontal stacked bar chart (Fig. 3).Each problem is divided into its component steps by black lines, and each step is further divided into separate interactions.We use the same color scheme as the feedback provided in the tutor interface (Section 3), where green segments represent a correct response, red segments represent an incorrect response, while yellow segments indicate when students asked for a hint (C2).The same color palette is used for all visualizations in VisTA, and was selected to be colorblind safe.Hovering over a segment shows a tooltip with details of the interaction, such as the KC associated with the step and student input value (C2).An asterisk (*) indicates that the student completed a problem correctly, otherwise, the problem was incomplete or not attempted (C2).In this visualization, the width of each segment represents the duration of each interaction, while the total width of all segments reflects total time spent by the student on the problem (C3).

Details view.
Finally, teachers can click on a single path in the Problem Type view or use the drop-down menu to see the details of how a single student attempted a single practice problem (Fig. 4, Right).This view is a combination of the stacked bar chart from the Student view and the step line chart from the Problem Type view for a single problem.In this visualization, the steps for the problem are listed on the y-axis and the time spent is represented on the x-axis.As a student works through a problem, the line descends from the top left corner to the bottom right corner, with each step broken down further into detailed interactions and correctness.Taken together, this Details view provides information on accuracy, time, and interaction sequence (C2, C3), allowing teachers to not just evaluate how well students can solve a problem, but also identify where mistakes occur.

Use Cases
In the previous sections, we have described the visualizations in VisTA and how they address the challenges surfaced during our focus group session.We now walk through three use cases demonstrating the insights that teachers might gain from using VisTA.The Problem Type view uses a step line chart to depict all student attempts of a particular problem type.In this example, we see that while most attempts were completed in under 5 minutes, there was one student attempt that took more than 15 minutes and was left incomplete.Right: Clicking on a path brings up the Details view, which shows the breakdown of each step.Here, the student appears to have made multiple mistakes on the "new_first_expression" and "new_second_expression" steps before leaving the problem incomplete.

Understanding student engagement.
Ana is a math teacher who has made the Apprentice Tutors available to her students who want extra practice on the topics covered in class.She is now interested in whether the tutors have been helpful to her students, so she navigates to the Analytics tab of the Apprentice Tutors platform, and begins exploring the visualizations available.From the Overview (Fig. 2), she sees that her students have practiced a large number of problems about exponents.Comparatively, problems about factoring and radicals received less practice, and there was a greater proportion of inaccurate responses for those tutors.

Understanding skill mastery from practice.
Recalling that there is a student in her class who has been doing well on factoring, Ana further investigates whether the student has been practicing with the Apprentice Tutors.She selects their name from the dropdown menu and, from the Student view, finds that the student has indeed practiced a large number of problems related to factoring (Fig. 3).She notices that the student had struggled on the first two questions, making multiple inaccurate inputs.However, they appeared to learn from the practice and solved three subsequent problems quickly and accurately.Ana then notices that while the student was able to complete all factoring problems when the first coefficient was 1, as the Apprentice Tutors system started introducing problem types with new KCs (polynomials with first coefficient >1), those problems were left incomplete.Ana makes a note to follow up with the student to see if they need help practicing skills to solve such problems.She then decides to look at how other students have performed on the same problem type to see if this is a common challenge she needs to address in her upcoming class.

Comparing usage patterns and problem solving strategies.
When looking at the detailed interactions of another student in her class using the Student view, Ana comes across an interesting strategy (Fig. 5).She realizes that this student has been using multiple hints to help them solve problems.Majority of the hint usage is concentrated in the first problem, and as the student gained familiarity with the problem type, the number of hints requested also decreased.She then contrasts this visualization with the student from Fig. 3.She realizes that the previous student rarely used hints, even when hints might have helped them correct inaccuracies and progress with the problem (Fig. 3).Ana makes a note to remind students of the hint feature.She also wonders if the tutors can be improved by automatically prompting students to request hints based on detected student behavior.

EVALUATION
We conducted a second focus group session and presented VisTA back to the same group of five teachers (see Section 4) for feedback and evaluation.Participants were compensated with a $25 gift card for their time.The sharing of anonymous student data and the details of the focus group protocol were approved by our Institutional Review Board (IRB) prior to the study.Participants also provided their documented consent before the session.
During the session, we first demonstrated VisTA and all visualization views to the teachers.All student names were omitted from the demonstration to preserve student privacy.The subsequent discussion was broadly structured around the following questions: -Is there anything in the visualization prototypes that is more interesting/useful for you?-Is there anything in the visualization prototypes that is not interesting/useful for you?
-Is there something you would like to know that is currently not shown in the visualizations?
During the session, we followed up on the feedback provided, particularly comments about how teachers might use the visualizations, any visual patterns they found interesting, and areas for improvement.Finally, teachers were invited to ask any questions they had of us.The evaluation focus group lasted around an hour.

Results
Overall, participants found VisTA to be useful.We used an affinity diagramming process to categorize their feedback into two major themes that emerged during the focus group: tailoring responses to student performance, and identifying steps in a problem that require more classroom instruction.
6.1.1Visualizing provenance helped supplement summary metrics and motivated nuanced responses to student performance.Teachers in our study found particular value in being able to see the details of the student problem-solving provenance (Student view, Problem Type view and Details view) combined with summary statistics about the overall accuracy (Overview).By breaking down each problem into individual steps and interactions, it was "useful to see what part of the process that they get tripped on" (P3).It also helped teachers identify if and when a student needed more assistance.As P1 described, if it was just one step with an inaccurate input, "it might be a goofy mistake... but if I see red and red and red, then I need to contact the student and send them to tutors... we need to figure out what's going on".Similarly, P3 mentioned that a single inaccurate response may not mean that "they're lacking the skills to complete the process, maybe it's that they don't know how to [use the intelligent tutor]".In such cases, a student may make mistakes on problems in the beginning, but show improvements in their performance on subsequent questions as they gained familiarity with the tutor interface (such as in Fig. 3).As such, by inspecting the stepwise provenance of how a student solved problems, teachers can gain a more nuanced interpretation of when and why students make mistakes, and tailor their response to different students appropriately.As P2 summarized: "I think it's good to see all the students together, but it's also good to see one student and how they do all over on all the tutors that they use".
6.1.2Aggregating and comparing solution paths helped highlight steps (KCs) that require more instruction.Teachers in our evaluation study found the Problem Type view to be particularly useful.By visualizing the details of multiple student attempts on the same chart, the system helped call attention to the steps and KCs that need more instruction.Teachers can see "how many students are getting the middle part [and] which part I need to focus on it more" (P2).This helped them review their teaching and proactively consider how they might address the common challenges their students face.In some cases, it might also prompt reflections on the design of the intelligent tutor and how it might be improved.As P5 described, her insights after viewing the visualizations were such that "if several students are getting hints or missing in the same place, I would go back and look at the tutor and see if it's something in the tutor, or is it that they don't understand the terminology, or do I need to reteach that area.It makes me think all three of those." Taken together, we see how the Problem Type view, in particular, provides valuable insight into how students interact with the intelligent tutor, helping teachers reflect on both tutor design and individual student needs.

DISCUSSION
In addition to providing feedback on the usability of the VisTA system, the teachers in our study also raised interesting avenues of future work, where VisTA might be extended to support new features and user groups.

Temporal Sequence Query and Detection
During the evaluation study, teachers agreed that not all patterns or errors in the visualization were equally important.For example, a single inaccurate input could be an accident, but other patterns such as a sequence of red-red-red inputs would indicate that they need to follow-up with the student (see Section 6.1.1).Furthermore, a teacher who sees a concerning pattern of interactions may find it helpful to identify other students who also display this sequence.This suggests one promising direction of future work into the automatic highlighting of interesting data-facts and salient event sequences.Teachers can input user-defined temporal sequence queries to search for patterns of interest that are then brought to their attention.For instance, while one teacher may be concerned about inaccuracies, another may be more interested in how their students have been using the hint feature.This user-driven query function can complement manual data exploration, helping to surface insights specific to different user contexts.Existing work into temporal sequence analysis [17,19,20,27,46,50] and automatic data-fact generation [6,8,10,28,41] can provide guidance for how VisTA might be extended to automatically detect and visualize teacher-defined data facts.In future work, we plan to build on these existing studies to implement a temporal sequence query feature to help teachers define and automatically highlight interesting patterns in intelligent tutor data.

Supporting Self-Directed Learning
Multiple teachers in our evaluation study also raised the potential for VisTA to be useful for their students as well.P2 mentioned that the visualizations might be "more useful for the students than [for] the teachers" because the visualizations could help students identify where they went wrong.Seeing their accuracy for each step of a question might also lead students to reflect that "maybe I was on the right track, but I'm missing some part of this" (P3) or that "they made a mistake [and] have to review that part before the test" (P1).This feedback is supported by theories of self-directed learning where learners are encouraged to set their own goals, monitor their performance, and seek out feedback [14,36].Prior work implementing such visualizations had demonstrated that showing students visualizations of their own performance prompted them to be more reflective of their behavior and reduced attempts to game the system [22,48].These studies also found that visualizing step-wise temporal information and peer-related data were important factors that influenced self-reflection.This suggests that many of the metrics and views in VisTA may be similarly effective for supporting self-directed learning in students.Future studies can further explore this by expanding visualization design and development to include students using the Apprentice Tutors as well.

Limitations and Future Deployment
Due to scheduling constraints and teaching commitments, we tried to be mindful of the teachers' time when planning for the design study and conducted the requirements gathering and evaluations through two focus-group sessions instead of individual interviews.Furthermore, since the same group of teachers participated in both sessions, it is likely that VisTA may not generalize well to other teachers and courses.To address these concerns and to ensure that teachers obtain hands-on experience with VisTA instead of simply viewing a demonstration, we have currently deployed VisTA to all teachers at our partner institution.Based on this long-term deployment, we plan to conduct a more in-depth evaluation of how, and to what extent, the VisTA system helps teachers adjust their classroom instruction, and whether this results in measurable outcomes in student classroom performance.

CONCLUSION
In this paper, we have provided a characterization of the challenges for teachers interested in integrating intelligent tutor data in their teaching.Based on these challenges, we designed the VisTA system to visualize logs of student interactions with the Apprentice Tutor intelligent tutoring system.VisTA is, to the best of our knowledge, the first analytics tool developed to visualize student provenance data from intelligent tutors.In a user evaluation with five teachers, we demonstrated VisTA helps them better interpret intelligent tutor data, gain insights into student problem-solving provenance, and tailor their follow-up responses to student needs during classroom instruction.We have currently implemented VisTA on the Apprentice Tutors platform.In future work, we plan to look at how VisTA is used in a long-term deployment over the course of a semester.We also plan to examine how VisTA might be extended to support context-dependent sequence detection for teachers and self-directed learning in students.

Figure 1 :
Figure 1: Top: An example of two rows of data collected by Apprentice Tutors.Bottom: How data columns map to different parts of the system.Each row records a single interaction by a student.From left to right, each interaction has 11 attributes: ID, userID (anonymized student ID), tutor, interface (problem type), start_state (problem), selection (input box/step being attempted), action (interaction type), input (student input), correctness (correctness of input), kc_labels (knowledge component/skill associated with this step), and time (recorded time of student interaction).

Figure 2 :
Figure 2: The Overview visualization showing proportion of correct (green), incorrect (red), and skipped (gray) questions attempted by students for each problem type.Users can click on the respective labels in the legend to select and deselect the groups they would like to include in the histogram.The control panel on the left can be used to select a particular problem type or student to visualize.

Figure 3 :
Figure 3: The Student view showing all problems of a particular problem type attempted by a single student.We can see that the student spent the most time and made the most mistakes on the first two problems, but solved subsequent problems quickly and more accurately.When new problem types with new knowledge components (solving polynomials with first coefficient >1) were introduced, the student started to make more mistakes and left the problems incomplete.

5. 1 . 3
Problem Type view.From the same drop-down menu on the left, teachers can also select a particular problem type to analyze.For each problem type, we visualize all student attempts in a single step line chart (Fig. 4, Left).The steps of the problem are listed along the y-axis from top to bottom.Each student attempt is represented by a single blue line.The amount of time spent on a step (C3) is visualized along the x-axis, and corresponds to the length of the horizontal line segment.As a student works through a problem, we expect to see their line descend from the top left corner to the bottom right corner, demonstrating progression through the problem (C1).Lines that terminate in the middle of the chart indicate an incomplete attempt.

Figure 4 :
Figure 4: Left:The Problem Type view uses a step line chart to depict all student attempts of a particular problem type.In this example, we see that while most attempts were completed in under 5 minutes, there was one student attempt that took more than 15 minutes and was left incomplete.Right: Clicking on a path brings up the Details view, which shows the breakdown of each step.Here, the student appears to have made multiple mistakes on the "new_first_expression" and "new_second_expression" steps before leaving the problem incomplete.

Figure 5 :
Figure 5: Student view for a student whose interaction patterns show that they frequently asked for hints while working through the problems.