Behind the Screens: Exploring Eye Movement Visualization to Optimize Online Teaching and Learning

The effective delivery of e-learning depends on the continuous monitoring and management of student attention. While instructors in traditional classroom settings can easily assess crowd attention through gaze cues, these cues are largely unavailable in online learning environments. To address this challenge and highlight the significance of our study, we collected eye movement data from twenty students and developed four visualization methods: (a) a heat map, (b) an ellipse map, (c) two moving

A user study with 13 teachers evaluated the visualizations' impact on social connectedness with students, as well as their perceived usefulness, usability, and cognitive demand.

INTRODUCTION
In 2020, the Covid-19 pandemic triggered a massive shift in the educational landscape, as institutions worldwide were forced to transition to online learning [22].As online learning environments continue to gain popularity and maintain a significant share in education, designing effective online learning experiences becomes a pressing concern [11].This issue extends the challenge of maintaining students' attention beyond traditional classroom settings.
A key factor in student engagement, retention, and learning outcomes is the quality and extent of teacher-student interactions.However, these interactions are limited in online environments due to the absence of cues such as raised hands or facial expressions [43].Additionally, online learning demands that students self-regulate their attention, which may prove challenging [37].Thus, a method to visualize students' attentional states using gaze information could be invaluable, allowing teachers to identify where students are focusing their attention even in an online setting.
Previous research has explored various methods to provide student feedback, such as MudSlide for detecting confusion [16] and Electroencephalography (EEG) for attention monitoring [2,38].While these methods offer indirect attentional feedback, eye-tracking technology provides direct insight into visual attention.Moreover, Sun et al. demonstrated the feasibility of presenting dynamic learning states to presenters in real-time [43].Consequently, integrating eye-tracking technology into online learning environments could significantly enhance teacher-student interactions and improve the overall effectiveness of online education for the human-computer interaction community.
Understanding not only the average gaze position of an entire audience but also the variability of students' gaze is crucial for gauging crowd attentiveness in human-computer interaction contexts.Heat maps are widely recognized and intuitively understood as visual representations of eye movements, often serving as a standard approach in gaze visualization for individuals over time [4,5,7,32,33].Alternatively, less intrusive information about the audience can be provided using bars at the edge of the screen [43].However, this visualization lacks spatial information, necessitating additional mental processing for interpretation.
As heat maps can be intrusive and obscure content, and bars or other abstract representations may not provide sufficient information, we designed and evaluated two intermediate visualizations: (1) an ellipse (Figure 1 (b)), a 'less cluttered' version of a heat map that displays average position and variances in students' gazes, and (2) moving bars (Figure 1 (c)), with a vertical bar on the right and a horizontal bar at the bottom.Both heat maps and ellipse maps create spatial overlay, partially hiding content for the teacher.We compared these visualizations with well-established heat maps (Figure 1 (a)) and bar plots (Figure 1 (d)) to determine the most suitable visualizations for live online sessions and retrospective analyses within the human-computer interaction domain.By optimizing gaze visualization techniques, we aim to improve educators' understanding of student attention in online learning environments, ultimately enhancing the overall learning experience.
This work aims to use gaze tracking for the direct assessment of crowd attention and reveal the optimal trade-off between being informative and distracting for communicating crowd attention in online teaching scenarios.
In this study, we aimed to enhance the understanding of crowd attention in online educational settings.To achieve this, we collected gaze data from twenty students as they watched educational presentations.Using this data, we implemented and evaluated four visualizations of crowd attention, presenting them to the thirteen respective lecturers for assessment.Our evaluation focused on various aspects, including subjective connectedness, perceived usefulness, usability, cognitive demand for lecturers, acceptance of the visualizations, and their ability to foster a connection with the audience.The results revealed consistent positive assessments from lecturers for concrete on-slide visualizations.However, the heat map was also perceived as distracting.Consequently, we propose that the ellipse map represents an effective compromise between fostering audience connection and minimizing distraction.By collecting and visualizing students' eye movements, we can provide a powerful tool for educators to better perceive crowd attention and feel more connected to their students.This visualization tool can be beneficial for retrospective analyses to improve visuals and delivery of online lectures, during live online teaching or other presentations, and for further research on the effects of crowd attention within the human-computer interaction domain.
In summary, the contributions of our work are: (1) We visualize students' crowd attention based on measures of central tendency and variability of students' eye movements to improve online teaching.
(2) Based on an online webcam-based eye tracking study with 20 students, we developed and implemented four different visualizations of crowd attention, using established visualizations (heat map, vertical bar) and new suggestions (ellipse, moving bars).
(3) In a study with 13 lecturers, we evaluated the subjective connectedness, perceived usefulness, usability, and cognitive effort of our four visualizations overlaid onto their instructional video, indicating that the ellipse visualization can establish an optimal trade-off between concrete and abstract visualizations.

RELATED WORK
Our work is mainly related to three fields of research.First we discuss how gaze information shapes interaction, then we detail technologies and sensors that help to foster a teacher-student connection, and lastly we discuss visualizing eye-based student feedback.

How Gaze Information Shapes Interaction
Learning can be inferred from gaze [6,21].Variations in fixations and saccades, as well as other temporal patterns in eye movements, may reveal the attentional foci of student interactions with a learning environment [13].In technical setting, eye gazing was used to imbue virtual agents with the ability to capture attention, retain interest, and increase conversational flow with human users [8].Gaze information is not only important in human interaction, but also in human machine interaction.The eye-tracking technology offers educational researchers a viable way to link learning outcomes to cognitive processes [24].Eye tracking could also be used for multimedia learning [12], reading processes [1] and even psychological disorders [17].Video conferencing platforms have become popular in recent years.People are still learning how to use them efficiently.
Studies have also investigated ways to enhance the usage of video conferencing tools [27,30].Providing the power of eye tracking data, it could be useful to measure eye movements and provide feedback to the presenters who are using video conferencing tools or online teaching platforms.In our work, we focus on the crowd attention concept, which informs the teacher about the attention of a crowd.This is different than the idea of joint attention that could involve several cognitive skills and processes [41].

Teacher-Student Connection in Video Conferencing Scenarios
Our aim is to convey student feedback to the lecturers such that they get an overall sense of the students' crowd attention without requiring explicit actions of the students.Therefore, we discuss systems that focus on the use of physiological sensors to build a connection between audience and presenter in both real-life and video conferencing settings.Our work faces the specific challenge that in an online lecture scenario the teacher typically cannot see the audience, because students often turn off their cameras or their videos are shown in a small gallery view, which makes it difficult to see details.Therefore, visualizing students' attention is the main channel that conveys this attention to teachers.
To establish a connection between audience and presenter during live-stream lectures, Sun et al. present a system that predicts flow-related psychological states [43].The system interprets the flow-related states boredom, flow, and anxiety based on facial expressions of the audience.This information is then aggregated and visualized to the presenter using a bar plot and a line chart.The system was evaluated with eight participants, who each gave a 40-min lecture about a topic of their expertise.The system was perceived positively by the presenters, noting that it helped them find problems in their lectures, and helped them to take adjustments to specific aspects of their content.However, some of them also mentioned that the feedback introduced additional load, as they had to shift their attention to the feedback.Yet, they agreed that the feedback helped to make the online lecture "more similar to traditional real-world teaching".With AffectiveSpotlight, Murali et al. follow a different approach [28].Instead of making the virtual lecture more real-world-like, they explore putting an artificial spotlight on selected members of the audience, following the idea of television shows where selected audience members are shown to convey a certain emotion.Using facial expression recognition, the system evaluates emotional states of the audience members and selects a salient audience member to show to the presenter to give them a general idea of the audience's current affective state.In an evaluation with 14 presenters, the authors found that the presenters were significantly more aware of the audience when using the AffectiveSpotlight system in comparison to a control condition.
While these systems focus on general audiences and analyze their affective and flow states, we propose the use of eye tracking visualizations to give teachers feedback about the current attentive state of their students.Furthermore, in contrast to visualizing one mean value [43,45] or one person like Murali et al. [28], we visualized the inter-individual variability among students' eye movements.As such, we are able to show more detailed information about the students' attention levels than a mean value without revealing their individual identities.

Visualizing Eye-Based Student Feedback
Heat maps are a widespread technique for visualizing eye-based feedback [5,7,32].They are used as analytic tools to gain detailed information about the visual attention of viewers or an audience [4].In teaching and learning contexts, heat maps are often applied for retrospective analysis, e.g., to analyse students' gaze behavior in massive open online courses [40] or for the analysis of the usability of learning material [9].To the best of our knowledge, investigations that explore to what extent heat maps can be used for real-time analysis of students' visual attention are currently missing.The previous works on physiological sensing of audience feedback that were discussed in the previous sections, used simple bar or line graphs to visualize audience feedback [18,43].
One recent study investigated a similar research question: The authors collected eye movements in a webcam-based study [35], created and evaluated four different visualizations [19]: a deviation map, a disk map, a horizontal bar and a vertical bar.Their deviation map was special in the sense that it does not portray the mean fixation positions but rather the points in space at which the deviation between eye fixation is highest (based on Isokoski et al. [20] deviation map).However, while the deviation map turned out to be preferred for retrospective analysis, it remained unclear whether the participants truly understood the meaning of the deviation map.
It was also rated as quite distracting.One important caveat of their study was: the participants who evaluated the visualizations were not the lecturers themselves, but students who did not know the content of the presentation before, so their judgements of the visualizations might be skewed because they also tried to attend to the content.
With our work we touch on the aforementioned visualization techniques.We designed and evaluated four gaze visualizations which visualize students' eye movements across the learning content.As for on-content visualizations we used a classical heat map that reveals students' eye movements directly on the content.In contrast, for side-based visualizations, we visualize the variability of students' eye movements as a bar plot, similar to the previous works of Sun et al. [43] and Hassib et al. [18].

RESEARCH OBJECTIVE
This work aims to improve online teaching by visualizing student crowd attention to teachers.Crowd attention is measured using the gazes of the students.Specifically, we designed and implemented four gaze visualizations following hitherto approaches: heat map, ellipse, moving bars and bar plot (see Figure 2).The aim of the following investigation is to judge whether the established methods of gaze (heat map) and cognitive state (bar plot) visualizations are visualizations that are preferred by lecturers or whether the newer 'intermediate' suggestions are viable alternatives.
Concrete representations of crowd attention, such as heat maps [20] and the ellipse, might be easy for the teacher to interpret due to their spatial proximity to the content.I.e., the information about the gazes is shown directly on the presentation material (often presentation slides), helping a direct assessment of crowd attention but impairing the readability of the slides.The moving bars show the same information as the heat maps, but do not overlay content.On the other hand, a compact and abstract visualization of attention values, such as bar plots [18,43], might be less distracting.However, they require more cognitive demand to process and interpret.
This research was carried out in three methodical phases.In the first phase, we selected a set of presentation videos and conducted an online eye-tracking study involving 20 students.During this study, we recorded the eye movements of the participants as they viewed the presentations.In the second phase, we utilized the collected eye movement data to generate four distinct gaze visualizations, each designed to provide insight into students' attention and focus during the presentations.In the final phase, we presented the generated visualizations superimposed on the learning material to the lecturers who had originally conducted the respective presentations.We then solicited their feedback regarding the perceived usefulness, usability, and cognitive demand of the visualizations, as well as their opinions on the potential of these visualizations to foster social connectedness between them and their students.To further explore these aspects, we conducted structured qualitative interviews with the lecturers.

LAB-BASED EYE TRACKING STUDY WITH STUDENTS
For gathering eye movement data for the four distinct gaze visualizations, we performed a lab-based eye tracking study to collect students' eye movement data while they watched short videos.

Online Videos
Thirteen videos were collected from either the Virtual German CHI (Conference on Human Factors in Computing Systems) 2020 playlist or the TeaP (Conference of Experimental Psychologists) 2021.To start with, We pre-selected videos that lasted between 2 to 3 minutes.Due to the availability of the presenters, the final set of videos included videos with a length ranging from 2 to 11 minutes.
We then asked for the video presenters' consent that We could use their videos and whether they were willing to be interviewed by us.All videos were in English.The videos come with different resolutions, and some were with an on-screen instructor.We set the videos resolution to 1280 x 720 pixels and covered the on-screen instructor with a white static image.After the pre-processing of the videos, the eye-tracking data collection proceeded.This data was to be used to overlay visualizations on the videos (see Figure 1).These videos were used for the evaluation of the visualizations with the teacher participants.

Study Design
We collected the eye movement data in an eye tracking lab in Ulm University.The experiment was implemented using OpenSesame [26].This part of the study targeted to collect precise eye movement data of students watching the videos.To keep the study duration short (and not cause fatigue), we divided the 13 videos into two groups.The participant with odd participant ID watched 7 videos, and those with even participant ID watched 6 videos each.
All participants passed the eye-tracking calibration procedure.We aimed to collect eye movement data of at least 10 participants for each video since this is about a representative number of students who join an advanced University seminar.In the end, 23 sessions were conducted.The videos in Group 1 were watched by 12 participants, while those in Group 2 were watched by 11 participants.

Student Participants
We recruited 20 participants (14 female, 5 male, 1 prefer not to say) with an average age of 25.65 ( = 21,  = 35) via email and social media announcements mainly from the postgraduate student population at Ulm University.Inclusion criteria were minimum 18 years of age, good command in English, proof of COVID-19 compliance, and normal or corrected to normal vision.19 of the participants were enrolled as students.

Procedure
The participants registered their participation through terminplaner4.dfn.de.When they reached the lab, they were briefed about the study and signed the consent form.As they sit on the participant chair, we instructed them to make themselves comfortable while putting their chins on the chin rest.Then, they answered a series of demographics questions about age, gender, English level, highest educational level, and whether they are students.Later, the calibration procedure by Eyelink 1000 Plus was performed (only the right eye was tracked).During the calibration, the participants needed to fixate on a black dot in the middle of the screen, followed by eight dots that moved around the screen.The process for validation was similar.After the calibration, depending on which group they were assigned to, they watched the videos in random order.Before each video was played, a drift check was done.After watching each video, two questions about difficulty and comprehension were asked.The total duration of the study was about 40 minutes, and the participants were compensated with 8 EUR per session.

VISUALIZING CROWD ATTENTION
We introduced four distinct gaze visualizations, namely heat map, ellipse, moving bars, and bar (see Figure 2 for an overview of the visualizations).Crowd attention is presented by visualizing the eye movements of students while they were watching the videos.When majority of the students are looking at the same screen position, the variability of gaze points on the screen is low.The variability is high when students looking at different positions on the screen.As the first visualization we choose a heat map, which is a popular technique to visualize eye-tracking data that provides rich information [42].The next visualization, defined as ellipse, is a simplified version of the heat map that also carries spatial information but less distracting than the heat map.Lastly, the bar plot based visualizations adapted from Sun et al. [43] and Hassib et al. [18], represent the variability of students' gaze points.The moving bars provide spatial information about the mean position of gaze points on the screen.In contrast, the vertical bar only visualizes the total variability of the gaze points.

Implementation
The four visualizations were implemented using Python and Processing [15].Visualization frames were generated from students' recorded eye movements while watching short videos, which were then combined and overlaid onto the original videos.Heat maps were created using a modified version of the PyGaze heat map plotter [10,34].In our implementation, colors range from purple to yellow, indicating the concentration of gaze points at a specific location.The ellipse's x-and y-radii are based on the interquartile range (IQR) of x-and y-positions of gaze points on the screen, encompassing the middle 50% of gaze points.This visualization consists of two gray bars: the horizontal bar at the screen's bottom represents the horizontal IQR of gaze points, while the vertical bar at the right margin indicates the vertical IQR.The intersection of these bars' centers marks the mean gaze position.Appearing as a stationary vertical bar on the right side of the screen, this visualization represents the sum of the vertical and horizontal IQRs of gaze points but does not provide spatial information.After overlaying the videos with gaze visualizations, we conducted a user study involving 13 teachers who viewed the selected videos.The study aimed to evaluate their experiences with each visualization, measuring perceived usefulness, usability, mental effort, social connectedness, and ranking of the four gaze visualizations.

Study Design and Procedure
The study design and procedure involved informing participants about the online study and subsequent interviews via email.Participants booked a time slot using an online scheduling platform.Teachers joined the study through a link, provided informed consent, and were introduced to the study and its procedure.They then answered a pre-questionnaire about their lecture experience before watching their own video presentations, overlaid with one of the four gaze visualizations.Teachers watched their video four times in total, each time with a different, randomized gaze visualization.After each video, they completed an intermediate questionnaire assessing perceived usability, usefulness, and cognitive effort.Following all four videos, teachers answered a post-questionnaire comparing the visualizations.The online study concluded with an interview conducted over the video conferencing system Zoom Meetings, with most interviews lasting 15 to 20 minutes, while some extended beyond 30 minutes due to additional teacher input.

Questionnaires
5.3.1 Pre Questionnaire.At the beginning of the study, we asked participants to indicate their experience with giving online lectures on a 5-point scale, reaching from "no experience" to "regularly giving online lectures".

Intermediate Questionnaires.
Perceived Subjective Connectedness.To assess how the teachers perceived how much the visualizations created a personal connection to the audience, we asked them to rate three questions on a 7-point scale ("not at all" to "very much").The questions are based on Parmar and Bickmore's questions to evaluate an augmented reality real-time feedback system [31].In addition, the participants answered the Inclusion of Other in the Self Scale [3], which measures how close a person feels with another individual or group.The scale is a Venn diagram, i.e., the two roles "self" and "other" are shown as two circles.The overlap between the two circles indicates the relationship between "self" and "other".The scale reaches from two separated circles to two almost completely overlapping circles.
Perceived Usefulness.To evaluate how teachers rate the perceived usefulness of the visualizations, we asked them to answer ten questions adapted from the "perceived usefulness"-scale of the "technology acceptance model for empirically testing new end-user information systems" [14]."Perceived usefulness" is hereby defined as "the degree to which an individual believes that using a particular system would enhance his or her job performance." [14].The scale is a 7-point scale, reaching from "strongly disagree" to "strongly agree".
Perceived Usability.To evaluate teachers' general perception of the usability of the visualizations, we used a usability survey consisting of seven questions on a 7-point scale, reaching from "not at all" to "very much".For this questionnaire, we adapted the questions provided by Murali et al. for the evaluation of a public speaking support interface [29] and the evaluation of the AffectiveSpotlight system [28].
Perceived Cognitive Demand.We assessed cognitive demand with the Rating Scale Mental Effort (RSME) by Zijlstra and Van Doorn [46].This scale is a unidimensional rating scale that stretches from "0'"to "150" and contains nine anchor points stretching from "absolutely no effort" to "extreme effort".Respondents indicated their current state of mental effort by positioning a slider to the perceived mental effort level.

Post
Questionnaire.In the post-questionnaire, participants were asked to rank the visualizations based on five criteria: (C1) preference for use during online lectures, (C2) preference for use in retrospective analysis of recorded online lectures, (C3) perceived helpfulness for online lectures, (C4) ability to create the closest connection to the audience, and (C5) perceived level of distraction during online lectures.Rankings were given from most preferred, helpful, or close connection to least, as well as most to least distracting.

Instructor Participants
Originally, there were 13 teachers.One of the participants was dropped due to technical issues preventing the interview.Therefore, 12 teachers (9 male, 3 female; 12 White) with an average age of 32.25 ( = 27,  = 39) joined the study.They completed the online study as well as the semi-structured interviews.The teachers were lecturers from Universities in Germany.During data collection, two of the teachers noted auditory technical issues (no or quiet sound).Both expressed that this technical issue did not affect their user experience and rating of the visualizations.

Data Analysis
We employed the non-parametric Friedman test for a global comparison of the four visualizations, analyzing individual questions within each questionnaire or scale.This approach was chosen because most questionnaires were not designed to be calculated for specific constructs.If the Friedman test revealed statistical significance, we conducted non-parametric pairwise Wilcoxon signedrank tests to statistically compare the individual visualizations.The Bonferroni-Holm correction method was used to adjust the p-value.For qualitative analysis (interviews), all but one participant agreed to have their sessions recorded.Recorded sessions were transcribed verbatim and hand-coded.For the participant who declined recording, we took notes during the interview.

Quantitative Results
5.6.1 Pre-Questionnaire: Online Lecture Experiences.All participants had experience with online lecturing, with the majority giving online lectures occasionally.The distribution of experience was as follows: 17 5.6.2Intermediate Questionnaires.Perceived Subjective Connectedness: Friedman tests revealed significant differences in all three questions.For the first (How much of a personal connection did you feel with the audience?)and second question (How easy was it to see the non-verbal feedback from the audience?),no significant differences were found in the post hoc Wilcoxon tests.For the third question (How easy do you feel it would be to respond to the nonverbal feedback from the audience?),significant differences emerged between ellipse vs. bar plot and heat map vs. bar plot.The bar plot visualization received significantly lower scores than the other two.Figure 3 displays the results.
Inclusion of Others in the Self : This scale was assessed using a Venn diagram.The mean scores (with standard deviations in brackets) were 2.17 (0.7) for the bar, 3.75 (1.22) for the ellipse, 4.08 (1.68) for the heat map, and 2.83 (1.27) for the moving bars.The Friedman test revealed a significant difference, and the post hoc Wilcoxon tests showed a significant difference between the heat map and the bar.The bar visualization had significantly lower scores than the heat map. Figure 4 presents the results for this scale.
Perceived Usefulness Ten questions were rated on this scale, ranging from not at all (1) to very much (7).Friedman tests identified significant differences in all questions.However, post hoc Wilcoxon tests revealed significant differences in PU1 (Using the visualization would improve the quality of the work We do), PU2 (Using the visualization would give me greater control over my work), PU6 (Using the visualization would increase my productivity), PU9 (Using the visualization would make it easier to do my job), and PU10 (Overall, I find the visualization useful in my job).Figure 5 presents the results for these measurements.
For PU1, a significant difference was found between the bar and ellipse.Compared to the bar, the ellipse significantly improved the quality of the work.For PU2, significant differences were identified between the bar and ellipse as well as the moving bars and bar.In comparison to the bar, the ellipse gave the participants less control over their work, and the moving bars gave less control than the bar.For PU6, a significant difference emerged between the bar and ellipse, indicating that participants believed the ellipse could increase their productivity more than the bar.For PU9, significant differences were detected between the bar and ellipse as well as the moving bars and bar.Using the ellipse would make it easier for participants to do their work compared to the bar, and the moving bars would make it easier than the bar.For the last question, PU10, a significant difference was found between the bar and ellipse, suggesting that the ellipse was perceived as more useful in their job compared to the bar.
Perceived Usability This measurements consisted of seven questions rated on a scale from 1 (not at all) to 7 (very much.Friedman tests discovered significant differences in U3 (How much do you  feel the visualization would help you deliver the lecture?), U4 (How distracting do you think the visualization would be when delivering a lecture?), U5 (How satisfied are you with the visualization?) and U6 (How much would you like to give future lectures with the visualization?).Nevertheless, post hoc Wilcoxon tests revealed significant differences in U4 and U5.For U4, significant differences were found between ellipse and heat map as well as moving bars and heat map.The mean were 5 (1.48) for bar, 3.5 (1.57) for ellipse, 1 (1.57) for heat map, and 4.5 (1.51) for moving bars.This scale was re-coded to present the graph more meaningfully.From the mean, it seems that heat map was very much distracting.As compared to moving bars and ellipse, heat map was significantly more distracting.In relative to U5, a significant difference was discovered between heat map and ellipse.The mean were 2.  with ellipse than heat map. Figure 11 shows the results for this measurements.
Perceived Cognitive Demand The cognitive demand was assessed by the RSME scale for mental effort.The scale ranges from 0 (absolutely no effort) to 150 (extreme effort).The mean were 45 (293) for bar, 31.7 (21.1) for ellipse, 33.9 (27.8) for heat map, and 58.4 (27.1) for moving bars (see Figure 4 right panel).A Friedman test showed a significant difference between the visualization, with the p-value being very close to 0.05; however, no significant difference was found for post hoc Wilcoxon test.

Qualitative Results
Only the content that was mentioned at least twice during the interviews was included in this analysis.This section is grouped into several parts: teaching experiences, user experiences with the visualizations, two use cases, factors discouraging usage, and improvement suggestions.

Teaching Experience.
The participants reported that they were generally having a hybrid teaching mode, with a mixture of online and offline teaching experiences.Most of the participants gave practical sessions and lectures.In terms of the online tools they utilized, they mostly used Zoom, Miro boards, and Power Point.Other tools included OBS, Moodle, Big Blue Button, and Discord.The participants shared about their online teaching experiences: P(1) ". . .a mini version of myself talking while We was presenting the slides. . ." P(11) ". . .all participants turned off the camera and that you were sitting there without any kind of feedback."

User Experiences With the Visualizations. :
The participants revealed that, with the visualizations, they felt more connected to the audience and could know whether the students were paying attention.Some highlighted they would be happy to use them for retrospective analysis.The visualizations also provided inspirations for their professional teaching life and were helpful with retrospective analysis.The following showed how the participants were inspired: P( 12) ". . . on the first slide, they only, um, We saw that, the heatmap was on the title and nobody looked at the authors.And on the last slide We saw that it was the opposite, actually, that everybody was looking at the authors' names and nobody actually looked at the title anymore. . .pretty insightful." P(1) "There were a lot of gazes still looking at this content that they were not supposed to look at yet...So this is like an aha moment for me." Throughout the interviews, the heat map and ellipse received most attention, as P12 commented: "I prefer mostly the, uh, heat map and bubble (ellipse)." P8 also replied, "...I would prefer something like the blop visualization ... feels like there's more of a connection to the audience ... " Most participants found heat map and ellipse to be helpful.
Heat map.As shown in the quantitative analysis, heat map was ranked the highest for retrospective analysis.Similar trend is seen in the inteviews.P1 reported, "I chose the heat map because it just gave me the most information."Similarly, P4 also commented, "I think the heap map is the best or most, like, precise one." Ellipse.Ellipse was ranked as the most preferred for online lectures in quantitative analysis.In the interview, P13 reported, "...I think it is unobstrusive and can work in the live situation and you have still a quite high level of information."P4 commented similarly, "... this balance of it's not too distracting because it's not like occluding anything.It has no color and so on, but it still provides me a good, two dimensional estimation of where attention seems to be." Moving Bars.Some participants commented that it was hard to comprehend moving bars.P4 said, "... that was pretty hard for me to like manage internally.There was like quite a bit of a mental effort to find out like this bottom.I have to look at the bottom one.I have to look at the side one, and then I have to look at a slide where it kinds of matches." P8 also commented, "Yeah, I didn't like that one as much...I kept trying to figure out like... because you have to then like, infer like this, this, this one is here." P1 also said that it was hard to process.However, interestingly, P3 mentioned that he would like to have moving bars enabled the whole time.He could imagine moving bars to be on all the time without distracting him because they were not displayed on the slides.
Bar. Bar visualization was least mentioned throughout the interviews.P1 mentioned that bar offered very little information as compared to the others.P5 felt that this visualization did not give him a reference on how far the audiences were reading the slides and that the moving bars is more valuable.

Two Use Cases.
In terms of the use case, We asked questions about the usage in live teaching and retrospective analysis.
Live Teaching.For live teaching, the participants were less likely to choose heat map.They preferred moving bars and ellipse, or even bar.P1 and P7 chose ellipse, while P11 and P12 chose bar.P3 would like to have moving bars on at all time that he would not need to take the efforts to switch it on and off.He thought this was workable because moving bars were not overlaid on top of the slides.
Retrospective Analysis.The participants generally preferred visualizations that offered more information, namely heat map and ellipse.P8 mentioned that she would prefer both heat map and ellipse.P7 preferred ellipse as it gave fastest feedback and did not have too much noise like heat map.P4 and P2 both thought that heat map was good for this use case because it provided most information.

Factors Discouraging Usage.
In terms of what would discourage the participants from using the visualizations, they reported data privacy concerns, distractions, and additional equipment.Most participants concerned about the students' willingness to be eyetracked, as well as whether they would be comfortable to attend online classes with eye tracking feature.P4 thought that it should be more transparent and open that the students could see the visualizations as well.Also, if the students had to install additional hardware to be eye-tracked, they would not use them.They preferred to have ease of use.Furthermore, the participants would like to be comfortable for them to use the visualizations, such as having less time pressured lectures (P8) or familiarity with the content (P10).They also commented that if the visualizations were too distracting, they would not use it.This leads to the point that if they could not turn the visualizations off when they wanted, they would not use them at all.The following shows examples of the participants' response: P( 2) "If it's always moving, then maybe I'll just, I'll be too distracted." P(5) ". . .student, they could at some points try to look at some certain parts or certain things ... I don't know...I think it's just too distracting to use the whole time." 5.7.5 Improvement Suggestions.The participants gave plenty of suggestions to improve the current visualizations.First, they would like to have a toggle to switch the visualizations on and off on the fly.Second, they would like to have minimal setup time, ease of use, and to switch through all kinds of visualizations easily.Third, they would like to have the flexibility to adjust the transparency of the visualizations.Fourth, it is essential for the visualizations to work on all kinds of slides, including dark and white slides, as commented by P13, ". . .I think it works best for quiet, bright slides with white background ... sometimes I was not possible to see the visualization ... " Most participants raised their concerns about the visibility of the visualizations on dark background.
Also, some participants wanted to have more than one cluster for bars and ellipse.They noticed that ellipse provided average gaze points and reckoned that there could be more than one cluster at a point of time after watching the heat map visualization.P3 commented, "I would want them actually to split up to at least two bars."P4 said, "... some kind of thresholds at which you separate them and maybe show two clusters." Another improvement suggestion is to have analytics dashboards to offer critical insights, especially for retrospective analysis.P6 and P11 both mentioned that they would not like to re-watch the whole 90-minute lectures and suggested to have some sort of aggregated information like analytics dashboards to show the insightful key points for them to look at.

DISCUSSION
The primary objective of this study was to enhance e-learning by visualizing students' eye movements for instructors.Drawing upon established visualizations of physiological measures and eyebased visualizations, specifically the heat map directly overlaying the content and a vertical bar on the side of the screen, we tested four different visualizations: the typical heat map, an ellipse, a vertical bar, and two bars.Each visualization was derived from real eye movements of a student sample collected in a lab-based study.Subsequently, we presented these visualizations to 12 instructors, overlaid on their own e-learning content.We concluded the study by gathering both quantitative and qualitative feedback from the instructors regarding their experiences with the visualizations.
In terms of perceived subjective connectedness, participants found the feedback from the more concrete ellipse and heat map visualizations easier to interpret, and these visualizations fostered a closer connection to the audience.This connection was reflected in the inclusion of others in the self scale, the ranks (see also Figure 7.( 5)), and partially the perceived subjective connectedness (Figure 4) compared to the visualizations that were presented on the side of the screen (moving bars and vertical bar).This finding is particularly intriguing since the moving bars convey the same information as the ellipse, albeit with a different visualization.For fostering a strong instructor-student connection, the directness of the information-visualization (i.e., visualization directly on the content and not on the side) seems to be crucial.These results align with Hirzle et al. [19], wherein heat maps and "disk maps" outperformed bar plots.
While fostering a connection to the audience is essential, additional visualizations should not be so cognitively demanding as to overshadow these benefits.Although no significant differences were found in post-Wilcoxon tests, the ellipse numerically required the least effort, and the moving bars required the most.This could potentially explain why participants preferred the ellipse over the heat map.The finding regarding moving bars is consistent with qualitative results, wherein some participants reported difficulty in comprehending the mechanism behind the moving bars, necessitating considerable cognitive effort to interpret them.
The heat map was reported as more distracting than the bar plots in terms of overall perceived usability, which is consistent with the qualitative results and prior studies ( [19]).In fact, participants preferred the ellipse as a simplified version of the heat map, as it provides the benefits of information content on the slide without being overly dynamic or distracting.
Although heat maps were evaluated as easy to understand, instructors preferred the ellipse for real-time e-learning to improve the quality of their work, increase productivity, and enhance usability.For retrospective evaluation of their teaching and learning materials, however, the heat map was considered the top choice, as its distracting nature has less impact in post-hoc analyses of e-learning content.
In terms of further qualitative assessment, participants reported that all visualizations enabled them to feel more connected to the audience and provided insightful teaching experiences, suggesting that any visualization of crowd attention is better than none at all.Gaze information allowed instructors to verify whether their elearning content was on track.Notably, users expressed a desire to toggle visualizations on and off, particularly during live e-learning sessions, and adjust the transparency of the visualizations to suit their needs or preferences.We propose adapting the design in terms of line thickness and color to match the layout of the slides.
For retrospective evaluation, the current setup requires instructors to re-watch every minute of their e-learning content.Participants suggested incorporating summary statistics as an initial overview for such analyses, such as an overall deviation score indicating whether students often focused simultaneously on the same location or if their gaze points were distributed across different parts of the slide.The latter could indicate a need for improvement in the design of the e-learning slides, as the focus is unclear.This is especially important because it has been shown that participants who deviate significantly from the average (group) gaze point have significantly lower performance in a subsequent quiz [36].
In conclusion, our study demonstrated the value of visualizing students' eye movements for instructors in e-learning settings.The findings suggest that direct on-content visualizations, such as the ellipse, foster a closer connection to the audience and are less cognitively demanding for instructors.By incorporating these visualizations and adapting them to the layout and preferences of individual instructors, e-learning experiences can be improved, fostering better connections between instructors and students, and ultimately enhancing the overall teaching and learning process.Future research could explore additional visualization options and refine summary statistics to further optimize the use of gaze data in e-learning environments.

Comparison with related prior work
Overall, the more abstract visualizations (bar and moving bars) appear to offer limited practical advantages compared to the concrete on-slide visualizations in the context of e-learning.While they were not found to be disruptive, they also seemed to provide little benefit to instructors.This is somewhat surprising, as this presentation format is frequently employed in related work [18,43] for presenting audience feedback to presenters.For instance, Sun et al. utilized a bar plot to visualize feedback in online lectures [43], displaying the extent (i.e., a mean value) of various states, such as flow or boredom.In contrast to our work, the authors found the bar plot to be intuitively usable.This discrepancy might be attributed to the more complex information in our case, as measures of variability may be more challenging to interpret than a mean score of a specific concept.Furthermore, we did not compare the visualizations to a control condition without any visualization, so it is possible that the abstract visualizations still offer benefits when compared to no visualization at all.
In contrast to Murali et al., we did not provide feedback on individual audience members [28] but instead focused on visualizing crowd attention as variability in gaze data.We identified two issues with displaying individual values: First, visualizing individual gaze information raises privacy concerns [23].Second, presenting each student's individual gaze point was deemed to provide too much detailed information to be processed during instruction.By visualizing the variability of eye movements, although we do not display individual data, we still offer a measure of inter-individual variability in attention.This approach has been suggested as a core factor in crowd attention to serve its functions [44].

Limitations and Future Directions
Our work encompasses both quantitative and qualitative findings concerning visualizations of crowd attention in e-learning contexts.We recorded eye movements from approximately 12 participants per video.While this sample size is sufficient for evaluating visualizations in a typical seminar or advanced course setting, further investigations involving more students are necessary to assess the generalizability of our inferences to larger audiences.However, given that measures of variability typically do not change significantly as the number of participants increases, we are confident that our current results provide a reasonable approximation for larger e-learning crowds.
At present, the heat map is the only visualization capable of depicting more than one "area of focus."In larger crowds, being able to display multiple areas of focus may be important for better addressing the diversity of eye movements within such groups.Consequently, future research should explore the possibility of using multiple ellipses in a manner similar to heat-map-based visualizations.This task, however, is not trivial and would require systematically probing which clustering method works best.Some existing clustering algorithms include hierarchical, k-means, and DBSCAN [25].Nonetheless, Isokoski et al. argued that clustering is only meaningful when there is a substantial amount of data to cluster [20], which critically depends on group size.
As a future step, we aim to implement and examine crowd attention in actual instructional settings.This endeavor will necessitate tracking the eyes of students simultaneously in real-world environments.Although real-time and in-the-wild eye tracking remains a challenge for current technology, even small eye movement datasets can be useful for retrospective analysis of e-learning sessions.

Two Recommendations for Two Use Cases
In summary, two distinct "winners" have emerged for different use cases involving gaze-based feedback in e-learning contexts.For live online teaching scenarios, we recommend employing an ellipse-like visualization of students' gaze points.This visualization is less intrusive than a heat map, yet it is perceived as providing a comparable amount of information.For retrospective analysis, a heat map-like visualization is the clear preference, as it offers the most information.However, when implementing this, customizable user settings (e.g., adjusting the transparency of the heat map or enabling/disabling it) seem to be necessary for widespread adoption.
Our vision for crowd attention visualizations involves a real-time system that tracks the gazes of all student audience members outside the lab, calculates the visualization in real-time, and presents this visualization to the instructor in real-time, thereby assisting them in responding to the attentional state of the audience.Nevertheless, there remain several challenges to overcome in order to realize such a system in the current landscape of e-learning.

Proposed Third Use Case: Facilitating Self-Reflection
Given the increasing affordability of display-mounted eye-trackers, a further noteworthy application of eye-tracking visualizations emerges in the context of facilitating self-reflection for students.By providing students with visual feedback on their own attention patterns, they can be better equipped to understand their learning behaviors and identify potential areas for improvement.Notably, these reflective insights can have significant implications on their learning processes and subsequent performance.The value of selfreflection in learning and cognitive development has been well established in educational psychology ( [39]).
When combining this concept with the use of bio-signal data and eye-tracking, as outlined in Andujar and Gilbert and Schneegass et al., it provides an exciting direction for future research ( [2], [38]).For instance, physiological measures such as heart rate, skin conductance, and EEG could offer additional insights into students' emotional and cognitive states during learning.These measures could be combined with gaze data to provide a more comprehensive understanding of student engagement, enabling further personalized feedback.
It is worth noting, however, that this proposed use case brings along its own set of unique challenges and considerations.Notably, the privacy and ethical aspects of tracking and sharing personal data are paramount.To ensure ethical use, it would be essential to obtain informed consent from students and implement measures to protect their data.Furthermore, the interpretation of such visualizations would need to be guided and supervised by educators or trained professionals to prevent potential misinterpretations or unnecessary stress for students.
In summary, while our study has mainly focused on the value of visualizing students' eye movements for instructors, the potential of these visualizations in supporting self-reflection for students themselves is a promising avenue for future exploration.By integrating insights from cognitive psychology and human-computer interaction, the future of e-learning could further be revolutionized, contributing to personalized and effective learning experiences.

CONCLUSION
In the realm of effective e-learning, understanding and visualizing crowd attention is of paramount importance.Our study examined four distinct visualizations of students' eye movements during online learning sessions.The simplified ellipse emerged as the preferred visualization for instructors, striking a balance between the spatial proximity of concrete visualizations and the unobtrusiveness of abstract ones.In contrast, the traditionally employed heat map, although easily understood, was less desirable for live online teaching.However, it demonstrated its value in retrospective analysis of e-learning sessions.The implementation of the ellipse visualization in live online teaching scenarios has the potential to enhance instructors' understanding of crowd attention, leading to more effective and engaging learning experiences.Additionally, the visualizations positively influenced instructors' perceived social connectedness with their audience, which is crucial for fostering emotional connectedness in remote online environments.By employing these insights, e-learning platforms can be adapted to better meet the needs of both instructors and students, ultimately driving the evolution and improvement of online education.

Figure 1 :
Figure 1: This study measures students' eye movements during online learning to estimate their attentional state and provide instructors with information about crowd attention.Four gaze visualization methods were developed: heat map (a), ellipse (b), moving bars (c), and vertical bar (d).A user study with 13 teachers evaluated the visualizations' impact on social connectedness with students, as well as their perceived usefulness, usability, and cognitive demand.

Figure 2 :
Figure 2: Four gaze visualization techniques were implemented in this study: (1) The heat map displays students' gaze points on specific screen locations, with colors ranging from purple to yellow indicating the number of students looking at the same location simultaneously.This implementation is based on a modified version of the heat map plotter from PyGaze [10, 34].(2) The ellipse represents the central 50% of students' gaze points (IQR of x-and y-positions), centered at the mean position of the gaze points.(3) The moving bars, located at the screen's margin, correspond to the mean x-and y-positions of students' gaze points, with their width and height representing the middle 50% of gaze points.(4) The bar's height reflects the sum of the IQR of x-and ypositions of students' gaze points but does not indicate the gaze points' distribution.Two examples of each visualization are provided: the top row illustrates high variability of gaze points, while the bottom row displays low variability of gaze points.

Figure 3 :
Figure 3: Results from the Perceived Subjective Connectedness scale.A1 (How much of a personal connection did you feel with the audience?),A2 (How easy was it to see the non-verbal feedback from the audience?)and A3 (How easy do you feel it would be to respond to the non-verbal feedback from the audience?).The bars depict the mean points on the scale for each of the four different visualizations separately.Significant differences are highlighted ( *  < .05).Error bars show the standard deviation.

Figure 4 :
Figure 4: Left panel: results for the Inclusion of Others in the Self scale, 1 indicates no relationship between the teacher and the students, and 7 indicates a very close relationship between the teacher and the students.Right panel: results for the mental effort question.Significant differences are highlighted ( *  < .05).Error bars show the standard deviation.

Figure 5 :
Figure 5: Results of the Perceived Usefulness on a scale from 1 (not at all) to 7 (very much).Significant differences are highlighted ( *  < .05).Error bars show the standard deviation.

Figure 6 :
Figure 6: Results of the Perceived Usability scale.This scale ranges from 1 (not at all) to 7 (very much).U4 = How distracting do you think the visualization would be when delivering a lecture?; U5 = How satisfied are you with the visualization?) U5 (How satisfied are you with the visualization?).Significant differences are highlighted ( *  < .05).Error bars show the standard deviation.

Figure 7 :
Figure 7: Results for the ranking of the visualizations.The teachers ranked the visualizations with regard to the five shown criteria.For each criterion, we show the mean rank of each visualization from 1 to 4. The horizontal dashed line indicates the middle rank assignment expected from a random distribution.Note that C1-C4 indicate that the desired outcome is on rank 1, while for C5 the desired outcome is rank 4. Significant differences are highlighted ( *  < .05,* *  < .001).Error bars show the standard deviation.