Shaping and evaluating a system for affective computing in online higher education using a participatory design and the system usability scale

Online learning’s popularity has surged. However, teachers face the challenge of the lack of non-verbal communication with students, making it difficult to perceive their learning-centered affective states (LCAS), leading to missed intervention opportunities. Addressing this challenge requires a system that detects students’ LCAS from their non-verbal cues and informs teachers in an actionable way. To design such a system, it is essential to explore field experts’ needs and requirements. Therefore, we conducted design-based research focus groups with teachers to determine which LCAS they find important to know during online lectures and their preferred communication methods. The results indicated that confusion, engagement, boredom, frustration, and curiosity are the most important LCAS and that the proposed system should take into account teachers’ cognitive load and give them autonomy in the choice of content and frequency of the information. Considering the obtained feedback, a prototype of two versions was developed. The prototype was evaluated by teachers utilizing the System Usability Scale (SUS). Results indicated an average SUS score of 80.5 and 74.5 for each version, suggesting acceptable usability. These findings can guide the design and development of a system that can help teachers recognize students’ LCAS, thus improving synchronous online learning.


INTRODUCTION
Online learning enables a large number of students to enroll in classes and benefit from educational opportunities due to its flexibility [31].Specifically, online learning allows students who face obstacles attending a classroom physically to participate online.During the pandemic, most students faced such limitations in physically attending their classrooms.Therefore, as the utilization of online learning remarkably increased, its challenges [15] became visible for both students and teachers.
One important challenge that arises in online learning is the limitations of non-verbal communication during synchronous educational activities.This leads to a lack of awareness of the affective states of the students, i.e., states of feeling like emotions and moods [23].The affective states of students have an important effect on their learning experience by influencing their motivation to learn, engagement, and self-regulation [21].Therefore, their recognition by teachers is important since it gives them the opportunity to adjust their teaching approach accordingly and keep learning optimal [18].In traditional classrooms, teachers can recognize the affective states of students by observing their nonverbal cues.Nevertheless, in online classrooms, there is limited to no nonverbal communication.Therefore, it is not possible for teachers to detect the affective states that are related to learning, the learning-centered affective states (LCASs) such as boredom, engagement, and anxiety [4].Consequently, developing systems that intensify teachers' awareness of their students' LCASs can improve education quality and address an essential shortcoming of online education.
The research presented in this paper covers the design, prototypical development, and usability evaluation of the user interface for a system that detects the LCASs of students in an online classroom and presents them to the teacher in real time.Regarding the design, a crucial step is to explore and elicit the needs and requirements of the stakeholders.Regarding the evaluation of the prototype's usability, it is necessary to gain user-experience insights from representative stakeholders.In this study, we focus on teachers in online higher education and involve them in the participatory design of the proposed system as well as an evaluation of the system's prototype.
In our study, we aim to answer the following research questions: • RQ1 What content of the feedback regarding the students' learning-centered affective states is the most important for teachers in online classrooms?
• RQ2 How can the feedback be provided to teachers in a meaningful manner?• RQ3 What is the usability of the system prototype designed based on the design ideas gathered from RQ1 and RQ2?
For this purpose, we designed two group activities to occur within focus groups, namely, a card-sorting (RQ1) and co-design (RQ2) activity as well as an evaluation session (RQ3).
The remainder of this paper is structured as follows.Section 2 provides a review of relevant research on emotions in learning and co-designing platforms for online education.Section 3 describes the details of our study's methodology.Section 4 discloses the results of the focus groups, the development, and the evaluation of a prototype.Section 5 provides a discussion of our contributions and their implications suggesting future research directions as well as our limitations.Finally, Section 6, summarizes and concludes the paper.

RELATED WORK
In this section, we provide an overview of previous studies that focus on emotions in learning and on participatory design for platforms for online education.

Emotions in Learning
Emotions are not isolated internal states but are rather related to the context in which they are expressed [2,14].In the field of education sciences, that context is the classroom, an environment where students experience different emotions, i.e., complex reaction patterns involving experiential, behavioral, and physiological elements 1 .Pekrun defines academic emotions as the emotions that are relevant for students' learning [21] and categorizes them into four groups, achievement, epistemic, topic, and social emotions.As the utilization of online learning rapidly increases, there is a growing number of studies focusing on emotions in online classrooms [1].There are studies proposing models [34] and systems to detect various sets of affective states in online classrooms synchronously [6,17] and retrospectively [13].A meta-analysis of 24 studies explored the affective states that students experience in technology-enhanced learning context [9].Their results consistently showed that engagement was found to be frequent in the included studies while affective states such as contempt, anger, disgust, sadness, anxiety, delight, fear, and surprise were not and that there were differences between studies regarding the relevance of boredom, confusion, curiosity, happiness, and frustration.In our study, we aim to gain insights from teachers on which LCAS of their students they find relevant and useful to be aware of during an online lecture.

Participatory Design for Online Learning Platform
Participatory design can stimulate creativity and collaboration while offering the opportunity to gain insights from field experts and prospective users [10].It has been widely used in designing platforms for online learning [30].There have been studies that propose methods and tools for participatory design for capturing the conceptual design of the educational interactive systems [28] and 1 https://dictionary.apa.org/emotiondesigning learning analytics dashboards [29].However, a review of the recent literature [30] shows that in the majority of the studies, the participatory design activities and aspects such as recruitment are not thoroughly described.In our study, we provide a detailed description of the research design, process, and background of each focus group activity.

METHODOLOGY
To answer research questions RQ1 and RQ2, we employed Participatory Design Research (PDR) [24], a process that includes participants in various stages of the design of a deliverable system [26].The study was approved by the Research Ethics Committee of our university, and participants received an information letter and provided informed consent.Participants joined a focus group where two group activities took place.Each focus group session had a fixed duration of 90 minutes.Multiple focus groups were run with different participants but with the same structure and group activities.The size of a focus group was three to five participants.The lower boundary of the group size was based on the fact that participants are regarded as experts in the field who are highly engaged with education and interested in participating voluntarily.
The aim of the upper boundary of the group's size was to give everyone the opportunity to speak and express themselves given the restricted time [25].
To answer the research question RQ3, we developed a prototype of the proposed system following the insights gained from the PDR study.The prototype was simulating a short educational activity of a lecture on introduction to probability using a whiteboard in a class of 10 students.The simulation included artificial data of the students' LCAS per second.The prototype was evaluated regarding its usability using the System Usability Scale method [7] by ten representative users, specifically, teachers in online classrooms.Similar to the PDR, our university's Research Ethics Committee approved this study, participants were given an informational letter and subsequently provided their informed consent.
A visual representation of the methodology is shown in Fig. 1.The following sections provide insights into the participants' recruitment, the procedure of the PDR group activities, details regarding the development of the prototype, and the procedure of the evaluation.Shaping and evaluating a system for affective computing in online higher education using a participatory design and the system usability scale LAK '24, March 18-22, 2024, Kyoto, Japan

Recruitment Procedures
For the PDR study, the focus group sessions took place in two rounds with different recruitment processes.The first round was during the thematic workshop "Multimodality and AI in Online Education" at the 16th EATEL Summer School on Technology Enhanced Learning.The participants were junior and senior researchers in the field of technology-enhanced learning and education.We chose to recruit participants from that audience due to their extensive academic and research expertise in this area, which equips them with the skills necessary for deeply exploring and understanding the implications of educational technologies.Attendance at the workshop was open to all the participants of the summer school without any further inclusion or exclusion criteria.Participants registered for the workshop after reading its corresponding description.The workshop consisted of a presentation and a focus group session.
During the focus group session, participants divided themselves into groups of their choice after being informed of the desired group size.The groups worked simultaneously on two activities (Section 3.2).The researcher further explained the group activities during the workshop, where the participants had the opportunity to ask clarification questions.The second round was with academic staff from the Faculty Of Educational Sciences of the Open University of the Netherlands, a distance university, where courses are given online by design.Therefore, all teachers have experience with online teaching.Furthermore, we selected the faculty of education to leverage the specialized knowledge of its staff which enables them to offer more insightful and critical evaluations of the use of technological tools in pedagogy.Call for participation emails were sent to a mailing list of the academic staff, including an information letter with the purpose and details of the study.According to the availability, the groups were formed, and participants were invited for the face-to-face focus group activities.
For the prototype's evaluation, call for participation emails were sent to professionals from the same mailing list used in the second round of the PDR study.Therefore, a teacher who participated in the former groups could also participate in the latter.

Participatory Design Research Focus Groups
Each focus group was structured around two activities aligned with the study's first two research questions (RQ1, RQ2).First, a cardsorting activity was conducted to discern which students' LCAS teachers find important to know during an online classroom.Second, a co-design activity was utilized to gather expert perspectives on the design of the proposed system.Participants were asked to consider the didactic scenario of giving a traditional lecture online.Both group activities had the same duration, 45 minutes, including the time used by the researcher to give a reminder on the purpose and instruction of the activity and answer the participants' questions.The details of each activity can be found in the respective sections below.

Card sorting.
The first group activity was titled "Learning-Centered Affective States." The participants were given a set of 16 cards, each card corresponding to one learning-centered affective state.The set of those learning-centered affective states was chosen from the prominent states found in the relevant literature [4] as well as the universal set of emotions [11], namely: anger, anxiety, boredom, confusion, contempt, curiosity, delight, disgust, engagement, eureka, excitement, fear, frustration, happiness, sadness, and surprise.Each card had on its front side a learning-centered affective state and on its back a short description of it (Fig. 2a).The content of the cards is shown in Table 1.In the first round, we used definitions from dictionaries [8,12,22].Based on the questions and feedback that we received from the participants of the first round, we updated the definitions using terms that were more clear and more relevant to education [5,12,14,16,27].The participants of each group were asked to discuss the importance of the knowledge of each LCAS of their students in an online lecture.Specifically, they were asked to come up with a sorted list in descending order of importance.There was no restriction on the number of states that the participants could choose for a position in the ranking.For example, if they found that two states are equally important, they could rank them at the same level.In addition to our provided list, participants were given the opportunity to include and categorize any emotional states they deemed relevant.This approach was taken to leverage their expert insights regarding the role of emotions in the educational process.
Through this activity, we collected two types of data, the sorting of the LCASs that the groups generated and the observations noted during their group discussions.For the first one, each LCAS yielded a score, the "importance_score" as defined in Eq. 1.For example, if the group sorted the cards in 10 levels, the states in the 1  position would gain _ = 1 the states on the 2  position _ = 9  10 , while the states in the last position _ = 1  10 .A threshold of 0.8 average importance_score over the groups was used to consider a learningcentered affective state as important.For the second one, the researcher kept notes during the group discussions, and the comments that were common throughout the groups are reported in the Section.4.1 As for the group discussions, the researcher's notes were analyzed through a manual thematic approach aiming to uncover noteworthy insights and ideas, as well as to identify areas of consensus and divergence among the groups.

𝑖𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒_𝑠𝑐𝑜𝑟𝑒
where: n = the levels of importance that the participants' sorting concluded in p = the position of the state in the participants' sorting 3.2.2Co-design.The next focus group activity, titled "Communication of feedback," was designed to understand teachers' views on the proposed system's design.Participants discussed different features of the system and sketched a user interface to represent these features.The main points of discussion included position, modality, intervention, and visualization.After the feedback and questions from the first round, the aspect frequency was added in the second round.The position was related to the location on the screen where they would propose the feedback to be placed.Modality refers to the way they suggest the feedback to be communicated, for example, text or image.Intervention related to whether they would prefer the tool to give tips and suggestions regarding teaching based on the detected information and how they should be.Visualization refers to the mean the data is going to be displayed, for example, a pie chart or time series graph.At the beginning of the session, the researcher explained the activity and presented an example wire-frame, which is shown in Figure 2b.She subsequently answered the participants' questions.Through this activity, two types of data were collected, the participants' interface design drawings and the group discussion points, which were observed and analyzed thematically, similar to the Card Sorting activity.

Prototype Development and Evaluation
3.3.1 Prototype Development.The prototype was developed on a simulation of the widely used video platform Kaltura2 that was created to be used as a template.On that template, the prototype of the proposed system was integrated.The prototype was designed and developed considering the input received from the participatory design research focus group activities regarding the feedback's position, modality, type of intervention, type of visualisation, and

Prototype Evaluation.
The prototype was evaluated in individual sessions with teachers.Each meeting followed a specific procedure.First, the researcher gave a short introduction to the purpose of the system, followed by a demonstration of the prototype and an explanation of its functions.Following the demonstration, teachers were able to test the system, pose questions, and offer feedback.Teachers were requested to complete a survey using LimeSurvey 3 based on the System Usability Scale (SUS) methodology.In this approach, participants rate statements concerning the system's usability on a Likert scale ranging from 1 to 5 (1: strongly disagree, 2: disagree, 3: neutral, 4: agree, and 5: strongly agree).The statements included are the following: 3 https://www.limesurvey.org/(1) Q1: "I think that I would like to use this system frequently, " (2) Q2: "I found the system unnecessarily complex, " (3) Q3: "I thought the system was easy to use, " (4) Q4: "I think that I would need the support of a technical person to be able to use this system, " (5) Q5: "I found the various functions in this system were well integrated, " (6) Q6: "I thought there was too much inconsistency in this system, " (7) Q7: "I would imagine that most people would learn to use this system very quickly, " (8) Q8: "I found the system very cumbersome to use, " (9) Q9: "I felt very confident using the system", (10) Q10:"I needed to learn a lot of things before I could get going with this system." In evaluating system usability, the SUS score is computed using the formula: SUS= 2.5 × (  ∈ {1,3,5,7,9} (  − 1) This yields a score within the range of 0-100, indicative of the system's overall usability [7].Through this activity, two types of data were collected, the participants' scores on the system usability scale, as well as the points raised and the insights expressed during the evaluation session.

RESULTS
In the two rounds of the PDR study's focus groups, a total of 24 individuals participated, segmented into six distinct groups.Each round comprised three subsets with five, four, and three members, respectively.During the evaluation sessions, ten teachers participated, of whom six were previously acquainted with the subject, having also been involved in the participatory design focus groups.
In this section, we are going to present the results of the system's design, development, and evaluation stage.

Card Sorting
The descriptive statistics, i.e., the mean and standard deviation of each learning-centered affective state's _, are shown in Table 2.Moreover, the box plot in Fig. 3 depicts how the _ of the LCAS are distributed over the six focus groups.
The affective states that got an average importance_score larger than a threshold of 0.8 are confusion, engagement, boredom, frustration, and curiosity.None of the six universal emotions was given an importance_score above the threshold.Specifically, four of them namely, disgust, sadness, happiness, and fear, were placed in the lowest levels of importance.
In the boxplots, we can see some outliers as dots in the graph.Those outliers represent the _s given to an LCAS from one group that is not in line with the average of the rest of the focus groups.Specifically, contempt was given a high importance score in a unique group, where the participants claimed that when students feel contempt towards the teacher or their fellow students, this can lead to a lack of attention, therefore blocking the learning process.The engagement was rated lower in importance in one group compared to the rest.Participants in that group argued that engagement is not necessary for the teacher to be aware of since, as a positive state, it does not require action.Regarding eureka, the five groups found it not relevant since it does not last long and it occurs   what they already know, therefore they considered it important for the current lecture's content.Lastly, in one group, it was mentioned that fear can inhibit learning.Thus, it is important for the teacher to know it and aim to investigate it through discussion, while the rest of the groups found that it is not highly related to learning.There were several emotions and states that were suggested by the participants that were not included in the set that was provided (Table 1).Specifically, the states "pride" and "shame" were mentioned by participants of two groups with the same _s 0.75, 0.5, respectively.Additionally, "confidence" and "dissapointment" were mentioned in one group and were given importance_score 5  6 .The same group highlighted the importance of "trust" in learning, and, by extension, the teachers' knowledge of it, giving it an _ = 1.Lastly, "hope" was suggested by one group with an _ of 0.25.

Group Discussion
. The most prominent comment from all groups was that teachers are interested in being aware of affective states that are related to learning.Moreover, all the groups mentioned that they would prefer knowing an affective state that they can either regulate with an intervention, e.g., manage boredom by introducing an interactive activity, or use as an indication that the course is going as planned, e.g., students are engaged in the topic.It was mentioned that it is important to consider that there are strong relationships between affective states.Therefore, the knowledge and regulation of one can prevent another one and subsequently prevent blocking the learning.For example, confusion can lead to frustration which can then lead to anger which can be responsible for a student dropping out of a class.All groups highlighted the importance of the educational context and specific learning activity in order to decide which affective states are more important for teachers.For example, in some contexts, confusion can be used by the teacher to confront learners and grab their attention, while in others, it can indicate weakness in the presented material.Two groups supported that the most beneficial for teachers is to be aware of the achievement emotions [20], supporting that those are more informative for education.

Co-design
In this section, we present the generated designs of the interface as well as the discussion points of the participants' comments and insights.For presentation purposes, a replication of the generated drawings of the design is presented in Fig. 4. As seen in the figures, all focus groups emphasized the importance of flexibility to the teacher.
Groups 1,2,4,5, and 6 indicated that the most important characteristic of the position of the feedback's information window is that it is minimizable and opens after the teacher clicks on it.The same set of groups suggested that teachers should have the ability to choose the LCASs that they want to focus on among a list of possible ones.In that way, they would be able to alter the settings for each different learning activity while regulating the cognitive load of the feedback information.Specifically, Group 5 suggested that there is a maximum of five learning-centered affective states to not overload the teacher with information and keep the cognitive load in mind.
The preference for the use of emoticons varied among the groups.Specifically, Groups 4 and 5 indicated that they might increase the cognitive load considering that their interpretation requires some extra attention from the teacher since, for some, the link between emoticons and affective states is not straightforward or universal.On the contrary, Group 5 proposed the use of emoticons combined with text.
Regarding the type of intervention, four of the six groups found that notification can be easily disturbing.Therefore, they suggested a discreet pop-up or nudge that can be turned off by the teacher.On the contrary, Group 3 suggested a bloody screen 4 that would capture the attention of the teacher when drastic changes in the affective states of the students are detected.All groups agreed that the use of sound in the notification could be disturbing and interrupting.Therefore, they suggested that it be avoided.
A time series graph, color-coded for each affective state, was proposed as a visualization along with a pie chart from Group 1.A pie chart was also proposed as a visualization from Group 3.Both Groups 4 and 5 described a color-coded thermometer-like bar chart indicating the intensity of each affective state.On the other hand, color coding was not preferred by other groups that mentioned it could exclude teachers who are color-blind.Instead, Groups 2 and 6 suggest text, with Group 2 proposing a three-scale (all, some, none) indicator for each affective state and Group 5 the exact number of percentages.Lastly, on visualization, participants in Group 6 indicated that apart from the percentage of each affective state in the current moment, the change can also be informative for the teacher.Therefore, they suggested a bar chart that visualizes the change in the percentage and indicates with an emoticon whether it is positive or negative.For example, a low percentage of engaged students can be considered positive if it was previously even lower or zero.
In the instructions for this activity, we included the aspect of frequency in the second round (Groups 4, 5, and 6).All three groups proposed the frequency of the feedback to be adjustable in order to not be disturbing during the course.

Group Discussion
. All focus groups emphasized that the cognitive load should be considered, given that the online classroom can be an overwhelming and stressful environment for teachers.Similar to card sorting, participants highlighted the importance of the learning context in order to decide on the design.Additionally, all the groups mentioned that it should be taken into account by both the designers and the teachers using the proposed system that teachers do not act purely on the feedback from the system and lose their spontaneity but rather use it as assistive addition.

Prototype Development.
A prototype was developed based on feedback from the focus groups on the different aspects of the suggested system.Within this prototype, the suggested system merges with the online learning platform through the inclusion of a "Sense the Classroom" button in the navigation bar.Notably, with many groups emphasizing the value of autonomy, the system operates such that it's activated when this button is engaged.Upon activating the button, the system enables notifications that have been configured to be silent.This design decision was informed by the participants' concerns about the auditory notifications being a distraction to the online classroom.Moreover, the button activation initiates the display of a navigation bar on the right-hand side of the screen.This allows teachers to view students' LCAS in a graphical representation by activating the navigation bar.Recognizing feedback from teachers who prefer notifications only for significant events, we incorporated a feature permitting them to specify a criterion for getting notified.To maintain uniformity and simplicity during the evaluation phase, we pre-defined this criterion as " notify when more than 40% of the students are confused." Given the variety of design suggestions, we consolidated them into two primary   Figure 5: The two versions of the prototype.In both versions, the "Sense the Classroom" button is activated, which enables the notifications.The navigation bar is also activated, which triggers the graph display.
versions.The first incorporates a pie chart visualization accompanied by a pop-up icon positioned at the screen's top right corner for notification (Fig. 5a).The second version employs a bar chart visualization and, for notification, a pop-up text detailing the rationale behind the notification and a colored border surrounding the presentation (Fig. 5b).During the evaluation sessions, participants were informed that the pairing of visualization and notification type in the two distinct versions was arbitrary, with no underlying rationale for their combination.

Prototype Evaluation.
SUS Scores.The primary objective of having two versions was to encompass all design ideas rather than to compare them since both versions can coexist in a system where users select their preferred visualization and notification type.However, to account for potential strong preferences that might influence responses, participants were provided with separate SUS surveys for each version.The responses to each survey question, along with the SUS scores for the ten participants, are presented in Table 3.The average SUS scores for Versions 1 and 2 were 80.5 and 74.5, respectively.According to [19], both scores are within the acceptable range.
Discussion points.Five out of ten teachers expressed concerns that the second-by-second visualization could be distracting and offer insights they couldn't immediately act on.Instead, they recommended presenting the LCAS in a more extended timelapse for better utility.Additionally, three out of ten teachers indicated a preference for choosing the colors associated with each LCAS, suggesting that it would facilitate quicker graph interpretation.Six of the teachers expressed interest in reviewing this information postclass for detailed analysis, mirroring a point previously highlighted in the co-design activity.Lastly, individual feedback included suggestions like adopting a thermometer-style visualization for the LCAS on a scale and examining potential correlations between the teacher's spoken transcript and LCAS detection.

DISCUSSION, LIMITATIONS AND FUTURE WORK
Our study aimed to gather feedback from teachers on the design of an affective computing system for online higher education and assess the usability of an initial prototype derived from their insights.Therefore, we employed a participatory design approach and the system usability scale method to answer our research questions.The first research question was "What content of the feedback regarding the students' learning-centered affective states is the most important for teachers in online classrooms?".To answer this research question, a sorting of LCAS was performed by the participants through a card sorting activity.The results revealed the set of LCAS that teachers indicated as important to know while teaching online, namely, confusion, engagement, boredom, frustration, and curiosity.This set can be used as a starting point for future research that aims at affective computing in online education.Moreover, further investigations involving individuals who are exclusively and currently active in online teaching could be beneficial and complement our study, which primarily involved participants with additional theoretical knowledge of technology-enhanced learning.These results show that teachers in online learning are highly interested in epistemic emotions, i.e., emotions that are stimulated by cognitive problems, such as confusion, frustration, and curiosity [21].The indicated high importance of boredom as well as the suggestions for shame and pride from two groups, show that teachers are also interested in being aware of achievement emotions, i.e., emotions related to achievement activities and their success and failure outcomes.These results are in line with the research of Pekrun [21] that highlights the importance of epistemic and achievement emotions in learning.Social emotions such as contempt and anger were not indicated as important in online learning by the majority of the groups.Social emotions are considered to be important in teacherstudent interactions and group learning [21].Therefore, our results were aligned with the expectation that these are not important in our lecture setting.Nevertheless, further research is required for different lecture settings.Lastly, these results showed that the basic six emotions were not indicated as relevant in the given lecture setting by the teachers, implying that further research is needed in the detection of affective states beyond the six basic emotions [11].The second research question was "How can the feedback be provided to teachers in a meaningful manner?".To answer this research question, generated designs of the proposed system were created by the groups in a co-designing activity.Groups worked on four to five aspects relevant to the design.On the aspect of the position, the majority of the groups stated the most important is that the system is minimizable in order not to disturb their lecture.Consequently, designers of agents for online education should consider giving teachers that flexibility.Concerning the modality, the options listed by the groups were either a visualization or a visualization enriched with text.Various types of visualization were suggested, such as time series graphs, pie charts, or bar charts.Multiple groups proposed the use of colors in graphs, while some other groups found that the use of color could exclude color-blind teachers.Text enrichments that were suggested by the groups were percentages and scale indicators.The majority of the groups indicated that the interventions should be discreet and only activated by teachers.For the aspect of frequency, which was discussed in three out of the six groups, the property of flexibility was indicated as essential.To sum up, these results imply that teachers value flexibility regarding the different aspects of a system that can be embedded in an existing conferencing software while emphasizing the importance of not being overloaded with information which aligns with the cognitive load theory [32] and its implications in online environments [33].
Beyond the qualitative data from the focus group activities, participants offered further recommendations, which, while outside this study's scope, are valuable for future research.Most participants saw significant merit in individualized feedback on students' LCAS, particularly in small groups.In that way, teachers can provide personalized assistance to each individual student based on their needs, making this aspect valuable to be studied in the future.A majority of participants emphasized the advantages of obtaining a post-lesson report from the suggested system.Such feedback could influence future course design, structure, and teaching methods.We plan to delve into the design of post-event feedback on classroom affective states in future research.Our study's participants, being domain experts, provided insights from a teacher's vantage point.Yet, students might also gain from the suggested system by leveraging detected information for emotion regulation during online learning, potentially optimizing their learning experience [3].Consequently, incorporating students in subsequent participatory design research would be enriching.Lastly, the sample for this study is considered highly knowledgeable and critical due to the participants' background in the field of education.Future research should contemplate including online education teachers with a more foundational level of expertise.
The third research question posed was, "How usable is the system prototype designed based on the insights from RQ1 and RQ2?"In addressing this, we developed a prototype in two versions inspired by RQ1 and RQ2 and assessed it using the System Usability Scale method.The average SUS scores for Versions 1 and 2 were 80.5 and 74.5, respectively, both of which fall within the acceptable range according to [19].Notably, out of ten teachers, nine rated complexity low and ease-of-use high, suggesting a system viewed as straightforward and user-friendly.Additionally, the system's integration into the existing online learning environment was highly rated by all teachers.Lastly, six teachers indicated a strong likelihood of using Version 1, and five of using Version 2. The remaining teachers provided a subdued rating, suggesting a need for modifications, as detailed in the discussion points of the evaluation session.Among the dominant recommendations were extending the LCAS visualization beyond one-second intervals, offering customization of colors for each LCAS, and allowing users to select their preferred visualization and notification styles.
These outcomes can be used for the further development of affective computing systems designed for integration into online classrooms.While the specific model for this system falls outside the scope of our current study, it has been a focal point of related research in the field, as detailed in Section 2.1.Additionally, they hold the potential for real-time visualization of various learning analytics within online settings.

CONCLUSION
In this research, we present higher education teachers' insights on a system designed to provide feedback about students' LCAS in online classrooms.Additionally, we assess a prototype crafted from these insights.We collected those insights through participatory design in focus group activities and performed evaluation through the system usability scale method.The results show that teachers find that the knowledge of confusion, engagement, boredom, frustration, and curiosity of their students is important to keep the learning experience optimal.Moreover, the paper presents the teachers' generated ideas on the design of the interface of the proposed agent regarding the aspects of position, modality, intervention, visualization, and frequency.Lastly, both prototype versions received scores denoting them as acceptable, though feedback highlighted areas for potential refinement, including the system's autonomy and potential for distraction.In summary, this study's data offers insights from domain experts about the proposed system, serving as a foundation for future investigations into affective computing and learning analytics systems in synchronous online classrooms.

Figure 1 :
Figure 1: A visual representation of the study's methodology stages (a) Card sorting setting (b) Co-design setting

Figure 2 :
Figure 2: The setting of the focus groups, (a) the cards with the learning-centered affective states that participants were asked to sort based on how important they find their knowledge during an online lecture, and (b) an example of a generated design

Figure 3 :
Figure 3: The box plot showing the distribution of the _ of each learning-centered affective state over the six focus groups.

Figure 4 :
Figure 4: Designs generated by the focus groups.
(a) Prototype Version 1.In this version, students' LCAS are visualized using a pie chart, and the notification is indicated by a bell-shaped yellow pop-up icon.(b) Prototype Version 2. In this version, students' LCAS are visualized using a bar chart, and the notification includes a pop-up text explaining the reasoning behind the notification and a colored border of the presentation.

Table 1 :
The learning-centered affective states for the card sorting focus group activity

Table 2 :
The descriptive statistics of the learning-centered affective states' sorting.The numbers are rounded up to the 3  decimal.The top five ranked learning-centered affective states appear in bold.

Table 3 :
Each participant is assigned a unique identifier (ID) for anonymity.Rows display their Likert scale responses to the ten statements (labeled Q1-Q10) and the corresponding SUS scores calculated as described in Section 3.3.2.