Gamified metacognitive prompts in a higher education flipped classroom

Flipped classroom teaching approaches have increased in popularity in recent years. A common problem in these models is that students do not prepare properly for class. This study seeks to address this problem from the perspective of metacognitive reflection in order to equip students to be more capable of managing their own learning. A custom website was developed for use in a university-level flipped classroom. It provided students with access to their course content and also included three versions of metacognitive prompts, two of which included gamification. One version used structured gamification and the other made use of an open-ended gamification design. A between-subjects experiment was conducted across two undergraduate courses (n=58) over five weeks. The results showed no change in metacognitive awareness for the student group as a whole. However, the open-ended gamification group showed a significant difference compared to the guided gamification group. Furthermore, the structured gamification group showed a decrease in their regulation of cognition skills. This highlights the potential for bottom-up, open-ended gamification designs to be effective in educational situations where reflection is important. The article concludes with a discussion of the context-specific nature of gamification, as the potential gamification design implications based on these results.


INTRODUCTION
The popularity of student-centered learning approaches has risen in recent years as constructivism became the dominant pedagogical theory leading to an increasing focus on active learning [45].Advocates for a pure constructivist approach also called for a move away from traditional teaching-centred methods towards more progressive, democratized educational ideals [11,25,32,62].This means that there is a shift in the power balance within the class and more responsibility on the student to handle learning in their own way [82].Furthermore, as the world develops, there is a growing need for students to be equipped to handle these changes through the application of 21st-century skills, such as critical thinking [43] and lifelong learning [4,42].
The flipped classroom (FC) is one approach to student-centred learning [2] which has seen an increase in popularity since the COVID-19 pandemic forced many institutions to move to online or hybrid learning [21,66].This approach allows class time to be used more effectively by allowing time for active learning [18].However, the FC relies on students being capable of managing their learning so that pre-class time is used effectively [15].
This study focuses on addressing the problem of students not being prepared for class in a flipped classroom through the use of gamification and metacognitive prompting to improve metacognitive awareness.A website was developed to give university students access to their course content to prepare for class.In addition, the website included metacognitive prompts to stimulate students' reflection about their learning processes.Three versions of the prompts were developed, two of which were gamified using either elements to guide the student through the process (structured gamification), or elements that left the reflection process open-ended and provided the freedom to choose how to interact (open-ended).The effect of the three conditions of prompting on metacognitive awareness was tested.The website was used in two flipped classrooms with 58 students over five weeks.The results showed a significant difference between the two gamification groups in terms of their effect on metacognitive awareness.

Hypotheses
Metacognitive prompts have been shown to improve metacognitive awareness [8,9].As a result, the assumption is that including these kinds of prompts and requiring students to interact with them regularly should improve their metacognitive awareness.This provides a benefit for the flipped classroom since students who are more aware of their learning are likely to be better prepared for class [83].Therefore, H1 is concerned with the overall effect on metacognitive awareness in each condition: H1: Each of the three conditions (control, structured, open-ended) individually shows a significant increase from pre-test to post-test scores for overall metacognitive awareness, regulation of cognition, and knowledge of cognition.
Gamification has been linked to improved motivation [6] and engagement [38] in educational contexts.We, therefore, hypothesise that the gamified (structured and open-ended) versions of the metacognitive prompts will elicit greater engagement from students than the non-gamified (control) version and thus have a greater effect on metacognitive awareness as the students will spend more time engaging with the prompts in these conditions.Therefore, H2-H4 are articulated as: H2: The gamification conditions have a significantly greater improvement in metacognitive awareness from pre-test to post-test compared to the no gamification condition.
H3: Open-ended gamification has a significantly greater improvement in metacognitive awareness from pre-test to post-test compared to the no gamification condition.H4: Structured gamification has a significantly greater improvement in metacognitive awareness from pre-test to post-test compared to the no gamification condition.
Finally, when looking specifically at each of the gamified conditions, it is hypothesized that the open-ended gamification condition will have a positive effect on metacognitive awareness due to its open-ended nature which should allow for greater reflection.This is a key component of metacognition [23].Furthermore, while we expect the structured gamification condition to have some effect on metacognitive awareness due to the increased novelty of the condition compared to the control (hence H4), since it is guided and rule-driven it may not serve to enhance the exploratory nature of metacognition.We, therefore, hypothesise that the openended gamification condition will have a greater positive effect on metacognitive awareness than the structured gamification condition: H5: Open-ended gamification has a significantly greater improvement in metacognitive awareness from pre-test to post-test compared to structured gamification.

BACKGROUND 2.1 The flipped classroom
The flipped classroom has seen an increase in popularity in recent years as the importance of student-centred teaching methods moves to the fore [13,78].The term describes an approach to teaching and learning where the content that is usually taught during contact class time is moved outside the classroom.As a result, class time can be used for active learning and consolidation of understanding through practical exercises [1,10,53].Put differently, class time is used for higher-order skills, while time at home is spent on lowerorder skills [16,46].The benefits of this model are that instead of passive, teacher-centred approaches, students become the focus and the balance of power in the classroom shifts towards them [82].The relationship between teacher and student becomes more reciprocal [16,32].
However, the only way for class time to be used in this way is if students prepare for class by engaging with the learning content in their own time [2].This out-of-class learning usually takes place through pre-recorded lecture videos, textbooks and lecture slides.This allows students to learn at their own pace [69].On the other hand, since students are required to spend a considerable amount of time learning on their own at home, they need to be capable of both self-management strategies which involve exercising external control over the learning environment, as well as self-monitoring strategies which require exercising control over their internal learning processes [26,61].Self-monitoring strategies involve self-regulation and metacognitive processes which allow the student to determine the effectiveness of their learning and adjust their efforts accordingly [65].Without an understanding of these processes, the student will not be capable of taking control of their learning as they will be dependent on external factors such as teachers to indicate their level of success.Metacognition therefore plays a large role in the out-of-class learning that takes place in flipped classrooms [26].The increased freedom offered to students in a flipped classroom environment can give them more opportunities to reflect on their learning processes and more control over their learning environment [75,76] and existing work has shown that the FC benefits metacognitive awareness [51].Therefore, the relationship between metacognition and the flipped classroom seems to be mutually beneficial.

Metacognition
Metacognition is usually described as a person's knowledge of their cognitive processes [24].It is an important component of constructivist learning theories since learners must think about what they already know and how what they are learning relates to it [67].It is commonly divided into two main componentsknowledge of cognition and regulation of cognition [30,74].The former refers to a person's understanding of their own thought and learning processes.The latter describes a person's ability to exercise control over those processes to achieve certain learning outcomes.The more a person is capable of monitoring and regulating their thoughts during learning [72], the more metacognitively aware they are said to be [73].
Metacognitive awareness plays a role in learner achievement [19,58].Several types of support strategies have been developed to facilitate this process [7,9].This support can be direct in that it is explicitly taught to students through metacognitive training, or indirect in that it is embedded in the learning environment and encountered by students as they engage with the learning content [7].Metacognitive prompts are an example of an indirect method and are useful when a student already possesses the required metacognitive skills but does not execute them spontaneously (referred to as a production deficit [7]), as can be expected from a university-level student who already has a fair amount of learning experience [8].Prompts therefore serve as questions that focus students' attention on their thought processes and learning activities [17,52].There is evidence that metacognitive prompts can result in improved learning outcomes [7,8,17,52], but there is also evidence that prompts have little to no effect on learning outcomes [23,29,64], and this may be due to non-compliance [7].Thus, encouraging students to engage with metacognitive prompts is a challenge, one which gamification may be well-suited to address [79].

Gamification in education
Education is one of the most popular fields for the application of gamification [44].Within education, instead of trying to improve learning outcomes directly, gamification is usually used to affect the behaviours that act as mediators between the learning content and the learning outcomes [47].The most commonly studied behaviours within gamified education are motivation and engagement [54,55,57].Gamification has also been investigated in conjunction with flipped classrooms [22,27,35,37,56], but there are not many studies investigating the combination of gamification with the out-of-class component [37] and even fewer focusing on the intersection of gamification and metacognition [60].
The field of gamification has long faced criticism for being too focused on a narrow set of "elements" rather than on designing the experience of the user [12,41,48,50,80].Many examples of gamification rely on boiling the game experience down to these concrete elements that supposedly constitute the same experience afforded by games, which can easily be added to a situation, usually in the form of points, badges and leaderboards [48].While this may cause a system to look like a game at first glance, the underlying motivational and experiential outcomes are vastly different to the rich variety offered by games [41,81].One step towards designing for the same psychological experiences as games regardless of which elements are used [41] is to consider the gamification as affordances [84] rather than concrete game elements [20], which give rise to psychological outcomes and thereby associated behavioural outcomes [31,44,49].The former categorisation offers a greater variety of opportunities for gameful experiences to arise than trying to condense the complex experiential nature of games into specific components [80].
Another perspective on designing for diverse game-like experiences can be found in the categorisation of play activities defined by Caillois [14], where the concept of "ludic activities" or activities relating to play [39] is placed on a continuum ranging from ludus (systemic rule-bound, regulated play) to paidia (wild, free-form, improvisational play) [71].While the ludic component of play has been normalised by modern games, paidia is an aspect that is often lacking in the current conceptualisation of gameful experiences [80] and is usually confined to concepts such as playful design [5].Thus, scholars are calling for a change in perspective in gamification design with a look towards broadening the understanding of what constitutes a gameful experience to give rise to more varied experiences [80,81].
Therefore, this study adopts the ludus/paidia conceptualisation of play experiences.It focuses on the design of two types of gameful experiences, one based on ludus-oriented play and the other on paidia-oriented play.The ludus-oriented design (structured gamification) is concerned with the more formalised perspective of play experiences, usually embodied by games with clear rule structures [71].The paidia-oriented design (open-ended gamification) leans towards a bottom-up approach where the meaning of the experience is defined by the user, rather than the designer [80].It should be stressed that these two gamification versions are only oriented towards either ludus or paidia.We do not claim to have fully embodied either concept in the design, but rather we aimed to elicit feelings relating to each experience using specific design choices.For ease of reference, these conditions will be referred to as ludus and paidia in the following sections.The designs of the prompts are discussed further in section 3.

DESIGN OF THE WEBSITE 3.1 Overview
A website called Flip Quest was developed for this study.It was designed to be used in higher education flipped classroom courses.The primary objective of the website was to facilitate out-of-class preparation in courses that follow a flipped classroom methodology.It did this by giving students access to lecture videos and other course materials (such as lecture slides), and by including metacognitive prompts to aid students' awareness of their understanding and preparation.
The portion of the website that provided access to lecture videos and course materials was the same for all conditions.Videos were divided into weeks to align with the course schedule and kept as short as possible (between 10 and 20 minutes).Any associated course material such as lecture notes could also be downloaded.A, B and C in the appendix.

Metacognitive prompts.
The novel component of the website, and the section under study in this experiment, was the addition of metacognitive prompts to the website.The metacognitive prompts were given to the students in three sets each week -before, during, and after their interaction with the weekly content, according to the components of metacognitive regulation [3].
The "before" set of prompts was called planning prompts [74].Their purpose was to help the students plan their weekly learning by selecting appropriate strategies and allocating available resources; therefore, they were shown when the student logged on for the first time in a new course week.The planning prompts primarily required the students to think about the strategies that they would use to approach their learning during the week.
The "during" prompts were called monitoring prompts [72].They were intended to encourage the students to think about how their learning was going and to adjust their actions where necessary.The final set of prompts was called evaluating prompts [74].Their purpose was to allow the students to reflect on their learning after it had taken place to determine how effectively they implemented the strategies that they chose for themselves during the planning phase.This prompt set was shown to the students when they had watched all the required videos for the week.
Prompts were presented as popups which had to be completed to be dismissed.In this way, students were required by the website to engage with the prompts to continue watching the lecture videos.To prevent frustration, the prompts were kept short, requiring only about 5 minutes to complete each set, thus resulting in 15 minutes of reflection across the whole week.However, the students could spend as much time on the prompts as they wanted to.
In addition to being divided into three sets, the prompts themselves were also divided into three categories derived from [52] and [17].Firstly, pure metacognitive prompts required students to select learning strategies for the week and then to rate themselves on how well they implemented those strategies throughout the week.
Secondly, emotion-based metacognitive prompts asked students to reflect on their feelings about the course as a way of gaining insight into how their emotions affect their learning (E.g., "How did your feelings towards [course] affect your learning this week?").
Lastly, cognitive-based metacognitive prompts directed students to think about the learning content and their level of mastery of it (E.g., "What questions do you have about the content so far, and what will you do to address this?").
All three prompt types can be considered metacognitive, but some focus more on the emotional-or cognitive-related concepts of metacognition.Hereafter they will be collectively referred to as metacognitive prompts.
The three prompt sets (planning, monitoring, evaluating) included two to four prompts each.The monitoring and evaluation set contained at least one prompt from each of the three categories.The planning set did not contain cognitive prompts since cognitive reflection would only be effective once the student had engaged with some of the week's course content.Figure 1 shows a flowchart of the prompting process during the week.The full list of prompts is included in Appendix B.
After completing a prompt set, a summary of the student's responses was displayed on the main page of the website (see Figure C in the appendix) and a full record was shown on a separate reflection page.Thus, the student could browse their prompt history for previous weeks as well as keep track of what they had reflected on during the current week.
3.1.2Gamified prompt design.The purpose of this study was to examine the short-term effects of adding gamification to metacognitive prompts.Therefore, the website was divided into three conditions -control (non-gamified prompts), ludus (structured, feedbackfocused prompts) and paidia (open-ended, freeform prompts).All three conditions had the same prompts for each set.The difference between the conditions was in how the student could respond to the prompt.
In the control (non-gamified) version of the website, students responded using simple text/numeric input boxes and sliders.The second condition, ludus, focused on showing the students their progress through the reflection process in a structured way.Therefore, this version primarily made use of feedback affordances like gauges and modified the prompt response options to be more guided, using limited input boxes interspersed with pre-generated responses instead of large textboxes (see Figure 3).Sliders in the base version were changed to show more interactive and granular feedback in the form of a percentage and at the end of the week these ratings were represented on a graph.The strategy-selection component divided each strategy into core attributes which were represented by gauges that changed as the students rated themselves on their implementation of the strategy.Thus, the gauges became a visual way of tracking learning progress during the week and were intended to help the students focus on following their chosen learning strategies.When asked to evaluate their learning during the week, the students were presented with a selection of phrases linked to the nine common emotions experienced during learning [63] and allowed to select a set of phrases that best describe them.The use of pre-generated prompt responses, charts, and gauges was intended to guide the students through the reflection process in a structured way while providing them with feedback on their progress.
The paidia-oriented version was based on the conception of play as a free, unstructured activity within a set of guidelines that provides freedom for creative expression [77].Inspiration was drawn from the PLEX framework for playful experiences [5] and the Dadaist art movement, which embodies nonsense and irrationality.When asked to reflect on their emotions, students were given an emoji picker and a textbox without any guidelines; they were left to choose how to interpret these tools and to express themselves freely.In this sense, the paidia version was slightly less user-friendly, although this is not always a bad thing when play is involved [80].Textbox-based prompts were replaced with sentences containing words that could be clicked to be modified; this interaction was indicated through subtle, playful animation to encourage exploration of the interface.The result was often a humorous sentence constructed in parts.Each learning strategy was represented by a random image of an object, animal or person that appeared on a preloaded canvas containing a background image.The student could interact with the canvas by dragging these images around, rotating and resizing them to create a nonsensical composition of their own.In this way, the strategy choice was represented in a random, irrational way.Later on, rating one's strategy with emojis resulted in a change to the corresponding image component in terms of its style.When asked to reflect on their week's learning, students could "build their reflection" by combining the following components: an emoji, a colour, a GIF and text input.Figure 4 shows some examples of these prompts.
In summary, the website was designed to encourage metacognitive reflection through the use of prompts.These prompts were presented to the student in one of three ways, depending on the condition in which they were placed.The control condition presented the prompts in a standard way similar to a survey, the ludus condition aimed to guide the student through the reflection process, and the paidia condition aimed to give the student the freedom to reflect in any way that appealed to them.The difference between the conditions was how the student could interact with the prompt.
It should be noted that the control condition could be perceived as gameful in its own right since it also contains sliders and input boxes like the ludus version.The website was designed by first choosing the prompt questions and then assigning each prompt type a standard response type (such as an input box or slider) based on how it might be done in a digital survey.This became the control version.Then the control version was modified in the style of ludus-and paidia-oriented gamification to change how the user interacts with the prompt and receives feedback from it.For this reason, the control version is a control for this website and not necessarily "gamification-free".Since the perception of the gameful experience is subjective [41], it is not possible to design a completely un-gamified version of the prompts.
The gamification, albeit subtle, has been used to modify the experience of the user by modifying the hedonic qualities of the prompts themselves to encourage user interaction which would ideally serve to improve metacognitive reflection.The hedonic quality of an interface has been shown to have a positive effect on user experience [36].When viewed from a game design perspective, the different interfaces of the prompts in the three conditions are akin to the mechanics of a game, which give rise to the dynamics of interaction at runtime and then to the accompanying aesthetics as experienced by the user [40].In summary, the ludus and paidia conditions were aimed at eliciting two different types of experiences when interacting with the metacognitive prompts, and the goal of this study was to determine the effectiveness of those experiences as they relate to metacognitive awareness.

Participants
The Flip Quest website was used in two courses in the same undergraduate computer science-related degree at a university in South Africa.Course A was a first-year course aimed at teaching the basic principles of website development.Course B was a thirdyear course focusing on human-computer interaction theory.Both courses made use of the flipped classroom methodology by requiring students to watch lecture videos and consult class materials before class to attend class prepared to apply the material that they had learnt in practical exercises to consolidate their knowledge.
Before the study, ethical clearance was granted by the university ethics committee.Permission was not granted to gather demographic details about participants.However, it can be assumed that since course A was a first-year undergraduate course, the students were around 18-19 years old.Course B was a third-year course which meant that most students were around 21 years old.The purpose of this experiment was to test the effects of the metacognitive prompt conditions (control, ludus and paidia) on the metacognitive awareness of the students.Therefore, this study made use of an abbreviated version [28,33] of the metacognitive awareness inventory (MAI) [74].The abbreviated MAI contains 19 items that measure metacognitive awareness along two dimensions -knowledge of cognition (8 items) and regulation of cognition (11 items).The abbreviated version was used to keep the survey time manageable and because there is limited evidence confirming the 8-factor structure of the original MAI [31].
A 7-point Likert scale was used in this survey, where 1 mapped to "very unlike me" and 7 mapped to "very like me".The MAI had acceptable internal consistency in our sample (Cronbach's alpha values: pre-test data  = 0.856; post-test data  = 0.878).Internal consistency was similarly high for each of the subscales (pre-test data: Knowledge  = 0.803; Regulation  = 0.766; post-test data: Knowledge  = 0.857; Regulation  = 0.796) 4.3 Procedure 4.3.1 Experiment design.The students were randomly assigned to one of the three experiment conditions (hereafter referred to as control, ludus, and paidia).For the duration of this experiment, each student only experienced a single condition, thus making it a between-subjects study.On the first day of the semester during class time, the Flip Quest website was introduced to the students using a pre-recorded video to ensure that both courses received the same treatment.The video explained the reason for the inclusion of the metacognitive prompts since prior research shows that explaining the importance of metacognition to students can help facilitate the process [7].The video did not refer to the different conditions to avoid introducing bias.
A pre-test survey was conducted on the first day of the semester during class time to record a baseline metacognitive awareness.After five weeks, a post-test was conducted using the same survey.The survey was conducted during class to ensure the highest possible response rate.To avoid biasing the results no compensation was given for completing it.Furthermore, informed consent was obtained beforehand and if a student chose not to provide the consent, they were not asked to complete the survey.Students were also free to withdraw from completing the survey at any time.

Descriptive statistics
The survey responses were analysed using R and RStudio.Table 1shows the number of students in each condition group within each course and how many of these students completed the surveys.The final sample size of 58 students (19 in the control group, 23 in the  Course B suffered from a low response rate for the post-test.The completion of the survey was conducted during class time and class attendance for course B was low on the day when the post-test data were gathered and some students chose not to complete the survey.It should be noted that these missing students did not drop out of the course entirely.
The varying numbers of students in the conditions can be explained by the fact that students were randomly assigned to a condition and to begin with, they could not be divided exactly equally among three groups.Furthermore, when the post-test was conducted, some students had dropped out of the course causing further differences in the number of students per condition.
All statistical tests were conducted on the combined sample from both courses.It is important to note that although the sample included students from different years, the proportion of course A and course B students in was consistent across all three conditions.Moreover, the students were randomly assigned to each condition, eliminating the need for a baseline test in this study [34].A one-tailed paired t-test was conducted on the overall metacognitive awareness score for each condition.The assumption of normality was not violated for any of the three groups, as assessed by Shapiro-Wilk's test (p > .05).Two outliers were detected for the control condition, but inspection of their values did not reveal them to be extreme and they were kept in the analysis.Table 2 shows the results of the analysis.The two-tailed paired-sample t-test was conducted on each of the subscales of the metacognitive awareness inventory for each condition.The assumptions of normality were not violated, as assessed by Shapiro-Wilk's test (p > .05).The results are shown in Tables 3 and 4.
Based on these results, H1 is not supported.However, it should be noted that the ludus condition showed a significant negative effect on the regulation of cognition subscale with a moderate effect size, where the post-test scores were lower than the pre-test scores by a mean difference of .443.

H2:
The gamification conditions show a significantly greater change in metacognitive awareness from pre-test to post-test compared to the no gamification condition The ludus and paidia results were grouped to form a "gamification" group (n=39).This was compared to the results of the control group (n=19) to determine whether the gamified conditions had a greater effect on metacognitive awareness.An independent-sample t-test was run to determine if there were differences in the metacognitive awareness change between the two groups.Data are mean ± standard deviation unless otherwise stated.Metacognitive awareness difference scores for each group were normally distributed, as assessed by Shapiro-Wilk's test (p > .05),and there was homogeneity of variances, as assessed by Levene's test for equality of variances (p = .68).Three outliers were detected that were more than 1.5 box-lengths from the edge of the box in a boxplot.Inspection of Figure 5: Boxplots showing the differences between the condition groups according to the differences between the preand post-test their values did not reveal them to be extreme and they were kept in the analysis.The gamified group (-.067 ± 0.67) did not differ significantly from the control group (0.06 ± .83),t(56) = -.631,p = .53(adjusted p-value: .53;Cohen's d = -.17).We are therefore unable to reject the null hypothesis and H2 is not supported.There is no significant difference between the gamification group and the non-gamification group in terms of the change in metacognitive awareness.

H3-H5: Differences between conditions
To address H3-5, a one-way ANOVA test was conducted to investigate the differences between the conditions.The difference between the post-test and pre-test scores was calculated (post minus pre) for each condition and these values were then compared using the ANOVA test.Two outliers were detected for the control condition, but inspection of their values did not reveal them to be extreme and they were kept in the analysis.The data was normally distributed for each condition, as assessed by the Shapiro-Wilk test (p > .05)and there was homogeneity of variances, as assessed by Levene's test (p = .357).Data are presented as mean ± standard deviation.
Metacognitive awareness difference scores were statistically significantly different between the different conditions, F(2, 55) = 4.678; p = 0.013; (adjusted p-value: 0.026*; 2 = 0.145).The mean difference increased from the ludus condition (-.339 ± .48) to the control condition (.06 ± .83) to the paidia condition (.332 ± .73), in that order (Figure 5).A Tukey post hoc analysis revealed that the difference between ludus and paidia was statistically significant (p = .012),but no other group differences were statistically significant.We are therefore unable to reject the null hypothesis for H3 and H4 since there are no significant differences between the paidia and control groups or the ludus and control groups.However, H5 is supported as the paidia group shows a significantly greater improvement than the ludus group.

DISCUSSION
This study focused on the effect of three versions of metacognitive prompts on metacognitive awareness in a flipped classroom environment.Previous research has shown mixed results regarding the effectiveness of metacognitive prompts, including problems with non-compliance from students [7,17,23].This study made use of two types of gamification to determine whether the way in which the prompt was presented would affect its effectiveness.This was a short-term study, conducted over five weeks with a relatively small sample.
Based on previous work [7,8,17,52], it was hypothesized that the inclusion of the prompts would result in increased metacognitive awareness in students across all three conditions.However, our results (H1) do not show any statistically significant change in the overall metacognitive awareness of the students in each condition, but there is a slight increase in the mean scores of the paidia and control conditions.This may be due to the short-term nature of the experiment; five weeks may not be sufficient time for lasting changes in metacognitive awareness to become apparent.It may also be that the students did not interact with the prompts deeply.Although the website did require the students to move through each set of prompts, students could choose to input dummy data to get past the screen.System log data or interviews could serve to gain a deeper understanding of how students interacted with the prompts.
As a whole, the two gamified groups' combined scores did not differ significantly from the control group (H2).However, upon closer inspection of the differences between the three groups, the most interesting result is that although neither gamified group differed significantly from the control (H3, H4), the two gamified groups differed significantly from each other (H5).Therefore, the results of H2 may be explained by the fact that the overall lower scores of ludus and the higher scores of paidia combined were even closer to the control.In addition, the design of the control condition prompts compared to the ludus and paidia prompts may have caused them to be more informative in terms of metacognition as they were more straightforward in their representation.This may have resulted in them being more effective than it was assumed they would be.
The significant difference found between ludus and paidia (H5) will be explored in more detail below.

Effects of paidia-oriented gamification
The design of the paidia prompts included more opportunities to explore the prompt interfaces through small interactions and animations.The prompts in this condition were also specifically designed to encourage open-ended reflection by allowing students to ascribe their own meanings to interface components, such as using colour to rate their understanding of a topic.Gamification based on paidia-like interaction is not seen as often as ludus-oriented gamification.This may be because making an activity more playful and open-ended also often makes it vaguer.
This open-ended nature can confuse the user if they are not sure how they are meant to interact with it.This was a design challenge which was encountered when designing the paidia prompts.While the goal was to keep the interaction open to interpretation, this meant the user a metacognitive prompt and a means of interacting with it without any further instruction.An example of this is the canvas interaction (Figure F3 in the appendix) which generated a random image on a canvas each time the user chose a learning strategy.This interaction was inspired by the Dadaist art movement and intended to allow the students to "build" a visual representation of their weekly strategies by dragging the images around and resizing and rotating them.However, no instructions are given concerning this as the goal is to allow the user to explore and ascribe their own meaning to the interaction.
It is possible that the exploratory nature of these prompts was better at supporting the reflective aspect of metacognition, causing the students to spend more time engaging with the prompts and subconsciously thinking about their approach to the course and their learning.However, it is also possible that the novelty of these types of interactions led to increased interaction which may fade over time [68].Qualitative data from interviews is needed to understand more about why this condition was more successful than ludus.

Effects of ludus-oriented gamification
For the ludus group, there was a significant difference in the regulation of cognition subscale scores and the values showed that the post-test was lower than the pre-test.This subscale is concerned with a student's knowledge about their use of specific learning strategies and their ability to evaluate this use [73].Thus, their ability to regulate their learning.
The design of the ludus prompts presented an interesting gamification design challenge.Ludus-oriented gamification would generally be expected to be goal-driven and feedback-focused.It commonly makes use of feedback mechanisms such as progress bars or points to highlight goals within the system, and these components then result in some sort of reward.For example, in Duolingo, a user is given the goal of acquiring a certain amount of XP per day.They receive XP by completing language lessons.When they gather the required amount, there is generally some sort of reward provided in the form of in-game items like gems or simply through visceral feedback such as sound and quirky animations.The overall effect of this is that the user is given the goal of trying to gather XP and is rewarded when they achieve this goal.The system therefore makes use of these mechanisms to encourage users to engage in language learning lessons and progress in the system.
In terms of the design of the ludus-oriented prompts for this study, the primary challenge was that the activity of metacognitive reflection itself could not simply be gamified through the use of feedback mechanisms.This is because as soon as the activity itself is turned into a goal for the user, there is the risk of the user rushing through the activity to receive the reward.In the context of metacognitive reflection, this would nullify the effect of the prompt itself as metacognition requires ample time for reflection.For this reason, mechanisms that rewarded the completion of prompts were avoided in the ludus condition, even though this would have been the simplest way to implement this form of gamification.Instead, the focus was on trying to use feedback mechanisms such as gauges to show a user the effect of their reflection to try to make the reflection process itself the reward.In this way, the gamification components serve an informational purpose to support the role of the prompt.Put simply, the goal of the ludus condition was to reward the process of reflecting by providing information to the user about their choices.The downside of this approach is that unless the user engages with these feedback items by internalizing the information that they provide, their effect will also be lost.
H1 shows that the mean post-test scores for the regulation of cognition subscale were lower than the pre-test scores.It is possible that the design of the ludus condition was confusing for students and unless they had the internal motivation to engage with the information provided, the effect might have been damaging to how they think about implementing their learning strategies.
The implication of this is that consistent with existing literature, gamification is not a one-size-fits-all solution.It can have vastly different effects depending on the context in which it is situated [38,59,70].While the popularity of the ludus-oriented mechanisms of points, badges and leaderboards remains high, it is worth noting that despite the ease of implementation of these components, not all educational contexts lend themselves to this type of gamification.In the case of this study, the activity of reflection is not about encouraging the student to simply complete it but to actively engage with it and ludus-oriented mechanisms might not be the best fit for supporting this.However, further research, including qualitative interviews with students, is needed to fully understand the effects of these elements in the context of metacognitive reflection.

LIMITATIONS AND FUTURE DIRECTIONS
Due to the small sample size and use of a custom-made website, the results of this study are not necessarily generalizable to other student populations.However, the results regarding the effects of the ludus and paidia conditions may serve to guide the design of other educational systems and contribute to the current understanding of the effects of gamification in education, specifically regarding metacognition.The analysis was also conducted using the combined data from both courses.The course content or year level of the course may have had an impact on the results as each course was conducted slightly differently and with a different instructor.Care was taken to keep as many things as similar as possible across both courses, for example, the students in both were given a weekly class test on the course content to encourage them to prepare before class.
Future work in this area includes conducting a longer withinsubjects study in which each student moves through every condition on the website over four to five months.This would shed light on the longer-term effects of gamified metacognitive prompts on multiple additional constructs such as intrinsic motivation and autonomous learning.Furthermore, this will also be supplemented with qualitative data from interviews with students after the experiment to gain a richer understanding of the specific effects and opinions regarding the prompts and the different conditions.
Beyond the use of the Flip Quest website, the paidia-oriented gamification presented in this study represents an under-studied area of gamification as most current examples of gamification in education focus primarily on ludus-oriented interactions, with points/badges/leaderboard implementations being a prime example of this [38].Work focusing on the design and effects of paidiaoriented gamification would serve to broaden the understanding of gamification as a whole, as well as increase the variety in terms of the type of gamified systems that are designed in the future.

CONCLUSION
This study has aimed to address the need for improving metacognitive awareness in a flipped classroom model to assist the preparation of students in the out-of-class component.A website was developed to be used at a university level in the out-of-class component of a flipped class.It included three versions of metacognitive prompts, two of which were gamified using structured and open-ended affordances.The website was used in a between-subjects study in two undergraduate courses over five weeks and the effect on metacognitive awareness was measured using a pre-test/post-test approach.
The results showed that metacognitive awareness did not improve as a whole for all students, but there was a significant difference in the effects of the two gamified conditions.Furthermore, the students in the structured gamification condition showed a decrease in their regulation of cognition.These initial findings speak to the importance of considering the context when implementing gamification in educational settings as well as the potential benefit of making use of more open-ended and bottom-up gamification designs in settings where freedom of reflection is important.
Teaching students to become more aware of their thinking and more capable of independent learning is a critical skill for success in today's fast-changing world.This study has presented a more novel approach to educational gamification design than existing studies, and it has aimed to address an area of education that has not yet seen many gamified interventions.

Figure 1 :
Figure 1: The structure of the metacognitive prompts during the week Figure 2 shows two such examples and additional examples of all the prompts are shown in Appendix A.

Figure 2 :
Figure 2: Two examples of prompts in the control condition.An emotion-based prompt with a basic slider (top) and a cognitive prompt with a textbox (bottom) (see also appendix A for more examples)

Figure 3 :
Figure 3: Three examples of prompts in the ludus condition.An emotion-based prompt allows for more fine-grained reflection with a percentage (top); a cognitive prompt with a predefined response containing smaller textboxes (middle) and a structured way of reflecting on the emotions experienced during the week using an emotion wheel and predefined phrases (bottom) (see also appendix A for more examples)

Figure 4 :
Figure 4: Three examples of prompts in the paidia condition.An emotion-based prompt requiring an emoji and a single word (top); a cognitive prompt with a predefined response containing smaller textboxes and clickable words that could be changed (middle) and the strategy-selection component consisting of a canvas of editable images (bottom) (see also appendix A for more examples)

Table 1 :
The number of students in each course and condition who completed the pre-and post-test surveys

Table 3 :
TableThe results of the paired t-test conducted on the overall metacognitive awareness score for each condition to determine whether there was a statistically significant increase in scores from pre-test to post-test; Holm-Bonferroni adjustment method used (stats R package, version 4.2.0)The results of the paired-sample t-test conducted on the knowledge of cognition subscale of the metacognitive awareness inventory for each condition.*p < 0.05; Holm-Bonferroni adjustment method used (stats R package, version 4.2.0)

Table 4 :
The results of the paired-sample t-test conducted on the regulation of cognition subscale of the metacognitive awareness inventory for each condition.*p < 0.05; Holm-Bonferroni adjustment method used (stats R package, version 4.2.0)