Joie: a Joy-based Brain-Computer Interface (BCI)

The size and cost of electroencephalography (EEG) headsets have been decreasing at a steadfast pace. Prefrontal cortical activity is a promising input source that is also important for affect regulation. We created Joie, a joy-based EEG brain-computer interface (BCI) which uses prefrontal asymmetries associated with joyful thoughts as input to an endless runner game where the user’s character collects coins in response. In a lab study (20 participants, 15 training sessions per participant, up to two weeks of training), we found that our experiment group instructed to imagine positive music, winning awards, and similar strategies, demonstrated significantly greater ability in activating asymmetry compared to our placebo and control groups. In our analysis, Joie demonstrates the ability for prefrontal asymmetries to be used as input to an affective BCI and builds upon prior work in this area. Training these asymmetries can teach mental strategies that have applications in mental health.


INTRODUCTION
In recent years the size and cost of electroencephalography (EEG) headsets have been decreasing at a steadfast pace, leading to EEG being integrated into eyeglasses, headphones, AR/VR headets, and more [5,27,39].This trend means that brain-computer interfacing (BCI) could now beneft a wider number of users.In humancomputer interaction (HCI), BCIs have been explored for cognitive enhancement [21,27], meditation [60], personalized learning [61], gaming [5], and more.However, emotion regulation and modulation has received little to no attention, despite the growing challenge of anxiety and depression that impacts up to 32.3% of U.S. adults [50].
In afective computing [40], sensors are employed to measure data that reliably correlates with human emotional experience.Once afect detection is performed, afective information can be provided to users to support emotion regulation and modulation.Interfaces can detect expressive afect by performing facial recognition, motion capture and speech emotion detection, embodied afect by detecting changes in heart-rate, breathing rate, skin conductance, stomach activity, etc., and neural afect by detecting brain data [6].Expressive and embodied approaches have received relatively greater attention, with much fewer examples of neural afective interfaces [47].
Creating afective neural interfaces or afective BCIs presents a number of challenges.This includes difculty in non-invasively recording afect that may originate from deep brain structures [25], the requirement of high-density EEG recordings or complex machine learning models to achieve sufcient afect detection accuracy [23], and the lack of a well-defned interaction utilizing afective information.
Hemispheric diferences in frontal cortical activity have been associated with afect [20], are detectable with as few as two EEG electrodes and alpha frequency power analysis, and are conceptualized under a well-defned model: approach and withdrawal motivation.Approach motivation is an organism's tendency to go towards a stimulus and is associated with happiness, anger [4], and increased relative left frontal cortical activity.
We created Joie, a joy-based EEG-BCI.Joie uses prefrontal asymmetry associated with approach motivated "joy and excitement" as input to an infnite runner video game (Fig 1).Joie is intended to help users learn mental strategies to modulate frontal brain asymmetries for emotion.The model underlying Joie has been used to improve symptoms of anxiety and depression in clinical studies [35,52] and is one of the most well-studied neurofeedback protocols of its kind [53].Joie aims to apply this model to help individuals learn mental strategies to modulate frontal brain asymmetries, as they relate to "joy and excitement." Lastly, since EEG systems can be integrated into glasses, headphones, or headbands, the technology has potential to move out of the laboratory and into the wild.
Prior at UIST, Aranyi et al. 's "Anger-based BCI using fNIRS Neurofeedback" demonstrated how prefrontal cortical asymmetry could be used to detect anger and serve as input to a BCI [4].Functional near infrared spectroscopy (fNIRS) was used to measure increased oxygenated hemoglobin in the left prefrontal region associated with anger.The oxygenated hemoglobin served as an index of neural metabolic activity, which be summarized as increased left frontal activity.We demonstrate that with the same model of approach motivation and the same area of left frontal brain activation, subjective "joy and excitement" can be used as an input to an EEG-based BCI.
In this paper, we present the Joie system with results from a placebo-controlled validation study.In this study, we had an experiment, placebo, and control group perform fve neurofeedback training days with Joie.We sought to investigate if focusing on "joy" or "excitement" mental strategies can enable the user to better activate relative left frontal asymmetry versus other strategies, and further, if repeated neurofeedback sessions lead to an improvement in relative left frontal asymmetry.We also began an initial investigation into user perspectives on afective BCIs, such as perceived agency, understandability and comfort.The primary contributions of this work are: (1) a proof-of-concept for a novel joy-based BCI, which helps users learn strategies for eliciting positive emotions, with a two-week placebo-controlled evaluation (2) using EEG to enable "joy" as a physiological input modality, where similar work based on approach-withdrawal was shown prior with fNIRS but not EEG (3) user perspectives on emotion-based BCI, including a oneweek follow up interview

BACKGROUND
One model used to understand prefrontal cortical asymmetry is motivational direction, also called approach and withdrawal motivation [20].Approach motivation refers to whether an organism will go toward a stimulus and is associated with greater relative left prefrontal cortical activity; withdrawal motivation refers to whether it will go away from it and is associated with greater relative right prefrontal cortical activity.Both positive and negative emotions are associated with either type of motivation.However, approach motivation has been associated with happiness, joy, excitement, anger, and jealousy [20].
Frontal EEG asymmetries are believed to be linked to approach and withdrawal motivation.In EEG measurements, the alpha frequency band is used to determine whether greater relative left activity (approach motivation) or right activity (withdrawal motivation) is occurring.Increased alpha is associated with decreased activity in a brain region [10].Thus, the region with lower alpha power is considered the more dominant region during a specifed time window [20].Research using brain lesions, sedative barbituates such as sodium amytal, brain stimulation, and fMRI has indicated that the asymmetry is functionally important.
Greater relative left activity or greater relative right EEG-alpha is thought to indicate approach motivation.Thus, greater relative left activity measured from EEG is associated with approach-related emotions such as joy and anger [4].Baseline self-reported mood may not have a relationship with frontal alpha asymmetries, and/or they may not be an indicator of general mood.However, frontal alpha asymmetries are distinct for individuals with mental illness, particularly anxiety and depression.Frontal EEG asymmetries have been of interest as a biomarker for depression and are known to be impacted by anxiety as well [9,20].Alpha asymmetry neurofeedback training studies have found reduced depressive symptoms in patient populations.In a study by Choi et al., results indicated that their neurofeedback training successfully led to an increase in alpha power over right areas, which was associated with less symptomatic depression [9].However, on a neuroscientifc level, the association between the motivational circumstances and afective responses is not yet fully understood [44].

RELATED WORK
Here, we discuss afective computing with a focus on closed-loop interfaces, which return emotion feedback to the user for a variety of applications.We also discuss BCI research with a focus on affective BCIs or BCIs investigated from HCI perspectives (e.g.see Solovey et al. [49]).As Joie combines all of these areas, we discuss and compare varying approaches.

Afect Recognition and Feedback Modalities
We defne a closed-loop afective interface as a system that employs sensors to detect embodied, neural or expressive afect [6] and returns feedback to the user.Sensing methods for embodied afect include electrodermal activity (EDA) [34], breathing rate [18], heart-rate or electrocardiogram [11], and stomach signals or electrogastrography (EGG) [58]; for expressive afect, facial recognition [37,45,51] and voice recognition [12]; and for neural afect, EEG [28] and fNIRS [4].Approaches can be active, where the user attends to the afective biofeedback as part of the interaction, or passive, where an interface changes with respect to the detected afective state without requiring the direct attention of the user.

Embodied and Expressive
Interfaces.Embodied and expressive approaches have been more investigated than neural approaches.Embodied and expressive approaches tend to be less costly, have improved signal-to-noise ratio (SNR), and be less intrusive than neural approaches.
Many active approaches have been demonstrated for helping users regulate stress levels, improve emotional awareness, or queue refection.BioFidget, for example, adapted heart and breathing-rate sensing into a fdget spinner, and was shown to help users learn breathing techniques for stress reduction [30].MoodLight used electrodermal activity (EDA) to detect a user's level of arousal and alter the color of ambient lighting [34]; in the implementation, users applied mental strategies or breathing techniques to increase or decrease their stress levels as associated with specifc colors of light.Mirror Ritual used facial recognition to detect expressive afect, generate poetry based on the classifcation, and invoke afective self-refection in the user [45].
Passive afective interfaces have helped users regulate afect while actively attending to another task.EmotionCheck provided users with false heart-rate biofeedback during stressful presentations to infuence their emotional appraisal; specifcally, false lower heart-rate led to lower self-perceived stress levels [11].BrightBeat subtly changed the brightness of a computer screen to synchronize with goal breathing rate [18].And aSpire used pneumatic feedback to help users regulate breathing rate while attending to driving [8].The main limitation of passive approaches is that they do not help users learn emotion regulation strategies on their own.
Embodied and expressive approaches are limited by the type of afective information they can record.Embodied approaches are typically limited to recording stress and arousal levels.Valence, whether an emotion is positive or negative, is signifcantly more difcult to estimate.On the other hand, expressive approaches can accurately detect valence (e.g. a smiling face, or an angry voice).
However, expressive approaches have decreased performance when afect is not visible or audible, and can lead to privacy or legal concerns with camera and audio recording.Thus, there are many environments where expressive approaches may not be suitable.

Neural Interfaces.
Neural approaches ofer the unique beneft of recording indices of executive functioning, which includes afect regulation.Neural interfaces can be used to detect frontal cortical asymmetries that are associated with approach and withdrawal motivation.Cortisol and olfactory activity have also been documented to relate to approach and withdrawal responses [15,24].However, neural activity is most realistic for use in HCI applications.
Though approach and withdrawal-related asymmetries are not specifcally valenced, they can be activated by positive-thinking strategies such as positive autobiographical memories [63] and negative strategies such as anger [4].Prior work on anger-based BCI using fNIRS neurofeedback [4] is particularly important for Joie since it demonstrated that approach and withdrawal could be used to create an afective BCI.The system demonstrated that participants could imagine becoming angry to make a villain in a virtual environment disappear, where the opacity was controlled by greater relative left fNIRS activity.Participants could do so with minimal neurofeedback training.However, the system did not train participants to help them better manage their anger levels and gave active feedback only while they focused their attention on the screen.
The primary disadvantages of neural interfaces are signal quality and form factor. EEG-BCIs tend to be prone to electromagnetic interference (EMI) and fNIRS-BCIs tend to be prone to light-based artifacts.Both are prone to motion artifacts, though fNIRS is more resilient [61].A number of EEG denoising techniques have been explored to improve SNR in the presence of motion artifacts, particularly common average referencing [22], however, SNR remains lower when compared to embodied and expressive interfaces.Further, head-mounted devices are more intrusive than wrist-mounted or non-contact devices.Bulky EEG and fNIRS devices may not be suitable to be worn outside of lab for many users, leading researchers to investigate the potential of wearable form factors for each [7,42,57].This is expanded upon in the next section.

BCI Application Domains
BCIs have been explored for nearly two decades in HCI research.Applications have focused on cognitive enhancement, including attention [21,27], meditation [60], personalized learning [61], and gaming [5,16].Though Joie was implemented with a research-grade EEG cap, many examples have used low-cost, of-the-shelf EEG devices.Vi and Subramanian demonstrated how low-cost devices could be used to detect Error-Related Negativity (ERN) to assist users with design tasks [56].EEG alpha, beta and theata activity has also been used to create adaptive interfaces that respond to user mental states without explicit commands, particularly for engagement [43].Similar to Joie, the "Teegi" real-time interface [17] aimed to help users learn about EEG signals, using a tangible character to show brain activity based on the collected data.Further, "PEANUT" demonstrated how social robots can be integrated with BCIs to teach mental imagery strategies [41].However, these were not specifc to afect.fNIRS has been used to determine cognitive workload to allow dynamic systems to respond to this workload [49,54,61].Learn Piano with BACh [61] presented a dynamic piano learning application that could use the mental workload to adjust the difculty of musical pieces that users had to play on a piano.Determining cognitive workload also encouraged researchers to study multitasking and better support it with fNIRS systems.Brainput [49], a BCI system that involves users controlling two robots simultaneously, was used to study this and demonstrated that it could improve performance metrics appreciably.

Afective BCIs.
Afective Brain-Computer Interfaces (aBCI) use brain activity to estimate ongoing afective processes and communicate estimations of human emotion to computers, and have been explored for cognitive enhancement, music, art, and gaming applications [36].Joie is an example of a novel type of aBCI.The frst demonstration of a musical emotion BCI [31,32] showed how discrete emotions (sadness, happiness, etc.) could be detected using mental imagery and output to a synthesized instrument as part of a musical quartet.Other examples of sonic aBCIs have followed, with Leslie et al. creating an aBCI to teach an indiviual about their emotional state through music [29] and also creating a collaborative sonic aBCI where two users modulate their attention and relaxation levels for shared music generation [28].Aranyi et al.'s anger-based BCI demonstrated how aBCI could be integrated into virtual environments like video games.And Dickinson et al. designed "machine_in_the_middle, " a media art piece where the facial muscles of an artist were stimulated to correspond with detected emotions [14].
Outside of the HCI and BCI felds, neuroscience studies have been conducted in afective neurofeedback training for anxiety and depression [48,53].Though improvements in mental health were not a focus of this evaluation, Joie integrates methodology from these studies.Prior HCI studies did not incorporate placebocontrolled evaluations in their protocols, which can help us control for the placebo efect of the neurofeedback interface.The neuroscience community believes neurofeedback is susceptible to placebo, motivating us to control for it [59].

Wearable and Non-wearable Form Factors.
Numerous wearable EEG devices have been produced in recent years.The research device AttentivU [27] system integrates electrodes into eyeglass frames, computes an EEG-based attention index, and returns feedback on attention levels to the user in a real-time, closed-loop system.The Muse headset from Interaxon ofers an EEG headband with real-time neurofeedback for meditation [38].Neurable has produced a pair of headphones with around-ear EEG-sensors to monitor brain signals related to "performance" and ofer users summarized feedback on performance levels throughout the day [39].And OpenBCI has created Galea, an EEG-integrated virtual reality (VR) headset [5].Thus, an advantage of creating the Joie interface with EEG is that it could be integrated into smaller, wearable form factors. Compared with fNIRS, EEG is less costly and ofers more reliable recording in wearable form factors [42].Though non-wearable form factors are less intrusive, they bring social implications [34] or privacy and legal concerns when involving cameras or microphones.

DESIGN
Joie is a neurofeedback system which maps brain asymmetries to a player's character which must collect coins and avoid obstacles as it runs down a path.Joie is intended to help users learn to increase relative left frontal brain activity.Joie is designed based on operant conditioning, a common method to reinforce EEG features in neurofeedback systems.To collect coins and be "rewarded," a user must increase their relative asymmetry by 0.85 standard deviations from baseline.To run into obstacles and be "punished," a user must decrease their asymmetry by 0.85 standard deviations from baseline.The thresholds were adapted from a study on EEG frontal asymmetry neurofeedback and afect [1].While in between these thresholds, the character does not encounter objects in the environment.The pipeline and neurofeedback mappings are given in Figure 2.
This model does not necessarily have to be a video game, hence, Joie is not intended necessarily to be a gaming contribution.We chose a video game to increase participant engagement while maintaining the experiment structure used in previous work [1,44].The video game format can be iterated on to be more engaging.Operant training can be designed to be haptic [55], olfactory [2], integrated into a separate task (e.g.programming software), and more.
Joie is a software system that can accept input from two or more frontal electrodes in the international 10-20 or 10-10 systems of EEG.Joie connects to any EEG headset that can communicate via LabStreamingLayer (LSL), a networking and time synchronization system used frequently in BCI research.LSL handles the networking, time-synchronization and near real-time access to the collected data.The use of LSL makes Joie compatible with commercial EEG headsets of many sizes and formats, with the minimum requirement that frontal electrodes are recorded.The most commonly used electrodes for frontal asymmetries are F3 and F4 [20].
We designed methods to strictly follow operant training -i.e.provide the same audiovisual stimulus for each neurofeedback condition -but help maintain engagement.Our aim was to keep only three conditions -positive, neutral, and negative -consistent throughout the game.We decided to anthropomorphize the data with virtual characters that would interact with a game environment, programmed in Unity (C#).We also added personalization, allowing users to select a character they would relate to or enjoy playing as, as well as a sky and background selection.We also added instructions to the game to stand up, stretch, and walk around in between sessions, and in addition animated the virtual character to demonstrate this action.

Signal Processing
Our signal acquisition, processing and feedback pipeline is displayed in Figure 2 and described in this section.EEG signals were collected, processed, and output in Python.The data was collected via a LabStreamingLayer inlet and programmed in Python with the pylsl library.EEG classes were output every 5 seconds via a Websocket communicating with a Unity frontend.
For every 5 seconds of EEG data, using the MNE library [19], we applied a linear detrend, a 60 Hz FIR notch flter with zerodouble phase for power line noise, and a 3 to 40 Hz bandpass FIR flter with zero-double phase.For noisy signals, we implemented a  [20] which relies on the diference in alpha power between left and right frontal areas.Operant conditioning feedback was provided based on difernces in alpha asymmetry from baseline: coin collection with a positive chime for relative left activity, car crashing with a negative chime for relative right activity, and an empty lane for sub-threshold asymmetry.
peak-detection algorithm activated by peaks exceeding +/-100 microvolts.Windows exceeding the peak detection threshold caused the character to stop moving, and were not included in training results.
We used the most well-studied alpha asymmetry index from cognitive science literature on approach and withdrawal motivation [20], given in equation 1.The index involves calcaluating the common log of the mean alpha power for left and right frontal electrodes in a given epoch, then subtracting left from right to receive the asymmetry result.We also added a linear weighted moving average of asymmetry scores [44].The 5-second window was further split into two-second windows with 50% overlap, for a total of four sub-scores.For each of these two-second windows, we computed the power spectral density (PSD) using Welch's method in the alpha frequency range (8 to 13 Hz) and took the common log of the mean of the values.The fnal score took the 4 sub-scores and computed a linear weighted average of the scores in order, with weights of 0.25, 0.5, 0.75, and 1.The threshold was set as the mean asymmetry score calculated from the 4-minute windows of EEG data collected during Levels 1 and 2, totaling 8 minutes.The fnal result was determined by whether the training asymmetry score exceeded the threshold by +/-0.85 standard deviations, with left being greater and right being less [1].

METHODS
We conducted a placebo-controlled evaluation and follow-up interview.We hypothesized frst that H1.Participants with instructions to imagine positive approach motivated content and real feedback will perform signifcantly better in the game over multiple sessions.Our rationale was that positive approach-related emotions such as "joy" should enable greater relative left frontal activity [4,20].Second, we hypothesized that H2.Participants with instructions to imagine positive approach motivation content and real feedback will have greater post-game resting asymmetry scores.Our rationale was that resting asymmetry can indicate neurofeedback efcacy, as it is a period in which cognitive load from the task does not infuence alpha values [44].Resting asymmetry should increase in successfully trained participants.Further, we had the following research questions: R1.What is the user experience of Joie?Our motivation was to begin to understand user perspectives on on agency, self-learning, trust, understandability, usability, comfort, and convenience.Next, R2.Which mental strategies did participants use, and which did they use outside the study?We sought to begin exploring neurofeedback transfer, which is the ability for mental strategies to be used outside of the neurofeedback training environment to modulate emotion in real-world settings.We also sought to explore if "joy and excitement" mental strategies would have greater success in neurofeedback transfer versus neutral controls.
In our evaluation, we had two metrics: Trainability (game performance and resting asymmetry values) and user experience (surveys and interviews).We wanted to see whether using mental strategies related to positive mental imagery -such as pets, winning achievements, or upbeat music -would improve or change trainability and user experience.This informed the design of our participant groups, which were randomly assigned in a single-blind trial: (1) Experiment Group participants received an informational video with instructions on mental strategies to use (2) Placebo Group participants received the informational video with placebo feedback, the feedback was matched to a pilot participant who increased their scores over 5 sessions (3) Control Group participants did not receive the informational video, and instead focused on a mental strategy of their own thinking, such as being calm or performing mental math

Neurofeedback Instructions
We provided experiment and placebo group participants with an instructional video on approach and withdrawal motivation and example mental strategies that may be used in the game.Control group participants did not receive the instructional video.The video described that the "left frontal hemisphere is associated with approach-related emotions.These include happiness, excitement, and even anger." It instructed the participants that in order to collect coins in the game, you need greater activity in the "left hemisphere." Participants were provided with example mental strategies that "worked well for other people." These included 1) "Thinking about a dog or cat that they love.For example, imagining their memories, playing, walking, in a specifc place", 2) "Imagining songs in their head that are upbeat, exciting, or energizing.Songs they have memorized worked better than songs they do not know well." 3) "Remembering happy moments or imagining future happy moments, such as achievements" and 4) "You can come up with your own strategies too!" Giving participants examples of mental strategies to increase their left frontal activity is a key design component of our neurofeedback protocol.While mental strategies for neurofeedback in general are not necessary and can even hamper learning and performance, for emotion neurofeedback it is recommended [35] as some strategies may be harmful but indistinguishable in terms of brain activity.In our case, importantly, left frontal activity is also associated with anger [4].Because our neurofeedback mechanism did not account for emotional valence, we provided participants with an instructional video on the approach and withdrawal motivation model.

Procedure and Environment
We asked participants to come in for 5 days over 1 to 2 weeks, and then participate in a follow-up Zoom call one week after the last day of neurofeedback training.Experiments were performed in an ofce environment with controlled temperature and a single glass wall to the left of the participant.The experimenter observed each session from the outside, back left of the participant.
Sessions 1 and 5 were 120 minutes long, and Sessions 2-4 were 60 minutes long.Sessions were scheduled with 24 to 48 hours between.At all days, there was a game of Joie, which has three training sessions.At the frst session, participants were informed that: it would take multiple neurofeedback sessions to see improvement.They should focus on mental strategies for as long as they can, and not change them too frequently.They should not move their head or shoulders too much during the game, nor smile, sing, talk, or activate facial muscles during their strategies.Participants were encouraged to stand up and stretch in between sessions, in particular if they felt fatigued by the sessions.If participants were in the experiment or placebo group, they were shown the neurofeedback instruction video on the frst day only.
In Figure 4 there is a block diagram of each training day.Here, we describe each section of the training day: 5.2.1 Setup.A Neuroelectrics Enobio 32 EEG cap was placed on the participant; details are in the following section.

5.2.2
Pre-survey.The GAD-7 and Kessler-10 were taken by the participant before the frst session.The GAD-7 (Generalized Anxiety Disorder) is a 7-item anxiety scale and the Kessler-10 is a 10-item psychological distress scale; they were used to gauge selfreported anxiety and depression levels respectively [46,62].Before each gameplay session, participants took a survey to determine caffeine intake, sleepiness and waking hours, and two questionnaires: a short PANAS, and a STAI-6 (with an additional item added).The STAI 6 (short form state trait anxiety inventory) is a condensed form of the 40-item STAI used to measure self-report state anxiety and trait anxiety [33].The PANAS (Positive and Negative Afect Schedule) was used to determine self-reported afect [13].

Gameplay.
Gameplay is shown in Figure 3 and described here in text.For each session, the participant selected a character and a scene; afterwards, the player character started running down the path.For the frst two rounds, baseline data was collected as the participant's mind wandered and the character moved randomly.Each baseline round was 4 minutes long with 1 minute of rest in between.After this, the neurofeedback rounds began, and the participant was instructed to try mental strategies; the player character would move with the recorded real-time neurofeedback.Each round of neurofeedback training (3 total) was 8 minutes long with 1 minute rests in between.Baseline rounds were then recorded once more with the same structure as the initial baseline rounds.

5.2.4
Post-Survey and Takedown.The STAI-6 (and additional item) was taken again by the participant, and participants reported sleepiness, belief in neurofeedback, and descriptions of the mental strategies used.All equipment was taken down and cleaned of.

Participants
All procedures were approved by the Massachusetts Institute of Technology (MIT) Committee on the Use of Humans as Experimental Subjects (COUHES) and all participants provided informed consent.Participants (N = 20, ages 21 to 44, age_mean = 27.65,11 males, 8 females, 1 nonbinary) reported having normal to correctedto-normal hearing and vision, no known neurological or psychiatric disorders except for anxiety conditions, no known active prescription to antipsychotics, antidepressants, anxiolytics, stimulants or mood stabilizers.Participants were compensated for their participation.

Electroencephalography (EEG)
EEG signals were recorded with a Neuroelectrics Enobio 32, with a fexible neoprene cap with cutouts for removable electrode placements.The sampling rate was 500 Hz.The electrode sites Fp1, Fp2, F3, F4, F7, F8, AF3, and AF4 from the 10-10 system were recorded.Ground and reference were on an ear clip, with reference on the front and ground on the back.Each site had hair parted to reveal the scalp and Spectra 360 salt-and chloride-free electrode gel was placed in Ag/AgCl wells, 12mm wide, at each location.In addition, a small amount of Ten20 paste was placed on both ear clip electrodes.
EEG channels were visually inspected at the start of each session after setup.Each participant was asked to perform three blinks and a jaw clench to test the response of the headset.The wireless connection was restarted if a delay greater than 10 to 15 seconds was observed, until a delay of 5 seconds or less was observed.Using equation 2, electrode locations were kept at QI < 0.5 where less QI signifes less noise.The ofset is the mean value of the waveform, the main noise is the signal power (uV2) of the 1 to 40 Hz frequency band (standard EEG band), and the line noise is the signal power in the US power line noise frequency band 60±1 Hz.

One Week Follow-Up
Participants were asked to sign up for a video call 7 days after their day 5 lab session.Calls were recorded and auto-transcribed using Zoom.The interview surveyed the user experience and feelings towards the interface.Topics included: unprompted, agency, selflearning, trust, understandability, usability, comfort, convenience.We also asked participants about whether they related with the mental strategies between sessions or after the last session.Finally, we asked participants about their feelings towards BCIs.
We used the following interview questions: (1) How would you describe your experience using the interface (within and across sessions) in general?(2) Did you feel like the interface responded to your conscious control?Do you feel like you could trust the output of the interface?
(3) Do you feel like you learned about yourself while using it?(4) How did using the game infuence your sense of agency?(5) How did you relate to the mental strategies between sessions?(6) How have you related to the mental strategies since the study?(7) How do you feel about brain computer interfaces in general?We provided participants with the following defnition of agency: "Agency is the sense of control that you feel in your life, your capacity to infuence your own thoughts and behavior, and have faith in your ability to handle a wide range of tasks and situations."

RESULTS
We sought to investigate 1) how well are users able to activate relative left frontal activity as input to the interface, particularly when comparing "joy and excitement" versus other mental strategies 2) is there a relationship between repeated training and an increase in relative left asymmetry, and if so, which group performed best at creating this relative increase and what are the potential explanations and 3) what are user perspectives on agency, self-learning, trust, understandability, usability, comfort and convenience, as it relates to an afective BCI.
We evaluated trainability (resting asymmetry values and game performance) and user experience (surveys and interviews).Resting asymmetry values were defned by alpha asymmetries recorded during pre-and post-baseline sessions.Game performance was defned by "hits" (left, no threshold, or right) secured during gameplay.The "left hits" represent relative left activation and led to an increase in points during gameplay.
The abbreviations used for experiment groups in this section are: • Experiment group (MSF): participants who received instructions for mental strategies (MS) with real feedback (F) • Placebo group (MS): participants who received instructions for mental strategies with placebo feedback • Control group (F): participants who did not receive instructions for mental strategies with real feedback (F) Three sessions had to be removed from analysis due to technical issues: P15 Session 4, P2 Session 2, P5 Session 3.

Change in Resting Asymmetry
We wanted to observe if playing Joie could create changes in "resting" baseline asymmetry scores.Resting asymmetry can serve as an indicator of neurofeedback efcacy, as it is a period in which alpha scores are not infuenced by attention levels, and alpha suppresion occurs during cognitive tasks [26].In the research study from which we modeled our methods, it was observed that a linear increase in alpha asymmetry occurred only at rest for a group instructed to increase their relative right asymmetry scores and no signifcant diference was observed for the relative left group [44].
In our study we recorded two pre-baselines, each four minutes, and one post-baseline, four minutes, for our sessions.The fnal pre-baseline value was the average of the two four-minute sessions.
First we tested if the pre-baselines are normally distributed using a Shapiro-Wilk normality test.We observed that the pre-baseline for the control group (F) was non-normally distributed (p = 0.048, p < 0.05*).Thus, we performed a Wilcoxon rank sum test (with continuity correction) when comparing the control (F) group and otherwise used a two-sided Welch two sample t-test to compare starting asymmetry indices.Welch was used due to unequal variances.No signifcant diference was observed between the experiment (MSF) and placebo (MS) groups (t = -0.11031,df = 188.47,p = 0.91) nor with the control (F) group, though the signifcance trended higher.Between the placebo (MS) and control (F) groups, a weak signifcant diference in starting baselines was observed (W = 3490.5,p = 0.041, p < 0.05 *) (Fig. 5).The control group (F) on average began with higher baseline asymmetry scores when compared to the experiment (MSF) and placebo (MS) groups, though the experiment group had a wider range of minimum and maximum scores (Fig. 5).Given data that signifes some signifcant inter-individual and inter-group variability, we are motivated to evaluate and compare the efcacy of Joie using change from baseline indices.
The change in resting asymmetry scores is defned as the withinsession change between pre-and post-baseline alpha asymmetry indices.Using this metric, we are able to better control for intersession and inter-participant diferences in EEG alpha power.We repeated the Shapiro-Wilk normality test for the change scores and found that all distributions were normal.With this, we performed a Welch Two Sample t-test to compare changes between groups to observe whether our experimental group or placebo group had a greater change in resting alpha, based on practicing "joy and excitement" focused mental strategies.We observed that the experiment (MSF) group had signifcantly greater resting increase than control (F) (t = 2.2855, df = 58.699,p = 0.012, p < 0.05 *), but not placebo (MS) (t = 1.1583, df = 60.554,p = 0.12).The placebo (MS) group also did not have a signifcantly greater increase than the control (F) group (t = 1.2698, df = 54.645,p = 0.10).

Change in Relative Left Hits with Training
Game performance is defned as the ability for a user to improve number of relative left "hits" while playing Joie across all training sessions.Participants completed fve training days with three training sessions each, totaling 15 training sessions.The group-level results and regression ft are shown in Figure 6.Left, middle and right "hits" are defned as whether an individual was able to increase or decrease their asymmetry index in the correct direction in a 5-second epoch (see Figure 2).We performed an analysis with a linear mixed efects model due to our repeated measures study design i.e. taking multiple observations from the same participants.We repeated our Shapiro-Wilk tests for normality and found that the control (F) group scores were non-normally distributed.For this group, we took the logaritihm before creating the regression model.We found that number of training sessions signifcantly predicted increased baseline-adjusted left-activation for our experimental group (MSF: p< 0.05*) while for the placebo (MS) and control (F) groups no signifcant change was observed.The complete statistics are in table 1.

User Experience
We performed 20 follow up interviews with participants who completed all 5 sessions of the study.We were able to collect transcriptions from 19 of 20 interviews; unfortunately, the audio from the interview with P15 did not save.For this session, we refer to detailed notes that the experimenter collected during the calls.Phrases that were edited for clarity are placed in brackets.We primarily report on common themes between interviews, unless otherwise noted.
With an open-ended prompt (Q1), most participants brought up positive and negative aspects of Joie.They discussed how they felt wearing the cap and how they felt towards the game.Many individuals brought up a negative feeling towards negative reinforcement.They reported it could cause distraction and anxiety, and felt it could worsen their perceived performance.Some participants (N=4) reported that the negative feedback through car hits was "frustrating." In future iterations of the prototype, this feature could be removed as negative feedback is not required to produce a reward association.

Group
Std  1: Group diferences for game performance defned as the predictive power of multiple training sessions to increase left "hit" increases For most participants (N=18) it was their frst experience with a BCI -which led them to initially describe it as "interesting", or "fun" and "exciting", though most participants agreed Joie was "tiring" after multiple sessions.One participant (P12) described the game as boring, though most other participants described being interested in the game as a learning tool despite fatigue.One participant (P5) expressed interest in continuing to use the game after the study to apply techniques learned during cognitive behavioral therapy (CBT).
Most participants (N=17) reported they could trust Joie and believed it responded to their conscious control (Q2), with the remaining participants (N=3) reporting a hesitation with trusting the interface due to its inconsistency, and a remaining participant (P6) in the placebo group who was uncertain the interface was providing real neurofeedback, and could believe it was real or not real.Importantly, the remaining placebo participants (N=6) reported belief in conscious control and trust of Joie.Many participants mentioned they felt a "latency" or "delay" between their thoughts and Joie's response.
Participants provided mixed responses for the question on "self learning" (Q3).All but one participant who reported an interest to apply their strategies in real life also responded "yes" to the self-learning question (N=7).The remaining participants provided a mixture of responses.If a participant felt that the interface was too "inconsistent", they were more likely to not report learning (P23).Other participants felt they learned how "taxing it is to focus on one thing at at time" (P15); on a related note, other participants learned how important their circadian rhythm is for their mental functioning.Some learned their best time of day for cognitive efort (P18).
Most participants felt the game did not change their sense of agency (N=10) or felt it improved their sense of agency (N=8).The remaining participants (N=2) felt the game negatively impacted their sense of agency.Participants always associated an increase in agency with a perceived game score increase.Participants associated a decrease in agency with a decrease in game score; or, a perceived lack of control often infuenced by fatigue.P8 said their sense of agency increased as they felt they could control the game with their mental strategies; however, they described it as a "struggle" and if they were tired, they felt the interface had no impact on their sense of agency.
A total of 7 participants had goals to, attempted to, or successfully used their strategy outside of the study.Importantly, participants with instructions to use positive mental strategies were the highest predictors of transfer, with a total of 6 of 7 participants being in the MSF or MS groups.The strategies individuals used included thinking of people who like and respect them, focused on a positive song and mindset, thinking about proud or happy moments of the past and exciting plans for the future, and counting breaths.Most participants who didn't use a mental strategy, discussed how the study caused them to have more meta-awareness for their thoughts and observe what they are thinking.
Participants expressed feelings of excitement (N=4), curiosity (N=13), or neutrality (N=2) towards BCIs.Neutrality is defned in this context as believing the harm of technology comes from the social context where humans apply it, and not the technology itself.Participants did not express negative feelings, fear, or hesitation towards BCIs.Note: participants all came from a large technical university and consented to having their brain recorded for a research study, thus, these opinions likely do not represent society as a whole.

Mental Strategies.
At the end of each session, participants completed a free-form text entry describing the mental strategy they used.Participant responses were each assigned a single code which described the mental imagery content and tabulated.In the experiment group, the top mental strategies included travel (N = 21 instances), positive mindset (N=18), happy memories (N = 18), pets and people (N=15), and music (N=15).For the placebo grup, the top mental strategies included happy memories (N=21), music (N=18), future successes (N=15), and game-based strategies (e.g., focusing on the reward sound) (N=6).Finally, for the control group, the top mental strategies included breathing (N=18), mind-wandering (N=12), neutral memories (N=9), and game-based (similar to the placebo group) (N=9).The results show that strategies were infuenced by the neurofeedback instructions provided to participants at the start of the experiment.
Example mental strategies used by participants in the experiment and placebo groups include thinking about positive interactions with people who appreciate and respect them, imagining uplifting songs playing in their minds, recalling a proud moment of winning an award in the past, envisioning exciting plans to relocate to a new city after graduation, and engaging in mindfulness practices like counting breaths and reciting prayers.

DISCUSSION
In this section, we discuss the changes in left frontal activity that occurred using Joie during gameplay and at rest.We contextualize and compare our work to other BCI works in HCI, aBCIs, and closed-loop afective interfaces.Further, we discuss applications, ethics, limitations and future work.

Modulation of Prefrontal Alpha Asymmetry
Our experiment group which received information on the approach and withdrawal model plus example mental strategies (e.g."think of music that you enjoy, think of future accomplishments") (see Methods) was the only group to demonstrate an improvement in relative left asymmetric activation as predicted by repeated training sessions.Placebo group participants with example mental strategies and placebo feedback matched to a prior participant, had a greater increase in left prrefrontal activity than the control group but no signifcant improvement was seen across all training sessions.Repeated training sessions also did not predict increased left asymmetric activation for the control group.
In the protocol we modeled our experiment after [44], the "left" training group was unable to signifcantly improve their resting or gameplay prefrontal alpha asymmetry with repeated sessions.The previous study participants did not receive mental strategies similar to our participants, and more closely match the results of our control group.Our control group also did not see a signifcant increase in left frontal activity during gameplay or during rest.The right prefrontal training group in the previous study saw a significant diference only at rest, but not during the training sessions.As a future consideration, the role of right prefrontal activity in neurofeedback interface should be explored more.
Interestingly, the experiment and placebo groups in our study began with slightly lower left frontal activity scores than the control group during session 1.After multiple sessions, the experiment and placebo groups were able to improve their scores, but ended with similar scores observed in the control group in session 1.This efect is very interesting as it was seen in both experiment and placebo participants (N = 14) though participants were randomly assigned to our experiment groups.One possible explanation would be diferences in anxiety and depression levels between the groups that impact frontal alpha asymmetries [20].However, each of these groups have relatively balanced anxiety levels and low depression levels, as measured by the GAD-7 and Kessler-10 inventories administered on the frst and last day of the study.
A further explanation could be the impact of imagining mental strategies on brain signals.Cognitive tasks can induce alpha suppression [26].It it possible that being given specifc mental strategies that are detailed and/or vivid -such as "walking your dog in a park" -can cause an alpha suppression induced by cognitive efort.This suppression is subsequently recovered by repeated training sessions.Increases in alpha are inversely associated with activity in a brain region, as measured by blood fow to that brain region [20].The goal of Joie was to cause right alpha to be greater than left alpha, such that relative activity in the left frontal area is greater.The cognitive efort required to imagine could have suppressed the desired increase in right frontal alpha, but with further practice the imagining required less efort and selection of mental strategies improved.

Relations with Embodied and Expressive Interfaces
Prior research in embodied and expressive interfaces (see Related Work) focused primarily on systems which performed afect detection using face and body signals and using an active or passive approach to help users with emotion regulation.Similar to active approaches such as BioFidget, MoodLight and Mirror Ritual, Joie required the user to attend to real-time feedback that helped the user devise strategies to change their afective state according with the feedback.Passive approaches such as Brightbeat, EmotionCheck and aSpire helped infuence afect during other tasks.The advantage of these systems were that they were either wearable or unobtrusive, in the form of a wrist watch, integrated into a computer screen, or an object in the environment.Though the Joie also seeks to help with emotion regulation, it is intended to be used in a separate environment for learning rather than real-time intervention.An advantage of this approach is that the user does not need Joie on hand to reduce their anxiety in the wild, and can instead "take" their mental strategies with them anywhere.However, some participants remarked in interviews that it can be difcult to train for all of the variables of a real-life situation during a video game.For this reason, a future direction that could be considered would be a wearable, mobile implementation of the EEG headset and the neurofeedback.

Relations with Neural Interfaces
Aranyi et al. 's Anger-based BCI using fNIRS [4] reported the most similar afect-based results to Joie.Both Joie and the Anger-based BCI are active BCIs and rely on frontal asymmetries to perform their emotion classifcation.As mentioned prior, left frontal activity is not specifcally valenced and can be associated with both happiness or anger.In Aranyi et al., 11 participants in a single session were able to increase their left frontal brain activity by increasing their perceived anger towards an "evil" character.This brings into question how the single-session success could be attributed.Users of Joie required multiple sessions to learn how to increase positive left frontal activity, thus a possible explanation may be that focusing on positive emotions increases the difculty of the task.Another explanation is that fNIRS has the advantage of higher spatial resolution, and may be more suitable for detecting hemisphereic asymmetries.However, simultaneous EEG-fMRI has shown that similar asymmetry features can be detected with EEG [63].Further, Aranyi et al. reported that "physical" strategies i.e. imagining angry physical confrontation was most successful (75% success rate) versus combination verbal and physical or only verbal strategies.In comparison, participants in our study did not report strategies specifc to motor imagery.In future studies, positive strategies that use motor imagery versus non-motor imagery could be compared as part of an investigation to determine the infuence of the sensorimotor cortex.Motor imagery involving the left or right arm generates asymmetric sensorimotor mu rhythms that could potentially be detected from frontal EEG electrodes.
Joie's learning protocol with multiple sessions is most similar to the structure of learning in Mind-Full [3].Both Joie and Mind-Full used a EEG game to help users regulate their afect and positively reinforced learning over multiple sessions, though Mind-Full focused on children and adopted a 16-week feld evaluation including parents and teachers.Mind-Full successfully reduced anxiety scores for their users, however they did not report on whether usage of the app changed the EEG features they were recording in their study.The ability for Joie to reduce self-reported anxiety was not a focus of our initial study.
Participants told us in interviews that Joie provided value in learning, exploration, and increased meta-awareness of thoughts.Teegi [17] provided similar value to its users, and was meant to be an educational and exploratory tool.In addition to surveys, Frey et al. used behavioral and observational measures to evaluate the user experience.Their results indicated that interacting with the interface enhanced awareness and curiosity, which is similar to what our participants described to us during post-session interviews.Similarly, boredom was reported at a low instance rate: 2 of 10 participants for Teegi, 1 of 20 participants for Joie.They analyzed study session videos and recorded facial sentiments to reach this conclusion.Lastly, their participants also reported learning while participating in the study -though their measure of learning was for learning in general beyond learning about oneself.
Other BCI in HCI works discussed in this paper used passive and not active protocols to help their users achieve a goal state, such as Prinzel et al. [43], Learn Piano with BACh [61], and BrainPut [49].These studies showed how passive detection of brain states can be seamlessly integrated into a task to improve learning and performance.For the asymmetry model used in Joie, we can consider an alternative use case of passively detecting high-threshold changes in left or right frontal activity.These changes can be used as an emotional input into a passive BCI.

Limitations
The number of sessions over which our study occurred can be low in comparison to what is needed to achieve long-term reinforcement [53].Further, the protocol we modeled for our methodology was run with 20 participants in the left-increase and 20 participant in the placebo group.Our total participant count per group was much smaller, with less than 10 per each study group.Further analysis must also be conducted to evaluate the potential impact of Joie's method on self-reported afect, and whether it can eventually be used to help with emotion regulation in-the-wild.Lastly, the underlying mechanism of frontal alpha asymmetries and emotion regulation are still not completely agreed upon in neuroscience [20].

Future Work and Conclusion
In addition to being associated with approach-motivated emotion, increased activity in the relative left prefrontal brain region has been associated with improved symptoms of anxiety and depression [9,53].In future work, Joie can be evaluated in longitudinal studies for improving mental health.Society is experiencing a growing challenge of mental health and wellbeing.This year, 32.3% of U.S. adults or over 100 million individuals reported symptoms of anxiety or depression, with higher rates among younger individuals, the highest being 45.4% for individuals aged 18 to 29 years [50].Though there are systemic and environmental causes that can be difcult to address with technical approaches, human-computer interaction (HCI) research ofers an opportunity to improve individual, interpersonal, and community levels of mental health.Mental health and wellbeing has grown rapidly as an area in HCI in recent years [47]; most research has been conducted in diagnosis or self-tracking, leaving unexplored opportunities to adopt novel technologies and therapies into interfaces that can assist individuals with mental health challenges.
Bio-and neurofeedback have been shown in meta-analyses to be efective in improving symptoms of anxiety and depression [48,53].Specifcally, electroencephalography (EEG) and breathing rate biofeedback have shown greatest efcacy for anxiety disorders in comparison to electromyography (EMG), heart rate, electrodermal activity (EDA), and skin temperature biofeedback [52].So, EEG can ofer greater efcacy for symptom improvement in anxiety disorders than body-based electrophysiology methods.This motivates us to further investigate the design and usability of neurofeedback for mental health in the future.
Joie demonstrates how applying the approach and withdrawal motivation model can be used to create afective BCIs.This insight builds from prior work at UIST on anger-based BCI with a new modality that is lower in cost and found in many wearable form factors.In our study, we asked participants to perform mental strategies of "joy" and "excitement" while comparing with participants who did not receive any instruction.We observed a signifcant increase in left prefrontal activity for the group in our study which received information on mental strategies.Follow-up surveys demonstrated further that participants who had information on mental strategies were much more likely to use them unprompted outside of the study environment.Insights into the user experience demonstrated that individuals found value in increased meta-awareness and observation of their own thoughts, regardless of their study group.The next step for Joie would be to evaluate if the increases in left prefrontal activity also had an impact on emotion.With successful modulation of brain activity related to emotion, individuals with anxiety and depression could potentially have reduced symptoms after using Joie for an extended period.The principle of Joie can be applied to situations where learning mental strategies to modulate prefrontal brain asymmetries for emotion is important, including well-being, art, and more.

Figure 2 :
Figure 2: Joie's neurofeedback design.The system uses the most well-studied asymmetry index from psychophysiology literature[20] which relies on the diference in alpha power between left and right frontal areas.Operant conditioning feedback was provided based on difernces in alpha asymmetry from baseline: coin collection with a positive chime for relative left activity, car crashing with a negative chime for relative right activity, and an empty lane for sub-threshold asymmetry.

Figure 3 :
Figure 3: Joie gameplay fow.Users begin by selecting a customized character and environment, then exploring the virtual environment by running down a path.The frst two levels are baseline rounds where users are instructed to mind wander, each lasting four minutes.Then, there are three neurofeedback rounds.The game concludes with a post-baseline "rest" which repeats the pre-game baseline.

Figure 4 :
Figure 4: Experimental procedure for neurofeedback training days.Each participant performed fve neurofeedback days with three training sessions each, for a total of 15 training sessions.

Figure 5 :
Figure 5: Comparison between starting resting baseline asymmetry between groups, as well as the pre-post change in resting asymmetry across all sessions.

Figure 6 :
Figure 6: Linear regressions for left frontal "hits" across 15 training sessions, all participants, for the experiment (MSF) group, placebo (MS) and control (F) groups.