Cognitive load assessment based on VR eye-tracking and biosensors

In this paper I present the status of my doctoral research project, a general overview of the research topic and future developments. The main research focus will be to study and develop an extended reality solution for cognitive load assessment in adaptive virtual environments, based on eye tracking and bio-signals. The main objective is to respond to the need for healthcare and training becoming more personalized and location- and time-independent. The end goal is to establish a framework that serves as a quantitative basis for adaptive rehabilitation and training by pushing cognitive load assessment towards ubiquitous computing through immersive technologies.

With the growing interest in virtual reality (VR) applications in the medical field there is a greater emphasis on medicine and digital therapies becoming more personalized and tailored for individual patient needs.From this PhD research project's perspective, of interest is the use of VR in conjunction with digital biomarkers such as eye-tracking and biosensors for cognitive load assessment.While inside a virtual experience, the mental focus seems to lie on the elements of the digital environment and, as such, measurable cognitive parameters are fully controllable by the applications algorithms.In cognitive load assessment, particularly neurodegenerative disorders, current immersive digital experiences have the main advantage of providing an alternative screening modality for cognition.The main disadvantage is that they are only based on VR environments alone, not using integrated approaches such as VR eye-tracking pair or bio-signal combinations.To tackle this disadvantage and to further enhance location-and time-independent nature of immersive technologies, the PhD research aims to answer the following questions: • How can immersive technologies be improved by eye tracking and biosensors?• How can cognitive load assessment be used as feedback for adaptive virtual environments?• What is the impact of such a cognitive load based adaptive environment on training and rehabilitation?
The result provided by this PhD research will enable an integrated approach towards adaptive environments based on physiological data.This strategy of applying a unified optimized methodology holds for the integration of feedback mechanisms to create novel immersive environments, increasing the chances of a positive outcome.

RELATED WORK
Several studies have shown that VR technologies can be implemented in clinical settings and trials targeting cognitive assessments.Cognitive load assessment using VR environments is being used for the assessment of mood disorders [1], psychosis [1], schizophrenia [2], and neurodegenerative disorders [4] such as Parkinson's disease [3] [7] or dementia [8].These findings prove that immersive and especially VR experiences are well-fitting applications for adapting to a user's cognitive metrics.
The current state of the art regarding VR based cognitive load assessment involves the use of CAVIR (Cognitive Assessment in Virtual Reality) [1] [5], an immersive VR cognitive assessment test of everyday life functions performed in a virtual kitchen, AGT (Art Gallery Test) [6], a VR based cognitive assessment based on visual search in an art gallery scenario, Virtual Reality Functional Capacity Assessment Tool (VRFCAT-SL) [7], real-world tasks implemented in virtual environment, Box and Blocks Test [3] [8], Box and Blocks test [3], University of California Performance based Skills Assessment (UPSA) [7], Complex Task Performance Assessment (CTPA) [7], or Functional Assessment Short Test (FAST) [1].

PROBLEM STATEMENT AND HYPOTHESIS
The PhD study will focus on developing an adaptive system that makes use of both virtual reality and digital biomarkers for cognitive assessment.The main hypothesis is that by integrating eye tracking, biosignals and virtual reality into a multimodal system, and using the user's cognitive assessment as feedback, then a big leap forward for adaptive immersive technologies can be achieved.
Combining immersive environments with eye-tracking and sensor fusion for evaluation purposes is a novel trend that has only emerged in the last five years, with the increase in VR devices that have integrated eye-tracking capabilities [10].The PhD research proposes to collect and analyze digital biomarkers by means of wearable devices, and eye-tracking, and use the result as feedback for adapting training and rehabilitation to the user during the VR session.The detection of stress levels, fatigue, and estimation of cognitive load using bio-signals from wearable physiological sensors is an established area of research.The most studied data modalities taken into consideration for the PhD are skin conductance and temperature, heart rate and heart rate variability, and photoplethysmogram [11].Also, stress indicators will be determined from eye-tracking data, including blink rate, pupil dilation, fixation, and saccades [12].
The proposed research will be conducted in two use cases: rehabilitation in neurodegenerative diseases, and adaptive training for healthcare providers.The main motivation for selecting these use cases is the current need for the medical field to become personalized and location independent.In both use cases the immersive environment will be adaptive by using a cognitive load-digital biomarker pair as feedback.

CURRENT STATUS AND FUTURE WORK 4.1 Current Status
I am currently working on developing a fully functional data acquisition framework.Up until this point I have managed to successfully integrate VR environments with eye tracking and biosensors, and to perform several tests.The test scenario involves the use of a virtual garage in which the test subject must find five objects.Once an object is found, the subject must focus his gaze on that specific object for 3-5 seconds for that object to be considered "found".Also,  during session run, at a random moment in time a distraction occurs, to better simulate real life instances.The distraction involves one of the objects inside the garage falling, followed by a loud noise.The main purpose of this entire setup is to evaluate the cognitive load during a visual search and focus task, and to evaluate the cognitive response of the subject during a distraction.It is worth mentioning that during this immersive experience the subject's heart rate and galvanic skin response are measured using wearable devices, while the eye tracking parameters area is measured with the built-in eye tracker of the VR headset.
The initial results point to the fact that while inside an immersive environment, there is a link in changes occurring in cognitive load, heart rate variation and galvanic skin response.Figure 1 presents digital biomarkers' evolution in time during a virtual reality setting, while figure 2 displays the virtual environment.
In figure 1 the red line highlights the moment where the distraction occurred.As seen after the distraction, the cognitive load (yellow graph) and the galvanic skin response and heart rate increase.Also, it is worth noting that each peak on the cognitive load graph, before the red line, corresponds to the test subject finding an object and focusing on that respective object.In this instance the galvanic skin response and heart rate increase as well.In figure 2 the virtual environment is displayed.The interaction between the user and the virtual setting is done through gaze, represented as white circle, by highlighting the objects that need to be found.

Future Work
Future work will involve building an updated framework that will enable real-time sensor data acquisition and processing.Currently, sensor data is processed at the end of the running session.Also, future work will also include a gamified approach towards training and rehabilitation using cognitive load as feedback.One research idea is to have multiple visual search and focus levels inside the same virtual environment with various stages of difficulty, the access to superior levels and subsequent update of the virtual scene being enabled by the cognitive load assessment of the subject at the end of each session.

BROADER IMPACT
The core mission of the PhD is to perform research in the field of adaptive technologies for healthcare with the purpose of developing immersive environments tailored to user needs.By the end of my PhD studies in the second half of 2027, the proposed system is to be delivered and functional.The integration of immersive technologies and digital biomarkers, together with a gamified approach based on user cognitive load towards clinical rehabilitation and training represent unique and novel aspects of the PhD research.

1 INTRODUCTIONI
am doing my PhD at NOVA University Lisbon, Portugal under the supervision of Prof. Rui Neves Madeira in his research group at NOVA LINCS, a research lab focused on Computer Science.I am currently in my second year of PhD having started in March 2023 and aim to finish it in 2027.In Portugal, the PhD Thesis usually consists of 4 or more number of publications in journals or conferences and then a summary linking them all together.The research component of my PhD is carried out at the Center for Digital Health and Social Innovation of the St. Pölten University of Applied Sciences (STPUAS), where I am employed as a Junior Researcher.At STPUAS I work closely with Dr. Vanessa Leung, my secondary PhD supervisor, in EyeQTrack, an Austrian-funded project to develop innovative solutions in adaptive XR training and rehabilitation.The joint supervision of my doctoral studies was made possible by the European University Alliance E 3 UDRES 2 , to which both Prof. Madeira and Dr. Leung belong.
, a fully immersive VR based version of the Box and Blocks test for upper limb function in Parkinson's, and Virtual multiple errands test (VMET) [9], an immersive exploration based assessment placed in a virtual supermarket, for executive functions deficit.Trial based studies of the current state-of-the-art solutions have shown that VR tests for cognitive load assessment display the same statistical relevance as well-established tests such as Montreal Cognitive Assessment test (MoCA) [5] [6], Abbreviated Mental Test (AMT) [8], Mini-Mental State Examination (MMSE)

Figure 1 :
Figure 1: Digital biomarkers during a virtual reality setting.