Designing a Digital Twin for Adaptive Serious Games-based Therapy

Smart healthcare platforms continuously gather user data, allowing the application of artificial intelligence techniques for the adaptation of solutions, and providing acquired knowledge that can be shared among clinicians, therapists, and patients. Data analytics and machine learning techniques are used to extract information from the sampled data, generating knowledge so systems can learn from it. Simulation of therapeutic scenarios becomes possible, allowing inference of possible outcomes and providing tools for decision support. Therefore, this research focuses on the design of an AI-augmented digital twin towards adaptive serious-games therapy for people with disabilities (e.g., cerebral palsy, speech disorders, Parkinson’s disease) to provide solutions toward positive therapy results. Finding an adequate digital twin for therapy with people with disabilities is relevant, aiming at prediction and simulation in the context of using serious games for therapy engagement.


INTRODUCTION
Usually, people with disabilities (e.g., cerebral palsy, speech disorders, Parkinson's) should perform repetitive exercises to regain functionality [22].Gamification and serious games (SG) are tools to promote engagement and motivation to maintain long and tedious exercise routines [4,6].Smart Healthcare (SH) uses modern technologies and approaches for therapy and patient monitoring, including individualization (grouping, history, and analysis), mobile medicine, continuous health monitoring (using IoT devices), telemedicine, and disease prevention.The pervasive and ubiquitous presence of current technology provides tools for the creation of smart environments, continuously collecting data, which can be used as a source for Machine Learning (ML) algorithms and other Artificial Intelligence (AI) techniques for adaptation of solutions, according to therapeutic goals [5].Digital transformation in healthcare, with digitization of information and digitalization of processes, already provides data in a way that is better adapted for AI and ML models [13].Electronic Health Records (EHR) platforms gather large volumes of heterogeneous multimodal data (e.g., demographic, biometric, clinical analysis and reports, exercise results, audio, video, images, text, and sensor data), that quickly becomes unmanageable for interpretation by domain experts.A challenge with big data in healthcare is making data manageable and more easily interpretable.AI and ML provide solutions to cope with the volume of data by extracting relevant information and finding patterns among these elements [10].ML uses include diabetes prediction and drug response prediction relating to cancer, while image classification allows retinopathy and cancer detection [3].Deep Learning (DL) presents challenges and opportunities and multiple algorithms, with current uses including clinical imaging, EHR, genomics, and wearable device data [14].Several works use DL for information extraction and representation learning from EHR data, allowing outcome prediction, phenotyping, and clinical data de-identification.There is still work to do regarding model interpretability, data heterogeneity, and lack of universal benchmarks.
All these methods and technologies can contribute to the design and development of a Digital Twin (DT), that is defined as a digital model of a real entity [19].The concept has been applied in multiple research fields, such as science, engineering, and healthcare.A DT is essentially a software system based on the integration of AI, data analytics, IoT, virtual and augmented reality (VR/AR), allowing real-time status assessment, data analysis, risk management, cost reduction, simulation, and prediction [7].Therapists could benefit from a DT of patients allowing application of scenarios, simulation and prediction.SGs application can be more effective if relying on an adequate DT.

RESEARCH CONTEXT
The DT concept, in healthcare, is not clearly defined mainly due to the extreme complexity of humans that must be modeled digitally [20], so existing applications focus on specific biologic parts and functions [16].A Human Digital Twin (HDT) provides extra tools for personalized treatment and preventive care [15].A DT should gather multimodal data and provide a persistent representation of that acquired knowledge, allowing the application of ML algorithms for inference of possible therapeutic outcomes [17].Requirements for DT platforms are synchronization, real-time behavior, automation protocols, systems interoperability, and domain-expert involvement, among others [11].DT design guidelines include clear data visualization, plain language, and clinical workflow integration.DT application examples are decision support systems for therapy, forecast of patient's medical conditions, and personal health management for older adults.AI uses in DT focus mostly on precision and preventive health, predictive analysis, and explainability [8].
SG applications provide an interface for users, promoting motivation and adhesion to therapies while implementing sensor data capture logic, with several distinct adaptation techniques being used for personalization [9].SG and DT concepts have been integrated successfully for learning [21] but, so far as we know, not for therapy.

RESEARCH STATEMENT
The following research statement can summarize all the research work: "An AI-augmented digital twin should support an adaptive serious games-based therapy for people with disabilities, providing tools for personalized therapy results".This thesis will try to find answers to the following research questions: • RQ1: How can the concept of DT be successfully applied to SG-based therapy, producing a meaningful representation of the patient?• RQ2: Which distinct therapeutic scenarios can be applied to the DT for the prediction of possible outcomes?• RQ3: What ML methods and techniques can be effectively used to help achieve therapeutic results?

WORKPLAN
This thesis aims to research a DT for therapy with people with disabilities, taking into consideration the specific characteristics of these users, and focusing pathologies such as cerebral palsy, autism spectrum disorder, speech sound disorders, Parkinson and Alzheimer diseases.The concept will be explored, focusing on the DT's smart layer, for research and implementation of methods to produce knowledge.These AI and ML-based methods will be applied for adaptation of solutions with the goal of supporting therapy to obtain positive results.The DT will offer an expanded visualization of a patient representation, allowing application of distinct therapy scenarios for simulation of therapeutic interventions, and inference of possible outcomes.Work includes the design and implementation of DT, with its ML models and techniques, for therapy solutions focusing on disabilities, searching for adaptation of solutions.Expected contributions include: a) Systematic review of Digital Twins for therapy, b) DT proposal towards SG-based therapy for people with disabilities, c) Design and implementation of DT platform prototype, d) User studies for assessment and validation of results, e) publications at relevant conferences and journals.This work started in 2022 and is currently in its second year, expected to last until 2026.The work plan follows the scheduling and checkpoints of the Doctoral Program at Nova FCT, based on three phases: a) preparation, b) research, and c) conclusion.The preparation phase was expected to last 12 months, with the remaining 36 months focusing on research and document writing.During the preparation phase, the curricular part of the program was already completed successfully.Elaboration of the thesis plan started with a literature review on topics such as DT, Multimodal Systems, HCI, and ML, and helped define the research questions.The thesis plan document is being written to be submitted for evaluation by an advisory commission.The research phase includes analysis, design, and implementation of prototypes.Tests with stakeholders will happen at various development stages.The thesis proposal will be written during the last year of the program, followed by the document submission and discussion.Dissemination papers will be written and submitted to relevant journals and conferences.This is achievable under the supervision of Prof. Dr. Rui Neves Madeira (orcid 0000-0001-7360-3855), from Instituto Politécnico de Setúbal, and the co-supervision of Prof. Dr. Octavian Adrian Postolache (orcid 0000-0001-5055-6347), from Instituto Universitário de Lisboa.

Work in Progress
Relevant work has been done providing tools for the current research.
• PLAY platform -Previous work allowed the design and development of a computational framework and Web-based platform [2].PLAY provides tools for SG design and a repository of data from external applications.The platform allows the management of a patient profile, providing a basis for a DT of each patient.• SG Integration -Several SG use the PLAY framework through the available API, providing samples from distinct sensors, and using the platform as backend.JustPhysioKidding (JPK) [12] allows gross and fine motor rehabilitation for the upper limbs, using Kinect, Leap Motion, and Shimmer IMU.
PlayMatFun [18] focus on motor rehabilitation for the lower limbs, using a Pressure Mat and the Wii Balance.Fanima is an SG focusing on phonetic-phonological assessment for speech therapy, where PLAY provides a therapist web interface for real-time interaction during the exercises.

Publications
A paper focusing on the application of AI techniques was published during the PhD curricular studies [1].A paper introducing the work on Fanima and exposing preliminary results was submitted to Pervasive Health'23.The research has been presented at several workshops and submitted to the local IEEE IMS Doctoral Consortium, at ISCTE, Lisbon, Portugal, providing relevant insights on further development.
• Neurofeedback project -Participation in a European project focusing on adapting therapy solutions based on Neurofeedback.Different data captured during exercises is processed by PLAY.