Holistic Patient Assessment System using Digital Twin for XR Medical Teleconsultation

Assessing a patient’s social and environmental factors is crucial in medical treatment due to their impact on overall quality of life and treatment outcomes. However, current industrial solutions and research on augmented reality and virtual reality telehealth have primarily focused on training or physician collaboration, thereby overlooking the importance of assessing patients’ non-medical factors in the consulting environment. To address this issue, we propose a holistic patient assessment system in an extended reality (XR) teleconsultation environment, based on clinical evidence from the literature. By utilizing the digital twin (DT) of the patient’s environment and a physical examination assistance tool, healthcare professionals can effectively assess non-medical factors and enhance diagnosis accuracy. Through evaluations by current healthcare professionals, our system has been medically validated for its adequate integration of crucial factors in medicine and potential for real-world deployment. We believe that our system could envision a new line of research in future XR telehealth, particularly in the context of patient-centered and inclusive care.


INTRODUCTION
The proliferation of healthcare technology undoubtedly holds great promise for the general population's treatment.Artificial intelligencepowered diagnostic tools assist healthcare professionals in the early detection of growing cancer in their patients, while wearable devices allow people to track their health on a daily basis.A novel cardiopulmonary bypass technology can save an infant's life [40], and a new surgical technique presents a brand new life to a little girl who once believed her lost arm was irreversible [29].While the mentioned technologies guarantee a cure for diseases, there is a growing population of healthcare professionals arguing that another key factor should be included in the treatment [50].This key factor, is called social determinants of health, also mentioned as the "upstream factor" [5,33,48] (More details are provided in section 2.1).To illustrate the importance of assessing this key factor, we present a short scenario.
Let us imagine John, a 40-year-old man living alone in a rural area.He has been experiencing a persistent cough and shortness of breath for the past two months.His doctor, Jane has conducted various medical examinations, including a pulmonary function test, and prescribed traditional asthma medication.While the symptoms improved, the cough could not be completely eliminated.Jane attempted to increase the dosage of the medication, but the problem persisted.However, when Jane visited John's house for home treatment, she discovered that the pollen from a pine tree was the triggering factor for his symptoms.Notably, when John went on a business trip, his symptoms disappeared.This non-medical factor (in this case, living environment with the pine tree), which surrounds the patient and influences their behavior, is a significant aspect to address in medical treatment.To achieve a holistic assessment of the patient, which is considered to be the primary goal of medicine [19], healthcare professionals are encouraged to investigate the mentioned factors during their medical examination.
However, assessment of the non-medical factor is demanding because of its time and cost-intensive nature of the process [10].In this regard, telehealth is thought to be an adequate solution due to its unique characteristics in overcoming physical distance [31].Telehealth, which is defined as the use of information technology to deliver critical information for the diagnosis, treatment, disease and injury prevention, and education [34], holds numerous benefits.Some of them include overcoming the physical distance between patients and healthcare professionals, reducing the chance of pathogen infection, the cost of disease treatment, and fewer hospital visits.In addition, telehealth enables accurate diagnosis, and effective communication between stakeholders [7,23].Along with the development of augmented reality (AR) and virtual reality (VR) technology, VR-based psychotherapy [3] and AR-based surgical monitoring [13] have been proposed, and companies have introduced various telehealth platforms [2,51] with the preservation of traditional telephone-based teleconsultation [27].
While telehealth can fully leverage its potential to overcome physical barriers, we could not find any research papers or industrial solutions that enable the assessment of non-medical factors.
Additionally, telehealth has not yet become widespread in the current medical system [12].Although existing research has highlighted and made technical advancements in physician-physician collaboration, they often neglected to introduce the patient in the consulting environment [13,26,52].This resulted in overlooking the importance of holistic assessment.To address this research gap and issue, we have designed and developed a holistic patient assessment system for medical teleconsultation environments.Through the literature survey, we have identified key clinical factors and integrated them into our system.We have utilized DT technology to facilitate the assessment of non-medical factors and have implemented an interactive assistant tool to visualize the patient's personalized data for healthcare professionals and patients.We evaluated our system's medical validity with current healthcare professionals, who have extensive work experience in diverse medical settings.Our key contribution threefolds; • Developed a holistic patient assessment teleconsultation system by addressing the needs of healthcare stakeholders.
• Integrated digital twin and assistance tool that amplifies holistic assessment capability and diagnosis accuracy.
• Medical validation by current healthcare professionals with extensive work experiences from diverse settings.
The following details will be provided: First, we summarize the related research and industrial solutions that have contributed to the proliferation of telehealth.Second, we introduce our system along with its design rationale and illustrate both the user and system flowcharts.Next, we mention the medical validation and analysis results conducted by healthcare professionals.Finally, we conclude by describing our limitations and future works to address.

RELATED WORK 2.1 Importance of Holistic Assessment in Medical Context
The term "holistic" in the context of medicine refers to comprehensively viewing physiological, interpersonal, societal, environmental, and spiritual signals from a human being [14,30].Also known as biopsychosocial or patient-centered care [28], holistic care involves holistic assessment, which assesses patients from these dimensions in a longitudinal perspective [35,36].For example, a healthcare professional can holistically assess a patient by exploring their interpersonal relationships to determine whether they receive sufficient emotional support, and by examining their living conditions for any potential allergens that could trigger an asthma attack.In this medical context, each non-medical factor is defined as a social determinant of health, or more commonly, upstream factors [4,48].This upstream factor was emphasized by the U.S. Department of Health and Human Services because of its significant impact on the overall health of the general population [46].Bradley et al. [18] further stated that both local and governmental health systems should address these upstream factors, as they are crucial to public health.The benefits of holistic assessment include overall enhancement in a person's well-being [22], resulting in increased quality of life [49] and reduced medical costs.In a qualitative study with 18 healthcare professionals, it was emphasized that providing holistic care based on holistic assessment to patients is more important than solely addressing medical issues [6].However, in the current medical environment, holistic assessment is not being initiated vigorously due to medical costs and a shortage of healthcare professionals [11].
To add, healthcare professionals have identified four categories of causes for this phenomenon.These include extensive workload due to comprehensive management and traveling to patients' homes, city-centralized care organization, and healthcare professionals' safety during the visits [39].Despite the proven benefits of holistic assessment in terms of both financial and medical outcomes, as well as from the perspectives of patients and society, these pressing needs are not being adequately addressed.

Current XR Telehealth Research
Abundant research has been conducted in the field of XR telehealth, focusing primarily on two domains: training and remote collaboration.In the training domain, Sun et al [42] introduced a novel approach to train acupuncture in an MR environment.Interactivity played a crucial role in heart [41] and nursing education [32] in a VR environment.On the other hand, to facilitate effective remote collaboration among healthcare professionals, [26] utilized identical displays within users in an AR setting.An annotation tool for remote and local users in an asymmetric MR environment guided healthcare professionals in effective communication [13], and its medical effectiveness was further evaluated in [53].One of the most promising advantages of the XR environment is its limitless usage of infinite space.Previous works have shown exceptional examples of utilizing this advantage, shedding light on the emerging telehealth system.By overcoming physical constraints, XR technology expands the capacity of medicine by reaching vulnerable populations (e.g., people with mobility impairments), thus achieving health equity.This advantage is maximized when combined with DT technology [8].A good example is shown in [52], where the authors copied the 3D region of the local user and reconstructed it onto the remote user during the medical education session.However, to the best of our knowledge, there is a lack of research that primarily focuses on the interaction between patients and healthcare professionals through the use of DT.

Current Teleconsultation Research and Industrial Solution
Excluding XR technology, there are numerous research works and industrial solutions that help facilitate effective medical consultations in physically remote settings.The most common medium used is mobile phones, which enable the transmission of both audio (voice) and images.Despite not being physically co-located in the hospital, patients preferred teleconsultation for three reasons: improved outcomes, reduced treatment costs, and decreased travel time [7,23,47].However, these platforms still face challenges.Teleconsultation does not guarantee the same level of presence, and healthcare professionals cannot accurately assess the exact location of the painful area due to the lack of depth information [12].The work closest to ours is [25], which introduced a social VR clinic in a consultation setting.While this work accurately pinpointed the needs of healthcare professionals and incorporated adequate visualization into their system, still the need to integrate the assessment of upstream factors was unresolved.Therefore, in order to fulfill the needs of the medical environment by leveraging XR technology in full capacity, and to address the research gap, we propose the development of an evidence-based holistic patient assessment system for XR medical teleconsultation.
3 HOLISTIC PATIENT ASSESSMENT SYSTEM 3.1 System Overview 3.1.1System Design.Our system operates in both VR and AR environments.Enabled by advanced VR technology, patients can access healthcare services from any location, benefiting from consultations in a well-designed virtual setting.Healthcare professionals leverage sophisticated tools, including data analysis and visualization systems, for precise and immediate diagnoses.To address accessibility challenges, the system introduces augmented consultation rooms for individuals unable to use immersive VR devices due to symptoms like dizziness.Utilizing AR glasses, patients access the system, with the doctor's holographic avatar augmented by assistant tools and data analysis in their physical living environment.
3.1.2Spatial Design.The overall design of the space in the VR system is in observance of the design guidelines and previous research results about the interior and exterior design of the hospital that is beneficial for patients.The potential influence of the virtual environment's interior is stated in [9].The holistic patient assessment system consists of three spaces: a waiting room, a consultation room, and a patient's living environment.A waiting room is a space for a patient to wait for a doctor and prepare for the consultation.This room is designed as a virtual space in a VR system, the interior design should comply with a hospital interior design guidelines in order for a patient to feel comfortable and relaxed.Several interactions with virtual objects can also be designed for patients to relieve stress.In the AR system, patients can see the preliminary examination window augmented on their physical environment.In addition, there may be several virtual objects additionally augmented on the environment for patients' adaptation to it.A healthcare professional and a patient start meeting in the consultation room.The virtual waiting room aims for patient comfort, adhering to hospital design guidelines, with stress-relief virtual interactions [1,15,20,24,43].AR consultation rooms, an alternative to VR, use holographic avatars and spatial matching technology.At the same time, a healthcare professional's perspective is connected with the virtual version of the patient's living environment, in which the model is pre-scanned before the consultation.Patient living environments, crucial for accurate diagnoses, are scanned using portable devices for a DT model in both VR and AR systems.This information enhances comprehensive consultations in both environments.

Functional Design.
Two key supportive systems facilitate the consultation process: an assistant tool and a medical data analysis program.The 3D assistant tool offers a visual representation of the human body with layers like the skeletal, muscular, circulatory, digestive, and nervous systems.It helps patients to accurately describe symptoms, enhancing healthcare professionals' understanding of the patient's symptoms.On the other hand, healthcare professionals communicate with patients much more easily and effectively by using the tool to explain diagnoses and instructions.This tool improves communication between both parties.The data analysis system provides up-to-date information for healthcare professionals to assess a patient's current status comprehensively.It includes data collection, analysis, and visualization functions, gathering medical history and preliminary examination data before consultations.Results are presented as texts with infographics on a floating screen next to the patient's avatar, aiding healthcare professionals during consultations.After consultation, the prescription is updated in the database, and a summary with instructions is displayed in front of the patient for easy comprehension.

System Development
In a VR-based medical teleconsultation system, we considered a deltoid pain consultation scenario.The patient, a graduate student experiencing shoulder pain, seeks orthopedic care. 1) Patient enters a waiting room: Log in, and review a preliminary examination survey before meeting the healthcare professional.
2) The healthcare professional and patient enter a consultation room: Summarized medical history and examination results are displayed.The patient describes symptoms using a 3D tool, followed by the healthcare professional's explanation.They move to the patient's living environment.3) Healthcare professional and patient move to a patient's DT environment: Explore the DT space, assess environmental conditions, and identify causes of shoulder pain based on patient habits.4) Healthcare professional and patient return to a consultation room, and healthcare professional prescribes and provides instructions based on collected data.Communication continues using a 3D tool and data visualization screen.

Spatial Design.
In order to conduct the user scenario, we developed a prototype of the system in Unity [45].Avatars were generated using Ready Player Me [37], and the credit for the 3dimensional model of the human body goes into [44].The waiting and consultation rooms feature an urbanscape backdrop resembling the patient's environment, with a nature view curtain wall for healing [15,38,43].The interior design mimics reality (Figure 2: (A), (B)), representing a consultation room and a patient waiting room, respectively.The patient's living environment is a 3D DT of his/her lab (Figure 1: (A)), uploaded and connected to the virtual consultation room.Healthcare professionals transition seamlessly between the two spaces using avatars (Figure 2: (C)) 3.2.2Functional Design.A 3D assistant tool, positioned between a healthcare professional's and a patient's avatar, features buttons representing human biosystems that can be toggled.On the desk, both users can utilize three marker colors to draw on the 3D tool (Figure 3).Pre-uploaded medical history and examination data are visually presented beside the patient's avatar, with an additional post-conversation survey to quantitatively assess the pain extent.
For the patient's medical history and treatment-related information, it will be stored and managed in the database, categorized by each patient, in both text and image file formats.
In a patient's living environment, the survey for checking the environmental condition is shown in a healthcare professional's perspective as well (Figure 1: (B)), and all data is collected for the comprehensive analysis and summary.After all examination is finished, the total summary of the diagnosis is visualized in front of both users' perspectives, and a healthcare professional gives a patient prescription based on the visualized data (Figure 1: (C)).

MEDICAL VALIDATION 4.1 Methodology
4.1.1Expert Demographics.We recruited three healthcare professionals using convenience sampling, as per the method used in [16].Below is a detailed description of each expert's demographics.Expert A is a 58-year-old female medical doctor specializing in family medicine.With 33 years and 9 months of work experience, she currently operates her own clinic.Her primary focus is on treating patients with chronic diseases such as hypertension and diabetes mellitus type 2, as well as musculoskeletal system disorders, such as degenerative arthritis.During the COVID-19 period, she gained experience in remote medical consultation by prescribing medications through telephone calls to patients.Expert B is a 31-year-old male medical doctor specialized in orthopedics.Currently serving as an army surgeon, he previously worked for 5 years in a tertiary hospital with over 700 beds.He has been treating patients for 6 years and 9 months.Due to COVID-19, he also has experience in remote medical consulting through telephone calls.Expert C is a 26-year-old female registered nurse with a master's degree in community nursing.She currently works as a school nurse.She has teaching experience in community nursing at a domestic university and previously worked for 2 years in a neonatal intensive care unit in a tertiary hospital with over 2400 beds.Her total work experience is 4 years and 9 months.Expert C does not have experience in remote medical consulting.
4.1.2Procedure.The purpose of this study was to validate whether our prototype fulfilled two objectives: 1) reflecting major components in a medical environment, and 2) envisioning possibilities for real-world deployment.To achieve this, we conducted a validation session through a remote meeting interview using Zoom [54].This method was also used in a previous study by [21].Once the experts agreed to participate in the validation session, author A scheduled an appointment.After confirming the session time, author B sent previously recorded video material, which lasted approximately 5 minutes, via email.All experts were instructed to thoroughly review the video before participating in the session.At the start of the session, authors A and B greeted the experts, expressed gratitude for their participation, and provided information about the compensation fee as per school regulations.We followed a semi-structured interview process, encouraging the experts to interject at any time if they had additional questions or feedback to provide.All sessions were transcribed with the full consent of the experts.The session lasted approximately 40 minutes in total.

Results and Analysis
In the following section, all citations were translated into English.We preserved the original citation, except for correcting grammar and adding context in parenthesis.E1, E2, and E3 denote Expert A, Expert B, and Expert C, respectively.

Assessing Upstream
Factor Through Digital Twin.All experts expressed positive reactions regarding the assessment of the upstream factor using the DT.E3 stated, "upstream factor is assessing those (social and environmental) factors by looking at the patient's environment and lifestyle.I was surprised to see it implemented in the video."E2 also expressed amusement, saying, "(showing patient's room) This was the most novel part." In addition, he emphasized the novelty of our prototype by saying, "Giving guidelines on nonpharmacologic interventions such as lifestyle and exercise routines, as well as goals, was novel."Furthermore, E1 noted the efficacy of our prototype and expanded on the potential of using the DT to assess upstream factors, stating, "This (assessing upstream factors through DT) is good.When we (doctors) make house calls, we assess the cleanliness of the patient's toilet and check for any possible hazardous materials.Through this system, we can coach things (medical advice) regarding the patient's living situation."These quotes demonstrate that all three experts unanimously complimented the effectiveness of using the DT to assess the patient's upstream factors, finding it medically useful in such a setting.

Enhancing Diagnosis
Accuracy Through Annotation Tool.Regarding the annotation tool, experts agreed that it enhances the accuracy of diagnosis through vivid physical examination.E2 noted its intuitiveness in assessing both the patient's target pain area and surrounding body structure, stating "I can see the patient's painful area intuitively.(...) It is useful in addressing the relations with other (body) structures."Our system showed an advantage in physical examination compared to existing remote teleconsultation systems (telephone calls).This was confirmed by E1, who said, "Telephone consulting (the current teleconsultation method) cannot conduct such things (physical examination).However, through this system, the patient can explain the pain severity, as well as pointing the painful area."E3 also expressed her preference for visualization, stating, "It will be helpful to healthcare professionals.It is better to see the visualization." Although none of the experts participated in video or VR teleconsultations due to the constraint of legal regulations, we can conclude that our system has an extra advantage in conducting physical examinations.With the use of 3D human models, healthcare professionals can gather additional depth information about the area, which is currently not transferrable through audio or video streams.

Holistic Assessment Through
Visualization.Holistic assessment, which comprehensively analyzes a patient's social and environmental aspects from a longitudinal perspective, was found to be feasible within our system.E3 expressed a limitation of the current medical consultation system, stating, "When it comes to students visiting the school clinic, there are some students who come often.I roughly remember their (medical) characteristics, but I used to think about something like (giving information about) what the symptoms of this student were, and what the problem was.Past information becomes vague." It was clear that our system helped healthcare professionals gather the patient's past medical information.E1 added, "From the perspective of a medical doctor, (visualizing the patient's history) does help a lot." The possibility of assessing a patient's overall medical history and upstream factors in a longitudinal perspective was also acknowledged by E2, stating, "The median value is derived from the data.It will be effective in comparing median values." Experts also noted that our visualization technique had strengths in assessing the patient from an overall perspective, compared to the current electronic medical record system: "It is helpful to see general things" (E2), "It will be helpful for patients with complex medical histories" (E3).

DISCUSSION AND FUTURE WORK
By reviewing medical literature, we discovered the importance of assessing the social and environmental factors of a patient in medical consultations.While the upstream factor is a key component in planning and administering medical interventions, this was often overlooked due to financial and time constraints.We utilized DT to assess the upstream factor in a timely and effective manner, for example, healthcare professionals can assess a patient's compliance by observing the location of the prescribed medication.Furthermore, based on the upstream factor assessment through DT, they can also administer nonpharmacologic interventions tailored to the patient.For instance, healthcare professionals can recommend the patient to replace their office chair after the detection of a misaligned height between the chair and the table, as this can cause extra burden on the muscles.Healthcare professionals agreed upon its function and were amazed to see such an important factor integrated into the proposed system.Meanwhile, current remote teleconsultation research and industrial solutions have not been able to fully meet the needs of users (both patients and healthcare professionals) or fully utilize the underlying technology.These solutions lacked a sense of presence due to their limited immersion in environments such as telephone or video calls.Another challenge was the limited ability to perform physical examinations due to the small display sizes or data input formats.To address this, our annotation tool allows patients to precisely indicate the location of pain on a 3D human body, enhancing the accuracy of diagnosis.Additionally, our visualization tool assists healthcare professionals in making holistic and longitudinal assessments, thereby yielding opportunities toward patient-centered care.
Our work poses several limitations.First, we used convenience sampling, thus our work should be careful in generalizing the results to the entire population of medical professionals.Although the experts in our study have diverse work experience in various clinical settings, still they do not represent the entire medical profession.In the same context, the small number of participants is another limitation of our study.Third, there is a chance of missing the subtle connotations of words due to the translation process.We made efforts to stay within the same context as much as possible, but unintentional neglect of such nuances is still possible.Finally, due to the nature of a feasibility study, we did not conduct performance evaluations specifically related to the system itself, such as the System Usability Scale or the NASA Task Load Index.
While there are dimensions that require further inspection, we plan to conduct future studies based on the valuable insights gathered from our experts, as well as acknowledging the mentioned inspection.Our first objective is to integrate a local map that depicts medical clinics within the patient's living area.This suggestion was made by E3, who stated, "When the patient needs physical therapy or an invasive procedure, it would be nice (if the system could) to recommend such places.For example, when (the system) receives the patient's house GPS information, a map could pop out, and the healthcare professional could say like, 'When the pain gets worse, you should visit this clinic.'"E2 also expressed their expectation for a seamless connection between remote and local healthcare professionals, saying, "I wish there were a link with the local medical service."Based on the feedback provided by our healthcare professionals, we expect our next system can inform healthcare professionals about the local healthcare sources available to the patients.As locating the patient's access to adequate health services is part of upstream factor assessment, we aim for our system to fulfill our initial motivation more strongly.While our current work has validated the possibilities and feasibility of our system, our next step is to evaluate its performance and establish patient safety parameters.Additionally, we will address legal considerations in a stepwise manner, starting at the domestic level.

CONCLUSION
Our work is, to the best of our knowledge, the first research to integrate the assessment of upstream factors through DT and include an assistance tool to enhance precise diagnosis in a longitudinal perspective.We believe our system shows its finest utility in two populations: first, people living abroad who face challenges in receiving local medical services, and second, elderly people with mobility impairments who require routine medication for chronic diseases such as hypertension, diabetes, or pain disorders.However, the system is certainly not limited to the aforementioned population groups.As the medical paradigm shifts from treatment to diagnosis [17], we assert that our system has the potential to be integrated into the medical system, along with technical improvements.We cannot claim that our system can provide answers to the complex questions asked in healthcare settings.However, in our constant pursuit of achieving health equity and our desire to effectively utilize technology, we hope that our system can contribute to the search for complete answers.As a first step, we sincerely present the design and development process of our initial prototype and its medical validity results.

Figure 3 :
Figure 3: Left: The markers on the desk.Right: Patient marking on 3D body.