How Much Decision Power Should (A)I Have?: Investigating Patients’ Preferences Towards AI Autonomy in Healthcare Decision Making

Despite the growing potential of artificial intelligence (AI) in improving clinical decision making, patients' perspectives on the use of AI for their care decision making are underexplored. In this paper, we investigate patients’ preferences towards the autonomy of AI in assisting healthcare decision making. We conducted interviews and an online survey using an interactive narrative and speculative AI prototypes to elicit participants’ preferred choices of using AI in a pregnancy care context. The analysis of the interviews and in-story responses reveals that patients’ preferences for AI autonomy vary per person and context, and may change over time. This finding suggests the need for involving patients in defining and reassessing the appropriate level of AI assistance for healthcare decision making. Departing from these varied preferences for AI autonomy, we discuss implications for incorporating patient-centeredness in designing AI-powered healthcare decision making.


INTRODUCTION
With artificial intelligence (AI) entering the healthcare domain, growing attention is given to the potential roles of AI in specific medical functions such as diagnosis and clinical treatment [1].The introduction of AI can significantly change the roles and interactions of healthcare stakeholders [3,22].For instance, AI can take over repetitive tasks of healthcare providers (HCPs) [4] and take part in more complex tasks such as clinical decision making, potentially as a new actor next to patients and healthcare providers [39,42].The integration of AI into clinical decision making is anticipated to improve diagnostic accuracy and advance the quality of personalized and preventive care for patients [32].
Researchers in human-computer interaction (HCI) have investigated clinicians' perceptions and experiences of AI-powered clinical decision-support tools (DSTs), shifting the research focus from labbased evaluation of algorithmic performance towards real-world uptake by stakeholders in practice [29,40,41].Their studies revealed various cognitive, psychological, and contextual issues that clinicians face in adopting AI tools in their practices (e.g., issues of intelligibility, transparency, trust, professional autonomy, etc.) [18,39,40].To overcome these challenges, there is ongoing research on improving the explainability of AI [29], which can help clinicians understand the underlying mechanisms of algorithmic output better and set a realistic level of trust in AI.Also, more concrete design concepts, such as "Unremarkable AI" [41], have been suggested to mitigate poor contextual fit and integrate AI recommendations in a way that is not obtrusive to the existing routines and practices of clinicians.
While these previous works have generated an in-depth understanding of how experiences with AI-powered DSTs can be improved on the clinician side, it is underexplored how patients-the eventual beneficiaries of AI-powered decision making-perceive and prefer the use of AI in their healthcare decision making.In a recent study on patients' perceptions of human-AI interaction in healthcare [17], the authors concluded that patients may not be ready to accept and use clinical applications based on AI due to concerns regarding various potential risks.This highlights the necessity of including the voice of patients in envisioning desirable implementations of decision-support AI in healthcare.The need for including patient perspectives has never been neglected in HCI research.For instance, previous DST research has emphasized that healthcare decision making is the process of active patientphysician interactions.It argues that patients' preferences should be considered in the design of AI-powered DSTs to realize the core value of patient-centered, shared decision making [18,40].Nevertheless, since present HCI research has primarily focused on AI developed for clinicians [3,29,40,41,43], the needs of patients were discussed mainly from the perspective of healthcare providers, missing input from patients themselves.
In this paper, we aimed to address this research gap (i.e., patients' needs) by inviting patients to explore potential of AI-powered DSTs and enabling them to express their perspectives and preferences regarding the use of AI-powered DSTs for their healthcare decision making.The central research questions we aimed to answer were: (i) What extent of AI autonomy do patients find preferable in assisting healthcare decision making?and (ii) How do patients want to use AI for their healthcare decision making?To investigate these questions, we developed a fictional story and speculative AI prototypes demonstrating three levels of AI autonomy for decision making in a near-future pregnancy care context.By leveraging the prototypes, we elicited patients' preferred choices of using AI in an interactive narrative.This fiction-based, speculative design approach [12] enabled us to demonstrate probable, plausible, and possible futures of AI-powered healthcare decision making with concrete design artifacts and narratives.Also, this speculative enactment [14] helped us to evoke participants' active reflections and discussions towards preferable futures of AI-based healthcare decision making.Through semi-structured interviews, we collected responses of 12 women who experienced pregnancy in the last three years.After they went through the interactive narrative, we interviewed them to discuss the motivation behind their choices in the story and their preferences for AI-powered decision making in healthcare.The patterns in their in-story choices were complemented with additional responses of 15 women who went through the same interactive narrative and shared their responses via an online survey.We analyzed the common patterns in participants' instory choices and emerging themes in the interviews.The analysis revealed that participants' preferences for AI autonomy varied per person and context, and may change over time.Based on these findings, we discuss potential ways to incorporate patient-centeredness in the design of AI-powered healthcare decision making.

BACKGROUND & RELATED WORKS 2.1 The Scope of Decision-Support AI & Research Context
According to Yang et al. [42], clinical decision support tools (DSTs) can assist in three tasks: making a diagnosis (diagnostic DST), selecting a treatment option (therapeutic DST), or making prognostic predictions of a disease or outcome of a treatment (prognostic DST).
In this paper, we focus on prognostic AI that supports decision making with predictions on the risk of a disease and likely outcomes of potential treatment options.This focus is motivated by the Digital Twin project in which we initiated this study.The notion of a Digital Twin (DT), defined as a dynamic virtual representation of a physical system [37], originates from the field of engineering, but is now also emerging in the healthcare domain as a novel technology that uses holistic biomedical data of a person to advance personalized and preventive care for individual patients [5,6,28,35].This vision motivated collaborations between medical researchers, healthcare providers (e.g., gynecologists), data scientists, and designers in our project to use this technology to improve the care of pregnant women who are at risk of developing pre-eclampsia-persistent high blood pressure during pregnancy.Within this research context, we explored how the DT-based predictions on the risk of pre-eclampsia could be used to support preventive care decision making for pre-eclampsia.DSTs can support both clinicians and patients [42].Clinicianfacing DSTs mainly focus on reducing cognitive errors and burden and supporting clinical problem solving.Patient-facing DSTs focus on educating patients about their situation and treatment options.Our primary focus in this paper is on AI-powered decision making from the patients' perspective.This focus is motivated by the increased adoption of Shared Decision Making in healthcare which respects patients' preferences, autonomy, and values in clinical decision-making processes [15].This is a big change in clinical practices, which shift from a paternalistic approach in which a clinician makes decisions for the patient towards a patient-centered approach in which clinicians actively involve patients in health decision making and support patients in making a decision that aligns with their personal preferences and values [16,24].When AI-powered DSTs are used in this patient-centered approach, it is not only important to ask patients for their preferences regarding treatment options, but also for how they want to use AI in their decision making (e.g., Do they want to use AI?If so, how would they want to use it?).
There is ongoing research on patient-facing DSTs that are developed to educate patients and help them reflect on their treatment preferences and values to support shared decision making.For instance, Embodied Conversational Agents-anthropomorphic conversational interfaces-have been developed to deliver educational information to patients by simulating face-to-face conversations with HCPs [7,45].Also, Raj et al. [31] developed a tool to support diabetic patients' care decision making by enabling them to navigate and reflect on multidimensional health data based on episodic narratives.More recently, a few studies have explored the potential of prognostic AI specifically designed for patients [10,11].For instance, Jayakumar et al. [19] developed a personalized report that includes machine learning(ML)-based predictions on the benefits, risks, and the likely improvements of the quality of life and bodily functions to support patients in considering knee replacement surgery.This experiment revealed significant improvements in decision quality, the level of shared decision making, and patient satisfaction among the intervention group (with the personalized report) as compared to the control group which received educational materials only.This shows the potential benefits of patient-facing prognostic AI in improving shared decision making.Nevertheless, if and to what extent patients are willing to adopt the use of such AIdriven tools in their care decision making is still an open question considering various perceived risks ascribed to AI (e.g., reduced patient-HCP communications due to the implementation of AI, lack of human-like empathy and compassion in AI, trust and accountability issues, and privacy concerns) [17].It is also yet to be fully explored to what extent patients would allow AI to take over their own decision-making autonomy.

Configuring Human-AI Autonomy in Healthcare Decision Making
The importance of designing the right level of computer autonomyor technical autonomy-and human control has been noted in various domains of intelligent and autonomous systems, ranging from recommender systems to autonomous vehicles, and autonomous weapons [25,33,34,38].Designing AI autonomy determines the distribution of control, agency, and responsibilities among human and non-human actors involved in decision making [34].To provide a systematic way of investigating and configuring the appropriate level of human and computer autonomy, the levels of autonomy have been defined with varying granularity.For instance, Mackeprang et al. [25] suggested ten levels of automation in the context of a human collaborating with a computer for ideation.According to their definition, the levels of AI assistance can be as low as Level 1, where the computer offers no assistance and the human takes all decision and actions, and as high as Level 10, where the computer decides everything and acts autonomously, ignoring the human.Simmler and Frischknecht [34] proposed five levels of technical autonomy, considering transparency (i.e., whether the system is transparent) and openness (i.e., whether the output of a system is deterministic or changeable) as key dimensions to determine technical autonomy.According to their categorization, a system with the lowest technical autonomy is fully transparent and its output is predetermined.A system with higher levels of autonomy can be less deterministic and transparent in algorithmic processing, more adaptive in learning from experience, and scalable to other contexts.
Although the benefits of AI-driven insights are well recognized in a healthcare context, there are concerns regarding the risks of delegating autonomy to AI in healthcare decision making.Reasons for this are potential technical limitations in algorithmic decision making [26] as well as the risks of AI challenging clinicians' autonomy and authority, for instance, when predictions lead to suggestions that are in conflict with clinician's decisions [40,43].For these reasons, increased attention has been paid to support clinician-AI collaborations in clinical decision making in order to augment clinicians' decision making rather than substituting their roles [18,29,30,41].
In this paper, we aim to expand this discourse on the right level of AI autonomy in healthcare decision making by including the perspective and preferences of patients.For this investigation, we adopted Mackeprang et al. [25]'s approach to define the levels of technical autonomy, as it enables a clearer focus on the interplays between humans and AI resulting from the relational autonomy that is given to AI [34].Based on the lens of Mackeprang et al. [25], we defined four levels of AI autonomy for this study: • Level 0 (L0): No AI assistance.Patients do not make use of AI in decision making.
We acknowledge that these levels can be determined with more granularity.However, since the potential forms of interactions between patients and decision-support AI are not yet fully explored and developed for our study context, we start with this basic classification and explore if different levels of AI autonomy emerge in our discussions with patients.

The Roles of HCP in AI-Assisted Shared Decision Making
Previous research on clinician-facing AI-powered DSTs highlights the roles of HCPs in exercising caution and taking accountability in using AI in clinical decision making.For example, Mainali et al. emphasized the importance of awareness among clinicians regarding the intended use and limitations of any ML algorithms to avoid inaccurate data interpretation [26].While limited, there are a few recent works that show patients' expectations on the role of HCPs in leveraging AI in healthcare practices.For instance, the result of a recent interview study with patients in varying healthcare contexts (e.g., primary care, cardiac care, or other therapeutic care) highlights three types of skills that patients believe are essential for HCPs in an AI-enabled future: cultivating patients' trust in AI embedded in healthcare, fostering patient engagement in shared decision making, and establishing data governance and validation of AI technologies [20].Also, patients' perceptions on the scenarios of using clinical AI applications with and without physician interactions (namely, scenarios of 'AI as augmenting technology' and 'AI as substituting technology' respectively) were examined in an experimental study focusing on both acute and chronic healthcare contexts [17].
The study revealed that patients have great concerns regarding accountability and transparency of regulatory standards in both scenarios including the situation where physicians are present.This result implies the necessity to further investigate desirable boundaries between the roles of AI and HCPs in shared decision making in which patients can feel safe and have trust in the AI-powered decision-making processes.

METHOD: AN INTERACTIVE NARRATIVE & SPECULATIVE AI PROTOTYPES
To investigate patients' preferences for AI autonomy in healthcare decision making, we developed prototypes of three speculative AI interfaces, namely, the DT Calculator, the DT Virtual Advisor, and the DT Virtual Doctor.These AI prototypes were designed to demonstrate three levels of AI autonomy (from L1 to L3 respectively) as defined in 2.2.The AI prototypes were carefully situated into a fictional story to make the use of the AI-driven tools believable and suspend any disbeliefs our participants might have about the technical feasibility or performance.In this study, we wanted the participants to vividly imagine specific decision-making moments and to reflect on their preferences regarding AI involvement in these moments.With these goals in mind, we designed a fiction probe-an interactive narrative combined with the speculative AI prototypes-using Twine1 , an authoring tool for web-based interactive fictions.We structured the story in three parts (pre-story, main story, and exit story) with in-story choices to probe participants' preferences.The story was narrated from a second-person perspective (e.g., "you and your partner have dreamed of having a baby") to directly address the participant and receive responses from their own perspective.After going through the interactive narrative, the participants were interviewed to discuss their choices and responses to the AI prototypes.In what follows, we elaborate on how we designed the interactive narrative and staged the prototypes of AI interfaces with in-story choices for this study.

Pre-Story
The interactive narrative first sets the context for a near-future scenario of pregnancy care and probes participants' general attitudes towards interactions with HCPs and the use of healthcare technologies.The story draws participants into the situation of a woman trying to have a baby for 6 months without being successful.To elicit participants' general attitudes towards interactions with HCPs, we asked if they want to visit a preconception care center to consult a midwife's advice or prefer to wait for a while and not see a midwife yet.After this, the story jumps to the moment where the woman finds out that she is pregnant.In the story, she calls a midwifery practice to arrange a first appointment.A midwife explains the DT technology and recommends considering data collection with a DT during the 5-week waiting period to get predictive insights of AI regarding potential complications during pregnancy.The participant is asked to choose whether she wants to start the data collection now, later, or never.We used this question to measure participants' general openness to data-driven health technologies.

Main Story
The main story engages participants in two decision-making situations related to the care of pre-eclampsia (Figure 1): (i) a decision on nutritional supplement intervention to lower the risk of pre-eclampsia (Decision 1), and (ii) a decision on the timing of pregnancy termination (i.e., delivering the baby prematurely) to prevent further progress of pre-eclampsia (Decision 2).We addressed the two decision-making situations at the relevant weeks of pregnancy: Decision 1 at the 20th week of pregnancy and Decision 2 at the 30th week of pregnancy.We chose the weeks of the events in close consultation with gynecologists to align the story with reality (e.g., the needs for intervention for pre-eclampsia arise around and after the 20 th week of pregnancy) and to make the decision-making challenging (e.g., the onset of pre-eclampsia at the 37 th week of pregnancy is less challenging than in the 30 th week).

Decision 1.
The first decision on nutritional supplement intervention becomes necessary when there is a concerning increase in blood pressure of a pregnant woman.In practice, the woman will usually receive a prescription for nutritional supplements, such as Aspirin or calcium supplements, to lower the blood pressure [9].This seemingly simple and low-risk decision on supplement intervention can create a dilemma because some women might prefer reducing the consumption of any artificial supplement during pregnancy.Additionally, the general clinical guidelines may not fit the individual nutritional requirements of every woman.
We envisioned that DT-based AI can be helpful in this decision making in addition to the advice of HCP; based on the woman's personal DT monitoring data, the AI can provide personalized predictions on the woman's reduced risk of pre-eclampsia depending on the amount of calcium intake.In the story, we narrated this possibility in the situation where the increased blood pressure of the woman is detected by the monitoring of her digital twin, and her midwife advises the woman to take a 1000mg calcium tablet based on a clinical standard.The woman is given an option to use the AI prototypes to adjust or decide on the amount of her calcium intake.

Decision 2.
The second decision on the timing of pregnancy termination is required when a pregnant woman develops preeclampsia.In practice, when pre-eclampsia occurs, the woman will be hospitalized immediately and monitored constantly.The delivery will be induced as soon as the pre-eclampsia reaches a severe level.Although terminating pregnancy is the only way to cure severe preeclampsia, deciding on the right timing for pregnancy termination is a tough decision [13,36].On the one hand, premature babies do not always survive and typically experience various kinds of severe complications.On the other hand, the health of a pregnant woman can be severely affected (e.g., life-long damage to organs) if the pregnancy is terminated too late.In addition, the experience of the woman can be overwhelming, because the pre-eclampsia can progress rapidly and then the decision on pregnancy termination should be made in no time.Facilitating shared decision making in such a time-pressing situation can be difficult for HCPs.
We envisioned that the DT-based AI predictions can be helpful for the woman in exploring the decision options much earlier through AI's prognosis of health risks depending on the timing of delivery.In the story, we narrated this possibility in the situation where the woman experiences an increased risk of pre-eclampsia.The woman is referred to a gynecologist and advised to be monitored through the digital twin and wait at home until any symptoms of pre-eclampsia arise.In the meantime, the woman is given an option to use the AI prototypes to explore the options to respond to pre-eclampsia.Notably, the risk in this second decision is lifethreatening and much higher than the first one.We included these low-risk and high-risk situations to see how participants' perceptions and preferences for AI may differ in the two decisions.

3.2.3
The Speculative AI Prototypes.The three speculative AI prototypes, the DT Calculator, the DT Virtual Advisor, and the DT Virtual Doctor, were developed for the two decisions in the forms of interfaces for patients (Figure 2).
The DT Calculator, based on level-1 AI autonomy, is designed to exert minimal autonomy.It gives a woman higher autonomy in exploring decision options by enabling her to formulate requests for specific AI output.For instance, the woman can formulate potential calcium intake plans (Decision 1) or set a potential delivery date (Decision 2) herself and check the effects of the chosen option on the reduction of health risks (These effects are informed by the developed algorithms in our project).With this level of autonomy, the DT Calculator represents the potential role of AI as an instrumental tool in health decision making.
The DT Virtual Advisor, based on level-2 AI autonomy, is designed to suggest a range of care options proactively, while respecting the woman's autonomy for the final decision.For example, the Virtual Advisor recommends three effective calcium intake plans (Decision 1) or potential delivery dates (Decision 2), and the woman can chat with the Virtual Advisor to explore the implications of other care options and make her final decision.With this degree of autonomy, the DT Virtual Advisor represents the potential role of AI as a co-decision maker in health decision making.
The DT Virtual Doctor, based on level-3 AI autonomy, is designed to perform autonomous decision making with minimal human involvement.For instance, the Virtual Doctor makes the decision for the woman by automatically giving the exact dose of calcium that she needs based on the real-time monitoring of a calcium sensor on her body (Decision 1) or by setting the delivery date with minimum risks for the health of the baby and the woman.With this degree of autonomy, the DT Virtual Doctor represents the potential role of AI as an authority in health decision making, like authoritative doctors.This authoritative role of AI is intentionally designed to be provocative to explore participants' perceptions and preferences on this high-level AI autonomy.To prevent immediate rejection of the design, we included minimal but essential human involvement in the autonomous decision-making processes of the DT Virtual Doctor (e.g., approval of initial activation, possibility to stop or overwrite algorithmic decision before executing).
The detailed descriptions of the AI prototypes are summarized in Table 1.We named the concepts for level-2 and level-3 AI autonomy

DT Virtual Advisor
The DT Virtual Advisor recommends calcium intake plans that are effective for you, but you decide which calcium intake plan you will follow.
The DT Virtual Advisor suggests an optimal delivery date based on its recommended level of acceptable risks for you and your baby.You can negotiate with the DT on alternative options and make the final decision yourself.
Level 3: AI as an authority

DT Virtual Doctor
The DT Virtual Doctor makes the decision for you by automatically giving you the exact dose of calcium that you need based on the real-time monitoring of a calcium sensor on your body.You no longer have to worry about your calcium intake and risk of hypertension.
The DT Virtual Doctor decides for you on your delivery date.The DT Virtual Doctor predicts the moment that you will develop pre-eclampsia and define the optimal delivery date by minimizing the health risks for you and your baby.You will receive a notification as soon as the optimal delivery date is defined with 95% certainty.At that moment, the DT Virtual Doctor will also communicate the delivery date to your gynecologist to prepare for the follow-up procedures.
as the DT 'virtual' advisor and 'virtual' doctor to clarify that they do not represent real humans.For readability, however, we will use the shorten forms to refer to these concepts in the rest of the paper where necessary (i.e., the DT Advisor and the DT Doctor).

3.2.4
In-Story Choices.For each decision-making situation, we first asked if the participant likes to consult DT to make her decision or follow the given advice of the HCP in the story (midwife/gynecologist) (Figure 3; left).We did this to collect participants' general preferences between AI and HCP before they see the specific AI prototypes we developed.Next, the story introduces the AI prototypes in the form of demo videos (Figure 2) to show the interaction possibilities.The descriptions for each design were also provided.We decided to use a video format instead of fully interactive prototypes to ensure that all participants would experience the same set of interactions.In the next step, the participant is asked to choose the preferred way of using the AI prototypes in the story situation (Figure 3; right), considering the level of AI autonomy (i.e., DT Calculator, DT Virtual Advisor, DT Virtual Doctor, or no use of DT) and how they want to make the decision (i.e., with or without HCP).The summary of the options and descriptions that were presented to the participants are included in Appendix.

Exit Story
Upon completion of the main story, the story jumps to three years after the baby is born and the women is considering to have a second child.With this exit story, we wanted to check if the participant's general willingness to adopt the decision-support AI is consistent with their previous choices.Therefore, we ask if they would like to use the presented AI prototypes for their next pregnancy.Participants responded on a 5-point scale ranging from 'absolutely' to 'absolutely not'.

Story Experience Design
Participants were able to follow each passage of the web-based interactive narrative.We used voice narrations and graphical illustrations as supportive media for storytelling to better engage the participants in the story.We designed the interactive narrative to proceed to a positive result no matter which choice is made by the participants (e.g., "You made a good choice.Your pregnancy is progressing with low risk of getting hypertension.").This was to build a positive relationship between the participant and the story and to prevent participants from withdrawing the explorations of speculative AI prototypes due to negative experiences in the preceding story.When participants chose not to use AI in the first in-story choice, we acknowledged their choice first and then asked them to still explore the AI prototypes to continue with the study.Then, they were redirected to the same passages that those who chose to use AI went through.We wanted to know people's preferences in the case that they would actually start to use AI support, and this is mainly plausible when trust is initially high.We therefore tried to create a maximal trust baseline through the optimistic scenarios.

User Study
We used the fiction probe in semi-structured interviews as well as an online survey to collect responses of women who had a recent experience of pregnancy.The full study was approved by the Ethical Review board of the university where this study took place.
3.5.1 Semi-Structured Interviews.We conducted semi-structured interviews with 12 women (P1-P12) (30-39 years old) who recently experienced a pregnancy in The Netherlands (11 participants gave birth 6-22 months ago, 1 participant was in the 17 th week of her pregnancy).We considered the place and recency of the pregnancy experience in our recruitment to make sure that participants could understand the presented story based on their experience with the Dutch healthcare system and relate to the story by reflecting on their recent pregnancy experience.There were five non-Dutch participants (1 European, 4 Asians).Three participants experienced complications (e.g., gestational diabetes) during their own pregnancy.They were not patients themselves at the time of study participation, but we interviewed them as potential users of healthcare AI.To prevent potential discomfort from the story regarding complications, we took extra care to communicate this upfront and resolve any related questions and concerns of the participants before proceeding with the interview.The interview started with general questions regarding participants' pregnancy experience (e.g., pregnancy history, experience of complications during pregnancy) to set out a general understanding of each participant.Then, the participants went through the interactive narrative at their own pace.To minimize the influence of the interviewer on the participants' choices in the story, we provided participants personal space to finish the story alone and did not check the participants' responses on the fly.Also, we instructed participants to respond to the story in the way they wanted (e.g., there is no right or wrong answer) and provided only technical support when needed.Most of the participants finished the story in 10-15 minutes.After they finished the story, we asked a set of questions including (i) their general story experience (e.g., 'Was the story easy to follow?', 'Were the events in the story easy to imagine?', etc.) to check how much they were able to situate themselves in the fictional story, (ii) the reasons behind their choices to gain deeper understandings of underlying expectations, concerns, and values, and (iii) their wishes regarding AI-assisted decision making in the healthcare context to collectively envision its 'preferable' futures [12].Before concluding the interview, we asked the participants if they wanted to change their choices to see how their perception might have changed throughout the discussion during the interview.If they said yes, we asked their changed choice and what reasons they had for this.The individual interviews (7 face-to-face, 5 online) lasted between 40-60 minutes including going through the story.The participants were rewarded with 10 euros for their participation.The interviews were audio-recorded and transcribed verbatim for analysis.
3.5.2Online Survey.Due to the limited availability of women who could participate in our interviews, we also collected responses from additional women via an online survey to complement the responses from the small number of interview participants.The survey participants experienced the same interactive narrative and video demos of the AI prototypes that were given to the interview participants.This same format allowed us to combine the in-story responses made by participant in the interviews and survey, legitimately increasing the number of total responses.The survey participants were recruited by our online invitation within the communities of local universities as well as in local social media groups of parents.We added an additional set of exit questions in the survey to gather the same background information we collected from the interview participants (i.e., demographic background (e.g., age, nationality, etc.) and previous pregnancy experience).We also collected how confident survey participants were with their choices in the story and only included the responses with a high confidence level.The online survey was fully anonymous and voluntary.In total, 17 women responded, and two responses were excluded due to a low reported confidence level.The included 15 survey participants (S1-S15) were between 28 and 40 years old.Five respondents were non-Dutch (2 Asians, 3 Europeans).Seven participants experienced complications during pregnancy including pre-eclampsia (n=3).

Data Analysis
We first analyzed interview transcripts, using thematic analysis [8], to find any emergent patterns regarding participants' preferences for the level of AI autonomy in healthcare decision making.For those initial patterns, we further investigated the interview transcripts to understand the underlying reasons behind these preferences.We developed open codes for our participants' explanations on why they chose (not) to use the AI prototypes, the reasons they gave for choosing a certain level of AI autonomy, and the reasons for the chosen ways of using AI (e.g., with/without HCP).In addition, we open coded participants' concerns and wishes regarding using AI for their healthcare decision making.We tried to find the connections among these initial codes and developed themes for discussion.Although our focus was mainly on qualitative findings, we analyzed the quantitative data (i.e., the in-story choices made by all participants from the interviews and survey and combined for analysis) to investigate any statistical evidence to strengthen our qualitative findings.For instance, we explored statistical differences in participants' preferences across two decision-making situations (e.g., the most preferred AI prototypes in D1 vs D2), general trends in preferences for the three AI prototypes, and potential influence of personal factors (e.g., the history of complications) on preferences for AI autonomy.
In the following sections, we report the emergent patterns found from the qualitative analysis and support them with quantitative results if available.Then, we discuss the implications of our findings in designing AI-assisted decision-support tools from the perspective of patient-centered care.

FINDINGS: PATIENTS' PREFERRED WAYS OF USING AI IN HEALTHCARE DECISION MAKING
The analysis of participants' choices and interview transcripts revealed not only their preferences for the three types of AI presented in the story, but also a broader understanding of why they want to use different levels of AI autonomy.Also, we were able to understand participants' expectations of the roles of HCPs in AI-assisted healthcare decision making.In this section, we present four emergent patterns in the choices of participants that lead to discussions on how we may design AI-powered applications for patient-centered healthcare decision making.

HCP Involvement is Desired in All Levels of AI Autonomy
The most salient pattern that emerged from the interviews is the desire to involve an HCP when choosing to use AI in health decision making.This pattern was well reflected in the choices of all participants in the story (e.g., None of the participants chose the option to use AI without HCP involvement in D2, and only 6 out of 24 participants chose to use AI without HCP involvement in D1).Most of the interview participants often mentioned that they would not blindly follow a DT's predictions and suggestions without consulting an HCP to better understand the meanings and validity of these predictions.We found that this pattern is closely related to their perception of AI's knowledge.For instance, P2 said, "She (the midwife) may see stuff which the virtual advisor or virtual doctor may not see."Like P2, participants tended to consider AI "less knowledgeable" (P2, P3, P5, P8) than HCPs in understanding individual differences among patients.Related to this, the participants frequently emphasized how pregnancy can differ per person, recalling their own pregnancy experiences.For instance, some participants shared stories where they found the recommended clinical guidelines (e.g., nutritional supplements) not effective for them, or where their delivery went unexpectedly difficult or was delayed without any clear reasons (P3, P5, P9).They perceived that AI lacks an understanding of such personal differences and dynamics in pregnancy as its predictions are mainly based on the objective observations of one's health data (P2, P5), and it works based on "standard cutoff points" (P6) that determine the best treatment option for "average women." (P6) In contrast, HCPs were perceived more "knowledgeable, " "experienced, " and "accountable" (P3, P6) in this regard.This was because participants used to share subjective symptoms and experiences (e.g., pains, stress, etc.) with their midwife or gynecologist during regular meetings.They perceived that such nuanced information cannot easily be measured and "objectified" (P3) by AI.The personal interactions with HCPs seemed to create a solid foundation for our participants' trust and belief in their HCPs knowledge of their individual needs and corresponding ability to make the best decision for them.In addition to that, participants highlighted the communication ability of HCPs as another reason for considering HCP involvement essential: "With the virtual advisor, it is like you communicate with Google.You need to know which words, which keywords he understands.If you do not give good keywords, you may be directed to a completely different direction than where you want to go.With the midwife, you can catch up quicker if there is any misunderstanding" (P2).Due to the perceived knowledge and capacity gaps between AI and HCPs, participants prioritized the autonomy of AI in decision making lower and preferred to involve HCPs for additional confirmation and discussions regarding AI's predictions.
Nevertheless, our interviews revealed that patients' concerns about the trustworthiness of AI may not always mean that patients want to reject the use of AI entirely.Instead, many of the participants who expressed concerns about the trustworthiness of AI still wanted to include AI in their discussions with HCPs because they expected AI to bring unique benefits, such as a precision possibility for individuals (P2, P5, P6, P7) and completeness in monitoring health data (P4, P5, P10).Therefore, participants wanted to complement the potential limitations of AI with the insights of HCPs and vice versa.
The ways participants envisioned how the knowledge of AI and HCP can be complemented showed their expectations on different levels of HCP involvement.For high-risk decision-making like in Decision 2, participants wanted their HCPs to play an active role by explaining the meanings of AI predictions and confirming their validity.Although the insights from AI were considered less reliable than the knowledge of HCPs, participants wanted their HCPs to still take the advice of AI seriously and be open to discussing potential discrepancies between the advice of AI and their own advice: "I would definitely like the midwife to explain why she is thinking differently than the app, and I imagine I'll get a good answer."(P1) For low-risk decision making, like in Decision 1, some participants thought that minimal involvement of HCPs will work as well, for example, through their quick confirmations or approval before taking the advice of the DT.Taking a step further, P8 preferred to use and adopt the results of the DT Calculator by herself and only wanted to inform her midwife afterward so that her midwife would not miss important information that might become relevant later.
To summarize, our findings show a general preference among participants for HCP involvement when using AI in healthcare decision making.However, the expectations regarding the level of HCP involvement were more nuanced than a binary choice of all or none.

Preferences for AI Autonomy Vary per Perceived Risk in Decision Making
From the analysis of the choices of all participants, we found a difference in participants' preferences for the level of AI autonomy in Decision 1 and in Decision 2. In Decision 1, both the DT Calculator (L1) and DT Advisor (L2) were most preferred (11 and 9 out of the 27 responses) (see Table 2).In Decision 2, the DT Advisor (L2) was chosen much more frequently (n=16) than the DT Calculator (L1) (n=4).We conducted a chi-square test of independence to examine the statistical relation between the risk perception (low vs. high) and the choice of AI autonomy (DT Calculator vs. DT Advisor).The result showed that their relation was significant,  2 (1, N = 20) = 5.23, p = .0022,indicating that patients are more likely to choose the DT Advisor (L2) than the DT Calculator (L1) in high-risk situations.Regarding this difference, the interview participants indeed noted that their choices were based on the perceived risk in Decision 2 which was substantially higher than Decision 1.In Decision 1, participants perceived a lower risk in decision making because they knew that their decision on nutrition, even if it went wrong, would not have any life-threatening consequences.In Decision 2, they considered the stake substantially higher because they do not possess clinical knowledge to make such a "big decision" (P6, P11), and the decision poses significant risks for the health of both woman and baby.These risk perceptions may relate to more than half of the participants both in the interviews and the survey (i.e., 9 out of the 12 interview participants and 9 out of the 15 survey respondents) choosing for different levels of AI autonomy for the two types of decisions.
Among participants who chose different levels of AI autonomy in D1 and D2, we found an interesting pattern that divides them into two groups.On the one hand, there was a group of participants (n=10; 6 interviewees and 4 survey respondents) who wanted higher levels of AI autonomy in high-risk decision making (Decision 2).For instance, most of them (n=8) chose the DT Calculator (L1) in Decision 1 and the DT Advisor (L2) in Decision 2. The common reason for favoring the DT Calculator (L1) in Decision 1 was that it allows patients the freedom to individually explore other care options.It gave them a sense of control, transparency, and agency in their decision making: "With the calculator, you can check everything.[. ..]It's more YOU have everything in hand, and you know where it comes from.It's not just like [you get] an answer and you follow that."(P2) However, the sense of control that was preferred in Decision 1 was perceived as a risk factor in Decision 2. P2, for instance, explained that "If there is really a problem, the calculator has the risks that you miss out one option, which is the good one.[The DT Calculator has] too many possibilities at the expense of risks or unsureness to miss THE possibility." In similar vein, these participants mentioned that they preferred the DT Advisor (L2) in Decision 2 because it provides more suggestive information by telling them "what the better option is based on your data" (P1) and showing what "a golden standard" (P5) for the acceptable range of risk is in a given situation.They thought that the information from the DT Advisor would be helpful in making such a difficult decision.For a similar reason, two other participants also preferred higher levels of AI autonomy in Decision 2 than in Decision 1 by changing their choices of AI autonomy from none (L0) to the DT Advisor (L1) (P8) and from the DT Advisor (L2) to the DT Doctor (L3) (S13).
On the other hand, there was the other group of participants (n=8; 3 interviewees and 5 survey respondents) who preferred the lower levels of AI autonomy in high-risk decision making.For instance, two participants (P6, S14) who preferred the DT Advisor (L2) in Decision 1 chose the DT Calculator (L1) in Decision 2 because the lower level of AI autonomy of the DT Calculator gives them an opportunity to explore the consequences and make their own decision before asking for the opinions of others (e.g., DT or HCPs).In addition, three participants (P4, P10, S12) who chose the DT Doctor (L3) in Decision 1 changed their choices to the DT Advisor (L2) in Decision 2. Regarding this, P4 explained that she would like to have more room for discussion in high-risk decision making by asking the DT Advisor (L2) about other possibilities instead of just relying on the autonomous decision making by the DT Doctor (L3) that she preferred in Decision 1. P10 also mentioned that the deterministic, "to-the-point" approach of the DT Doctor would be less preferred in Decision 2. In addition, there were a few participants (S5, S9, S15) who completely rejected the use of AI (L0) in Decision 2 and preferred to discuss only with HCPs in a traditional in-person visit.Unfortunately, we did not have a chance to collect further explanations for this choice as they were the anonymous respondents to the survey.Still, it suggests the need for understanding if and why AI would be less valued in high-risk decision making and what it would imply in designing AI-supported healthcare decision making.
To further investigate potential factors contributing to the differences in these two groups, we examined the potential statistical relationships between several personal variables and their choice in different risk situations (higher AI autonomy vs. lower AI autonomy in the high-risk situation).The personal variables that we examined include participants' previous experience of complications during pregnancy and number of pregnancies they experienced.Although there were no statistical relationships to explain what might have resulted in the differences between the two groups, the personal factors elaborated in 4.3 might be one of the reasons for the difference in preferences.
Overall, it seems that participants wanted to balance the distribution of decision power among different decision-making stakeholders (i.e., AI, HCPs, and themselves as patients), either by giving more agency to the other actors (AI, HCPs) or having more control themselves.By doing so, participants seemed to reduce the risks and uncertainties to their acceptable levels in the given decision-making situations.
Table 2: Frequency of the levels of AI autonomy chosen by the participants per different risk perceptions (Left) and health history (Right).Numbers in the parentheses mean the frequency of the choices in Decision 1 and Decision 2.

Risk Perception
Experience of Complications Low

Preferences for AI Autonomy Vary per Person
As partly addressed in the previous section, there were individual differences in the preferred levels of AI autonomy among our participants.We found three personal factors that seemed to contribute to these individual differences.Firstly, we found that participants who had a health history tended to rely more on AI in general.For instance, all the 11 participants who experienced complications preferred to use AI in both decision-making situations, whereas 6 out of 16 participants who had a healthy pregnancy did not choose to make use of AI at all (L0) in either one of the two situations or both.In addition, two participants that experienced complications chose to use the autonomous decision making of the DT Virtual Doctor (L3) in highrisk decision making (Decision 2), showing their high-level reliance on decision-support AI (Table 2).
Secondly, we noticed that personal attitudes towards healthcare decision making might have affected participants' preferences for AI autonomy.We found this pattern in the four interviews where participants (P3, P5, P8, P11) explicitly mentioned that they like to have a high level of control in healthcare decision making.For instance, they all mentioned how actively they had been participating in the meetings with their HCPs by searching for reliable source of information on the Internet (e.g., the websites of hospitals, public health organizations, or scientific literature).They were keen on doing their own 'research' and being informed sufficiently regarding their situations and the advice of HCPs.Reflecting their preferences to control, none of them chose to delegate their decision to the DT Virtual Doctor (L3) in the two types of decisions.P3 especially showed strong repulsion to the idea of AI taking over important health decision making: "I think it is really something that should not happen.How can an app know when it is best to deliver my kid just based on aggregated data of myself?"In contrast, P4 expressed her strong preference to the autonomous decision making by the DT Virtual Doctor (although only in Decision 1; the low-risk situation), explaining how her previous experience of gestational diabetes had made her become less proactive in healthcare decision making: "If I'm imagining myself in this situation, it would be really stressful.I believe, in this situation, you would be referred to the gynecologist, and then you will have so many appointments and so many people monitoring you.The less you can do, the better." She said she would like to rely on the DT Virtual Doctor as it can take her mind off a problem that would otherwise constantly occupy her cognitive and emotional resources for decision making.
Relating to personal attitudes, lastly, participants had different preferences for risk communication.For instance, while there were participants who liked to be well informed of every possible health risk during pregnancy to feel reassured (P3, P5), there were also a few participants who would like to be informed of only a critical risk.Due to the worries about potential negative impact of a DT on her psychological wellbeing, P9 wanted to involve a DT only for a limited period of the pregnancy: "I get stressed out sometimes about the things giving me too much information.Sometimes I'd like to be ignorant and not know everything.I don't know if it would help me or if it would make me more nervous about the pregnancy.I'm not sure right now what it [the DT] would do to me, so maybe I would try to use it, but only in the last stage of pregnancy." While patients' preferences for AI involvement in healthcare decision making need to be respected, these personal factors reflect the potential vulnerability of patients in decision making which could lead to the over-reliance on AI unless designed with caution.In the discussion section, we discuss further how this issue might be addressed in designing AI-powered DSTs for patients.

Preferences for AI Autonomy May Change
During the discussions with the interview participants, we also observed that their preferences for AI autonomy could change.For instance, we found that participants who were reluctant to accept the involvement of AI in general became more open to accept it after seeing the exact possibilities of AI through the prototypes and the flexibilities they could have in taking the advice of AI.Based on these reasons, two participants (P3, P11) who initially rejected the use of a DT embraced the benefits of utilizing the DT's predictions in the end and wanted to use the DT more actively in their discussion with an HCP.
In addition, there were several participants who explicitly highlighted that their preferences for AI autonomy were tentative and thus, could change.One of the major reasons for this was that their perception on the reliability of AI can change, for example, based on the results of their chosen DT for certain decision making: "Maybe if something negative would happen, or if it would give me an advice that is wrong, I would throw it off immediately.That's my hesitation (to decide now whether to use the DT for my future pregnancy).If everything went well, and all the advice made sense and were correct, I would definitely use it again.But if there was a piece of advice [from the DT] that didn't work out, I wouldn't trust it anymore." (P8) P7 also wanted to reassess the benefits of the DT in her next pregnancy and would like to decide whether to use it again or not: "In the story, it gave definitely a positive experience, so I think my answer will be These examples show the potential dynamics in patients' preferences for AI autonomy when considering the repeated use of decision-support AI for a long time.This finding shed light on the need for regular reassessments of patients' preferences towards the level of AI autonomy not only to make the assistance of AI adaptive to patients' preferences but also to recover patients' trust in AI if necessary.

Potential Modes of Patient-HCP-AI Interaction for Shared Decision Making
By synthesizing the patterns observed in our study, we identified four different modes of patient-HCP-AI interaction appreciated by our participants that might give a clue to how the roles and interactions among patient, HCP, and AI could be shaped for healthcare decision making.Each mode of interaction is presented in the order of increasing levels of AI autonomy in shared decision making (Figure 4).

Traditional patient-HCP mode.
The traditional patient-HCP interaction without the of AI (L0) was preferred when our participants did not find additional value in using AI in their decision making.In this mode, the patient consults with the HCP, and the HCP leverages their knowledge to provide sufficient information and support to help the patient make the best decision for their values and preferences.The patient's right to refuse the use of AI for their healthcare decision making is operationalized in action and by designs in AI-powered DSTs that support the patient to switch the interaction to the traditional patient-HCP mode.
4.5.2Patient-led mode.In the patient-led mode, the patient as a decision maker actively utilizes decision-support AI and makes a final decision themselves.The AI plays the role as an instrumental tool (like the DT Calculator) and exerts a low-level autonomy (L1) in decision making by generating output upon the request of the patient.The involvement of the HCP can be minimal, for example, by providing additional confirmation on the patient's decision upon request.In our study, the patient-led mode was preferred in the low-risk situation (D1).Participants appreciated the patient-led mode because of the freedom of exploring alternative care options and a precision possibility for individuals.However, the patient-led mode of interaction was less preferred in the high-risk situation (D2) due to the fear of missing the best option at the expense of having a high degree of controllability.The patient-led mode of interaction might be suitable for low-risk lifestyle decision making where patients have a better understanding of their preferences (e.g., what works best for them) and can share the responsibility of care.

Collective mode.
In the collective mode, the patient consults both AI and HCP for advice and discusses the final decision together.
The AI plays the role as a co-decision maker (like the DT Virtual Advisor) both for the patient and the HCP.The interaction can be tailored to the patient and the HCP depending on their expertise and information needs.In our study, the collective decision making mode was well received in both low-risk and high-risk situations.
In the high-risk situation, participants appreciated this mode due to the inevitable uncertainty in decision making.As different capabilities of different actors (HCP, AI, patients themselves) can make unique contributions, participants liked to involve all actors to distribute decision power and leverage each of their capability to manage the uncertainty in decision making.Potential conflicts of opinions and values among three actors pose new design possibility.
4.5.4AI-led mode.In the AI-led mode, the patient delegates their decision to the AI.The AI plays the role as an authority (like the DT Virtual Doctor) and decides for the patient, demonstrating a high-level autonomy (L3).To make this mode of interaction safe and legitimate, the HCP's approval is essential.Based on the agreement with the patient, the HCP also delegates their decision to the AI and supervises autonomous decision making of AI where necessary.Although very limited, a few participants in our study chose the AI-led mode in Decision 1 to reduce decision fatigue and stress of self-care.The AI-led mode of interaction might benefit patients and HCPs in the decisions of which possible choices and results are well known, thus has less risk in delegating the decision-autonomy to AI.However, caution should be exercised to prevent any misuse and over-reliance on AI.We were able to find these modes of interactions as we had given our participants the option to choose the involvement of the three actors (i.e.patient, HCP, and AI) with varying levels of granularity (Appendix).This conceptualization of various relations that patients can have with decision-support AI and HCP made it possible to examine patients' perception of AI-powered healthcare decision making, including highly speculative roles of AI (e.g., the DT Virtual Doctor).Interestingly, there was no option that was never chosen by the participants.This may imply that every interaction mode has the possibility to be realized and valued depending on decision making situations, suggesting opportunities for further research.We note, however, that these four modes of patient-HCP-AI interaction are not an exhaustive list of relationships among the three actors.There can be even more granularity in the above-described modes.Also, since our focus was on patients' preferences, a potentially missing mode of interaction can be the one led by the HCP.Given the growing attention to patient-centered decision making in healthcare, it is important to investigate carefully how the HCP-led mode might benefit the core values of patient-centered decision making in healthcare.We expect this initial set of patient-HCP-AI interaction modes to initiate further investigations into the various relationships among the stakeholders in AI-powered shared decision making.

DISCUSSION
So far, we have presented our findings, illustrating participants' preferences for AI involvement in healthcare decision making.In this section, we first reflect on general contributions of our findings within the context of existing literature.Subsequently, we discuss the implications of our findings in incorporating patientcenteredness in the design of AI-powered healthcare decision making by highlighting potential ways to address patients' varying preferences for AI autonomy, potential areas for patient-HCP-AI collaborations, and the framing of decision-support AI as a shared technology between patients and HCP.

General Reflection
Our findings add to the current understanding of patients' perspectives of using AI in healthcare decision making in several ways.
First, our findings show patients' perception of the need for HCP involvement in AI-powered healthcare decision making because of perceived limitations in the knowledge of AI.This is not a surprising result given the concerns raised among both patients [17] and healthcare professionals [3,18,22] about the trustworthiness of AI in healthcare.Previous research argued that users' perceived risks of AI due to various concerns (e.g., reduced patient-HCI communication and interaction, trust issues, and concerns regarding privacy, transparency, and accountability [17]) hinder the general adoption of AI applications in healthcare.Beyond this discourse, our findings revealed that users' adoption of AI can be more nuanced rather than binary depending on their level of trust in AI.Our participants showcased a perspective in which the knowledge of AI and the insights of HCPs both entail value and risks.Thus, by leveraging their respective advantages, it is expected to complement their weaknesses and reduce risk as a whole.
Second, our findings highlight patients' preferences for AI involvement in low-risk and high-risk decision-making situations.The general preference of participants for the DT Virtual Advisor (L2) over the DT Calculator (L1) in the high-risk situation resonates with the result of a recent survey experiment which revealed that AI decisions were perceived as less risky and more useful than human experts in high-impact scenarios [2].In their experiment, AI decisions were perceived less risky than human experts in high-impact scenarios in three different application domains (e.g., AI applications for decision making on Media, Health, and Justice), whereas there were no differences observed between AI and human experts in low-impact scenarios.Also, AI decisions in high-impact scenarios were perceived as more useful than human experts (e.g., AI decisions on medical treatments than on fitness recommendations).Given the high-level of uncertainty and the impact of adverse consequences, it seems reasonable to attribute greater value to leveraging AI insights in high-risk decision making.The distinctive preferences for AI autonomy in the high-risk decision making (e.g., either higher or lower AI autonomy) found in our study highlights interesting differences between two groups of patients that have not yet been readily addressed in existing literature.The characteristics of the two groups could be further investigated to explore possibilities for tailoring interactions with design-support AI based on these differences.
Third, we found varied preferences for AI autonomy depending on several personal factors, such as individual attitudes toward healthcare decision making, personal health history, and preferences for health risk communications.This finding adds to the theoretical model of human trust in autonomous systems [23], underscoring these three personal factors as contributing elements influencing individuals' inclination to trust autonomous systems, particularly within the healthcare context.
Lastly, our findings indicate that patients' preferences for AI involvement in healthcare decision making could change over time based on their repeated interactions with AI and changing clinical situations.This finding emphasizes the need to consider patients' preferences for AI involvement as a dynamic construct that provisionally determines the appropriate level of AI involvement in a given context rather than a static element that is pre-determined and stable.This also implies that applying one-off preference settings in AI-powered shared decision making may not suffice.

Addressing Patients' Varied Preferences for AI Autonomy
5.2.1 Enabling patients to communicate preferences for AI autonomy through use.Our findings showed that patients' preferences for AI autonomy in healthcare decision making can vary per person, context, and change over time.These patterns are more than a matter of personal taste regarding AI.Instead, it can be understood as the expression of patients' trust in AI.These findings may imply that what is perceived as a preferable or appropriate level of AI assistance in healthcare decision making can be redefined constantly.This reflection leads to fundamental design questions, such as:'Who can define the right level of AI assistance in healthcare decision making?' and 'How?'.In the increasing examples of AIpowered DSTs for clinicians, the levels and forms of AI assistance are usually defined during 'the design time' [21] often informed by existing research on clinicians' perceptions and experiences of AI.This includes the insights usually from clinicians gathered through user studies and participatory design processes [18,29,41].Once the design decisions on the levels and forms AI assistance are made (by designers based on user input) and implemented in the designs, their appropriateness is rarely reexamined during 'the use time' [21].However, our findings suggest that preferable or appropriate levels of AI assistance would need to be reassessed considering various personal and contextual factors, and dynamic aspects of patients' preferences towards AI autonomy.In such a process, we believe that patients can be involved more actively to communicate their preferences during the use time.
'Variable autonomy' [27] may be a relevant concept to realize such patient-AI interactions.In [27], Methnani et al. proposed to develop intelligent systems with variable autonomy-dynamically adjustable levels of autonomy-as a means to operationalize and ensure meaningful human control.They argue that meaningful human control over a system is not achieved by simple human presence in the loop of autonomous processes (e.g., human authorization).Instead, it requires active user-system interactions that are collaborative and transparent.They argue that a system with variable autonomy can support such interaction for meaningful human control and embody critical ethical guidelines for AI such as accountability, responsibility, and transparency.Although they did not discuss the notion of variable autonomy in healthcare contexts, we believe that it is a valuable and inspiring perspective to take into account in designing AI-powered healthcare decision making.For instance, AI-based DSTs can engage patients with the interfaces that allow them to communicate their varying and changing preferences and adjust the levels of assistance accordingly.The variable autonomy of AI, however, will need to be discussed and executed based on mutual agreement among patients and their care providers, especially in high-risk decision making scenarios, to make sure that addressing patients' preferences does not lead to poor decision making, resulting in adverse consequences to patients' health.If done properly, the AI-powered healthcare decision-making processes could be designed more trustworthy and safer for patients from a broader perspective.The ways to realize these possibilities will need to be further researched.

Respecting patients' preferences for AI autonomy with caution.
Our findings reveal several factors which could put patients in vulnerable positions when choosing the extent to which they rely on AI for healthcare decisions.Factors including stress, personal health history, and low interests of participating in healthcare decision making can lead patients to over-rely on AI for the sake of other values (e.g., psychological well-being).Yet, patients should know that AI can never be error proof and thus, they share the responsibility for their own health and the decision on how they use AI's insights.AI-powered DSTs for patients should be designed in a way that can safeguard patients' vulnerability from over-reliance on AI.Previous research on clinical AI applications has emphasized the importance of training users (mostly healthcare providers) and providing explanations for algorithmic predictions to help users understand the limitations of AI models better [18,29].The increased transparency of AI is expected to help users balance their level of trust in AI and the actual capability of AI [44].Our findings suggest the potential of adding more granularity in this trust calibration based on the understanding of patients' psychological status, personal health history, and health beliefs.If patients have a high-level of vulnerability, more careful steps can be implemented in delegating their health decisions to AI.For example, the AI system can redirect patients to consult their decision with HCPs.Also, it can ask patients to revisit their choice of AI autonomy occasionally to confirm or evaluate their satisfaction in the ways they use AI for health decisions.By doing so, it will be able to respect patients' psychological challenges regarding their healthcare decision making in a way that can reduce the risks of misleading them to over-rely on AI.

5.2.3
Enhancing the roles of decision-support AI for constant preference reassessments and values alignment among stakeholders.Preferences are prone to change as clinical situations change.As we found in the study, there can be various factors that can change stakeholders' preferences to AI involvement.In this sense, their preferences tend to be provisional, conditional, and unstable.Epstein and Gramling [16] argued that "enacting treatment decisions that are based on provisional preferences requires doctors and patients to 'check in' periodically to reassess the effectiveness of the plan and whether that plan continues to reflect preferences as clinical situations change." We believe the same applies to the decisions on AI's involvement in healthcare.Choosing to use decision-support AI based on provisional preferences requires patients and HCPs to check in periodically to reassess the effectiveness of the chosen interaction mode and whether that interaction mode continues to reflect patients' preferences as their perceptions on decisionsupport AI and clinical situations change.However, as also noted by Epstein and Gramling, implementing such a reassessment in the context of growing clinical demands is challenging and will cost a lot of time and attention from HCPs, which could make the overall healthcare unsustainable.To address this challenge, preference reassessments can be prompted and facilitated by AI where necessary.Like this example, the potential roles of decision-support AI for constant preference reassessments and values alignment among stakeholders need to be actively sought out.

Potential Areas for Patient-HCP-AI Collaborations in AI-Powered Shared Healthcare Decision Making
Reflecting on our findings, we found two potential areas for patient-HCP-AI collaborations which could make AI-powered healthcare decision making more trustworthy and sustainable.

5.
3.1 Building a shared knowledge base.In this study, we found that our participants perceived AI as less knowledgeable than HCPs in general.It is interesting to discuss this user perception because AI usually works with more data samples than HCPs, and therefore, it is often expected to advance current personalized care by HCPs.The participants' perceptions on the capability of AI and HCPs were closely related to the perceived characteristics of data that AI and HCPs work with.Participants considered the data input for AI as big and thin which made them consider AI lacking personal relevance to individual patients.In contrast, the data input for HCPs were considered as small but thick which includes information about individual patients' subjective, lived experience as well as their personal characteristics (beliefs, attitudes) and domestic circumstances.Our participants experienced this process of information gathering by their midwife by sharing their symptoms, feelings, and other goodto-know information during their in-person visits.As participants had accumulated a rich knowledge base for their health decision making together with their midwife, it was preferable for them to trust and rely on HCPs more than AI.
This finding suggests an opportunity for patient-HCP-AI collaborations in developing a shared knowledge base for healthcare decision making.An AI-powered system can complement the patient-HCP collaboration by sharing data and recommendations.For instance, the expertise of AI in analyzing population data could be compared and augmented with HCP's insights on the uniqueness of the patient based on their clinical experience with various patients and so-called outliers.Also, if patients could contribute to the development of such a shared knowledge base by reporting their personal lifestyles and subjective experiences, it will be a great way to add more nuance to AI-or HCP-collected data.By making this process of data collection more transparent and collaborative, the decision support AI could be more meaningfully implemented and trusted within patient-HCP relationships.
5.3.2Health risk management.Our participants often illustrated how AI's monitoring of health data can reassure their safety when they go through changes in their body due to the unfamiliar health conditions, like pregnancy.This functional possibility of AI was appreciated by our participants because they do not have to "bother" their HCP every time they have small questions.reliable reassuring of AI was expected to benefit patients in this context and help both patient and HCP to know when to take action if a risk is present.This suggests another opportunity to support patient-HCI-AI collaborations in managing health risks.For instance, different roles of the three actors in the 1 st and the 2 nd line of care can be explored in this regard.In the first-line care, where a patient's health risk is relatively low or not yet present, the AI could play a role as a compassionate expert next to the patient by reassuring them based on the reliable health monitoring data and guiding their self-care.When alerting signs are present, the AI could inform both patient and HCP in a way that they can take necessary actions in a timely manner.In the second-line care, where the risk is higher, and the decision entails more uncertainty due to changing clinical conditions, the AI could engage the patient and HCPs more intensively to support their shared decision making.This way of patient-AI-HCP interactions could support patients' responsibility in taking part in managing their health risks and enhance patients health ownership.Also, it will help HCPs to invest their time and resources in supporting patients in more critical and risk-laden decision-making situations.This will reduce the overall burden in healthcare and thus, contribute to making healthcare more sustainable in the long term.

Framing Decision-Support AI as a Shared Technology
Previously, AI-powered DSTs have often been considered as a tool for experts (i.e., clinicians).Due to the clinician-oriented focus, patients have rarely been asked whether and how they want to leverage AI in their healthcare decision making, although they are the eventual beneficiaries of AI-powered healthcare decision making.In our study, we positioned patients and clinicians as equally important users of AI-powered DSTs.This perspective enabled us to frame AI-powered DSTs as a shared technology that can play a role between patients and HCPs.This notion makes the following conceptual contributions to current research on human-AI configurations within the specific context of healthcare.
Firstly, it provides more nuanced understandings of the involvement of HCP and AI, as illustrated in 4.1 and 4.5.Previously, the notions of 'AI as augmenting technology' and 'AI as substituting technology' have often been discussed for the HCP-AI interaction.This resulted in somewhat dichotomic views in current AI-powered DST research.Research often questioned: Does AI perform better than HCP? Do people prefer AI or HCP?As reflected in the four modes of interaction in 4.5, many of our participants acknowledged in the interviews that they appreciated being invited as an important actor to decide on whether and how to use AI technology for their healthcare, instead of not being aware of the potential use of decision-support AI by HCPs or being given rather passive options to decide on technology use (e.g., giving one-off consent).Participants' intentions to leverage the decision-support AI were generally very constructive (e.g., using AI to do more research before consulting the HCP, using AI insights to reduce the burden for the HCP while still being reassured by AI).Unlike the fears of HCPs that are often highlighted in literature (e.g., fear of being replaced by AI, fear of professional autonomy being challenged by AI) [40], the use scenarios envisioned by participants were inclining more toward potential positive changes in healthcare decision making.
Secondly, our approach redefines the humans in the human-inthe-loop decision making in health.The notions of human-in-theloop, human-on-the-loop, and human-out-of-the-loop have been discussed as different human-AI configurations in various application domains [2].In previous AI-powered DST research, it has always been clinicians (i.e. the domain experts) who were considered as human who can be 'in' or 'on' the loop of decision-making processes, correcting and improving the AI models.Considerations of multiple human actors in AI-powered healthcare DSTs have recently emerged in HCI research.For example, the work of [43] examined the interplays between different types of clinicians.However, they still address the sub-groups of the same actor (i.e.clinicians), and the role of patients is framed as passive recipient of the decision result rather than as active decision maker.By framing decision-support AI as a shared technology in our study, we propose that patients represent another human actor in such interactions.Their role in the human-in-the-loop decision making is not to improve the system performance per se but more to decide on the most favorable role for AI in a given context, helping systems to function best for them.Our framing of decision-support AI as a shared technology suggests that patients might play different roles as another human in the loop in AI-powered shared decision making.
We anticipate that framing AI-powered DSTs as shared technology between patients and HCPs will serve as a valuable approach for integrating patient-centered perspectives into the design of AIpowered healthcare decision-making.This conceptual lens will direct more attention toward patients' roles and their interactions with HCP and AI in shared decision making, inspiring meaningful discourses beyond the current mainstream research emphasis on HCP-AI interactions.

Limitations & Future Work
While our findings provide initial insights into patients' perspectives in AI-powered decision making in healthcare, we would like to acknowledge the limitations of this work which can be studied further in future research.First, our findings are based on the responses of a small number of participants, thus further validation with larger groups of patients is needed to strengthen the generalizability of our findings.Also, while we intended to focus on only a few possible forms of AI applications for patient-facing DSTs in this study, more specific design choices for configuring human-AI autonomy will be worthwhile to be studied further.There are several variables that might have influenced the results.For example, participants' in-story choices could have been affected by cultural differences in understanding healthcare, types of diseases (acute vs. chronic diseases), and the method of becoming pregnant.Next to that, the presence of the researcher during the interviews might have influenced participants to report more preferable responses for the topic of the study (preferring AI) resulting in a skewed view in the result.Also, we focused on highly performing AI to create maximal trust baseline to explore participants' preferences for AI which might have resulted in different results otherwise.The relationships among these variables and patients' choices on AI can be further explored.Lastly, although we mainly focused on the preference of patients (i.e., women), the opinions of other important actors (e.g., family members, partners) can play a substantial role in healthcare decision making.The role of significant others of patients can be furthered studied in the context of AI-powered healthcare decision making.

CONCLUSION
In this paper, we investigated patients' preferences for the level of AI autonomy in healthcare decision making using an interactive narrative and speculative AI prototypes.The results showed our participants' general preferences for the advisory level of AI assistance in which they can actively search for risks in health decision options.They preferred to use it with HCPs to discuss for further clarification and negotiation.On top of this general preference, the study revealed that preferable levels of AI autonomy in healthcare decision making can vary depending on patients' personal attitudes, health history, perceived risks in decision making, and changing perceptions of AI.This dynamics suggests the need for respecting and incorporating patients' preferences through variable autonomy of AI in healthcare decision making and safeguarding patients' vulnerability from over-reliance on AI.We hope that these insights provide a valuable starting point to address patient-centeredness in the design of future AI-powered healthcare decision making.Q2: How would you like to decide on your calcium intake?By myself (without DT): • a.I would make my calcium intake plan by myself and do not make use of the DT functions.With DT: • b.I would make my own calcium intake plan based on DT calculator.• c.I would follow the recommendations of my DT virtual advisor.• d.I would decide to install a calcium sensor to use the DT virtual doctor's autonomous calcium management.With DT & HCP: • e.I would use and discuss the DT calculator's predictions with my midwife.• f.I would discuss the DT virtual advisor's recommendations with my midwife.• g.I would ask my midwife to approve the installation of a calcium sensor to use the DT virtual doctor's autonomous calcium management.With HCP (without DT): • h.I would follow my midwife's advice and do not make use of the DT functions.

DECISION 2
In-story questions and choices for the decision on the timing of the pregnancy termination to prevent further progress of preeclampsia: Q1: What would you do next?
• a.I would wait and follow my gynecologist's recommendations on the optimal timing of responding to the preeclampsia.• b.I would consult the predictions of my DT to see when the optimal timing would be to respond to the pre-eclampsia.
Q2: How would you like to decide on the timing of your delivery?
By myself (without DT): • a.I decide whether and when to terminate the pregnancy based on how I feel and do not make use of the DT functions.With DT: • b.I choose the delivery date and time based on the DT calculator.• c.I follow the DT virtual advisor's recommendation.
• d.I activate the autonomous delivery planning by the DT virtual doctor.With DT & HCP: • e.I use and discuss the DT calculator with my gynecologist.
• f.I discuss the DT virtual advisor's recommendations with my gynecologist.• g.I ask my gynecologist to approve autonomous delivery planning by the DT virtual doctor.With HCP (without DT): • h.I follow my gynecologist's advice and do not make use of the DT functions.

Figure 2 :
Figure 2: The three AI prototypes introduced in the second decision-making situation.

Figure 3 :
Figure 3: Questions to elicit participants' choices in the story.

Figure 4 :
Figure 4: Potential modes of patient-HCP-AI interactions for shared decision making from the perspective of patients

Table 1 :
Overview of Speculative AI Design Concepts