A Taxonomy of Robot Autonomy for Human-Robot Interaction

Robot autonomy is an influential and ubiquitous factor in human-robot interaction (HRI), but it is rarely discussed beyond a one-dimensional measure of the degree to which a robot operates without human intervention. As robots become more sophisticated, this simple view of autonomy could be expanded to capture the variety of autonomous behaviors robots can exhibit and to match the rich literature on human autonomy in philosophy, psychology, and other fields. In this paper, we conduct a systematic literature review of robot autonomy in HRI and integrate this with the broader literature into a taxonomy of six distinct forms of autonomy: those based on robot and human involvement at runtime (operational autonomy, intentional autonomy, shared autonomy), human involvement before runtime (non-deterministic autonomy), and expressions of autonomy at runtime (cognitive autonomy, physical autonomy). We discuss future considerations for autonomy in HRI that emerge from this study, including moral consequences, the idealization of "full" robot autonomy, and connections to agency and free will.


INTRODUCTION
Robot autonomy strongly and unavoidably shapes user experiences in human-robot interaction (HRI).For example, users who fully control a robot's movements via teleoperation (e.g., drones, telepresence robots) tend to view the robot as merely a tool rather than a "colleague" [157].On the other hand, users who interact with robots that operate independently (e.g., robot vacuums, robots that give directions in airports) tend to view them as distinct agents with their own intentions and goals [191].Users who interact with robots that exhibit higher levels of cognitive autonomy (e.g., moral judgement [49], cheating [104], achieving individual goals [98]) readily anthropomorphize and thus respond to them in ways that they might respond to humans [52,122].
Despite the importance of robot autonomy, there are currently two key challenges with the conceptualization and usage of the term "autonomy" in HRI.First, it is conceptually ambiguous.Many papers in HRI use autonomy only as a descriptor that applies to all robots in a general paradigm (e.g., "autonomous vehicle") [96,105,186], while other papers describe robot autonomy as a complex feature intertwined with notions of mind [188], agency [84], or free will [46].These conceptual differences make certain empirical questions intractable because different conceptualizations yield different answers.For example, the finding that people are hesitant to enter an interaction with a robot may be explained by a reduction in user control if autonomy is conceptualized as a lack thereof [132] or explained by a threatening level of robot intelligence if autonomy is conceptualized as a sophisticated mental faculty [160].
Second, over the past two decades in HRI, when autonomy has been made conceptually precise, it has predominantly been a onedimensional scale of the level of human involvement in robot operation, as in seminal taxonomies from Beer et al. [15] and Huang et al. [85].This narrow view of robot autonomy stands in contrast to the literature outside HRI with rich and complex conceptualizations of human autonomy in philosophy, law, medicine, social science, and even in human-computer interaction (HCI), such as autonomy as a basic human right [66] and autonomy as the essence of what it means to be a person [59].In HRI, we seem to be conflating, rather than articulating, critically different cases of autonomy.Consider that one study shows that increased robot autonomy leads to greater trust in the robot [154] while another describes how increased robot autonomy leads to less trust [10].How can an increase in autonomy lead to opposite outcomes?We propose that both conclusions can be true simultaneously because the two works are referring to different forms of autonomy altogether.While the former appears to be measuring autonomy as the level of human intervention, the latter seems to be measuring it in terms of the robot's cognitive capabilities.If we can tease apart these cases, we will have a powerful conceptual tool to understand and predict user experience outcomes (e.g., user situational awareness [146]) and robot ethics [47].
Our approach addresses three research questions: RQ1: How have other fields (e.g., philosophy, psychology) conceptualized and used the term "autonomy"?
RQ2: How has the field of HRI conceptualized and used the term "autonomy"?
RQ3: How can we develop a taxonomy that integrates these literatures to allow the HRI community to more precisely understand and communicate distinct forms of robot autonomy?
To answer these research questions, we first review popular conceptualizations of autonomy from outside the HRI literature and present a systematic literature review of the usage of "autonomy" in HRI.We then propose a comprehensive taxonomy based on these literatures with six distinct forms of autonomy summarized in Table 1: those based on robot and human involvement at runtime (operational autonomy, intentional autonomy, shared autonomy), human involvement before runtime (non-deterministic autonomy), and expressions of autonomy at runtime (cognitive autonomy, physical autonomy).Finally, we discuss how our taxonomy could allow the HRI community to better manage, predict, and revise expectations for human interactions with autonomous robots.

CONCEPTUALIZATIONS OF HUMAN AUTONOMY
The predominant definition of autonomy in HRI has been the degree to which a robot operates without continuous supervision or intervention from a human operator [7,170].High robot autonomy can thus be defined as low levels of human operator involvement (i.e., high levels of human operator disinvolvement), and low robot autonomy as high levels of human operator involvement [85].This relatively narrow conceptualization in HRI is a sharp contrast with the varied and rich conceptualizations of human autonomy outside of robotics and HRI.Etymologically, "autonomy" is being subject to one's own laws, derived from the Greek autos, "self, " and nomos, "law" or "governance." Scholars have interpreted this in myriad ways.
We answer RQ1 with a brief overview, developed through the first and second authors aggregating various definitions of autonomy by and across disciplines into conceptual foci.

Self-Determination
Self-determination can refer to one's actions being caused by oneself rather than external forces [106].Friedrich Nietzsche viewed a person as autonomous if and only if they are causa sui, or "selfcausing" [61].However, this arguably means no agent is truly autonomous, in the sense that all events have past causes [61].Most other accounts are substantially weaker, only requiring, for example, that one is able to have some control over whether to act on certain desires.In the "hierarchical" model of autonomy [50], the self is identified with an internal freedom to endorse or reject first-order desires, which are desires for anything other than a desire, such as "I want to not overeat, " but not higher-order desires, such as, "I want to not want to overeat" [50,59].To have "reflective self-evaluation" is to be aware of first-order desires and able to choose which to act upon based on higher-order desires [59,71,177].The closely related "substantive" model of autonomy states that an absolute endorsement of our actions is not essential to autonomy.Instead, autonomy entails the capacity to revise our decisions with moral reasoning [50,184].Endorsement of actions need not be entirely self-originating in order to constitute selfdetermination, as the hierarchical model suggests, but one must be able to challenge the external forces that would otherwise determine one's own actions.
Self-determination is also central to psychological definitions of autonomy that stress the importance of a sense of self [44,159].Self-determination theory (SDT) is one of the most popular psychological frameworks for human motivation that comprises three basic needs: autonomy, competence (i.e., the ability to implement autonomy oneself), and relatedness (i.e., the ability to implement autonomy with others).The central distinction in SDT is between "autonomous" and "controlled" motivation.Autonomy is characterized in SDT as "volition, or a self-endorsement of their actions" [45].Autonomous motivation originates from an internal locus of causality, which entails self-endorsement and the ability to choose one's decisions.The person's behavior is thus propelled by the identification of their actions with intrinsic values.Controlled motivation, by contrast, is dictated by external events and particularly by avoiding punishment from other people [149].

Independence
Many conceptualizations of autonomy focus instead on the direct independence of an individual's behavior from external forces, rather than the internalized sense of self and identity, though independence from external forces plays a role in that as well.These conceptualizations directly argue that the hallmark of a truly autonomous agent is to have liberty of judgment and choice from external dependencies-not just physical, but cognitive as well [40].Philosophers of antiquity took this view.For example, Aristotle argued the self-sufficient person was not dependent on any other being or external force to guide their own happiness [50].
Legal and political conceptualizations also tend to emphasize independence from external forces, such as oppressive governments [9,41].HCI researchers primarily conceive of human autonomy as the human's degree of independence from the computer's restrictions [16].Medical definitions emphasize the lack of external "hindrances" to the patient's cognition and physical activity [63].

Other Conceptualizations
In political theory, autonomy is also used to describe the selfgovernance of collective entities.Autonomy can be an attribute of political states, ethnic groups, territories, and institutions if they are able to operate without the control of a higher level of government, and is typically protected by international law [56].Political autonomy can also entail the autonomy of individuals within the collective body, particularly the freedom for individuals to selfgovern as long as it does not infringe on the autonomy of others.Philosopher Jean-Jacques Rousseau famously conceptualized the "general will," the interests of a collective that are derived from the individuals therein [17].Thompson [165] argued that this is a form of autonomy.Through liberating oneself from individualistic, possibly hedonistic, motives and desires (i.e., the "private will") and assimilating into the general will, one preserves her individual autonomy [165].It is only through the social and cultural environments to which one chooses to respond that an individual is able to critically evaluate her own values, beliefs, and desires.Under this view, individual autonomy cannot be conceived through self-evaluation alone [34].
Autonomy is frequently associated with "freedom of the will" or free will-which is distinct from Rousseau's idea of the general will.Free will is generally regarded as a degree of power or control over one's actions [125].The nature of the relationship between free will and autonomy is rigorously debated.In some theories, free will is stronger than autonomy: an agent can be autonomous without having free will but cannot have free will without being autonomous [28].In other theories, autonomy is the capacity to reflect upon and execute free will, providing the "self" that enables self-evaluation and self-governance.Kant is known for this view, arguing broadly that an autonomous will is a free one [90].In Kantian ethics, autonomy is "made possible" by exerting free will to follow the universal moral law, or the "categorical imperative", and not being subject to any external moral laws or one's own first-order desires or inclinations [69,91].To be autonomous is to have a certain moral independence and freedom to fulfill one's duty [40,91].

Takeaways from the Conceptualizations of Human Autonomy
We divide the usage of autonomy into two conceptual foci: selfdetermination and independence.Self-determination is a sense of direct internal control over one's actions, choices, and identity.Independence is the freedom from external control over one's cognitive and physical actions.Additionally, political autonomy is a way that a body of individuals can be collectively autonomous.
The relationship between free will and autonomy raises discussion on what kinds and degrees of cognitive and moral capacities qualify one as autonomous.As we will later show, our taxonomy can leverage this literature on human autonomy to both broaden the conceptualization of robot autonomy in HRI and to make it more precise.

SYSTEMATIC LITERATURE REVIEW
To answer RQ2, we examined how robot autonomy is used within the HRI community.
Cognitive ( 34) (a) Usage of the term "autonomy" in the corpus (b) Forms of autonomy defined in the corpus Interaction (THRI).We reviewed all items with "auton*" (e.g., autonomy, autonomous) in the title, abstract, or keywords through July 31st, 2023.Because our focus is full research articles, we excluded the 262 HRI publications in other formats (e.g., extended abstracts, videos, late-breaking reports), resulting in 500 included papers.
A subset of 40 papers (10 from each venue) was annotated by four researchers to reach consensus on procedure and categorization.Once consensus was reached, the other 460 papers were each annotated by one researcher except when a categorization was ambiguous and needed input from others.For each paper that either defined autonomy (e.g., "the robot acted autonomously (without intervention from a human operator or Wizard of Oz, WoZ)") or implied a definition (e.g., "the robot must not require any human operators and must operate fully autonomously"), it was categorized into one or more forms of autonomy in HRI (e.g., operational, cognitive), which we developed over the course of the review process and describe in detail in Section 4. Each other paper was placed in one of three categories based on whether the term "auton*" was only used as a label for a robot (e.g., "autonomous social robot"), a summary of related work, or a reference to human rather than robot autonomy.Additionally, if any of the 500 papers included a human-subjects experiment, we annotated how autonomy was explicitly used as an independent variable or dependent variable.Finally, we took note of any papers that had a notable discussion of "agency" or derivative terms.The complete literature review annotations are available as a supplemental document.As shown in Figure 1, there was a notable increase in the number of papers mentioning autonomy from 32 in 2008 to 75 in 2022.The year 2023 (30 papers) was not included in the graph because there was not a full year of data available.As shown in Figure 2(a), we found 235 explicit or implied definitions of autonomy (98 explicit, 137 implied).Another 238 papers only had robot autonomy as a label (e.g., "autonomous robot, " "autonomous vehicle"); 16 only had robot autonomy as a summary of related work; and 11 referred only to human autonomy (whether defined, used as a label, or only referenced in a summary of related work).As displayed in Figure 2(b), of the different forms of robot autonomy, we categorized 178 as operational, 49 as shared, 34 as cognitive, 18 as non-deterministic, four as physical, and three as intentional.Of the 238 papers that only had autonomy as a label, which were most common in RO-MAN (60% of papers), we did not place them in one of these categories because they did not explicitly state or imply a definition, but we assumed that the papers' authors had the operational definition, if any, in mind.For example, when authors only used the label "autonomous vehicles, " they seemed to have the definition in mind of a vehicle that travels without a human operator [20,30,92].

TAXONOMY OF ROBOT AUTONOMY
To answer RQ3, we propose a taxonomy that characterizes distinct forms of robot autonomy (see Table 1) by integrating the rich, interdisciplinary discussions of human autonomy with the results of our systematic literature review.We conceptualize three domains of robot autonomy.Human and robot involvement at runtime includes operational, shared, and intentional autonomy; human involvement before runtime includes non-deterministic autonomy; and expressions of robot autonomy at runtime include cognitive and physical autonomy.

Human and Robot Involvement at Runtime
The first domain of robot autonomy in the taxonomy is the degree of human and robot involvement at runtime, which includes three forms of autonomy: operational, intentional, and shared autonomy (see Figure 3).This domain is general-sum, incorporating both human and robot involvement as independent factors.This perspective stands in contrast to the zero-sum perspective in many HRI models, which assumes, for example, that 90% human involvement implies 10% robot involvement [85,123].Some recent work in HCI has found a general-sum approach more useful for characterizing human-AI interactions, defining AI roles based on the levels of both "human involvement" and "AI autonomy" [93].
This conceptualization is displayed in Figure 3.We depict operational autonomy as the degree of human operator disinvolvement at runtime (the horizontal axis) and intentional autonomy as the degree of robot goal-oriented involvement at runtime (the vertical axis).This allows us to categorize common robots in terms of their location in the space.Teleoperated robots have low robot involvement (i.e., low intentional autonomy) and low human operator disinvolvement (i.e., high human involvement; low operational autonomy) [132]; they are essentially vehicles for their human operators and would be idle without one.On the other hand, a drone swarm typically has high robot involvement and low human operator disinvolvement [95].The ability to differentiate cases such as teleoperated robots and drone swarms-each of which has low human disinvolvement, but different levels of robot involvement-is the key benefit of our general-sum taxonomy.
Shared autonomy is represented by the grey square, requiring at least some intentional involvement by both the robot and human.If either the robot or human has no control over operation (low intentional or high operational autonomy, respectively), there is no shared autonomy.Shared control paradigms, such as "symbiotic autonomy", where humans and robots "fill in" the other's mutual weaknesses [176], and negotiation, where both the human and and robot goal-oriented involvement (intentional autonomy).
Intentional involvement from both human and robot at runtime represents shared autonomy, depicted as the grey rectangular area.
robot "negotiate" their level of autonomy given the other's [89], are examples of shared autonomy.
4.1.1Operational autonomy.We define operational autonomy as the degree of human operator disinvolvement at the robot's runtime.We refer to "disinvolvement" instead of "involvement" for consistency with the other forms, in which "high" denotes high autonomy and "low" denotes low autonomy.Full operational autonomy occurs when there is full human operator disinvolvement (i.e.no human operator involvement), so the robot is able to function and execute its tasks entirely on its own.For example, a Roomba robot vacuum can clean an assigned area without any human intervention.On the other hand, in the Wizard of Oz (WoZ) paradigm or with telepresence robots, there is little to no operational autonomy because all robot actions are conducted by a human operator.Olatunji et al. [131] say that autonomy is "typically considered as a continuous or discrete spectrum, with direct human control and full autonomy at either end, and any number of intermediate levels in between." Our conceptualization of operational autonomy is inspired by the aforementioned conceptual focus of autonomy as "independence" from external factors.Relative to the robot, human intervention is external, even if it is in response to the robot's needs within its local environment (e.g., a broken wheel).Even when operationally autonomous, robots are usually inexorably tied to the context of their environment [24] and human operators [82].
Operational autonomy is the dominant conceptualization of autonomy in the HRI literature.Figure 2 shows that 178 out of 235, or 75.7%, of papers that define autonomy do so as operational.Explicit definitions focus on the omission or inclusion of human involvement, using phrases such as "amount of/without Term Definition Example Associated Terms Human and Robot Involvement at Runtime • "no human intervention" [42] • "do its tasks without a human's input" [53] • "sense its environment, plan based on that environment, and act upon that environment with the intent of reaching some taskspecific goal (either given to or created by the robot) without external control" [15] control, input, the degree of human operator disinvolvement at runtime intervention, involvement, remote, tele-operation, tele-presence, Wizard-of-Oz

Operational Autonomy
the degree of robot goal-oriented involvement at runtime deliberate, directional, goal, random, wandering

Intentional Autonomy
• "several types of in-the-wild studies of mobile robots can be run without autonomous navigation, using wandering instead" [120] the degree to which the human operator and robot are both intentionally involved at runtime • "joint control of a robot by a human user and an autonomous control system" [26] • "reduce the amount of control people need to exert by 1) predicting people's goals and 2) taking assistive actions toward those goals" [7] adjustable, degrees, levels, semi-autonomous, shared control, symbiotic

Shared Autonomy
Human Involvement Before Runtime the degree to which a robot's behavior is not specified prior to runtime • "autonomous or pre-programmed" [84] • "neither remote controlled by a human, nor scripted previously, but is instead generated by an algorithm that reacts to external sensor stimuli" [151] adaptive, determine, pre-programmed, predict, scripted, stimuli

Non-deterministic Autonomy
Expressions of Autonomy at Runtime the degree to which the robot takes cognitive action • "to make ethical choices" [49] • "to reason, or understand the sensory information it is receiving, and plan the most appropriate action based on this input" [189] Cognitive Autonomy choice, decision, goal, moral, reason, think • "e.g., [...] is the robot confined to a pen or does it wander and get stuck under television cabinets" [150] • "energetic autonomy, i.e., the capability to cover long distances without the need to be recharged" [115] the degree to which the robot takes physical action

Physical Autonomy
confined, dance, drive, move, stuck Table 1: We propose a comprehensive taxonomy of six distinct forms of robot autonomy based on our systematic literature review and integration with the literatures on human autonomy in other fields (e.g., philosophy, psychology).
Similar scales exist in medical robotics [187] and unmanned systems research [85].
Operational autonomy is also the form of autonomy most frequently manipulated in HRI studies [48,109,163].Manipulations tend to present discrete levels of robot autonomy as defined in the LoRA scale [15], either varying on whether humans are involved at runtime at all [170] or on a continuum of low, partial, or high input [4,18,48,65,109,121,123,131,163,188].A common set-up compares an autonomous robot with one that is WoZ or teleoperated [33,81,119,121,168].Some papers describe a "semiautonomous" approach, wherein a robot can recover from errors made in its autonomous state through operator intervention [94,156].Brooks et al. [27] varied the length of time in which the human operator "ignores" or "neglects" the robot with longer times constituting higher autonomy.Studies have found various effects of operational autonomy, such as Torre et al. 's [167] finding that participants are less likely to swerve out of the way to avoid collision with autonomous robots than teleoperated robots.Outcomes of interest include performance [32,131] and perceptions of operational autonomy and robot reliability [48].For instance, Rosenthal-von der Pütten et al. [143] found that more human-like robot appearance and the inclusion of a narrative about the robot, rather than just an instruction manual, led to evaluations of the robot as more autonomous and intelligent.Reuben et al. [147] tested participants' judgements of whether a robot was autonomous or teleoperated based on their mental model of the robot over a six-week in-thewild study and found that participants' judgements were impacted by their understanding of possible behaviors of autonomous robots.

Intentional autonomy.
We define intentional autonomy as the degree of robot goal-oriented involvement, or simply robot involvement, at runtime.Intentional autonomy is analogous to operational autonomy, but it is determined within the robot and without direct reference to external factors.Our conceptualization of intentional autonomy is thus associated with self-determination, because intentional autonomy is about the robot's internal, rather than externally imposed, goals.Because the robot is necessarily involved with its own behavior in the sense that it is the one exhibiting the behavior, we specify "goal-oriented involvement" as the robot being both oriented towards a particular goal and having knowledge of this goal.We see the robot knowing its goal, as opposed to accidentally achieving it, as having the necessary internal representation to stay targeted at the goal.For example, Nanavati et al. [120] describe a "wandering" robot that arbitrarily explores its local environment with no directionality.In this case, while there is full operational autonomy (i.e., full human operator disinvolvement), if the robot ascribes no purpose or internal representation to its "wandering", there is no intentional autonomy.In the absence of any involvement, behavior is left to random chance or environmental factors (e.g., wind, bystanders).Robots with high intentional autonomy exhibit non-random behavior at runtime [103].For example, an effective game-playing robot that evaluates the game state and selects actions to optimize its chance of winning [2] would have high intentional autonomy.
In this taxonomy, intentional autonomy does not require specific mental faculties, such as self-awareness or agency, as "intention" is sometimes conceptualized outside of HRI [182], only that the robot knows its goal in the previously defined sense.One could argue that no robots achieve this because they are by definition created by humans and take action according to human inputs.However, it is clear in the human case that, when humans are willfully or forcibly constrained based on another person's intentions (e.g., a child grounded by their parents), they maintain some autonomy.To speak in terms of our conceptual foci, their independence may be limited, but their self-determining faculties (e.g., cognitive abilities, goal-setting) can still manifest.With intentional autonomy, the idea is that even though robots may not escape the influence of a human at any point in its operation, robots can still express purposeful behavior to achieve some goal.We can observe this when, for example, a robot self-determines the best "level of autonomy" to use [146].

Human Involvement Before Runtime
4.1.3Shared autonomy.We define shared autonomy as the degree to which the human operator and robot are both intentionally involved at runtime.Shared autonomy is unique in our taxonomy, in that it is a composite of operational and intentional autonomy, as shown in Figure 3. High shared autonomy occurs when the human and robot are more equally involved, and low shared autonomy is when one is much more involved than the other.This concept of shared autonomy corresponds to myriad terms in the HRI literature: symbiotic autonomy [12,176,179], shared-autonomy teleoperation [62], adjustable autonomy [64,77,108,118], adaptive autonomy [11,75], shared control [26,58,89,127,172], assistive robot control [7], traded control [126], semi-autonomy [13,21,38,94,99,107,114,138,139,156,158,169,175,185], confidence-based autonomy [161], collective autonomy [35], and shared autonomy [7,26,72,98,116,124].Our conceptualization is partly influenced by the outside literature on collective autonomy.Some studies in HRI refer to collective autonomy in the context of "teams" or "swarms" of more than two systems collaborating [95,110,134], even explicitly circumscribing collective autonomy as being under the command of a single operator [35].Outside of HRI, however, collective autonomy can encompass a unique sort of autonomy altogether for collections of individuals (e.g., the general public).
Shared autonomy manifests in two ways: simultaneous control and exchanged control.In simultaneous control, the human operator and the robot take action at the same time [94,138,185].For instance, a robotic arm picking items off of a shelf may perform the movements themselves without human involvement, but with the direction and speed of the task being simultaneously dictated by the human operator.In exchanged control, only one party takes action at a time [6,26,89,123].For example, if a human picks up their robot vacuum and sets it in each room of a house, where the robot vacuum cleans without human involvement, this is a case of exchanged control between the human while moving between rooms and the robot while in a room.Most papers that define shared autonomy, or a closely associated concept, emphasize its benefits, particularly its flexibility in either the robot or human deciding when to take or yield control.Veloso [179] defines symbiotic autonomy as "the robots explicitly include actions to ask for help from humans in their behavior policies." Similarly, Naghsh et al. [118] define adjustable autonomy as one in which, "the robot entity needs not make all decisions autonomously, rather it can choose to reduce its own autonomy level and transfer decision making control to other users or agents."Supervision can also serve to correct errors [112], and many researchers describe the human-robot cooperation as a "blend" of user and robot involvement [26,32,57,116,127].As robots become more sophisticated and collaborative (e.g., "Human-Autonomy Teams" [1]), questions of collective autonomy will become more pertinent-not just for robot autonomy, but also how robots should be deployed when their autonomous operation may affect the collective autonomy of political states and other institutions.
During runtime (i.e., the period in which the robot is operational), both the human operator and robot can have input and control over the robot's autonomy.Prior to runtime, however, only the human  takes action since the robot is not operational.Therefore, there is only one axis of autonomy, which we characterize in this section as non-deterministic autonomy.

Non-deterministic autonomy.
We define non-deterministic autonomy as the degree to which the robot's behavior is not specified prior to runtime. Figure 4 (left) illustrates a robot with low non-deterministic autonomy because it is given exact instructions to reach its destination; there is no leeway for the robot to further optimize or diverge from its navigation path.Examples of low nondeterministic autonomy in the literature include robots that are explicitly said to be "pre-programmed, " [3,84,102] "scripted previ-" [151] or have "predefined" [31] or "manually designed" [111] behavior.On the other hand, the robot with high non-deterministic autonomy in Figure 4 (right) is not provided the exact sequence and execution of its runtime operation.Instead, the robot can decide the optimal path during runtime to reach its destination and adapt to novel situations at runtime, such as "what information is needed and at what time and place it is requested" [73], in order to accomplish its goal.Examples of highly non-deterministic robots in the literature include those that are able to adapt to their environment [31,78,140,162] or adapt to people [79,142] in real-time.
In practice, non-deterministic autonomy may correlate strongly with intentional autonomy.However, we view these two forms of autonomy as distinct.For example, a robot that has extensive specification of behavior prior to runtime may behave in a goal-oriented way during runtime (e.g., navigating to a specific location despite alternative rewards), but that behavior has still been specified prior to runtime.The robot would thus have low non-deterministic autonomy but high intentional autonomy.
It remains unclear exactly how non-deterministic autonomy could be operationalized or measured.One could measure the degree of conditionality (e.g., number of "if" statements in the code), the diversity of possible runtime environments, or the sophistication of action selection subprocesses, but in each case the robot's programming would still determine its actions in a given context.Nonetheless, this concept seems to capture an important sense of autonomy in the HRI literature that varies even if the robot has full operational or intentional autonomy.

Expressions of Robot Autonomy at Runtime
In addition to considering robot and human involvement in a robot's operation before and during runtime, there are different ways in which robot autonomy is expressed at runtime, which we categorize as cognitive and physical.4.3.1 Cognitive autonomy.We define cognitive autonomy as the degree to which the robot takes cognitive action.Generally, cognition is defined as "[t]he action or faculty of knowing taken in its widest sense, including sensation, perception, conception, etc., as distinguished from feeling and volition" [36].Robot cognition includes the ability to make and execute decisions [135,148,155], set and achieve goals [15,98], understand others' mental and affective states [14,108,128], act upon one's senses and planning [15,37,101], and learn from the environment and past experiences [19,86,144,189].Discussion of robot cognition is rife with the same sort of disagreement as has persisted for decades in the human cognition literature [97].Some have explicitly defined robot cognition as synonymous with artificial intelligence and autonomy [77].We view cognitive autonomy as not requiring intentional autonomy.It is possible for a robot to reason about the world in complex ways without a particular goal in mind.
In our literature review, forms of cognitive autonomy fell naturally into two categories: moral autonomy and decisional autonomy.Moral autonomy means the robot is capable of using moral reasoning to select actions, such as distinguishing right from wrong [25,49,129,174] and resolving morally ambiguous scenarios [160].Moral autonomy often involves an awareness of ethical and social norms prior to operation [68,181] and of moral stakeholders during operation [133,171].Decisional autonomy is a robot's decision-making ability, such as the "ability to observe and act on their environment, as well as conduct activities toward achieving both individual and collective goals" [98] or to "generate its own strategies for action" [37] and independently evaluate or deliberate on those actions [55,101,135].For example, decisionally autonomous robots can independently reason about "manipulation goals" when predefined goals by the human operator may fail [87].A particularly important decision-making ability is the "ability to disregard human commands" [2], which was only utilized in one paper in our review [2], but several included papers cited Złotowski et al. [191], who utilized this form of decision-making.

Physical autonomy.
We define physical autonomy as the degree to which the robot takes physical action.The paradigmatic case of restricting physical autonomy is confining the robot in a physical space, such as a cage [150], which we refer to as movement autonomy.Levillain et al. [100] discuss autonomy as "bursting into sudden movements, " which would require significant freedom of movement.Physical autonomy can also manifest through the resource requirements of the robot, particularly its energy source, which we refer to as energetic autonomy.An example of a robot with high energetic autonomy is a drone that can recharge itself to fly indefinitely [115].Physical autonomy can be increased or decreased based on the desired robot behavior.In a study by Horn et al. [83], the same robot could be used either in a "passive mode" as a walking aid or in an an "autonomous mode" wherein the robot can move towards the user without human assistance.

FUTURE CONSIDERATIONS FOR ROBOT AUTONOMY
In this section, we discuss future considerations for autonomy in HRI that emerged from this work: moral consequences, the idealization of "full" robot autonomy, and connections to agency and free will.Developing these perspectives will allow the HRI community to better manage, predict, and revise expectations for the scaling and enrichment of human interactions with autonomous robots.

Moral Consequences of Robot Actions
We defined moral autonomy as a robot's capability to select actions through moral reasoning, but there is a wide spectrum between merely having behavior with moral consequence, such as being pre-programmed to encourage a human patient to take medication [174], and human-level consideration of right and wrong, such as in Kantian moral theory through adherence to universal moral law.In both cases, HRI designers face difficult decisions in choosing between ethical frameworks like utilitarianism and deontology, and these challenges will only increase as increased autonomy involves robots making more decisions with significant moral implications [49,181].An initial step towards tackling these challenges is addressing how robot autonomy can both erode and enhance the autonomy of humans.Robots shifting roles from "tool" to "colleague" [157] and gaining humans' trust [153] and moral consideration [5,76,137] may make humans feel as though their autonomy is threatened by and traded off with robot autonomy [18], or it could complement and preserve human autonomy by providing aid [183] and maximizing the user's capabilities, especially on command [43,178].

Moving Towards "Full Autonomy"
Within their study, Wright et al. [186] state "full autonomy is an essential objective, " and it is generally regarded as desirable but "not yet practical" [29] or "far from reality" [62].Robot autonomy will scale towards this "fullness" in different ways and towards different forms.Operational autonomy is relatively straightforward: full operational autonomy refers to states where the degree of human operator disinvolvement is near 100% [51].However, "full" autonomy is unclear for the other five forms we describe.In general, HRI researchers have expressed that robots are not, and should not be, able to possess free will [46] or commit harm against humans [113].And yet, some argue that, "as the cognitive, perceptual, and motor capabilities of robots expand, they will be expected to be explicit ethical agents" [174].As HRI research strives to keep pace with the real-world increase in robot autonomy, the ideals of human autonomy can be a useful yardstick.For instance, just as some accounts of human autonomy are based in reflective self-evaluation, more "advanced" robot autonomy, and its regulation, could be based in the robot's ability to reflect upon and criticize its choices.

Robot Agency and Free Will
We focus on autonomy because of its importance in HRI, its relative concreteness, and the lack of existing comprehensive frameworks.However, there are related concepts that also bear on the future of HRI.Agency is arguably the most closely related concept, as reflected in the numerous discussions of agency in the reviewed items.For example, Zafari and Koeszegi [188] define agency as "the capacity to perform a goal-oriented task to an extent autonomously on the environment, " implying that agency requires autonomy, but others define autonomy and agency as separate concepts [132,145].In many papers focused on reinforcement learning, the word "agent" is used merely to differentiate the subject of the model from its environment, which presumably does not require autonomy in any of the definitions considered in this review.This minimal sense of agency is also the most common outside of HRI, in which an agent is merely a system that takes action [152].Bennett et al. [16] provide a thorough and timely review of agency and autonomy in the context of HCI, though their focus is on that of the human, rather than the computational or robotic system.We did not often find the concept of "free will" in the reviewed items.The outside literature has many different conceptualizations of free will, and there seems to be agreement that it is a more sophisticated and less common property compared to autonomy.This helps explain the lack of discussion in HRI and robotics, because while robots are increasingly complex and capable, few would argue that they are sophisticated enough to possess free will [46].If we adapted the Kantian notion of autonomy to robots, such that autonomy relied on free will, then robot autonomy is not even a possibility for current systems.Human-like robots such as Sophia, the world's first robot with legal citizenship, make salient the possible futures of robots with roles, responsibilities, and even rights [74].HRI needs rigorous frameworks of autonomy, agency, and free will to answer impending questions of moral responsibility, fairness, accountability, and transparency.

CONCLUSION
Robot autonomy influentially and ubiquitously shapes the user experience in HRI, yet its conceptualization and usage within HRI faces two primary challenges: conceptual ambiguity and unidimensionality.In this paper, we addressed these challenges by broadening the conceptualization of robot autonomy in HRI.We conducted a systematic literature review of robot autonomy in HRI and integrated this with interdisciplinary literatures outside of HRI to propose a taxonomy with six distinct forms of autonomy: those based on robot and human involvement at runtime (operational autonomy, intentional autonomy, shared autonomy), human involvement before runtime (non-deterministic autonomy), and expressions of autonomy at runtime (cognitive autonomy, physical autonomy).A number of future considerations for autonomy in HRI emerged from this process, including moral consequences, the idealization of "full" robot autonomy, and connections to agency and free will.Going forward, we encourage HRI researchers to clarify which sort of autonomy they are analyzing in their conceptual and empirical studies, as well as to extend and critique this taxonomy to better evaluate the evolving nature and effects of autonomous robots in human society.

Figure 1 :
Figure 1: The total count of items from HRI, RO-MAN, IJSR, and THRI with "auton*" in the title, keywords, or abstract.

Figure 2 :
Figure 2: Usage and forms of "autonomy" in the reviewed corpus.

Figure 4 :
Figure4: Non-deterministic autonomy is the degree to which a robot's behavior is not specified prior to runtime, as shown by the example of a navigational robot.