A Scoping Survey on Cross-reality Systems

Immersive technologies such as Virtual Reality (VR) and Augmented Reality (AR) empower users to experience digital realities. Known as distinct technology classes, the lines between them are becoming increasingly blurry with recent technological advancements. New systems enable users to interact across technology classes or transition between them—referred to as cross-reality systems. Nevertheless, these systems are not well understood. Hence, in this article, we conducted a scoping literature review to classify and analyze cross-reality systems proposed in previous work. First, we define these systems by distinguishing three different types. Thereafter, we compile a literature corpus of 306 relevant publications, analyze the proposed systems, and present a comprehensive classification, including research topics, involved environments, and transition types. Based on the gathered literature, we extract nine guiding principles that can inform the development of cross-reality systems. We conclude with research challenges and opportunities.


INTRODUCTION
Over the last three decades, devices that deliver immersive, digital experiences like Virtual Reality (VR) and Augmented Reality (AR) have reduced in size from bulky hardware [54,76] to today's consumer-friendly devices (e.g., Oculus Quest 2, Microsoft Hololens 2).Nowadays, it has become easier to provide great experiences and immersion in a variety of different professional [15,41] or social settings [184,321].In the past, many of these experiences were created around specific manifestations of the Reality-Virtuality Continuum [199], meaning they are limited to concrete technology classes.Here, examples include training in VR [80,100,174], enhancing the real world with AR [2,81,175,250], and vice versa enhancing virtual environments with parts of Augmented reality (AR).AR alters reality by overlaying digital information.Superimposing information empowers users to interact with virtual objects within the real world [200].Thus, AR is the manifestation closest to reality, as it results in users perceiving the physical environment to a stronger degree than they do virtual aspects.According to Azuma et al., AR has three characteristics that need to be fulfilled: AR (1) combines real and virtual elements, (2) is interactive in real time, and (3) is registered in 3D [20].A persistent challenge of AR systems is using and interacting with physical objects [152,343], which is of particular interest for cross-reality research.

Augmented virtuality (AV).
In AV, users are immersed in a virtual environment; however, parts of reality are incorporated into the digital experience [192,200].In comparison to AR, AV relates more to the virtual environment, while AR relates more to the real environment.With the support of see-through modes in current VR devices, AV has recently gained popularity and is, for example, used to configure the play area for the latest VR devices.

Virtual reality (VR).
In VR, users experience an entirely virtual environment with as little interference from the real-world environment as possible.This digital world is not directly bound to the laws of physics and, therefore, can exceed these boundaries [199].Although one could argue that VR represents virtuality on the continuum, current VR experiences do not completely immerse the user into a virtual environment and, thus, do not represent virtuality.For example, users may bump into walls or get motion sickness if the real-world and VR experiences do not align.Hence, we understand VR as a part of MR.VR can be seen as a mode of reality that exists together with the physical reality to provide its users new forms of experiences [333].

Mixed reality (MR).
MR is not a term describing a particular manifestation on the continuum; instead, it represents all possible manifestations on the continuum that involve both reality and virtuality to some extent.In other words, every experience that lies between reality and virtuality is considered to be MR [198,200].Three years ago, Speicher et al. [278] published a paper addressing the following question: "What is Mixed Reality?"They conducted interviews with experts and analyzed 68 related papers, finding that different definitions of MR exist.Hence, in our article, we use MR as an umbrella term that represents all manifestations of the continuum, such as AR, AV, and VR.Furthermore, four experts interviewed by Speicher et al. stated that "five or ten years from now, we will not distinguish between AR, MR, and VR anymore." In other words, there could be one merged category of devices that supports different manifestations.In the future, this category of devices will form the ultimate cross-reality systems.

Actualities
Some cross-reality systems allow for seamless transitions on the continuum, for example, to allow users to transition from the real world into VR [137,258,284] or to integrate parts of reality into their VR experience [59,111,192].Here, the existing term manifestation is too inflexible to reflect such experiences and, more importantly, does not allow to describe changes in these experiences over time.Moreover, reality and virtuality are used to describe the extremes, and thus, their use to describe such experiences could be ambiguous (e.g., the user's reality).Thus, we argue for the term actuality to depict the currently experienced reality of a user.The term actuality goes back to the concept of potentiality and actuality introduced by Aristotle [260].In short, Aristotle stated that potentiality is a not-yet-realized possibility of all possibilities that can happen, and actuality is the realization of a specific potentiality-the actual thing that became real.The English word actuality is derived from the Latin word actualitas, which translates to in existence or currently happening.Thus, an actuality describes the current reality-the things that currently seem to be facts for a user.In the context of reality and virtuality and all their combinations, we can use the word actuality to describe the actual experience of a user.For example, we can consider two users-one using VR and one just standing nearby.The actuality for the VR user would be a virtual, digital experience, while for the bystander, the actuality is just reality.Here, two actualities exist, whereas each actuality is described by one point on the Reality-Virtuality Continuum.Moreover, when a user transitions, for example, from reality to VR, we can say that the actuality of this user changes over time.We use actuality as the universal term to refer to the individual experiences that users of cross-reality systems are having at a specific point in time.Our definition aligns with Eissele et al., who suggests using actuality to describe virtual experiences [68].
Definition 1: Actuality An actuality refers to the current experienced reality of a user on the Reality-Virtuality Continuum.At each point in time, the actuality of a user can be represented by one point on the continuum.The actuality of a user can change over time, allowing one to experience different degrees of virtuality.

Subjects and Objects
Cross-reality systems involve different entities: subjects and objects.The difference between both entities is that subjects have ways of perceiving their environment, while objects have no perception (e.g., a user, bystander, or animal would be a subject, while a table, keyboard, or vacuum cleaner would be an object).Hence, subjects can experience their environment; an actuality that describes their current experience exists.However, besides this difference, subjects and objects also have attributes in common.Primarily, both can exist physically in the real environment, digitally in the virtual environment, or simultaneously in both environments.In previous work, researchers focused mainly on the role of subjects in cross-reality systems.Nevertheless, we believe that objects also play an important role (cf.Section 2.5).

Definition 2: Subject and Object
Cross-reality systems can consist of two types of entities: subjects and objects.They differ in the sense that for subjects an actuality exists that describes their current experience, while objects have no perception of their environments, and thus, no actuality is assigned.

Definition of Cross-reality Systems
Simeone et al. categorized cross-reality systems into two types that either involve (1) a smooth transition between systems using different degrees of virtuality or (2) collaboration between users using different systems with different degrees of virtuality [273].Following this definition, the role that objects can play in cross-reality systems is somewhat neglected, as the definition focuses on the perspectives of the subjects.Nevertheless, the interaction between subjects and objects should be considered in cross-reality systems as well, especially if the object is not intended purely for the subject's actuality but instead was repurposed and integrated into the user's experience (substitutional reality).Following this definition, a haptic prop specifically designed for a VR experience should not be considered a cross-reality system; however, if a real-world object such as a vacuum cleaner is repurposed for a VR experience, we consider it a cross-reality system (e.g., Wang et al. [315]).Therefore, we distinguish three different types of cross-reality systems, which can be defined through the following definition.

REVIEW METHOD
This scoping review [233] presents the first compilation of a literature corpus that analyzes crossreality systems and interactions.While the first publications describing cross-reality systems appeared recently (e.g., for the design space of transitional interfaces [313]), they focus on specific types of cross-reality systems and do not provide a holistic overview of the topic.Following our definition of cross-reality systems, we considered a broader range of literature that focused on research involving: (i) A subject changes its actuality (e.g., a user transitions into VR [29,30]): Type 1 (Transitional).(ii) There is an interaction between at least one subject and at least one object that is repurposed for the current actuality (e.g., a physical keyboard brought into VR for typing [192]): Type 2 (Substitutional).(iii) There is an interaction between at least one subject and at least one other subject, experiencing different actualities each (e.g., users collaborate using AR and VR [41]): Type 3 (Multi-user).
An initial investigation revealed that a systematic search-term-based literature review (e.g., PRISMA1 ) would not be possible, as terms to describe cross-reality systems are not yet fully established.Furthermore, relevant aspects are often hidden within a research prototype or system, are a smaller part of a broader research agenda, or seem too marginal for the scope of the corresponding publication to be described by the authors.An example would be the paper from Ruvimova et al. in which a user is distracted by the noise of an open office space and, therefore, transitions into VR for an isolated experience [258].Here, the developed system was not explicitly described as a cross-reality system; however, it is an intrinsic part of the approach.Hence, to present the most complete literature corpus, we individually screened our initial literature set manually.
For our literature review, we performed the following steps (see Figure 1): (1) We started by manually going through the proceedings from 2015 to 2022 of the five leading conferences in which related cross-reality system papers were published (in parentheses: corresponding publication count): ACM CHI (5131), ACM UIST (767), ACM VRST (627), IEEE VR (1,539), and IEEE ISMAR (373).The corresponding digital libraries account for 8,437 entries for these venues in the given time frame.All authors together checked the title of each paper to identify off-topic research.We considered only full papers, while other types of publications were excluded (e.g., workshop publications, demos, and posters).(2) We then individually read the abstracts (and further sections if necessary) of all remaining publications to identify if the publications fit the scope of our literature review (meaning the three inclusion criteria hold; see Figure 1) and gathered them in a spreadsheet similar to Doherty and Doherty [61].If the relevance of a publication was not clear to the Fig. 1.Literature selection process: The initial literature corpus from five leading conferences was screened based on our inclusion criteria.Then, referenced and citing literature was screened and added based on the same criteria.We repeated this process until we did not find more relevant literature.
screening author, it was discussed with all authors and a mutual decision was made.In total, we identified 160 papers that are relevant for this review.(3) After that, we looked at all references and all citing papers of the already gathered literature to identify further relevant papers, an approach that others have also applied, e.g., Katsini et al. [146].We applied this process recursively, going through the references and citing papers of newly added ones until we could not find any more relevant publications.In this step, we went through 11,465 references and 13,620 citations and found 103 additional referenced papers and 43 additional cited papers (n = 146).(4) In total, we found 306 relevant papers describing a cross-reality system, which we further classified to extract their core features and identify common themes.
The initial literature corpus was compiled using Google Scholar as the main search engine for citing papers while also relying heavily on the IEEE DL and ACM DL.At this point, it is worth mentioning that this strategy does not guarantee one will identify all relevant papers.We screened a tremendous number of publications, and while our literature corpus grew substantial in size, there is a chance that we missed some relevant publications due to human error.However, strict database queries suffer from similar issues, especially when the terminology of the research field is unclear or not yet fully established.Therefore, we argue that our approach was able to identify more relevant research publications than an automatic approach.
The final publication corpus (n = 306) served as the basis for understanding the interplay among different subjects and their actualities and corresponding objects that manifest across the Reality-Virtuality Continuum.For the publication corpus, we went through all publications and identified important features relevant to this survey to obtain a holistic view of the review corpus.Here, we identified features like the research topic and keywords that briefly describe the given research and involved scenarios as well as the purpose of the scenario (e.g., collaboration, leisure activity).Furthermore, we categorized the scenario together with involved subjects and objects.Therefore, we identified and quantified the involved entities (e.g., users, objects/artifacts) and how they were integrated into their scenarios (e.g., real-world objects brought into VR).Further, we extracted the form-factors (i.e., type of used devices) and modalities (i.e., visual, auditive, or haptic).We then identified how different entities relate to one another across the Reality-Virtuality Continuum and how they manifest on the continuum (e.g., VR, AV, AR).A complete version of our literature corpus, including a classification concerning different features, can be found as supplementary material.
Descriptive Summary of Literature Corpus.Over the last decade, we see a clear uptick of publications proposing cross-reality systems (see Figure 2(a)), indicating a growing interest in the research community.While the publication count before 2015 may be inaccurate because we did not screen conference proceedings before that year, a clear trend between 2015 and 2022 remains recognizable.Nevertheless, in 2021, a dip in publications is observable, which is likely an artifact of the global Covid-19 pandemic, as in the year after, the publication count recovers.Furthermore, besides the identified five leading conferences, we identified the IEEE journal Transactions on Visualization and Computer Graphics and the ACM SIGGRAPH conference as highly relevant venues (see Figure 2(b)).Finally, our corpus revealed that a few authors have around 10 publications published on the topic already.Here, Mark Billinghurst is taking the lead with over 20 publications (see Figure 2(c)).

CLASSIFICATION OF RESEARCH PROPOSING CROSS-REALITY SYSTEMS
Our classification of previous research focuses on the user interactions taking place in cross-reality systems.To understand and classify the gathered research, we analyzed our previously collected 306 papers.Each publication presents an artifact contribution (research prototype or system) that involves more than one manifestation of the Reality-Virtuality Continuum.Next, we present the classification of our research corpus concerning the three types of cross-reality systems and their research topics (see Section 4.1).Thereafter, we analyze the involved real and virtual environments (see Section 4.2).Finally, we examine the different transitions taking place (see Section 5).

Types of Cross-reality Systems and Their Research Topics
We started analyzing all 306 papers by assigning categories to each paper, following an opencoding approach with all authors involved (e.g., we assigned the category "HMD user transitions into VR" to the following paper [284]).Thereafter, we applied the method of card sorting [279], clustering the identified categories and assigning a research topic to each cluster (e.g., we clustered "HMD user transitions into VR" into the research topic "transitional interface").Then, we grouped the categories within each research topic into additional types to further classify the different papers (e.g., "HMD user transitions into VR" into the type "automatic transition").Here, it is important to note that a paper can be sorted into multiple research topics and types.Finally, we assigned each research topic to one of the three cross-reality systems types defined in Section 2.5.
In the following, we describe the research topics within the three cross-reality system types.

Type 1 (Transitional): Subjects Transitioning on the Continuum Experiencing a Changing Actuality.
For the first type, we identified one research topic as relevant: transitional interfaces.In sum, we identified 48 of 306 papers (15.69%) that investigate Type 1 systems.
Transitional Interfaces.A transitional interface is a system designed to empower users to transition on the Reality-Virtuality Continuum and experience its various manifestations, proposing a new way to interact and collaborate among these manifestations [23,124].An early example is the MagicBook from Billinghurst et al. [29,30].The book can be read in reality, augmented with virtual objects in AR, or used as a companion in immersive VR.With AR-and VR-enabled devices becoming part of everyday life, it is imaginable that transitional interfaces will become ubiquitous.In the past, two different categories have been explored (see Table 1): interfaces controlled by the user (36) and interfaces with an automatic transition (12).
User-controlled transitional interfaces allow users to manage shifts between manifestations.Different form factors of these interfaces have been explored in the past, ranging from headset-(e.g., [55,253,308]), handheld-(e.g., [23,29,58]), and projection-based devices (e.g., CAVEs [268]) to a combination of various form factors (e.g., [124,234,255]).The second type of transitional interfaces allow for an automatic transition between manifestations on the continuum, meaning the user may initiate the transition, but then the interface automatically transitions the user to the target manifestation.So far, the majority of investigated transitions are limited to those between reality and VR, investigating transitions into VR (e.g., [4,280,284]) or out of VR (e.g., [157,277]).Also, some of the investigated automatic transitions involve users who transition to AV (e.g., [37,323]).

Type 2 (Substitutional): Subjects Interacting with Objects That
Are Repurposed for the Subject's Actuality.For the second type of cross-reality systems, we found that 158 of 306 papers (51.63%) are relevant that are distributed over two different research topics: object utilization (124) and collision avoidance (39).In the following, we present each of the research topics in detail.
Object Utilization.The 124 papers that address object utilization investigated users experiencing a concrete manifestation (e.g., VR) in which they lack relevant objects, for example, real-world objects.Important is that these objects are not components specifically designed for being used in VR such as VR controllers.These controllers have no real purpose in the real world because they are only used to interact with the virtual environment.Hence, to fulfill our definition of Type 2 cross-reality systems, we focus on objects that have specific semantics in the real world (or virtual environment) and are repurposed for the user's experience.A typical example of this category is a VR user who wants to use a physical keyboard within the VR environment (cf.[192,306]).In this example, the keyboard is not designed for VR but instead is used to operate a computer in the real world.A counter-example are VR haptic props (cf.[13]).Here, the haptic props are designed to enhance the virtual experience but have no meaning in the real world, similar to VR controllers that exist with the sole purpose of interacting with the virtual environment.In all papers investigating object utilization, real-world entities are integrated into either VR (100) or AR (24).An overview of all these papers and their categorization is shown in Table 2.
The integrated real-world objects include mostly physical objects from the real world-for VR to deliver passive haptics (23), integrate handheld devices (17), or include input devices (12) such as keyboards, mouses, or instruments, or for AR to utilize them (6) or include handheld devices (5).Furthermore, often they utilize parts of the user's environment to create more realistic haptic sensations in VR.Other approaches range from integrating specific real-world objects [38,74,324] to annexing any kind of object automatically [117,275] or with the help of another user [179].A side effect of including physical objects is that users are more aware of their presence and are less likely to bump into them.Besides physical objects, previous work investigated the influence of other more abstract objects such as motion or notifications.Integrating real-world motion empowers users to experience VR in moving vehicles without getting motion sickness [118,193,194,231].In addition, studies have shown that enjoyment and immersion significantly increase with included motion [118].Finally, various studies have investigated how to integrate notifications without negatively affecting immersion [94,122,222,259].This can be accomplished, for example, by seamlessly integrating notifications into the virtual environment as diegetic elements [259].
Collision Avoidance.When users are immersed in virtual environments, obstacles in the real world are no longer visible.In order to solve this problem, various collision avoidance approaches have been explored.While these approaches have mostly investigated VR scenarios, the problem is not exclusive to immersive virtual environments [140,141].Overall, previous work presents three main strategies for avoiding collisions in VR and AR experiences: manipulating the user (16), manipulating the experience (14), or providing warnings that alert users (9).All approaches previously researched and found in our literature review can be seen in Table 3. Unlike warnings, which are designed to gain the user's attention, approaches that manipulate the environment or user often incorporate unnoticeable changes into the experience, empowering users to walk around infinite virtual environments without being aware of it (e.g., [16,65,205,328]).These approaches currently have their limitations (e.g., mainly resulting from the induced illusions that only work to a certain degree), making collision warning approaches useful additions to VR scenarios or alternatives for non-VR scenarios (e.g., auditive warnings [1]).

Type 3 (Multi-user): Multiple Subjects Experiencing Different Actualities.
In total, we found that 133 of the 306 papers (43.46%) investigated Type 3 cross-reality systems.For these papers, we identified the following research topics (in descending order): collaboration (93), bystander inclusion (34), and isolated experiences (11).In the following, we present these topics.
Collaboration.The most frequently researched topic of Type 3 cross-reality systems is collaboration, with a total of 93 publications.Here, collaboration between users experiencing the same manifestation on the Reality-Virtuality Continuum was not included in our literature review (as it does not fulfill the definition of Type 3).Thus, we only included publications involving two or more manifestations on the continuum, so-called asymmetric collaboration [82,285].We identified two types of asymmetric collaboration: remote (67) and co-located collaboration (27).In Table 4, all of these publications are listed in their respective categories.
Compared to co-located collaboration, remote collaboration is the more extensively researched topic, with a share of over 70.1% of all collaboration-related publications.Different remote collaboration approaches have been investigated, with collaboration between VR and AR headset users being the most frequent (24).The reason for this is that expert-novice scenarios are explored frequently, with the expert in VR and the novice on-site in AR.Other approaches typically involve a headset in combination with another form factor.Here, the most frequently used form factors are traditional 2D displays involved in eight of the remote collaboration approaches as well as a handheld device (8), for example, a mobile touchscreen device [225].Besides users experiencing concrete manifestations, transitional interfaces have been explored for collaboration as well.They allow users to switch between augmented and virtual views of one collaborator's space [156] or to use the transition to switch between the spaces of both collaborators [297].Moreover, others have investigated various combinations that involve tabletops [282,283], handhelds [78,84,185,282], or projections [82] to enable remote collaboration.
For co-located collaboration, the most frequent combination of form factors is a VR headset combined with an AR handheld device (e.g., [98,164,215]).Another observable trend is that in most co-located collaboration a VR headset is involved (15).However, compared to remote collaboration, utilizing users that experience different actualities has been explored less frequently, with only 27 publications (29.03%).Some of these papers explore highly unique concepts that are difficult to group with other publications, such as work from Baudisch et al. [24].In this paper, the authors investigate multiple users collaborating in the same real-world space; however, they play with a virtual ball that can only occasionally be perceived.We believe this work is relevant because, while the collaborators experience the same manifestation, the scenario still integrates an object that has a different manifestation.Especially interesting here is that the object exists in virtuality, not reality.
Bystander Inclusion.In many publications, researchers investigated a range of approaches to include bystanders in the MR experience (oftentimes of an HMD user).Unlike collaboration scenarios, the bystander is a real-world person who does not participate in all aspects of the experience, but rather interacts with the user as needed.Overall, we identified 34 of 133 Type 3 cross-reality system publications as relevant (25.56%) to this research topic.These publications can be classified into three different approaches: bystanders contribute to the user's experience without a channel back to themselves-unidirectional (16), the user interacts with a bystander-bidirectional (10), or the user shares their experience with a bystander who does not interact with it-unidirectional (8).In Table 5, all publications researching bystander inclusion are listed with their respective categories.For scenarios in which bystanders are involved in the VR experience, it is always a VR user for whom the bystanders create haptic sensations [43,45] or to whom bystanders are shown [75,209,307].For interaction between bystanders and users, all approaches describe the interaction between a head-mounted VR user and their bystanders, with two approaches being most frequent: using a 2D display that helps bystanders to participate in the experience [132,173,318] or using no technology at all [69,224,346].When sharing an experience with bystanders in two publications, an augmented environment was shared [112,325].A VR user often shares their experience using a CAVE [128,129] or headset display facing bystanders [183,242].
Isolated Experiences.Isolated experiences aim to separate two users on the Virtuality-Reality Continuum as far as possible from each other.In total, we found 11 publications investigating one of two different scenarios: users share the same physical space while at least one is immersed in a specific manifestation of the continuum, for example, VR (10), or users are immersed into a manifestation to escape reality (1).All scenarios are listed in Table 6.In most cases, VR users share the same space and need to be redirected to avoid collisions between them.This is similar to collision avoidance, except that here two users are involved.For user isolation, an interesting idea has been presented by Ruvimova et al. [258].They suggest using VR as a solution to evade a crowded office space.

Summary.
When reflecting on all investigated 306 publications, we identified that different entities are involved in the explored research topics.To describe these entities, we suggest a classification into two groups: subjects and objects.Subjects can be users or bystanders that perceive their environment and can experience different manifestations.Their very own perspective on the scenario depends on these manifestations (e.g., AR or VR) and, therefore, forms their actuality-that what is "currently happening" for them.This can be individual for each subject.In contrast, objects can be various things, such as real-world, physical objects; information (e.g., notifications); or even motion.Essential for the classification as an object is that they do not have a perception of the environment.In the investigated publications, we found all three types of crossreality systems, but with different frequency.It is worth mentioning that a cross-reality system does not have to be limited to one specific type but can be classified as multiple types at the same time (e.g., ARchitect [179], in which users can transition between AR and VR (Type 1), repurpose physical real-world objects for the VR experience (Type 2), and experience different actualities at the same time (Type 3)).In sum, we found 48 publications (15.69%) that investigated Type 1 systems that involve subjects transitioning on the continuum and, thereby, experiencing different actualities.For Type 2 and Type 3, we found 156 (50.98%) and 133 (43.46%) publications, respectively.Both types involve multiple entities, with Type 2 systems including at least one subject and one object, while Type 3 systems involve two or more subjects.Furthermore, during our analysis, we observed that there are similarities between Type 2 and Type 3 cross-reality systems.For both types, there are research topics that aim to increase the distance between the entities on the Reality-Virtuality Continuum, while there are other research topics that investigate how to decrease the distance between different entities on the continuum (see Table 7).For the research topics of collision avoidance and isolated experiences, the entities should repel each other, meaning that the interaction between the entities is decreasing, while in the topics of object utilization, bystander inclusion, and collaboration, the entities should attract each other on the continuum, thereby increasing their interaction.Interestingly, we observed that the majority of publications investigate aspects of entities attracting each other, 251 of 306 (82.57%), while the minority looks at increasing the distance between entities, 50 of 306 (16.34%)-entities that repel each other.Each publication is counted once.Summing numbers from different topics may result in higher totals due to overlapping topics.Publications solely in the topic of transitional interface are excluded.

Combinations of Environments in Cross-reality Systems
Experiences on the Reality-Virtuality Continuum involve different environments.Per definition, these include at least one real environment and one virtual environment between which the continuum spans.They are entangled with each other or otherwise there would not be any influence from one into the other environment.The most simple example is a VR user who experiences some form of digital world but still stands on the real, physical floor.Nevertheless, in a minority of publications, more than two environments are involved (e.g., two VR users in the same physical space who experience different virtual environments [18]).Overall, we found three different environment constellations: scenarios involving one reality and one virtuality (230), scenarios involving multiple real-world environments and one virtuality (67), and scenarios involving multiple virtualities and one real-world environment (9).

Multiple
Real-world Environments.Scenarios of this category involve at least two realworld locations (i.e., different geographical areas) between which physical entities do not move, for example, an expert user joining a novice user from a different real-world location [238].Overall, we identified 67 publications as relevant for this category (21.9%).While reviewing publications involving multiple real-world environments, we found that they mainly address remote collaboration (64), followed by object utilization (6), as the underlying research topics.Object utilization investigated various approaches including the integration of information from the real world, such as notifications or messages (3) [122,189,259].

Multiple Virtual
Environments.We found nine publications involving multiple virtual environments (2.9%).The main research scenario in eight of these publications involved multiple VR users who share the same physical space but not the same virtual experience [18,21,62,64,167,188,240,263].In this case, every user has a distinct actuality that differed from the actualities of the other users.Corresponding publications also focus on avoiding collisions between co-located VR users and assume that these users want to engage solely in their individual experiences.On the contrary, Wang et al. [308] recently proposed a transitional interface that allows a user to view other co-located VR players' experiences.Finally, the number of virtual environments can also be higher than two, for example, if more users are involved and need to share the same physical space [64].

Summary.
We identified the different environment constellations presented in the screened publications.The majority, 75.2%, of the publications investigated scenarios with one real and one virtual environment.When multiple environments are involved, these are often physical locations located apart from each other and are digitally connected mainly for the purpose of collaboration.We also identified publications that aimed for isolated experiences of users with different virtual experiences.Here, these users were located in the same physical space.Hence, the research aimed for providing isolated experiences and those closely related because of an inevitable interaction or influence, avoiding collisions.When multiple virtual environments were deployed, we found that most approaches aimed for providing users with an isolated experience, which aimed for less interaction with co-located users.Along with that, collision avoidance was investigated to reduce the number of encounters with other persons to preserve the isolation.Eventually, we did not find any systems that use multiple real-world and multiple virtual environments.

ANALYZING CHANGING ACTUALITIES IN CROSS-REALITY SYSTEMS
When using a Type 1 system, the actuality of a user changes over time due to a transition along the Reality-Virtuality Continuum.However, numerous systems in the literature are not introduced as cross-reality systems, nor are the transitions highlighted in particular because the presented research did not investigate the cross-reality aspects in itself but, for example, topics like user perception [254] or collision avoidance [1].Therefore, we conducted an in-depth analysis of the literature to find Type 1 systems and corresponding transitions that are not obvious to readers.We identified 118 relevant publications that introduced systems that changed the actualities of its users.Continuing our overview presented in Section 4.1.1,we present our in-depth analysis of these transitions in the following.First, we analyzed the involved manifestations in the described systems (see Section 5.1).Here, we limited ourselves to the distinct manifestation previously introduced: VR, AV, and AR, including transitions involving the Real World (RW).Thereafter, we identify the cause of these transitions (see Section 5.2).Finally, we conclude with a summary (see Section 5.2.9).

Transitions between Manifestations
As seen in Table 8, subjects transition along the Reality-Virtuality Continuum from and to various manifestations.Here, the perception of the transition is dependent on the perspective of a subject-the actuality (e.g., a VR user experiencing VR or a bystander experiencing reality).For example, a bystander could walk by a VR user and is shown to the VR user in the virtual environment when being close [192].The bystander's actuality does not change as the bystander still perceives the RW while crossing the area around the VR user.However, the VR user sees the bystander in the virtual environment; therefore, the VR user's actuality changes with a transition from VR to AV.This is because the virtual environment is augmented with objects from the real world and therefore is no longer purely virtual, in this case, with the bystander.In the following, we introduce the different manifestations involved in the transitions that we found in the literature.

Transitions to Real
World.We found eight (2.61%) publications that involved a transition to the RW.Here, taking a glimpse at a bystander while being in VR results in a transition from VR to the real world [40].This can be useful when immersed VR users want to interact with surrounding persons for a brief moment.To avoid collisions when using AR obstacle detection and accompanying alerts that make users aware of these obstacles forms a transition from AR to the RW [141].When taking the VR-HMD, and thereby transitioning to the RW, users report that they, for example, felt disoriented [157].Therefore, gradual exit procedures could help VR users to exit their virtual experience more comfortably and safely.Likewise, one could use metaphors like a door to the real world to exit virtual experiences [277].

Transitions to Augmented
Reality.We identified 12 (3.95%)publications that investigate switches from the RW to AR. Editing the real world with AR's help can be seen as a transition from a real environment to AR [338].Likewise, overlaying virtual objects onto real ones lets a user transition from RW to AR as soon as the overlays are brought into place [117].Also, sharing content with a bystander can be seen as a transition from the RW to AR [112].Here, the bystander is the transitioning subject.

Transitions to Augmented
Virtuality.Overall, we found 60 (19.74%) publications that involved transitions to AV.The most common transition within the type are publications investigating transitions from VR to AV (54, 17.76%).Bringing in real objects like a cup for drinking, a keyboard for typing [192], or a smartphone [59] when needed depicts a transition from VR to AV.Also, integrating approaching bystanders into the virtual world in order to create awareness or foster interaction results in a transition from pure VR to AV [305] or when actively interacting with them [104].Further, while in VR, partially showing the RW would result in a transition from VR to AV [111].Further, transitions from VR to AV can occur in a non-obvious manner and often rely heavily on the visual sense.For example, for two users that use redirected walking to meet each other to shake hands while being immersed in VR [201], as soon as they are redirected toward each other and shake hands, their VR is externally influenced through the handshake, which is part of the real world.In this case, they transition for a brief moment from VR to AV.Additionally, we found five (1.65%) that investigated transitions from the RW to AV.Here, a bystander could enter a VR user's experience and thereby augment the virtual experiences with their appearance [308].

Transitions to Virtual Reality.
In sum, we found 37 (12.17%)publications that involved transitions to VR.We identified 10 (3.29%) publications that investigate transitions from AR to VR. Users could start in AR and then, for example, decide to transition to VR [254,256], to exchange information between the two manifestations [253], or to collaborate [99].Further, we identified 20 publications (6.58%) involving a transition from RW to VR.For example, Steinicke et al. introduced an approach for transitioning into VR through a portal metaphor.They provided a portal from the real environment to VR to the user.The user could enter the portal to enter the virtual environment [284].Also, it could be shown that a smooth transition into VR helps the user to create awareness of the virtual environment [303].5.1.5Transitions to Multiple Manifestations.We found eight (2.63%) publications that focused on interfaces for transitions along the whole continuum from the RW to AR, then further to AV, and finally to VR.In these scenarios, users transitioned step by step from the real world to the virtual.Each step involved different objects or actions taken by the user [255].

Summary.
We investigated 118 publications that introduce transitions on the continuum and identified involved manifestations.We found that most transitions (54) are from VR to AV, followed by transitions from the real world to VR (20).Some transition categories are underrepresented, like transitions from AR to the RW or from AR to AV.Moreover, the presented transitions can be non-obvious at first (e.g., VR users transitioning to AV when they meet and shake hands [201]).

Causes of Transitions
Transitions on the Reality-Virtuality Continuum can have different causes.We identified several causes for transitions (see Table 9).In the following, we introduce these causes in greater detail.

Substitution of Physical
Object.We found 26 (8.55%) publications that substituted physical objects with virtual ones.For instance, providing a realistic walking experience and at the same time enhancing VR can be accomplished by constantly scanning the real-world environment and adapting the virtual world accordingly to let the user walk in the automatically generated world [44].Here, the user transitions from VR when not adapted to AV when the virtual world is adapted to the surrounding physical environment; in other words, the physical environment is substituted by the virtual environment.Furthermore, real-world objects can be substituted to provide haptic feedback to virtual objects that share similar haptic properties [117].

Change
Actuality.We found 22 (9.62%) publications that introduce transitions on the continuum that are deliberately caused by the user to access virtual objects or to enter a virtual environment.Such transitions can enhance presence [284].For example, when entering a virtual environment, transitioning gradually from the RW to VR makes users feel more presence [137].This can be accomplished by gradually blending out real-world objects and at the same time blending in the virtual environment.Users may also exit VR, which causes a transition from VR to the real world.Here, Knibbe et al. investigated which factors influence transitions out of the virtual experience [157].The results pointed out that the virtual experiences influences the users beyond the point of exit and therefore need further consideration.To exit virtual experiences, metaphors like portals [308] or curtains [161] can be used to indicate the possibility of a transition between VR and the RW.Traversing on the continuum can be accomplished by different user actions or using objects [255].

Bystander Inclusion.
Including bystanders can also be a cause for transitions.We identified 21 (6.91%) publications that investigate transitions caused by bystanders.For example, a transition from the real world to AV can be caused if the bystander enters the tracking space of a VR user [305].Here, the bystander is integrated visually into the virtual environment.A bystander could also cause a transition from the real world to AR when projections are used to give access to the virtual content that an AR user experiences [112].Breaking the VR isolation can be done by enabling bystanders to interact with the VR user [104].Here, the bystander can actively participate in the VR user's activity and influence the virtual environment.In this scenario, the VR users transition from VR to AV when interacting physically with the bystander.From the perspective of the bystanders, they can see floor projections in the RW and can use a display to enter the virtual experience, which also can be seen as a transition from the RW to VR.Other ways to include bystanders into virtual experiences utilize audio to allow for communication between VR users and bystanders [224].

Interaction with Physical
Object.We found that most transitions occur due to interactions with physical objects.Here, we found 19 (6.25%) publications.Interaction with the real world can cause transitions, for example, from VR to AV [192].Users transition when they want to drink or eat something while experiencing VR [37].Further, we found that the usage of an external device causes transitions [59].Users could check a smartphone for messages [3] or using a tablet [125].For using a smartphone, one could capture it in the RW by video.Then, the smartphone can be cropped out of the video feed and presented to the VR user.This augments the VR experience, making it AV.Similarly, using a physical object such as a keyboard in VR constitutes a cause for a transition [266].Here, the VR user is transitioning from VR to AV when using the keyboard.

Collision
Avoidance.We found 10 (3.27%) publications in which obstacle avoidance caused transitions of users.Providing such safety features can cause transitions along the continuum, like creating awareness of obstacles in the VR user's proximity [140,322].Modalities other than the visual were also investigated, e.g., auditive feedback, which lets the user transition out of VR to AV as the virtual environment is augmented with auditive warnings of real-world objects [1].

5.2.6
Collaboration.We found eight (2.61%) publications in which the cause for a transition was the collaboration among users.Often, collaborators transition from AR to VR when creating a collaborative solution [99,156,179].For instance, they shape a maze in AR and then use the created maze to play a game in VR [179].

Providing Haptic
Feedback.We found eight (2.63%) publications that introduced transitions when providing haptic feedback.For example, to enhance typing in VR, one can integrate a physical keyboard [102,159] or smartphone [114].Users also transition when using physical objects around them to mimic the haptics of virtual objects, for example, through haptic retargeting [232].

Interacting Virtual
Object.We identified four (1.32%) publications that introduce transitions that allow for the interaction with virtual objects, for instance, when a real-world environment is scanned and edited in AR [317].Further, a transition can be caused when combining a physical environment with a virtual one [55], or when the real environment is occluded, a user could use a virtual copy of the same to get a better overview [23].

83:19
5.2.9 Summary.We investigated 118 publications that introduce transitions on the continuum and identified their corresponding transition causes.We found that most transitions (26) occurred when physical objects were substituted in virtual experiences, for example, to design virtual environments on the basis of the physical world [275].This is followed by 22 publications that introduced transitions that occurred when there was the need to deliberately change the actuality, for example, when leaving a virtual experience [157,277].The third highest cause of transitions was bystander inclusion into the virtual experience, with 21 publications.Here, bystanders were brought into the virtual experience of, for example, a VR user to create awareness of their presence, thereby making the VR experience an AV experience [305].

NINE GUIDING PRINCIPLES OF CROSS-REALITY SYSTEMS
In an interview study, Ashtari et al. identified eight key barriers that MR creators face today [12].An important barrier noted by the different groups interviewed (i.e., hobbyists, domain experts, and professional designers) is the lack of concrete design guidelines and examples.Therefore, following our previous section that investigated and described current research on cross-reality systems, we continue with the introduction of nine guiding principles for designing and implementing such systems that we derived from our analysis.We categorized the principles according to the three different cross-reality system types introduced in Section 2.5.We grounded our rules in the literature, thereby providing the underlying rationale together with examples of how the rule can benefit the design and implementation of cross-reality systems.

Type 1 (Transitional): Subjects Transitioning on the Continuum
Principle 1: Allow for Smooth Transitions When Changing the User's Actuality.Allowing users to slowly and gradually transition into a target manifestation can benefit their understanding of what is going on.For example, slowly transitioning into VR allows users to keep an awareness of their physical environment [303], improve the sense of body ownership [137], and increase presence [4,284], while slowly transitioning out of VR can mitigate disorientation [157] and should be designed non-interactively [120].A slow and gradual transition can, for example, be implemented by morphing real objects into virtual objects one after another in the target environment [303].
Principle 2: Use Suitable Metaphors to Make Transitions Intelligible and Believable.A possibility to transition should be indicated by a metaphor to help users understand possible actuality changes (e.g., portals [92,244,307,308]).This helps to peek into other manifestations and increases presence [284] and immersion.Also, tokens that allow for a transition can be employed as such metaphors (e.g., books [29,30] or smartphones [92]).Important is that the deployed metaphor communicates its affordance to users.
Principle 3: Give Users Control Over Transitions.Transitions are a powerful technique of crossreality systems as they enable users to change their actuality.However, they can result in severe issues for users if they are deployed wrong (e.g., a system that automatically transitions from AR to VR while the user navigates traffic would put its users at risk).Following the golden rule "support internal locus of control" from Shneiderman et al. [270], designers and developers should consider three primary aspects to give users control over transitions: (1) users can initiate the transition (e.g., by following a metaphor [29,30,55,92,244,308]), (2) users can control the transition (e.g., speed of transition adjusted by the user [103,303]), and (3) if multiple manifestations can be visited, the user should be able to identify and choose the target manifestation (e.g., [29,30,55,103,255,280]).If automatic transitions are deployed, users should understand the transitions' trigger.

Type 2 (Substitutional): Subjects Interacting with Substitutional Objects
Principle 4: Consider Surrounding Physical Objects to Avoid Collisions.Every object physically existing in the user's environment should be considered in the experience to avoid collisions [44,109,140,187,330]. Here, one can either bring over the physical object to the user's current actuality to raise awareness, for example, by substituting physical objects with feasible digital representations [275,302], or one can use solutions that redirect users around the physical obstacles [16,52,121,296].If immersion is not of high importance, designers and developers can also deploy warnings using various modalities to help users avoid collisions (e.g., visual, auditory, or multimodal alerts [1,91,141,195]).
Principle 5: Integrate Relevant Physical Objects to Enrich Experiences.Every object that is relevant to the user should be integrated into the user's experience [192].For example, one can enable users to enjoy a drink or use a keyboard [37,155,159,192,306] or mouse [345] without taking off the VR headset.Here, it is relevant to reduce the mismatch between the real and virtual world by finding a suitable virtual representation of physical objects (e.g., not showing the correct amount of liquid in a glass can result in problems [37]).Furthermore, we consider relevant objects to be more than physical bodies.Objects are also abstract information like notifications [259] or physical phenomena like motion [107].These objects surround us and, thus, influence our perception in various ways.For example, if we experience VR inside a car as a passenger, we need to take the motion into account that is caused by the car driving [118,193,194,231], similarly for VR experienced on board an airplane [321].If physical phenomena are neglected, it can degrade the experience of users.
Principle 6: Provide Opportunities to Interact with Object in Every Possible Actuality.When objects are present in the experience of users, there should be an interaction possibility for these objects [75].Furthermore, if the user's actuality changes throughout the experience, it is valuable to provide interaction possibilities with objects throughout all these actuality changes [29,30,168,255].These interaction possibilities cannot necessarily remain the same across the changed actuality but often require designers/developers to adapt them [208] (e.g., a book that enables transitions changes its appearance in different manifestations [29,30]).

Type 3 Multi-user: Multiple Subjects Experiencing Different Actualities
Principle 7: Allowing for Isolated Experiences.It can be helpful to opt out of a social context, for example, to gain a distraction-free environment for better working conditions [258].If surrounding users should be excluded from the experience of a user (i.e., a mute on social so to speak), one can utilize the different methods provided by collision avoidance research [240] and adapt them while keeping in mind that other users move and are not static.Overall, three different approaches exist: manipulate the experience [167,258,296], manipulate the user [263], and give collision warnings [91,140].
Principle 8: Include Bystanders in Closed Experiences.Experiencing a manifestation of MR in a head-mounted device often excludes bystanders from the experience [14,105].Hence, cross-reality systems should be capable of including bystanders in the HMD user's experience.Depending on the goal, a cross-reality system can bridge the actualities of the HMD user and bystander by either providing a representation of the bystander in the MR experience [31,115,182,192,195,209,240,286,305,307,331] or sharing the MR experience with bystanders [112,131,318,325].Here, allowing bidirectional communication is possible as well and offers the foundation for collaboration [10,40,105,106,181,346].

83:21
Principle 9: Enable Collaborators to Understand Each Other's Actualities.As cross-reality systems enable users with different actualities to collaborate, it is beneficial to communicate these actualities, helping collaborators to understand the individual perspectives involved.Designers and developers of cross-reality systems have three ways to apply this rule: (1) they can allow collaborators to switch into each other's perspectives [179,298], (2) they can allow collaborators to get a glimpse at each other's perspectives (e.g., in the form of portals [307,308]), or they can integrate the elements of each other's perspectives in their own actuality [41,75,297,326].

RESEARCH CHALLENGES AND OPPORTUNITIES
Based on our literature review, it is evident that there has been an uptick in research around crossreality systems (cf. Figure 2).In recent years, we can see a strongly increasing interest in this topic, with larger numbers of actualities involved and a trend toward more dynamic actualities that frequently change over time.Our literature review revealed that it is difficult to identify relevant research, especially Type 1 (Transitional) cross-reality systems, as occurring transitions on the continuum are often not the focus of the work.Thus, they are not prominently described (see Section 7.1).Further, we found that cross-reality systems can become rather complex due to the different perspectives involved (see Section 7.2).Moreover, we identified that current cross-reality systems partially neglect AR devices (see Section 7.3) and a trend toward AV solutions becomes visible (see Section 7.4).To address the increasing complexity of cross-reality systems, we conclude this section by discussing novel prototyping methods of cross-reality systems as an opportunity to make the field more inclusive and allow for quicker iterations (see Section 7.5).

Implicit Transitions
Many of the surveyed papers contain transitions on the continuum, meaning they change users' actuality over time.However, the presented evaluations did not or only vaguely investigated the transition, in particular, cf.[91,183].Often, authors do not explicitly describe the transition that takes place on the continuum, for example, when the underlying research instead focuses on haptic feedback through the inclusion of real-world objects [159,275].Nevertheless, these transitions can be manifold, as they potentially involve multiple actualities and can affect various subjects that interact with the cross-reality system.We refer to these transitions as implicit transitions since they are a byproduct of the proposed system and not the focus of the introduced research.As these implicit transitions between actualities are complex, we found that they are difficult to grasp and hard to articulate.But due to their strong impact, they should be considered.Here, we found that common ground to describe these transitions has not yet been established.As a result, it is tough to extract the transitions' essence, making an evaluation and comparison nontrivial.To make implicit transitions comprehensible and comparable, we recommend investigating visualization methods that enable one to convey the transitions taking place within a cross-reality system.Finally, cross-reality systems often do not investigate the transitions of their proposed systems.For example, research evaluating different approaches to display a physical keyboard in VR assumes the keyboard is always present [159,266].Thereby, these works focus more strongly on interacting with the keyboard in VR but less strongly on the transition between the keyboard being present or not.While it makes sense to focus on interacting with the keyboard, the aspect of how to transition between these states of the keyboard received less attention.

Multiple Actualities
We identified several research topics that involve multiple users and bystanders (cf.Section 4.1.3),which we refer to as Type 3 cross-reality systems.Here, both users and bystanders have different actualities and can transition along the continuum.Thereby, they can change their actuality, resulting in more complex interactions.For example, von Willich et al. introduced a cross-reality system in which from the VR user's perspective, a bystander enters VR and thereby transitions closer to the VR user; however, from the bystander's perspective, there is no transition into VR, meaning the bystander still experiences the real world [305].Thus, all perspectives need to be taken into account as they contribute to an all-encompassing understanding of the scenario.However, it remains challenging to grasp and convey users' and bystanders' perspectives and actualities to an audience that has not experienced the system itself.Again, we recommend investigating visualization methods; nevertheless, we emphasize that such visualizations need to consider the different actualities of the users involved in Type 3 cross-reality systems.

Missing Research on Augmented Reality
We revealed that current research investigations mainly focus on cross-reality systems that shape around VR users.We found only a smaller number of systems that proposed cross-reality experiences with AR users (VR is present in 236 papers, while AR only exists in 111 papers-less than half).We believe that the tendency of immersive VR to blend out the visual information from the real world while auditory or haptic sensations remain perceivable inherently offers more conflict potential, which previous work has aimed to address.Nonetheless, previous work has demonstrated that AR suffers from similar problems-just to a smaller degree [140,141].Still, neglecting these issues can cause severe problems, especially when cross-reality systems are operated in more dangerous environments (e.g., while navigating traffic [136]).Hence, more investigations into head-mounted AR systems are needed, especially as these systems already provide the possibility to communicate more easily with bystanders, but the digital content is hidden similar to VR systems.Novel approaches introduced conceptual solutions to these issues [69].However, especially for crossreality systems that allow users to transition on the continuum, more hardware is required as only very few devices allow transitioning between AR and VR.Currently, these devices are also limited to video see-through AR.

Trend toward Augmented Virtuality
Current VR systems aim for immersive experiences; however, the physical environment of VR users continues to have an impact [187].For example, VR users need to be careful not to bump into bystanders or furniture [192].Thus, in recent years, research has shifted toward cross-reality systems that include parts of the VR users' environment on demand, meaning they temporally or permanently transition users toward AV.In this work, we define such systems as Type 2 crossreality systems (or Type 3 if they include other users).Commercial products have followed this trend, for example, Oculus with the release of its Pass-through API.Thereby, researchers have acknowledged the shortcomings of current VR systems and started embracing the opportunities cross-reality systems do offer.In the future, more research is needed to systematically investigate which aspects of users' real environments need to be introduced to VR experiences and, more importantly, when and how users transition to AV with the goal to incorporate these aspects into their experiences.Finally, integrating real-world objects into the experience requires considering many different objects.If we manage to find computational approaches to integrate them automatically (e.g., [117]), it will enable users to engage with more objects.

Prototyping Cross-reality Systems
Prototyping and developing cross-reality systems is still challenging [214] and can be a timeintensive process that often requires software and hardware prototyping expertise [12].Especially, the creation of cross-reality hardware prototypes (e.g., [44,104,105,192]) has a high entry barrier and requires the use of various hardware components (e.g., displays, projectors, sensors), engineering skills (e.g., electrical engineering, software development), and design expertise (e.g., rapid prototyping).Enabling fast and low-effort prototyping of cross-reality systems could support researchers, developers, and designers of cross-reality systems to quickly iterate their ideas and designs without the need to fully implement the entire system in both software and hardware (e.g., by avoiding a hardware implementation).We argue that more novel prototyping methods are required to help develop cross-reality systems.Recently, Gruenefeld et al. published VRception, a prototyping concept and toolkit that allows for rapid creation of cross-reality systems entirely in VR [103].With this system, multiple users can remotely join one virtual environment.In this environment, they can use various pre-defined virtual components to build cross-reality systems and prototype their functionality in VR.A useful addition to this would be a modular hardware system that allows users to create cross-reality systems with less effort and without the need for extensive software and hardware experience.Such a system could include modular hardware components that can be easily integrated with each other (e.g., small projectors, displays, cameras) and software components that allow for easy integration into virtual environments.Moreover, researchers have proposed various prototyping tools relevant to cross-reality systems [214].For example, they have presented approaches utilizing VR to prototype AR applications [97,166] or to enact futuristic interfaces [272].While these approaches are not directly targeting cross-reality systems, they can still be valuable for the prototyping process of these systems.

GENERAL DISCUSSION
In this section, we discuss the current state of cross-reality system research, thereby answering our guiding question: How can we align the language across communities and establish a solid foundation for future work that benefits both researchers and practitioners?For each extracted research question, we have a dedicated paragraph below that aims at discussing our related findings.
Classification of Cross-reality Systems.The field of cross-reality systems is a relatively young research area.Hence, a well-established terminology is not yet present in the relevant research communities.We argue that it is timely to establish a common terminology as we see an increasing number of publications that introduce cross-reality systems and research.Through our review, we aimed to provide a terminology that allows one to classify cross-reality systems.This can foster research by providing terms that make such systems more comparable or ease the communication of novel ideas.In this context, we argued for the term actuality to describe the current experience of cross-reality system users.Through this term, we can clearly describe what a user is currently experiencing (e.g., the actuality of a user is VR).Further, we introduced a clear distinction between subjects and objects.Subjects are conscious and can perceive their environment, or in other words, they have an actuality.For example, a person in the real world perceives the physical environment; therefore, the actuality for this person is the real world.When the person uses a VR-HMD, the actuality would be VR.To describe cross-reality systems that allow one to transition between different manifestations on the Reality-Virtuality Continuum [200], we introduced Type 1 cross-reality systems.Transitional interfaces [29,30,300] can be classified as Type 1 cross-reality systems as they allow their users to transition between various manifestations (e.g., the real world, AR, VR), thereby changing the actuality of their users.Objects play a key role and, with their utilization, form an important new category within cross-reality systems.We have identified a large number of publications that utilize objects within cross-reality systems (158 out of 306 publications).Therefore, we introduced Type 2 cross-reality systems.These types of systems allow one to repurpose objects, for example, from the real world in virtual experiences [192].Through Type 2 systems, we can describe all systems that integrate objects from another manifestation into the current actuality (e.g., a smartphone into VR [6]).We limit ourselves not only to physical tangible objects.Also, systems that make use of physical phenomena like heat [291] or motion [48,193,194] can be categorized as Type 2 cross-reality systems.To describe systems that involve multiple subjects, each of which experiences different actualities, we introduced Type 3 cross-reality systems.A typical scenario would be users collaborating using AR and VR [220] or bystander inclusion [104,192,305].We argue that this classification allows for structuring the field of cross-reality systems, thereby allowing one to get a better understanding of current trends and even recognize research that is not explicitly introduced as part of the cross-reality domain, for instance, utilizing objects within the user's actuality for haptics [206] or integrating real-world motion into VR [48].We believe that along these types, we can establish useful terminology and guidelines for researchers and practitioners in the area of cross-reality systems.In this sense, we introduced nine guiding principles for the design of cross-reality systems.
Nine Guiding Principles for Cross-reality Systems.As suggested by the literature, there are entry barriers for the development of AR/VR applications [12].At the same time, MR applications are envisioned to become more relevant in the future [278].Through our review, we observed a strong rise in contributions to the field of cross-reality systems, yet we lack guidelines that help to design and implement novel cross-reality systems and experiences.At this time, we strongly believe that it is important to propose a set of rules for cross-reality system design.With our nine guiding principles, we proposed such a fundamental set along our three types of cross-reality systems that are grounded in a large literature corpus.Although these rules may be partly familiar to crossreality experts, formalizing and communicating such a set can benefit the field of cross-reality systems.Novice researchers or practitioners can benefit from years of research distilled into a crisp set of rules that serve as useful guidelines in many practical and educational contexts.The nine guiding principles we have proposed are backed by our extensive literature review.Nevertheless, they are not verified through empirical evaluations.In this sense, future research is necessary to assess their overall applicability.Still, we strongly believe that the rules in their current state form an important starting point for future and well-established guidelines.
Research Challenges and Opportunities.We extracted promising research challenges and opportunities for future work through our literature review.The field of cross-reality systems is manifold, ranging from introducing implicit transitions that were not part of the underlying research question [159] to bystander inclusion that focuses primarily on immersed users and less on bystanders [305].Therefore, little is known about their effects on the corresponding scenario.We see numerous research opportunities here that can help to shape the understanding of cross-reality systems and their effects on all involved users.
Limitations.We acknowledge the following limitations to our survey.We intentionally opted for a manual screening approach to compile our literature corpus because it allowed us to include a larger, more diverse set of publications.On the one hand, this procedure can introduce human error (e.g., overlooking a publication) as our corpus grew substantially in size (overall we screened 33,522 publications).On the other hand, our manual approach allowed for the identification of publications that investigated cross-reality systems but did not use common terminology or present the research as a cross-reality-related evaluation.An automated approach like a database query would have suffered from the same limitations.Hence, we believe that our manual approach led to the compilation of a literature corpus that represents current research in greater detail than an automated one.Further, we compiled the literature corpus starting with HCI-related conferences.Consequently, literature that introduced cross-reality systems in other venues might not be considered in our literature corpus.As this survey approaches cross-reality systems from an interaction perspective, we started with HCI venues.Other venues (e.g., TVCG or SIGGRAPH) often present graphic-focused publications and might lack the interaction part that is of interest to this survey.Nonetheless, through checking references and citing papers iteratively, we identified a huge amount of cross-reality systems published in other venues.Finally, we did not investigate the underlying population of the corresponding user studies in the reviewed papers.Therefore, our survey does not address possible novelty effects introduced by the presented systems.

CONCLUSION
Due to the increasing interest in cross-reality systems, we conducted a scoping literature review, surveying existing publications that propose such systems.Here, we conducted an in-depth literature review by surveying more than 8,437 papers as an initial pool of papers in this domain, from 2015 to 2022.By following their referenced papers and papers that cited them, we surveyed around 25,000 additional papers (as citing or referenced publications).In sum, we identified 306 papers that describe implementations of cross-reality systems (e.g., [137,192,255]).These served as a corpus for classifying their research topics and identifying shared properties.While we see a growing interest in cross-reality systems, we could not identify common terminology.However, to describe cross-reality systems and the aforementioned interplay among different actualities, such terminology should be established.Hence, in our work, we answer the following research question: How can we align the language across communities and establish a solid foundation for future work that benefits both researchers and practitioners?In particular, we contribute a classification of cross-reality systems into three different types: Type 1: Subjects transitioning on the continuum experiencing a changing actuality; Type 2: Subjects interacting with objects that are repurposed for the subject's actuality; and Type 3: Multiple subjects experiencing different actualities.Furthermore, we contribute to a better understanding of these systems by identifying shared properties and providing nine guiding principles that should be followed when implementing these systems.Finally, we conclude our work with research challenges and opportunities that can benefit crossreality systems.Here, we address current shortcomings and propose future research perspectives, including visualization and prototyping methods for these systems.

Definition 3 :
Cross-reality SystemsWe define three types of cross-reality systems: Type 1 (Transitional): Subjects transitioning on the continuum experiencing a changing actuality.Type 2 (Substitutional): Subjects interacting with objects repurposed for the subject's actuality.Type 3 (Multi-user): Multiple subjects experiencing different actualities.

Fig. 2 .
Fig. 2. Descriptive statistics regarding the corpus of literature gathered in our literature review.For the subfigures b and c, we included all venues and authors with at least eight relevant publications.

Table 1 .
Publications Representing Research That Investigates Transitional Interfaces

Table 2 .
Publications Representing Research That Investigates Object Utilization

Table 3 .
Publications Representing Research That Investigates Collision Avoidance

Table 4 .
Publications Representing Research That Investigates Collaboration between Users

Table 5 .
Publications Representing Research That Investigates Bystander Inclusion

Table 6 .
Publications Representing Research That Investigates Isolated Experiences

Table 7 .
Overview of All Research Topics Involving Multiple Entities (Subjects/Objects) and Their Relationship on the Reality-Virtuality Continuum, Covering Both Type 2 and Type 3 Cross-reality Systems

Table 8 .
Transitions of the Subjects along the Reality-Virtuality Continuum Involved Manifestations: Real World (RW ), Augmented Reality (AR), Augmented Virtuality (AV ), and Virtual Reality (V R).

Table 9 .
Transition Causes for Transitions of Subjects along the Reality-Virtuality Continuum