Investigating the Design of Augmented Narrative Spaces Through Virtual-Real Connections: A Systematic Literature Review

Augmented Reality (AR) is regarded as an innovative storytelling medium that presents novel experiences by layering a virtual narrative space over a real 3D space. However, understanding of how the virtual narrative space and the real space are connected with one another in the design of augmented narrative spaces has been limited. For this, we conducted a systematic literature review of 64 articles featuring AR storytelling applications and systems in HCI, AR, and MR research. We investigated how virtual narrative spaces have been paired, functionalized, placed, and registered in relation to the real spaces they target. Based on these connections, we identified eight dominant types of augmented narrative spaces that are primarily categorized by whether they virtually narrativize reality or realize the virtual narrative. We discuss our findings to propose design recommendations on how virtual-real connections can be incorporated into a more structured approach to AR storytelling.


INTRODUCTION
Hailed as an innovative medium for storytelling, Augmented Reality (AR) enables immersive and interactive narrative experiences by superimposing virtual elements on the real 3D space [11,27].Through AR, the virtual space in which digital narrative content is contained can take on a concrete form by merging with the real space along the Mixed Reality (MR) continuum [87], ofering compelling opportunities to blur the lines between fction and reality [78,93].In achieving this efect, the virtual narrative space and the real space are connected on many levels to constitute an augmented narrative space in the user's immediate surroundings [72,117].
Therefore, it is crucial to understand the structure of the augmented narrative space and how the virtual and the real elements shape it.This establishes the foundation for AR storytelling, both in theory and practice [73,111,112].With the advancement of AR technologies, attempts have been made to conceptualize the ways in which these technologies facilitate relationships between the virtual, the real, and ultimately the user in HCI, AR, and MR research [9,55].While they provide insight into the design of augmented narrative spaces based on how the virtual and real are combined, eforts to apply and further contemplate the concepts to survey and analyze approaches to AR as a spatial storytelling medium within these felds have been lacking.
To address this issue, we conducted a systematic literature review targeting research articles that presented AR storytelling systems and applications in the HCI, AR, and MR domains from 2003 to 2023.For a total of 64 articles, we investigated virtual-real connections in augmented narrative spaces across Pairing, Registration, Function, and Placement Rule, deriving patterns for each of them.Based on the ways that the four virtual-real connections were linked to one another in all cases, we identifed and categorized dominant types of augmented narrative spaces.
The eight dominant types of augmented narrative spaces we found were classifed as either those that virtually narrativize reality or those that realize virtual narratives.By analyzing common traits of cases that comprised each major link between virtualreal connections, we conceptualized each type in the former category as Historicized Outdoor Spaces, Enhanced Indoor Spaces, Extended Object Spaces, and Flexible Backdrop Spaces, and those in the latter as Transported Scene-Based Spaces, Transformed Room-Scale Spaces, Transplanted Tabletop Spaces, and Compatible Portal Spaces.
The contributions of our systematic literature review are threefold: First, we provide a framework of virtual-real connections for a deeper understanding of the spatial and narrative structures of the augmented narrative space.Second, we present a landscape overview of augmented narrative space design within HCI, AR, and MR research, spanning from the earlier to most recent cases.Third, we propose design recommendations to integrate our fndings into a methodical approach that leverages the common characteristics of diferent augmented narrative space types as conventions of a newly emerging medium for storytelling.

BACKGROUND 2.1 AR as a Spatial Storytelling Medium
Amidst mounting expectations for the technological advancement and sociocultural acceptance of AR, discussions on its potential application areas have largely focused on the capacity of AR as a paradigm-shifting medium for immersive and interactive storytelling rooted in space [54,103,124].AR succeeds and builds upon conventions of previous media, such as theater, flm, video games, and VR, which emphasize the spatial arrangement and representation of narratives [107,117].It also transcends their limitations by providing a unique combination of both spatial and narrative characteristics that enable unprecedented possibilities for spatial storytelling [80].
The frst of these characteristics is that the virtual narrative space is housed in the real space.A term established in narratology studies, narrative space is where narrative as the presentation or telling of a story-a sequence of events involving characters-occurs [56,106].In abstract forms of media such as written text, narrative space is often relegated to the background for storytelling [50]: Its representation is left to mental concoctions as the temporal progression of the narrative takes center stage [16].In digital storytelling, however, narrative space becomes a concrete, visible entity integrated into the virtual environment, thereby constituting a virtual narrative space [1,57,104].AR takes one step further in establishing the virtual narrative space as an essential element for storytelling, situating it within the real space that shapes the user's narrative experience [3,5,81].
Second, the virtual narrative space and the real space are connected through various dimensions to create an augmented narrative space.Deriving from Manovich's concept of augmented space, augmented narrative space refers to the real space overlaid with a virtual narrative space in this study [84].AR is in essence the technological means by which the virtual narrative space and the real space are aligned, connected, and merged with one another [10].Furthermore, the features of the two spaces comprising augmented narrative space also determine how mutual relationships between them create efective narrative experiences.Regarding the real 3D space, factors such as physical confgurations, spatial afordances, and meaning or afect as factors of place identity [110] create a sense of placeness [28] that ultimately impacts the virtual narrative space [33].In the virtual narrative space, the purpose and type of the story told, along with its spatial design, are associated with the real space it targets [120].
Third, the user can interact with the augmented narrative space within its real space.AR enables the user to simultaneously perceive and spatially interact with both the real and virtual during a narrative experience.MacIntyre et al. [80] state that the personal nature of AR interfaces induces the user to implicitly interact with the augmented narrative space they are situated in, even when the virtual narrative space itself is non-interactive.In this situation, the user's sense of place is created by the augmented narrative space as a whole.Consequently, how the virtual narrative space and real space are constructed in relation to one another to achieve a "perceived realism [40]" afects various aspects of narrative experience, such as spatial presence, engagement, and agency [118,121,122].
Taking these characteristics into account, we recognize the virtual narrative space, the real space, and the user situated within their spheres as fundamental and interdependent elements in the design of augmented narrative space.Therefore, we aim to investigate the various dimensions on which the virtual narrative space and the real space have been connected to construct an augmented narrative space.Furthermore, we discuss how these connections ultimately condition the spatial and narrative aspects of user experience.

Conceptualizing Virtual-Real Connections in Augmented Narrative Spaces
Many previous works have presented concepts, frameworks, and taxonomies for AR that provide insight into how the virtual space and the real space are connected, some with a specifc focus on storytelling.Here, we review these works with special attention to the spatial and narrative aspects that these connections are based on.
The spatial aspects of virtual-real connections in AR are concerned mainly with how the physical properties of the real space afect the visual manifestation of the virtual space.The extent to which the virtual space and the real space can be paired with one another in materializing as an augmented space is one of these aspects [48,131].This is all the more important for AR storytelling, as a certain pairing determines not only the spatial form but also the type of narrative confgured in a given virtual narrative space, along with the form, scale, and number of real spaces that it can target [44].Shilkrot et al. [117] categorized AR narratives as situated, location-based, and world-level by where the augmented narrative space takes place.While they did not provide precise defnitions for these terms or explain them through virtual-real relationships, these classifcations-location-based in particular-are often used to emphasize how a real space is targeted and paired with a virtual narrative space.
Whereas location-based AR narratives refer to virtual narrative spaces tied to specifc locations, the terms location-independent [31,41] and play-anywhere [49,108,135] are frequently used in contrast for those that can be experienced everywhere.Additionally, adaptive (or space-adaptive) [43,67,121,123] is another term that describes virtual narrative spaces accommodated in various real spaces.However, these concepts are not entirely orthogonal: location-based and adaptive are sometimes used in parallel to convey the idea of augmented narrative spaces that adapt to a specifc location [20,70].As a systematic taxonomy of AR narrative types has not yet been proposed, we aim to inductively derive categories for virtual-real connections through our study.
Another spatial aspect is registration, which is the technological process of aligning the virtual space with the real space [141].Many works have discussed conceptualizations of diferent AR registration methods.In a taxonomy of virtual information representation in AR, Tonnis et al. [129] defned registration as the spatial relationship between virtual and real objects.However, terms that distinguish types of registration were not given.In most approaches, registration is categorized by the types of computer vision and sensor technologies employed, such as marker and markerless tracking, feature tracking, plane detection, SLAM (Simultaneous Localization and Mapping), GPS, and sensor fusion [4,13,17,144].On the other hand, Normand et al. [96] provided a taxonomy based on the degree of freedom as opposed to the type of technology used to focus on the device's position and user.In our study, we also take an approach to registration that is not oriented in technology.We focus on the spatial structures of connections between the virtual and the real that registration methods facilitate to examine their association with the design of augmented narrative spaces.
Regarding narrative aspects of virtual-real connections, the function the virtual narrative space serves with regard to the real space is one issue.For this, Azuma [2,6] put forth three useful concepts: Remembering, Reinforcing, and Reskinning.Remembering uses historical events and memories associated with a location to enrich AR narratives.Reinforcing draws upon qualities presently inherent in the real space to tell stories that complement and amplify them.In contrast to Remembering and Reinforcing, Reskinning employs augmentations to recharacterize reality and make it suit the objectives of the virtual narrative being told.While Azuma introduced these ideas in the narrower context of location-based AR storytelling, it provides a meaningful point of reference for the various ways in which the virtual narrative space can afect the real space it targets.
Lastly, the placement of virtual narrative elements, such as virtual objects and spatially designed scenes as their collective arrangement, is determined by various features of the real space.Murray [92] stated that the immersive power of MR storytelling lies in conventionalizing virtual boundaries in relation to the spatial presentation of reality.Kalawsky et al. [61] stressed the importance of providing efective links between an AR application and the real space it is applied to.These links were specifed as location complexity, content complexity, and semantic relevance by Wither et al. [137].In more recent works that explored placement in AR for sensemaking and storytelling, the physical and semantic features of the real world, such as specifc types of furniture [76] and space syntax [83,123] were considered as crucial factors.How these concepts can be efectively applied to identify meaningful virtual-real connections in AR storytelling applications and systems will be one of the main issues for this study.

METHODOLOGY
In this study, we frst sought to investigate the various ways in which spatial and narrative aspects of the virtual narrative space have been connected to the real space for AR storytelling within the scope of HCI, AR, and MR research.Second, we wished to uncover how these connections can provide the basis for establishing distinct categories of augmented narrative spaces.To achieve these goals, we formulated the following research questions: RQ1.How have virtual narrative spaces been paired, registered, functionalized, and placed with regard to their target real spaces for the design of augmented narrative spaces in HCI, AR, and MR research?RQ2.Based on the ways in which virtual narrative spaces have been connected to their target real spaces, what dominant types of augmented narrative spaces can be identifed in HCI, AR, and MR research?
To answer these questions, we conducted a systematic literature review of AR storytelling applications and systems that have been presented in HCI, AR, and MR research over the last 20 years.

Article Selection
In selecting the articles, we adhered to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement [101], which is a standardized reporting guideline that has been widely adopted.For the given procedure, we determined three terms: 1) the list of research venues most pertinent to the goals and scope of our study; 2) sets of search keywords to fne-comb through databases for the most relevant results; and 3) inclusion/exclusion principles for accurate deliberation.Details on the selection procedure are given in Figure 1.
Our systematic literature review initially targeted academic journals and conferences included in the top 20 Google Scholar ranking for the HCI category, updated as of July 2023, which were CHI, PACMHCI, IMWUT, TAC, IJHCS, BIT, HRI, IJHCI, Springer VR, IUI, IJIMT, DIS, UIST, IEEE VR, THMS, UAIS, TOCHI, HCII, IJCCI, and MTI (listed in order of rank).Regarding venues not included above and specializing in AR and MR research, which are domains not categorized by any metrics-based academic resource, we added IS-MAR, TVCG, and SIGGRAPH on account of their strong relevance to the scope of our study, along with their highly held reputations in academic circles.
Next, we conducted test search trials with various combinations of keywords to cast a wide and dense net that could accurately flter the most ftting collection of articles.We fnalized 12 sets with combinations of the following terms: (Augmented Reality OR Mixed Reality) AND (story OR narrative OR experience OR scenario OR game OR tour).These sets were used to search abstracts of peerreviewed full papers from the chosen venues in the databases of ACM, IEEE, Taylor & Francis, Springer, and Elsevier.This led to the exclusion of MTI from our target venue, which is published by MDPI and ranked 20 on the HCI Google Scholar ranking.We set the search period to include articles from 2003 to 2023, which we deemed suitable to sufciently cover the body of works relevant to our study.Identical search conditions were applied to all fve databases, and the preliminary search results retrieved 1,657 articles from them.After duplicate results from each query and articles that were not full papers were removed, 927 remained.
We then began a manual elimination process conducted in two stages, which were based on three inclusion/exclusion principles given as follows: (3) The application or system in the articles must be demonstrated by a specifc narrative content or experience that satisfes the most widely accepted defnition of a story as epitomized by Jahn [56]: A sequence of events involving characters.
We frst applied these principles to the titles and abstracts of the remaining papers, which resulted in 280 articles eligible for the next and fnal process of perusing the articles' content to determine whether they satisfed all three principles.As the goal was to identify individual cases of augmented narrative spaces for analysis, articles that proposed AR storytelling systems as authoring tools and applications as single narrative experiences were considered equally so long as the articles in question showcased specifc stories that were facilitated by those systems or applications.
To minimize bias and observe objectivity, the frst, second, and third authors contributed equally to the manual elimination process: An article was included or excluded only when a unanimous decision was reached by the authors.To achieve a consensus for each article, conficts were resolved through extensive discussions between the authors.Ultimately, a total of 64 articles were fltered and selected for analysis.The venues and numbers of the fnal set of articles are presented in Figure 2.

Article Analysis
We applied grounded theory to our analysis, which is a qualitative research method that scrutinizes data without predefned frameworks and inductively derives codes for specifc attributes [32,91].Regarding RQ1, the coding of articles was centered on content that provided information on the criteria of connections we selected-Pairing, Registration, Function, and Placement Rule.Following Braun and Clark's six stages of thematic analysis [14], we frst conducted multiple readings of sections detailing the development and implementation of the augmented narrative space presented by the AR storytelling application or system in each article.
Second, we established initial codes through open coding.For this, we listed information concerning our criteria by creating a table for spatial and narrative attributes of the real space (e.g., spatial range, location, characteristic), the virtual narrative space (e.g., spatial range, types of augmentations, scene construction), technology (e.g., device environment, tracking method, system), and user experience (e.g., user role, interaction, movement) that consist the augmented narrative space in each article.In addition, we summarized the narrative content or experience provided, which was based on factors such as genre, character, premise, storyline, narrative setting, and linearity.The frst, second, and third authors coded 30, 19, and 18 articles separately and combined the results to form the initial codes.
Third, we corroborated the initial codes to examine common elements, terms, and descriptions concerning the four criteria, which emerged as potential patterns.Fourth, we cross-reviewed these patterns by engaging in discussion to deliberate those that could best categorize the data for each criterion.Fifth, we fnalized the patterns and named each of them.Lastly, we organized the data to report the patterns for each criterion.For RQ2, patterns assigned to the four virtual-real connections in each augmented narrative space were charted in list form to identify major branches of links between them.This was to examine whether certain categories for Pairing, Registration, Function, and Placement Rule were associated with one another to constitute distinct types of augmented narrative spaces.

RESULTS
In this section, we frst present patterns inductively derived for virtual-real connections in augmented narrative space (RQ1).Patterns for each connection, their defnition, and the corresponding articles are given in Table 1.Based on the links between the virtualreal connections for every case, we analyze eight dominant types of augmented narrative spaces that were identifed (RQ2).

Virtual-Real Connections in Augmented
Narrative Spaces 4.1.1Pairing.For Pairing, which refers to how a given virtual narrative space is spatially matched with the real space it targets, three patterns were identifed: Exclusive, Universal and Conditional.Exclusive pairings were found for 28 of the 64 articles(43.75%),where the augmented narrative space consisted of a given virtual narrative space that can only be connected to a single, designated real space.
A virtual narrative space consisting of an evacuation training game for a specifc building would not be operable anywhere else [18].In 19 Universal pairings (29.69%), a given virtual narrative space could be situated in any real space, so long as its dimensions could physically accommodate the virtual narrative space.Conditional pairings constituted the remaining 17 cases (26.56%).In these cases, a virtual narrative space could be overlaid on an unspecifed number of real spaces, which, so long as the minimum standards of augmentation as stated for a universal pairing are met, must satisfy one or more requirements concerning their spatial attributes.For example, any shipping container can be paired with a virtual narrative space that tells stories related to shipping containers [7].

Registration.
Registration concerns how a given virtual narrative space is attached to the target real space.For this connection, Spatial Mapping, Multiple Anchoring, and Single Anchoring were patterns that emerged.The distinction between registration patterns was made by the spatial form of attachment, which resulted from the application of diferent tracking technologies.Spatial Mapping indicates that the entire spatial range of the virtual narrative space is layered over the target real space.This was the case for eight articles (12.5%) whose systems or applications employed tracking methods that interpret the geometric structure of real spaces, such as SLAM [127].In Harmonize, the target real space is scanned in its entirety to facilitate a real-scale frst-person shooter game [99].Multiple Anchoring refers to the 40 cases (62.5%)where augmentations in the virtual narrative space were intermittently attached to numerous specifed points in the target real space.This was usually achieved through image or marker detection, object tracking, and GPS-based alignment.For example, various landmarks or monuments along the Singapore River connected the virtual narrative space to the whole area [65].The remaining 16 connections (25%) categorized as Single Anchoring employed the same methods to establish a single attachment point within the spatial range of the target real space.In one application, multiple scenes of a story were augmented on a fxed position over a map of an ancient city [114].

Function.
For Function, which pertains to how the narrative content or experience provided by the virtual narrative space affects the target real space, the patterns ultimately followed Azuma's classifcation: Reinforcing, Remembering, and Reskinning [2].We adopted and applied these terms to the patterns we identifed because the codes derived for function were similar to one of these three in their meaning.In 28 cases (43.75%) of Reinforcing, virtual narrative spaces complemented the physical and semantic features presently inherent in their target real spaces.A treasure hunt game augmenting quizzes and tasks on the main attractions of a natural history museum is one example [90].The codes for these cases included keywords such as enhance, strengthen, bring out, emphasize, and contextualize.There were nine cases (14.06%) of Remembering, in which virtual narrative spaces represented past events and memories related to the target real space 1 .Recall, evoke, commemorate, treasure, and historicize were some of the descriptions associated with this function.In one case, scenes from a period TV drama were augmented at real-world locations where the events they portrayed took place in the 16th century [102].
The remaining 27 cases (42.19%) for Reskinning involved virtual narrative spaces that are not associated with features of the target real space in any way.Common descriptors included transform, create, turn into, and alter.For Reskinning, three characteristics could be further found.In most cases, the virtual narrative space reskinned the target real space to immerse users in entirely fabricated stories (fctional).Whereas most of them were games, one where users fght one another at an imaginary battlefeld being an example [99], some adapted parts of known novels or movies [26,36].Furthermore, the target real space became the grounds where practical scenarios for certain situations could be carried out (situational).These cases included training [39,77,138], learning [52,133], and simulation [95] scenarios that are irrelevant to features of their target real spaces.
In other cases, the virtual space brought narrative elements from another real space such as the south pole [68], a museum exhibit [60], or a now-demolished historic site [105] to unrelated target real spaces, placing users in a completely diferent location from their immediate reality (translocative).However, such traits could not be established as independent categories of function, as more than one of them could be applied to some of the cases at the same time.For instance, a virtual narrative space ofering neurofeedback training via a bird-egg collecting game is both situational and fctional [77] in reskinning the real space.Similarly, a virtual narrative space consisting of an exhibition room at a real museum at an unrelated target space to tell the story of a made-up family during the Holocaust is both translocative and fctional [60].For this reason, we fnalized Reskinning as an indivisible category of function.
4.1.4Placement Rule.Placement Rule specifes the type of information the virtual narrative space retrieves from the real space to determine where and how its elements are placed.For this connection, two patterns emerged-Physical Feature and Semantic Relationship.In the 17 cases (26.56%) for Physical Feature, elements of the virtual narrative space were placed according to the geometric features of the real space components they were connected to, such as those of planar surfaces, images, markers, or geographic features such as GPS coordinates.In one case, a 2D image of a game character is augmented with its 3D version based on the visual features [77].In the other 47 cases (73.44%), the Semantic Relationship between elements of the virtual narrative space and the physical features of the real space was the basis on which the former was placed.These relationships are based on the functions, meanings, and afordances of the target real space as a whole and also of individual objects and specifc locations within their spatial range.For example, specifc locations within a university campus identifed through GPS coordinates are connected to virtual content that describes their history and meaning [132].

Types of Augmented Narrative Spaces
Once we identifed patterns for Pairing and Registration as connections between spatial aspects of the virtual narrative space and real space, along with Function and Placement Rule as those between narrative aspects of the two spaces, we linked the patterns found for these four connections in all 64 articles and visualized the data in an alluvial diagram, as shown in Figure 3. Often utilized as a graphical summary of multidimensional data for systematic reviews [15,139], the alluvial diagram depicts prevailing tendencies for virtual-real connections that can collectively defne and distinguish diferent categories of augmented narrative spaces.
The order in which the four connections are linked in the diagram was based on a space-oriented authoring process of AR narratives proposed by Shin and Woo [119,120].For a virtual narrative space, 1) selecting the type of real space to target (Pairing) is frst considered in 2) determining the function it would perform with regard to that target space (Function).Then, 3) narrative elements would be chosen and placed in relation to features of the target real space (Placement Rule) and 4) registered as a whole to lock the virtual narrative space onto the target real space (Registration).This specifc order enabled a more efective visualization of how the four virtual-real connections are considered in the design fow of augmented narrative spaces and what patterns of links between them emerge as a result of that fow.
Consequently, we confrmed eight dominant links between the virtual-real connections.Dominant links were defned as those consisting of more than 5% of the total number of articles.These eight links accounted for 57 cases of the 64 articles reviewed (89.06%).Furthermore, we found that they could be placed in two subsuming categories of augmented narrative spaces: 1) Spaces of Virtually Narrativized Reality and 2) Spaces of Realized Virtual Narratives.The list and number of articles categorized by the types of augmented narrative space are given in Table 2.

Spaces of Virtually Narrativized
Reality.32 articles (50%) comprising four dominant branches of links between virtual-real connections commonly prioritized the target real space in their construction of augmented narrative spaces.In these cases, the virtual narrative space is born out of the target real space, with connections between them all originating from the real: The narrative setting exists within the target real space.Hence, the primary goal of storytelling is to engage the user more deeply with the real space by narrativizing it with the support of augmentations.Within this category, branches are distinguished mainly by the environment and scale of the target real space.

1) Historicized Outdoor Spaces
In the frst type of augmented narrative space that virtually narrativizes reality, the virtual narrative space is exclusively paired with the target real space to tell stories from the past that are specifically associated with it.The placement of its narrative elements is related to the real space and its components on semantic levels, and the space as a whole is registered onto the target space through numerous augmentation points (Exclusive-Remembering-Semantic Relationship-Multiple Anchoring).This type comprised eight out of the 64 cases reviewed (12.5%).In these augmented narrative spaces, stories the virtual narrative space overlays are commonly about the history of the target real space, which may not be readily apparent: On the outset, the real space itself does not directly reveal or contain elements of the virtual narrative space.Instead, it provides the setting for them through specifc features such as landmarks and locations that were once the actual and physical backgrounds of events that occurred in the past.These features, or PoIs (Point of Interests) [128], serve as the anchoring points that augmentations of the virtual narrative space are registered onto.They constitute story nodes that are sequentially ordered within an outdoor area of a singular, socially recognized identity, such as cultural and natural heritage sites [22,29,34,102,132], regional districts (e.g., the Westwood neighborhood in Los Angeles [136] and the Singapore River area [65]), and university campus grounds [58].Therefore, we termed this link between virtual-real connections as Historicized Outdoor Spaces.In all cases for this type, augmented narrative spaces provided tours of the target real spaces through stories that were essentially linear.
2) Enhanced Indoor Spaces In this type, the virtual narrative space establishes a 1:1 pairing with the target real space to narrativize and emphasize characteristics that presently defne it.The placement of narrative elements as augmentations is based on their semantic relevance to the real space and its components, and multiple points within the real space serve as anchors that the virtual narrative space is registered to (Exclusive-Reinforcing-Semantic Relationship-Multiple Anchoring).This was the case for another eight articles (12.5%).The only diference between Historicized Outdoor Spaces and Enhanced Indoor Spaces is their function: the latter reinforces the present state of a real space instead of remembering its past.For the articles we analyzed, however, this diference also led to discrepancies between the categories in the environment of the target real space-outdoors vs. indoors.
Here, the narrative setting is clearly provided by the present confguration of the target real space or by semantic attributes inherent to it.The anchoring points of the real space are objects or specifed locations with unique attributes, essentially forming distinct subareas.They are augmented with narrative elements that elaborate their contexts and are pieced together for a continuous narrative experience with varying degrees of narrative linearity.For this reason, six out of the eight cases targeted exhibition rooms in museums to support, complement, and thereby enhance the content of individual artifacts in a story that binds the whole virtual narrative space to the entire spatial range of the target real space [30,35,45,53,90,115].In the other two cases, the virtual narrative space enabled users to experience training scenarios involving the context of specifc locations and attributes of the real target space [18,85].
3) Extended Object Spaces This type consists of augmented narrative spaces where a virtual narrative space is matched with a single, specifc target real space to reinforce its current attributes.Augmentations bear relevance to the target real space on semantic levels and are registered through a single corresponding point (Exclusive-Reinforcing-Semantic Relationship-Single Anchoring). 10 cases (15.63%) were identifed for this type.Whereas the only diference between Historicized Outdoor Spaces and Enhanced Indoor Spaces is their function, registration is the sole factor distinguishing Enhanced Indoor Spaces from this type.Here, units of the virtual narrative space are overlaid on a single object, regardless of how they are spatially and temporally arranged.
In fve of these cases, the augmented narrative space targeted a 2D image board with unique designs that signifed a specifc space, such as a fctional urban area [125], a driving route [47], a zoo [145], and predefned reference images [74].In three other cases, scenes related to the content of certain books were augmented on their pages [38,42,51].The remaining two cases featured larger physical objects, such as a rhinoceros statue at an art exhibit [64] and a functioning hydrogen fuel tank [134] as target areas of the virtual narrative space.In all these cases, the virtual narrative space was downscaled according to the confned spatial range of the target objects.However, it produced the efect of extending the dimensions of the target real object regarding the physical space it occupied, the meaning it conveyed, and the function it performed.

4) Flexible Backdrop Spaces
In the last type of augmented narrative space prioritizing reality, the virtual narrative space was paired with an unspecifed number of real spaces that satisfed one or more spatial attributes and emphasized those very attributes through the content or experience it provided.The placement of augmentations depended on the semantic relationship between the virtual and real, registered through multiple objects or locations in the real space (Conditional-Reinforcing-Semantic Relationship-Multiple Anchoring).This was the case for six articles (9.37%).By providing more fexibility in determining the target real space, these space type ultimately leverages the virtual narrative space to interact with elements of the real space, incorporating them as signifcant backdrops in the story.
Unlike other types that virtually narrativize reality, these spaces are not distinguishable in terms of their spatial environment or scale: Within this type, target real spaces greatly vary in these aspects, ranging from 2D image boards [140] to life-sized objects [66], rooms [7,88], and outdoor locations [24,75].What sets them apart is that the narrative setting, which is rooted in reality, is less concrete.The characteristics of the anchoring points are viewed more as abstract ideas that loosely infuence the construction and content of the virtual narrative space.Therefore, engagement with the narrative setting in the target real space occurs on a more symbolic or metaphorical level.This results in stories that are more context-driven than context-specifc.For example, standardized simulations of clinical breast examination can be augmented on any real mannequin [66], and an AR learning session delivering general information about a eucalyptus tree can be held at any outdoor location with that tree [24].

Spaces of Realized Virtual
Narratives.In 25 articles (39.06%) that accounted for the other four dominant links between virtualreal connections, the virtual narrative space took precedence over the real space.As opposed to those that narrativize reality, all narrative elements are rooted in the virtual narrative space in these types of augmented narrative spaces.The narrative setting wholly exists in the virtual narrative space or merely borrows props from the real space to enhance its power in reskinning the target real space: Reality is a means through which the virtual narrative space can materialize.The stories are often entirely fctional and are granted greater imaginative freedom, with spatial confgurations of the real world playing a secondary role compared to the virtual elements.As a result, users are immersed more deeply in the manifestation of the virtual within the real than the real itself.For this category, types are mainly distinguished by the dimensions in which the real space is transfgured by the virtual narrative space.

1) Transported Scene-Based Spaces
In these spaces, the virtual narrative space can be paired with any target real space, provided that the minimum physical area needed to augment its spatial range is secured.It serves to transport a narrative experience that bears no relevance to the placeness of the target real space.The placement of its elements is based on the geometry or geography of the real, which correspond to specifc points in reality (Universal-Reskinning-Physical Features-Multiple Anchoring).Six cases (9.37%) were identifed as establishing this particular link.
In four of the cases, all narrative elements, including the setting, were enclosed within the virtual narrative space.The virtual narrative space was then transported to the real space through multiple augmentation points organized as scenes.The augmented narrative spaces facilitated narrative progression through these scenes in varied spatial environments, ranges, and narrative linearity.A miniature Asian garden was augmented onto diferent markers on an empty tabletop space for sequential interactions between virtual 3D characters [94].A fully virtual space was designed for an escape room game that can be implemented in any indoor space at a life-size scale, with various anchors augmenting diferent scenes [143].For a larger spatial range outdoors, an in-car game featuring a virtual 3D character set to diferent GPS locations was implemented [146], and select PoIs of Antarctica were transported to ft a smaller open area [68].In two other cases, scenes out of the novel Wrinkle in Time [26] and the flm Princess Bride [36] were augmented to be reenacted anywhere in reality.
2) Transformed Room-Scale Spaces For this type, there are no restrictions on how the virtual narrative space is paired with the target real space beyond the minimum requirement for augmentation.The augmented narrative space provides a story that is irrelevant to the target real space's features and ultimately reskins it, registered over its whole region by processing the spatial confguration of the designated area (Universal-Reskinning-Physical Feature-Spatial Mapping).Ultimately, the virtual narrative space transforms the target real space into a distinctly diferent entity to the point of erasing its original attributes.This type accounted for six articles (9.37%), which all targeted room-scale indoor spaces.
Five cases presented game-based narratives utilizing the entire spatial range of the target real space at room scale.For example, Ninja on a Plane leveraged the planar surfaces detected in the corner of an unspecifed ofce room to turn the whole area into a ninja's escape route [21].Any room could become a galaxy of planets to be conquered in Startag [97], a battlefeld in Harmonize [99], a language-learning space in Arigato [133], or a coin-collecting game site [89].SceneAR [23], which is the one other case for this type, detects the surfaces of any target indoor space to spatially arrange a sequence of virtual narrative scenes over the entire spatial range of the target real space, ultimately transforming it into a setting for the virtual narrative space.
3) Transplanted Tabletop Spaces Transplanted Tabletop Spaces are diferent from Transported Scene-Based Spaces and Transformed Room-Scale Spaces in their registration only: the augmented narrative spaces for this type register virtual narrative spaces to the real target space through a single marker, image, or location, which is not associated with the story in any meaningful way (Universal-Reskinning-Physical Feature-Single Anchoring).We identifed four articles (6.3%) for this type.Compared to Transformed Room-Scaled Spaces, where a real room with specifc semantic features is given an entirely diferent function, the spaces in this category select a confned area within a real space as a purely geometrical entity with no preassigned meaning.Subsequently, a virtual narrative space entirely disconnected from the physical space is directly transplanted onto this semantically vacant area.
Cases that fall under this type all presented augmented narrative spaces that target empty tabletop spaces as their real spatial range.The virtual narrative spaces augmented on them commonly provided content for interactive board games that users engage in through virtual agents at a miniature scale.Eggly [77] is an educational game for children with autism set at a farm, and Le-gionARius [105] is a strategy game based on ancient Roman military camps and Germanic settlements.These games can be augmented on any space that fts the physical dimensions of the entirely virtual game boards.Woll et al. [138] features a fctional space with an astronaut under alien attack in a completely virtual board game setting.Sekhava et al. [116] augmented a virtual Zigorat (a maze tower) on a board-game-sized surface, which a ladybug character must climb to release her friends.

4) Compatible Portal Spaces
In the last type of realized virtual narratives, virtual narrative spaces are paired with real spaces with one or more specifed attributes and serve to reskin them.The placement of virtual elements is based on semantic features of the real target space, and these elements are augmented on multiple points within the real spatial range (Conditional-Reskinning-Semantic Relationship-Multiple Anchoring).Nine cases (14.06%) were found for this particular link.In all these cases, the real objects and locations to which the virtual narrative space was registered were interchangeable with others that shared the specifed attributes.Moreover, these elements functioned as portals through which a virtual narrative space bearing no relevance to the target real space could spatially manifest.
Six cases required certain physical objects to be placed in the real space to open up reskinning narrative spaces: BubbleU [63] augmented narrative elements on nine predefned objects, and certain image cards were needed in an AR game for children that could be played on any tabletop space [52].Be Our Guest [113] specifed that the object leading to virtual spaces narrating various tea-drinking cultures should be a cup, and a story of the Holocaust [60] needed the same type of furniture to be set up in the target real space as its original setting at a museum exhibit.Similarly, a room with a table and chairs were the stated conditions of an AR narrative that changes such real spaces into a setting for Three Angry Men, an interactive play [79].Another case replaced detected objects in a real-world scene with other virtual characters through augmentations to create fctional stories [62].In the other three cases, portals into the virtual narrative space for outdoor environments were specifed, such as any location that resembles the narrative setting of scenes from the movie Nosferatu [37] or any theme park attraction queue for an AR game ofering spatial and temporal escape [142].Li et al.'s location-aware adaptation method for virtual narrative spaces set predefned parameters such as location compatibility and walking distance in a given real space [70].

DISCUSSION
In this analysis, we examine the results to understand how the spatial and narrative aspects of the virtual-real connections identifed (RQ1) are associated, resulting in the eight dominant types of augmented narrative spaces within two higher-level categories: Spaces of Virtually Narrativized Reality and Realized Virtual Narratives (RQ2).We then investigate how these types facilitate various modes of spatial and narrative experiences.Based on our analysis, we present design recommendations to leverage these virtual-real connections and types of augmented narrative spaces to establish a more systematic approach to AR storytelling.Additionally, we acknowledge the limitations of our approach.

Analysis of the Results
5.1.1Spatial and narrative associations between virtual-real connections.The distribution of patterns for Pairing (Exclusive, Universal, Conditional) and Function (Reinforcing, Remembering, Reskinning), along with the links between them, shows that there are clear associations between how the virtual narrative space and the real space initially fnd one another to establish the main objective of the augmented narrative space they comprise.For pairing, exclusive connections accounted for the most number of cases (28, 43.75%).This indicates that the target real space has largely been designated and fxed to a specifc, meaningful location or area for the design of augmented narrative spaces.
In these cases, the virtual narrative spaces functioned either to reinforce or remember the real space, with considerably more cases for the former (19).For all but 2 cases among the 19 Universal pairings, the virtual reskinned the real.In the remaining 17 cases of Conditional pairings, the virtual narrative space's function was distributed among Reskinning (10) and Reinforcing (7).This means that depending on the extent to which a target space is specifed, the narrative content confguring the virtual is determined: the more exclusive the spatial pairing, the more bounded the story's setting and premise are to the target real space.
Then, the problem is how elements of stories in the virtual narrative space, as determined by Pairing and Function, are realized through patterns found in Placement Rule (Physical Feature and Semantic Relationship) and Registration (Spatial Mapping, Multiple Anchoring, Single Anchoring) to implement the augmented narrative space as a spatialized narrative experience in reality.For all but two cases, Exclusive or Conditional pairings augmented narrative elements of the virtual based on their Semantic Relationship with the target real space, which was the majority of cases (47, 73.44%).None of these augmented narrative spaces were registered to the real space through Spatial Mapping: For the articles reviewed, semantic associations were mostly recognized through specifc points of features in the real space instead of a holistic understanding of its spatial structure.
In the other 17 cases relying on Physical Features for placement, 16 of which were Universal pairings, registration patterns showed an even distribution overall among Multiple Anchoring (6), Single Anchoring (4), and Spatial Mapping (7).This indicates that Placement Rule and Registration are connections that are also impacted by the specifcity of the target real space and the narrative content it dictates: Whereas most Exclusive and some Conditional pairings establish meaningful points of connections for stories bound to the target space, Universal pairings generally entail elements of more free-ranging narratives to adapt to the physical features of the real space in various ways.Ultimately, these associations between the four virtual-real connections are the basis on which the eight dominant types of augmented narrative space can be discussed.
5.1.2Spatial and narrative experiences of diferent augmented narrative space types.Space of Virtually Narrativized Reality outnumbered those of Realized Virtual Narratives by seven cases.This owes primarily to the prevalence of Exclusive pairings, which is one of the defning features of augmented narrative spaces prioritizing the real.Furthermore, three out of four types with the most number of cases belonged to Spaces of Virtually Narrativized Reality: Extended Object Spaces (10), Historicized Outdoor Spaces (8), and Enhanced Indoor Spaces (8).Among Spaces of Realized Virtual Narratives, the highest number of cases were Compatible Portal Spaces (9).On the other hand, the remaining four types-Flexible Backdrop Spaces (6), Transported Scene-based Spaces (6), Transformed Room-scaled Spaces (6), and Transplanted Tabletop Spaces (4)-did not difer greatly in number.
These augmented narrative space types illustrate the major ways in which newly developing technologies and ideas for spatial narrative experiences in AR have been implemented in HCI, AR, and MR research.Through Extended Object Spaces, AR storytelling has aimed to narrativize a limited area contained within the user's immediate surroundings through specifc, tangible objects.As the physical dimension of the augmented narrative space is not large enough to situate the user in it, the user takes a fxed position outside its bounds to become a part of an interactive narrative through physical or virtual agents in miniature or to observe a linear story's progression.An example of the former is ChildAR [47], which presents an augmented narrative space where users drive along diferent routes by moving an augmented miniature car on a tabletop gameboard space.In one of the latter cases, a book on Korean temple bells is augmented with their history, and the user engages with the narrative outside of it through interactions that do not afect the narrative being told [42].
Conversely, Historicized Outdoor Spaces and Enhanced Indoor Spaces envelope and physically move the user along a spatially arranged narrative in various trajectories [8] and have mostly been developed for the cultural heritage domain.They focused on enlivening narratives inherent in heritage sites and museum exhibits.The Svevo Tour takes the user to various locations in the Italian city of Trieste through a narrative centered on the novelist Italo Svevo [34], and Hammady et al. 's MuseumEye MR application implemented an augmented narrative space for the Tutankhamun exhibit at Cairo's Egyptian Museum with three storytelling scenes [45].Compared with the three aforementioned space types, Flexible Backdrop Spaces virtually narrativize reality more indirectly and metaphorically.Consequently, the context provided by the real space, rather than its unique placeness, is embellished when reality is brought to the fore as the narrative setting.For instance, LINA engages the sociocultural context embedded in elementary school classrooms for an educational mystery-solving game that centers on a missing classmate.It can be utilized in classrooms anywhere to reinforce their original function [88].
Turning to Spaces of Realized Virtual Narratives, Compatible Portal Spaces, the type with the second highest number of cases, leverage certain traits that can be found across various real spaces as portals into virtual narrative spaces.This allows users to evade their surroundings while being situated in them.As long as there are some chairs and a table in any real space, a completely fctional story of a jury's confict that is disassociated with the target space can be integrated into the reality that the user is inhabiting [79].Unlike the three other augmented narrative space types in the same category (Transported Scene-Based Spaces, Transformed Room-Scale Spaces, and Tranplanted Tabletop Spaces), Compatible Portal Spaces establish a semantic relationship between narrative elements of the virtual and features of the real: augmented characters sit on the real chairs around the real table in the above example.This indicates that together with Spaces of Virtually Narrativized Reality, this type represents a major format that employs AR narratives for specifed real spaces.
For the three other space types of Realized Virtual Narratives, the distinction between them is made by how each type registers the virtual narrative space to the target space, as other connections are the same.This leads to diferences in the spatial scale of the narratives being realized to some extent, which is in turn associated with the type of stories being told.Spatial Mapping for Transformed Room-Scale Spaces and Single Anchoring for Transplanted Tabletop Spaces not only give the space types their names but also confgure how the diferent spatial scales are employed within the common narrative genre they represent.Almost all cases in the two space types are games except for SceneAR, which presents a linear story of bees arguing over fowers [23].The fve other cases of Transformed Room-Scale Spaces are frst-person role-playing games where users themselves conduct gaming actions at a real scale, experiencing the space through various perspectives and modes of input.In Star Tag, the user is encouraged to move quickly and uses voice and gestures to approach and conquer virtual planets dispersed in space [97].
On the other hand, the spatial dimensions of the four Transplanted Tabletop Spaces dictate that they all take on the form of board games, which limits the user's spatial interactions to a mobile interface.In terms of their story worlds, both space types tend to represent those that are entirely removed from reality itself and not just their target spaces.For Transported Scene-Based Spaces, which vary in their spatial scale, Multiple Anchoring as the registration method sets the space type apart from Transformed Room-Scale Spaces and Transplanted Tabletop Spaces in that the scenes anchored to various points in the target real space elicit linear narratives as opposed to games.Users experience narrative progression by encountering diferent scenes remediated within the real space.This means that the augmented narrative space integrates existing multimedia resources within a newly devised narrative sequence [12].For example, IFiction [26] and ARKaraoke [36] take a select number of scenes from the novel Wrinkle in Time and the flm Princess Bride respectively out of their original context and position them in diferent locations to fctionally reskin the real space.These scenes ofer varying levels of immersion according to their spatial scale, which varies within allowing users to occupy the virtual narrative space while moving through the physical environment.

Strategize based on priority-the virtual or the real?
In designing augmented narrative spaces, the foremost step should be to determine whether it is the virtual narrative space or the real space that plays a more central role in its dual composition, i.e., which of the two layers should prevail over the other.Depending on this decision, how the two spatial entities are connected falls into place.If the virtual should be more prominent, stories of creative versatility can be adapted to a diverse range of real spaces via their physical features to showcase the transformative power of the virtual narrative space.If the focus is instead on bringing a certain real space to the fore, the virtual narrative space can recede to the background through a story that illuminates the hidden or apparent features of the real space.This can be achieved by connecting the virtual narrative space to specifc and meaningful areas, locations, or objects.In essence, the choice between real and virtual dominance in an augmented narrative space should dictate the main objective of the user experience it provides and build the foundation upon which ensuing design decisions are made.5.2.2Consider narrative tropes against augmented narrative space types.Once the main objective for the design of an augmented narrative space is set, outlining the specifcs of the narrative itself through parameters such as genre, character, sequence, and the degree of linearity to constitute certain narrative tropes for that objective is required before it can be spatialized in the virtual and the real.At this stage, features that characterize dominant types of augmented narrative spaces through links between virtual-real connections can be utilized as conventions of AR storytelling.This is to say that certain augmented narrative space types can be referred to and refected in creating certain narrative tropes.For example, suppose the goal is to realize the virtual narrative space through an immersive and interactive indoor experience set in a fantasy world.In this case, the setting is consistent with Transformed Room-Scale Spaces and can be further developed by following or playing with conventions for this type.The genre of the narrative would typically be a game that is highly interactive and non-linear, with users participating in the narrative as characters with the agency to trigger the sequence of events.

5.2.3
Balance the virtual and the real through spatial user interaction.When details of the narrative are confrmed, the ensuing design choice mainly involves spatializing narrative elements in the virtual narrative space with regard to the target real space.Our analysis revealed that according to an augmented narrative space's main objective and link between virtual-real connections, the composition of virtual and real elements difered in their amount and presence: dominantly virtual spaces are more noticeably virtual, and real ones more real.However, it is crucial for users to perceive a balanced integration of the virtual and the real to enhance their immersion during an AR narrative experience [122], either to suspend their disbelief in virtuality [86,121] or engage themselves better in reality [130].To mitigate the inevitable imbalance between the virtual and the real in certain types of augmented narrative spaces, we recommend facilitating user interaction with the less dominant elements as part of the design strategy.For instance, enabling users to manipulate real objects to trigger virtual narrative sequences, placed to encourage active movement in the real space of a Transformative Room-Scale Space that prioritizes virtuality, would be benefcial for narrative experience by ensuring that users engage with both the virtual and the real.

Limitations
There are some limitations that should be addressed in subsequent studies.First, our systematic literature review primarily targeted research articles in the HCI domain, with a few additions of venues representing the most relevant works on AR and MR.This was an intentional decision made to investigate how the design of augmented narrative spaces has been understood and practiced in academic spheres.However, it led to the exclusion of commercial cases that have greatly contributed to the advancement and deployment of AR storytelling.Ingress [25,82], Pokémon Go [100,109], and Fragments [59,121] are some of the most notable examples.Considering that these cases present augmented narrative spaces that defy the scope of those identifed in our current review in some aspects, analyses of virtual-real connections in them will provide deeper insight into this issue.
Second, we employed a selective number of virtual-real connections to be identifed and utilized in defning types of augmented narrative spaces.This served as a practical and efective framework that encompasses both spatial and narrative aspects of relationships between the virtual narrative space and the real space as two essential components of the augmented narrative space.Nevertheless, they should be considered alongside other traits, such as those that involve the user's spatial conduct and narrative experience as another meaningful element that ultimately establishes the augmented narrative space as grounds for an embodied experience [19,98].This may lead to a more comprehensive understanding of the holistic design principles for AR storytelling.

CONCLUSION AND FUTURE WORK
In this work, we conducted a systematic literature review of 64 articles that presented AR storytelling systems or applications in HCI, AR, and MR research over the last 20 years.We inductively derived patterns for Pairing, Registration, Function, and Placement Rule as connections between the virtual and the real in the design of augmented narrative spaces.Furthermore, we examined links between the virtual-real connections to identify eight dominant types of augmented narrative spaces.Through a comparative analysis of each type's characteristics, we conclude that the design of augmented narrative space is fundamentally shaped by how connections between the virtual and the real defne, support, and leverage the degree of dominance of one spatial entity over the other.
In the future, we plan to expand the scope of our review to a wider range of augmented narrative spaces to include commercial applications as well and analyze them through a more rigorous framework.In addition, we intend to integrate our fndings on virtual-real connections and augmented narrative space types into the refnement and enhancement of AR storytelling systems.Ultimately, our objective is to glean more detailed and in-depth insights for a more systematic approach to the design of augmented narrative spaces that can result in enhanced user experiences for them.

Figure 2 :
Figure 2: The venues and numbers of the fnal selected articles

Figure 3 :
Figure 3: An alluvial diagram depicting links between patterns of four virtual-real connections in augmented narrative space-Pairing, Function, Placement Rule, and Registration-for the 64 articles reviewed

Table 2 :
Types of augmented narrative spaces defned by eight dominant links between real-virtual connections