RoboVisAR: Immersive Authoring of Condition-based AR Robot Visualisations

We introduce RoboVisAR, an immersive augmented reality (AR) authoring tool for in-situ robot visualisations. AR robot visualisations, such as the robot's movement path, status, and safety zones, have been shown to benefit human-robot collaboration. However, their creation requires extensive skills in both robotics and AR programming. To address this, RoboVisAR allows users to create custom AR robot visualisations without programming. By recording an example robot behaviour, users can design, combine, and test visualisations in-situ within a mixed reality environment. RoboVisAR currently supports six types of visualisations (Path, Point of Interest, Safety Zone, Robot State, Message, Force/Torque) and four types of conditions for when they are displayed (Robot State, Proximity, Box, Force/Torque). With this tool, users can easily present different visualisations on demand and make them context-aware to avoid visual clutter. An expert user study with three participants suggests that users appreciate the customizability of the visualisations, which could easily be authored in less than ten minutes.


INTRODUCTION
Robots are increasingly used to automate tasks in the manufacturing industry.In particular, collaborative robots (cobots) [29] are gaining popularity in many application scenarios, as they allow human workers to work alongside, without the need for fences and other safety barriers [13].However, studies fnd that cobots are not being utilised to their full potential, as they are often treated merely as uncaged industrial robots [38,39].In order to harness the full potential of cobots and truly achieve the human-robot collaboration (HRC), it is essential to support better communication and convey an understanding of the robot's real-time intentions [38].
In fact, understanding the robot's intentions has gained increasing attention within human-robot interaction (HRI) [44].Researchers have explored various ways to communicate the robot's intent such as using lights [49], anthropomorphic efects [18,21], and haptic feedback [42].Recently, augmented reality (AR) has in particular been explored as a useful technology to communicate the robot's intent and visualise safety information [15,16,23,26,47].With these systems, visualising the robot's intent has been shown to increase user awareness [20], the feeling of safety [20,37], and the user's understanding of the robot behaviour when working close together [36].However, creating these AR robot visualisations requires a high level of software and robotics skills.
In this work, we present RoboVisAR, an immersive AR authoring tool to create in-situ robot visualisations.RoboVisAR allows users to create AR robot visualisations within a mixed reality environment (HoloLens 2) without the need for programming.To author visualisations, users frst record an example of the robot's behaviour, which then serves as the basis to explore diferent AR visualisations.Users can rapidly scroll through the recorded data to see the robot movement on a virtual robot replica.To this the user can add different visualisations and instantly see how they appear throughout the robot program.Once satisfed with the authored visualisations, users can test these AR visualisations by applying them to the physical robot in their workspace.To enable the easy and quick authoring process, RoboVisAR supports six types of common AR robot visualisations: Path, Point of Interest, Robot State, Message, Safety Zone, and Force/Torque.Each type of visualisation has different properties to customise its visual appearance.Furthermore, the system has a custom condition feature, which allows users to defne when these visualisations appear.RoboVisAR supports four types of conditions: Robot State, Proximity, Box, and Force/Torque.By combining these conditions and visualisation types, users can easily author various AR visualisations, such as showing the safety zone when the user approaches the robot, or rendering a movement trajectory path for a certain task.
To evaluate our system, we conducted an expert user study with three participants.Based on a predefned robot program, participants were asked to perform two tasks: 1) create a predefned set of visualisations, and 2) freely design and explore own visualisations.Results from our study show that users can create useful AR robot visualisations within 10 minutes.Furthermore, we found that the immersive authoring approach is more useful for testing visualisations within the context, compared to traditional authoring workfows that require programming environments like Unity3D.
In summary, we contribute with: (1) RoboVisAR, a novel immersive authoring tool to create insitu robot visualisation using real data directly from a robot.(2) A condition-based approach to robot visualisations that allows users to defne when they appear.(3) Results from an expert user study that highlight opportunities for immersive AR robot visualisations.

RELATED WORK
In the following we discuss related work from three areas: (1) AR robot visualisations to support human-robot interaction (HRI), (2) authoring tools within HRI, and (3) general AR authoring tools.

Visualisations to Support Human-Robot Interaction
For efective human-robot interaction, it is essential to comprehend and predict robot's intentions, and researchers have explored various strategies to relay these to users.These methodologies include motion-based cues, such as the robot's movement trajectory [17], visual cues, like a robot's "gaze" direction [40], and auditory or light signals [7,34].More recently, Pascher et al. [44] conducted a review on how to communicate robot motion intent, where AR was found to be a prominent medium.Furthermore, they proposed four diferent types of robot intent that can express a robots intentions to a human co-worker: motion, attention, state, and instruction.Motion intent visualises the future motion of the robot, e,g., as a movement path, attention intent seeks to draw human attention before an action is initiated, state intent communicates internal state, such as the robot's current mode (moving or stopped) or data from sensors, and fnally instruction intent communicates instructions for assisting the robot in continuing its task.Pascher et al. further argue that robot motion intent is a rather vague term, which has not been clearly defned yet [44].
In general, augmented reality has been shown to be a promising technology for communicating robot intent to humans.Suzuki et al. [48] outline fve domains where AR can enhance human-robot interaction: facilitate programming, support real-time control, improve safety, communicate intent, and increase expressiveness [48].For example, Andersen et al. [1] found that projection-based intent visualisations were perceived more positively by participants collaborating with a robot than traditional text-or screen-based interfaces.Rosen et al. [47] investigated AR-based visualisations to communicate robot motion intent for an obstacle avoidance task.Comparing AR visualisations on a head-mounted display (HMD), visualisations on a 2D display, and no-visualisations, they found that AR can reduce the perceived workload and enhance collision predictability.Hietanen et al. [26] explored how projection-based and HMD-based AR could support human-robot collaboration, and found that AR in general can improve the collaborative experience.Other researchers have compared diferent AR visualisations: Gruenefeld et al. [23] compared three visualisations on an HMD, Arevalo Arboleda et al. [3] compared no cues, basic cues, and advances cues for robot teleportation on a HMD, and Hetherington et al. [25] investigated three legibility cues for a moving robot.Tsamis et al. utilised multiple AR visualisations in their system, including safety bubble, robot's path, silhouette preview of the robot, and warning messages [51].Similarly, Andronas et al. combined several diferent AR visualisations for human-robot collaboration tasks [2].Lunding et al. found that users had diferent preferences when combining diferent visualisations in a single system.E.g., while some users found path visualisations useful, others found it redundant as the robot's next task was also visualised [36].
While these studies give insights about how diferent AR intent visualisations perform in certain tasks, it is worth noting that the results might not be universally applicable across diferent contexts [3,23,25].This highlights the importance of developing authoring tools that allow users to easily design AR robot visualisations relevant to their specifc contexts and scenarios.Furthermore, researchers have emphasised the need for creating more intuitive authoring tools to support non-technical users to develop and test diferent visualisations [39,41,48], which also applies to robot intent visualisations.Currently, the design and creation of these visualisations is largely limited to programmers.In this work, we aim to broaden the space for AR robot intent visualisations, by creating an authoring tool to allow non-technical users to participate in the development of these visualisations.

Authoring Tools for Human-Robot Interaction
Within human-robot interaction, authoring tools have primarily focused on supporting non-technical users in robot programming.Existing authoring tools include V.Ra [10], KineticAR [19], GhostAR [9], and Figaro [45].However, most of these tools focus on the robot programming aspect, whereas our focus is on authoring visualisations to help users understand the robot's intent.Similar to our goal, there are a few tools that focus on authoring robot visualisations.For example, RViz (short for "ROS visualisation") is a 3D visualisation and debugging tool, that enables robot programmers to visualise the robot's sensor data and internal state when using the ROS ecosystem [24].In addition, several tools have been made that support visualisations in AR [4,27,28].For creating robot intent visualisations, Unity's Robotics Visualisation Package [50], allows users to create visualisations in Unity based on ROS-messages.However, these tools are embedded in programming environments and they are based on the ROS ecosystem.Hence, users need extensive technical skills to create and use the visualisations.Moreover, these tools do not support immersive authoring within a mixed reality environment, thus, users need to go back and forth between the computer screens and physical environment to test the visualisation.In contrast, RoboVisAR allows users to create in-situ robot intent visualisations within a mixed reality environment through an immersive authoring interface.

Augmented Reality Authoring Tools
In human-computer interaction, there has been an increased focus on augmented reality authoring tools, which enable users to create their own AR applications without programming.For example, Pronto [33], Rapido [32], and 360proto [43] allow designers to prototype new AR applications.The Meta-AR-App [52] and Augmented Math [14] allow teachers to create augmented teaching material for students, and Augmented Reality Scratch [46], ExposAR [35], and StoryMakAR [22] allow children to author their own AR applications.While these tools are a great inspiration for designing authoring tools, they do not work in the context of robot intent, which requires more seamless integration between the robot data and AR visualisations.
A few tools have explored AR authoring to integrate sensors and actuation in the physical world.For example, in CAPturAR [54] and ProGesAR [55], users can prototype context-aware IoT applications by recording the human's movement or by using proxies through AR-based interfaces.In MechARSpace [56], users can author AR enhanced toys with a two-way biding between AR content and physical sensors and actuators from a plug-and-play IoT toolkit.While these tools do not focus on visualisation aspects, there are some that do: in PapARVis Designer [12] users can extend static visualisations in physical books by augmenting them with virtual content.The MIRIA toolkit [8] allows users to create AR visualisations for analysing interaction data in AR.However, these tools only work with pre-existing data.In our work, we focus on creating visualisations based on live data that directly comes from the robot, the human behaviour, and the physical space.

ROBOVISAR
This section introduces RoboVisAR, an immersive authoring tool designed for creating AR-based robot visualisations.RoboVisAR supports a rapid and iterative design process, allowing users to create and combine a diverse set of visualisations.Furthermore, the tool enables users to generate functional visualisations consistently across various robot programs and confgurations.
The authoring process can be divided into the following three steps.(1) Record: capturing an example robot behaviour, (2) Design: creating timeline-based visualisations and modify their appearance with context-aware conditions based on the recorded example, and (3) Play: testing visualisations with the original or a diferent robot program.Consequently, RoboVisAR operates in one of three modes: record mode, design mode, or play mode.The user interface adapts based on the chosen mode, providing assistance tailored to the specifc task.The data source shifts between live-data directly from the robot and other data sources in record mode and play mode, to stored-data of a recorded example in design mode.Design mode ofers the most comprehensive interaction possibilities with the system.
Before users start authoring with RoboVisAR, basic information about the system must be confgured.This includes markers, robot(s), and the name of the project.To switch between the diferent modes, the mode selection interface is used (Figure 2).Project setup is achieved by pressing the Settings button.

Record Mode
Before creating visualisations, RoboVisAR needs example data to initiate the authoring process.This recorded data enables previews of potential visualisations to aid the design of dynamic visualisations.Users can scroll through the recorded example to confrm the desired behaviour (Figure 4), thereby obtaining a more efcient feedback loop compared to operating the actual robot during the design process.
When record mode is activated, the user is asked to start a recording.While recording, the robot executes its program, and the system captures the relevant data.The menu can be interacted with using the opposite hand.
Multiple recordings can be created, such as one showcasing a 'normal' execution of the robot program and another representing an 'error-case'.However, only one recording can be utilised in the design mode at any given moment.The recording interface is displayed in Figure 3.The record menu presented to the user when RoboVisAR is in record mode, which can be achieved using the mode selection interface, Figure 2.
RoboVisAR records live data directly from the robot, the HMD, and potentially other devices or software components.The following data is recorded in a generic format from the robot: robot state, joint angles, tool-centre point, force/torque values, and robot path.Furthermore, the hand and head positions of the user are recorded through the HMD.As RoboVisAR is a standalone application, all data is stored on the HMD.
Once the recording ends, either due to user intervention or automatically when the robot's state changes to stopped, the system transitions to the design interface, as illustrated in Figure 4.

Design Mode
After recording, the user can transition to the design mode to create context-aware visualisations based on the captured behaviour.In this mode, a panel shows a timeline highlighting when specifc visualisations are active within the given example, as depicted in Figure 4. Additionally, this panel ofers options to add, modify, and remove visualisations, to set and adjust conditions, and to generate support objects that can serve as inputs for these conditions.Play/pause button for replaying recorded data with the created visualisations.The slider can be used to navigate to specifc times in the example.On the right side of the menu are (e) editable properties for the selected (Point of Interest) visualisation, and (f) conditions for the selected visualisation.Further conditions, marked with a +-symbol in the upper right corner, can be added.

Exploring the Timeline.
Inspired by other AR authoring tools such as Pronto [33] and Rapido [32] RoboVisAR support a timelineview of the recorded data (see Figure 4).The timeline show which visualisations have been added and when they are active.A slider facilitates navigation to specifc times within the recorded example.Additionally, the replay button allows users to preview the created visualisations on a virtual robot that refects the robot's position during the recording.

Adding Visualisations.
A user can add a new visualisations by clicking on the add visualisation button, as shown in Figure 4. Upon clicking, they are presented with in-situ previews of all available visualisations, based on the time frame of recorded data selected with the timeline slider.Once a selected visualisation is confrmed, it is added to the timeline and becomes available for further refnement, such as changing properties (see Figure 6) and defning conditions as described below.RoboVisAR currently supports the following six types of visualisations, which represent some of the most common approaches used in related work.An example of each can be seen in Figure 5.
• Path visualisation: a line indicates the robot's next movement path.Line width and colour can be modifed.• Point of Interest: a marker highlights a certain location of interest.This could e.g., be where the robot is about to pick or place an object.The user can edit the shape, size, line width, and line colour, and enable a pulsing animation.
• Robot State visualisation: a panel shows the robot's state as "Running", "Paused", or "Stopped".The user can modify the anchor and ofset, i.e, where the panel is placed spatially, and whether the panel should face the user.• Message visualisation: a panel that indicates either a "stepback", "help", or "wait" message.The user can again modify the anchor and ofset, and whether the panel should face the user, as well as the message type.

Creating Conditions.
Visualisation conditions determine when a particular visualisation becomes active and visible.While all visualisations are automatically shown by default, the user can also limit the visibility of visualisations to certain times, to avoid visual clutter.To support this, RoboVisAR can currently track and trigger four types of conditions, which can be customised and combined.A visualisation appears when its corresponding condition is fulflled based on the following: • Robot State Condition: activates based on the robot's operational state ("Running", "Paused", "Stopped").• Proximity Condition: becomes active when two anchor points (described below) come within a certain distance of each other.• Box Condition: activates when an anchor point resides within a specifed area delimited by the box.A box is a modifable cuboid, allowing the user to defne a specifc volumetric area in 3D space.• Force/Torque Condition: becomes active when either the measured force or torque are greater than a threshold specifed for either the x, y, or z-axis or all axes combined.
All conditions have an option to fip when they are active.For example, the Proximity condition can be set to become active when the two anchor points are separated by a distance greater than the predefned threshold (see Figure 6, c).
Beyond controlling the visibility and activation of a visualisation, conditions can also dynamically modify properties of a visualisation when they're active.For example, the colour of a Safety Zone visualisation can change based on the robot state, to be red when running and otherwise green.Or a Point of Interest visualisation can change shape depending on the type of robot action (if the action types are physically separable), which could for example be achieved with a Box condition.

Adding Anchors.
By default, a scene contains several anchors, which are used to establish spatial relationships.All visualisations have an anchor property and the user can redefne which anchor a visualisation is attached to.However, this is only meaningful for some visualisations.For example, a Path visualisation is always anchored to the robot's base and cannot be changed, since that would break the inherent spatial relationship between the robot and the Path visualisation.In contrast, the panel showing the Robot State, which does not have an obvious pre-existing spatial relationship to the robot, can be modifed by the user.
In addition to repositioning visualisations, anchors can also be incorporated in spatial conditions, such as the Proximity and Box conditions described previously.Typical available anchors include the main environment (centred on the QR-code used to synchronise coordinate systems between the HoloLens and the robot), parts of the robot (i.e., robot base, robot tool centre point aka."fngertips"), and body parts of the user (i.e., left hand, right hand, and head).Beyond these, users can specify new anchors.The functionality to add and modify anchors is shown in Figure 4.

Play Mode
Play mode can be used to test the authored visualisations with the real robot, or simply when the authoring process is complete and the visualisations are ready for deployment.In play mode, data is transmitted directly from the robot to the system.
Since conditions are specifed with regards to the context but not tied to a specifc robot program, all visualisations should behave consistently, even if there are modifcations to the robot program.Their behaviour is contingent on events (i.e., data changes) rather than on specifc timelines.
If the user is unsatisfed with the outcome, they can either revert to design mode to adjust and refne existing visualisations, or create a new recording that more aptly captures the context wherein the desired behaviours should manifest.
RoboVisAR is implemented as a framework, such that adding visualisations, conditions, and robots requires little work for the developer.For example, a condition must implement three methods: Active(Frame), IsInitialized(), and Properties().The Active method takes a data-frame, i.e, a collection of all data at time , as input and returns a boolean-value for whether the condition is active.IsInitialized() returns true, if all properties have meaningful values.Properties() returns a list of properties, which can be edited by the user.A new visualisation likewise requires three methods to be implemented and, optionally, to override the appropriate methods for receiving data, e.g., NewRobotState() or NewJointState().

Shared Coordinate System.
As is well known, various systems use diferent conventions for describing poses.Unity3D uses a left-handed, Y-up coordinate system, whereas that in ROS is righthanded, and the coordinate system defned for the UR5e robot is right-handed and Z-up.To ensure consistent representations, we have decided that all coordinates used in RoboVisAR must be converted into the Unity3D format.Thus, all robot adaptors must handle the conversion, if necessary, before publishing messages to shared MQTT-topics.
Furthermore, to align the origins of the coordinate systems for the HMD, the robot, and the workspace, we use QR-codes.These are easy to implement on the HoloLens 2 and we have found that they aford sufciently stable tracking, given a QR Code with sufcient level of detail.

Geting Data into the System (and Recording It). Since Robo-
VisAR depends on data captured from a real robot, it is necessary to establish a connection to the robot.This "robot adaptor" consists of a MQTT communication layer between the HMD and robot.While other options, like ROS, could ofer viable alternatives, we opted for MQTT due to its simplicity and ease of implementation across platforms and programming languages.
We have defned a protocol that specifes how data should be formatted and which topic it is expected to be published to.An implementation of a robot adaptor is not required to support the entire protocol, as this may not be possible.However, this might limit the available visualisations and conditions.While we provide an adaptor for Universal Robots e-series, RoboVisAR will also support KUKA iiwa and KUKA KR120 robots, with the corresponding adaptors.

AR EXPERT REVIEW
To evaluate RoboVisAR (v.0.3), we conducted an expert review study with AR professionals who have experience with HRI.Our goals were to (1) observe and assess how experts use RoboVisAR to create robot visualisations and (2) examine the qualitative usability and utility of our authoring tool.This evaluation strategy falls into the category of "usage evaluation", as described in the HCI toolkit evaluation strategy classifcation [31].
A more controlled experiment to compare RoboVisAR with a baseline seems difcult, as no clear baseline exist.Currently, to our knowledge and experience, robot visualisations are hand-crafted for each project and no authoring tools are available for this purpose.As RoboVisAR introduces a new concept for robot visualisations, we did not test it with novice users, but relied on experts to gain frst insights.We expect that this initial feedback will help highlight the benefts and areas for improvement of RoboVisAR for future iterations.
We recruited three AR professionals (n = 3, zero female), based on their prior experience with human-robot interaction.The participants self-identifed as PhD-student (P1) and software-developers (P2, P3).Their age ranged from 26 to 32 years.

Procedure
Participants were informed about the study purpose and they gave their written consent to participate.They provided background information through a questionnaire, and then received an introduction to the overall study environment and tasks, including information on the robot's built-in safety measures.
The study was split into three consecutive phases: tutorial, fxed task, and free task.These are described in more detail below.The study concluded with a semi-structured interview, in which participants were asked to refect on the experience of creating the robot visualisations, to gather insights about the participants' overall conception and understanding of the interface and the task.The overall study duration was about 75 minutes on average and participants received a small compensation for their time.
4.1.1Task 0: Tutorial.The following shared task between the robot and participant was demonstrated: the robot would pick a DUPLO brick and place it in a small cardboard box, the participant should then close and rotate the box, before pressing a button to confrm that the box was ready for the robot to pickup and place in a larger cardboard box.
After the demonstration, participants started a recording from RoboVisAR, before running the robot program once again.The tutorial then guided them through creating three diferent visualisations (Robot State, Message, and Safety Zone) in design mode.Each visualisation was furthermore styled and a condition was added to each: respectively Robot State condition, Proximity condition, and Box condition.The tutorial concluded by running the robot program once again, this time in play mode, to verify that the created visualisations worked as intended.4.1.3Task 2: Free task.A continuation of the task was demonstrated to the participant: the robot would pick and place two DU-PLO bricks in the large cardboard box, while the participant was instructed to pick and place some as well.The box was then closed and fipped by the participant, before the robot simulated applying tape to seal the box.Participants spent on average 15 minutes freely exploring the opportunities of RoboVisAR to support this task.

Main fndings
Our fndings are based on interviews with the AR experts and can be summarised as: all participants reported that the tool works well and that they like it.In the following we report in more detail on the individual features.
The timeline was described as useful and participants especially appreciated that it shows when visualisations are active.They also valued the ability to scroll through it, e.g., to before a visualisation become active, and get a preview of how the visualisation will turn out.P2 mentioned that it resembles a video editor.
The ability to create visualisations situated directly in the application context was highlighted by all participants.Here, P2 added that it is hard to relate to the real context in e.g., Unity3D, but when using RoboVisAR, you see where the robot is and will be, where your hands are, and so on.
All participants agreed that the conditions make really good sense and they were generally happy with the available options.P2 and P3 mentioned the beneft of styling taking immediate effect, whereas their traditional workfow consists of trying out one variant at a time, with a (minutes-) long build-process in between.Further, each participant proposed some variants of new conditions that could be implemented.P1 suggested a 'robot-is-waiting-forinput'-condition.P2 suggested an 'active-when-hands-are-close'condition, which could be achieved by automatically adding an anchor to all visualisations (new implementation) and then using the Proximity condition.P3 suggested a 'gaze'-condition.
All participants found button presses and the general menu interaction difcult with the HoloLens.Besides issues relating to mid-air interaction with HoloLens, some usability issues where reported: it is not possible to combine conditions with AND and OR (P1, P3), the colour picker is very limited (P1, P2, P3), the selection of anchors would be easier if they had a label (P1), and, fnally, is it not possible to delete anchors (P3).A last improvement requested by P1 and P2 is an extension to the timeline, such that events from recorded data, e.g., when the robot state is changing, could be seen.
Finally, we observed that participants spend on average 9:30 minutes on Task 1: fxed task (P1: 8:30, P2: 7:00, P3: 13:00).It should be noted that P3 had less experience with the HoloLens and spent substantially more time struggling with button clicks for menu interaction.Overall, this suggests that with RoboVisAR users can create four visualisations, change their style, and add a condition to each visualisation within just around 10 minutes.

DISCUSSION
Based on our expert review study, we will discuss condition-based robot visualisations, in-situ authoring of robot visualisations, and, lastly, we will refect on the limitations of our current work and future research directions.

Condition-based robot visualisations
In this work, we propose making the robot visualisations conditionbased, such that they are only presented when needed.This allows users to combine diferent visualisations, while avoiding visual clutter.The idea of making content availability based on conditions has already been widely explored in other domains e.g., for proxemic interaction [5] researchers investigated how users' position and orientation can be used as input to create interactive systems.However, in the feld of HRI this concept has not yet been widely applied.
In most existing work on robot intent and safety visualisations (e.g., [3,23,26,47]) the virtual content is perpetually visible.In other works (e.g., [2,36,51]) diferent visualisations are combined in HRI systems, and Chacko and Kapila even use sensor data to change their visualisations.We go beyond this with our condition-based approach, and allow users to easily explore when diferent visualisations should be presented or how they should change based on the context.Furthermore, our conditions allow the visualisations to be robust to changes in the robot program, such that they can be applied across programs or when the robot behaviour changes dynamically.
The participants in our expert study found that the implemented conditions provide a lot of opportunities.Further, they could imagine additional conditions, such as a 'gaze'-condition, or a 'robot-iswaiting-for-input'-condition to be useful.More work is needed to explore new conditions, or how users may create custom-conditions based on already available data.Another interesting extension to condition-based robot visualisations, are personalised visualisations.One could imagine that a predefned set of visualisations are created by a designer using RoboVisAR, as part of the system implementation.The system could then be extended by the operators, who modify visualisations based explicitly on their personal preferences.A slightly diferent approach would be a condition based on the operator's personal preferences and expertise.The relevance of this becomes apparent, when considering that, in real use, the operator's expertise will change over time.

In-situ authoring of robot visualisations
In our expert review study, participants found that creating visualisations in-situ was a great beneft, as it directly related to the context, which the visualisations was being designed for.This has also been reported by others, as it typically results in a faster feedback loop [30].We expected the in-situ authoring to have concrete advantages, e.g., in picking suitable colours for visualisations, which are often important to make them clearly distinguishable from the environment.Another example is the verifcation of conditions, especially those relying on distance, which RoboVisAR facilitates.
However, the above-mentioned benefts come at some cost.For most, if not all users, interaction is still faster and more fuent with a keyboard and mouse, compared to mid-air menu interaction.An AR interface is typically also more restricted in functionality than a desktop solution would be.However, the use of gaze and gestures opens up new ways of interaction that can be more accurate and may allow more complex inputs.However, customisation to these new forms of interaction is needed, both for users but also interface developers.
The ability to easily author robot visualisations in-situ might also open new areas of exploration.RoboVisAR can make it easier to explore complementary visualisations, i.e, how diferent types of visualisations can efectively be used in combination.This will most likely include investigations into redundancy, i.e, diferent visualisations that convey the same information.For example, we observed in an earlier study [36] that, if a robot's task and path are both visualised separately, one of these visualisations may be perceived as redundant, since they both convey some of the same information (namely the destination of the robot).While it remains to be investigated, whether this is detrimental (e.g., due to increasing visual clutter), or whether redundancy has benefts, a promising approach may be to allow the user to toggle visualisations dynamically.

Limitations and future work
5.3.1 Limitations of the expert review study.While the recruited experts were generally excited about our system, the study had certain limitations.Firstly, the number of participants is obviously low and, consequently, the diversity of our sample is poor.This is mainly due to our requirement of the very specialised background in AR and HRI.Furthermore, the purpose of our study was to evaluate RoboVisAR, but not to investigate task performance or the overall interaction with menus and buttons on the HoloLens [36,53,56].Thus, some prior experience with HoloLens was also desired.Secondly, the experts were presented with predefned robot programs in a lab-based HRI case.While this design made it possible for the experts to be exposed to most features of the system, it would be interesting to explore how the authoring tool would be used in an actual industrial HRI case.However, as participants consistently reported similar experiences, we think our study has its merits as a frst evaluation of RoboVisAR, despite these limitations.

Hardware limitations.
The current prototype is created for the HoloLens 2 HMD, which has some restrictions in both physical properties and performance.The limited feld of view is often mentioned as a limitation for AR-HRI systems [6,23].This should be taken into consideration, when designing visualisations -at least until the current state of HMDs improves.However, extending the display area may pose new challenges, such as visual clutter, if the designers fll the available space with visualisations, without consideration of their contextual relevance.We believe RoboVisAR is a promising tool to address this issue also in future.

5.3.3
Support other robot types.The current system is designed for a particular type of robots, namely a robot arm.Although only one brand of robot has physically been tested, the system can be extended to other brands by creating respective adaptors, as described in 3.4.2.Furthermore, while many of the same visualisations are applicable for other types of robots, such as drones and mobile robots, designing specialised visualisations for these with the current system might introduce new challenges, due to the user's distance to the robots and the size of the space they are moving in.More work is needed there.

Supporting diferent modalities.
In our research, we have focused on creating visual outputs.This naturally has limitations, such as information needing to be in the user's feld of view to be visible, and the critical problem of visual clutter, when the user also needs to see the physical world to safely collaborate with the robot.It would be interesting to explore how visualisations can be complemented by other modalities, such as sound and haptic feedback.One could, for example, imagine that, instead of using a message to express that the robot needs the user's input, the user could be "nudged" through haptic feedback.Future research could explore how these modalities can be incorporated into the authoring process.

Enabling two-way communication.
To foster collaboration between humans and robots, communication should not only fow from the robot to the user, but also the other way around.The already implemented conditions could play a part in such a setup: instead of controlling when a visualisation is presented to the user, they could instead trigger when a signal is sent to the robot.E.g., when the user approaches the robot, a proximity condition can trigger the robot to move at a slower speed, or the user could acknowledge a request from the robot by gazing at a virtual object.

CONCLUSION
In this work, we present RoboVisAR, an immersive authoring tool that enables users to create condition-based AR robot visualisations.RoboVisAR can record live data of a robot, while it is executing its program.The recorded example is then used to design conditionbased AR robot visualisations in a timeline-based editor.The designed and possible additional visualisations can be tested in-situ, as they are applied to a virtual replica of the robot.Adding conditions creates an interactive fow, where visualisations are enabled or disabled based on relevant context.Through an expert review study, we showed that users can author a group of visualisations within just 10 minutes, including initial recording and playing the fnal result.Furthermore, participants found the in-situ authoring highly useful, described the conditions as relevant functionality, and agreed that the timeline provides a good overview of when visualisation are active.Thus, RoboVisAR provides the HRI community with a new way to author condition-based robot visualisations that are directly situated in the intended application scenario.

Figure 2 :
Figure 2: Screenshot of the mode selection interface.The menu is active when the palm of either hand is facing up.The menu can be interacted with using the opposite hand.

Figure 3 :
Figure 3: Screenshot of the Record menu.(a) Is clicked to start/stop a recording.(b) Is an option to automatically record only when the robot is executing its program.(c) provides a list of all recordings from which the user can select.The menu is visible when in record mode (see Figure 2)

Figure 4 :
Figure 4: Screenshot of the Design panel shown in design mode containing: (a) a list of visualisations including the timeline showing when each is active, (b) a button to add new visualisations, (c) controls to create anchor points and new boxes, which are used for some conditions, and (d) aPlay/pause button for replaying recorded data with the created visualisations.The slider can be used to navigate to specifc times in the example.On the right side of the menu are (e) editable properties for the selected (Point of Interest) visualisation, and (f) conditions for the selected visualisation.Further conditions, marked with a +-symbol in the upper right corner, can be added.

•
Safety Zone: a circular line indicates a safety-zone around the robot.The user can modify the line width, line colour and padding (i.e., zone size).• Force/Torque visualisation: a vector indicates the current measured force or torque values.The length of the vector corresponds to the respective measured value.The user can can choose the measurement to visualise, and modify whether each axis (x, y, z) is to be shown individually or combined.

Figure 6 :
Figure 6: (a) Panel to customise the 'width' property of a visualisation, such as a Path and Point of Interest.(b) Panel to edit a Robot State condition.(c) Panel to edit a Proximity condition.The red border indicates that one or more properties need initialisation, as the second anchor is not yet set.

4. 1 . 2
Task 1: Fixed task.The participants were instructed to create four visualisations (Robot State, Path, Point of Interest, and Message) based on the scenario from the tutorial.Each visualisation should have at least one property changed and one meaningful condition.