Unlocking Potentials of Virtual Reality as a Research Tool in Human-Robot Interaction: A Wizard-of-Oz Approach

Wizard-of-Oz (WoZ) systems represent a widespread method in HRI research. While they are cost-effective, flexible and are often preferred over developing autonomous dialogs in experimental settings, they are typically tailored to specific use cases. In addition, WoZ systems are mainly used in lab studies that deviate from real world scenarios. Here, virtual reality (VR) can be used to immerse the user in a real world interaction scenario with robots. This article highlights the necessity for a modularized and customizable WoZ system, using the benefits of VR. The proposed system integrates well-established features like speech and gesture control, while expanding functionality to encompass a data dashboard and dynamic robot navigation using VR technology. The discussion emphasizes the importance of developing technical systems, like the WoZ system, in a modularized and customizable way, particularly for non-technical researchers. Overcoming usability hurdles is crucial to establishing this tool's role in the HRI research field.


INTRODUCTION
Robots become more present in our daily lives, particularly in public spaces such as libraries, public transport, and museums, where they serve complementary functions [9,19,21].This trend has led to the identifcation of numerous application scenarios for robots, each presenting unique challenges and requirements for humanrobot interactions [5,6].These challenges may include the need for case-specifc responses from robots or the utilization of diverse communication channels.Addressing such challenges in an autonomous system can be complex, afecting the seamless fow of interaction.Consequently, a common approach to studying these interactions in Human-Robot Interaction (HRI) is through "Wizardof-Oz" (WoZ) experiments [3,18].This method has proven to be more cost-efective [20], fexible [10] and is often preferred over the development of dialogs, as technical systems do not have to operate autonomously [14].In WoZ experiments, a human operator, known as "the Wizard," remotely controls the robot while monitoring aspects such as its movements and its speech.Importantly, individuals interacting with the robot are not aware that a human is orchestrating its actions.
While the WoZ method is useful for projects in HRI, the technical systems for studies are typically customized for specifc settings.Adapting or reprogramming these systems for alternative studies entails considerable programming efort.This situation needs to be critically reconsidered, as diverse application scenarios have case-specifc infuences on robot perception [2].Specialized WoZ systems not only diminishes the possibility to use it for other application scenarios but also prevents non-technical researchers from autonomously conducting studies tailored to their specifc needs.It consequently represents a hurdle in the generation of research results.We suggest that, lowering this barrier by making it more easy, time saving and efcient (even for technically skilled people), a modularized and customizable WoZ system that reliably facilitates case-specifc interactions within diverse scenarios can be benefcial for HRI research.
Nevertheless, using a customized WoZ system in specifc environments, particularly public study settings, might be difcult to use.Those environments pose unique challenges, such as regulatory constraints or challenging controllability, as the risk of external factors, like auditory and visual stimuli could possibly infuence participants' perceptions of the robot.Virtual reality has become an increasingly accepted solution in recent years to deal with these limitations and problems.Our approach combines both, a classical (but modularized and customizable and thus fexible) WoZ with the benefts of VR.The reasons of employing VR are threefold.Firstly, it enables the recreation of real environments in an authentic way, facilitating the simulation of diverse use cases [16].This approach makes it possible to use environments for studies without having to be physically present.Additionally, virtual recreations of real environments can be manipulated as required for specifc study settings [1].Virtual representations of physical robots can also be used in VR for research purposes, which, besides other advantages, facilitates access for scientists who do not have physical robots at their disposal (e.g.young researchers, institutions lacking money).Secondly, VR stands out for its cost-efectiveness, location-independent usability, and boundless creative possibilities in shaping virtual environments and objects, such as robots [11,15].This fexibility makes VR a dynamic research tool adaptable to a spectrum of requirements for studies.Thirdly, the use of VR signifcantly enhances observability and controllability compared to (physical) real-world environments [4,7,13].These advantages not only contribute to the reproducibility of studies but also guarantee ongoing and precise data collection through diverse measurement techniques, including motion tracking, eye-tracking, and physiological data collection.
The idea to combine a WoZ approach with a VR solution originates in needs of the interdisciplinary robotics competence center RuhrBots, in which social robots are to be researched in public spaces.As one example of a public space allowing access to all kinds of people potentially interacting with robots, a public library was designed in the VR environment.A sophisticated and prototypical implementation was developed to meet the requirements of feld studies for the RuhrBots project, such as observing human behavior in response to specifc robot utterances, which may be challenging to address in real-world environments.The project aims not only to use the benefts of VR for HRI research but also to establish a platform for conducting case-specifc studies in various public spaces (e.g.citizen service, museums).

RELATED WORK
Although the "Wizard-of-Oz" method was originally employed as a natural language processing system [8], today the paradigm is also widely used in Human-Robot Interaction.This is primarily because by using a WoZ setup, the functionalities of robots can be simulated autonomously, saving development time and costs.This possibility culminates in various research approaches, such as for natural language processing, non-verbal behavior, and navigation scenarios, as outlined by Riek in his review [17].In this context, Rietz et al. have attempted to create a WoZ tool for various use cases with "WoZ4U" [18].While this represents a novel approach to generalize the functions of WoZ systems, it falls short in handling case-specifc infuences of various application scenarios in HRI.
With regard to Wizard-of-Oz systems in VR for HRI, there are initial approaches that utilize the functions and possibilities of VR technologies.For example, Tran et al. developed a teleoperation interface for WoZ systems [22].By using a VR headset and its controller tracking, it was possible to transfer various gestures to the Pepper robot in real time through physical movements of the wizard.Miniota et al. also explored the teleoperation aspect of social robots [12].They compared the approach of a VR-WoZ with that of a traditional WoZ without VR in a conversational task with the Furhat robot, where various facial expressions were transferred.They concluded that the VR condition represents a more enjoyable and socially immersive alternative for transferring realistic facial expressions to Furhat.
These WoZ systems intend to make robot interactions appear more realistic through the use of VR.While these approaches are novel and can advance the medium VR as a research tool in HRI, they do not solve the need for a modularized and customizable WoZ to bridge the gap for researchers with less expertise or for reasons of time and cost efectiveness.To mitigate the challenges of studying human-robot interactions in specifc environments and to take advantage of VR for research, an easily accessible solution for researchers, regardless of their professional background, is essential.This approach could add value to HRI research by providing researchers with a cheaper, faster and more fexible alternative research method.The potential of such a research tool could thus lead to more efective and informed research in the feld of HRI.

TECHNICAL IMPLEMENTATION
To develop our VR WoZ system, we collected several requirements among the interdisciplinary researchers of the RuhrBots competence center.Firstly, it needs to encompass the functionalities of existing WoZ systems, including features like providing robots with voice output, displaying external content via a tablet, and performing gestures.Secondly, the system should enhance study settings by incorporating VR, introducing a pathfnding system for robots, and implementing a data measurement system capable of continuously and reliably collecting various data from methods such as motion tracking, eye-tracking, and physiological data collection.Thirdly, the WoZ system should be adaptable to case-specifc environments, utilizing VR to recreate diverse public spaces and provide realistic settings for studies.
Regarding the technical implementation, Unity 3D was chosen for developing the WoZ system in VR due to its XR Interaction Toolkit, facilitating cross-platform use of VR devices.Typically, VR applications are projected onto a VR device and mirrored on the computer screen during runtime.In VR WoZ studies, this setup is advantageous as it allows the wizard to observe the participant's perspective in real-time.The WoZ system utilizes a second screen projection for the user interface (UI), chosen for functional considerations.It employs various customizable UI elements projected through a second camera in Unity.Scripts are written in a generalized manner, enabling dynamic adjustments and modifcations through Unity Inspector variables, such as text snippets or tablet contents.

Virtual Robot Environment
Currently, the VR application comprises a specifc library environment where various aspects of human-robot interaction can be explored.The library scenario was chosen, as this has arisen from the context of the RuhrBots project.The library is dynamically adaptable in terms of its objects (e.g., books, posters) and their arrangement.Additionally, the social robots Pepper and Temi are available.The robots are modeled after their real counterparts in terms of functionality and possess optional enhancements in their virtual representation, such as environment-sensitive navigability and more freedom in executing gestures.

WoZ Features
The WoZ system can be employed to control and manipulate various functionalities of the robots.An exemplary confguration of a WoZ for the library scenario is illustrated in Figure 2. To provide a better overview of the features, they have been divided into diferent categories, which are explained in the following chapters (Robot Communication, Robot Navigation, Robot Design, Data Measurement System).

Robot Communication.
The communication features for robots encompass voice output (category 1), displaying content via a tablet (category 2), and performing gestures (category 3).These categories are fundamental for basic interaction in HRI.In category 1 ("Robot Voice" & "Dynamic Robot Voice"), users can send pre-prepared voice commands or dynamically create commands using a text feld or microphone.This fexibility accommodates common commands and allows spontaneous responses to participant input.Category 2 ("Tablet Content") enables the display of diverse content on the robot's tablet, facilitating additional information presentation and control panel creation.Category 3 ("Gestures") serves as an execution category for pre-animated gestures, enhancing interaction capabilities.

Robot Navigation.
Since it is important for some studies that robots can navigate to certain locations, a navigation system was implemented.This is particularly important because physical robots can experience limitations in navigating in real space, especially in changing and dynamic locations.Thus, with the functions of Category 4 ("Robot Movement") it is possible to move the robot to predefned locations in the virtual environment.For instance, using the "To human" button allows the robot to navigate to the participant from any location.Additionally, static locations can be predetermined before WoZ experiments, and the robot can approach them during the experiment.The robots' navigation is context-sensitive concerning the objects in the environment, ensuring the robot fnds the shortest path based on the obstacles and objects in the virtual environment.

Robot Design.
To ensure a certain variability and dynamic adaptation in the appearance of the robots during the course of the study, it is possible to adapt various aspects of the robot during the runtime.Category 6 allows for the coloring of various objects in predefned colors.In the context of robots, specifc applications could involve changing the eye color, such as in Pepper, or altering the color of the robot itself.The areas to be colored can be defned in the Unity Inspector.Category 8 allows for the adjustment of the speech output of the robots and the speed.This feature enables diverse and inclusive responses to humans.

Data Measurement
System.When using VR, a lot of data is generated in the areas of motion, eye tracking and physiological tracking.This data can be collected continuously and reliably during the study period without being noticed by the participant.For this reason, a data dashboard was implemented to present various data in real time (category 7).Many newer VR devices come equipped with eye-tracking sensors.By default, these sensors can be controlled and utilized through a framework from Tobii.This enables the tracking of eye-tracking metrics like fxation duration or fxation count across a range of devices, and these metrics can be displayed in the WoZ.Additionally, the presented WoZ allows various objects to be selected for eye-tracking functionalities.This, for example, enables testing the attention of participants toward individual books in the context of the library.With current VR headsets from Hewlett-Packard (Reverb G2 Omnicept Edition), it is also possible to measure pulse, heart rate variability, and various pupillometry metrics.This provides additional insights into subtle reactions of people interacting with the virtual robot.

DISCUSSION AND FUTURE WORK
Based on requirements collected from an interdisciplinary group of researchers in the RuhrBots competence center, this paper presents a WoZ system for VR environments.The system, initially implemented within a library environment, is fully functional and currently deployed in studies on human-robot interaction in public spaces.Complete dialog structures can be constructed and complemented with gestures, tablet content, and other functionalities.However, while we are working on a version closer to a real authoring tool, these functionalities still need adjustment through the Unity development environment in the current version.For example, it is necessary to manually enter text descriptions for individual voice commands into the existing property variables in the Unity inspector.While this process works and might pose no problem for researchers experienced with Unity and similar programs, it may present a hurdle for non-technical researchers.Although the WoZ system presented was developed in a participatory manner with researchers from the felds of computer science and psychology, further iterations are needed in order to make the WoZ system accessible to everyone.The next steps include expert interviews with established researchers and focus groups with doctoral candidates in the feld of Human-Robot Interaction to explore the benefts, as well as the opportunities and risks they see in using such a system for their studies.Together, tools, methods, and techniques will be curated in a toolbox, along with the appropriate instruments, to make a technical system as understandable and usable as possible.
In this context, it would be conceivable to expand the technical concept of a Wizard-of-Oz system into a previously mentioned authoring tool.This way, non-technical researchers could not only beneft from the advantages of a generalized WoZ system in HRI but also utilize a comprehensive tool for study execution.The implication of this shift would be moving from adapting interaction during runtime to customizing and preparing the interaction before the study execution.This approach would provide researchers not only with the opportunity to test specifc interaction sequences in advance but also to use the advanced measurement methods that VR ofers and incorporate them into the research design.Complete study executions could be orchestrated, creating a locationindependent, fexible, and comparatively more afordable alternative to feld studies.It would be essential to ofer various environments (in addition to the library) since non-technical researchers may lack the expertise to create such virtual spaces.The creation of a "ready-to-use" authoring tool could thus generate signifcant added value in making VR usable for HRI research.This approach could empower non-technical researchers to set up studies using VR technology, thereby harnessing the full range of advantages that VR ofers for research and contributing to higher accessibility and equal opportunities in participating in research.

CONCLUSION
This paper introduced a modularized and customizable WoZ system for studies involving VR in HRI.It presents a fully functional virtual library environment, including two robots, along with a WoZ system that enables control and customization of the robots.Limitations in the current version include the selection of environments (currently only the library) and available robots (currently Pepper and Temi).Future iterations aim to address these limitations by introducing various case scenarios (e.g., public space scenarios like museums) and robots (e.g., Navel) to enable case-specifc research.
The overarching goal of this research is twofold: frstly, to enable the use of WoZ systems while harnessing the benefts of VR, and secondly, to explore ways to enhance the accessibility of technical systems, such as the presented WoZ for VR research projects, facilitating their use across a diverse range of disciplines.This concerns not only to the WoZ approach but also as a potential authoring tool.In the course of rapid technical development, it is therefore primarily a matter of researching what such systems need in the long term and what generally needs to be taken into account in the context of HRI research.Catalogs of requirements must be developed independently of the system in order to guarantee the sustainable use of such a research tool in HRI.

Figure 1 :
Figure 1: Illustration of the division of the application into two screens (left the participant POV and right the WoZ)

Figure 2 :
Figure 2: An example of the Wizard-of-Oz interface where settings can be adjusted at runtime.