VR-based Assistance System for Semi-Autonomous Robotic Boats

In this paper we present the concept for a teleoperation system for semi-autonomous robotic boats using virtual reality. This system can be used for monitoring autonomous driving as well as for direct manual control. The integration of live sensor data is possible as well as the integration of past measurement results and their correct registration within the virtual representation. Initial field tests have shown that the technical concept can be used for the envisaged areas of application, from which specific challenges can be derived that are addressed in this paper.


INTRODUCTION
Mobile robotic systems that are suitable for use in open terrain have been establishing themselves for several years in various areas of application such as environmental monitoring, infrastructure inspection and agricultural land management.ASVs are used in Agriculture 4.0, for example, while drones are used to inspect photovoltaic systems or survey open-cast mines.Another innovative example -and the focus of this paper -is robotic boats, which are e.g.used in the sensory recording of water quality or in search missions [20].
The fully autonomous operation of such outdoor robots often comes up against limits under difcult operating conditions.The robot itself must be able to recognize critical situations and switch between autonomous, semi-autonomous and manual modes depending on the context.Using an (Artifcial Intelligence (AI)-based adaptable) catalog of criteria, the robot identifes the most suitable mode depending on the situation and communicates this to the operator.Using Virtual Reality (VR) interfaces, the operator gains a comprehensive impression of the conditions on site, e.g.obstacles and sensor data.This form of immersive control reinforces the involvement of a remote operator in the current situations of ASV and measurement.The advantage of a VR system therefore lies in improving possible confict resolution strategies and avoiding problems of mixing diferent projections (full ASV view vs. camera image / map view / external view from the shore).We want to address the mentioned points for autonomous boats in a frst step with the development of a VR-based Human Machine Interface (HMI) prototype for remote control of ASVs, that includes the following features: • Streaming of camera images with boat position as pivot point to a remote operator with Head-Mounted Display (HMD) • Integration of additional data, e.g.location and depth sensing • Visualization of expected trajectory and bearing • Interactive control for live adjustment of path planning In this paper we present the technical prototype and conceptual work for the HMI.

BACKGROUND 2.1 Autonomous Surface Vehicles (ASVs)
By the term ASV we refer to robotic boats and robotic swimming platforms (surface) in this paper.A typical use-case for these robots is geo-environmental monitoring [4][5][6][7][8], especially for bathymetry measurements [20,23].For these, ASVs need specifc technical systems and sensors, like position determination systems, propulsion systems, Inertial measurement unit (IMU) and sensors for acquisition of data of the surroundings (cameras, depth sensors).One example of an ASV, a Clearpath Kingfsher 1 , which is the robotic platform of choice for the present paper.This ASV, with dimensions 350 x 980 x 320 mm, comes with Global Positioning System (GPS), IMU and front camera.The propulsion is done via electrical water jet and a drive power of 40 N thrust, which leads to a maximum speed of 1.7 m/s (3.3 kn).Its weight is 28 kg with 10 kg additional payload 2 .

(VR)-Teleoperation of ASVs
Teleoperation systems are a long-standing feld of research that has led to a rich literature and promising results for similar applications (e.g. for drones), e.g. in [2,3,12,[14][15][16]18]. Teleoperation can be used for remote controlling vehicles and robots, e.g.cars [19], drones [10] or ships [17].Commonly, a remotely located human operator is presented with the current state of the controlled system as perceived by its sensors through an HMI.It allows the operator to determine appropriate control commands sent to and executed by the controlled system.For this loop to be successful, the HMI has to enable the operator to correctly understand and assess the state and context of the remote system.To that end, existing concepts leverage the visual focus of operators and present them with video streams of onboard cameras, e.g. as in [9].However, to support a better sense of orientation and scale those streams can be processed to present enriched views as deployed by [22].Next to the data, the presentation itself is central to enable the operator to safely control the remote system.For that, [11] propose using HMDs to enhance situation awareness and task performance.[13,24] confrm that users working with VR/Augmented Reality (AR)-HMIs perform better in detecting collision or grounding situations of surface vehicles than users working with conventional HMIs or even 3D-GUI.The users rated the VR-GUI to be the easiest tool to use and at the same time the best expert tool.As before, the visual focus of human perception is exploited by allowing operators to use their natural behavior (e.g.turning their head) to create an immersive experience increasing situational awareness.For that, the style with which the 1 https://www.clearpathrobotics.com/wp-content/uploads/2013/08/Kingfsher_USV_Brochure_2015.pdf 2 https://github.com/kf/kingfsher/tree/indigo-devel/kingfsher_descriptioninformation is presented has to be designed carefully, especially for such VR-based systems.Inappropriate design can increase the risk of motion sickness of the operator.It emerges from disagreeing sensational stimuli and can cause, for instance, disorientation and nausea [1].We discuss this point in Section 5.

CONCEPTUAL WORK
To meet the requirements mentioned in Section 1 the Clearpath Kingfsher serves as a platform for lightweight technical setup.In the following sections both hard-and software concept are described.We plan to make the completed model publicly available, e.g. on GitHub.

Technical setup
The technical setup comprises a HMD, a comprehensive camera to capture the boat's surroundings, a more powerful board computer (as the standard of the Clearpath Kingfsher has not the capabilities for high-resolution video streaming) and depth sensors.The following systems are used within our concept: VR HMD The remote operator uses a HTC VIVE Focus 3, a standalone device with inside-out tracking.This ensures the mobile application in outdoor areas.Camera Using a telescopic pole, we attached a RICOH Theta Z1 360 • camera to the ASV.With a resolution of 23-megapixels (6720 x 3360 pixels) it ensures the capturing of image-based information of the boat's environment.Board computer To increase the computational power of onboard systems, but to also consider the power consumption of 12V, we chose an Intel NUC barebone system as board computer with Ubuntu Linux as Operating system (OS).Depth sensor For depth sensing we consider the usage of Light Detection and Ranging (LiDAR) (above water surface, e.g. for tunnel exploration) and Sound Navigation and Ranging (Sonar) (R2Sonic 2020 for underwater profling).The wireless connection of the on-board systems (board computer and camera) is ensured by an added router with low power consumption.The transfer of the camera image data to the board computer is established as User Datagram Protocol (UDP) connection based on GStreamer to avoid delays and react to data losses.The connection between the ASV and remote station is provided via the on-board antennas in a 2.4 WiFi network.For the frst prototype we consider distances from remote station to ASV of up to 100.Nevertheless, for bigger areas connectivity is an issue and mentioned in Section 5.

ROS setup
For the core of the model we choose Robot Operating System (ROS), as it is the de-facto standard middleware in robotics.In the ROS framework, nodes interact to each other through a publishsubscribe messaging and service mechanism.Messages are transfered through topics, which are containers with unique identifers to and from messages are push and pulled (e.g. for data transfer).In this paper we use ROS 1 (Noetic), because the ASV also runs on ROS 1 and we can use some of the nodes and topics.Nevertheless, we plan to use ROS 2 for all ASV-independent scripts, as the packages for the ROS-Unity communication are developed in ROS 1 and 2.  A connection between Unity and ROS is established by using a Transmission Control Protocol (TCP) endpoint and connectorboth packages are developed by Unity Technologies 3, 4 .The TCP connector needs to get integrated in the Unity project and enables the engine to receive and send messages (see Section 3.3).The counterpart is the TCP endpoint, a node within the ROS network, which communicates with the connector.Therefore the model is not a ROS node itself, but communicates with the network by generating a TCP connection with the ROS master over the TCP endpoint node.The basic structure is shown in Figure 2.
So basically, the on-board camera sends an image stream to the board computer, where the data is published to a ROS topic, which can be subscribed by other applications -e.g.Unity at the remote station.

Unity setup
After subscribing the ROS topic of the camera, the image data can be accessed within the Unity game engine5 .To do so, a VR camera game object is placed at (0, 0, 0) surrounded by a sphere.The 360 • image is mapped as textured on the sphere around virtual camera with the lower half is culled due to the augmentation of sensor data and the integration of further information, e.g.path (see Figure 3, without culling).
Below the virtual camera a low poly model of the ASV is placed with a compass attached (see Figure 4).The green line indicates trajectory and bearing, the grey points represent a former Sonar scan.
To provide a control panel, a Head-up-Display (HUD) is added, whose visibility can be switched on and of.The HUD provides input elements to adjust network and control settings and shows certain parameter of the model (network connection, delay time [], Frames per Second (FPS), current speed (from ROS).The HUD is shown in Figure 5.The ASV position is synchronized using the World Geodetic System (WGS 84) and the GPS data stream from ROS.The initial orientation can be adjusted in the application.

EXAMPLE
We tested the described model at an inland water (lake) with dimensions of roughly 100 × 100.Under normal conditions, the ASV navigates autonomously on the water and records relevant parameters.The vision is to integrate a context-based movement planning and execution supports the operator in distinguishing between the (1) autonomous normal mode (VR supervision) (2) diferent gradations of interactive semi-autonomous control up to the (VR supervision and interactive path planning) (3) manual mode (VR visualization and controller input).
The control in the feld trial was done in two ways: manually with an additional controller (Logitech F310 gamepad) and automatically with prepared waypoints.The Logitech controller replaced the HMD controller as their functionality for interactive path planning (2) has not been implemented yet.The technical setup of the ASV for the test run is shown in Figure 6, an example shot of the VR scene is depicted in Figure 7.To analyze the observed correlation between distance (ASVremote station) and delay we plan to conduct a comprehensive user study to evaluate the system's usability and efectiveness in realworld scenarios to show its benefts and analyze its weaknesses.The focus of these studies is connectivity, reliability and usability of the application.

CONCLUSION & FUTURE WORK
In this paper, we have presented the technical concept of a VRbased teleoperation system for robotic boats.The system includes mobile hardware, communicates with ROS and is scalable in quality and performance.The feld tests have shown that it can be used as a basis for further developments as a monitoring and later also as an interaction interface.The following problems and challenges arose in the course of development and testing: Connection The provision of suitable bandwidth for the transfer of high-resolution video fles in a comparatively large operating range is elementary for the use of the system.While the scalability of the system in our example was able to provide a remedy for a small body of water, this point must be taken into account in future experiments and to ensure suitable FPS for live control in teleoperation.As an alternative, we plan to integrate 5G in a future application.Sea & motion sickness Like in other VR applications [21,25], permanent use is difcult here too, which can -among other things -be described by motion sickness.In addition, sea sickness must be taken into account when using water surface vehicles, which also makes continuous teleoperation difcult.As a prototype, we have already implemented initial approaches to balance the camera using IMU data given by the ASV or using a 3-axis gimbal.Assistance handler The operator is supported by a set of assistance functions, e.g. an exploration, planning or collision assistant, which are activated on a context-driven basis.Depending on the situation, the assistance functions merely make suggestions, for example regarding the trajectory, which the operator has to confrm.On the other hand, the operator can suggest a route, which is checked by the system and corrected if necessary.Interactive path planning using VR controllers requires the fusion of image and depth data.Since the camera image is planar, no waypoints can be placed on it, so a way must be found to link the visualization of GPS-based waypoints with the representation of the image data.One option here would be to display image and depth information separately rather than overlaying them.Context identifcation A state machine is designed to describe the context situations, whereby the transition functions between the control modes are concretized based on an initial setting with AI approaches.The input variables are the environment model for navigation and data collection, as well as the intended movement specifcations.In addition to water depth, shoreline edges, etc., the navigation model also includes recognized or potential obstacles as well as current and wind information.The model for data collection includes the actual target variables of the measurement campaign.These can be, for example, hydrological, chemical or physical phenomena (fow velocities, pH values, temperatures).
We see various ways of addressing these problems and will adapt and improve the application accordingly.In addition to feld tests, we are planning a user study to investigate the behavior of various operators in forced confict situations for the ASV.

Figure 2 :
Figure 2: Schematic overview of the technical setup and the used components.

Figure 3 :
Figure 3: Mapping of 360• image (represented by a section) on a sphere, in which centre the virtual camera is positioned according to the ofset of the telescopic pole.

Figure 4 :
Figure 4: Interface design in Unity with a 3D model of the boat with compass and green trajectory.The grey points in the background visualize a point cloud of a previous sonar scan.

Figure 5 :
Figure 5: Interface design in Unity with canvas display with additional information.The blue lines with yellow points represent the input controller of the HMD.

Figure 6 :
Figure 6: Technical setup of the ASV for feld trials.

Figure 7 :
Figure 7: VR scene with 360 • image on surrounding sphere, HUD and virtual boat model.