Mobile Robots Meet Augmented Reality Technologies: Transforming Human-Robot Interaction in Industry 4.0 Scenarios

Mobile robots combine digital technologies and automation, enabling a new era of operations in Industry 4.0 scenarios. These robots can navigate through assembly lines and logistics warehouse autonomously, making them catalysts for increased efficiency and productivity. Despite this, various challenges arise, including maintaining multiple robots up and running over large spaces, handling navigation during unexpected situations, ensuring operators safety, among others. To help overcome these, Augmented Reality (AR), a key pillar of Industry 4.0 can be used to assist with Human-Robot Interaction (HRI), contributing to a more productive and optimized logistics environment. This work proposes a framework for enhancing the understanding of the status of logistics robots through the use of a mobile AR tool. Its goal is to provide operators with an effective way of obtaining more information about the surrounding robots, as well as remote control them. The framework was iteratively tested during the various phases of design and development, allowing to integrate the feedback collected in each stage.


INTRODUCTION
Industry 4.0, often referred to as the Fourth Industrial Revolution, represents a paradigm shift in manufacturing and industrial processes that leverages advanced technologies to create smart, interconnected, and highly automated systems.At its core, Industry 4.0 aims to merge the physical and digital worlds to optimize efciency, fexibility, and productivity across various industries.Among the key technologies driving this transformation, robotics, especially intelligent mobile robots, play a pivotal role in redefning how industries operate These robots are characterized by the integration of digital technologies, automation, and data-driven decision-making into manufacturing and industrial processes (Figure 1).Their ability to work alongside humans, adapt to changing conditions, and contribute to increased efciency makes them a critical asset for modern manufacturing and logistics operations [4,16,18,31].
To elaborate on some advantages of mobile robots, they may be used for automating a wide range of tasks, such as material handling, transportation, and repetitive operations.They can work 24/7, navigating autonomously in large spaces, increasing the speed and accuracy of order processing, thus improving efciency and reducing operational costs.Moreover, they may be equipped with sensors and advanced control systems, often including AI-driven decisionmaking capabilities, that allow them to interact with objects, as well as detect and respond to human presence, thus enhancing Human-Robot Collaboration (HRC) on the shop foor.These robots can also be equipped with vision systems that perform quality inspections, identifying defects or inconsistencies in products, sometimes, more accurately and consistently than human inspectors.Equally relevant is their ability to collect real-time data about the production process, inventory levels, and equipment status.This data can be used for predictive maintenance, quality control, and process optimization-Mobile robots can also optimize energy consumption by using efcient routes and minimizing idle time, contributing to sustainability goals in Industry 4.0 [2,3,[11][12][13]15].
Although using intelligent mobile robots in Industry 4.0 ofer numerous benefts, it also come with their own set of challenges.These include navigating dynamic and cluttered environments, containing various obstacles, narrow aisles, changing layouts, and potentially slippery surfaces, afecting safety and movement efciency.Likewise, maintaining efciency and coordination while being part of a large feet of robots.Precise localization within the industrial environment is also critical for mobile robots, which can be afected by lighting conditions and signal interference, impacting their ability to maintain accurate positioning.Another challenge is associated with ensuring safe and efective interaction in the presence of both other robots and human workers.Thus facilitating seamless cooperation, collision avoidance and prevent accidents.The challenges even go beyond the robots.Introducing mobile systems might require retraining the existing workforce to operate alongside and interact with robots.All in all, although these robots are designed to be highly fexible, due to these reasons, sometimes, they need human operators to assist them overcome such constraints, which imply the use of additional technological solutions.In fact, human operators are also essential to conduct maintenance procedures on mobile robots, contributing to their reliability and preventing disruptions in the shop foor operations [7,14,17,22,34].
To help overcome some of these limitations, one possible solution is the use of Augmented Reality (AR), one of the nine pillars of Industry 4.0 [6,19,20,25,29].This technology has the capacity to improve human operators decision-making, as well as facilitate interaction with surrounding robots by enhancing spatial understanding and awareness through responsive computer-generated information that is superimposed over the real-world environment [26,27,32].Solutions using AR have been explored over multiple felds (e.g., industry, healthcare, education, entertainment, serious games, among others) [5,7,8,10,21,28], and, in recent years, started to expand to address Human-to-Human Co-located and Remote Collaboration [23,24], as well as Human-Robot Interaction (HRI) or HRC thanks to the advancements in recent hardware and software solutions.For industry scenarios, AR may signifcantly improve overall operational efciency, safety, and the quality of interactions between human operators and robotic systems [1,9,17,22].
This work proposes a framework for improving comprehension of mobile robots status.It uses a mobile AR tool, ofering operators the means of acquiring enhanced insights into the surrounding robots, and facilitating data visualization and remote control over them.The framework underwent iterative testing across the diferent stages of its conception and development, afording the seamless incorporation of feedback garnered at each stage.

FRAMEWORK USING MOBILE AR
Figure 2 shows some features of the proposed framework, while Figure 3 displays an overview of the main modules.It was designed to facilitate how human operators' conduct specifc procedures on industrial mobile robots.It seamlessly integrates AR interfaces with remote control capabilities and real-time data visualization, contributing to improve the operator's ability to monitor, control, and maintain mobile robots efectively.The core functionalities of this framework are described next, and revolve around utilizing a mobile device's camera to track a mobile robot marker, triggering a set of AR interfaces to enhance the operator's capabilities.
Marker detection and confdence zone: When the mobile device's camera detects the visual marker of the mobile robot, an array of AR interfaces is triggered, providing the operator with realtime data and control options.This feature also enable selection of various robots, enabling operators to manage multiple robots efectively and collaborate seamlessly.Besides, the marker detection confdence zone is illustrated to give operators insight into the accuracy and reliability of the robot's marker recognition (Figure 2-1).This allows operators to manage robots from a safe distance, reducing the risk of injuries in hazardous environments.AR can also be used to display emergency protocols, enabling swift and accurate responses to unforeseen situations.
Status visualization: Real-time visualization of robot's status like sensor readings (e.g., battery levels, task progress, and environmental conditions), object recognition results, alerts, and navigation paths is available to provide operators with a comprehensive understanding of the robot's characteristics (Figure 2).This helps gain insights into robot performance, efciency, and areas for improvement.Plus, by providing operators with information directly within their feld of view, AR reduces the need for them to switch between remote screens.This streamlined access to information can help operators make informed decisions, leading to increased efciency and productivity.
Sensor management: Operators can also view a list of available sensors on the robot and toggle them on or of as needed, allowing for fexible data collection and analysis.This way, AR interfaces can be customized to display information relevant to specifc tasks or scenarios.For example, the representation of the signs identifed by the robot may be enabled, allowing to verify if the identifcation was correct or incorrect quicker than having to check the robot log later on a distinct device.This adaptability ensures that operators receive the most pertinent information for their current context.
Bumper Status Display: The robot's bumper status may be also exhibited, emphasizing any changes in its physical interaction with the environment (Figure 2-2).AR can display data from other sensors to enhance human and robots safety by alerting operators to potential collisions or hazards in real time.Visual overlays can provide warnings or guidelines to help operators steer robots away from obstacles.
Trajectory and path planning: Various indicators are displayed to aid operators in understanding the robot's trajectory throughout the environment, including its speed, direction and rotation.By using AR, operators can gain better situational awareness of their surroundings and the robot's environment.This is especially crucial in busy industrial settings where multiple robots Figure 2: Mobile robot data visualization using AR: 1-illustration of the tracking confdence zone -from bad (red) to optimal tracking (green); 2-bumper sensor augmentation.Red represents a detected collision, while green no collision; 3-augmentation of the wheel drop sensor.Red represents that the wheels are being pushed up, while green is its default state; 4-remote control using joystick buttons.Also, charts illustrating angular speed (blue), linear speed (orange).and operators are working simultaneously.Moreover, AR can assist in optimizing paths for mobile robots by overlaying potential routes onto the environment foor.This helps operators visualize and plan efcient routes, avoiding congestion and minimizing collisions, making navigation and path planning more straightforward in case of malfunction of collision.
Component Highlight: Various components of the robot may be highlighted, supporting maintenance, component replacements, or quality control procedures by pinpointing specifc areas of interest by superimposing step-by-step instructions (Figure 2-3).AR's ability to overlay instructional content onto the real-world environment can beneft training and onboarding processes.Likewise, new operators can learn how to interact with robots and perform tasks efciently by following selected AR-guided instructions, reducing the learning curve (faster onboarding), while leading to faster skill acquisition and greater confdence among operators.
Remote Control Mechanisms: The framework ofers mechanisms to remotely control the robot's position within the environment, granting operators the ability to navigate it precisely through out the environment.Operators can utilize AR joysticks to control robot trajectory, speed and rotation, while navigating complex and dynamic environments, streamlining coordination and task allocation (Figure 2-4).
Overall, incorporating AR into industry scenarios has the potential to bring numerous benefts.These include real-time data overlay, improved quality of interactions through intuitive interfaces, enhanced training and simplifed execution of complex tasks, more efcient operations and monitoring, reduced downtime, increased safety for both operators and robots, as well as customization/adaptability.This holistic improvement may contribute to a more productive and optimized logistics environment.Also, contribute to decrease human errors while operating or performing specifc procedures on the mobile robots.This is particularly valuable in logistics operations where precision and accuracy are paramount.
The framework was created through a Human-Centered Design (HCD) methodology, starting with problem understanding, requirement elicitation, creation of low fdelity mock-ups.Later, design and development of the proposed prototype using the Unity 3D game engine, based on C# scripts and the ROS# plugin, enabling communicate with the mobile robots.The virtual content was placed in the real-world environment through the Vuforia library.
The framework includes interaction and visualization managers responsible for sharing all respective changes to the mobile robot.Also, communicate efectively with the data processing manager, responsible for processing the data generated by the operator interactions.It then shares and stores this information in a dedicated server.Following, the robot's status is promptly updated to refect the alterations.This real-time feedback loop ensures that the operator's commands and visualizations are accurately refected in the robot's behavior.This approach is generic enough so that the data processing module can be connected with a Digital Twin (DT) of an industrial environment, allowing to have a live virtual representation of the environment, as well as all mobile robots and their current status at any given moment.

FIRST IMPRESSIONS
An initial user study was conducted to evaluate the features of the proposed tool.Participants needed to complete a set of procedures on a TurtleBot mobile robot over a large environment while equipped with a mobile device, simulating a logistics warehouse.Tasks included detecting the marker and connecting to the robot dashboard; analyse the current status of a set of indicators; highlight some components of the robot structure illustrating how to conduct a replacement procedure; control the robot and change its trajectory over a large space; collide the robot into a wall and verify the change of the bumper sensor; among others.
Participants were instructed on the setup, the task, and gave their informed consent.Then, the AR-based tool was introduced and an adaptation period was provided.After fnishing the tasks, participants flled in a survey regarding the tool.Plus, a small interview also occurred.To this end, 15 individuals were recruited whose ages ranged from 20 to 55 years old.Participants had various professions, e.g., Faculty members, Researchers, as well as PhD and Master students from diferent felds.From these, 1 participant had no background knowledge in either robotics or AR, 9 participants had previously interacted in AR environments and the remaining participants were both familiar with robotics and AR.
This approach allowed us to gather valuable insights into the framework's usability and efectiveness in a realistic scenario.This controlled experimental design allowed us obtain reliable and objective data regarding the use of AR in the described scenario.Additionally, draw meaningful conclusions and make informed decisions for further improvements.Overall, all participants were able to fulfl the intended tasks.Participants with prior robotics experience exhibited a preference for the practical aspects of the application, namely being able to control the robot, as well as the component highlights.Participants with less experience showed a greater inclination towards the status visualization as well as the trajectory and path planning.They also emphasized that AR allowed to comprehend concepts that would other wise be more difcult to assimilate.Likewise, they suggested that the AR interfaces were user-friendly, making it accessible to those without extensive technical knowledge.In general, all participants reported the importance of having such a tool to address the tasks at hand, highlighting that in more complex and dynamical environments, the proposed tool would be of even greater assistance.Regarding future updates, participants expressed interest in more diversifed data visualizations, as well as more detailed models.This indicates a strong desire to gain additional insights into a robot's status through the use of AR.They were also curious about experience the proposed tool in a real-life environment with multiple robots acting at the same time.
Moving forward, a feld study with human operators and other potential target users is being prepared, allowing to improve the ecologic validity of the proposed solution.In addition, integrating remote features, allowing to collaborate with distributed experts for handling specifc activities, i.e., maintenance procedures is always being considered [23,30,33].

FINAL REMARKS AND FUTURE WORK
This paper proposed a framework for visualization of mobile robots real-time data.Using AR allows operators to easily understand robot status, paths, and other relevant data, facilitating better communication and collaboration between humans and robots.By incorporating AR into HRI in industry scenarios, we expect to signifcantly improve overall operational efciency, safety, and the quality of interactions with robotic systems.
Moving forward, the framework will be extended to support AR headsets, given the growing adoption of such devices in Industry scenarios, allowing to conduct activities in a hands-free setting.Afterward, we intend to conduct a real-life user study in an industrial scenario to properly validate the proposed framework with a larger, more diversifed audience, a scenario that will provide a more ecological setting.Additionally, explore AR-based remote features for supporting of-site assistance, an activity sometimes required to overcome specifc maintenance procedures.

Figure 3 :
Figure 3: Framework overview for assisting human operators with data visualization and remote control of mobile robots in industrial scenarios.Assets from iconfnder.com.
This research was supported by IEETA in the context of the project [UIDB/00127/2020], and by the RN21 Integrated Project -Innovation in the Natural Resin Sector for the Reinforcement of the National Bioeconomy, co-fnanced by the Environmental Fund through Component 12 -Promotion of Sustainable Bioeconomy (Investment TC-C12-i01 -Sustainable Bioeconomy No. 02 /C12-i01 / 202), European funds allocated to Portugal by the Recovery and Resilience Plan (PRR), under the European Union's (EU) Recovery and Resilience Mechanism (MRR), framed within the Next Generation EU, for the period from 2021 to 2026.