Towards Unlimited Task Coverage and Direct (far-off) Manipulation in eXtended Reality Remote Collaboration

The growing interest in remote collaboration is driven by multiple factors. Advances in digital technology have made it increasingly feasible to collaborate from anywhere, breaking down geographical barriers. Despite the rise of new tools several challenges still exist. These include limitations on the remote member's perspective, confined to what on-site members can capture, as well as potential shortcomings for interacting with on-site physical objects. This position paper discusses the potential of leveraging mobile robots within eXtended Reality (XR) environments to transform how remote collaboration is performed. Mobile robots, equipped with cameras, microphones, articulated arms and various sensors, can serve as physical avatars for remote team members, allowing to navigate through physical spaces, enabling real-time, immersive experiences. Combined with XR tools, these robots may empower on-site and remote collaborators to engage in shared activities. These and other arguments in favor of this position are presented and future directions are proposed.


INTRODUCTION
Remote collaboration refers to the process of individuals working together on shared tasks from diferent locations.These activities have become increasingly prevalent in scenarios like industrial, medical, and education, among others [3,9,15,16,24].
The landscape of remote collaboration continues to evolve, presenting new possibilities for seamless, efective, and inclusive work.One example is eXtended Reality (XR), leveraging immersive technologies to transcend physical distances [5,17,19,22].This umbrella term, encompasses Augmented Reality (AR) for on-site members and Virtual Reality (VR) for remote individuals, creating interactive/immersive environments that enable collaboration in ways that were once constrained by geographical boundaries.Solutions exploring XR allow to defne a common ground for situation understanding, discussion of ideas, and generation of instructions, merging the real and virtual world [2,4,18,21,27].
Despite this, XR is not without its limitations.The remote member view is limited to what on-site members can capture, which might not always provide a comprehensive understanding of the environment.This can hinder the efectiveness of remote guidance and decision-making.Moreover, XR falls short in interaction with onsite physical objects.In situations requiring detailed explanations, nuanced gestures, or precise annotations, XR may not fully capture the richness needed for efective collaboration [7,13,25,26,28].
To overcome these constraints, mobile robots may be used as dynamic agents (Figure 1), being the eyes and ears of remote members, extending beyond fxed perspectives.Its mobility allows exploration, enabling a more comprehensive understanding of the environment.They can also navigate challenging terrains or hazardous environments where human access is restricted.These robots can enhance interaction capabilities, being equipped with advanced sensors, allowing remote members to manipulate physical objects at a distance.This interactivity bridges the gap between physical and virtual realms, extending team member's abilities while fostering more efective communication [1,8,11,14,20,23].
Literature shows that VR has been used for tele-operation of mobile robots, and AR for controlling robotic arms.Despite this, combining remote-XR with mobile robots for Human-Human and Human-Robot Collaboration (HRC) has not been explored in detail [6,8,10,12,13,25].This position paper describes a conceptual model for remote scenarios using XR and a mobile robot.The arguments in favor are described, and future directions are drawn.

ARGUMENTS FOR OUR PROPOSAL
Next, various arguments to support our position on the use of XR-solutions and mobile robots for remote scenarios are described: • Enhance immersion -XR technologies can provide a high degree of immersion.Remote collaborators can feel as if they are physically present, improving the quality of interactions and understanding of the on-site environment.Plus, mobile robots may act as a tangible presence with interaction capabilities over the environment, extending the presence of remote collaborators into the physical space.This extension goes beyond a virtual representation, allowing navigation and manipulation of the environment in real-time through XR, creating a more engaging and interactive collaboration experience that closely resembles physical presence; • Extend capabilities -remote collaborators can use XR to annotate, document, and share insights directly within the shared virtual space, creating a more dynamic and interactive knowledge-sharing environment.Taking advantage of the robot features, on-site members can be guided through specifc procedures, providing step-by-step instructions, all while communicating in real-time through XR interfaces; • Move towards more natural and adaptive interaction -instead of relying solely on one mode of communication, collaborators can use multimodal interaction in the form of gestures or voice commands to control the robot or manipulate virtual objects, making interaction more natural and intuitive between humans and robots.This adaptability to convey information, replicating how humans naturally communicate, enhances the collaborative experience.It can also allow addressing particularly important aspects of communication concerning, e.g., focus of attention and proxemics, i.e. nonverbal communication Moreover, collaborators may even distribute tasks across distinct modalities as a way to adapt to context and manage cognitive workload; • Elicit multi-sensor data collection -collaborators can navigate robots to inspect diferent areas, gaining a comprehensive understanding of the physical surroundings.The distinct sensors of the mobile robots may be used for multidimensional data collection, including environmental data, temperature readings, and other relevant information.This will provide a comprehensive view for team members to take

CONCEPTUAL MODEL FOR REMOTE XR HUMAN-ROBOT COLLABORATION
To inform action towards the development of remote XR HRC, aligned with the arguments set forth in the previous section, it is important to identify the overall dimensions at stake. Figure 2 represents a conceptual model, incorporating various modules for supporting remote collaboration using XR and mobile robots.The scenario presented includes distributed team members, each in a distinct physical space working together towards a common goal.The communication and interaction process occurs through the use of a Collaborative XR Platform, allowing combine the physical and digital realms.Each team member may be equipped with distinct devices (better suited to their role in the collaborative process) that converge within the XR tool, fostering a rich environment for discussion and information exchange.This innovative platform serves as a shared common ground, facilitating real-time communication, data sharing, and decisionmaking.To elaborate, an on-site member or remote-controlled robot will start by collecting contextual data from the environment (e.g., a video feed, a virtual scene, tangible landmarks, or a reproduction of the physical context).In turn, a remote member will analyze this data and enter a discussion phase, from which a set of instructions will emerge to address the problem at hand.This will occur through the authoring of virtual content, which on-site members will have access to, and may visualize superimposed on top of the real-world environment or using distinct approaches (see Figure 1).
To complement, a mobile robot equipped with an articulated arm extending from its frame becomes an extension of the team members' capabilities, ofering a unique blend of physical and virtual interaction.Controlled seamlessly by either on-site or remote team members utilizing XR features, the robot becomes a versatile tool for capturing point-of-view perspectives, sharing high-quality video and audio feeds, projecting information in selected areas of interest or interacting with physical objects in the environment.
The XR-controlled robot adds a layer of immersion to the collaboration, allowing team members to remotely navigate physical spaces, manipulate objects, and engage in tasks as if they were present on-site.This integration of XR technology with a mobile robot transforms the traditional boundaries of remote collaboration, enabling an interactive and multi-sensory experience.
To implement this vision, the following modules are required: • Data processing -this module serves as a centralized component, managing the infux of information generated by various team members.It employs advanced algorithms to process, analyze, and structure data efciently, ensuring that the information exchanged through the XR is ready to be consumed by the remaining modules; • Stream -this module facilitates real-time streaming of audio and video feeds, enabling live communication, ensuring that team members can share their perspectives, demonstrations, or collaborative activities seamlessly; • Authoring -this module empowers on-site and remote members to create a diverse array of virtual content (e.g., text, drawings, shapes, voice recordings, 3D objects), fostering a rich and expressive collaborative environment; • Interaction -this module accommodates a spectrum of distinct devices with varied interaction capabilities, ensuring a harmonious collaborative experience by adapting to their unique possibilities; • Visualization -this module enables capturing the task context, while also allowing team members to position virtual content in precise locations without compromising situational understanding over time.Together with the authoring module, it presents visual data in the XR space in a dynamic and adaptable manner, enhancing the team's ability to visualize complex information in a relevant and evolving context; • Ownership -this module manages access and permissions to collaborative assets.It verifes access rights, version control, and attribution of contributions within the XR space, ensuring a secure and accountable collaborative environment, where team members have defned roles and responsibilities; • Synchronization -this module oversees updates across team members, ensuring changes made by one member are refected for others, maintaining a coherent collaborative experience, eliminating discrepancies, and maintaining a unifed perspective among all collaborators; • Data Storage -this module provides a repository for storing and retrieving assets within the XR platform, ensuring the integrity and accessibility of shared information over time; • Remote Control -this module empowers both on-site and remote team members to manipulate mobile articulated-arm robots, through the XR platform.This allows seamless interaction with physical spaces, enhancing the scope of remote contributions within the collaborative environment.
In this landscape, the convergence of XR tools, diverse devices, and a versatile mobile robot can promote an efcient collaborative process by extending the reach of team members.This sets the stage for a new era of remote work, where the boundaries between the physical and digital worlds blur, and collaboration thrives.

FUTURE DIRECTIONS: A PROPOSAL
The evolution of XR-remote collaboration and mobile robots is ripe with possibilities.Next, some topics of interest are described, which will be pivotal in unlocking the full scope of XR and mobile robots capabilities to address the characteristics of remote scenarios: • Design and development of frameworks -develop XR environments that adapt dynamically to the collaboration needs.Consider how the environment can change based on the type of task, the expertise of team members, the complexity of the problem being addressed or the mobile robot available to assist during the collaborative process; • Defne modular architectures -a scalable and modular architecture allows the integration of the latest technological advancements without major disruptions across various sectors.New XR devices, sensor technologies, or robot features can be incorporated, ensuring that the collaboration platform stays at the forefront of innovation; • Expand robots sensors -enhance the sensory capabilities of mobile robots by integrating advanced sensors such as LiDAR, thermal imaging, or air quality sensors.This would provide richer environmental data to remote collaborators; • Improve telepresence aesthetics -improve the visual and auditory representation of collaborators in XR environments.This could involve enhancing avatars, facial expressions, and voice modulation for a more lifelike presence; • Support emotion recognition -integrate methods for reproducing remote members emotions (e.g., through the use of avatars or implicit visual cues, etc.), contributing to a more empathetic and human-like interaction experience, where participants can truly understand and connect with each other on an emotional level, building stronger bonds; • Establishing interaction patterns -explore multi-modal interaction, in particular, address how mobile robots may perceive when a specifc team member is explicitly interacting with the robot, which is extremely important in multi-user co-located scenarios, to avoid miscommunication; • Integrate intelligence features -implement Machine Learning (ML) and Artifcial Intelligence (AI) algorithms to enable mobile robots to make more autonomous decisions, understand and predict collaborators intentions, and assist them more efectively during remote sessions.This may include obstacle avoidance, path planning, and other intelligent behaviors that reduce the cognitive load on the collaborators; • Enable multi-robot collaboration -investigate how multiple mobile robots can work together in complex scenarios supported by XR.Research on coordination algorithms, communication protocols, and task allocation in multi-robot remote collaboration would be valuable; • Incorporate remote monitoring -explore how XR and mobile robots can be employed in continuous data collection and analysis.This may allow to conduct in-depth studies on how humans perceive and interact with mobile robots through XR tools.For this, frst, it is important to establish relevant metrics for quantifying the quality of collaboration; • Assess collaborative decision-making: -evaluate how well distributed team members can collaborate efectively using the XR-supported remote collaboration system, in particular, focusing on the the impact of the mobile robot in facilitating discussions and decision-making.What is its effectiveness as a communication medium?; • Conduct long-term evaluation -study how collaborators adapt to XR and mobile robot collaboration over long periods (e.g., changes in user behavior, satisfaction, and productivity as they become more accustomed to the proposed paradigm); • Verify adaptability to varied conditions -evaluate how XR and mobile robots adapt to varied conditions like lighting environments, network outages, physical spaces, system failures or cyber-attacks.This may require exploring redundancy measures and fail-safe mechanisms to ensure continuous operation can be attained across diverse scenarios; • Consider ethical aspects -address data privacy, ownership of virtual assets, robot behaviour accountability and the implications of on social dynamics.This may imply establishing guidelines for the use of XR and mobile robots in various sectors, ensuring safety and compliance.
By delving into these aspects, researchers and practitioners can deepen their understanding of the complex interplay between humans, XR, and mobile robots for remote collaboration.This comprehensive exploration can result in more robust, adaptive, and user-friendly systems that cater the needs of various industries.

CONCLUDING REMARKS
In summary, an opportunity exists to explore the combination of eXtended Reality (XR) tools and mobile robots for scenarios of remote collaboration, as illustrated in this position paper.This dynamic synergy presents solutions that may address diverse contexts, transforming the way teams interact across distances.By bridging geographical gaps, XR-enabled solutions and mobile robots can bring experts to the feld without requiring them to be physically present, which is particularly valuable in remote, and sometimes hazardous environments.This approach ofers a multifaceted set of benefts that cater to a wide array of situations, ranging from improved efciency and cost savings to increased safety and environmental consciousness.

Figure 1 :
Figure 1: Example of a scenario of remote collaboration having a remote member (left) and multiple on-site individuals (right) communicating and sharing information through XR and a mobile robot.Inspired by the graphical design of [25].
into account during collaboration, leading to better decisionmaking and resource management over time; • Harness Global Talent for Local Problems -geographical boundaries become less restrictive.Accessing expertise from anywhere becomes feasible, making it easier to assemble diverse teams and draw upon global talent, which can significantly improve problem-solving and decision-making.It can lead to more efcient workfows and faster issue resolution; • Decrease costs -reduced travel costs can represent a substantial advantage while being more environmentally friendly.Combining XR and mobile robots minimizes the need for on-site visits, saving organizations money on transportation, accommodation, and other related expenses.Plus, fewer physical travels lead to a reduced carbon footprint; • Raise exploration of novel environments and solutions -robots can adapt to varied indoor/outdoor settings, making them versatile tools for remote collaboration.Combining them and XR opens up new possibilities for exploration, allowing for the investigation of more restrictive scenarios like archaeological sites, or space exploration; • Increase safety -in scenarios with inherent risks, such as industrial maintenance or hazardous environments, mobile robots provide a safer alternative to perform given tasks.By using XR tools, on-site and remote members can assess and guide the situation without exposing themselves to danger;

Figure 2 :
Figure 2: Conceptual model of a Collaborative platform illustrating the necessary elements for a scenario of XR-remote collaboration between an on-site and a remote member supported by a mobile robot.Assets obtained from iconfnder.com.
This research was supported by IEETA in the context of the project [UIDB/00127/2020], and by the RN21 Integrated Project -Innovation in the Natural Resin Sector for the Reinforcement of the National Bioeconomy, co-fnanced by the Environmental Fund through Component 12 -Promotion of Sustainable Bioeconomy (Investment TC-C12-i01 -Sustainable Bioeconomy No. 02 /C12-i01 / 202), European funds allocated to Portugal by the Recovery and Resilience Plan (PRR), under the European Union's (EU) Recovery and Resilience Mechanism (MRR), framed within the Next Generation EU, for the period from 2021 to 2026.