Getting Closer to Real-world: Monitoring Humans Working with Collaborative Industrial Robots

Detecting behavior and cognitive states of operators collaborating with industrial robots in the field is among the primary objectives and challenges in current human-robot collaboration research. To achieve this goal, it is first essential to introduce dynamic elements of the industrial settings into the lab to examine how to effectively detect, read, and interpret operators' psychophysical reactions associated with complex and dynamic workflows. As a first step toward this goal, we designed a realistic collaborative assembly task, encompassing common manufacturing operations performed jointly by humans and cobots. The task involves sequential sub-operations in a practical workflow (manual screwing, pick-and-place, supported screwing) and dual-tasking simulating conditions of fatigue and high workload. Early results cover users' performance, mental workload, and affective reactions, along with acceptance, engagement, and participants' perception of the experimenter's influence on the task execution. We finally discuss how such an experimental approach represents a first step toward field-based interventions.


INTRODUCTION 1.Transitioning from lab to industry feld in human-robot collaboration assessments
The study of human factors, behavior, and the cognitive and affective states of operators working closely with advanced production systems, such as collaborative robots (cobots), is emerging as a key area of research, particularly in industrial manufacturing [7,12,19,27,30,38].Following years of impressive technological advancements focused on enhancing efciency, improving quality, and reducing costs, Industry 5.0 "human-centric manufacturing" is now widening the focus to human operators, making central their well-being, needs, and capabilities [24].In this view, human factors, and behavioral and cognitive dynamics of operators interacting with such new technologies need to be monitored in industries.The ultimate goal is hence monitoring operators' cognition and perception in the feld, bringing traditionally lab-based methodologies and equipment into the industries.To achieve this goal, it is frst essential to introduce dynamic elements of the industrial settings into the lab to examine, in a semi-controlled setting, how to efectively detect, read, and interpret the cognitive dynamics associated with complex behaviors, such as those exhibited by operators working closely with cobots.Specifcally, we evidence two essential points.First, the experimental tasks need to be realistic and dynamic.Ideally, they must be composed of actions seamlessly integrated into a coherent workfow that is controllable yet avoids excessive simplifcation.Second, the experimenter(s) should be able to ensure the execution of a standardized and safe experiment without interfering with the task execution, simulating situations in which operators handle industrial tasks unassisted.
In this contribution, we thus examine the design of an industrial assembly task that aims to resemble a real industrial activity more than an experimental task.We also illustrate a dual-task paradigm that manipulates task demand, and consequently operators' mental load, by tackling similar cognitive marks as those involved in highlyfatigued industrial operators [1,35].Therefore, we present our early results on the performance, workload, and afective reactions of participants executing the task under various demands, on their levels of acceptance and engagement, and the perceived role of the experimenter within the lab setting.We fnally discuss how these fndings may be transposed to real-world industrial settings, assuming this contribution as a frst and necessary step toward feld-based monitoring of Human-Robot Collaboration (HRC).

Targeting realistic industrial tasks
As an initial step towards transitioning to feld-based studies, researchers can enhance lab methodologies and equipment to align with the complexity and dynamism of industrial tasks.For instance, Segura and colleagues [36] have reviewed the most common tasks performed in various manufacturing industry sectors, showing how collaborative tasks such as the assembly of small-sized components and the handling of heavy objects, are well-suited applications across various industry sectors.
Nonetheless, in traditional HRC lab studies, researchers tend to oversimplify industrial and production operations (e.g., [6,9]), focusing on segmented or single operations (e.g., picking, placing, etc.) that misrepresent the actual industrial activities.Several studies addressing human performance, workload, and/or UX in HRC leveraged repetitive pick-and-place tasks that overshadow the dynamicity typically required by advanced cobots [6,9,10].Although this approach reveals very subtle behavioral and cognitive mechanisms, it rarely produces directly applicable or relevant data for real-world advanced manufacturing contexts.
This addresses the frst point: creating dynamic tasks that strike a balance between experimental control and oversimplifcation.Rather than breaking down industrial operations into fragments, we advocate for integrating diverse operations into a seamless workfow that closely mirrors actual industrial work.This approach enhances ecological validity for studying industrial activities, particularly those performed with highly advanced collaborative workstations.Furthermore, literature suggests that, for studying cognition and action in an integrated way, we need to consider that diferent actions are inherently integrated into a continuous fow of ongoing behaviors and should be investigated as such [16,18].Then, we can use methodological procedures typical of experimental settings (e.g., randomization and counterbalanced order in which various actions are performed) to ensure the internal validity of our measures.
Additionally, in lab-based studies, experimenters are often present to guide participants in performing standardized procedures and to prevent physical collisions with the cobot.In the latter cases, they need to reassure participants before restarting the experiment.However, this involvement can infuence participants' perceptions of the cobot or task dynamics.It is thus crucial to establish standardized and safe settings while minimizing the experimenter's role, simulating scenarios where operators handle tasks unassisted, and allowing participants to form perceptions solely based on task and cobot dynamics.

Targeting human workload, stress and fatigue
Industrial tasks involving HRC often require workers to oversee multiple industrial operations simultaneously, coordinating different activities on the same production line [3,20], adapting to changing priorities when tasks are dynamically re-allocated based on real-time production [39], or performing precise and fne operations while keeping the pace with the productivity schedule [13].These situations have in common that the operator must often be aware of their own performance, of the robot's state, and of the overall production phase, likely inducing high workload, stress, and mental fatigue.These circumstances lead operators to experience psychological discomfort, slow the workfow, and increase the probability of running into errors or unsafe behaviors [5,11,40].For these reasons, monitoring psychophysical states in industrial HRC contexts is of crucial importance.
From a cognitive perspective, as industrial tasks often entail multiple operations simultaneously, operators can be required to retain procedural steps in their minds, inducing a burden on their working memory.This cognitive system is responsible for temporarily holding information in mind and manipulating it, and can be simulated in lab environments by introducing secondary mental tasks.The concurrent execution of primary (e.g., industrial assembly) and secondary tasks (e.g., mental counting) is known as dual-tasking or multitasking and stands out as a robust paradigm for studying workload dynamics both in HRC [14,20,27,28] and the broader literature [8,15,21,35,37,42].Among the secondary tasks, arithmetic tasks [17,35] are known to induce psychological (e.g., perceived demand and mental overload) and behavioral efects that ft those observed in work environments (e.g., worsening of job performance, undertaking unsafe behaviors) [23].Therefore, by intensifying the required cognitive resources for concurrently performing primary and secondary tasks, dual-tasking helps transpose conditions of high mental fatigue that may occur in industrial HRC contexts into the lab [26,35].To test the efectiveness of dual-tasking manipulations, self-report measures can help detect conscious psychophysical states after the execution of a workfow [11,12], and their utility, particularly for industries, may be empowered by physiological measures operating unobtrusively on a more unconscious level [35,41].

Technical set-up
The assembly workstation (Figure 1) was equipped with the UR10e collaborative robotic arm (6 DOF, 12.5 kg Payload).Designed to be height-adjustable, it was set about 15 cm below the fexed arm's elbow.The cobot, programmed in Polyscope (version 5.11), displayed task instructions on a touchscreen connected to the workstation.The work area featured two green and two red metal plates, and fve boxes on the table containing screws, bolts, two black masks, two transparent masks, and metal pieces of various shapes and colors.Two ambient cameras (VHD-V61CL FullHD) connected to a DELL PC (model XPS 2720, Intel Core i7-4790S processor, screen resolution 2560×1440, RAM 16 GB) recorded video footage for performance analysis using BORIS (version 8.20.4).Data processing and analysis were conducted in R Studio [33].

Task and procedure
A collaborative assembly task was designed following the framework presented in the introduction section.Following literature [36], we incorporated manual operations like manual screwing [12] (i.e., A in Figure 2), along with common cobot-assisted operations, such as handling [34] and pick-and-place [9] (i.e., composition B), and using industrial tools like an electric screwdriver [22] (i.e., supported screwing C).All operations were integrated into a cohesive workfow that composed a single trial.• Manual screwing (A): the participant prompted the cobot to fetch a green plate and place it on the worktable.They then selected six screws, inserted them into the plate's slots, and prompted the cobot to rotate the plate 90 degrees.After tightening the bolts, the cobot released the plate onto the workstation.Finally, they placed the completed plate into the designated housing.• Composition (B): the participant covered the green plate with a black mask, picked the metal pieces from the three boxes one at a time and placed them into the black mask, replicating a specifc confguration.The cobot assisted by lifting the boxes sequentially for accurate piece selection.Finally, the participant placed a transparent plate over the black mask.• Supported screwing (C): the cobot delivered a red plate to the participant, who assembled it onto the workpiece.After choosing six screws and inserting them, the participant instructed the cobot to position the workpiece under the screwdriver.This enabled them to tighten the screws and fnish the workpiece.The cobot then moved the workpiece away from the screwdriver, allowing the participant to take it of the workstation.
The operations in each trial were randomly ordered but structured to maintain their sequential integrity.While the starting and ending sub-tasks varied, the sequence remained consistent, with sub-task B always following A, and so on.Possible combinations thus included ABC, BCA, and CAB, preserving the sequential fow.Additionally, trial combinations were randomized for each participant, who performed trials in varied orders like BCA, CAB, ABC, or other permutations.This randomization helps assess the consistency of participants' psychophysical reactions across various operation orders.
The entire experiment took about one and a half hours.Participants started with a training session to learn safe and unsafe cobot interactions and familiarize themselves with the task.If well executed, this step also helps minimize the probability that participants will need the experimenters' assistance during the task recordings.The experiment then comprised three single-task and three dualtask trials.Text-and fgure-based instructions were displayed on the touchscreen desktop at the workstation.In the single-task condition, participants focused solely on the assembly task.In the dual-task condition, they performed the assembly task while continuously subtracting 3 from 1022, restarting the subtraction from 1022 in case of errors.

Measurements
The time on task was used as a measure of human performance by computing time spent on every operation through video analysis.Following the single-and dual-tasks, we then evaluated participants' workload using the NASA-TLX questionnaire [15] and afective reactions using the Self-Assessment Manikin (SAM) [4].Furthermore, acceptance and engagement were evaluated using a revised Technology Acceptance Model (TAM) designed specifcally for cobot interactions [31] and a User Engagement Scale (UES) [29].Finally, participants were asked to what extent they believed the experimenter contributed to their performance during the assembly task.They were prompted to rate it on a slider from 0 to 10.This question was included to gauge participants' perception of the experimenter's infuence on the task execution.

Statistical analysis
SAM, NASA-TLX, and time on task were analyzed using Generalized Linear Models (GLMs) in Rstudio [33].The models included the factor Condition (single-task, dual-task).SAM and NASA-TLX models also incorporated the factor Dimension, including the levels Mental demand (MD), Temporal demand (TD), Physical demand (PD), Performance (P), Efort (E), and Frustration (F) for the NASA-TLX, and Valence, Dominance, and Arousal for the SAM.Time on task models for each operation included the three-level factor Order (1 if the operation was executed as frst, 2 as second, 3 as third last operation).The participant was set as a random efect, and post-hoc tests were adjusted using the Bonferroni correction [2].
Perceived experimenter support.On a scale from 0 to 10, our participants rated the perceived support of the experimenter as 1.23 on average in the single-task (SD=1.88),and 1.58 in the dual-task (SD=1.73).

DISCUSSION AND CONCLUSION
Results showed how our dual-tasking, burdening working memory, signifcantly increased perceived workload during the assembly task (Figure 3), it was perceived as less pleasant and more activating, and induced a lower sense of control.The heightened arousal indicates increased engagement and attention required to handle multitasking.Conversely, the decreased pleasantness suggests that the overall task experience and interaction with the cobot may be less satisfying or rewarding when operators engage in more cognitively demanding tasks.This also appears to reduce their sense of control or dominance, which is known to be associated with psychological and physiological stress [32].
Notably, despite efectively simulating a psychologically demanding condition, the dual-task did not impact task execution time.Possible explanations include participants exerting efort to maintain performance levels or employing adaptive strategies to compensate for increased cognitive demand [25].A detailed qualitative analysis of operators' behaviors during the task execution, possibly along with other psychophysical measurements, is needed to confrm these hypotheses and better understand variations in task strategy.Remarkably, with careful randomizations, our study design enabled time-on-task analyses without disrupting the dynamic task fow, aligning with our goal of creating a realistic yet controllable experimental task.
In line with our second goal, results also indicated a particularly low perception of the experimenter's contribution to the participants' performance.This aligns with our aim to conduct a standardized, safe, and controlled experiment without perceiving any infuence of the experimenters with the task execution, resembling ecological conditions where operators handle industrial tasks unassisted.
While the research was conducted in a semi-controlled laboratory setting, its signifcance lies in laying the groundwork for future assessments in real-world feld settings.Our participants performed a realistic task with an uninterrupted workfow and without feeling any interference from the experimenters with the task execution.Additionally, we integrated load and multitasking conditions, often faced by workers in the industrial sector, and successfully detected diferences in workload and afective reactions as reported by the participants.The next step consists of integrating various psychophysiological measurements possibly on actual operators to test whether such psychophysical variations can be detected accurately and unobtrusively during the execution of such dynamic tasks via diferent sensors (e.g., eye, heart, or brain indices) in the lab, and most importantly in the industry.

ACKNOWLEDGEMENTS
This study was carried out within the PNRR research activities of the consortium iNEST (Interconnected North-Est Innovation Ecosystem) funded by the European Union Next-GenerationEU (Piano Nazionale di Ripresa e Resilienza (PNRR) -Missione 4 Componente 2, Investimento 1.5 -D.D. 1058 23/06/2022, ECS_00000043).This manuscript refects only the Authors' views and opinions, neither the European Union nor the European Commission can be considered responsible for them.

Figure 1 :
Figure 1: Technical set-up from the participants' view on the left, and from above on the right

Figure 3 :
Figure 3: Boxplots and data distribution of the participants' afective state (SAM) and workload (NASA-TLX) in each task condition (single-task, dual-task).