Electromyography-based Kinesthetic Teaching of Industrial Collaborative Robots

Current methods for robot teaching still lack intuitiveness and efficiency or require instrumentation of the environment (e.g., with cameras). This poses problems, especially for companies that cannot afford dedicated robot programmers. Classical online teaching with a teach pendant (TP) can be tedious and confusing, and kinesthetic teaching (KT) is often perceived as inefficient since toggling gravity compensation mode forces users to switch between interfaces or constrains them physically. We propose a novel robot teaching method that allows users to activate gravity compensation mode, confirm positions along trajectories, and manipulate the end effector. In our system, this is done via hand gestures %%that occur naturally while handling the robot and that we detect by equipping an operator with a wearable electromyography (EMG) armband. To evaluate our system, we compared it to a commercially available KT system in a user study that yielded statistical evidence that our approach is significantly faster while no difference regarding the perceived usability of the systems was found. Additionally, expert interviews confirm that the baseline system is state of the art and confirmed the market potential of EMG-based KT. Finally, we confirmed that the gesture classifier does not need to be re-trained for each user, which makes the proposed system a %%highly interesting option in practice that is cost-effective, efficient, and provides high teaching ergonomics.


INTRODUCTION
Use of robots across industries is steadily increasing, as companies strive to increase efciency and quality of production [1].Flexibility then is a key factor, especially given small lot sizes in companies that ofer mass-customized products, or in small-and medium-sized enterprises (SMEs) [2].To fexibly reprogram robots, companies often adopt ofine programming by trained employees [1].However, especially SMEs typically lack the required expertise and therefore cannot beneft from this fexibility [2,3].In such cases, efcient programming methods tailored to the use by untrained employees are required.A robot programming method which fulflls these requirements is kinesthetic teaching (KT).With KT, users manually program a robot by moving its end efector (EE) to desired positions in teach mode [4].This method simplifes robot programming for novices and non-experts [5] but lacks efciency: Switching between teach and standard modes typically requires the operator to either switch between interfaces or to use a body part exclusively for toggling modes [6,7].To overcome these shortcomings videobased gesture recognition methods have been proposed.These methods address the efciency related challenges, but require an environment which is equipped with cameras.We therefore propose EMG-based KT which enables gesture based teaching of robots without augmentation of the environment.

BACKGROUND AND RELATED WORK
A major barrier in robot programming is the limited available workforce with robot-programming know-how [3].Therefore, solutions that enable less qualifed individuals to program robots are needed [8].Ofine programming using computer-aided tools, which require specialized knowledge, is not always suitable for nonexpert users [9].On the other hand, online programming methods, such as a teaching-playback method with a teach pendant (TP) [10], through KT [4] or through graphical programming interfaces [9] allow users to intuitively guide the robot's EE to the desired positions.However, as programs become longer and more complex, the intuitiveness of the visual abstraction vanishes [9].In this context, EMG has the potential to improve such intuitiveness when combined with hand gesture recognition [11,12].

Teleoperation with a Teach Pendant (TP)
A popular approach to teaching robots is teleoperation, which does not require physical interaction between the operator and the robot.
Teaching via teleoperation typically uses a hand-held device-a TP [9].
With these TPs, robot's EEs can be moved to desired positions via a teaching-playback method, recording trajectories and waypoints for later execution.This method is suitable for simple tasks without overly complex geometry which are repeated frequently [13]-for more complex tasks, TPs support textual or visual programming languages.However, these approaches require knowledge about the robot's coordinate system and are therefore not easy to use for non-experts [4,5].Furthermore, given the low working efciency, TP-based teaching is not optimal in environments where tasks need to be reprogrammed frequently [10].

Kinesthetic Teaching (KT)
KT enables users to teach a robot without knowledge about the robot's coordinate system and without using a textual or visual programming language [13].This is achieved by using a robot's teach or free-drive mode, which enables users to manually guide the EE to desired positions [4].The waypoints or the complete path are then stored for playback [9].Most manufacturers today ofer KT on their collaborative robots [14].KT is intuitive and efcient, especially for teaching pick-and-place operations [15].It has been shown to be appropriate for teaching larger movements in contexts where it is safe for users to enter the robot's workspace [16].Activation of the teach mode is handled diferently by diferent manufacturers: Mostly, users are required to either switch between interfaces (e.g. a tablet and the robot), which is time consuming, or to press an enabling switch, which constrains them physically, as one hand is constantly bound to pressing the switch.On the Universal Robots e-Series [7] this mode is activated by holding an enabling switch on the TP device, and on the UFACTORY xArm series [6] it is activated from an external KT device or through Modbus.
From this survey of related work, it can be concluded that an efcient yet intuitively usable system to support KT, while allowing an inexperienced individual user to teach a robot with both hands and without being constrained by stationary devices (e.g., TPs, switches, foot pedals, etc.), is missing.

EMG-BASED KT
Aiming to overcome the above-discussed inefciencies while preserving the intuitiveness of KT, we propose EMG-based KT.This method enables users to teach a robot via intuitive hand gestures.To enable EMG-based KT, our system features a wearable EMG Capturing Device (see Sec. 3.1).The device is capable of wirelessly transmitting measured EMG data, so that users can move around freely during teaching.Furthermore, the system features a Gesture Classifer (see Sec. 3.2) that takes the EMG data as input and identifes the current gesture.Finally, our system contains a Robotic Arm (see Sec. 3.3) that is able to equip simple EEs.Our approach is distinguished from other research in this space (e.g., the method by Zhang et al. [17]) in that we do not use information from inertial measurement units but exclusively use EMG data.
To validate the proposed method, we developed a PoC system as depicted in Fig. 1 and performed several evaluations that we report about in this contribution.Finally, the robotic arm executes the command.

EMG Capturing Device
EMG data is measured using a commercial of-the-shelf EMG armband1 (see Fig. 2) and is sent to the EMG Collector via Bluetooth Low Energy (BLE).The armband collects EMG data through 8 electrodes which are attached around the forearm and measure electrical signals traveling through the forearm muscles at up to 200 Hz.The EMG Collector receives EMG data at a sampling frequency of 50 Hz and takes the mean absolute value and the standard deviation for each of the eight channels over a window of 10 consecutive measurements.This yields a 16 dimensional feature vector which is then transmitted to the Robot Controller at a rate of 5 Hz.

Gesture Classifer
The Robot Controller identifes user gestures by passing the feature vector transmitted by the EMG Collector to a -NN algorithm with distance metric.Subsequently, it sends appropriate commands to the robotic arm via the Robot Operating System (ROS) and the manufacturer SDK [18].
Our Robot Controller is able to identify four diferent gestures from received feature vectors and to map them to robot commands: (.) enter standard mode, (.) enter teach mode, (.) toggle EE, and (.) confrm position.We mapped four distinct gestures, which have been chosen in order to maximize intuitiveness.This is illustrated best through the mapping of the Grab gesture (see Fig. 2b): By grabbing the robot, the user activates the teach mode in which the robot's posture can be freely manipulated.Since the operator anyway needs to grab the robot to guide it to the desired position, this creates a seamless and highly ecologically valid integration between operator movement and robot instruction.Likewise, when the user relaxes their forearm in teach mode (Fig. 2a), the robot enters the standard mode in which the robot waits for commands from the Robot Controller.Relaxing the forearm in standard mode is associated with not performing any specifc action.The gestures to toggle EE (Fig. 2c) and confrm position (Fig. 2d) were selected so that the system can easily distinguish them from the other gestures (and from each other), due to difering muscle activation in the forearm.
To train the classifer, data from one of the authors was used.During 5 sessions conducted at diferent times, 300 data points were collected for each gesture and classifed using the -NN algorithm.To maximize average accuracy, we used each data set for validation while the other four were used for training, yielding an optimal = 15.

Robotic Arm and Controller
Our PoC system is adapted for the UFACTORY xArm7 collaborative robotic arm [6], which ofers

EVALUATION
We conducted a user study to evaluate the usability and efciency of EMG-based KT.In the study, we compared our EMG-based KT system with a commercially available KT system that comes with a visual programming interface.Specifcally, we compared the amount of time it took for participants to teach the robot a predetermined task with both systems.In addition, after completing the teaching task, participants flled out a System Usability Scale (SUS) [19] survey for both systems.The goal of this was to compare the overall usability of the two systems.In addition, we evaluated the robustness of the gesture classifer (see 3.2).Table 1 shows a summary of the participants' demography.All experiments were conducted in the laboratories of our institute.

Scenario: Pick-and-Place
For our study we designed the following simple pick-and-place scenario consisting of the following subtasks: () Move EE above the frst tennis ball on a ball feeder, () Open gripper to its maximum, () Move EE down to the tennis ball, () Close gripper to grab the ball, () Move EE (with ball) above ball feeder, () Move EE (with ball) above a tennis ball can, and () Open gripper to drop the ball into the can.

A Baseline KT System
For comparison, we set up a commercially available KT system using the xArm Studio software application provided by UFAC-TORY [6].This software provides a graphical programming interface that allows users to program an xArm by dragging and dropping code blocks based on Blockly [20].
In order to simplify the task for the experiment participants, we prepared a Blockly code template that consisted of all blocks required to complete the scenario described in Section 4.1 in the correct order.Therefore, in the experiment, the participants did not have to fnd and combine the right blocks in the right order but only edit the provided blocks (e.g., adjust the x,y,z coordinates of a block so that the EE navigates to the desired position).For editing, the experiment participants used the standard KT functionality that comes with the xArm7 and xArm Studio.

Methodology and Procedure
As a preparation, each participant received a printed overview of the task they should teach to the collaborative robot.The participant was then randomly assigned to teach the task described in Section 4.1 with one of teaching systems.Before starting, the participant received a manual for the respective system and could take as much time as needed to read the instructions and ask questions.
The time used to complete the task was measured from the time the participants signaled that they are ready to start the programming to the time they fnished teaching.After completing the teaching, they flled out a questionnaire with demographic questions, information about their self-perceived experience, and the SUS.The standard version of the SUS was chosen, as it has proven efective for comparing diferent interfaces with the same functional purpose [21].This procedure was then repeated for the second system.In each case, the following participant started the experiment with the other system than the participant before, so that a total of 11 participants started with EMG-based KT and 11 started with xARM Studio-based KT.

Results
The mean number of seconds and the mean SUS scores with standard deviation () from our study were compared via paired t-tests.Table 2 shows the results of this test for both EMG-based KT and xArm Studio-based KT.
4.4.1 Teaching Time.Our experiment shows strong statistical evidence ( = 0.002) that teaching a robot with EMG-based KT is faster than teaching with xArm Studio.This efect would likely be more accentuated if users were not provided with the Blockly template described in Section 4.3.Furthermore, in a real-world scenario no template could be provided for the xArm Studio-based KT.Therefore, users would need a basic understanding of programming in order to complete the task.With EMG-based KT this is not needed.

Usability.
No statistical evidence that the systems' SUS scores difer signifcantly was found.This result is in line with the authors' aspiration of increasing teaching speed while ofering usability which is comparable to a commercially available system.Although isolated SUS scores do not have the same meaning as their comparison, the extensive amount of available historical SUS data allows for the creation of a rating scale, on which EMG-based KT-with its mean SUS score of 76.7-would be considered within the acceptability range and an adjective rating above 'good'.It is noteworthy that one of the study participants ran into challenges when confrming positions with EMG-based KT. 4.4.3Classifier Performance.Our classifer yielded an average accuracy of 0.85, an average precision of 0.87, an average recall of 0.85, and an average f1 score of 0.85.For individual gestures accuracy varied between 0.91 and 0.97, precision between 0.73 and 0.91, recall between 0.70 and 0.99 and the f1 score between 0.79 and 0.94.For individual participants, the highest measured accuracy was 0.993 and the lowest 0.538, while the median was 0.88 (standard deviation: 0.12).This shows that in most cases our classifer is robust enough to be transferred to other users without retraining.Nonetheless, individually training the classifer for diferent users can be done quickly and improves the accuracy (e.g., retraining the classifer for two participants with low accuracy resulted in improvements from 0.538 and 0.686 to 0.973 and 0.977, respectively).

EXPERT INTERVIEWS AND DISCUSSION
Several limitations in our study need to be considered: First, it is not certain that the results obtained by comparing EMG-based KT to xArm Studio-based KT can be generalized to TP and other competing systems.Second, the study was conducted on novice users, which raises the question of whether the results can be transferred to experienced users.Therefore, we interviewed four experts in robot teaching: Three application technicians and a robotic system integrator, who are specialized in robot teaching and programming.
These experts were asked general questions about robot teaching and (collaborative) robots in industry.Thereby, they confrmed that the baseline system (see 4.2) used in our study is comparable or slightly superior to current industry standard.
Subsequently, we demonstrated EMG-based KT and the experts tested the system themselves after brief instruction.Based on these trials, the experts confrmed that the system could enable nonexpert users to teach robots on their own.According to them, one large advantage of EMG-based KT is the possibility to use both hands to manipulate the robot during teaching, since comparable robots that they were familiar with required holding an enabling switch, leaving only one hand for teaching.Overall, they believe that the novel system possesses potential in industry sectors where collaborative robots need to be frequently reprogrammed.Furthermore, they identifed potential for leveraging the collected EMG data to teach robots for force-sensitive applications (e.g., handling of sensitive goods).

CONCLUSIONS AND FUTURE WORK
EMG-based KT enables intuitive, efcient, and cost-efective robot teaching.The mapping of simple gestures to diferent functions allows KT with both hands and without having to switch interfaces to toggle gravity compensation mode.We conducted a study where novice users taught the same task to a 7-DoF collaborative robotic arm using EMG-based KT and on a commercially available baseline system.With EMG-based KT, teaching was signifcantly faster than on the baseline system while a comparison of the usability scores yields no statistical evidence of superiority of either of the systems.These results were confrmed by industry experts.
Further improvements to EMG-based KT should be the target of future work: A more accurate classifer can improve the gesture recognition, enabling the introduction of new gestures mapped to further functions.Finally, tailoring the system to force-sensitive use cases holds, according to experts, signifcant potential for an EMG-based KT system.

Figure 1 :
Figure1: EMG-based KT: A user performs hand gestures and wears an armband which collects EMG data.This data is transmitted to a Laptop via BLE.There, an EMG Collector samples the data, extracts features and classifes the gesture performed by the user.Subsequently, a robot controller transmits a robot command mapped to the identifed gesture via ROS to a server.This server forwards the command via Modbus/TCP to the control box of the robotic arm and plays audio feedback to inform the user about the state of the system.Finally, the robotic arm executes the command.

Figure 2 :
Figure 2: Our implementation of EMG-based KT features four gestures to teach a robot.

Table 2 :
Results of the paired t-tests.