Comparing a Mid-air Two-Hand Pinching Point-and-Click Technique with Mouse, Keyboard and TouchFree

Some of our daily activities are performed by interacting with public touchscreens, such as food kiosks, bank tellers and newsstands. Nonetheless, the physical contact with these screens that are used by different people may be considered unhygienic. To avoid contact, some screens already integrate one-hand contactless interaction technologies, i.e. Leap Motion Controller, though they may lead to arm fatigue and slow performance. We present LeapPointer, a mid-air two-hand pinching point-and-click technique. Specifically, this technique relies on a Leap Motion device to track both hands, and proposes a new software tool that allows bimanual selection through pointing and pinching gestures. A user study was performed to compare LeapPointer with two other techniques: the common mouse/keyboard and the current UltraLeap’s TouchFree technique. Task completion time and accuracy as well as subjective data were gathered. The analysis of these data suggested that LeapPointer is significantly faster than the other touchless technique although less accurate. Self-reported fatigue was less with LeapPointer than with TouchFree.


INTRODUCTION
Pointing and selection are two of the most commonly performed actions when using a computer interface.Pointing and selection in a computer are usually performed with a mouse.The techniques for pointing and selecting could be divided into contact and mid-air (also known as contactless) methods.In contact techniques, the user performs actions by physically interacting with the device [3] whereas in the contactless techniques actions are interpreted using a camera, ultrasound or other gadgets that do not require contact from the user.One of the advantages of the later relates to hygiene, as contact devices are prone to accumulate microorganisms [7] that may contribute to disease spread [9].
There are numerous studies about pointing [4] and selection techniques [1] in 3D environments.Markussen et al. proposed a single-hand technique using Optitrack [10].Regarding contactless techniques, Yi et al. came up with a mid-air bimanual technique for keyboard typing, though it leads to fatigue [11].Similarly, several studies have reported that Leap Motion commercial techniques lead to fatigue [5,6].
We present LeapPointer, a touchless bimanual technique based in gestures that are captured using a Leap Motion Controller.The software used in the project is covered in Section 2.2.To show the position and gestures of the hands, we have developed an application in Unity that captures the hands position, analyses the gestures to execute system actions, and draws a representation of the hands and the actions in the screen.
We have carried out a user study where we covered two tasks: pointing circles and text entry.Three techniques were compared in the studies: a contact technique, mouse and keyboard; an already existing commercial contactless technique, TouchFree; and our technique, LeapPointer.The details about the user study are covered in Section 3, including the collected objective and subjective data, which serves as the basis for the conclusions covered in Section 5. Lastly, in Section 6, we discuss limitations and future improvements of LeapPointer.

SYSTEM
LeapPointer is based on a commercial hardware, Leap Motion, and a software technique that we developed.The complete system is shown in Figure 1.The system is composed of a commercial hardware, Leap Motion (highlighted in green), and a developed software that attaches an interactive cursor to each hand.

Hardware
Leap Motion is a device composed of two infrared cameras with software that enables hand tracking.To use this controller, it just needs to be connected to a computer by USB.When executing its software, the system reconstructs a 3D skeleton of the hands in real time, making it possible to track both hands.

Software
Taking advantage of the Leap Motion tracking, LeapPointer has been developed as a way to interact with a computer using both hands, mapping the actions of the hands into the common actions of a mouse.This means, making selections by pointing and pinching gestures.The technique achieves an interaction similar to leftclicking a physical mouse, but with two active cursors to choose from (one at each hand).
LeapPointer consists in showing on screen the models of the tracked hands with a cursor in the form of a small sphere between the index and the thumb fingers.The cursor is coloured to show the system's state: if the hand is enabled (white), disabled (black) or clicking (blue).

Functionalities.
• Activation: the hand needs to be rotated 90 degrees along the Z axis (forearm).After that, the hand should be rotated back, into the initial position.The sphere changes from black to white to indicate the correct activation of that hand, as shown in Figure 2. • Pointing: the cursor follows any of the two hands, pointing at the virtual sphere that is between the index and the thumb.By default, the cursor follows the sphere of the right hand.• Clicking: to click at the position of the sphere, the index and the thumb should close, performing a pinching gesture.This gesture performs the same action as left-clicking with a physical mouse.Users can pinch with both hands, to click on separate locations of the screen.The click will remain active while the pinching is closed, and released when the fingers separate.This interaction is shown in Figure 3.
• Deactivation: the cursor of any of the hands can be disabled, so that the pinching has no effect.Having hands disabled is convenient for avoiding accidental clicks if the user wants to use the mouse or keyboard.The gestures for deactivation is the same as for activation.

USER STUDY
A user study was conducted to measure the interaction techniques on two basic tasks.The study consisted in using 3 different interaction techniques: the bimanual contactless technique named LeapPointer, the UltraLeap's contactless technique named Touch-Free and the common input technique with a mouse or a keyboard.Two tasks were performed by the user: a pointing task and text entry task.The pointing task consisted in clicking on a coloured circle each time it appeared on the screen in a random position, for a total of 29 circles.The writing task consisted in typing 6 words in a virtual keyboard for the contactless conditions, and in a physical keyboard for the other condition.A summary of the user study, indicating all techniques and tasks, is shown in Figure 4. of them presented physical or visual impairments.A LeapMotion controller, a mouse and a keyboard were used in addition to a computer and its screen monitor.Before starting the experiment, the participants received instructions about the techniques to use and the tasks to complete.The conditions were counterbalanced using a latin square to avoid order bias.Participants took an average of 15 minutes to complete the study, including the explanation time.

RESULTS
The measurements from the user study were analysed using ANOVA repeated measures to detect significant effects of the conditions.In addition, posthoc tests with Bonferroni correction were used to determine significant differences.

Objective Data
Objective data were analysed in terms of Task Completion Time (TCT) and accuracy of the participants while completing the proposed tasks using each of the three techniques.As can be seen in Figure 5, users completed both tasks in less time using the mouse or keyboard.Comparing between the two contactless techniques, LeapPointer is significantly faster than TouchFree in both tasks.In fact, this difference is even more significant in the case of the writing task.
The results for accuracy imply that the mouse and the keyboard are significantly more precise than the other two techniques for the pointing task, and more precise than LeapPointer only for the writing task.Focusing on the contactless techniques, TouchFree is more accurate than LeapPointer, though this difference is only significant in the case of the pointing task.

Subjective Data
Subjective data were gathered using NASA-TLX [8] and SUS questionnaires [2].These questionnaires contained 16 questions (6 from NASA TLX and 10 from SUS).They were given to the users after completing the experiments.Each of the questions was a sentence to which the user had to express their agreement for each of the three techniques, scoring it with a number between 1 (I totally disagree with the statement) and 7 (I totally agree with the statement).
The questions from the NASA-TLX questionnaire were as follows: Q1-The mental effort necessary to use this method is huge; Q2-The physical effort necessary to use this method is huge; Q3-The dynamic of the activity has been very fast; Q4-I have felt successful at doing the activities; Q5-I had to put lots of effort in order to do the activities; Q6-I felt frustrated while doing the activities.The questions from the SUS questionnaire were: Q7-I think that I would like to use this system frequently; Q8-I found the system unnecessarily complex; Q9-I thought the system was easy to use; Q10-I think that I would need the support of a technical person to be able to use this system; Q11-I found the various functions in this system were well integrated; Q12-I thought there was too much inconsistency in this system; Q13-I would imagine that most people would learn to use this system very quickly; Q14-I found the system very cumbersome to use; Q15-I felt very confident using the system; Q16-I needed to learn a lot of things before I could get going with this system.
The obtained results for the questionnaires are shown in Figure 6 (NASA-TLX) and Figure 7 (SUS).As can be seen, there are several reported differences between the techniques.For instance, from the contactless techniques, users find LeapPointer less physically demanding than TouchFree, and overall less demanding.Moreover, they find LeapPointer faster to use than TouchFree, as well as less frustrating.As a final overview, users show more interest towards using of LeapPointer than TouchFree in their daily interactions with touchscreens.

CONCLUSIONS
The commonly used mouse and keyboard are the fastest of the three interaction techniques.They also provide the best accuracy when performing pointing and writing tasks.In the case of the contactless techniques, LeapPointer is faster than TouchFree, yet it is less accurate.In conclusion, depending on the tasks to be  Agreement with the statement LeapPointer Mouse/Keyboard performed, time and accuracy should be considered to choose a preferred technique.

LIMITATIONS AND FUTURE WORK
LeapPointer has been evaluated by a preliminary user study.Further evaluation is needed to better refine the technique.It could be appropriate to conduct another study with more users, focusing on a wider participants background, as only evaluating device interaction with users related to computer science causes biased results.Related to the technique's performance, some aspects may need to be improved.For instance, incorporating haptic feedback into the system could make users more aware of the distance between their index and thumb fingers when performing pinching gestures.Specifically, having haptic props attached to the hands, such as clamps, tweezers or rubber marbles, could improve the pinching interactions.The use of intelligent materials with hardness variability could be explored.In this way, the task to be performed would be directly linked to the softness or hardness of the prop.For example, dragging a large file would provoke a hard sensation in the fingers when selecting it.
Another important feature to address is the precision of the system.As explained before, LeapPointer is significantly faster than the other contactless technique, but it is also less accurate.To tackle this issue, the system could incorporate new methods to keep the cursor still when no action is being performed, avoiding user's hand shakiness.
In terms of functionalities, other methods could be implemented to expand the system's interaction possibilities beyond pointing and selection.For example, automatically displaying a virtual keyboard or copy-pasting some text when some gestures are performed.

Figure 1 :
Figure 1: The system is composed of a commercial hardware, Leap Motion (highlighted in green), and a developed software that attaches an interactive cursor to each hand.

Figure 2 :Figure 3 :
Figure 2: Activation gesture: A) The black sphere indicates no interaction of that hand with the system; B) Rotation of 90 degrees around the Z-axis; C) Rotation of 90 degrees in the opposite direction around the Z-axis (going back to position in A) and the sphere changes to white, indicating that the clicking on that hand is enabled.

Figure 4 :
Figure 4: User study.Pointing task and a writing task using three different techniques: a mouse/keyboard, LeapPointer and TouchFree.

Figure 5 :
Figure 5: Objective measurements split by condition and task.A) Task Completion Time (TCT).B) Accuracy.Error bars indicate standard error.