Designing Haptic Feedback for Sequential Gestural Inputs

This work seeks to design and evaluate haptic feedback for sequential gestural inputs, where mid-air hand gestures are used to express system commands. Nine haptic patterns are first designed leveraging metaphors. To pursue efficient interaction, we examine the trade-off between pattern duration and recognition accuracy and find that durations as short as 0.3s-0.5s achieve roughly 80%-90% accuracy. We then examine the haptic design for sequential inputs, where we vary when the feedback for each gesture is provided, along with pattern duration, gesture sequence length, and age. Results show that providing haptic patterns right after detected hand gestures leads to significantly more efficient interaction compared with concatenating all haptic patterns after the gesture sequence. Moreover, the number of gestures had little impact on performance, but age is a significant predictor. Our results suggest that immediate feedback with 0.3s and 0.5s pattern duration would be recommended for younger and older users respectively.

Meanwhile, the emergence of mixed reality, powerful mobile devices, and wearable devices highlights the need to explore new interaction paradigms that support fexible virtual operations in 3D space.In this work, we focus on the application of mid-air gesture interaction and examine the communication performance of vibrotactile feedback delivered via wristband form factor.We foresee this interaction modality as a novel gesture-haptic UI, where gestural inputs and haptic feedback can be tightly coupled around the hand.More specifcally, as a complement to traditional graphical and voice UIs, we envision using this gesture-haptic UI for fast and convenient interactions across a wide range of applications, such as interacting with smart glasses while walking on the street, issuing commands in mixed-reality environments, or controlling the car's interface or mobile phone while driving.
Compared with other forms of hand-based input using touchscreens and hand-held controllers, mid-air hand gestures enable fexible remote interaction [5,22,93], and can be used for complex commands [74], language production [33], and 3D object manipulation [26] with less interruption [27,71].Moreover, although visual and audio feedback have been widely applied [38], these feedback modalities can be less efective or unavailable during many activities of daily life, especially when interacting with wearable devices, such as wristbands [64,80], smart rings [2,28,73], or smart glasses [68].In addition, visual and audio feedback could be distracting or overloading during information-rich tasks such as gaming or driving [39,82].Further, haptic feedback indeed aligns very well with hand-related interactions as tactile sensation and gesture performance both rely on the somatosensory system [25] and can mimic the tactile sensations we readily perceive when manipulating tangible objects.
In this work, hand gestures were utilized to express shortcut system actions.More specifcally, we intend to explore sequential gestural inputs that can be used for complex commands.Distinct haptic patterns are coupled with the actions/gestures as feedback for users to confrm the system response without adding more visual or auditory load.Similar to keyboard shortcuts, shortcut actions expressed by gestures can speed up interaction by completing diverse functions and complex tasks using a single gesture [47,61,62,76].Due to the increased cognitive load required to learn more gestures and their mappings, prior research has investigated grouping sequences of gestures together to improve the communication bandwidth [15,57,93].However, only the input design was explored and evaluated in those studies, leaving the feedback design unexplored for sequential inputs.
Using haptic feedback as a communication modality has been thoroughly investigated across many application felds, such as encoding movements and directions using spatiotemporal patterns [11,35,59,66], or conveying semantic meaning of interface actions [6,43,95], emotion or social messages [4,36,72,75,104], and speech-or text-related information [18,37,50,70,92].These applications, however, only deployed haptic feedback for systeminitiated interaction, where users passively received and decoded haptic messages.During user-initiated interaction, such as gestural inputs, not only is recognition accuracy crucial for providing satisfying user experiences, but so too is ensuring efcient feedback to not slow down the communication fow.Especially for sequential inputs, fuid interaction would be preferential without waiting for prolonged haptic patterns to playback between inputs.
As a step into investigating haptic feedback for sequential gestural inputs, in this work, we designed and compared diferent haptic patterns to achieve efective human-system communication with high efciency.We explored the haptic rendering space of a wristband with four vibrotactors equally spaced around the wrist.Haptic patterns were frst designed for predefned shortcut actions by leveraging metaphors to make them memorable and recognizable.To pursue efcient interaction, we then examined the trade-of between recognition accuracy and haptic pattern duration in single gesture interaction to identify the minimal duration without sacrifcing recognition accuracy.For sequential gesture integration, two feedback strategies were frst proposed: (i) the immediate feedback strategy was designed to send the haptic pattern immediately after each gesture was detected, and (ii) the concatenated feedback strategy was designed to send all haptic patterns after all gestures of a sequence were performed and detected.In addition to the feedback strategy, three more factors were also considered for the evaluation of sequential gesture interaction, which includes pattern duration (derived from single gesture interaction), the number of gestures in the sequence, and the age of users.By examining both recognition accuracy and overall interaction time, haptic design recommendations were fnally derived for two groups of users based on age ("younger user" < 45 and "older user" >= 45 years old), respectively.
The main contributions of this work are: • Metaphor-based haptic patterns rendered on the wrist to provide efective feedback for gesture-delivered shortcut commands.• The identifcation of the trade-of between recognition accuracy and haptic pattern duration when expressing semantic meanings using four vibrotactors.• Haptic design recommendations for sequential gesture interaction for younger and older users.

RELATED WORK
This work relates to prior literature on the perception of vibrotactile feedback while varying haptic rendering parameters.We also overview the designs of vibrotactile feedback in encoding and conveying semantic messages.

Perception of Vibrotactile Feedback
Within the context of vibrotactile feedback, its perceptual characterization has been carefully explored by tuning the signal parameters, including frequency, amplitude, rhythm, waveform, location, and duration [6,9,20,32,35,49,51,66,89,95].Amongst those studies, many would co-vary multiple signal parameters to increase the pattern distinctiveness [6,49,89].Meanwhile, it was recommended that fewer levels (usually 2-3) should be applied for each one of the covaried parameters to achieve better recognition performance [9,95].Moreover, spatiotemporal haptic patterns have been reported to outperform solely spatial patterns with changing parameters [59].Two metrics have been mainly employed to evaluate the role of vibrotactile parameters on perception, i.e., recognition rate and multidimensional scaling (MDS).MDS is often applied for identifying perceptual similarity amongst diferent stimuli [8,42,49,63,89,95], while recognition rate is more appropriate when examining the efectiveness of conveying predefned messages.More specifcally, the time duration of vibrotactile feedback typically varies between hundreds of milliseconds to several seconds.For motion efects rendered by vibrotactor arrays [35,99], the recognition accuracy of single-direction motions was reported to increase along with the pattern duration, which achieved roughly 70% for duration of 135 ms [59].Research on pattern speed of high-frequency lateral skin stretch has also shown that faster speeds with shorter pattern durations lead to lower recognition rates [49].For applications such as encoding language phonemes with vibrotactor arrays, 100 -400 ms and 220 -480 ms were employed for representing consonants and vowels respectively [70,92].Longer durations of 1 -2 seconds have also been employed for haptic patterns delivered by hand-held single-actuator devices [20,34,51,78,89].However, for more complex semantic meanings, such as system commands, how haptic pattern duration infuences the communication efectiveness is seldom reported, especially for the aim of shortening the pattern duration.
Another popular strategy of encoding semantic meanings using haptic feedback is to leverage metaphors [3, 6-8, 42-44, 81, 83], such as the pulses of heartbeats [42,72].Metaphors associated with haptic patterns are mental models that harness users' pre-existing experiences, e.g., audio, visual experiences, or other associations, with their encoded meanings to lower the cognitive load necessary to understand, remember, and retain their patterns.Prior work has also proposed pre-designed tacton databases [83] and design guidelines [3,43] to aid in the process of designing metaphor-based haptic patterns.
Emotion-and social-related messages are also common concepts conveyed by haptic devices, such as haptic sleeves [79], gloves [72], and wristbands [75].Indeed, afective social touch [53,54] is one of the major functionalities of our sense of touch that maintains social connections and provides emotional support [94].As emotional feelings of valence and arousal have been reported to be evoked by haptic signals with varying haptic parameters [84,103,104], encoded emotional messages could be intuitively recognized if haptic patterns trigger emotional feelings that match the messages, e.g., mapping gentle, comfortable haptic patterns to positive meanings.
While the aforementioned haptic messages were designed for system-initiated interactions, several studies have also focused on the haptic pattern as feedback for user-initiated input [45,80,91].However, they only provided basic haptic feedback for single inputs and did not evaluate the recognition performance.In this work, we seek to examine the communication efectiveness and efciency of haptic feedback for gesture-issued commands.Sequential gestures were focused to test the performance evolving with the increase of the input complexity.

HAPTIC PATTERNS FOR SHORTCUT ACTIONS
The shortcut actions were selected in our study to cover basic system commands that are frequently used across diferent systems and applications.We then narrowed them down to the ones appropriate for the gesture-haptic interaction scenario.Note that both action and gesture developments were not the main focus of this work.A subset of nine actions was fnally selected for user study experiments to explore the design of haptic patterns for single and sequential gestural inputs.To simplify the implementation of gesture detection, hand gestures were designed to be expressed by right hands only.

Shortcut Actions and Gestures
Nine shortcut actions and their corresponding hand gestures (Figure 1) were designed based on previous studies [56,67,102] and refned through internal discussions with UI and UX design experts.Similar to keyboard shortcuts and GUI buttons, the Confrm action would accept the previous action expressed by users or recommended by the system, whereas Cancel would reject the previous action.Inspired by prior research on user-defned hand gestures in augmented reality [67], thumbs up and thumbs down were assigned to the Confrm and Cancel actions, respectively.Save and Share actions were also related [102], where Save would be used to save fles, extracted information, or virtual targets locally, while Share would be used to share them with other users.Hand gestures for multi-touch tabletops have already been proposed for Save and Share actions [102] as "catching", i.e., quickly sliding a single point toward oneself, and "ficking", i.e., quickly sliding a single point away from oneself, respectively.As we implemented gesture tracking for frame-by-frame hand poses instead of hand movements, we revised these two gestures as a fst for Save and an open hand for Share.Capture was the action to take a picture, which would be convenient especially for controlling smart glasses.The symbolic gesture [56] of "pressing camera shutter" was utilized.
The Remind action would be used to set a reminder and its gesture incorporated a stretched thumb and index fnger to represent the hour and minute hands of a clock.It also utilized the metaphor that is associated with tying a string around one's fnger into a bow as a way to remember things.The Query action would be used to search for additional information amongst local fles or using a search engine and it was mapped to a hand gesture that resembled a question mark.The Tag action was designed for expressing social media reaction of 'like' or 'love' with the "mini-heart" hand gesture [100] applied to it.The Lucky action would be used to obtain recommendations from the system for potential next actions with the hand gesture of "fngers crossed".

Haptic Patterns
Haptic patterns were designed using an in-house wristband with four evenly distributed linear resonant actuators (LRA) [1] as vibrotactors (VT) (Figure 2).The resonance frequency of each LRA is 170 Hz.Among the nine haptic patterns, Save, Share, Remind, and Query were designed with motion efect by actuating each VT in a specifc order.For the rest of the haptic patterns, no motion efects were generated and all four VTs were actuated simultaneously.Metaphors were used when designing haptic patterns to increase their intuitiveness and memorability.Haptic pattern for the Confrm action, for example, consisted of one shorter and one longer two pulses to mimic the short and long strokes of a check mark.The haptic pattern for the Cancel action was designed to represent the feeling of throwing something away into a trash can and the trash bouncing in the trash can.Following the guidelines for triggering emotional feelings of valence and arousal using vibrotactile feedback [84,104,105], this pattern had multiple irregular peaks with high intensity to render an unpleasant feeling that is associated with canceling something [96].The motion efects [35,99] of moving inward and outward were applied to the Save and Share actions to represent "grabbing something toward oneself" and "sending something away from oneself".To facilitate the perception of direction, an intense and sharp peak was added to pinpoint where the movement ended, which was at the radial side and ulnar side  of the right wrist for Save and Share respectively.The haptic pattern for the Capture action mimicked the sound wave of pressing a camera shutter, i.e., a long decayed curve after two quick peaks.
The metaphor mapped to the Remind action was the "tick tock tick tock" sound of a clock, so four short, light pulses were emitted on the sides of the wrist with the alternation between left (VT-L), right (VT-R), left (VT-L), right (VT-R).To mimic the loading metaphor of a rotating circle for the Query action, the four vibrotactors were actuated successively in the clockwise order around the wrist.Inspired by the "like" and "love" meaning of the Tag action, a heartbeat metaphor was applied for its haptic pattern with two light continuous waves to generate a comfortable, pleasant feeling [96].For the Lucky action, a light bulb metaphor was applied for its connotation of "I suddenly have a good idea".Its haptic pattern had two light pulses followed by an intense pulse to represent the thinking process followed by an idea suddenly occurring.
A pilot study was conducted with eight participants including four haptic experts and four novices to test and iterate the haptic pattern design.Subjective feedback was collected about the selected metaphors and the generated haptic patterns.After multiple iterations of fne-tuning haptic parameters, including amplitude, frequency, waveform, location, rhythm, and shape modulation, we arrived at the fnal version of the haptic patterns described prior.As such, we designed each haptic pattern to be as distinct from each other as possible to ensure easy recognition and focused on the factors more relevant to the sequence design.

EXPERIMENT 1: PATTERN DURATION VS. RECOGNITION ACCURACY
The frst experiment was designed to understand how haptic pattern duration infuences participants' recognition accuracy.Four pattern durations of 0.1 s, 0.3 s, 0.5 s, and 0.8 s were adopted based on prior work [70,92] and a pilot study.As varying the duration of the same pattern would make it feel like a diferent one, pattern duration was set as a between-subjects variable.Three experimental phases were designed with the nine haptic patterns: self-paced exploration, learning, and testing [20,92].The experiment was conducted on the same day and took around one hour in total.Participants were informed to only memorize haptic patterns without memorizing hand gestures since pictures of hand gestures would be provided on the screen to guide each gesture performance.

Setup
Python GUIs were developed for the three experimental phases to provide instructions and collect responses.Hand tracking was implemented using the MediaPipe framework [48] in Python to extract 2D hand joint positions from the real-time video captured by a webcam.Hand gestures were detected using the relative joint positions and angles.For example, gestures with thumb pointing up and the other four fngers folded were detected as Confrm, and gestures with crossed index and middle fngers and the other three fngers folded were detected as Lucky.

Participants
Twenty participants were recruited aged 23 to 54 years (mean = 35 years, SD = 12 years; 13 male, 7 female).Participants provided informed-consent through an ethics protocol approved by the IRB at (anonymized), and were paid $75 for their participation in the study.Three participants reported expert-level experience in haptics; three participants reported moderate experience in haptics; seven participants reported limited experience; and seven participants reported no experience.Five participants were randomly assigned to each of the four pattern durations.

Experimental Procedure
Before the experiments, the facilitator explained the concept of gesture-haptic UI and the task of learning the mapping between haptic patterns and actions without needing to remember hand gestures.The hand tracking was then tested for participants to get familiar with the nine gestures.Participants were then instructed to put on the wristband on the right hand at approximately 3 cm away from the wrist joint.They were then asked to wear headphones that played pink noise to mask the sound of the vibration.

Exploration.
Nine buttons representing the nine actions were presented on the computer screen.When users clicked on a button, it would generate the corresponding haptic pattern on the wristband, and display the associated metaphor and haptic pattern visualization (Figure 3A) on the screen.Participants frst clicked on each of the nine buttons and perceived the haptic pattern while the facilitator explained the meaning of the actions and the metaphor linked to the haptic patterns.Afterward, participants put on headphones that played pink noise and began the self-paced exploration of haptic patterns by clicking on the nine buttons as many times as they wanted.Participants could end this phase when they felt ready to move to the learning phase.

Learning.
The nine haptic patterns were provided 45 times in a pseudo-random order with each pattern repeated fve times.
On each trial, the system sent the haptic pattern 250 ms after participants clicked on the 'Start' or 'Next' button.The haptic pattern could be felt only once.Participants then selected the action they deemed as correct.After submitting the selection using a 'Submit' button, they received visual feedback about the correctness of their selection.If the selection was incorrect, the correct action would be displayed together with the metaphor and hand gesture (Figure 3B).Regardless of whether their selection was correct or incorrect, a 'Replay' button was activated after the submission.Clicking on this button would resend the haptic pattern and it could be clicked as many times as wanted.By providing the correct answer and the option of replaying the haptic pattern, participants could reinforce the correct match between actions and haptic patterns.Participants could proceed to the next trial by clicking the 'Next' button.

Testing.
In the testing phase, a haptic pattern would be triggered by detecting only the instructed hand gesture performed by the participant.Nine actions were pseudo-randomly provided with each repeated four times, resulting in 36 trials in total.For each action with four repetitions, two correct and two wrong haptic patterns were randomly assigned.The wrong patterns were randomly picked from the rest of the eight haptic patterns.On each trial, the action and the corresponding hand gesture were shown on the screen (Figure 3C).A small real-time video was also displayed at the bottom left of the GUI showing the gesture detection results frame-by-frame.It was provided for participants to adjust their hand gestures if the hand was out of view or the gesture could not be detected immediately.After the instructed hand gesture was correctly performed by the participant and was detected by the system, either a correct or incorrect haptic pattern would be sent right away.
After receiving the haptic pattern, participants answered whether this pattern was correct or incorrect for the provided action by clicking the 'Yes' or 'No' button.Two more questions were answered by participants to collect subjective cognitive load for each trial.
No feedback was provided about the correctness of their answer.Afterward, participants could move to the next trial by clicking on the 'Next' button.Here, we designed the testing phase with yes-no questions instead of asking participants to select the correct action from the nine options.This approach better simulates the real-life application of this UI, where users receive the haptic pattern after performing the hand gesture and then identify whether it is the correct system response that they intended.

Results
The recognition accuracy was calculated for the testing phase and the last two repetitions of haptic patterns in the learning phase (Figure 4).A psychometric curve with a sigmoid curve cumulative Gaussian distribution function was ft to the testing and learning data using the Psignift toolbox in MATLAB [98].For the pattern duration of 0.8 s, participants achieved high recognition accuracy with the median of 92% during the testing phase, which was much higher than the chance level of 50% for two-option questions.A similar recognition accuracy with the median of 94% was found for the learning phase relative to the chance level of 11.1% for nine-option questions.This indicates that the designed nine haptic patterns could be efectively remembered and recognized.From the psychometric curve, we found that the pattern duration of 0.5 s already approached the saturation point in the testing phase, albeit with a median accuracy of 89%.Data from the learning phase exhibited a similar trend.More specifcally, 0.3 s was at the ramp of both curves with 81% median accuracy for the testing phase and 61% median accuracy for the learning phase.Tests of equal proportions [58] revealed that the recognition accuracy at the duration of 0.8 s was only signifcantly diferent from that of 0.1 s during the testing phase (p < 0.001 for 0.8 s vs. 0.1 s, p = 0.117 for 0.8 s vs. 0.3 s, p = 0.752 for 0.8 s vs. 0.5 s).When we pushed the pattern duration to be extremely short as 0.1 s, recognition accuracy derived from testing (67%) and learning (33%) phases, were still signifcantly higher than the chance levels of 50% and 11.1%, respectively (p < 0.0001 for both phases using a Binomial test).

Discussion
The high recognition accuracy from the 0.8 s pattern duration supports the efectiveness of relying on haptics as the main feedback for gestural inputs.Shortening haptic patterns to 0.5 s and 0.3 s still enables recognition rates at approximately 90% and 80%.Therefore, instead of long haptic patterns for several seconds, 0.3 -0.5 s pattern duration could be sufcient when confrming user inputs.Note that the above data was derived using wristbands with four vibrotactors located around the wrist.When pursuing concise and lightweight wearable devices, reducing the number of actuators is commonly considered to reduce the hardware complexity.However, this strategy is also expected to decrease the recognition performance accuracy, especially for spatiotemporal patterns.For example, the Save and Share actions shared the same motion efectbased haptic patterns, with the only diference being the motion direction.These two patterns would then be identical if only one vibrotactor was used.For the Remind pattern, which used four pulses alternating between VT-L and VT-R at a very fast pace, using only one vibrotactor or shortening the distance between the two vibrotactors, e.g., both placed in the watch face instead of on the band, might also infuence the recognition.As illustrated by the classic tau [30] and kappa efects [86], the spatial distance between stimuli would be perceived as smaller for shorter inter-stimulus times, and the time durations perceived between stimuli would be perceived as shorter for smaller inter-stimulus distances [24].More specifcally, it is suggested that only after the production of time and an exponential function of distance reaches a certain constant value, users will begin to discriminate two stimuli instead of a merged one [23].Such perception thresholds for spatiotemporal haptic patterns also explain the signifcant drop in recognition accuracy when haptic patterns were displayed for only 0.1 s.
Despite the spatiotemporal interplay, temporal perception sensibility could also constrain recognition accuracy when exploring the shortest possible pattern duration.Indeed, for 0.1 and 0.3 s durations, participants reported that the four pulses in the Remind pattern (P1, P8) and the three pulses in the Lucky pattern (P1, P10) were difcult to precisely detect and count.Meanwhile, the Remind pattern was also frequently confused with the Cancel pattern in the learning phase, likely because they both have multiple pulses that were difcult to count.For the temporal acuity of human tactile sensation, it has been reported that the threshold of temporal gap detection would increase with lower signal intensity [23] and lower signal frequency [55].More specifcally, minimum temporal gaps around 50 -80 ms could be detected when using 35 Hz and 250 Hz sinusoid vibration stimuli with 25 dB above the threshold intensity [55].Therefore, when using short haptic pattern durations, fewer or only one haptic pattern should be designed with the between-pulse gap lower than the gap detection threshold.If multiple were designed, other haptic parameters, e.g., amplitude or location, have to be used to help discriminate them.Moreover, although not applied to our haptic pattern design, the spatial acuity of human tactile sensation, which is around 1 cm for the hand dorsum and around 1.5 cm for the forearm with successive stimuli [52], should also be considered when pursuing a more compact hardware form factor.
Considering that tactile temporal acuity plays a role in haptic pattern perception and recognition, a psychometric analysis was applied to determine the relationship between recognition accuracy and pattern duration.Meanwhile, the ftted trend could also be infuenced by many other factors, such as the learning process [16], the total number of haptic patterns [77], the distinctiveness of the haptic patterns, and so on.While it is difcult to explore all of these factors at once, our fndings are an empirical reference for similar applications that value such interaction efciency.

EXPERIMENT 2: HAPTIC FEEDBACK FOR SEQUENTIAL GESTURAL INPUTS
This second experiment sought to evaluate multiple variables relevant to the haptic feedback design when sequences of gestural inputs were performed.By randomly sending either correct or incorrect haptic patterns as feedback, we aimed to examine if participants could accurately identify 1) if there was an incorrect haptic pattern in the sequence, and 2) which haptic pattern was the incorrect one, when varying the haptic design variables.

Experimental Design
Six of the nine actions were used to create gesture sequences for the experiment, i.e., Confrm, Cancel, Save, Share, Remind, and Tag.
The Lucky and Query actions were not included in the study as they required long responses, e.g., multiple recommended items when using Lucky to pull recommendations from the system, and would introduce unwanted gaps into the sequences.Capture was also not included as visual feedback would likely be available for this action and thus aforded a relatively lower priority for haptics-dominant interaction.
To pursue both the efectiveness and efciency of gesture-haptic communication, recognition accuracy and interaction time (overall interaction duration for a sequence) were adopted as the two main evaluation metrics.Two strategies were proposed to provide haptic feedback at diferent times during the gesture sequence.The immediate feedback strategy was designed to provide the haptic pattern immediately after each hand gesture was detected, which is similar to the feedback delivery for single inputs.The concatenated feedback strategy was designed to provide all haptic patterns in a concatenated manner after all hand gestures in the sequence were performed and detected.This second strategy was proposed to facilitate the smooth performance of gesture sequences.When using the immediate feedback strategy with prolonged haptic pattern durations, participants would have to pause between gestures, waiting for the end of the haptic pattern, instead of seamlessly performing the gesture sequence.Therefore, we anticipated that the concatenated feedback strategy could be preferred, especially by practiced users who are capable of performing gestures quickly and would prefer not to be interrupted by the feedback.
The variation of pattern duration was also considered, where 0.3 s and 0.5 s were selected based on fndings in Experiment 1.In addition, the number of gestures in a sequence (sequence length) was included as well to examine its infuence on communication performance.The numbers 2, 3, and 4 were picked as we foresaw sequences with 5 or more gestures would be less commonly used for this action set.Moreover, age is believed to have a large impact on haptic perception [10,16,31,85].Therefore, participants across a large range of ages were recruited as well.
Overall, four independent variables were designed for this experiment, which were haptic feedback strategy (2 levels), pattern duration (2 levels), sequence length (3 levels), and age.Among them, pattern duration and age were between-subjects variables, while feedback strategy and sequence length were within-subjects variables.Five experimental phases were designed including the same exploration and learning phases used in Experiment 1.Three testing phases were designed consisting of one single gesture interaction phase and two sequential gesture interaction phases with the two feedback strategies counterbalanced for diferent participants.

Setup
The same experimental setup that was used in Experiment 1 was used in Experiment 2. During the three testing phases, however, all responses were provided by pressing keys on a keyboard rather than using a mouse cursor to collect more accurate response times.

Participants
Thirty participants were recruited to participate in this experiment.One participant was removed from the study as their experiment exceeded the two-hour time limit that was approved by our IRB.Thus, data from the remaining 29 participants is reported herein (mean = 41 years, SD = 14 years, range = 19 to 65 years old, 18 participants < 45 years, 11 participants ≥ 45 years; 14 male, 15 female).Fourteen participants reported limited experience with haptics, fourteen participants reported no experience with haptics, and one participant reported moderate experience.The study received ethics approval from the IRB at (anonymized), and participants were paid for their participation in the study.Sixteen participants were assigned to the experiment with 0.3 s pattern duration and thirteen participants were assigned to the 0.5 s pattern duration.

Experimental Procedure
The experiment occurred over fve phases, i.e., exploration, learning, testing for single gesture interaction, testing for sequential gesture interaction with immediate feedback, and testing for sequential gesture interaction with concatenated feedback.The exploration and learning phases were similar to Experiment 1 with the exception that six actions rather than all nine were used.

5.4.1
Testing -Single Gesture Interaction.The frst testing phase was adapted from the testing phase in Experiment 1. Six actions repeated six times were pseudo-randomly provided together with their corresponding hand gestures.Either a correct or incorrect haptic pattern was sent when the instructed gesture was detected.We kept the only question of asking the correctness of the haptic pattern to better track the response time (Figure 5A).A "Yes" sticker was put on the "F" key and a "No" sticker was put on the "J" key for participants to provide their answers.A "Next" sticker was put on the space key for participants to proceed to the next trial.The "Yes" and "No" keys were activated after the end of the haptic pattern and the "Next" function was activated after either "Yes" or "No" was collected.The response time was recorded for each trial as the duration between the end of the haptic pattern and the time point of pressing either the "Yes" or "No" key.Before the phase started, six practice trials were performed with each action presented once for participants to become familiar with the keys and the interface.

Testing -Sequential
Gesture Interaction with Immediate Feedback.For each trial, participants were presented with a sequence of actions and their corresponding gestures on the screen (Figure 5B) and were asked to perform all gestures in the sequence one by one.A haptic pattern, either correct or incorrect, was provided immediately after each instructed gesture was detected.The corresponding action on the screen was also highlighted once the gesture was detected.Participants could control their gesture performance pace but were instructed to provide their answers as quickly and accurately as possible.After all gestures and haptic patterns for a given sequence concluded, participants indicated if all of the haptic patterns were correct or not by pressing the "Yes" or "No" key.If "Yes" was pressed, the "Next" key would be activated and participants could proceed to the next trial.If "No" was pressed, another question appeared on the screen and asked which haptic pattern was incorrect.Participants used the number keys on the keyboard to provide the position of the incorrect pattern.After inputting the number, the "Next" key was activated.
Forty-eight sequences were pseudo-randomly presented during this second testing phase, with 16 sequences for each of the 2-, 3-, and 4-sequence lengths.Actions were pseudo-randomly sequenced without the same actions or Confrm and Cancel occurring right after each other.For the 48 sequences, 24 were provided with correct haptic patterns for all actions in the sequence.For the remaining 24 sequences, only one of the actions in the sequence was provided with an incorrect haptic pattern.This incorrect action was randomly selected in the sequence and the incorrect pattern was randomly selected from the remaining fve haptic patterns.The sequences with correct or incorrect feedback were equally distributed amongst each sequence length (e.g., 8 of the 3-sequence length trials were correct and 8 were incorrect).Among the 144 provided actions (16 * (2 + 3 + 4)), the six actions of Confrm, Cancel, Save, Share, Remind, and Tag were repeated with the same frequency of 24 times.As only 24 actions would be provided with the wrong haptic pattern, 4 of the 24 repetitions were chosen for each of the six actions.Four sequences including 12 actions (i.e., twice for the six actions) were provided for practice before this phase.Prior to the sequential gesture testing phases, the facilitator specifed that each sequence would contain, at most, one incorrect pattern, and participants should input only one number to indicate the position of the incorrect pattern.
The overall interaction time of each trial was defned and recorded between the time when the "Next" key was pressed in the previous trial and the time when either the "Yes" or "No" key in the current trial was pressed.The time duration between the detection of two consecutive hand gestures was also collected.The response time was defned and collected as the duration between the end of the last haptic pattern and the time point of pressing either the "Yes" or "No" key.

Testing -Sequential Gesture Interaction with Concatenated
Feedback.For the third testing phase, 48 sequences were generated using the same procedure as in the second testing phase.However, instead of sending the haptic pattern immediately after each hand gesture was detected, a single pulse as short as 50 ms was provided to notify participants that the gesture was detected and they could move to the next gesture.The corresponding action on the screen was highlighted once the gesture was detected for notifcation as well (Figure 5B).After the last gesture was detected and a 0.9 s gap, all associated haptic patterns were sent in a concatenated way, with an inter-pattern gap of 0.7 s.These temporal gaps were decided based on the results from a pilot study with four participants.After the last haptic pattern was sent, the "Yes" and "No" keys were activated.The same questions were asked as those in the second testing phase.The overall interaction time, the time duration between two gestures, and the response time were also collected the same as in the second testing phase.Four practice trials with 12 actions were also provided for this phase.
The two feedback strategies for sequential gesture interaction, i.e., sequential strategy and concatenated strategy, were conducted in a counterbalanced order across participants.At the end of all fve phases, post-surveys were collected including task loads for both feedback strategies and participants' preference of actions.

Feedback Strategy & Patern Duration.
Recognition accuracy was calculated per participant for the two testing phases of sequential gesture interaction.Participants' responses were marked as correct in two cases: 1) if they accurately identifed the sequences with all correct haptic patterns, and 2) if they accurately identifed sequences with incorrect pattern as well as adequately identifed which one was the incorrect pattern.As feedback strategy was a within-subjects variable and pattern duration was a betweensubjects variable, the linear mixed efects model (LMEM) was applied using R lmerTest package [41] to determine their efects, if any, on recognition accuracy and interaction time.Feedback strategy and pattern duration were set as fxed efects and participants were set as random intercepts.
The results demonstrated that both the feedback strategy and the interaction between feedback strategy and pattern duration exhibited signifcant infuences on recognition accuracy (feedback strategy: p < 0.05, 2 = 0.04; pattern duration: p = 0.173, 2 = 0.07; interaction: p < 0.01, 2 = 0.06; Figure 6A).Post-hoc analysis with Benjamini-Hochberg (BH) correction found that recognition accuracy was signifcantly lower when using concatenated feedback with 0.3 s pattern duration than using concatenated feedback with ).This recognition accuracy was also signifcantly lower than using the immediate feedback strategy with 0.3 s pattern duration (p < 0.001).The potential reasons for a signifcant drop in recognition accuracy for the concatenated feedback strategy with 0.3 s pattern duration could be multifaceted.With a shorter duration, 0.3 s in our case, diferences in haptic patterns were more difcult to perceive and would require more time for participants to make a decision.Yet the interaction pace actually became faster without the fexibility for participants to pause and think as they could do in the immediate feedback trials.The next pattern could be sent while participants were still recalling the previous one, resulting in missing the next one or several patterns.
Feedback strategy and the interaction between feedback strategy and pattern duration were also found to have signifcant infuences on interaction time (feedback strategy: p < 0.0001, 2 = 0.07; pattern duration p = 0.59, 2 = 0.01; interaction: p < 0.01, 2 = 0.004; Figure 6B).Post-hoc analysis with BH correction demonstrated that the concatenated feedback strategy required signifcantly longer interaction time than the immediate feedback strategy for both 0.3 s and 0.5 s pattern durations (p < 0.0001 for both).Pattern duration of 0.3 s also led to signifcantly shorter interaction time compared with 0.5 s for immediate feedback strategy (p < 0.0001).Overall, feedback strategy and pattern duration exhibit strong interaction in infuencing sequential gesture interaction, while in general, 0.5 s pattern duration enabled higher recognition accuracy and immediate feedback strategy enabled shorter interaction time.5.5.2Sequence Length.The LMEM analysis was conducted for the four combinations of feedback strategies and pattern durations with sequence length being the fxed efect and participants being random intercepts.We found that sequence length did not infuence recognition accuracy (immediate feedback with 0.3 s pattern duration: p = 0.085, concatenated feedback with 0.3 s pattern duration: p = 0.084, immediate feedback with 0.5 s pattern duration: p = 0.685, and concatenated feedback with 0.5 s pattern duration: p = 0.059, Figure 7A, B).Interaction time increased when more actions were included as expected since more rounds of inputs and feedback were conducted.Therefore, those increasing trends were not the focus for the infuence of sequence length, while the trends of interaction time between the two feedback strategies were similar across diferent sequence lengths.Overall, results show that gesture length does not appear to have a large impact on either recognition accuracy or interaction time.

Age.
As shown in Figure 8A, recognition accuracy was negatively correlated with the participants' age for all three testing phases when using 0.3 s pattern duration (single gesture interaction: Pearson correlation p = 0.0012, 2 = 0.54, sequential gesture interaction with immediate feedback: Pearson correlation p < 0.01, 2 = 0.42, sequential gesture interaction with concatenated feedback: Pearson correlation p < 0.01, 2 = 0.4).Linear regression analysis demonstrated that with every 10-year increase in age, the recognition accuracy would decrease by 5.65%, 7.34%, and 9.61% for single gesture interaction, sequential gesture interaction with immediate feedback, and with concatenated feedback, respectively.For pattern duration of 0.5 s, no negative correlation was detected among all three testing phases (single gesture interaction: Pearson correlation p = 0.069, 2 = 0.27, sequential gesture interaction with immediate feedback: p = 0.77, 2 = 0.01, sequential gesture interaction with concatenated feedback: p = 0.66, 2 = 0.02, Figure 8B).For every 10-year increase in age, recognition would only decrease by 3.52%, 0.82%, and 1.4% for single gesture interaction, sequential gesture interaction with immediate feedback, and with concatenated feedback, respectively.It indicates that haptic patterns with 0.3 s pattern duration would be difcult for older participants to recognize.LMEM was used to compare the feedback strategies with the feedback strategy being fxed efect and participants being random intercepts.Mann-Whitley U test was used to compare the pattern durations.BH correction was used for multiple comparisons (Figure 9, Table 1).Results show that immediate feedback with 0.3 s pattern duration and immediate feedback with pattern 0.5 s pattern enabled signifcantly higher recognition rates and shorter interaction times for younger and older groups respectively.

Feedback Strategy Workflows.
As shown in previous analyses, the immediate feedback strategy aforded reliably shorter interaction time compared with the concatenated feedback strategy.
We then visualized their interaction workfows in Figure 10A.Only the condition with 0.3 s pattern duration and 4-gesture sequence length was shown as an example.For the concatenated feedback strategy, the average duration between consecutive gestures was approximately 1.3 s.Since no haptic patterns were provided after hand gestures were detected, this time duration could be used for only gesture performance and detection.For the immediate feedback strategy, the time duration for participants to confrm a correct pattern was approximately 0.3 s, which was obtained by subtracting the 1.3 s gap and 0.3 s pattern duration from 1.9 s.When the previous haptic pattern was identifed as the incorrect pattern, the time needed to confrm that the haptic pattern was wrong would be around 1 s instead, i.e., 2.6 s -1.3 s -0.3 s.This time diference between recognizing correct and incorrect haptic patterns was indeed signifcant regardless of where the incorrect pattern was provided in the sequence (p < 0.0001 when moving to the second gesture and moving to the third gesture, p = 0.0014 when moving to the fourth gesture, using LMEM after BH correction; Figure 10B).In addition, the recorded response time for the immediate feedback strategy was signifcantly shorter than that recorded from the concatenated feedback strategy and single gesture interaction (p < 0.0001 for both comparisons after BH correction) (Figure 10C).Subjective workloads reported by participants were compared between the two feedback strategies tested in sequential gesture interaction in Experiment 2. Seven-point Likert scale NASA-TLX [29] was used, including six dimensions, i.e., mental demand, physical demand, temporal demand, performance, efort, and frustration, where 1 implies 'perfect' and 7 implies 'failure' for performance, and 1 implies 'very low' and 7 implies 'very high' for the other fve dimensions.LMEM analysis was conducted with feedback strategy, pattern duration, and age group being fxed efects and participants being random intercepts.Results show that concatenated feedback required signifcantly higher workloads compared with the immediate feedback strategy (p < 0.0001), while age groups (p = 0.135) and pattern durations (p = 0.109) did not cause signifcant diferences.
To further examine how the changes in haptic design infuence each dimension of workload, post-hoc analysis was conducted with BH correction.Results show that for 0.3 s pattern duration, the immediate feedback strategy required signifcantly lower eforts than the concatenated strategy (p = 0.09, 0.35, 0.082, 0.31, 0.0051, and 0.11 for the six dimensions, respectively).For 0.5 s pattern duration, the immediate feedback strategy aforded signifcantly lower mental demands, temporal demands, efort, and frustration (p = 0.0051, 0.13, 0.012, 0.11, 0.022, and 0.0051 for the six dimensions, respectively).Large efect sizes were derived for all signifcant pairs ( 2 = 0.57, 0.59, 0.51, 0.44, 0.6 for the fve pairs respectively).Participants also elaborated on why they preferred one feedback strategy over another.For participants who preferred immediate feedback, they reported that this strategy required less memory load while it would be overwhelming to recognize multiple haptic patterns at one time.In particular, P19 reported that when receiving immediate feedback, "I could respond as the feedback happened instead of having to remember", while they "have to remember all the feedback" when patterns were concatenated.P29 also reported that "With the immediate feedback, once you notice the wrong haptic, you know the answer".These comments indicate that immediate feedback could be straightforward and intuitive for participants to make a decision without the efort of recall.Note that it could also be caused by our design of immediate feedback using the same interaction fow as the learning phase, where haptic feedback was provided right after gestures were detected.Similar to the interaction workfow observations, participants preferred immediate feedback as they could control the pace of interaction and had more time to evaluate the haptic pattern if needed, while P23 reported that they "cannot slow down for recognition" when using concatenated feedback.The immediate feedback also enabled participants to "focus less on the rest of the sequence since I had already identifed the incorrect pattern" (P6).In comparison, participants needed to focus more during the concatenated feedback phase and "cannot miss a haptic pattern" (P9).Meanwhile, one advantage of concatenated feedback was also reported as it allowed participants to compare haptic patterns in the same sequence to help identify the incorrect pattern (P6, P11, P20).While participants could not compare patterns when receiving immediate feedback.Participants' preferences of shortcut actions were collected from Experiment 2 using the question "Among the six actions, for which ones could you envision using this gesture-haptic UI in the future?" in the post-survey.Confrm, Cancel, Save, and Share actions were preferred across the two pattern durations (Figure 11 B).Participants reported that actions whose haptic pattern was difcult to distinguish were less preferred.Also, actions that they foresaw to be less commonly used or new to them were also less preferred.This suggests when proposing new concepts of actions, more distinct haptic patterns should be designed for them to mediate users' reluctance to use them.The Remind action was less preferred as participants believed that gestural inputs would not be able to provide the information needed when setting a reminder, such as the time, location, or specifc action items.This indicates that certain actions would be benefcial when the context is readily available.For example, invoking Share during a phone call while the receivers of the sharing item are easy to specify, or invoking Remind while looking at a poster when the time and location are available.
Moreover, in the post-survey, participants also reported their preference for the gap length between consecutive haptic patterns in the concatenated feedback strategy.Results show that 17 participants preferred for the gap to be longer (15 for slightly longer and 2 for much longer), 10 participants reported "no need to change", only 2 participants wanted it to be "slightly shorter", and none to be "much shorter".Based on the interplay between feedback strategy and pattern duration in infuencing the recognition accuracy, we would recommend avoiding using the combination of concatenated feedback and 0.3 s pattern duration, as it led to a large drop in accuracy compared with the other three combinations.Meanwhile, despite the interaction between the two factors unraveled by the LMEM analysis, 0.5 s would be recommended for pattern duration to achieve higher recognition accuracy.For interaction time, the immediate feedback strategy aforded reliably shorter interaction time than the concatenated feedback strategy and would be recommended when pursuing efcient sequential gesture interaction.

Discussion
Although one might assume that adding more gestures into a sequence would lead to worse interaction performance, sequence length was found to not signifcantly infuence recognition accuracy or interaction time.One reason could be that the memory load caused by actions and gestures was not taken into account.To eliminate the infuence of the learning process for actions and gestures, actions and gestures were always displayed on the screen during the testing phases.Participants could look at the actions while perceiving the haptic patterns and making decisions instead of remembering them.The memory load incurred when using sequences of actions and gestures would, however, play a role during real-world interactions and might lead to decreased performance, especially for the concatenated feedback strategy.Moreover, participants were likely to make a decision on each haptic pattern when receiving them instead of holding the memory of the whole sequence and making the decision at the end.This could be supported by Figure 10 C, where the fnal response time for concatenated feedback was similar to that for single gesture interactions.It suggests that longer gesture sequences would not cause additional short-term memory of haptic patterns either.
Moreover, detailed explanations of the metaphors before experiments could also facilitate participants' memory and recognition of those actions and haptic patterns.In future work, it would be benefcial to test the recognition performance between predefned metaphors and user-defned metaphors, while we would recommend providing explanations for predefned metaphors to facilitate the user experience in real-world applications.
5.6.2Age-Based Haptic Design Recommendations.Age is believed to impact tactile functions, due to changes in skin properties, touch perception, and its underlying nervous system [55].Focusing on tactile sensitivity, studies have shown that tactile thresholds increase signifcantly with age [90,101].More specifcally, for vibrotactile stimuli, younger adults aford lower detection thresholds for stimulus intensity [16], localization [10], and temporal gap [31].Similarly, in our second experiment, we also observed that the recognition accuracy of haptic patterns proportionally decreased with an increase of age for the pattern duration of 0.3 s.However, this decreasing trend did not hold for the pattern duration of 0.5 s.This suggests that in our case, a decrease in temporal sensitivity may play a more signifcant role in reducing recognition accuracy, compared to decreases in other tactile perception functionalities.From the cognition perspective, it also suggests that slower recognition speed with age might have a larger impact than the decrease of memory capabilities [12].
Moreover, we found that younger participants were more robust to diferent approaches of providing haptic feedback as their recognition accuracies were close to each other across diferent pattern durations and feedback strategies.Since immediate feedback with 0.3 s pattern duration led to signifcantly shorter interaction time, this combination is recommended for younger users.In contrast, older participants were sensitive to how haptic feedback was provided.More specifcally, changing from immediate feedback to concatenated feedback would signifcantly increase interaction time, and changing pattern duration from 0.5 s to 0.3 s would signifcantly lower recognition accuracy.Therefore, immediate feedback with 0.5 s pattern duration is recommended for older users.Please note that the 45-year-old threshold does not indicate an abrupt change in participants' interaction performance at this precise age.Instead, age afects performance in a gradual manner, with slight changes in this threshold leading to similar interaction performances between the two age groups.5.6.3Interaction Time across Feedback Strategies.We initially hypothesized that the concatenated feedback strategy would be more efcient, since it enables practiced users to swiftly perform the gesture sequence without being interrupted to pause by haptic patterns.Without constantly switching between the modalities of gesture performance and haptic perception, we anticipated gestures within a continuous gesture sequence to be performed right after the previous ones, while preparation time might be needed for gestures following haptic patterns.However, results from Experiment 2 demonstrated the opposite.One reason could be that the haptic patterns we used in experiments were very short.Immediate haptic feedback with 0.3 s to 0.5 s duration is likely to conclude before users begin the next hand gesture.Compared with longer haptic patterns, gesture performance would not be slowed down by such pattern durations as no additional time is needed to wait for the patterns to fnish.Another reason is that participants we recruited were all novices to this gesture-haptic UI.Even after completing the exploration, learning, and single-gesture testing phases, it was still challenging for participants to be thoroughly familiar with all gestures and haptic patterns, despite having the images of gestures displayed on the screen.During the concatenated feedback trials, the gesture performance was actually not as efcient as anticipated.Future work should be conducted to explore the infuence of familiarity on interaction time.
Moreover, based on the workfow observations, a signifcantly longer time would be needed to confrm an incorrect haptic pattern compared to confrming a correct pattern.This provides insights into the potential of predicting users' behavior ahead of time based on the gap between two consecutive gestures.For example, a longer gap within a fuent gesture sequence could be a signal for an incorrect haptic pattern being identifed by the user.In this case, the system could adjust the recognized command or suggest redoing the hand gesture.More importantly, it also implies that the fexibility of moving faster to the next gesture when a correct haptic pattern is confrmed contributes to the shorter interaction time when using the immediate feedback strategy.In comparison, concatenated feedback lacks this fexibility since the system has to apply the same gap between all haptic patterns.One might question that the gap we set between haptic patterns could be too long.However, 27 out of 29 participants reported in the post-survey that they preferred the gap to be longer or remain the same.Changing this inter-pattern gap or the frst gap before haptic patterns to be longer or short might also infuence both recognition accuracy and the overall interaction time, where future research is needed.
In addition, participants could skip the evaluation of the remaining haptic patterns if they already identifed an incorrect haptic pattern in the sequence.This could lead to less interaction time, especially for the immediate feedback strategy, since participants could quickly perform the rest hand gestures without thinking.In comparison, when using the concatenated feedback strategy, participants wouldn't be able to further accelerate the gesture performance or have the fexibility to change the pace of haptic feedback.This could be supported by signifcantly shorter response times of the immediate feedback strategy compared with those of the concatenated feedback strategy and single gesture interactions.This advantage of the immediate feedback strategy can also be applied to real-life interaction scenarios, where users could cancel sequential input in the middle of the sequence.In contrast, when using concatenated feedback, users have to perform all gestures in the sequence.Furthermore, it is likely that gesture performance, haptic feedback display, and haptic pattern evaluation might slightly overlap during the immediate feedback strategy, which could further shorten the interaction time.5.6.4Incorporating with Other Feedback Channels.While this study focused on interaction with solely haptic feedback, we believe this gesture-haptic UI also afords great potential while incorporating with other feedback channels.Although visual and audio feedback might serve as the primary feedback channels when available, integrating haptic feedback could always facilitate user-system communication.In scenarios where visual and auditory channels are already crowded or overloaded, such as gaming or noisy environments, haptic feedback could be more efective in conveying information by using an alternative channel.It could also better catch the user's attention with less interruption or distraction from the major tasks.As for scenarios where visual and audio feedback are efective, adding haptic feedback could also increase interactivity.For example, the haptic feedback for 'Share' generates a motion efect resembling something being sent out, which could make the interaction more entertaining and more immersive for especially XR applications.Additionally, in scenarios where loading time is needed for the visual or voice feedback, such as for actions like 'query' or 'lucky', haptic feedback serves to confrm the command reception and reassure users that the system is processing their request instead of being unresponsive.For such use-case scenarios where multiple feedback modalities were employed, our fndings in haptic design could serve as a reference, ofering insights into the haptic and overall feedback design.

CONCLUSION
This work explored a gesture-haptic UI for shortcut commands and examined the design of haptic feedback to achieve fast and accurate interactions.Metaphors were leveraged during the design of nine haptic patterns, which were validated to be readily recognized as feedback for hand gestures.By running user studies, we found that haptic patterns with the duration of 0.3 s could achieve recognition accuracy of approximately 80% when identifying incorrect patterns, and with approximately 90% accuracy when using 0.5 s pattern duration.Moreover, for sequential gesture interaction, the immediate feedback strategy exhibits reliably shorter interaction time compared with the concatenated feedback strategy.We also examined the workfows when using these two feedback strategies and elaborated on the details underlying the shorter interaction time of the immediate feedback strategy.Age was also found to have a signifcant impact on haptic feedback design.Based on the results from our experiments, when performing sequential gestural inputs, we would recommend using immediate feedback with 0.3 s pattern duration for younger users (i.e., < 45 years old) and immediate feedback with 0.5 s pattern duration for older users (i.e., >= 45 years old).Overall, this work provides guidance on the design of haptic feedback for user-initiated sequential gesture interaction and sheds light on the haptic pattern design for similar applications with sequential inputs.

Figure 1 :
Figure 1: Selected shortcut actions with their intended meanings and associated hand gestures.

Figure 2 :
Figure 2: The nine haptic patterns designed for the shortcut actions.Metaphors were leveraged to guide the pattern design.The visualization sketches show signal envelopes for haptic patterns whose four VT channels share the same signal without motion efects and show motion-efect haptic patterns relative to the hand gesture.The last column depicts four-channel haptic signals generated by the four vibrotactors, i.e., VT-T (top), VT-L (left), VT-B (bottom), VT-R (right).

Figure 3 :
Figure 3: GUIs for Experiment 1. (A) GUI for the exploration phase.This screenshot was taken when Remind was clicked.(B) GUI for the learning phase.This screenshot shows the GUI when the answer was incorrect.(C) GUI for the testing phase.

Figure 4 :
Figure 4: Recognition accuracy and psychometric curve ftted for Experiment 1. (A) Boxplots of recognition accuracy for the testing phase, where the numbers above each pattern duration denote the median values.(B) The psychometric curve for the testing phase.Some data points overlapped since their accuracies were the same.(C) Recognition accuracy for the learning phase.(D) The psychometric curve for the learning phase.Some data points overlapped since their accuracies were the same.

Figure 5 :
Figure 5: GUIs for Experiment 2. (A) GUI for the single gesture testing phase.(B) GUI for the two sequential gesture testing phases.Despite diferent interaction fows of the immediate feedback and concatenated feedback strategies, their GUI shared the same visual appearance.GUIs for the rest exploration and learning phases were the same as in Experiment 1.

Figure 7 :
Figure 7: The results of manipulating the sequence length on (A) the recognition accuracy for 0.3 s pattern duration, (B) the recognition accuracy for 0.5 s pattern duration, (C) the interaction time for 0.3 s pattern duration, and (D) the interaction time for 0.5 s pattern duration.Diamonds denote means, and error bars denote 95% confdence intervals.

Figure 8 :Figure 9 :
Figure 8: Correlations between recognition accuracy and age for 0.3 and 0.5 s pattern durations across the three testing phases.Dots represent data from individual participants and the bands around the regression lines denote 95% confdence intervals.

Table 1 :Figure 10 :
Figure 10: (A) Workfow of sequential gesture interaction using immediate or concatenated feedback strategy.The condition of 0.3 s pattern duration and sequence length of 4 gestures is shown here as an example.Numbers in the blocks are mean values collected from the experiment.G: gesture, H: haptic pattern.(B) Comparison of time duration between consecutive gestures when the previous haptic pattern was identifed as correct or wrong in the immediate feedback strategy.Only data from the 0.3 s pattern duration was included.(C) Comparison of response time between three testing phases for 0.3 s pattern duration.

Figure 11 :
Figure 11: (A) Mean NASA-TLX ratings for the two feedback strategies used in sequential gesture interaction.Error bars denote 95% confdence intervals.*, ** denote signifcance levels of p < 0.05, p < 0.01.(B) Count of actions collected from the question "Among the six actions, for which ones could you envision using this gesture-haptic UI in the future?"across two pattern durations.

5. 6 . 1
The Role of Feedback Strategy, Patern Duration, and Sequence Length in Haptic Feedback Design.