MAMMOTH: Mid-Air Mesh-based Modulation Optimization Toolkit for Haptics

Mid-air ultrasound can recreate the missing touch from contactless interactions, such as bare hand gestures in extended reality. But designing ultrasound haptics either relies on inadequate static sensations or experts who can create dynamic sensations. We introduce MAMMOTH, an open source toolkit for Unity that automatically generates dynamic ultrasound sensations for interactions with 3D objects. The haptic feedback is achieved by extending and generalizing a path-routing algorithm for intersections between meshes. We first describe how the toolkit works and then demonstrate how it builds on previous techniques. Finally, we present how to use the toolkit to implement three distinct use cases.


INTRODUCTION
Sensor technology has enabled the implementation of contactless interaction (e.g., hand tracking, proximity sensors, motion sensors).Unfortunately, the intrinsic lack of contact in such interactions has translated into a lack of haptic feedback.However, our sense of touch is, as Linden put it, "what makes us human" [10].Aiming to alleviate this issue, the HCI community has focused their effort on re-introducing haptic feedback to contactless interactions through so-called mid-air haptic technology.
One mid-air haptic technology enabling the re-introduction is ultrasound haptics.Ultrasound haptic devices consist of a 2D array of transducers that focus ultrasound in a single point, creating a haptic sensation on the skin.This focal point can be moved around rapidly, creating the illusion of a pattern or shape.
One recurring application of the technology is the ability to touch virtual objects in extended reality scenarios.For instance, Barreiro et al. proposed the path routing optimization technique (PRO-STM) to render haptics for a gaseous fluid interaction.However, this technique does not work for general objects and the approach renders larger areas sequentially, resulting in a sensation that is moving sensation instead of steady.
We build upon PRO-STM to create a new rendering technique and generalize it in a toolkit, MAMMOTH.The technique enables mid-air haptic designers to automatically create haptic interactions between tracked hands and 3D objects.The sensation changes in real time depending on the points of intersection between the hand and other 3D objects by computing the shortest path that travels all the points.The haptic sensation is rendered by interpolating between each point in the path and outputting it through the ultrasound device.Our technique includes optimizations such as dynamic frequency modulation to render long paths as a steady sensation.
This paper introduces the technique used in the MAMMOTH toolkit, and three use cases including two MAMMOTH instances at once, enabling mediated social touch.MAMMOTH will be released as an open source toolkit for Unity.Our implementation uses the Ultrahaptics device and Leap Motion, but the steps described in the technique can be followed for use with other ultrasound devices and engines.MAMMOTH enables both experts and novices to easily add dynamic mid-air haptics to their contactless interactions.

RELATED WORK
Many mid-air haptic technologies can create a sense of touch to contactless interactions.Mid-air feedback can be achieved through airflow, laser, heat, electrostatic, ultrasound, and more.These technologies all have advantages and disadvantages, pertaining to intensity, spatial resolution, ability to generate warmth, and accessibility.One of the leading forms of mid-air haptics uses focused ultrasound to stimulate our sense of touch.
Ultrasound haptics are achieved by timing the output of multiple ultrasonic transducers to collide at the same focal point.This focal point can be moved around to generate the sense of shapes and patterns on the palmar side of the hand.Although ultrasound haptics can stimulate other parts of the body such as the forearm [15,22], face [7,20], and feet [9], the palmar side of the hand is the main target for contactless applications.One reason is that the hands are the body's tool for everyday interactions (e.g., holding a coffee mug, shaking hands, typing this paper).Another is that the hands are one of the most densely innervated regions of our body [3], making the ultrasound sensations feel stronger.Ultrasound haptics have been used to create the contactless sensation of touching another person [13,19], a heart [17], mid-air buttons [12,14,21], fluids [8], gases [1] and more.There are two general approaches for modulating ultrasound haptics: amplitude modulation (AM) or spatiotemporal modulation (STM).MAMMOTH uses the spatiotemporal approach and we consider the frequency of each sensation to be its repetition rate [6] There are two primary types of ultrasound haptic sensations: static and dynamic sensations.Static sensations are time-bound and often repeating.They can be anchored to the hand, but do not conform to the interaction such as changing the size or pattern.Static sensations include pre-made sensations of circles and squares [24] and "Hapticon" sensations designed and exported through Ultraleap's Sensation Designer [18,25].The designer's sensations do not conform to objects, and apart from being anchored to the hand, are static once designed.Dynamic sensations can change in real time and conform to the shape of interactions.This includes bare hands interactions with objects [11], fluids [8], and gasses [1].Implementing the dynamic sensations requires expert knowledge of signal processing and geometry, and no existing tool is publicly available.

PATH ROUTING OPTIMIZATION
Our technique generates a sensation on the intersecting meshes based on the 2-opt path routing algorithm.This implementation is based on PRO-STM by Barreiro et al. [1].
The following six steps are taken to generate the haptic sensation: (1) Mesh Selection

Meshes
MAMMOTH works with most objects in Unity, such as standard 3D objects (e.g., cube, sphere in figure 2a) and custom 3D objects (such as the mammoth in figure 1).The meshes can move, overlap each other, and change shape and size, as long as the intersections are triggered by the collision system.The objects must have a collider component attached to them to recognize the intersection with the tracked hand.

Collisions with Discretized Hand
MAMMOTH uses the physics collision system in Unity to generate the intersecting points in a 3D space.Figure 2b shows the standard cube and sphere colliders, as well as a custom set of colliders created for the tracked hand in the MAMMOTH toolkit.The mesh of the tracked hand implemented with Leap Motion is fitted with sphere colliders on each limb.The positions of the sphere colliders are generated using a Poisson-Disc sampling over the hand mesh surface.Each sphere is then attached to a limb to follow the movements of the hand.Each sphere is 5 mm in diameter and is triggered as an intersecting point upon collision with another mesh collider.The other mesh colliders can be standard shapes like cubes or cylinders, or any other 3D mesh in Unity that can be used as a 3D collider as the mammoth in Figure 1.

Intersection Points
To capture intersections and enable haptics, a custom MammothInteractable component (i.e., C# script) is attached to each object.The component transmits the intersection points to a custom Mammoth-Renderer component placed anywhere in the Unity scene, which runs the algorithm and transmits the final path to the component connected to the haptic device.Intersection points from each Mam-mothInteractable component are combined to a single set of points in the MammothRenderer.Figure 2c shows the intersection points captured from the cube and sphere.All points captured used in the following steps are 3D coordinates.

Point Exclusion and Initial Path
To optimize the performance of the 2-Opt algorithm, several intersection points can be excluded to generate an initial path.We exclude contact points that are within a specified range of another point, keeping only one of them.Excluding points too close together is tolerable due to the focal point's size of 8.6 mm in diameter [6] and the two-point discrimination between approximately 15 mm (50 Hz) and 22 mm (200 Hz) for AM [2]. Figure 2d shows the resulting points using an exclusion range of 5 mm.The unoptimized initial path is seen in the path of Figure 2e.This path is not suited for ultrasound rendering because the path is too long and has overlapping edges, making it impossible to control the frequency of the sensation when considering one period to occur when the pattern repeats the same path.

2-Opt Algorithm
The 2-opt algorithm is a solution to the traveling salesman problem.With a set of points, the aim is to visit all points using the shortest path possible [4].To achieve this, the 2-Opt algorithm checks whether reordering two edges on the path results in a shorter path.This is repeated until no improvement is found.As 2-Opt can be an expensive  (!) operation [23] where  is the number of vertices, there are several steps to optimize the implementation.
In the initial 2-Opt algorithm, a 2-Opt "swap" is executed on every check of nodes, and the total length of the path is computed.The amount of swaps and length calculations can be reduced by calculating the difference in length between the edges about to be removed and the new edges that will exist due to the swap.If the new edges result in a shorter path, they will also result in a shorter total length, and the 2-Opt swap can be executed.If they do not, the algorithm can move on to checking the next possible edges without the swap and length computation.
The next optimization is to use the Euclidean squared distance when checking whether the new edges will result in a shorter total path length, as the square root removed in the distance computation has no impact on the result.
Once the 2-Opt algorithm is completed, it results in a reordered list of points that form the shortest possible path for the haptics sensation, as seen in Figure 2f.

Interpolation with Dynamic Frequency Modulation
To render the path, we must interpolate between each point.If the points were sent directly to the device's buffer, it would result in an inconsistent experience due to the changing length of the path and the distance between each point.By interpolating between each point with the same separation we achieves a more consistent experience.Barreiro et al. [1] introduced an interpolation separation of 0.175 mm.A path traversing the many contact points on the hand mesh can generate paths too long to render when using low separations.The lower the separation, the more points are sent to the buffer, resulting in a longer rendering time.The two-point discrimination threshold and focal point size allow us to use a higher interpolation separation.To render long paths, we dynamically modify the separation for each path.If paths are long and would result in a too long rendering time, the separation is increased.If paths are too short, the separation can be decreased.We determine the separation by bounding the rendering frequency (path repetition rate) between two values and computing a new interpolation separation whenever a new path is introduced.The interpolation separation  between each point is computed using the lower and upper frequency bounds   and   : where The default interpolation separation is denoted by  0 and the sampling rate is  (40Khz for the Ultrahaptics). 0 is the frequency for the given path using the default separation of  0 .The interpolated paths are shown in Figure 3 using static interpolation separation (top) and dynamic interpolation modulation (bottom).To render the haptic sensation, the interpolated points are sent to a buffer for the ultrasound device.

EVALUATION 4.1 Static vs. Dynamic Frequency
Barreiro et al. separated paths over 0.14 m into multiple paths of 0.14 m with 800 points 0.175 mm apart and rendered each path separately in 20 ms [1].With observed shortest paths of total length  over 2 m, this approach would result in approximately 14 paths rendered in 280 ms.Instead of feeling a steady sensation, it feels like moving sensations on smaller surface areas.Figure 3 shows the shortest path rendered with an interpolation separation of 0.175 mm (left).Even though the points are the same size as in the dynamic path (right), the proximity of the points (red) results in a visually uninterrupted line, and the haptic sensation feels like a moving point along the path.The interpolation separation  of 0.175 mm and no dynamic frequency modulation renders a 3.5 Hz sensation.Using dynamic frequency modulation and a lower frequency bound   of 50 Hz, the points are interpolated 2.5 mm apart as seen in figure 3.At 50 Hz ultrasound patterns feel like steady sensations.

Performance Comparison
As the exact 2-Opt optimizations are not visible from the paper of Barreiro et al. [1], we can not conduct timing measurements for a direct comparison.Dynamic frequency modulation allows us to eliminate multiple steps in the PRO-STM algorithm.To separate the points into different segments, they conduct K-means clustering before the 2-Opt and recurring 2-Opt runs if the shortest paths found are greater than 0.14 m.Since we keep the entire path, there is no need for this separation.
Calculating the dynamic frequency modulation separation is an inexpensive  (1) operation, and changing interpolation separation should not have an impact on performance.Point exclusion is a  ( 2 ) operation where  is the number of points.MAMMOTH computes the path using 3D points, which decreases the performance compared to Barreiro et al.'s nodes projected to a 2D plane.
Barreiro et al. include a final path refinement step in their algorithm to align the path with "pressure peaks and ridges".This affects the intensity of the felt sensation to emulate highs and lows in a pressure field.As the Ultrahaptics emits a low force, the perceived range of intensity is low.The intensity of MAMMOTH is mapped between 0 and 1 at an API level, thus saving the performance required for the path refinement operation.

Improvements
As this is the first version of MAMMOTH, multiple performance and feature improvements are to be added later.Being an open source project allows us to make improvements transparent for users and for users to add their improvements.

Performance.
Performance is key to creating a pleasant haptic experience, as an expensive algorithm results in both visual and haptic latency.The major contributors to decreased performance are the physics collision system using the discretized hand, and the 2-Opt algorithm.
We chose to use the collision system to generate the intersection points.This means Unity's physics system must register all collisions of the spheres in figure 2c placed on the tracked hand.Instead of using spheres, we could improve performance with mesh intersection algorithms such as 3D-EPUG [5].Other custom hands could be created and use MAMMOTH, as long as they have surface points that can act as a "trigger" in Unity's collision system.With the intersection mesh, we can sample points across the surface and use those as the input intersection points.The accuracy of the rendered haptics also relies on the performance of the hand-tracking, in this instance the Leap Motion.If the hand-tracking is inaccurate or delays occur, the haptics will feel misaligned or sluggish.
There are several ways to improve the 2-Opt algorithm.Other algorithms such as 2-Opt++ [23] and 3-Opt [16] have tackled the traveling salesman problem.The 2-Opt algorithm involves calculating the lengths of the edges multiple times upon each iteration.This can be avoided by pre-computing a length array with the lengths between all spheres on the discretized hand.For all spheres on the same limb, this distance remains static.For spheres on different limbs, the actual distance depends on the movements of the limbs, creating possible errors by using the pre-computed array.

Features.
We aim to continue updating the toolkit with new features.
One feature is the ability to specify that distinct objects should have individual paths, splitting the total path into smaller paths.This is not to segment the path like Barreiro et al. [1] but to eliminate the haptic feedback rendered in-between objects, as seen in the line between the sphere and cube in figures 2f and 3.
The addition of filters can change the sensation of the rendered path.This could be a simple moving average filter, which smooths the final path.A filter could also remove the inner parts of a path, keeping only the bounding outline.
Martinez et al. [11] explored ten variations of rendering haptics for objects in mid-air.This included approaches where haptics only was felt on the surface of objects.By setting up colliders on the boundary of objects, MAMMOTH can achieve a similar effect.But as our aim is a plug-and-play toolkit, a future feature is the option to enable surface-only haptics.This can be expanded upon with a feature to vary the intensity between the surface and the inside of objects.

USE CASES
In this section, we show three use cases for MAMMOTH.For all examples, the toolkit must be imported and use the custom discretized hand.

Single object
Rendering haptics for a single 3D object is simple with MAMMOTH.In this use case, the haptics match a 3D sphere resembling a ball.The sphere has the MammothInteractable attached and a sphere collider.Upon interacting with the ball, the user feels ultrasound haptic feedback.The feedback is felt both on the surface and inside the ball.Figure 4a shows the interaction with the ball.

Multi-object
Figure 4b shows an interaction with six objects.The objects are dynamic and move upon collision.MAMMOTH combines the intersection points of all the objects into a single set and computes the optimized 2-Opt.MAMMOTH works with moving objects since existing intersecting points are reevaluated at a specified interval.The interval is the update rate of the haptic pattern.

Dual-instance
Using two instances of the MammothRenderer component and specifying its use for independent sets of Leap Motion and Ultrahaptics enables a social interaction between users.The users feel active haptic feedback when touching each other's virtual hands.Figure 4c shows the reciprocal interaction with the intersection points highlighted in red.

CONCLUSION
Adding ultrasound mid-air haptics to contactless interactions is achieved with either static or dynamic sensations.The static sensations can be inadequate to capture the shape of custom models and the dynamic sensations require expert knowledge to implement.We have introduced MAMMOTH, a toolkit with a rendering technique that automatically generates interaction-based haptics for contactless interactions.Our approach builds on previous techniques to generate the shortest path between points of intersection and render the haptics using spatiotemporal modulation.The technique provides a generalizable solution that works with most 3D objects

Figure 2 :
Figure 2: The first steps in our technique.(a) shows the 3D objects, (b) are the colliders, (c) are the intersection points, (d) are the remaining points after exclusion, (e) shows the path before and (f) after the 2-Opt algorithm.

Figure 3 :
Figure 3: The final step in our technique is the dynamic interpolation separation.The top shows a 0.175 mm separation (static).The near proximities of the points resemble a continuous line.The bottom shows a 2.5 mm separation (dynamic), based on the path length, with visibly discrete points.

Figure 4 :
Figure 4: Use cases.The figure shows the application with (a) a single object, (b) multiple moving objects, and (c) two MAM-MOTH instances at once enabling mediated social touch.