Using 3D Mice to Control Robot Manipulators

Fluid 6DOF teleoperation of robot manipulators enables telemanipulation where autonomy is not possible, facilitates the collection of demonstration data, and aids routine robotics development. Amongst 6DOF input devices, 3D mice stand apart for their ergonomic design and low cost, but their sensitivity and users' relative inexperience with them require special design considerations. We contribute a web software package that makes integrating 3D mice in robot manipulation interfaces easy. The package consists of configurable input signal processing schemes that can make the device more forgiving by, for instance, rejecting small inputs or emphasizing a dominant axis, and an interactive visual representation of the device's 6DOF twist input, which helps with operator familiarization and provides a visual aide during teleoperation. We provide a demonstration interface illustrating a typical integration with a ROS/ROS2 robot system and give usage advice based on our research experience.


INTRODUCTION
Robot teleoperation systems have become vital in sectors like healthcare, space exploration, and manufacturing [13,17], where they enable colocated or remote operators to complete challenging manipulation tasks.While a rich literature has characterized the use of master-slave systems, where the operator interacts with a robot arm which supplies haptic feedback, there is now increasing interest in lower-end industrial or home use cases where less precision may be acceptable and a lower cost required.Cheaper teleoperation input devices such as teach pendants, joysticks and gamepads [21,24] can be cumbersome and lack the degrees of X Y Z Figure 1: The 3D mouse is a small desktop input device that supports simultaneous 3D translation and 3D rotation input.
freedom to enable comparable fuid motions.One device that addresses these requirements is the 3D mouse [23]-often called a spatial mouse or a SpaceMouse, after the brand-which is explicitly crafted for navigating and interacting within three-dimensional spaces [10,11].Similar to a joystick, the device, shown in Figure 1, is attached to a fxed base and the user supplies a displacement and/or a rotation, usually with a single hand.This relative input requires only small movements from the operator, enhancing its ergonomics for extended use [7], compared to a master-slave system or a motion capture system [8].
We explore the 3D mouse for controlling the end-efector of a robot.Previous works have compared 3D mice to haptic controllers and have also been used for collecting data for imitation learning, indicating an increased demand for 3D mice in research applications [4,5,14,25].Numerous studies have demonstrated the utility of a 3D mouse in teleoperation [12,15,16,22].While these works have explored various tasks using the 3D mouse, there remains a gap in the availability of open-source software solutions.We address this gap by providing software designed to enhance 3D mouse-based teleoperation systems.

PURPOSE
The objective of our software package is to make 3D mice usable for robot operators.While the short throw of common 3D mice-a maximal translation displacement of 1mm and maximal rotation of 5deg-makes the devices suitable for extended use, it also makes them less forgiving for novice or less dexterous operators, and is the root of several common difculties: (1) Sensitivity: Users are often surprised that the device begins registering input even for minute movements.Many fnd it difcult to rest their hand over the device without registering inadvertent input.
(2) Dimensional coupling: The device is free to move in any dimension and has no bias towards particular axes, so users may struggle when attempting to isolate a dimension.(3) Limited feedback: The device's mechanical design afords haptic feedback in the form of proportional counter-pressure, but the small scale of the feedback is often insufcient for new operators to interpret whether they are supplying the input they intended.Because this feedback is not controllable, 3D mice also cannot indicate to the operator that the robot controller is unable accomplish the commanded input, due, for example, to joint limits, singularities or obstacles.
We address these issues with a confgurable processing pipeline and a 3D visualization of the user's input.

CORE FUNCTIONALITY
3D mice produce a 6-dimensional signal s = [t; r] where vectors t = [ , , ] and r = [ , , ] represent translation and rotation displacement about axes of the device.Our package helps users understand their input and aids in the processing of the signal to produce a twist output S consisting of angular velocities and linear velocities suitable for commanding a robot manipulator's end efector.
Input normalization.We standardize the confguration of the input axes-X right, Y forward, Z up when seated behind the deviceand scale values to be in the range [-1, 1] with the device rest state at 0, creating a uniform basis across varying device specifcations.
Button normalization.Most 3D mice include one or more buttons, which can be used for arbitrary commands.Our package extracts, labels and forwards the state of the device's buttons alongside the processed control signal.

Signal Processing
Deadband.Inputs within a zone [−, ], with ∈ [0, 1] are nullifed to reduce the occurrence of minor, unintentional inputs.This is accomplished with flter function which removes values in the deadband, and applies a transformation (), scaling the remaining values appropriately so that input begins smoothly at the boundary of the dead region.Figure 2 plots Equation 1 with an identity transformation and = .1.
Input Sensitivity Curve.Users often struggle to provide input in the lower range of the device when teleoperating small motions.In contrast, users rarely need to diferentiate between similar near-maximal inputs.Following common practice in gaming applications, we redistribute input precision to favor smaller values using a cubic sensitivity transformation, () = 3 + (1 − ) [6].The parameter of this operation, ∈ [0, 1], interpolates between cubic and linear functions.A value of 0 results in a scaled linear deadband, while values approaching one bow the line toward the x-axis, with a value of 1 resulting in a smooth (diferentiable) transition across the deadband boundary.Figure 2 plots Equation 1 with cubic transformation and = .1.

Input
Figure 2: Left: A linear scaled deadband rejects input in a small region around 0, and scales the remaining input range to cover the full output range.Right: A cubic scaled deadband acts similarly, but applies a cubic function to increase input sensitivity for values.
Dimensional Weighting.Providing a bias in favor of axis-aligned unidimensional translation or rotation inputs helps novice operators overcome the difculty of isolating dimensions.We provide a softmax reweighting flter which increases the contrast between small and large values in a vector x, creating reweighed vector weighted = where exp( −1 | |) We apply this scheme to both r and t.If the device, for instance, registers a large value for translation component and small values for and , the softmax function assigns greater weight to , ensuring that the robot primarily moves in that direction.The degree of this emphasis is confgurable using the temperature parameter , resulting in no reweighting as → ∞, or in hard-max behavior as → 0 + .We preserve the magnitude of the input by scaling x weighted to have magnitude ||x|| 1 .
Temporal Smoothing.The near-instantaneous response of the device can make it feel jumpy to operators.Temporal fltering mitigates this issue, but by its nature, introduces a degree of latency to the control.We've found that an exponential moving average, also known as a low-pass flter, applied component-wise to s, setting

Visualization
Limited haptic feedback is an obstacle to 3D mouse usability, so we designed a twist visualization to provide feedback tailored to several common user needs.Figure 3 shows the visualization in several confgurations.
Representing High Dimensional Input.Users' main challenge is understanding whether they are inputting what they mean to be inputting.The device provides counterpressure opposite in direction and magnitude to the user's input, but it can be too subtle for some users.In order to represent both their 3D translation and 3D rotation input on a 2D screen, we render two axes markers in a 3D scene; a translucent anchor which serves as a reference point, and a moving target which represents their input, calculated as the pose target = exp(S).The target's motion is scaled to be several times larger than the physical device's true displacement to increase legibility.The scene is rendered using a perspective projection, causing parallel elements of the marker to converge to a point in the distance.To further enhance depth perception, the scene is rendered with "fog"; as the marker moves further from the camera, its color lightens and contrast is reduced, and as it moves closer, its colors darken and become richer.
Distinguishing Translation and Rotation.Rotation control is often novel and challenging for users, who can struggle to understand whether and how much rotation they are applying.The orientation of the target marker with respect to the reference encodes this information, but to make it more direct, we render a "twist trace"; a cord which connects the origins of the axes markers.This line, calculated by stepwise integration of the twist, bends into an arc as rotation is increased, creating a clear visual signature for translation inputs which include rotation.

CODE
3dmouse-hid is listed on NPM, the major Javascript package index, and its repository is available at https://github.com/hcrlab/3dmousehidunder the permissive BSD 2-Clause License.The core of the library consists of fewer than 1000 lines of code, and depends on only a single 3rd party package, factors which we believe will simplify community contribution and minimize maintenance burden.

USAGE NOTES
Installation instructions and inline documentation describe the details of our software, so in this section we highlight high-level usage considerations.The discussion in this section references an included teleoperation interface-shown in Figure 4-which serves as an illustration of a typical integration.The fow of information and division of responsibilities is charted in Figure 5.
Browser Support.Our package uses WebHID (Web Human Interface Device) to communicate with devices.Google Chrome and Microsoft Edge provide support, but WebHID remains a contested specifcation, and there are limited prospects for its adoption in other browsers [3].
Device Support.We include support for the current range of 3DConnexion SpaceMouse devices.Our code can work with any similar, HID-compliant device with the specifcation of button and axis bitfeld mappings.The included specifcations are provided in deviceSpecs.js.With further modifcation, this code could also be used to interface with alternative input devices which use the HID specifcation, including commercially available sip-and-puf devices.Note that using lower dimensional input devices would require the design of a modal control interface or an inferencebased assistive teleoperation system.While we do not include an example of such an interface in the package, there is signifcant prior work detailing the use of low dimensional input devices, such as 2D joysticks, to control 6D motion [1,9,19].
Getting Values.The ThreeDMouse class wraps the procedure for acquiring a WebHID device handle, receiving raw HID events, and processing them.It emits 3dmouseinput events which carry a timestamped frame of fltered output.
Buttons.Applications will have varying uses and needs for buttons, so we do not attempt to prescribe any functionality or fltering for them.In our applications, we have found it useful to map buttons to UI functions like changing the input mode or swapping camera views, or to simple robot functionality like homing the robot or toggling the gripper.More advanced uses would beneft from application-specifc button state management and debouncing.
Filtering.ThreeDMouse can be confgured with a flter object which can apply arbitrary operations to device signals before they are emitted.We provide a SmoothingDeadbandFilter class which exposes the full set of parameters described in Section 3.1, making it confgurable for many applications.The appropriate fltering confguration depends on the task and the users.While it is likely not appropriate to expose the full bank of parameters directly to users, it may be valuable to ofer a range of pre-tuned profles or a calibration procedure to ensure broad usability.
Mapping Input to Robot Motion.We've found that transforming the device's twist to a robot frame for control can require careful design consideration to match operator expectations.In our demonstration remote teleoperation interface, we provide a front facing view which tilts slightly downward to show the plane the robot rests on.Given such a view, users generally expect the axes of control to align with the axes of the robot's base.They implicitly map the X and Y axes in the base frame to the X and Y axes of the device, aligning them so that the motion of the device corresponds to the motion of the robot in the given camera view.Although "up" from the camera's perspective is tilted slightly away from the gravity axis, users still expect control to occur aligned to gravity.The interaction between the scene context, the position of the robot with respect to the camera, and varying user and task needs means that the input transformation needs to be designed per application.
Robot Controller.6DOF input is most suitable for end-efector velocity controllers, which directly accept 6DOF twist goals.In our teleoperation setup, we have used MoveIt Servo [2], which converts twist commands to joint velocities using diferential inverse kinematics.
Orienting the Visualization to Match Robot Motion.In order for the TwistViz visualization to serve as an efective feedback mechanism, it must always be perceived as matching the physical movement of the device and the movement of the robot, however this becomes impracticable when multiple, disparate camera views are used (as is commonly necessary to handle occlusion).Movement of the device can only naturally correspond with movement of the robot in a single view, so the input transformation must be modal and change according to the active view.It's possible for operators to become disoriented if they are controlling the robot while looking at a view that does not match the active input transformation.In our example interface, we create a clear visual hierarchy amongst views and we draw the twist visualization on top of the primary view to emphasize their connection.

Ethical Considerations
This package and other tools for remote teleoperation risk promoting the thoughtless removal of colocated operators, which can negatively impact safety by increasing worst case latency of human intervention.Further, applications of remote teleoperation may function to reduce worker power by dispersing human workforces and shifting them to areas with weaker labor protections.

Responsible Use Guidelines
Our package may exacerbate safety issues with teleoperation systems if misconfgured or applied without appropriate safeguards.In particular, failure to confgure appropriate output scaling may lead to the generation of large velocity commands.We recommend adopting a compliant controller, which can guarantee the robot will not exert excessive forces regardless of operator input.Basic safety limits like velocity and acceleration limits must also be enforced to protect the robot's actuators.
It is up to users to ensure that their interfaces adhere to web accessibility best practices [18] and to provide sufcient alternative input schemes for users with limited dexterity [20].

Utility and Relevance
The 3D mouse is the most widely available and most cost efective 6DOF input device.Our package democratizes its usage in robot teleoperation, addressing common issues facing operators with the use of signal processing and visual feedback.For researchers, this package may be useful in systems for Programming by Demonstration (PbD) or Learning from Demonstration (LfD), where it can facilitate data collection by reducing familiarization time for novice operators.The device's ergonomic benefts compound with usage duration, so it may be suitable for use by professional operators supervising or teaching feets of industrial manipulators.It may also be useful for bimanual teloperation, where the cost and ergonomic benefts are doubled.
Our package also flls a void for researchers already using 3D mice, providing a well-known reference implementation against which they can compare their implementations and-we hope-to which they can contribute their ideas and improvements.While more complex than day-to-day input devices, by providing a common core implementation we can promote efective, reproducible use of 3D mice in robotics research.
smooth prev = +(1−) , works well to address a range of situations with the appropriate confguration of decay weight ∈ [0, 1].Tasks which require smoother operation can set lower values, while those benefting from maximal responsiveness can disable the fltering with a value of 1.

Figure 3 :
Figure 3: The twist visualization used to provide feedback to operators, shown with diferent combinations of a translation (right column) and a rotation (bottom row).

Figure 4 :
Figure 4: The included demonstration web-interface provides an example of how the package's components can be integrated.

Figure 5 :
Figure 5: Software architecture in a typical usage