Textile Robotic Interaction for Designer-Robot Collaboration

This late-breaking report describes lab-based robot experiments involving two robot arms scanning and interaction with a set of 12 novel sustainable materials programmed with handfeel gestures inspired by how designers evaluate textile materials. The aim of gathering this data is to spur research in robot perception of soft materials and to contribute towards human-robot collaborative design systems. The complete dataset including scanned images, video of interactions accompanied by the robot motion paths is available with code at https://github.com/rca-msrc/textile-robotic-interaction-HRI2024.

and has been applied to textiles [2,6] and in classifying textile's structure and properties visually [5].Existing robotic-textile interaction research focused on predicting classes of textiles through tactile sensors [7,8], or shape of the textiles through image-based techniques [10]).Predicting physical, subjective and sensory properties of materials through visual analysis of robotic interactions inspired by a designer's handfeel gestures is an underexplored area which we aim to contribute towards with our experimental datasets.
The rest of the paper describes the methods and set-up of the experiments with initial evaluation of the data gathered.Section 3 describes the frst experiment's methods of scanning textiles and Section 4 describes the second experiment's methods handfeel experiments.Finally, Section 5 concludes with potential uses for the experiment's data for the HRI community.The technical details of the scripts and used to run the robot experiments are available at https://github.com/rca-msrc/textile-robotic-interaction-HRI2024.

ROBOTIC TEXTILE SCANNING
Both experiments utilised two UFactory xArm71 robot arms, to interact with 12 sustainable fabric materials (Table 1).In the frst experiment, described in this section, one robot methodically image scanned a draped fabric (held by another robot arm as in Figure 1).The aim is to have the robot classifying the fabric while scanning, thus exploring the physical space of proximity to the fabric.

Scanning Methods
One robot holds the fabric in a static draped position.The other robot arm then follows a static pre-programmed path with a tablet device attach on its end-efector.The device runs an image acquisition application that records time-stamped images at regular intervals (about 100ms).
3.1.1Defining the Path.Through manual and bespoke experimentation, key-points were recorded around the draped fabric and using the xArm's API2 , a linear path was interpolated (at a given speed) for the robot to guide the image-recording app.Positions were defned from the perspective of the camera application, experimentally (see robot_calibration.ipynb notebook in the supplementary code).Four key-point positions were defned to interpolate between for the robot to achieve good scanning coverage of the draped fabric: Top Center (where the camera was positioned pointing where the other robot held the fabric), Center Center (camera positioned directly at the center of the fabric), Center Right, and Center Left (camera positioned on the right and left facing the center height of fabric).
These four positions were defned for four levels of proximity to the fabric's surface (roughly stepped by 5-10mm): close, mid, normal, far.With these positions defned, a path (an ordering of these positions) was defned (see the the scan.pyscript in the supplementary code).
3.1.2Running the Scan.Each of the 12 fabric materials (Table 1) were scanned.The scanning robot moved from path position to position at a set speed (50 from a range of 1 to 250), and at each position the robot would pause for 1 second before moving to the next position.For each scan, the tablet-based image recording app recorded images at full resolution which was 1072x1072 pixels and at an input resolution 320x320 pixels which is a cropped region from the center of the image.
In addition, while scanning, the robots recorded their positions (broadcast from a network socket from each of the robot arms).These position frames were recorded at roughly 100ms interface (using the report.pyscript in the supplementary code).Since there are two robot arms, for each scan both robot's positions were recorded (the fabric holding robot was static for the entire scan).In order to mark-up the scanning routine and align with the robots' position, timestamped markers were recorded as the robot moved from position to position.
Finally, a top-down mobile camera recorded each experiment, providing an overhead video of the scan.

Running the Experiments.
Each fabric had at least 1 scan.Fabric 12 (ffe_12) had issues with the frst scan, and thus a second scan was recorded, and thus there are two sets of data for that fabric.Fabric 3 (ffe_03) ended up being partially scanned twice (due to performance issues with image acquisition application).
3.2.2Evaluation of Images.In total, 50,692 images were gathered for the 12 fabrics (median of 3637 images per fabric).As the robot arm and the image scanning application was decoupled, the arm moved during the scanning process.This resulted in a number of images experiencing motion blur.One rough metric for evaluating blurriness is calculating the variance of the Laplacian for the image's pixel values (less hard edges in an image results in a lower variance) [1]. Figure 2 shows a comparison of a clear and a blurry image for the same fabric, Fabric 9 (ffe_09), with their blurriness scores which are relative for each fabric material.Figure 3 shows the distribution of all the images for that fabric (of which 72% of the images are considered blurry, below threshold of 55).This indicates a tighter coupling of image scanning and robotic motion is necessary to reduce the blurriness of the scanned image dataset (see the blurri-ness_analysis.ipynb notebook in the supplementary code for further analysis)

ROBOTIC HANDFEEL EXPERIMENT
This experiment was inspired by videos of fabric handling used by fabric suppliers to convey the handfeel of the material.As opposed to displaying static images of their fabrics, a video 3 showing a standard set of gestures of a hand interacting with the fabrics is displayed.In this experiment, we program two robot arms to replicate a very simple gesture of two thumbs rubbing the surface of a fabric.This is recorded by video from two perspectives, a top-down and a side-angle view.

Method
Two robot arms are positioned symmetrically over a work area (Figure 4a.).The work area is roughly 20cm by 30cm and the surface 3 https://www.upwhk.com/shop/commodity/1140The robots interact with the fabric with a static end-efector "thumb", made of 3D printed ABS plastic, where it is rotated to roughly 45 degrees from the surface of the fabric and moves in a "pulling" motion along various axes (Figure 4b.).Because the aim it to replicated a "pulling apart" gesture, both robot arms move their thumbs in synchronous and symmetrical motions.
The thumbs have two sides to them, and the gestures are repeated for each side.One side is the "rounded" side of the thumb, and the other is a "sharp" nail side of the thumb (Figure 4c.).In early tests we found the sharp side to have more pull on the fabric, and to create a variation of interactions of the thumb with the fabric, ran experiments with both sides.
4.1.1Defining the Gestures.The robot arms conducted three different types of "pulling" motions on the fabric (thumbs starting at ca. 5 cm apart), each along 3 axes: X, Y, XY (in a diagonal direction).For directional fabrics, this mean that the motion was applied in both the warp and weft direction, as well as a 45 degree diagonal.
For each "pulling" motion, the robot arms applied increasing pressure of the thumb into the fabric material, determined through initial experimentation.This was done by setting the Z height of the thumb onto the fabric from barely grazing the surface of the fabric, and linearly increasing the Z position of the thumbs in multiple passes.
4.1.2Defining the Experiment.All 12 fabrics were tested twice in this experiment -one for each side of the fabric, designating one side A and one side B. The fabrics were pinned down to the foam surface of the work-area at each corner and side.For each fabric, the pulling motions are applied for 5 increasing levels of pressure, repeated for both ends ("round" and "sharp") of the thumb.

Handfeel Results
4.2.1 Running the Experiments.Each fabric had both sides evaluated with the robotic handfeel gesture, with one exception.Fabric 8 (ffe_08), a non-woven harder composite material, began to be scratched severely under increasing levels of pressure, thus the motions were cut of early to prevent destroying of the material.

Visual evaluation.
Visually comparing the resulting top-down video (Figure 5 shows a screenshot in mid-handfeel motion) with the video produced by the supplier 4 indicates similar displacement of the knit structures of Fabric 9 (ffe_09) (which is the only fabric for which supplier videos are available).However, in the supplier video, the thumb gesture is accompanied by an index fnger pinching motion which further expands the knit, which is an opportunity for future improvement in robotic manipulation.

CONCLUSIONS AND FUTURE WORK
The intention of this dataset is a preliminary exploration where a robot interacts with textile materials.The ultimate intention being to promote a collaborative system for designers to work with soft materials -notably textile materials for fashion and textile designers.In particular, robotic systems to help assess sensory properties of textile materials to help scale material libraries for designers.The data collected during these studies should be visually analysed further to gain a greater understanding of each textile's behaviour under the same controlled manipulation.Measuring stitch distortion and comparing warp and weft behaviours when handled, would suggest richer tactile data compared to still images.
Potential uses of these datasets of images and videos would be to train a robotic system to optimise motion trajectories for successful assessment and identifcation of textile materials and their properties.In particular, the motion paths in this dataset are statically defned, so this dataset might contribute towards dynamic path planning for a robot to assess a material.
More broadly, successful assessment of textile materials provides a robotic system with more information on how to better approach and manipulate these materials.Soft materials, such as textiles, are challenging for a robot system to grip and manipulate as opposed to rigid materials.For these tasks, allowing a robot to predict the type of material helps it select a better policy for handling and manipulating it.Future work would be to understand and predict sensory properties of textile materials through physical analysis with robots.Such data would also be useful for designers when sourcing materials online, especially when comparing like for like alternatives.

Figure 1 :
Figure 1: Robot scanning in-progress using tablet running bespoke image acquisition application ("center left" position)

Figure 3 :
Figure 3: Histogram of the blurriness scores (variance of Laplacian) for all of the images scanned for Fabric 9 (ffe_09).

Figure 5 :
Figure 5: Screenshot close-up from video recording of the robot thumb pulling Fabric 9 (ffe_09) in an analogous motion as in supplier video.

Table 1 :
Fabric materials used in this study (more detailed properties in materials.csv in dataset)