Analysis of Product Architectures of Pin Array Technologies for Tactile Displays

Refreshable tactile displays based on pin array technologies have a significant impact on the education of children with visual impairments, but they are prohibitively expensive. To better understand their design and the reason for the high cost, we created a database and analyzed the product architectures of 67 unique pin array technologies from literature and patents. We qualitatively coded their functional elements and analyzed the physical parts that execute the functions. Our findings highlight that pin array surfaces aim to achieve three key functions, i.e., raise and lower pins, lock pins, and create a large array. We also contribute a concise morphological chart that organises the various mechanisms for these three functions. Based on this, we discuss the reasons for the high cost and complexity of these surface haptic technologies and infer why larger displays and more affordable devices are not available. Our findings can be used to design new mechanisms for more affordable and scalable pin array display systems.


INTRODUCTION
In a pin-array, several pins reciprocate vertically across a flat surface in a compact array.The selective actuation of these pins in specific patterns produces tactile information.Twodimensional pin-arrays are the dominant surface haptic technology for refreshable braille and tactile display devices [11,22,39,52,54,62].However, their high cost has limited their use and adoption [47], significantly affecting the education and employment of people with visual impairments [29].Traditional pin-arrays were based on piezoelectric bimorph actuators, which were expensive to produce [89].Some newer pin-array systems use small motors, and electromagnetic actuators are then replicated for each pin [14].This modularized replication of motorized pins makes devices bulky, noisy, and impractical for large surfaces that are required for bimanual tactile exploration.For instance, to present just twenty braille characters on a refreshable braille display, 160 motors are required [90:20].
In their 2007 review article, Vidal-Verdu and Hafez [78] stated that for a comfortable tactile reading experience with both hands, a 2D tactile display of 32cm x 24cm will be required.These dimensions would accommodate 16 lines of 40 braille cells, with each pin being 2.5mm apart and raising up to 0.4-0.7mmabove the flat surface [56].Making the display the size of a standard braille book can also accommodate sufficient details for tactile graphics.If we imagine this one display capable of presenting both braille and tactile graphics, then it would need 9600 independently actuatable dots.The question is, what would it take for the existing designs to achieve the mechanical movement of 9600 dots practically, and will they be able to do so affordably?Unfortunately, no commercial product or research prototype has achieved it to date.
Therefore, our key research questions are "what is the solution space for the design of pin array technologies?"and "which product architecture is suitable for large surface pin arrays at an affordable cost and a manageable complexity?"To answer this question, we need to interpret and analyze the product architectures of pin array technologies.Product architecture defines how a product functions.It maps a product's functional elements to its physical and actual units or building blocks [75].The functional elements are individual operations in the product to produce its overall function.The physical elements of a product are actual parts, components and subassemblies that implement the functions.The arrangement of functional elements of a product into physical elements and how the physical elements interact with each other define the architecture of a product [75].
Previous literature reviews on pin arrays have captured and classified mechanisms [3,15,16,21,26,78] and materials [8,42,81] that enable the actuation of a pin-array tactile display.However, an analysis of their product architecture has not been discussed in literature.Analysing the architecture of complex systems such as pin-array technologies will help understand these systems' design complexity, provide a clearer overview of the solution space that will help to synthesize better ways to simplify future devices [55].Therefore, this paper contributes (i) a characterization of the product architectures of pin array display technologies including an overview of key functions, (ii) a critical analysis of the mechanisms for each function and their applications and (iii) reflections on the anatomy of these complex systems to discuss ways of simplifying the technology and design, specifically to make large surface pin arrays more feasible and affordable for people with visual impairments.

RELATED WORK
Interpreting the architectures of pin array displays requires a contextual understanding of their application.A significant majority of pin arrays are used in accessibility and education of students with visual impairments.The vertical selective, reciprocal movement of pins across a flat surface allows for differentiated tangential and normal forces to the fingertip as it slides over the surface [27].Moreover, the whole hand can be placed on the tactile image to gain a global shape and layout information [52].As a result, pin array displays replicate braille embossed on a surface [52].Beyond accessibility pin array systems are also found in art installations, sensing applications [57], robotic grippers and fixtures [45,63], VR interfaces [91,92] and shape-displays [17].However, our work focusses on developing refreshable tactile displays for tactile interactions.

Pin-Array Devices for Tactile Interactions
The pioneering invention to refresh pins on a single surface display goes more than a century back to 1916 [62].Today, refreshable tactile displays offer the advantage of providing quick access to the vast amount of information in the form of Braille and tactile graphics, without the need for storing them in voluminous printed media.Such display devices that can present both Braille and tactile graphics are highly aspirational among students, teachers and professionals who are visually impaired [52].There are now a growing number of interactive pin-array displays such as the HyperBraille, which is based on piezoelectric motors; Graphiti [93], a dynamic pin-array display which runs on repurposed coreless DC motors; DotBook [32] and BlindPad [85] which are based on electromagnetic forces to actuate an array of pins and HolyBraille, which is based on microfluidics [61].These notable approaches have created new possibilities for tactile access.However, their cost makes such devices prohibitive for most.For instance, the HyperBraille device comprises an array of 120x60 pins with an inter-pin spacing of 2.5mm that are actuated with piezoelectric actuators, costs $56,000 and is therefore unaffordable to most.The Graphiti display uses low-cost mass manufactured motors that reduces the cost of a 60x40 dots to $15,000 [94], but it is still beyond the reach of many individual tactile readers.The upcoming Dotbook also has an array of 60x40 dots with an interdot distance of 2.5 mm.The dots use electromagnetic attraction and repulsion combined with mechanical locking to raise, lower, and lock the pins in an array [95].It is one of the younger technologies and is expected to cost in the order of thousands of dollars.

Trends in Pin-Array Device Design
To address the challenge of affordability, several innovative designs are published in academic literature and can be found in patents.Brenner et al. in 2000 [12] presented a state of the art of actuators that are used in tactile graphic display systems.They categorised the state of the art into electromagnetic micro actuators, electrostatic microincubators, piezoelectric actuators, shape memory alloys, controllable fluids (electrorheological fluids and magnetorheological fluids) and engineering response gels.Chouvardas et al. in 2005 [16] presented an overview of tactile displays applications that included Braille and tactile graphic displays, medical applications, entertainment and educational applications, military applications and virtual environment applications.In 2007, Vidal-Verdu and Hafez contributed an updated survey of graphical tactile displays that mostly covered mechanical tactile displays [78].They reported the state of the art of different technologies using thermal, mechanical and electrical systems.In 2008, Chouvardas et al. presented a similar review [15].Both the reviews discussed the challenges of complexity of creating large pin array display surfaces and their high cost.While Vidal-Verdu and Hafez suggested multiplexing actuators, Chouvardas emphasized the exploration of new technologies using surface acoustic waves, ER and MR fluids.
Later in 2015, Ishizuka and Miki in [26] have surveyed different MEMS based methods to create tactile displays and categorised them by the type of haptic stimulation they produced on the skin and reported that most of the designs used deformation or vibration.Building on the momentum to investigate specific topics within the growing body of tactile display designs, Xie et al. [81] reviewed the updated state of the art for its use of smart materials in tactile actuators for information delivery.They presented an overview of the technologies and analysis of the advantages and disadvantages of each category which included motors, linear actuators, electroactive polymers, shape memory alloys, piezoelectric, pneumatic and carbon nanotubes.Biswas and Visell [8] recently in 2019, presented an updated review of the materials used in diverse haptic technologies that also included pin array displays.Their comprehensive overview surveys the emerging materials, principles of actuation, fabrication methods and the different designs of tactile displays.Taking examples of precomputing mechanical systems, the authors argue that as the advances in computing architecture from vacuum tubes to transistors has enabled powerful computers to be integrated into small and practical form factors. Similar breakthroughs are required for the tactile displays.The most recent survey on tactile displays for people with visual impairments was published by Yang et al. [83] They presented an updated state of the art and claim that they highlight and analyse the working principles of Proc.ACM Hum.-Comput.Interact., Vol. 7, No. ISS, Article 432.Publication date: December 2023.latch structures used in tactile displays that reduces power consumption but increase mechanical complexity.
These surveys are crucial for engineers as they provide an overview of materials and mechanism to develop pin arrays.However, people with visual impairment worldwide still wait for a technological breakthrough that can make full-page refreshable Braille and tactile graphics affordable.Therefore, a new perspective to understand and structure this state of the art in necessary, that is perhaps closer to a designerly way of working [75], enabling more creative designs of the pin array technology.

METHOD
To understand the design of pin array systems, particularly for tactile displays, we performed an analysis of designs published in different engineering venues and in patents.This includes an analysis of qualitatively reverse-engineered product architectures of the included designs.

Data Collection
Academic literature and patents of a proof-of-concept pin-array display was assimilated using the following search terms: 1. "Taxel", "Pin-array", "Display" from IEEE Explore, ACM Digital Library, Springer Link, SPIE Digital Library, Taylor and Francis Online, Science Direct and 2. "Braille" AND "Graphic" AND "Display" for the Espacenet Patent database.Taxel and Pin Array are the scientific terminologies that refer to technologies that create braille and tactile graphic displays, while the commercial terms for Braille and Tactile Graphic are more suitable for patents.Only publications in English could be included in the corpus (many patents in Japanese, Chinese, and Korean had to be excluded).We also added papers and patents from the reference lists of the literature reviews that have been completed on this topic from [3,12,15,16,21,39,78,83], [8,42,81], [26].We include each paper or patent that reports a unique design concept of a tactile pixel matrix along with its proof.Papers and patents that propose a concept but do not show any proof of concept; that discuss interaction design or human factors based on a previously developed pinarray; and those that discuss the design of actuating a single pixel, a single braille cell, and did not demonstrate a potential for a two-dimensional array were excluded from the database.We excluded single line braille cell technologies from the analysis because they cannot be used for Proc.ACM Hum.-Comput.Interact., Vol. 7, No. ISS, Article 432.Publication date: December 2023.multi-line braille and tactile graphic displays.However, those designs that employ a systematic arrangement of braille cells to create a large surface tactile display have been included in the analysis.Table 1 shows the sources of these papers and patents.
An initial search reported a total of 465 papers and 330 patents.After analysing the abstracts, working principles, schematic diagrams, array sizes and architectures of all the papers by two authors, 34 papers and 28 patents were included for the analysis.Five new designs were also added from existing literature reviews.In total, 67 unique designs were included in the study.As our method was qualitative and dependent on manual search and interpretations, we acknowledge that a small set of papers may not be included in our dataset, however, the included designs represent a rich diversity of literature in this domain.

Analysis Methods
Developing the product architecture is a 4-step process.In the first step, a schematic is developed that lays out the key functions of the product in the form of elements.In the second step, these functional elements are linked to physical systems.The physical systems are then semantically clustered into chunks.Finally, in the fourth step, the chunks are diagrammatically laid out, and the interactions between the chunks are defined.An analysis of architecture first requires the deconstruction of existing designs to identify the key functional elements and the fundamental interactions of the physical chunks [75].
3.2.1 Reverse Engineering Product Architectures.For every included display design, we noted the array size, technology and the key schematic or images that explained the design of the pin array system.We also extracted the paragraph from the paper that explained the working principle of the design.Based on this insight, we reverse-engineered [59] a visual diagram of the architecture from schematic diagrams, 3D renderings, prototypes, and a description of the working principle.
3.2.2Coding Product Architectures.As per the above method, each pin array system was analysed qualitatively to identify its functional categories and mechanisms based on inductive interpretation of the diagrammatic content [96].First, a small sample of data with 10 designs were coded for their function and mechanism.An initial set of themes were inductively derived from the reverse engineered data as no prior theory is available to classify the architectural schematic diagram of engineering systems.Once the researchers agreed with the initial set, with every new design, the themes were evaluated for their fit to the classification, using a constant comparison method [20].Through this process, all the included, interpreted designs were coded.The number of occurrences were also noted to provide a cursory quantitative understanding of the domain.
The classification of architecture was based on subjective judgments of authors, three of whom have a background in product design, industrial and mechanical engineering and have created many new designs both for research and industrial purposes.We conducted this analysis with a deep commitment to improve the designs and provision of tactile displays for people with visual impairments and acknowledge that our analysis reflects our design thinking and biases.

Morphological Landscape of Pin Array Tactile Display Mechanisms
Tactile display systems use diverse technologies, designs, and approaches to realize three key functions.The first function is to actuate selected pins up and down relative to a flat surface, the second is to lock the pins in either state, or the third is to expand the working function over a large surface that can allow for bimanual scanning.
To vertically reciprocate the pins in an array, existing state-of-the-art designs either use fluid pressure to blow up membranes or push selected pins, bend a deformable actuator or uses a material that changes its shape to push and pull a pin or its state to allow the pins to actuate.The more common approaches to actuating pins have been through electromagnetic interactions and motorized systems.To lock the pins in place, the existing state of the art either uses continuous energy to keep the material in its excited state or has a specific interface that locks the pins mechanically.Instead of mechanical locking, a few designs use materials that change their state to lock the pins.Finally, for creating a large array, the working principle can be replicated pin-by-pin to achieve the array.A cluster of pins can be organized in a module and replicated over the surface.Standard actuators across rows and columns have been employed to reduce the number of actuators, or a small array is translated over a large surface on an x-y gantry.One of the potential approaches to significantly reduce the number of components is to integrate functional layers to develop a substantial surface, with each layer covering the entire display.
Table 2.A morphological landscape of the different functions and architectural designs of pin array displays.

Functions Architectures
Raise and lower pins technologies have certain advantages and limitations (Table 3), and only a few of these approaches are suited for affordable large surface haptic displays.

Function 1 -Mechanical Interactions to Raise and Lower Pins
The first function of a pin-array is to raise and lower the pins of an array.This section describes the themes of mechanisms that achieve the function.
4.2.1 Fluid Pressure.Positive pressure exerted on a fixed elastic membrane causes it to expand to create a blister that creates an asperity on an otherwise flat surface [4,5,67].Instead of the flexible membranes, passive mechanical pins arranged per the display's resolution in airtight chambers can also create a tactile dot when pushed by a pressurized fluid [49,98].Typically, the elastic membrane recovers its original flat shape once the positive pressure is removed [9].Otherwise, the compressor can be modulated to create a negative pressure in the system, which sucks down the membrane blisters [5].With pins, the removal or positive pressure along with gravitational pull on the mass of the pin recedes them back, resetting the display [49].In such pneumatically driven tactile displays, high density and replicability of pins are possible due to the compact size of the tube and other small mechanical components at the surface of the display.Bulkier components, such as the compressor and valves, can be kept separately, allowing for wearable pin-array displays [74].
4.2.2Bending Materials.One of the most reliable pin-array devices has been based on piezoelectric bending actuators.These actuators support two layers of piezoelectric materials separated by a passive flexible beam.One side of the piezoelectric material gets contracted upon application of a high voltage.It causes the beam to deflect in that direction, which is resettled by providing voltage to the piezoelectric element on the other side.These bimorphs have been arranged horizontally to raise or lower a dot directly [99], and with a long enough beam, it is possible to achieve the height requirements for braille reading for a single dot [66].The vertical arrangement of piezoelectric bi-morphing actuators has been used in a commercially available display HyperBraille.These vertical piezo reeds bend on one side to push out a wedge connected to a pin, creating a tactile effect, and its bending to the other side recedes the pins back to their flat state [80].While piezoelectric materials bend under high voltages, permanent magnets attached to cantilever beams can also make passive beams bend under electromagnetic forces.For example, Mohammadi et al. [46] demonstrate a vibrating tactile array using a single coil driving mechanical resonance of individual tactile pixels.A ferromagnetic washer attached to a flexible beam gets attracted and repelled at its resonance frequency by controlling the voltage in the closely placed coil and the stiffness of the bending beam.The deflection creates the tactile effect; bending systems under electromagnetic forces can also create displays.

Shape Change.
Shape-changing smart materials can reversibly change their shape or dimensions in response to an external stimulus such as heat, light, pressure, or electricity.These materials can intrinsically act as actuators and are helpful in compact mechanical systems.For example, Nitinol (an alloy of Nickel and Titanium in nearly equal ratio), a shape memory alloy (SMA), has distinct advantages that are well suited for tactile displays.Nitinol's shape memory effect occurs due to the material's ability to reconfigure its crystal structure through a temperature-induced transformation from its malleable martensite to its memorized and stiffer austenite state [7].A shape memory wire hence can change its form when heated.This has been used to create several different tactile actuators.Blazie Engineering filed a patent for an SMA wire-based actuator array of pins in which two SMA wires in an antagonistic pair are attached perpendicularly to the direction of the pin's movement.Selective actuation of the wire by passing current through them contracts the wire, which pushes up a pin and bends the other SMA wire of the pair.When the SMA wire on the other side of the pin is heated, it forces the pin to recede [73].In other embodiments of SMA wires [41,76], springs made of SMA wires are attached in an antagonistic pair to actuate the pins to raise or lower them.These springs occupy a very limited space and can actuate the pins to achieve high stroke and force measurements.
There are a few designs that use SMA sheets to achieve tactile displays.SMA sheets have a major advantage over individual pin arraysthe entire array of tactile pixels can be fabricated at once [7].Mineta et al. [43] developed a planar MEMS-based tactile display with a meandering SMA thin film actuator glued to a bias sheet spring on one surface and a pin on the other.The bias spring pulls the SMA sheet down however, when it is actuated, the sheet tries to become flat and raises a pin.Vitushinsky et al. [79] developed another bistable mechanism using two different shape memory alloys.Their actuator consists of a thin passive buckled sheet metal raised and lowered by two thin layers of different shape memory alloys that operate with heat pulses of different temperatures.The moment of the SMA produces the force necessary to buckle the sheet in either direction, which is connected to raised or lowered pins.Bhatnagar et al. [7] also presented an innovative design in which, tactile pixels were laser cut on a single sheet of Nitinol and were trained to bend out of plane when selectively heated from an external source.
Shape memory polymers are programmable materials that change their phase, demonstrate changes in their stiffness when subjected to heat, and can store elastic energy [25].Nadine et al. [5,6] published and then patented a display device in which a shape memory polymer membrane integrated with a matrix of compliant carbon-silicone heaters, PCB and flexible fluid chambers created a flexible tactile display.Selective Joule heating of the shape memory polymer membrane locally creates a region of low stiffness which expands under the external global pressure supply creating a tactile blister.Upon application of negative pressure and heat, the membrane retracts to its other side, refreshing the display.Actuators made from soft polymer composites are also interesting because they are lightweight, inexpensive, and resilient to fracture and damage [1].Light-induced mechanical deformation can also be found in liquid crystal elastomer (LCE) polymeric compounds that have been used to develop tactile displays in which stress generated under the illumination from a bright LED light exerts a force on a pin to rise.Upon removing the illumination, the actuator recovers to its initial state and recedes the pin [72].Such have increased the possibility to made lightweight displays.
Like shape memory polymers, electroactive polymers exhibit a very noticeable change in volume when stimulated with a high voltage electric field instead of temperature.Electroactive polymers are sandwiched between two electrodes that generate a high potential difference in the material, which brings about the actuation in the polymer.The shape change induced by the high voltage is used as actuators in compact spaces [51].Bar-Choen [2] was the first to conceive a braille display driven with electroactive polymers and since then, there have been many other experiments to use DEAs for actuator arrays [35,40,58].The initial concept from Bar-Cohen had reading pins that were moulded as a dot but were lowered individually to produce a tactile Proc.ACM Hum.-Comput.Interact., Vol. 7, No. ISS, Article 432.Publication date: December 2023.pattern of high and low dots by contracting an EAP film under high voltage.These reading pins were mounted on the film to protect the user from electric shock [30].Newer designs use a constrained DEA membrane in a fixed mechanical setup.The constrained membrane strains out, creates a tactile blister under high voltage, and recovers to its initial flat state when the voltage is removed [35].Positive air pressure is also used to lock the dots in place to reduce the power consumption to maintain the deformation [58].
4.2.4State Change.Few tactile displays utilize materials with low melting and boiling points that change their state to either actuate or lock tactile pixels.The state change enables actuators to be compact when not in use but expand to create a tactile display when they are actuated.For example, Vidal-Verdu et al. [77] demonstrated a thermo-pneumatic actuator for affordable large surface displays with an array of diodes that would selectively heat a chamber filled with a low melting point liquid (methyl chloride).The liquid would turn into vapours trapped in a chamber with an extendible membrane, which blows up to create a tactile dot.The removal of heat then changes the state of the material back to its liquid form, which recedes the expanded membrane.In a patent from Mutsohiro et al. [48] and a paper from Nakashige et al. [49], the authors use a low melting point metal as a clutch to lock the pins in place in their actuated state.A constant air pressure drives the pins out from a flat surface.Selected chambers with the low melting point metal are first heated to melt the metal, allowing the specific pins to pop out under positive air pressure.At this state, the metal is allowed to cool and freezes the pins in their actuated state.By melting the metal again and through negative pressure, the pins are then pulled back to their receded state.
More advanced shape-changing materials like hydrogels solidify from a gel-like state upon heating.Carbon nanotubes expand under heat and have also been explored to create highresolution tactile displays through selective variation in input voltage that heats and expands the material locally [44].The transparent surface has been deformed by selectively changing the viscoelasticity of poly(N-isopropylacrylamide) gel through a heat [65], packaged between two transparent membranes.Selective heating of localized regions solidifies those areas and creates a tactile contrast.
4.2.5 Electromagnetism.A widely experimented and patented mechanism to create a pin array is based on the electromagnetic interaction between a permanent magnet and a changing magnetic field in a coil.A static electromagnet can axially repel a permanent magnet to create a tactile actuation and then attract the magnet to recede it.For example, Streque et al. [68] integrated a permanent magnet in a PDMS membrane and positioned it above an electromagnetic coil so that repulsive magnetic forces stretched and bent the membrane to create the tactile effect, while the elasticity of the membrane bright the display back to its flat state.In BlindPad [85,87,100], researchers explored a matrix of permanent pot magnets between two passive multilayer PCB planar coils.By introducing magnetic shielding between the actuators tightly arranged in an array, independent actuation was possible.The display was an array of 12x16 tactile pixels, had a 10ms refresh time, and a stroke length of 0.8mm.The bistable design was also developed previously by Bowles et al. [10] using a permanent magnet layer sandwiched between two PCB-based solenoid actuators.The bistability of the design was useful in decreasing the energy requirements as current is only required to actuate the pins in either direction but not to hold the pins at a static position.In a different mechanism to lock the pins in place, Leo [37] proposed electromagnets to vertically displace pins against the force of springs, which slide up and then to the side to lock like an inverted ballpoint click pen.The same repulsive force slips the pins out of the mechanical lock, and the spring recovers its initial position to lower the pins.
Dot is a recent commercial start-up that manufactures refreshable braille displays, cells, and tactile graphic displays [101].Their devices make use of an array of small electromagnetic actuators.For example, in the Dot actuator [32,33], an eccentrically placed permanent magnet attached to a pin flips to raise the pin when its electromagnetically repelled by a small coil.The Proc.ACM Hum.-Comput.Interact., Vol. 7, No. ISS, Article 432.Publication date: December 2023.mechanical design locks the pin in place; however, when the electromagnet attracts the permanent magnet, it flips back to its un-actuated state.This way, a tactile dot gets actuated and locked with little energy, and the working principle is replicated for the entire array.
4.2.6 Motorized Systems.In various ways, rotary motors and linear actuators have been packed in an array to drive the motion of pins.Their operability, control, reliable performance, and affordability have been attractive to seek their application in large surface pin array displays.For example, one method is to vertically position the motor's axis, which rotates a screw, which translates a nut connected to a pin which reciprocates it to go up and down [31,53,88,102].In addition, a large array of repurposed vibrational motors, which are simple coreless DC motors, has also been proposed to make a refreshable tactile display.The DC motors have a system of lead screws that translates their rotary motion into the vertical displacement of pins [14].
The rotation of a motor can be translated into linear motion through a crank system.A rotary arm from the motor can create a significant linear displacement when motors are cleverly arranged to form an array.For example, Huang et al. [24] demonstrated the Retroshape prototype, which had a 4x4 taxel grid displaying information on the wrist under a smartwatch.Servo motors were cleverly stacked and placed above each other to create a movable 4x4 grid of taxels using a crank arm from the motor.The crank translated the rotatory motion of the motor into linear movement of the taxel and locked the taxes in place.Building on gear-based transmissions and drive shafts, TextureTouch, in an array of 16 probes arranged in a 4x4 grid, is driven by 16 servo motors stacked in an external rack [91].A rack and pinion system transfers the rotation from each servo to linear travel.The linear motion is then translated perpendicularly using an additional rack and pinion system.
Another mechanism to translate rotary motion into linear motion is a cam system.Chari patented a cam-based array of motors that were vertically and horizontally placed to make a 2D grid of rotary shafts [14].In the system, rows consisted of worm gears that could mate with the passive gears under each pin while the columns were designed like cam rods that could engage or disengage the line of pins.With fewer motors, several pins along a row could be actuated with this design.This mechanism is perhaps used in the Orbit Graphiti display.
To reduce the number of actuators over a large surface, some research work and patents introduce the concept of a two-dimensional gantry system.The moveable system consists of a single or a small array of actuators that moves in a two-dimensional plane to actuate passive pins over a large surface.For example, a patent from Lee [38] demonstrates a display in which an array of passive pins is actuated by a single linear actuator that moves in a two-dimensional plane.So, to actuate a particular pin, the gantry translates the actuator to that pin and pushes it up.In another patent from Roberts et al. [60], a tracker is mounted on a 2D guidance system with a freely rotating metal ball at the end of the tube.The motion of the tracker rolls the ball, which, when it contacts with a pin, actuates it.A more intricate system from Taphouse [28] has a magazine filled with ball bearings.A single actuator releases the ball bearings through the opening on the aperture layers when the magazine and the aperture layer are slid relative to one another.The ball bearings are controlled to be interested in the apertures of the array, forming the display element.It is then reset by releasing the ball bearings and letting them fall by gravity.

Function 2 -Mechanical Interactions to Latch / Lock the Pins
The second function of the pin-array display is to lock or latch the pins in place.Latching pins in either position is critical for the entire time a user intends to read on the display and it has been achieved in three themes that are described ahead.
4.3.1 Continuous Energy.One of the basic ways to latch the pins in their actuated state is to provide the energy to maintain the position continuously.For example, the continuous flow of current in a coil generates a continuous electromagnetic force that repels a permanent magnet Proc.ACM Hum.-Comput.Interact., Vol. 7, No. ISS, Article 432.Publication date: December 2023.
to lock the pin attached to it [50,68,86].Similarly, a continuous flow of air under a selected membrane can keep a taxel inflated [6,58,77,84], and a continuous flow of energy through heat or voltage keeps the actuating material in its bent [19,30,36], contracted [76,82] or expanded [5,35,40].Providing continuous energy to maintain the position of the pin is a straightforward mechanism.However, there are two major challenges with this locking interaction.First, when the tactile interaction with people is expected to be for an extended period or for larger display sizes, which will typically be the case in practice, the system would draw significant energy that will impact its portability and practical use [86].Second, continuous energy input to a specific taxel in a large array can have unintended interactions with the nearby taxels [82].Such unintended consequences can be avoided if the dependency on continuous energy is reduced and a mechanical latching system is used, as described in the next section.
4.3.2Mechanical Latching / Holding.Mechanical systems such as wedges, cranks, and frictional surfaces can preserve energy and keep moving parts locked in place.Using such systems has resulted in a low-cost but effective product architecture that keeps the pin in place against gravity and forces from the interactions without the need for external energy.For example, Zhang [37] uses the mechanism of clicking ballpoint pen to lock pins in place over a large array.The actuators are forced to displace under the repulsive forces of an electromagnet and a passive mechanical embodiment guides the actuating pins to a wedge lock, like the ones in a click ball point pen.The system keeps the pins locked until the electromagnets are excited again to raise the pins and releases the pins from the locking embodiment.Instead of the clicking design, Kim et al [32] propose the flipping latch structure in which an eccentrically placed permanent magnet in a pin embodiment is flipped under an external electromagnetic field created by static electromagnets.The flipping works like a crank and causes a linear motion in the pin.Each pin in their system can be actuated in 5ms and draws 1W of energy only for the actuation, as the mechanical latch does not need any external energy, making the design efficient and practical.This has been used in a display called Dotpad [33,34].Tretiakoff and Tretiakoff [73] in their patent propose the use of a bistable spring attached to the centre of a pin which is actuated in either direction by the contraction of SMA wires attached in an antagonistic pair.The spring creates an initial resistance to movement, but when released in either direction, it snaps and keeps the pins in place until an eternal mechanical force overcomes its stiffness.Roberts et al. [60] propose using an intermediate thin foam plastic sheet with very fine cell structures to hold the pins utilizing friction between the pins and the material of the sheet.A single sheet is an efficient multi-level display approach, as one can be configured to support multiple pin displacements.
The pins were locked in architectures with individual motors or motorized systems, owning to the motor's holding torque used to drive them.Generating rotary motion out from a linear force of the pin can be challenging in a compact array, while achieving linear motion out from rotation is relatively straightforward.Hence, purely by the holding torque the of the motors, the position of the pins can be seized.In lead screw type actuation, the taxels are put in position by the rotation of DC motors [64,71,102], stepper motors [31,88].Based on a crank arm that is attached to a micro servo motor, rotational motion gets converted into a small linear motions [24].Rotational motion of an motor that is placed away from a pin is translated using cams, gears [14], rack and pinion assemblies [91] which also locks the potion of the motor in place depending on the rotation of the motor.

State Change.
Locking of pins in place has also been achieved by deliberately changing the state of a supplementary but integrated material in the product architecture of the display.The function of this additional component is to change its state under external stimuli to lock or release the pins in the place and reduce the need for constant energy output.For example, Bolzmacher et al. [9] made use of MR fluid as a clutch to either lock or release the pins.MR fluid changes its viscosity in an electromagnetic field.When unactuated, the fluid has high viscosity acting as a plug due to the presence of a permanent magnet in the dot.When the dot's Proc.ACM Hum.-Comput.Interact., Vol. 7, No. ISS, Article 432.Publication date: December 2023.respective electromagnet is powered, the magnetic field cancels each other, releasing the fluid and making it less viscous.An external air pressure can then easily expand the membrane which is then kept in place by removing the electromagnetic field and plugging the MR fluid in this new state once again.Using a low melting point alloy, Nakamo et al. [48] proposed that selective melting of the alloy will release the pins that are under and externally produced pressure and would either actuate the pins or recede them back through vacuum.The low melting point material hence acts as a latch to select which pins would be free to actuate, and which would maintain their original state.

Function 3 -Array Design
The largest display till date is a the HyperBraille from Metec that has an array of 104 x 60 independent dots comprised of 6,240 dots driven by vertical piezoelectric bimorphing actuators [56].Some of the recent displays that are being launched in the market supports a matrix size of 60 x 40 dots [14,33], that are 2,400 taxels which is roughly the size of an A5 sheet and half of the A4 size.Most research have demonstrated a small sample array of 3 x 3 to 32 x 24 taxels, while patents often do not claim a specific size.Given the state of the art, the following approaches demonstrate the ways in which large surface tactile displays have been practically achieved.
4.4.1 Pin-Wise Replication.One of the most prevalent methods to create an array has been the replication of the working principle to reciprocate a single pin.Replication seems straightforward as once a compact actuation scheme is designed, it can be mass manufactured and arranged in the array.For example, Szabo and Enikov [70] developed a 5x5 array of solenoids which repelled and attracted a permanent magnet in which each element of the actuation system was replication for the array.Instead of the electromagnetic interactions, reciprocation of pin on a lead screw driven by a motor also has a similar array design in which, each motor, screw, nut and pin architecture is replicated [31,53,64].Replication of SMA springs [76] wrapped around each pin in an array and SMA and antagonistic pair of SMA wires [73] also follows the same array design strategy.Pin wise replication has also been effective in developing arrays from bimorphing piezoelectric actuators that are either arranged radially [69] or in different layers [36] to accommodate pins in a compact array.
However, replication cannot be feasible for developing large arrays with thousands of pins.With every pin, replication would increase every component of that pin module, including its control electronics.This means that for an array of thousands of taxels, there would be 3-4 times that many parts.This complexity can be difficult to fabricate practically.Moreover, with an increase in pins, the cost of the display will also increase, which will likely make the final product unaffordable.Therefore, a pin-wise replication strategy cannot be adopted for developing a large display surface.

Module-Wise Replication.
A more practical replication strategy has been replicating a module of dots.Typically, a braille cell is an integrated module with 2x4 pins, and the braille cell modules are replicated to form a large array.Many commercial technologies use this replication method.For example, Dotpad [103] has and array of 60x40 dots [33] that is made of massmanufactured braille cell modules of 2x4 dots [32].The mass manufacturing of the module and its compact size aids in the affordability and portable design of the display.2x5 array of piezoelectric modules [80] are also used in the HyperBraille from Metec [89].The large surface contains hundreds of actuation modules to output information.Large displays from KGS corporation [23] also consisted of modules of 8x2 piezoelectrically driven actuators that are stacked together to form a large array.The strategy to replicate modules has been more effective than replicating single dots.Modules can be mass manufactured and becomes a common platform that cuts cost and can be used in multiple product offerings.Integrated modules also have an easier control system than dot-wise replication because part of the electronics can be integrated into the modules.Hence, this system can make addressing a specific dot slightly easier and faster.Moreover, to maintain and repair the system, modular Proc.ACM Hum.-Comput.Interact., Vol. 7, No. ISS, Article 432.Publication date: December 2023.architecture enables easy replacement and attachment of new modules.However, despite the advantages over dot-wise replication, replication of modules also faces the challenge of significant cost increase with the number of dots.

Common Actuators Across Rows and Columns.
To decrease the number of actuators required to display tactile information over a large surface, common actuating components across the rows and columns are used to combine and mate their actuations to address selected tactile pixels in an array.For example, in a patent from Chari [14] that is now used in the Orbit Graphiti tactile display, a column of motors driving a spur gear combines its motion with a row of cam shaft that actuates to mate the pins of the row with the spur gear column.The column gear creates a perpendicular motion in by mating with the gear of a pin.In this way, selected taxels in a row are actuated and the entire information can be presented row by row.This system greatly limits the number of actuators that are required in comparison to the replicationbased product architecture and has the potential for large surface scalability.However, increasing the array size by a single column or row of dots would require new actuating modules and hence can become increasingly complicated and expensive for larger surfaces.Furthermore, integrated actuation systems can be difficult to repair.If a motor is damaged, the entire row cannot be rendered.The many gears of the system can also get misplaced or misaligned due to wear over time and hence cause breakdowns.
4.4.4Pin-Array on a Movable Platform.Instead of actuating taxels over a large surface, some prototypes evaluate the use of movable platforms with only a small array that presents information according to the platform's location in the plane.This design approach again reduces the number of compact actuators required to present tactile information but allows the information to be presented over a large surface due to a movable platform.Chan et al. [13] demonstrated using a piezoelectric braille cell matrix that is kept on a carriage in an x-y gantry mechanism that runs on linear guides.Tactile graphics gets revealed when users move the carriage on a platform, and the pin array actuates according to the provided shape in the given region of the display.
These distributed actuation systems reduce the dependency on closely packed tactile actuators to develop a large surface display.It is like looking at the tactile graphic from a small window.As tactile reading is a sequential process, the fingers can also read part of the graphic or information in braille sequentially, making it possible to comprehend shapes through a small tactile array moving over a relatively large surface.However, creating gantry systems can make the eventual product bulky and stationary as portability of the gantry becomes an issue.Furthermore, bimanual scanning or getting a global overview of the image is hard to achieve in this mechanism, limiting the overall comprehension of the shape.Hence, a large actuatable surface is seemingly more appropriate than viewing tactile information through a small window.
4.4.5 Integrated Layer-by-Layer Architecture.To create multiple actuating taxels using fewer components, researchers have developed prototypes of displays based on functional layers of different materials stacked on top of each other to create a tactile display.Each material layer serves a particular function and can easily extend over large surfaces.The pioneering display design using an integrated approach for the entire array was made by Kato et al. [30].They demonstrated a large, flexible, lightweight sheet of refreshable braille cells driven by organic transistors and soft actuators.The display was mechanically flexible, lightweight, shock resistant, and potentially inexpensive to fabricate, making it suitable for portable mobile devices.The sheet-type display is made by laminating three layers of the organic transistor sheet, a polymeric actuator, and a cover layer.The transistor matrix selectively directed the voltage to ionic metal-polymer composite actuators that bend when excited by a voltage.Selective excitation through the transistor array created bending interactions in the actuator, which translated a dot vertically to create the tactile effect.However, using these actuators Proc.ACM Hum.-Comput.Interact., Vol. 7, No. ISS, Article 432.Publication date: December 2023.necessitated high currents to actuate the actuators quickly.The displacement was also small, 0.2mm, which limits braille reading.
Furthermore, the issues of stability of the IPMC actuators in air and the robustness of the organic transistors were limited for weeks.To design a more practical use case for a longer lifetime, the authors visualize a sophisticated method of encapsulation.To overcome these drawbacks, Fukuda et al. [19] developed another sheet like Braille display by integrating the transistor array from a static random access memory (SRAM) unit with carbon-nanotube-based actuator on top of it that is driven by organic thin-film transistors.The display consisted of three functional layers laminated to one another, the CNT actuator layer, the organic TFT layer, and the organic TFT-based SRAM.They reported better air stability, dot displacement and longterm use compared to the display presented by Kato [30].However, a large area of SRAM would be required for a large surface display, which will make the display more expensive.
The largest display published to date using this array design approach is 32x24 from Besse et al. [5,6].The display integrates a thin shape memory polymer membrane with a matrix of compliant carbon-silicone heaters on top of a flexible PCB and a flexible fluid chamber.The integration of these four functional layers enables the selective actuation of taxels, in which positive air pressure in the fluid chamber inflates a selected area of the shape memory polymer that is heated with the array of carbon-silicone heaters powered through the PCB.Only four components are hence responsible for the actuation of 768 independent taxels.However, if any single layer is missed or breaks down, the function of the entire display can be compromised.Feng and Hou [39] developed a 5x5 display prototype using ionic polymer metal composite actuators in a similar design.Each element of the layer of the IPMC actuator is controlled by delivering electrical signals to each actuator pair through a PCB.The signal bends the IPMC actuator, which is sandwiched between the PCB and a backing layer.The bending force from the actuator layer pushes up a PDMS dot to selectively create a tactile effect.

DISCUSSION
The specific requirements to create an array of thousands selectively addressable pins that are placed 2.5mm apart and can reciprocate to 0.4 -0.7 mm over a flat surface has led to a considerable design exploration over the past decades.The state of the art of materials and mechanical systems have been explored from time to time, but this survey that classifies the product architecture of pin array tactile displays.To help guide future research and design, we recap and discuss critical trends from our analysis and discuss the implication for two critical aspects of refreshable tactile displays have been the scalability of the display for bimanual exploration and the reduction in the number of parts to increase affordability.

Current Foci and Growth Opportunities
The foundation of our work is the question "what is the solution space for the design of pin array technologies?".These technologies have been published in many venues including patents as the technology touches mechanical engineering, electrical engineering, computer sciences, HCI and commercial devices.Our decision to filter technologies that are aimed towards interactive tactile displays has shaped our dataset and the conclusions we draw from it.In this framing, we acknowledge that we would have missed pin array systems that are aimed for different scale and applications but would have qualified for their mechanism, such as inForm [17].
However, within the context, our analysis reveals three key functions that are the core purpose of the technology, raise and lower pins, lock pins and create a large array.We reported six diverse methods to raise and lower pins but found that majority of architectures at this scale have used electromagnetic actuators (28.36%,N=19) and motorised systems (23.88%,N=16).A common and an obvious thread between these technologies is their availability in a miniature Proc.ACM Hum.-Comput.Interact., Vol. 7, No. ISS, Article 432.Publication date: December 2023.form factor.Most locking systems use continuous power (55.24%,N=35) while the most common array architecture is based on pin-wise replication (52.24%,N=35).We believe that certain combinations of subsystems, 'motorised + pin-wise replication' (17.9%,N=12) and 'electromagnetic + continuous power' (10.4%, N=7) can be avoided as they lead to high bulkiness, complexity and power requirements.
This research domain has been increasingly drawing from material research.Advances in materials, and miniature actuators often finds its demonstrative use to pin tactile display systems.SMA actuators, shape memory polymers and piezoelectric bending actuators are (34.32,N=23) are used frequently as they have favourable technical performances given the space constraints and can be fabricated in small sizes.However, these materials themselves are uncommon and when combined with pin-wise and module-wise replicated architectures (14.9%,N=10), they would also lead to higher costs of making pin array.Therefore, innovative combinations of materials and mechanisms for the three key functions are required to create new designs that are more affordable, less complex and at par with the required performance for braille and tactile graphic reading.We discuss the potential design solutions ahead that address our second research question.

Simplifying Pin-Array for Scalability and Affordability
Every included paper and patent is an attempt to address the challenge of affordability and scalability of pin arrays for tactile interactions.These two aspects are the most critical challenge in this research domain.Scalability is essential for bimanual exploration which leads to a meaningful tactile interactive experience [78] and affordability is necessary to make devices that reach its users [65].Therefore, it is necessary for design to find ways to simply the system so that it's possible to scale and make it more affordable.In fact, bending actuations with a layerby-layer architecture had some of the highest array sizes on average (10x10, 12x12, 12x24, 27x27 and 32x24).This insight indicates a potential architectural combination that may be used for large surface displays.For example, Besse et al. [5] have demonstrated a tactile display with four components that can actuate 768 independent taxels through a layer-by-layer architecture using shape memory polymeric actuators.Using a similar architecture, Fukuda et al. [19] demonstrated a 12x12 array of bending actuators based on carbon nanotubes.Hafez [21] has also discussed a concept of a tactile actuator using a single sheet of a shape memory alloy material with all the array actuators integrated into this one sheet.The actuators are small cantilever beams that work in bending to create a tactile asperity that can represent a braille dot.Bhatnagar et al. [7] demonstrated the feasibility of the concept using a thin sheet of Nitinol in which 729 tactile pixels were laser cut and trained on a single material.However, in the present state of the art, five of the six designs require continuous energy to maintain the position of the pins, and all are made of advanced materials (IPMC, CNT, SMA, Piezoelectric bimorphs and IPMS).We see this as a current limitation of the state of the art.
Affordability of the overall pin array display will be depended on the materials, number of parts, its fabrication process and design complexity [18].Expensive base materials, with many parts required to actuate each pin, complex fabrication processes and an exceedingly high amount of design complexity will continue to make pin arrays prohibitive.Bending actuators arranged in a layer-by-layer architectural combination has shown to reduce the number of parts required to make the display and reduce the complexity of the design.Future work can investigate mechanical locking designs, more affordable materials and manufacturing processes for this architectural combination.

Reflections on our Research Process, Position and Limitations
Rigorous curation of pin array systems meant for both refreshable tactile graphics and braille to generate a dataset and applying codes was a challenging and effortful process.Extracting these designs from multiple scientific libraries and patents was necessary because of the Proc.ACM Hum.-Comput.Interact., Vol. 7, No. ISS, Article 432.Publication date: December 2023.diversity of research domains and commercial interests that overlap this interactive surface modality.We manually screened each paper, understood every working principle of the included papers, reverse engineered the product architectures from the given data and coded it to contribute a new, morphological chart for classifying the state of the art.We have explained each category of the morphological chart of pin arrays with examples from the corpus that helps in understanding this new method of clustering.Based on this exercise, we identify and discuss a potential architectural combination can be used towards affordable and simpler tactile displays.
However, beyond the challenges and the contributions, our work has some limitations.First, the classification system developed reflects our position and approach to design new pin array tactile interfaces.Other researchers may have defined these codes and classifications differently.Second, although the corpus represents a wide variety of literature in this domain, it may not be exhaustive.There may be more large surface tactile displays designs that may use a different mechanism than what is presented in the morphological chart.This is unavoidable as new design solutions are continuously invented addressing each of the three functions.Third, for each design we did not analyse the technologies' performance in terms of refresh rates, force outputs and displacements.These can vary with materials and mechanisms and will significantly affect the usability of the interactive system.But we had to limit the scope of this paper to purely the mechanical architecture of these systems, considering that in future work, with design and development, the required performance parameters can be achieved with novel combinations of architectures.Finally, our survey is limited to a corpus of papers specific to pin array tactile displays.As discussed in the related works, pin array systems are used for a diversity of applications, from table-top interfaces [17] to interactive environments [92].The mechanisms used in these systems may be useful for inspiring the design of tactile systems but due to the significant difference in scale, performance requirements and application areas, we chose not to include them in the current scope of this review.
these in a visual format and summarizes all the possible solutions to each function from the existing state of the art.The morphological chart [97] captures the entire included landscape of mechanical designs to achieve the reciprocating motion of dots over a large surface.The different technological approaches in literature embody a unique combination of mechanisms to design a display for each of the three functions.However, all actuation Proc.ACM Hum.-Comput.Interact., Vol. 7, No. ISS, Article 432.Publication date: December 2023.

Table 1 .
Libraries, search terms and results

Table 3 .
Summary of advantages and disadvantages of actuation technologies We now discuss each function and mechanism in greater detail with examples.