A Way for Deaf and Hard of Hearing People to Enjoy Music by Exploring and Customizing Cross-modal Music Concepts

Deaf and hard of hearing (DHH) people enjoy music and access it using a music-sensory substitution system that delivers sound together with the corresponding visual and tactile feedback. However, it is often challenging for them to comprehend the colorful visuals and strong vibrations that are designed to represent music. We confirmed that it is necessary to conceptualize cross-modal mapping before experiencing music sensory substitution through focus group interviews with 24 DHH people. To improve the music appreciation experience, a cross-modal music conceptualization system was implemented herein, which is a prototype that allows DHH people to explore the visuals and vibrations associated with music to perceive and appreciate. An evaluation with 28 DHH individuals demonstrated the capability of the system to improve subjective music appreciation experience via music-sensory substitution. Eventually, DHH people with negative attitudes toward music became positive in the exploration and customization process with our system.


INTRODUCTION
Music is the art of combining sounds to create a composition that uses melody, rhythm, or other expressive content [90].It helps people express their emotions and enriches individual and social functions, such as intellectual gratification, communication, and social cohesion [28,30,41].Deaf and hard of hearing (DHH) people have difficulties experiencing music through conventional soundcentric media, but they can sense and enjoy music in various ways, such as by sensing the variations in the vibrations of a loudspeaker, playing instruments, or dancing [14,82].Using a music-sensory substitution system is one of the approaches to support music accessibility for DHH people; this system allows them to interpret and experience music by replacing musical elements with other senses, such as visual and vibrotactile feedback [46,80].However, intuitively designing cross-modal mappings that allow DHH people to enjoy music using other senses is difficult.To feel the substitute stimuli as music when designing a music-sensory substitution system, the complex relations among various musical and alternative sensory elements must be identified and interpreted [1,16,37].
As the most common music-sensory substitution system, tactile information has a positive effect on DHH people by allowing them to "feel" the music.However, complex musical elements, such as melody or timbre, which correspond to vibration elements can be better recognized by personal optimization of haptic sensitivity (intensity) and by training to interpret vibration elements [46,80].In graphic-based visualization, wherein musical properties, such as pitch and tempo, are presented as graphical metaphors, is more important about the process of training or personalizing the relationship with alternative senses than that the tactile approach [16,25,43].Fourney and Fels [17] reported that many audio-tovisual stimuli are transmitted as only monotonous stimuli if they are not signified by DHH people.They explained the importance of facilitating DHH people to interpret mapping relations when converting visual elements into musical elements.The internalization of musical elements that the alternative senses express must be first conceptualized to sufficiently perceive music.
Herein, a music-sensory substitution system is proposed based on a cross-modal conceptualization framework derived through focus group interviews with 24 DHH people; and validate design elements from the user study with 28 DHH people that do not overlap with previous interviews.By analyzing the potentials and challenges of existing methods for substituting music with visual and tactile feedback through focus group interviews, we confirmed that improving the satisfaction of music in music-sensory substitution systems requires a conceptualization phase that perceives crossmodal mapping and supports personalization.We implemented a cross-modal music conceptualization system that presents music with visual and tactile feedback.Validation with between-group analysis revealed that the proposed system enhanced both participants' perception of cross-modal mapping and the experience of music appreciation through sensory substitution.
The two main contributions of this study are as follows: • Comprehending the needs and challenges of music-sensory substitution systems for DHH people through focus group interviews, we found that a conceptualization framework is required to internalize the relation between music and sensory substitutes to appreciate as music.Thus, we derived two strategies: exploration and customization.• We implemented a system to help internally synchronize the cross-modal mapping based on the derived strategy.Through a between-group study with 28 DHH people, we verified that the conceptualization framework positively affects their actual music appreciation experience.

BACKGROUND 2.1 Music-sensory substitution system for DHH people
Alternative sensory channels, such as visual and tactile, have been used to support DHH people with barriers to experiencing music in activities ranging from appreciation to music to creative activities.
As summarized in Table 1, tactile feedback has been used in various ways for music appreciation [36,37,61,83], and instrumental performance [58,78,84].However, visual feedback is typically not used alone, but is combined with tactile feedback to provide detailed feedback on the informational aspects of music learning, such as playing [40,98], singing [101], training [27], and music-making [48,73,86], rather than music appreciation.For DHH people, the tactile sense is often more intuitive and natural used than the visual sense, allowing them to experience music through sensations, such as feeling the vibrations from loudspeakers [11].However, research on enabling DHH people to appreciate and express music through the sense of vibration has primarily focused on rhythmic elements-driven presentations using vibration [38,48,72,73].This is due to the limited bandwidth of the tactile vibrations that can be perceived and differentiated by humans compared to the auditory bandwidth [5,8,93].Therefore, to convey a broader range of musical elements, research is being conducted on whether the pitch or emotion of music can be distinguished through haptic frequency.For example, to convey melody to DHH people, Karam et al. [36,37] developed an Emoti-Chair that converts audio signals into a tactile channel using a "model human cochlea."In addition, several studies have examined whether DHH people can distinguish the melody and emotion of music with the vibration of the musical elements.However, the perception and differentiation of frequencies are personal, and some require training to make a clear distinction of music [46,80].
Graphic-based visualizers that convert musical elements into visuals have been proposed primarily for music education through aural training and music theory.For music training and theory learning, DHH people have been given simplified musical feedback by converting musical elements into visuals or by designing and presenting these elements as music scores.Zhou et al. [101] developed a mobile game with auditory training that maps pitch to the height and color of visual artifacts for children with cochlear implants, and Ling [49] proposed an easy-to-read visual music sheet using color and shape design to help deaf children better comprehend a chord and an interval.Although there are studies that have supported DHH people in music appreciation through visualization, there has yet to be an accepted music-visual mapping convention.Fourney and Fels [17] suggested 18 visual types using shape created and reported the difficulties and limitations involved in music enjoyment among DHH people.As the mapping of music to visual elements primarily aims to convey the informational aspects of music to obtain detailed feedback and intuitive visual notation, its application to the appreciation of music by DHH people is limited [25].Pouris et al. [74] reported that existing graphic-based visualizers are only practical for music experts who can analyze music visually, indicating that it is ineffective for audiences, such as DHH people, who do not understand music theory or cannot experience music professionally.

Exploration and customization of a music-sensory substitution system
Existing research has explored the use of visual and tactile senses to facilitate and enrich experience in musical activities, such as appreciating, playing instruments, and composing both for hearing individuals and for DHH people (Table 1).For hearing professionals, visual and tactile elements have been mainly used to enhance the understanding and analysis of music [4,9,13,21,24,29,47,54,79,85].Drawing and paintings have been used to enhance the accessibility of music, as well as its easy creation without musical experts, by non-professionals, especially children [12,96,97].Most of these prior studies about substitution system for hearing people focused on identifying mapping relationships between music (e.g., pitch, loudness, rhythm), visual (e.g., color, size, and shape) [9,24,29], and tactile (e.g., frequency) [4] elements to enable hearing people to understand and represent music intuitively with the sound [12,13,15,39,55,60,68,87].
For DHH people, challenges in accessing sound can potentially affect their ability to perceive the clarity of these mapping rules in musical experiences [81].Graphic visualizations for hearing people were primarily aimed at understanding and modeling the structure of music, so applying these to the music appreciation experience of DHH people may be challenging [17,29].Consequently, making DHH people to adapt to the relationships between the senses is necessary [3,16].As shown in Table 1, we surveyed previous works to see if their substitution system includes identifying the mapping relationships between music and assistive modalities (exploration) and designing substituted sensory according to personal taste and preferences (customization).More specifically, the systems were categorized according to whether their user was actively or passively involved in the exploration process.We found that most studies do not support the exploration phase as a tool or limitedly support it in a passive way, for example, by allowing the user to be informed about the mapping rules without voluntary participation.On the other hand, to help users with hearing impairments experience playing a musical instrument, the MuSS-Bits [73] provides visual and tactile feedback to actively engage users to explore different sounds, including musical instruments (e.g., drum) and external sounds (e.g., hand clapping).In addition to sound exploration strategies, the authors noted that personalization of tactile vibration is necessary for DHH users to be comfortable with the accepted music.However, it is difficult to see the cross-modal relationship of how exploring the mapping relationships between sensors affects the music appreciation experience.Although prior research suggests that customization phase can enhance the music-making experience [72,73], current music sensory substitution systems for appreciation are deficient in personalizing musical elements.
Existing studies lack that simultaneously supported an active exploration phase and customization to grasp the mapping relationships between alternative senses.By designing a system that supports active and easy navigation and customization, we aim to further enhance the music appreciation experience of DHH people by replacing the traditional visual and tactile approach with music.

UNDERSTANDING THE MUSICAL EXPERIENCE OF DHH PEOPLE 3.1 Method
To identify the needs, challenges, and strategies required to design a music-sensory substitution system, we conducted focus group interviews with a total of 24 DHH people, consisting of a group with three participants.To select the people for the interview, we recruited for specific conditions through social welfare facilities, and these conditions include hearing limitation, an interest in music, and experience with visual or tactile-based music-sensory substitution system; however, it was not necessary for participants to have musical training or a high level of musical understanding.The participant demographic is shown in Table A1 in the Appendix (5 males and 19 females; mean age = 35.79years [SD = 11.62]; and with hearing impairments [13 deaf and 11 hard of hearing]).All DHH participants had previous experience with music-sensory substitution systems, such as visualizers or tactile devices.Semistructured interviews were conducted mainly on the potentials and challenges of music-sensory substitution systems.The three participants within one group answered questions one-on-one through a sign language interpreter, and the interview lasted an average of 70 min.Each participant was compensated with $15.This study was approved by Gwangju Institute of Science and Technology's Institutional Review Board (IRB).
Video and audio recordings of all interviews were created with the participants' consent.Two researchers transcribed and coded all the recordings independently using thematic analysis [6].They developed a comprehensive codebook and reviewed the coding results with the co-authors to gain further insight into the music-sensory substitution experiences of DHH people.

Findings
3.2.1 Cross-modal mapping as a solution to making music more accessible to DHH people.Means of accessing music content.Participants answered that they typically consume music visually by watching subtitles, sign language, and dance videos.Half of the participants, however, complained that they could not enjoy many songs because of the lack of subtitles or additional visual contents (e.g., songs without lyrics or music videos).Interviewee I7 explained, "I watch many sign language music videos because it's hard to read the lyrics, but unfortunately there are only a few songs providing such videos".Similarly, I2 mentioned, "I wanted to know what song my hearing friend liked, so I looked it up, but it was hard to find it in the music video or lyrics".All interviewees agreed that they wanted to feel and enjoy music naturally, anytime, and anywhere.Additionally, I8 explained, "I think there should be a lot of options when it comes to music appreciation.Hearing people freely choose and experience the music they want, but DHH people cannot; DHH people give up music because they are only allowed to access music with subtitles or music videos and must try harder to find out what music they like".Thus, the accessibility of music for DHH people must be improved.More than two-third of the interviewees highlighted that music-sensory substitution systems, which automatically translate music into visuals and tactile sensations, can help address the difficulties with music accessibility.For instance, I3 and I5 mentioned, "The existing music-sensory substitution system automatically delivers any music into visual and tactile sensations, so I think I can get the songs I want freely".
Means of representing musical elements.Sixteen of the participants complained that songs described through mediums, such as lyrics and music videos, do not sufficiently express musical elements.They added that they wanted to receive information about the tempo or rhythm of the music.According to I14, "Receiving music through lyrics is more like reading a book than appreciating music.That's why I enjoy music more often with the rhythm delivered to the speakers".Additionally, I24 said that he sometimes looks for videos that graphically present musical elements to experience music.Similarly, I17 mentioned, "I prefer to feel the excitement of the inner rhythm conveyed by music, so the vibration from the speakers in the karaoke room feels like music to me than the lyrics".He said he would "like to create a haptic device to feel the vibration of the music beat of any music, anytime, anywhere".These alternative sensory systems can describe musical elements, such as melody and rhythm, more effectively than music videos or lyrics, and haptic feedback is valuable as a complementary means to expressing the musical rhythm of other visual elements.Meaningless information due to lack of cross-modal perception.When participants are not previously exposed to or experienced with the cross-modal mapping of the music information being delivered via the sensory-substitution feedback, they are more likely to receive the generated content (e.g., visual or tactile feedback) as meaningless stimuli, resulting in boredom and fatigue.More than half of DHH interviewees said that the graphic-based visualizer was boring or too complicated, reporting no feeling or any sense of music.For instance, I11 said, "I think the shapes represent music, but I don't know what they mean specifically, I don't think I would look at it more than once because it is boring".They often viewed the graphics as cryptic animations, stemming from their unfamiliarity with the relationship between music and animation.Additionally, I10 and I14 mentioned that "I think it is more like looking at a picture than a visualization [graphic-based visualization] expressing a sound, and it is no fun" (I10) and "If one has no concept of music at all, the visual feedback is only a boring stimulus" (I14).Ten interviewees noted that training in cross-modal mapping would be helpful and necessary before consuming the musical content: "If something suggests visual and tactile symbols that deaf people understand like sign language, it will help us enjoy music" (I24).Thus, DHH people first need to learn the stimuli linked to their senses and familiarize themselves with useful tools that will help them in learning and exploring the cross-modal sensations.Individual preference for visual and tactile senses.Even if identical content was presented, each individual perceived it differently according to their personal experience of similar types of stimuli.For example, half of the participants felt that the music visualization was too complex, whereas the other half said it was too simple.Particularly, the individual differences on the perception of haptic intensity was large.Individual preferences were shared by the participants in response to visuals and vibrations, and sensory preference played important roles in stimulating the sensations (rhythmic and others) and emotions expressed through music.Some interviewees stated: "It is better to use the visual elements of intense design that convey a significant change in music" (I15) and "I cannot feel the vibration very well, and I feel like my sense of vibration is getting dull over time, so I want to design very strong tactile patterns" (I18).As each individual wanted to experience musical and sensory elements differently, the stimuli must be designed such that they match the sensitivities and preferences of each individual.

Design goal: conceptualization of music-sensory substitution
Based on the interviews, we identified music-sensory substitution systems as an important means of increasing the access of DHH people to music experiences, but there are issues around cross-modal sensory perception and personalization that need to be addressed.To achieve the application of a music-sensory substitution system to the music experience of DHH people, first, a process of adequate awareness and familiarity on the relationships between music-related sensory needs to be implemented.Second, there should be a process of identifying sensory preferences and personalizing them accordingly.In this study, we defined the internalization of musical information through these two processes as conceptualization.Thus, we derived design goals to realize the conceptualization of music-sensory substitution based on the comments of the interviewees and the findings of related studies.
• Active sensory exploration for perceiving cross-modal mapping through playing instrument.When experiencing music through alternative senses, the information may be meaningless unless the relationship between musical elements and the mapped senses is fully perceived.Becoming familiar with cross-modal relationships requires a natural and active exploration process.Exploration is the act of being curious and discovering new information about the environment [2].Active exploration is important for internalizing knowledge and behaviors compared to passive exploration [75].Particularly, musical knowledge and skills can be mainly acquired by playing instruments, which can also be used to discover personal interpretations and expressions through explorations [72,73].Most of the DHH participants suggested that the basic elements of music must be understood by playing an instrument rather than simply following an explanation of the mapping rules.They believed that they could enhance their grasp of visual and tactile representation of musical elements by exploring the mapping rules as they create their expressions by playing instruments.The exploration process is aimed at developing the ability to appreciate music through alternative senses; thus, the relationship between visual and tactile elements mapped to musical elements by playing instruments, such as piano and drums, can be understood via active exploration to conceptualize the perceptual relationship between audio and alternative senses.• Customization of cross-modal design for individualization.Preferences for alternative senses in music vary between individuals, and this personalization may be an important factor in the music experience of DHH people.For example, while sensitive users may experience discomfort, desensitized users may not notice a difference and have a negative music experience, indicating that cross-modal senses should be personalized.In existing research, customization is the act of modifying to suit the needs or desires of individuals.Indeed, customization is particularly valuable in multisensory interfaces, such as visual, tactile, and auditory, owing to the inherent variability in human sensory perception and experiences [35,45,67].Our interviews revealed that customization is also necessary for music-sensory substation systems, and it may consist of two main parts: 1) mapping between music and sensory substitution and 2) designing the detailed parameters of visual and tactile stimuli.For a more natural and intuitive presentation of musical elements, a method for individually preferred sensory representation must be devised.This method will enable the design of visual elements, such as shape, color, texture, and vibration elements, such as sensitivity and intensity.

CONCEPTUALIZATION FOR MUSIC APPRECIATION
To enable the conceptualization of cross-modal mapping by DHH people and improve their music appreciation experience, we proposed a conceptualization framework.Based on the aforementioned design goals, this framework comprises of two parts: 1) cross-modal conceptualization, including exploration and customization, and 2) appreciation.A detailed description of the system implementation is given in Appendix A.2.

Cross-modal conceptualization (Figure 1-1)
Exploration (Figure 1a) is designed to help users conceptualize cross-modal mapping rules by observing and representing them while playing he instrument.As shown in Table 1, existing research has been conducted on the intuitive mappings between music and visual and tactile elements.In previous research, visual color, position, and size were mapped to musical elements, whereas tactile was primarily mapped to intensity because frequency requires a learning period.Additionally, our system also follows the intuitive and congruent mapping rules of these studies, using the piano to conceptualize melodic elements (pitch, rhythm, and volume) and the drum to conceptualize rhythmic elements (rhythm, tempo, and beat).Moreover, the system relies on the concept of super-additive processing, a phenomenon in which multimodal senses acting simultaneously produce an effect greater than the sum of their parts, to deliver music to three senses at once: sound, visual and tactile [66,88,94,95].Our design phase triggered the expectation that multimodality would be more effective than unimodality, as studies have demonstrated that multimodality facilitates the processing of different stimuli, increases detection accuracy, and reduces reaction time to feedback [70].In this study, visual and tactile sensations were conveyed simultaneously as a user explores cross-modal mapping through drumming and piano playing.When a user clicks the drum image, a dot appears in the middle of the screen, and the haptic device vibrates according to the cross-modal mapping rules.
For the piano input, the user can tap a total of 36 piano key buttons (12 scales × 3 octaves), and dots sequentially appear at positions with colors corresponding to the pitch.To support the exploration, the task of playing freely or playing while looking at the sheet of a specific music was included in this part.Free playing allows the user to express visual and tactile senses using an instrument, spontaneously creating a personal interpretation of the musical elements beyond the learned cross-modal mapping rule or basic music theory.With a visual-based music sheet, the user has to play the piano or drum at the right moment just as in a rhythm game, such as Perfect Piano1 .A transparent gray progress bar over the dots moves from left to right for a total of 15 s, indicating when to play.Under these conditions, users must play the piano or drum correctly to match the pitch and rhythm of the given music (i.e., the visual stimuli) with the timing of the progress bar.Customization (Figure 1b) is a process of changing the stimuli on the exploration display through cross-modal mapping based on the preference of DHH individuals.In our system, users can create and express the sensory representation-an expression in which musical elements are replaced with visual and tactile senses-to match their preferences.The visual elements that design the melody and beat can be adjusted for size, color, texture, shape, and spacing between pitches.They can also change the sensitivity and intensity with respect to the tactile feedback.Sensitivity corresponds to the least sound that can be ignored when setting the threshold for haptic feedback.If the sensitivity is set to the highest level, tactile feedback occurs even for very small sounds.In contrast, if the sensitivity is set to the lowest level, haptic feedback appears for only loud sounds.Intensity determines how strongly the tool generates the vibrations.Customization enables the users to explore cross-modal mapping rules while selecting and designing multimodal sensory options.Additionally, it provides a process for users to explore and design sensations that work for them, making the experience of alternative senses more intuitive and comfortable.

Music appreciation (Figure 1-2)
After conceptualization through exploration and customization, music appreciation allows users to appreciate music with the corresponding multimodal stimuli.The musical elements, such as timbre, pitch, and volume, of the songs analyzed are converted into visual and tactile sensations based on cross-modal mapping and user customization.Users can import their desired mp3 files of any music they want to experience and appreciate the music based on three cross-modal stimuli: sound, visual, and tactile stimuli.Tactile stimuli were rendered using Tactosy for Arms, from bHaptics 2 , by receiving a beat and rhythm signal from our system.This commercial product offers an application programming interface (API) for customizing the intensity and sensitivity of the tactile signal, indicating its suitability for implementing our framework.The appreciation interface can automatically translate any music into visual and tactile sensations to help DHH users identify and feel music, dance videos, and other music-related content.

USER STUDY 5.1 Participants
For this study, we recruited 28 DHH with hearing impairments ranging from hearing loss to deafness, with previous experience music activities, and who had not participated in the previous interviews.There were no restrictions on hearing status, gender, or age, but we excluded people who had no experience or familiarity with computers.Thereafter, they were divided into an experimental group and a control group (i.e., with and without the conceptualization phase, respectively) considering their age, identity, auditory ability, and previous music experience through a preliminary questionnaire (Table 2).The average ages of the participants in the control and experimental groups were 33 (SD = 7.75) and 35.8 years (SD = 8.05), respectively.Both groups included an equal proportion of six deaf people with severe to profound hearing loss (85+ dB), who could recognize nearby loud sounds, and eight hard-of-hearing people who did not have difficulty hearing common sounds after wearing a hearing aid but found it difficult to enjoy music due to sound distortion.

Materials and measurements
We conducted a counter-balanced, between-subject evaluation to verify if the iterative usage of our system creates an adequate conceptualization of cross-modal mapping and enhances the music appreciation experience.Conceptualization involves the exploration of the mapping relations between visual and tactile elements by playing a virtual instrument, followed by a customization phase in which detailed visual and tactile elements are designed.The users appreciated the music through exploration and personalized sensory substitutions.For music appreciation, we selected 20 music samples of each genre (Pop, R&B, Rock, and Classic) from Jamendo Music, a free music-sharing website, and then selected four music samples that the participants had not experienced through presurvey [32].
To confirm the satisfaction of the participants with the visual and tactile feedback-based music appreciation, a questionnaire that measures the participants' subjective experience of music appreciation was designed.Based on the questions from a previous study, we designed the music experience questionnaire (MEQ) to enable sign language interpretation for participants who have difficulty reading [59].As shown in Table 3, our subjective MEQ includes nine questions in three categories: awareness of changes in musical elements (three items), satisfaction (four items), and emotion (two items) with the music appreciation experience, measured on a 7-point Likert scale.Herein, emotions were not measured on a per-music sample basis, but rather, attention was paid to the overall sensory experience.For example, participants were asked to rate whether they felt more positive (valence) or more excited (arousal) about the overall experience of the alternative sensory music.
The post-interview consisted of open-ended questions divided into the conceptualization phase and the appreciation phase.In the conceptualization phase, questions were asked about whether the exploration and customization processes supported their perception and representation of cross-modal mappings (e.g., "How did you perceive the cross-modal mapping relationships?", "How did you design the sensations?").For the appreciation phase, the questions focused on the reasons for positive/negative feelings about the appreciation experience (e.g., "Why was the music appreciation positive?","How did the conceptualization phase influence your feelings?").

Procedures
Participants in both groups engaged in five sessions on different days [76].Both groups used the same system, but their sessions were structured differently depending on whether they performed the conceptualization phase.Both groups appreciated music through multimodal senses in the first session for baseline measurement.The type of task differed across groups in the second, third, and fourth sessions, and the tasks focused on usage related to crossmodal conceptualization for music appreciation.The experimental group appreciated music after 20 min of exploration and customization, whereas the control group appreciated music directly through the alternative senses without cross-modal conceptualization.In the fifth session, both groups were requested to repeat the same task from the first session.As DHH people may listen to music with residual hearing during actual music experiences, sound was included for all conditions.The detailed process for each session in the experimental group was described as follows, with the control group repeating the first session of the experimental group five times: 1 How did you detect the change in pitch in the stimuli?
2 How did you detect the change in loudness in the stimuli?
3 How did you detect the change in rhythm in the stimuli?

Satisfaction
Evaluate whether the experience felt like music or noise and whether you enjoyed the experience.
4 How much did the stimuli feel like music?
5 How much did you enjoy the stimuli?
Evaluate whether musical sensations, such as energy and kinetics, were conveyed through the experience.
6 How much internal energy did you feel from the stimuli?Evaluate what emotion (arousal) the experience conveyed.9 What would you rate your music appreciation experience as excitement?
The evaluation metric used a 7-point Likert scale ranging from 1="Not at all" to 7="Very much".
• Session 1: A baseline music appreciation experience was measured to determine the mediating effect of our system on the music experience of the participants from both groups.First, the relationship between music, visual, and tactile feedback was explained, and participants appreciated the music with multimodality combinations for the four songs and answered the MEQ.• Session 2: After the exploration phase, in which the participants explored the meaning and relations between the sensations by playing a virtual instrument (20 min), they appreciated the music, as in the first session, and answered the MEQ.• Sessions 3 and 4: The same procedure as the second session was followed in sessions 3 and 4, but these sessions included the customization of visual (e.g., color and shape) and tactile (e.g., sensitivity and intensity) elements to allow participants to customize their sensory representation (this customization activity is included in the 20-min exploration).
• Session 5: To analyze the effect of conceptualization on the music experience of participants over five sessions, the same experiment as the first session was repeated.In the last session, however, the customization of sensory substitution was included before the evaluation.
Before the experiment, participants signed a consent form and completed a demographic questionnaire.In all sessions, we explained the purpose of the experiment and conducted it while seated in a quiet room with a sign language interpreter.During the experiment, the sign language interpreter was asked to interpret everything, including the conversations of users among themselves.Musical stimuli were ordered by counterbalance using a random Latin Square.After each 30s music clip ended, participants were given a cool-down period of 40s in line with the music clip time.After the main experiment, participants participated individually in a post-interview lasting approximately 10 min.It was a semistructured interview, divided into the phases of conceptualization and appreciation phases, as mentioned above, and was conducted one-on-one with the help of an interpreter who had helped carry out the experiment.All interviews were recorded with the consent of the participants.Each participant was compensated with $20 per session, and this study was approved by Gwangju Institute of Science and Technology's Institutional Review Board (IRB).

Quantitative analysis.
Of the 28 participants across five sessions of the user study, we collected 560 responses from the MEQ ([14 experimental group participants + 14 control group participants] × 5 sessions × 4 songs).Satisfaction in MEQ was measured by four questions (Table 3, items Q4-7), and we performed scale reliability tests.Cronbach's alpha showed strong reliability in the music satisfaction category ( = 0.96), indicating consistent answers.The goal of the analysis was to confirm whether our conceptualization framework of cross-modal mapping enhances the music appreciation experience.
As the level of appreciation was subjectively assessed using a 7-point Likert scale, each individual's residual hearing and musical comprehension may have produced different results.Based on the Shapiro-Wilk test, it was found that all MEQ scores, except for satisfaction in session 3 in the experimental group (p=0.106),did not follow a normal distribution (p<0.05).This suggests that each participant had a different level of appreciation for the music, rather than a similar level of appreciation.To balance this individual difference, the ratings for each participant from the first session were used as a baseline to calculate a score difference, facilitating the analysis of subsequent session rating changes.As the baseline was established at zero, the scores in the first session were uniformly zero, as shown in Figure 2.
To evaluate the efficiency of our conceptualization framework in enriching music appreciation, we implemented two analytical approaches.First, to investigate whether the observed disparities in the MEQ scores were attributable to our framework, we analyzed the score differences between the experimental and control groups during the fifth session.As every score difference during the fifth session did not conform to a normal distribution, as verified through Shapiro-Wilk test (p<0.05for every score in the fifth session), , we applied the Mann-Whitney U test for the analysis.Second, similar to the within-group analysis, we compared the MEQ score differences between the first and fifth sessions for each of the two groups.These differences also exhibited a non-normal distribution; hence, we utilized a non-parametric method, the Wilcoxon signed-rank test.The results are comprehensively detailed in Table 4.

Qualitative analysis.
All audio components were transcribed, and four researchers conducted an initial review of the experimental session notes using  thematic analysis [6].The researchers extracted keywords from participants' responses in each domain, focusing primarily on how the conceptualizations of cross-modal mapping of the DHH participants was related to music appreciation experiences.They used an inductive process to organize sub-themes and specifics.Higher level categories were created by grouping common or similar terms.The two researchers developed a comprehensive codebook to resolve areas of disagreement, and reviewed the results of three coding discussions with a co-author to gain further insight into the findings that enhanced the music appreciation experience.

RESULT 6.1 Quantitative results of the music appreciation experience
Figure 2 shows the results of the awareness of changes in musical elements and the improvement in satisfaction and emotion during the five sessions under the multimodality condition.The trend in the score differences between sessions differed for the experimental and control groups: data from the experimental group exhibited a positive trajectory, whereas data from the control group displayed a slight decrease.Using the Mann-Whitney U test to compare the data from experimental and control groups, we found statistically significant differences in all categories during the fifth session (Table 4).
In particular, pitch (p=0.0001),loudness (p=0.0009), and satisfaction (p=0.0001) were highly statistically significant.This suggests that our approach, aimed at promoting conceptualization through exploration and customization, was successful in improving music appreciation across repeated sessions.Music appreciation of the experimental group gradually improved over five sessions.However, participants' subjective overall music experience of exploration and customization activities was rated lower in the second and third sessions than in the first session without any activities.Participants required considerable effort to comprehend the cross-modal mapping relations in a short time and internalize them into a complete musical experience.In the initial sessions, participants indicated a disconnect with the concept and design process of cross-modal mapping.This may have resulted in a slightly diminished musical experience.A more detailed explanation is provided in Section 6.2.
Thus, the experience in the fifth session improved compared to the first session.For all categories of music appreciation, the comparison between the first and fifth sessions demonstrated statistical significance with the Wilcoxon signed-rank test (Table 4).Especially, pitch (p=0.00063) and loudness (p=0.00059)showed a positive and highly statistically significant development in the fifth session.In contrast, for the control group, pitch (p=0.015),valence (p=0.027),arousal (p=0.012), and satisfaction (p=0.046)all showed a statistically significant decrease from the first session to the fifth session, as determined by a Wilcoxon signed-rank test (Table 4).Despite continued exposure, the participants did not appear to internalize the musical elements that were altered by the cross-modal mapping.Wilcoxon signed-rank test results revealed that the awareness of loudness and rhythm did not show statistical significance in the fifth session compared to the first session; even pitch exhibited a statistically significant decrease.

Qualitative results of the music appreciation experience
6.2.1 Ongoing conceptualization process: increasing the interest and adaption of the DHH people to cross-modal mapping.
First session: difficulty in the perception of cross-modal mapping.Participants in the experimental group reported that the music replaced by cross-modal mapping failed to elicit the perception of music.Instead, they experienced negative responses to the visualizations and tactile, which were described as "signal processing or an alarm (E2)", "machine noise like a beep (E14)", and "feeling like a massage (E4)".Five participants (E5, E8, E11, E13, and E14) expressed that the conversion of music information into cross-modal mapping was distracting: "feeling uncertain about where to concentrate my attention (E5)", and "my senses felt unfocused (E11)".Eleven participants expressed negative sentiment towards the cross-modal mapping, stating that it did not feel like music.However, three participants (E7, E10, E12) expressed positive sentiments due to their interest in the new information being presented: "I'm not sure what kind of music it is, but it is interesting to see the sound visually (E7)".It is noteworthy that, despite the positive feedback received, not all participants perceived the cross-modal mapping as music in the first session.Although the novel technique garnered interest, it did not unequivocally augment their music appreciation experience.
Second and third sessions: struggling with grasping the concepts of cross-modal mapping.The second and third sessions mark the initiation of experiments and the onset of substantial conceptualization.As mentioned in Section 6.1, the music appreciation experience of the experimental group did not exhibit any immediate improvement and instead showed signs of deterioration.This may be attributed to the design of the mapping, which is not easily perceivable, and the inherent challenge of adapting to a new system.Ten participants (E1-E4, E6, E8-E10, E12, and E14) reported that they struggled with the process.For instance, E1 remarked, "It is challenging to use.It appears that I need to acclimatize to discerning the high and low notes;" E4 echoed, "I think it is hard to recognize elements of music with tactile".Two participants (E3 and E6) reported feeling like they were studying, rather than identifying the crossmodal mapping to enable them to appreciate music.Furthermore, five participants (E1, E3, E9, E12, E14) expressed dissatisfaction with the process: "I feel sleepy and bored (E9 and E12)", "I'm not interested yet (E1 and E14)", and "It seems vague (E3)".
Fourth and fifth sessions: improved perception and adaptation of the cross-modal mapping.By the fourth and fifth sessions, cross-modal mapping began to be perceived as an integrated experience through conceptualization.All fourteen participants reported that the cross-modal mapping facilitated their ability to effectively identify musical elements.This finding is in contrasts with the subjective assessment of distraction that participants reported during the first session of the study.This may be attributed to the customization process, which possibly assisted individuals in recognizing musical elements more intuitively."It was good to be able to distinguish the beats after customization (E3)" and "I was able to reflect the feeling, mood, and texture of the notes (E7)".All participants also reported that they felt accustomed to cross-modal mapping, which uses both visualization and tactile feedback.Adapting themselves with music as an alternative sensation enabled them to perceive cross-modal mappings as a unified musical experience rather than only as information: "At first, despite knowing in my head that the visual cues and tactile vibrations were connected to the music, I was bored because the different songs all felt the same to me.But once I adjusted the visual cues and tactile vibrations to be more in tune with each other, it turned into a unified experience where I could recognize the vibrations as rhythmic beats (E12)", "I thought it would be difficult to feel the music by simply visualizing the pitches, but now I know the concept of how music was expressed.By designing the senses to suit my preference, I was able to get used to it and enjoy it more (E6)".
Control group: struggles with adapting to cross-modal mapping.In the first session, participants of the control group were curious and satisfied with the appreciation experience of expressing music with alternative sensory stimuli.However, following the same process in the subsequent sessions to the fifth was regarded as a boring, meaningless content: "I don't know why I must keep going this way (C2)"; "I think it is just meaningless visual movement and vibration (C9)." The control group study confirmed that simple repetitive exposure to cross-modal mapping is not a substitute for music appreciation.These findings indicate that the exploration and customization phases of the conceptualization framework have a positive impact on the music appreciation experience.In conclusion, conceptualization is a prerequisite for DHH people to improve their music experience by exploiting the alternative sensory system.

Creative investment in customization: enriching experience by actively reflecting their musical taste .
How DHH people customized the cross-modal mapping to express themselves musically.Participants customized the visual and tactile feedback based on their preferences, and these processes assisted them in reflecting their own musical taste with creativity.For instance, E9 expressed a preference for an uplifting mood and thus increased the intensity of the tactile feedback, and E5 opted for the shape of a firecracker (decagram) for the rhythmic note to get a better sense of the drum.With respect to the customization of color, the selections of the participants changed depending on the purpose of the music appreciation.If a user desired accurate information, the rainbow color set would be a good option for facilitating pitch distinction: "I chose rainbow colors because they're noticeable and easy to distinguish (E7)".If a user wanted to reflect their taste, then the monotonous color set with a single hue may be the best option: "I chose a color set consisting of one color, which may not be easy to distinguish, but I am satisfied because I like this color very much (E1 and E10)".All participants reported positively receiving the customized experience, collectively regarding it as an essential aspect of their music appreciation.
What inspired DHH people to get involved in music: the tailored nature of the customization process.After realizing that the output would change in response to their choices, the participants felt they could participate and contribute more to the music.Ten participants (E1-E5, E8-E10, E12, and E14) indicated that they enjoyed the selection process.For example, E9, who had a great deal of interest in music but did not show any interest or willingness to participate in the first two sessions (the sessions without customization), became more enthusiastic after realizing that she could change the detailed representations on her own.Similarly, E3, who mentioned that her music experience had been very flat in the first two sessions, also became more involved in enjoying music cross-modally.Overall, we found that customization enhances the motivation and attitude of DHH users toward music to some extent by allowing them to experience music in a novel way and participate in music-related activities.In conclusion, our framework did not only improve the participants' awareness of the cross-modal mapping relation but also improved their satisfaction in music appreciation with the tailoring process.

Insights and feedback from the DHH people on our implementation.
Avoiding challenges during early conceptualization: intuitive design for perceiving musical elements.As noted in Section 6.2.1, ten participants experienced challenges during the second and third sessions.The commonality in their feedback indicated difficulty in comprehending the foundational concepts of music.E5 suggested that a comprehensive introduction might mitigate the initial challenges: "A detailed explanation seems necessary before beginning the exploration."Moreover, participants expressed a desire for a more textual and intuitive method to differentiate pitches.E10 suggested the inclusion of note identifiers in the visualization elements to facilitate easier pitch distinction, whereas E1 expressed a preference for written scales to aid in differentiation.Although these challenges became more gradual as the sessions progressed, designing an intuitive interface in the initial stages appears crucial in mitigating participant frustration and bolstering their motivation.
Customization: a call for diverse expressions.Our implementation was initially aimed at portraying musical elements.However, as the participants adapted to the cross-modal mapping, they desired to explore broader expressive elements.They highlighted the need for more aesthetic and mood-congruent representations in separate interviews.E4 suggested enhancing the emotional impact by including aesthetic elements, rather than limiting representations to geometric shapes.Similarly, E9 envisioned a more dynamic experience with the addition of colorful, club-like lighting elements.Meanwhile, E11 proposed mood-reflective color themes, such as yellow for upbeat tracks and blue for more somber tunes.E13 and E7 independently expressed a wish for additional contextual elements, such as lyrics and the visual representation of the performing artist, respectively, to facilitate a deeper connection with the music.These insights suggest a community preference for tools that not only implement the technical aspects of representing musical elements as alternative sensations but also embrace the rich emotional nuances of music.

DISCUSSION AND FUTURE WORK
Music appreciation systems through alternative senses for DHH people have provided mapping relationships between music and visual [17,61,71,74] or tactile elements [36-38, 53, 61, 69, 71, 83].However, previous studies rarely applied exploration and customization of mapping relationships, which we have identified as a significant gap.In this study, we showed that these two phases are critical for allowing the interaction of DHH people with music in a way that is meaningful to them.Active exploration enabled their engagement with music dynamically, discovering and learning at their own pace, while customization allows for the adaptation of the system to fit individual preferences and the desire to create their own set for music appreciation.In this section, we present these two key actionable takeaways, with a focus on (1) motivating DHH people to participate in the exploration phase actively, and (2) empowering musical experience quality through personalization and aesthetic tuning.We also discuss the scalability for future applications in education and composition beyond music appreciation.

Motivating the DHH people into active participation by solving the remaining issues during exploration
The participants in the experimental group engaged in exploration and developed a conceptual image of the cross-modal mapping of music.In the early stages of their exploration, they revealed that they struggled with accepting the basic concept of music.Additionally, the observation of the control group indicated that struggles in the exploration phase could affect the entire music experience of DHH people with a sour first expression, resulting in restricted access to the system and eventually leading them to lose motivation for the music and further activities.To prevent this, the music-sensory substitution system needs to provide a sufficient and accessible exploration phase in the early stage.Moreover, we suggest that this would enhance the internal motivation of DHH people, potentially giving us three design implications in the early stages: 1) assisting them to perceive each musical element first easily, 2) reinforcing the exploration to catch the cross-modal mapping more intuitively, and 3) keeping them motivated in their exploration with various music-related gamified quests or interaction with others.Typically, DHH people require sufficient exploration chances to achieve proficiency in the fundamental principles of every musical element; this is an important prerequisite for the accurate conceptualization of cross-modal mapping.Acquainting DHH people with musical elements through visual and tactile representations can aid their perceiving of basic concepts of musical elements, which can be accomplished in several ways.For example, musical elements can be perceived by presenting various sound samples (scales and octaves) and the corresponding explicit visual examples (a ladder expressing height) [1] to make users accept the basic concept of pitch.If DHH people perceive musical elements using these approaches, it will be easier for them to conceptualize how the musical elements correspond to specific visual and tactile elements.
It is also important to design alternative sensory stimuli so that DHH people can intuitively recognize mapping rules when exploring cross-modal mappings.However, Table 1 indicates that there are no definitive rules for the alternative senses that map these musical elements.This suggests that creating an intuitive, crossmodal mapping design is a challenging task.An abstract relation in cross-modal mapping makes it difficult for individuals to intuitively identify the musical elements that are represented by the notes appearing during the playback of a track.Using visual cues, such as color and height, to represent pitches will be helpful, but some participants found it unclear during the initial state of the exploration.Different methods must be developed to explain the notes to DHH people and help them perceive the notes better.Several participants suggested using text for differentiating pitches.We believe these efforts will help DHH people adapt to cross-modal mapping more quickly.
Finally, they must be motivated throughout the exploration process.According to participants, the exploration process can feel like studying music.To prevent them from dropping out, it may be beneficial to make the exploration process enjoyable.For instance, gamification [18,56] can be effectively integrated during the exploration phase, using elements such as levels and progression [63,64], as well as achievement badges [20,23], to enhance musical motivation and engagement.Similarly, peer interaction techniques [10,22] involving collaborative learning [33,91] and group problem-solving activities [77,89] can also be employed, inspiring the process of exploring the musical elements with friends or families.By utilizing these implications, DHH people can complete the exploration process while being motivated, embracing the experience of appreciating music positively without the tediousness of the process.

Empowering musical experience quality through aesthetic tuning
Letting DHH people explore these things in the system by themselves can make them seek and desire more about the components related to the modalities, leading them to design their own mapping rules and visualizations.This will make them feel closer to music.Previous studies on the personalization needs of DHH people reported automated sound recognition tools [19,31], hand gesture-based communication devices [92], and vibration feedback [72,73], which are focused on unimodality.In this study, we proposed a framework for the exploration and customization phases of cross-modalities: sound, visuals, and tactile feedback.Participants customized the cross-modality to enjoy music in a way that felt harmonious to them; thus, they could comprehend and embrace the expression of music more subjectively, rather than solely acquire musical information.DHH people are more free from the dominance of hearing in music appreciation than hearing people, so they can project their own creative expression into music through cross-modality.Therefore, it is essential to evaluate the diverse needs of DHH people and reflect them in the customization.
In addition to accepting the musical expression, participants noted the importance of aesthetic enhancement.Future studies should consider incorporating elements, such as abstract animations synchronized with music [44] or integrating nuanced haptic patterns with game sound effects [99], to cultivate an immersive and pleasurable musical experience for DHH people.Combinations involving the sentiments of music [47,57], symbolic audio features [51], emotions related to visual features [44], metaphors about tactile features [42], art related to musical features [52], and generating dances from music with machine learning [50] should be considered to enhance the aesthetic aspect of cross-modality.Cross-modal designers and assistive technology developers will need to find more optimized cross-modal matching combinations during the design process, which will provide richer options for DHH people to fully experience music through aesthetic tuning.

Scalability of cross-modal music conceptualization framework
Although our framework was designed for the music appreciation experience, it can be extended to enjoy music for various purposes.An increasing number of studies are exploring the cross-modal use of music in teaching and composing music.Emile Jaques-Dalcroze [34] developed a pedagogy that employs the body and voice to enhance musical rhythm, melody, and pitch, and the understanding of music notation and theory, going beyond the boundaries of "sound." Zoltán Kodály's methodology [26] underlines the importance of developing an intuitive sense of music by creating one's music, often in conjunction with dance, to emphasize a holistic artistic experience.The Kodály hand sign, developed by Cuwen [100], is instrumental in ensuring precise pitch recognition when instructing choirs.The music-making and creative process [62] project enables self-directed exploration of music by collaborating and sharing outcomes and by providing personalized learning approaches based on the interests and levels of the student.Although this was not designed for DHH people, research suggests that the teaching and creation of music in a cross-modal environment can considerably enhance the perception and interpretation of music.Note that in our framework, cross-modal mapping rules are not fixed, but rather changeable.Therefore, it can be applied to the cross-modal mapping constructs used in music education and composing; particularly, we expect that customization can be used to motivate music-related activities.Given that experiences and education in diverse settings often leave a profound impression on how music is comprehended and construed [65], our framework may encourage DHH people and underpin their musical pursuits with the system's scalability in the future.

Limitations and future work
In this study, we argued the importance of the conceptualization of cross-modal music mapping for the DHH people's enjoyment of music.However, we acknowledge that some limitations exist.First, although participants were satisfied with the current system, we only used basic musical, visual, and tactile elements in the crossmodal mapping.We also mapped the visual and tactile elements to musical elements one-to-one and designed the basic exploration and customization elements.Our future work will introduce new musical elements, such as harmony [7] or timbre, and more sophisticated cross-modal mapping for DHH individuals.Combining various elements in mapping has the potential to enhance crossmodal representation.
Secondly, this study utilized only a piano to present the melody.This approach was intended to avoid confusion in cross-modal mapping, but it may have limitations in representing other musical elements, such as timbre.Thus, a new approach will be developed that features a seamless cross-modal music conceptualization method, allowing the inclusion of various instruments, such as voice and cello.
Thirdly, this study had limitations in experimenting with the entire spectrum of DHH individuals, each with distinct hearing conditions.DHH individuals with hearing losses ranging from 20 dB to 94 dB were included in our study, but this may not be representative of all types of hearing loss.To enhance the generalizability of our findings, it is necessary to study the effectiveness of the crossmodal conceptualization framework in a more diverse population of subjects with varying hearing abilities.
Lastly, the current version of our system has limitations related to its platform.Natural tool accessibility with portable devices or app-based versions would enable greater immersion in the musicsensory-substitution system, thereby allowing people to enjoy music without limitations.As we continue this research, based on the cross-modal conceptualization verified in this study, we will develop improved visual and tactile feedback devices by referring to various visualization elements with good accessibility, such as the Phazr 3 app.

CONCLUSION
In this study, cross-modal music conceptualization was proposed to improve the music appreciation experience for DHH people.We designed a framework based on feedback from existing musicsensory-substitution systems with 24 DHH participants.Although some issues remain to be addressed, we demonstrated that our proposed exploration and customization approach could help DHH people improve their music appreciation experience by conducting a between-group experiment with 28 DHH participants.The cross-modal conceptualization can positively influence the music appreciation experience of DHH people by helping them feel comfortable with music using their alternative senses.This study verifies that the simple concept of cross-modal music conceptualization can increase the music experience of many previously developed music-sensory-substitution systems.Our system further verifies the possibility of changing participants' interests and attitudes toward music and how they enjoy it. 3https://apps.apple.com/us/app/id1472113449?mt=8

A.2 Detailed Explanation of Implementing of Cross-modal Music Conceptualization
Multimodality.Each musical (melodic and rhythmic) note is visually presented as one dot in the main display panel and tactilely expressed as a vibration.Pitch is visually mapped to the vertical position and the color of dots but is not tactilely mapped.Three methods-frequency, area, and vibration pattern-have previously been employed to express pitch tactilely, but typically several weeks of training are required to be able to adeptly distinguish pitch to sufficiently enjoy a song with these methods.We decided, therefore, to focus on expressing rhythm rather than pitch through tactile stimuli.The range of octaves in our system lies between 3 and 5; thus, C3 (3-octave do) maps to the vertically lowest position, and B5 (5 octave si) is placed at the highest position.Pitch class, such as do (C3, C4, and C5), is represented by the same color-that is, three dos (C3, C4, and C5) appear as red dots.The octaves are distinguished by the height of dots.Loudness can be identified by the size of a dot or the intensity of a vibration.Our system deals with melodic instrument (piano) and rhythmic instrument (drum).As shown in Figure 2, these instruments are visually distinguished by the shape, texture, and color of the dots.The horizontal position on the screen signifies time, while the tactile stimuli do not manifest time.The tempo of the music can be recognized by the speed of the dots being displayed.The time it takes to move from the left to the right edge of the screen is equivalent to four beats when the beats per minute of the music is measured; thus, if the tempo of music is faster, then the movement from left to right on the screen will be faster.The rhythm of music is expressed through the length of notes and rests.
In CMP, only the length of the rest is expressed.The length of the rest is visually represented as the horizontal interval between dots, and tactilely as the time interval between vibrations.At this time, the shorter and more frequent the rest, the faster the rhythm.Exploration.We had users play the drums and piano to represent rhythmic and melodic instruments, respectively.We used one snare drum, and the piano was designed to play 36 notes (12 scales × 3 octaves).As the users play the instruments, they can manipulate the size of the dots or the intensity of the vibrations by adjusting the loudness of the drums and piano.These instruments can be played freely or along with a given sheet music.We wanted users to understand the basic concepts of musical elements, so we presented them with sheet music that included rising melodies, melodies with octave differences, nursery rhymes, and so forth.
Customization.The sensory representation-when musical elements are replaced with visual and tactile senses-can be customized to match users' preferences.Users' can realize their preferred visual sensory representation of dots by selecting from among five shapes, four textures, and 11 color sets for piano, and from five shapes, four textures, and a color-which is selected by the user-for drum.These representation options of shape, texture, and color set are included according to the criteria of complexity; a circle, for example, is simpler than a star.Additionally, users can adjust the vertical spacing between pitches and decide whether to use connecting lines between the dots to improve the legibility.The connecting lines between points may clarify the change in pitch more intuitively.With respect to tactile feedback, users can change the sensitivity and intensity.Sensitivity corresponds to how low a sound can be ignored when setting the threshold for haptic feedback.If the sensitivity is set to the highest level, then tactile feedback occurs even for very small sounds.In contrast, if the sensitivity is set to the lowest level, then haptic feedback appears only for loud sounds.Intensity determines how strongly the tool generates the vibrations.
We set the default option of sensory representation that appears before user's customization to not be too complicated and strong yet still enable the user to feel the change in musical elements.The default dot shape was set to 'circle' and the texture was set to 'none'.The default color set of the piano and the color of the drum were 'rainbow' and 'white,' respectively.The pitch interval was set at 70% of the maximum, and the connecting lines between the pitches were automatically displayed.The sensitivity and intensity of tactile feedback were set to 30% and 5% of the maximums, respectively.

Figure 1 :
Figure 1: Overall architecture of the music-sensory substitution system.(1) shows the system's user interface to support the conceptualization framework and (2) shows the music appreciation interface.In the actual interface, (a) and (b) are on the same page, and the piano and drums are explored on different pages.

Figure 2 :
Figure 2: Tendency of the MEQ over five sessions.

Table 1 :
Music-sensory substitution systems for DHH people

Table 2 :
Demographic information of the participants in the user study

Table 3 :
Description of the MEQ.

Table 4 :
Comparison of significance figures by session and group a The p-value of [1st vs 5th sessions] was obtained by Wilcoxon signed-rank test for the score differences of the 1st and 5th sessions of the same group (standard deviations in parentheses).b The p-value of [EXP (experimental group) vs CTR (control group)] was calculated by Mann-Whitney U test for the score differences of the 5th session of two groups (*p<0.05,**p<0.01,***p<0.001)

Table A2 :
Dimensions of cross-modal mapping of music