Developing an Emoji-based User Experience Questionnaire: UEQ-Emoji

Emojis are increasingly used in tools to evaluate user opinions, typically through happy-to-sad rating scales. We extended this concept by developing an emoji-based User Experience Questionnaire (UEQ), a well-established UX evaluation tool. Our design process involved selecting emojis corresponding to UEQ adjectives and creating bipolar emoji scales. The final tool covered 11 scales but could not represent the novelty subscale due to limitations of the emoji language. A comparative user study showed statistically similar results for 7 out of 11 scales between the emoji-based and word-based UEQs. Although the word-based version was preferred, our findings suggest that emoji-based scales can be a viable alternative for UX evaluation. Our work provides new insights into the potential and limitations of using emojis in UX research. Supplementary materials include a ready-to-use UEQ-Emoji form and an analysis tool.


INTRODUCTION
Emojis have become ubiquitous in digital communication, with over 3,521 emoji currently recognized by the Unicode Consortium.For example, the smiling-face-with-smiling-eyes emoji (Unicode: U+1F60A name: smiling face with smiling eyes) is widely used to convey happiness, friendliness, or gratitude.Emojis can express emotions in a concise and visually engaging manner [7].When included as part of text-based messaging, emojis can add tone of voice or facial expressions and form a crucial element in correctly understanding the meaning of a message [39].A considerable body of research, including Lu et al. [24], has delved into the interpretation of emojis by individuals, underscoring that emojis do not constitute a universal language.Emojis have also become increasingly popular as a response method, particularly online surveys.Popular survey tools such as Google Forms, SurveyMonkey, and Qualtrics include the option to use emojis as a response method, allowing participants to choose from a pre-selected list of emojis when answering questions.
As well as selecting individual emojis, happy-to-sad rating scales using emojis have become a ubiquitous form of opinion data collection, e.g., physical smiley feedback terminals are a common sight in shops and public service facilities and present a scale using four emoji buttons [16].Where such bipolar differential rating scales, i.e., with reference terms at both ends to measure both the positive and negative sides of the response spectrum, have been implemented using emojis, they almost exclusively also present variations on a happy-to-sad scale [3,4].Commonly used UX evaluation tools that utilize a bipolar differential rating scales format include Attrakdiff and the User Experience Questionnaire (UEQ) [23,33].In the UEQ, respondents are asked to rate their overall experience using bipolar adjective pairs, such as annoying-enjoyable or complicated-easy.Each pair forms a bipolar scale, and the respondent's rating on each scale contributes to the overall evaluation of the user experience [23,33].
It is interesting to note that, similarly to the well-discussed differences in understanding the meaning of emojis, even words, the basic units of language, are subject to varying interpretations.For instance, seemingly straightforward words like penguin can be understood differently [25].This suggests that words used in existing user experience evaluation scales, such as conventional or unpredictable, will likely be interpreted differently.Such terms may appear straightforward but are complex and nuanced, open to various interpretations.
Given the propensity of emojis to communicate feelings, we explore the development of a UX evaluation tool using emojis as the endpoints of bipolar differential rating scales.To ground the design, we based our development on the well-established UEQ tool.Through an iterative process, see Figure 3, we first established a set of emojis that are understood to represent the individual adjectives in the UEQ.We then formed scales using bipolar emoji pairs and collated these to produce an emoji version of the UEQ.We report that the novelty subscale of the UEQ is not easily represented by emojis.Based on a user study comparing word and emoji-based UEQs, we report statistically similar results in 7 of 11 rating scales.While there were no quantifiable differences between participants' performance in the two cases, the word-based tool was preferred.We contribute by: (1) Exploring emoji-based bipolar differential rating scales beyond the ubiquitous happy-to-sad scale.For example, using emojis to represent exciting-to-boring.(2) Developing a UX evaluation tool comprised of a set of emojibased bipolar differential scales to provide data on several facets of the user experience.(3) Validating the developed emoji-based UEQ through a comparative user study.(4) We provide a ready-to-use UEQ-Emoji form and analysis tool as supplementary materials to this paper.

RELATED WORK
To position our work, we provide background by referencing relevant works in UX measurement tools and emoji use in communication.We then discuss prior works that have utilized emojis as user feedback tools and highlight our contribution.

UX Measurement Tools
In the realm of UX measurement, Hassenzahl and Tractinsky [17] laid the foundational framework, emphasizing that both practical (usability) and hedonic dimensions contribute to the overall UX.UX can be measured using qualitative methods such as interviews, text-based questionnaires, and the think-aloud method when using a product or service [38].For quantitative UX research, rating scales and Likert scales are the basic tools [35].In Likert scales, respondents express their agreement with statements on a scale, e.g., ranging from 1 (totally disagree) to 7 (fully agree).Building on these basic rating scales, several widely used UX evaluation instruments have emerged, e.g., the System Usability Scale (SUS), the NASA Task Load Index (NASA-TLX), AttrakDiff, and the User Experience Questionnaire (UEQ) [11].While the SUS and NASA-TLX are tailored to focus on the usability of products and services, AttrakDiff and the UEQ are designed to capture a holistic view of the UX, encompassing both usability and hedonic dimensions of the experience [11].Rating scales may be either unipolar or bipolar.Unipolar rating scales present a continuum, such as the degree of agreement, from zero to a defined maximum.In contrast, Bipolar scales present a continuum between opposites (positive and negative), such as the degree of agreement and disagreement [18].Of the previously mentioned UX evaluation tools, only NASA-TLX utilizes a unipolar format, e.g., rating mental demand from very low to very high.The SUS uses a bipolar scale with ratings from strongly disagree to strongly agree, and both AttrakDiff and the UEQ use opposing adjectives as the scale endpoints.DeCastellarnau [10] presents a useful literature review of response scale characteristics, noting that while bipolar scales can measure neutrality, the direction, and the intensity of opinion, unipolar scales only measure the intensity.Prior work has reported that bipolar rating scales tend to have a positive bias, as respondents are reluctant to choose negative responses, while unipolar scales have a center bias [10,18].

Emojis in Communication
Emojis have emerged as part of the online messaging culture to enrich textual communication, and their use has a role in counteracting the lack of non-verbal cues in online messaging channels [40].Early works on emoticons and emojis report how smiley faces started being used in discussion forums in the 1980s to indicate that a comment is a joke [12], and people placing them to indicate laughter, thus making the discussion more closely resemble the patterns in oral, embodied communication [27].
When the basic character strings consisting of ASCII characters started giving away to the colorful pictorial images of emojis, created in Japan in the late 1990s [21], the expressive power of emojis increased.Emojis are used as a short way to convey pragmatic information through imagery [5], for aiding personal expression and reducing the ambiguity of discourse of the message [22], and as non-verbal communication cues [41].The phenomenon of emoji has raised discussion about whether it is evolving to a visual language of its own, and its resemblance to early historical visual communication systems, such as Egyptian hieroglyphs, has been pointed out [5].It has been argued that graphic emoticons will lead in the future to the emergence of a universal symbolic language [6], especially considering face-based emojis, due to the universality of facial expressions [5].While there has been much discussion on cultural differences in emoji use [24], this primarily applies to 'symbol' icons while there is relative similarity for 'smileys' and 'people' icons [14].

Emojis as User Feedback Tools
Aiming to tap into emojis' ability for emotional communication, emojis have been increasingly integrated into research tools exploring users' perceptions of products and services.The most prevalent approach has been the development of various smiley face Likert scales, typically constructed with icons of 5 faces with expressions ranging from sad to happy, e.g.[15,29], see Figure 2. Alismail and Zhang [4] studied the use of smiley face Likert scales in online surveys, reporting no significant differences between numeric and emoji versions, but suggested using emoji-based questionnaires for assessing subjective feelings.Similarly, Toepoel et al. [36] found no difference in results from radio button and smiley face based Likert scales but reported that participants preferred the smiley face version.In earlier work, Alismail and Zhang [3] developed a version of the UEQ using smiley face Likert scales instead of the usual radio buttons.However, this implementation retained words to define the rating criteria and used the same happy-to-sad set of emojis for each of the subscales [3], see Figure 2.
Another approach to using emojis as a feedback mechanism is requiring participants to select one or more emojis from a pre-defined list of emojis, e.g., with use cases in the domains of orthodontics [26] and food preferences [19,20].Typically, these methods present a list of between 30 and 50 emojis from which the survey participants click all that apply (CATA) [26,32], see Figure 2. In particular, a large body of work using this approach has focused on its use in assessing food preferences, e.g., [19,20,32].When used in a mobile web survey with millenials as participants, Bosch and Revilla [8] reported a clear preference for answering open-ended questions with CATA-style emoji responses.
One step in developing emoji-based UX feedback tools is selecting emojis universally understood to represent the emotions or characteristics being evaluated.To determine the set of emojis to use in an online feedback tool Sun et al. [34] conducted an online survey where participants were presented with 2 or 3 expertselected emojis and asked to rate how well each emoji represented a specific emotion.Similarly, Scherr et al. [30] curated a set of emojis for use in gathering feedback and noted that most participants had a very homogeneous understanding of the sentiments and emotions represented by the emojis.

Our Contribution
There has been a huge amount of prior research in the HCI domain on using emojis in conversational usage, particularly exploring differences in understanding the meaning of emojis, e.g., Togans et al. [37].However, there has been relatively little research on emoji use as part of UX evaluation tools in HCI.In the HCI domain, prior work has almost exclusively focused on using smiley face Likert scales [4,15,29,36].As the smiley face Likert scale only uses five emojis, it does not leverage the capability of the larger set of emojis Left: A typical example of the ubiquitous happy-to-sad rating scale, the Emojiscore scale [1] is also used in [3,4].Right: Emoji-based click all that apply (CATA) questionnaires [32].Alternative presentations of this approach include an emoji keyboard [8].
typically used in conversational settings to explore emotional perceptions of products and services.The use of a larger set of emojis for feedback has been little researched in HCI.Existing UX tools such as AttrakDiff and the UEQ have been systematically designed to provide assessment across the multiple aspects contributing to the overall UX, and are well-established tools in the HCI domain.It now seems timely to explore the bringing together of these threads, combining the word-based UEQ with the potential for emotional expression supported by an extended set of emojis.Our research introduces several novel approaches: (1) Employing a diverse range of emojis in feedback scales, moving beyond the commonly used 5-emoji happy-to-sad scale.
(2) Utilizing emojis to represent the endpoints of bipolar rating scales.(3) Adapting the UEQ, a systematically designed word-based UX feedback tool, to use pairs of emojis instead of adjective pairs.

MAPPING UEQ TERMS TO EMOJIS
The first step in developing an emoji-based UEQ was mapping individual UEQ terms to emojis that are understood to have a similar meaning.This was done by first creating a proposed set of wordemoji mappings that were then validated through a user study.

Defining Representative Emojis
To identify broadly understood emoji representations for the terms in the UEQ, a set of potential mappings was created, which would be validated in a user study.The primary criterion for matching a term with an emoji was the definition of the emoji provided by Emojipedia [13].The emoji meanings on Emojipedia are written by emoji experts and lexicographers and include both the originally intended emoji meaning and the current real-world use [13].Two researchers independently identified one or more emojis whose definition they considered best represented each term.A discussion phase followed, aiming to reach a consensus on the single best emoji representation.For example, for the UEQ term boring the yawning face emoji was selected as its definition includes "... to imply boredom with a person or topic" [13].Similarly, interesting was mapped to face with monocle, defined as "... pondering, considering, or questioning something, ... or encouraging a closer look at some content" [13].For terms such as leading edge this required a broader interpretive approach, e.g.Rocket emoji "... to  the moon" [13] metaphorically representing technological advancement.As well as manually searching the meaning descriptions, the researchers used OpenAI's ChatGPT-3.5 to suggest potential mappings, which were then reviewed.For some terms, no potential emoji representations were identified.
With the exception of 2 emojis (soon and prohibited), all the selected emojis were ranked in the top 100 used emojis in 2021 [9].Of the 26 rating items in the full UEQ, 15 were selected as having potential emoji representations.As each rating item consisted of 2 opposing terms this corresponded to 30 emojis.For the terms usual and impractical 2 alternative emojis were included, and for interesting 3 alternative emojis were included.Thus the proposal list contained 34 word-emoji pairs.The selected list aimed to provide full coverage of the 6 subscales in the UEQ (Attractiveness, Perspicuity, Efficiency, Dependability, Stimulation, and Novelty), including at least 2 proposals in each subscale (see Table 2).
This list was then evaluated in an online survey to identify how well the proposed mappings were understood.This process was carried out in 2 phases, 1) using only the 16 terms in the 8-item short version of the UEQ (UEQ-S) and 2) including pairs of terms selected from the full 26-item version of the UEQ (total 52 terms).

User Evaluation of Word-Emoji Mappings
The user evaluation was conducted as an online survey, using participants recruited via Prolific [2].The survey was divided into two phases, the first evaluating 16 word-emoji mappings, and the second a different set of 19 word-emoji mappings.40 participants completed each survey.The participants in each phase were different.For each word-emoji pair, participants rated how well the emoji represented the word using a 7-point rating scale.The presentation order of the items was randomized to reduce bias effects.
Participants were selected as native English speakers, located in the UK who identified themselves as regular users of chat and messaging apps.This sample aimed to provide a participant set with a common understanding of the UEQ terms.Prior works translating the UEQ between different languages, e.g.German to Spanish, have noted that a one-to-one translation of the terms is not possible [28].Participants' mean age was 37.6 years (sd = 13.5) with 74% identifying as female and 26% as male.The survey took a median time of 100 sec to complete, for which participants were compensated 0.50€.The compensation level was suggested by the recruitment platform and corresponded to an hourly rate of approximately 17€.
Results from the survey are presented in Figure 4.

CONSTRUCTING AND EVALUATING THE EMOJI-BASED UEQ
Based on the word-emoji mapping ratings, we then formed a set of bipolar rating scales.

Constructing the Emoji-based UEQ
The ratings for the individual word-emoji mappings were combined as opposite pairs, and the mean rating for each pair was calculated, see Table 2.The best-performing emoji was selected where several different emojis had been proposed for a word.Based on the pairs with mean ratings on the positive side, an emoji-based UEQ was created, including 11 rating items, see Figure 1.
Although the target was to develop a version of the UEQ that addressed all the subscales represented in the UEQ, providing a balanced view of user experience, this was not possible.The 11item emoji UEQ does not include any items addressing the novelty subscale.The developed scale includes five items contributing to pragmatic quality and two contributing to hedonic quality [33].Additionally, the four attractiveness items contribute to both pragmatic and hedonic quality.

User Study
An online survey was arranged to evaluate the emoji-based UEQ.We compared the developed 11-item emoji-UEQ and the same 11 items presented in word format in a within-subjects study design.The within-subjects design enabled participants to select their preferred design in an end questionnaire.Following the method used to develop the short UEQ (UEQ-S) [33], the question "What is your opinion of the company Amazon?" was used.This subject is likely well-known to all participants and enabled our findings to be compared with those reported for the UEQ-S design.The presentation order of word and emoji versions was randomized to reduce bias effects.As well as demographics, to identify participants' familiarity with emoji use, participants provided information on the number of emojis they typically use.

Survey Participants
Participants for the online survey were recruited using the Prolific platform [2].Participants were selected as native English speakers located in the UK who identified themselves as regular users of chat and messaging apps.The survey took a median time of 2 m 5 s to complete, for which participants were compensated €0.80.The recruitment platform suggested the compensation level corresponding to an hourly rate of approximately €21.We excluded participants who provided the same rating to all scales in the questionnaire.We collected valid responses from 47 participants, 34  identifying as women and 13 as men.The participants' ages ranged from 19 to 77 years, with a mean age of 39.04 (SD = 13.74) and a median age of 37. Participants' range of emoji usage varied, with 20 participants reporting typically using between 0 and 5 different emojis, 19 using 6 -10, and 8 between 11 -30.Most participants (approximately 89%) completed the study on a mobile device.

Results
We present the survey's overall results in Figure 5.We first conducted a Shapiro-Wilk test for all measures and found that they were not normally distributed.Thus, we performed non-parametric tests in the following.Here, we report Wilcoxon signed-rank tests and additionally report the Bayesian version of the Wilcoxon signedrank to highlight the likelihood that the emoji-UEQ results yield  3. The analysis revealed significant differences between the emoji and word rankings for the scales understandable, easy clear, and interesting.In all cases, the ratings from the word-based condition were higher than those of the emoji condition.For the remaining 7 scales, no significant difference between the conditions was noted.
After checking for normality, we performed a Wilcoxon signedrank test to compare the task completion time (TCT) answering word and emoji versions.The test revealed no significant difference in the times for words ( = 30.76, = 13.44) and emojis ( = 30.24, = 10.71);see Table 3.To complete each UEQ, 11 clicks (or touchscreen taps) were required, with the number of clicks increasing above this if participants adjusted their ratings before submitting the form.Again, we performed the Wilcoxon signed-rank test to compare the number of clicks for words and emojis.The results revealed no significant difference in the number of clicks for words ( = 12.19,  = 1.57) and emojis ( = 12.47,  = 1.93); see Table 3.To identify the effects of participants' current emoji usage, independent samples t-tests were conducted between participants who reported typically using 0-5 different emojis and participants who reported using a wider selection of Note the 1-7 rating scale has been transposed to -3 to +3 as described in Laugwitz et al. [23].
emojis.The tests revealed no significant differences in the scores for words ( = −0.68, = .50)or emojis ( = 0.50,  = .62).This suggests that higher familiarity with emojis did not significantly affect the rating of the formats.Based on participants' ratings for ease of use of the two UEQ formats, a paired t-test was conducted.The test revealed a significant difference in the scores for words ( = 6.68,  = 0.69) and emojis ( = 5.57,  = 1.47);  = 4.54,  < .001,Cohen's d = 0.96.Hence, participants considered the word-based format significantly easier to use than the emoji format, and this effect was large.
Table 3: Comparison of word-based and emoji-based ratings using Wilcoxon signed-rank and Bayesian alternative methods (See Figure 5).* We tested the normality using the Shapiro-Wilk test.

DISCUSSION
In this section, we discuss further validation of our developed tool, highlight what we consider to be its strengths and weaknesses and reflect on the limitations of our design and research process.

Validating the UEQ-Emoji
When creating new evaluation scales, it is common to assess the internal validity of items that contribute to the same scale, e.g., using Cronbach's alpha.For the text-based UEQs, this internal validity check was a key part of the tool's validation process, e.g., [33].In the case of the UEQ-Emoji, we present Cronbach's alpha values for the Attractive, Practical, and Hedonic scales in Table 4.According to standard interpretations of Cronbach's alpha, these values indicate excellent internal consistency.These measures of internal validity are consistent with those reported for the text-based UEQ-S, which had Cronbach's alpha values of 0.85 for pragmatic quality and 0.81 for hedonic quality [33].
In the original validation of the UEQ-S, the authors compared the mean values of its scales to those of the full UEQ for the same  [33].As our study utilized the same question, rating the company Amazon, it is also interesting to compare with our findings -noting that opinions may have changed in the six years between the studies.Our findings showed similar values: 1.04 for Practical quality and 0.73 for Hedonic quality.In the words of the UEQ-S creators, the UEQ-Emoji "seems to approximate" to the word-based UEQs "expectedly well" [33].
For further validation, the creators of the UEQ conducted a user study that revealed a negative correlation between the time taken to complete a test task and participants' ratings on the UEQ's perspicuity scale.Interestingly, this correlation was not observed on the hedonic side [23, p. 9].As future work, a similar study could be completed with the UEQ-Emoji.

Strengths and Weaknesses of the UEQ-Emoji
The value of emojis in communicating emotions when used in communication is well-reported [39].While the scope of our study does not explicitly deliver evidence on the benefits of emojis over word-based UX evaluation tools, this has been consistently reported by numerous prior works.For example, exploring survey responses with freely selected emojis Bosch and Revilla [8] reported that the average information conveyed was significantly higher than words.We believe these benefits will also be reflected in our use of emojis.
One possible advantage of the UEQ-Emoji is its suitability for mobile use.Traditional word-based rating scales, like AttrakDiff and the UEQ, often struggle to fit on a smartphone screen in portrait mode, especially when bipolar scales with labels at both ends are used.This issue is further complicated by the lengthy labels often found in such tools.For example, the UEQ-S has been translated into over 36 languages, and in most Latin-based languages, the labels are too long to fit without requiring horizontal scrolling to read them fully, see Figure 6.In contrast, the UEQ-Emoji can be displayed in usable form on a smartphone screen in portrait orientation, enhancing its usability for mobile users.To emphasize the importance of mobile optimization, we note that 90% of the participants in our online survey used a smartphone rather than a laptop or desktop computer.However, despite the non-optimal visual presentation, participants using smartphones still expressed a significant preference for the text-based format.In a future study, it would be interesting to collect data identifying if participants scrolled the screen sideways to read the labels or guessed the positive label without it being fully visible.
We note that our study participants preferred the word-based tool over the emoji version when rating ease of use.We consider that the relatively old age of our sample, with a median age of 37 years, may have influenced this stated preference.As a comparison, Schouteten and Meiselman [31] evaluated children's food preferences using an emoji-based feedback tool and reported on the tool's potential with this user segment.
A significant limitation of our UEQ-Emoji is its inability to capture the perceived novelty of the product or service being evaluated, a feature in the text-based UEQ.This is due to the semantic limitations of the emoji language, as no emojis are available that are universally understood to mean conventional or leading edge.We suspect these terms in the word-based UEQ, are also subject to varied interpretations.It would be interesting to explore the performance of this scale in the word-based UEQ.
Several other challenges have been previously noted with emojibased questionnaires; a useful list is provided by Alismail and Zhang [3].For instance, some participants in prior studies felt that an emoji-based questionnaire appeared unprofessional.Others speculated that the emojis might introduce a positive bias in the results, a phenomenon we also observed to some extent in our study.The need for additional cognitive effort when interpreting emojis, akin to translating a second language, was also highlighted, leading to longer completion times for emoji-based questionnaires [8].In our study, we did not observe significant differences in the time taken to complete the survey.We note that the paid online participants in our study are likely motivated to complete the survey in the least amount of time possible.
Given these mixed strengths and weaknesses, further research is essential to understand the potential and limitations of the UEQ-Emoji fully.In particular, it would be interesting to utilize the tool with different demographic groups, such as children, who may have a different interaction experience.

Limitations of our Work
Our developed UEQ-Emoji is based on the well-established UEQ.However, reflecting on the impact of our process of selecting the used emojis for our final UEQ-Emoji is valuable.In particular, its initial stage required the researchers to pre-select a set of potential emoji-word mappings.We believe that the possibility of researcher bias in this step is minimal.The process required matching the meaning of UEQ terms with the defined meanings of emojis, i.e., semantic similarity.Focusing primarily on the top 100 most used emojis, 34 were selected for validation in a user study.We note that similar processes were used in selecting emojis for emoji-based choose all that apply (CATA) tools [19,20,32].

CONCLUSION
This paper has presented the design and initial validation study of the UEQ-Emoji, an emoji-based tool for evaluating user experience.The developed tool evaluates both pragmatic and hedonic aspects of the overall user experience.Following the text-based UEQ tools, the UEQ-Emoji includes scales measuring attractiveness, perspicuity, efficiency, dependability, and stimulation.However, due to the semantic limitations of the emoji language, no measure for novelty is included.
The meaning of the emojis used in the UEQ-Emoji was validated to ensure they were understood to have the same meaning as the terms in the text-based UEQ.Based on a limited evaluation (n = 47), the final version of the UEQ-Emoji shows satisfactory internal consistency and a reasonable degree of comparability to the established text-based UEQ.In a within-subjects study with participants with a median age of 37, the traditional word-based evaluation tool was preferred over the UEQ-Emoji.Further research is needed to fully understand the potential benefits of the UEQ-Emoji in the HCI context.Specifically, we speculate that the tool may be particularly beneficial when used with younger populations, such as children and adolescents.We provide a ready-to-use UEQ-Emoji form and analysis tool as supplementary materials to this paper.

Figure 3 :
Figure 3: The iterative steps in the research process

Figure 4 :
Figure 4: Ratings for how well the pre-selected emojis represent the UEQ terms.Items marked with an asterisk from study 1.

Figure 5 :
Figure5: Final questionnaire responses "Opinion of the company Amazon" using word and emoji-based rating scales.Note the 1-7 rating scale has been transposed to -3 to +3 as described in Laugwitz et al.[23].

Figure 6 :
Figure 6: The existing text-based UEQ and the UEQ-Emoji displayed on a smartphone.With a readable font size and usable size rating selection buttons, the text-based version does not fit in portrait but requires horizontal scrolling

Table 1 :
The 11 selected emoji pairs mapped to the six subscales in the UEQ.

Table 2 :
Mean ratings and standard deviations for emoji pairs (1-7 scale) and related UEQ subscale similar results to the original version, see Table

Table 4 :
Internal consistency of scales in the UEQ-Emoji measured by Cronbach's Alpha the company Amazon.They reported a mean value of 1.17 for Practical quality (which includes Efficiency, Perspicuity, and Dependability scales) and a mean value of 0.66 for Hedonic quality (comprising Stimulation and Originality scales)