Human Perception of Social Robot’s Emotional States via Facial and Thermal Expressions

Abstract

Facial and thermal expressions can be used by humans to interpret emotions. While facial expressions can be a voluntary reaction, the change of temperature in the body is often not. Thus, a facial expression may not always be consistent with the emotional state that is expressed by the body temperature. This article aims to study the human perception of the emotional expression and the emotional state of a robot that simultaneously uses its face and body temperature. To this end, a robot, named TherMoody, which has the capability to change its body temperature from 10–55℃, was used. Then, 25 combinations (5x5: one facial expression and one thermal expression for anger, joy, fear, sadness, and neutral state) were evaluated by 15 participants. Thermal expressions were designed based on the metaphors of emotions related to temperature. The results show people tend to base their judgment of the robot’s emotional expression exclusively in its facial expression, regardless of the change of its body temperature. However, there are combinations where the thermal expression predominates over the facial expression when judging the robot’s emotional state. Thus, these combinations may produce the perception that the robot’s emotional expression is, in fact, genuine, simulated, masked, or neutralized.

References

1. Alexis E. Block and Katherine J. Kuchenbecker. 2018. Emotionally supporting humans through robot hugs. In Proceedings of the Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 293--294. DOI:https://doi.org/10.1145/3173386.3176905
2. John-John Cabibihan and Sushil Singh Chauhan. 2017. Physiological responses to affective tele-touch during induced emotional stimuli. IEEE Transactions on Affective Computing 8, 1 (2017), 108--118. DOI:https://doi.org/10.1109/TAFFC.2015.2509985
3. Carlos Crivelli and Alan J. Fridlund. 2019. Inside-out: From basic emotions theory to the behavioral ecology view. Journal of Nonverbal Behavior 43 (2019), 161--194. DOI:https://doi.org/10.1007/s10919-019-00294-2
4. Charles Darwin. 1965. The Expression of the Emotions in Man and Animals. The University of Chicago Press.
5. Carl DiSalvo, Francine Gemperle, Jodi Forlizzi, and Elliott Montgomery. 2003. The hug: An exploration of robotic form for intimate communication. In Proceedings of the 12th IEEE International Workshop on Robot and Human Interactive Communication. IEEE, 403--408. DOI:https://doi.org/10.1109/ROMAN.2003.1251879
6. Chris Dodge. 1997. The bed: A medium for intimate communication. In Proceedings of the CHI’97 Extended Abstracts on Human Factors in Computing Systems. ACM, 371--372. DOI:https://doi.org/10.1145/1120212.1120439
7. Paul Ekman [n.d.]. The Ekman’s Atlas of Emotions. http://atlasofemotions.org.
8. Paul Ekman. 1992. An argument for basic emotions. Cognition and Emotion 6, 3--4 (1992), 169--200. DOI:https://doi.org/10.1080/02699939208411068
9. Paul Ekman and Wallace V. Friesen. 1975. Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues. Prentice-Hall.
10. Kerstin Fischer, Malte F. Jung, Lars Christian Jensen, and Maria Vanessa aus der Wieschen. 2019. Emotion expression in HRI: When and why. In Proceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction. IEEE, 29--38. DOI:https://doi.org/10.1109/HRI.2019.8673078
11. Daniel Gooch and Leon Watts. 2010. Communicating social presence through thermal hugs. In Proceedings of the 1st Workshop on Social Interaction in Spatially Separated Environments.
12. Daniel Gooch and Leon Watts. 2012. YourGloves, hothands and hotmits: Devices to hold hands at a distance. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology. ACM, 157--166. DOI:https://doi.org/10.1145/2380116.2380138
13. Mitchell S. Green. 2007. Self-Expression. Oxford University Press.
14. James J. Gross. 1999. Emotion regulation: Past, present, future. Cognition and Emotion 13, 5 (1999), 551--573. DOI:https://doi.org/10.1080/026999399379186
15. Markus Häring, Nikolaus Bee, and Elisabeth André. 2011. Creation and evaluation of emotion expression with body movement, sound and eye color for humanoid robots. In Proceedings of the 20th IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 204--209. DOI:https://doi.org/10.1109/ROMAN.2011.6005263
16. Yuhan Hu, Zhengnan Zhao, Abheek Vimal, and Guy Hoffman. 2018. Soft skin texture modulation for social robotics. In Proceedings of the 2018 IEEE International Conference on Soft Robotics. IEEE, 182--187. DOI:https://doi.org/10.1109/ROBOSOFT.2018.8404917
17. Rachael E. Jack, Oliver G. B. Garrod, and Philippe G. Schyns. 2014. Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time. Current Biology 24, 2 (2014), 187--192. DOI:https://doi.org/10.1016/j.cub.2013.11.064
18. Malte F. Jung. 2017. Affective grounding in human-robot interaction. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 263--273. DOI:https://doi.org/10.1145/2909824.3020224
19. Alisa Kalegina, Grace Schroeder, Aidan Allchin, Keara Berlin, and Maya Cakmak. 2018. Characterizing the design space of rendered robot faces. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 96--104. DOI:https://doi.org/10.1145/3171221.3171286
20. Naz Kaya and Helen H. Epps. 2004. Color-emotion associations: Past experience and personal preference. In Proceedings of the AIC 2004 Color and Paints, Interim Meeting of the International Color Association. AIC, 31--34.
21. Eileen Kennedy-Moore and Jeanne C. Watson. 2001. Expressing Emotion: Myths, Realities, and Therapeutic Strategies. Guilford Press.
22. Paul R. Kleinginna and Anne M. Kleinginna. 1981. A categorized list of emotion definitions, with suggestions for a consensual definition. Motivation and Emotion 5 (1981), 345--379. DOI:https://doi.org/10.1007/BF00992553
23. Zoltán Kövecses. 2000. Metaphor and Emotion: Language, Culture, and Body in Human Feeling. Cambridge University Press.
24. Zoltán Kövecses. 2005. Metaphor in Culture. Universality and Variation. Cambridge University Press. DOI:https://doi.org/10.1017/CBO9780511614408
25. Rikard Küller, Byron Mikellides, and Jan Janssens. 2009. Color, arousal, and performance -- A comparison of three experiments. Color Research 8 Application 34, 2 (2009), 141--152. DOI:https://doi.org/10.1002/col.20476
26. George Lakoff and Mark Johnson. 1980. Metaphors We Live By. University of Chicago Press.
27. Hee Rin Lee and Selma Sabanović. 2014. Culturally variable preferences for robot design and use in South Korea, Turkey, and the United States. In Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 17--24. DOI:https://doi.org/10.1145/2559636.2559676
28. Wonjun Lee and Youn-Kyung Lim. 2012. Explorative research on the heat as an expression medium: Focused on interpersonal communication. Personal and Ubiquitous Computing 16, 8 (2012), 1039--1049. DOI:https://doi.org/10.1007/s00779-011-0424-y
29. Diana Löffler. 2017. Color, Metaphor and Culture - Empirical Foundations for User Interface Design. Ph.D. Dissertation. Julius-Maximilians-Universität Würzburg, Germany. https://nbn-resolving.org/urn:nbn:de:bvb:20-opus-153782.
30. Diana Löffler, Nina Schmidt, and Robert Tscharn. 2018. Multimodal expression of artificial emotion in social robots using color, motion and sound. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 334--343. DOI:https://doi.org/10.1145/3171221.3171261
31. David Matsumoto and Linda Juang. 2016. Culture and Psychology (6th Edition). Cengage Learning.
32. Peter E. McKenna, Ayan Ghosh, Ruth Aylett, Frank Broz, and Gnanathusharan Rajendran. 2018. Cultural social signal interplay with an expressive robot. In Proceedings of the 18th International Conference on Intelligent Virtual Agents. ACM, 211--218. DOI:https://doi.org/10.1145/3267851.3267905
33. François Michaud, Paolo Pirjanian, Jonathan Audet, and Dominic Létourneau. 2000. Artificial emotion and social robotics. In Distributed Autonomous Robotic Systems 4. Springer, Tokyo, 121--130. DOI:https://doi.org/10.1007/978-4-431-67919-6_12
34. Hideyuki Nakanishi, Kazuaki Tanaka, and Yuya Wada. 2014. Remote handshaking: Touch enhances video-mediated social telepresence. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2143--2152. DOI:https://doi.org/10.1145/2556288.2557169
35. Jiaqi Nie, Michelle Pak, Angie Lorena Marin, and S. Shyam Sundar. 2012. Can you hold my hand? Physical warmth in human-robot interaction. In Proceedings of the 7th ACM/IEEE International Conference on Human-Robot Interaction. ACM, 201--202. DOI:https://doi.org/10.1145/2157689.2157755
36. Lauri Nummenmaa, Enrico Glerean, Riitta Hari, and Jari K. Hietanen. 2014. Bodily maps of emotions. Proceedings of the National Academy of Sciences 111, 2 (2014), 646--651. DOI:https://doi.org/10.1073/pnas.1321664111
37. Eunil Park and Jaeryoung Lee. 2014. I am a warm robot: The effects of temperature in physical human-robot interaction. Robotica 32, 1 (2014), 133--142. DOI:https://doi.org/10.1017/S026357471300074X
38. Denis Peña and Fumihide Tanaka. 2018. Touch to feel me: Designing a robot for thermo-emotional communication. In Proceedings of the Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 207--208. DOI:https://doi.org/10.1145/3173386.3177016
39. Denis Peña and Fumihide Tanaka. 2018. Validation of the design of a robot to study the thermo-emotional expression. In Social Robotics. ICSR 2018. Lecture Notes in Computer Science, vol. 11357, S. Ge et al. (Ed.). Springer, Cham, 75--85. DOI:https://doi.org/10.1007/978-3-030-05204-1_8
40. Katri Salminen, Veikko Surakka, Jukka Raisamo, Jani Lylykangas, Roope Raisamo, Kalle Mäkelä, and Teemu Ahmaniemi. 2013. Cold or hot? How thermal stimuli are related to human emotional system? In Haptic and Audio Interaction Design. HAID 2013. Lecture Notes in Computer Science, vol. 7989, I. Oakley and S. Brewster (Eds.). Springer, Berlin, 20--29. DOI:https://doi.org/10.1007/978-3-642-41068-0_3
41. Sichao Song and Seiji Yamada. 2017. Expressing emotions through color, sound, and vibration with an appearance-constrained social robot. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 2--11. DOI:https://doi.org/10.1145/2909824.3020239
42. Marjorie Fink Vargas. 1986. Louder Than Words: An Introduction to Nonverbal Communication. Iowa State University Press.
43. Christian J. A. M. Willemse, Dirk K. J. Heylen, and Jan B. F. van Erp. 2018. Communication via warm haptic interfaces does not increase social warmth. Journal on Multimodal User Interfaces 12 (2018), 329--344. DOI:https://doi.org/10.1007/s12193-018-0276-0
44. Graham Wilson, Dobromir Dobrev, and Stephen A. Brewster. 2016. Hot under the collar: Mapping thermal feedback to dimensional models of emotion. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4838--4849. DOI:https://doi.org/10.1145/2858036.2858205
45. Steve Yohanan and Karon E. MacLean. 2011. Design and assessment of the haptic creature’s affect display. In Proceedings of the 6th International Conference on Human-Robot Interaction. ACM, 473--480. DOI:https://doi.org/10.1145/1957656.1957820

Index Terms

• Computer systems organization

  • Embedded and cyber-physical systems

    • Robotics

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • User studies

    • Interaction devices

      • Haptic devices