skip to main content
research-article

Implicit Emotion Communication: EEG Classification and Haptic Feedback

Authors Info & Claims
Published:20 December 2017Publication History
Skip Abstract Section

Abstract

Today, ubiquitous digital communication systems do not have an intuitive, natural way of communicating emotion, which, in turn, affects the degree to which humans can emotionally connect and interact with one another. To address this problem, a more natural, intuitive, and implicit emotion communication system was designed and created that employs asymmetry-based EEG emotion classification for detecting the emotional state of the sender and haptic feedback (in the form of tactile gestures) for displaying emotions for a receiver. Emotions are modeled in terms of valence (positive/negative emotions) and arousal (intensity of the emotion). Performance analysis shows that the proposed EEG subject-dependent emotion classification model with Free Asymmetry features allows for more flexible feature-generation schemes than other existing algorithms and attains an average accuracy of 92.5% for valence and 96.5% for arousal, outperforming previous-generation schemes in high feature space. As for the haptic feedback, a tactile gesture authoring tool and a haptic jacket were developed to design tactile gestures that can intensify emotional reactions in terms of valence and arousal. Experimental study demonstrated that subject-independent emotion transmission through tactile gestures is effective for the arousal dimension of an emotion but is less effective for valence. Consistency in subject-dependent responses for both valence and arousal suggests that personalized tactile gestures would be more effective.

References

  1. Omar AlZoubi, Rafael A. Calvo, and Ronald H. Stevens. 2009. Classification of EEG for affect recognition: An adaptive approach. In Australasian Joint Conference on Artificial Intelligence. Springer, 52--61. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Bradley M. Appelhans and Linda J. Luecken. 2006. Heart rate variability as an index of regulated emotional responding. Review of General Psychology 10, 3, 229.Google ScholarGoogle ScholarCross RefCross Ref
  3. Faisal Arafsha, Kazi Masudul Alam, and Abdulmotaleb El Saddik. 2015. Design and development of a user centric affective haptic jacket. Multimedia Tools and Applications 74, 9, 3035--3052. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Jeremy N. Bailenson, Nick Yee, Scott Brave, Dan Merget, and David Koslow. 2007. Virtual interpersonal touch: Expressing and recognizing emotions through haptic devices. Human--Computer Interaction 22, 3, 325--353. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Tonio Ball, Markus Kern, Isabella Mutschler, Ad Aertsen, and Andreas Schulze-Bonhage. 2009. Signal quality of simultaneously recorded invasive and non-invasive EEG. Neuroimage 46, 3, 708--716.Google ScholarGoogle ScholarCross RefCross Ref
  6. Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25, 1, 49--59.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Lindsay Brown, Bernard Grundlehner, and Julien Penders. 2011. Towards wireless emotional valence detection from EEG. In Engineering in Medicine and Biology Society (EMBC’11) Annual International Conference of the IEEE. IEEE, 2188--2191.Google ScholarGoogle Scholar
  8. Ross Buck. 1994. The neuropsychology of communication: Spontaneous and symbolic aspects. Journal of Pragmatics 22, 3-4, 265--278.Google ScholarGoogle ScholarCross RefCross Ref
  9. Jongeun Cha, Mohamad Eid, Ahmad Barghout, ASM Rahman, and Abdulmotaleb El Saddik. 2009. HugMe: Synchronous haptic teleconferencing. In Proceedings of the 17th ACM International Conference on Multimedia. ACM, 1135--1136. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Monisha Chakraborty and Shreya Das. 2012. Determination of signal to noise ratio of electrocardiograms filtered by band pass and Savitzky-Golay filters. Procedia Technology 4, 830--833.Google ScholarGoogle ScholarCross RefCross Ref
  11. Manfred Clynes. 1977. Sentics: The Touch of Emotions. Anchor Press, New York, NY.Google ScholarGoogle Scholar
  12. Ruo-Nan Duan, Jia-Yi Zhu, and Bao-Liang Lu. 2013. Differential entropy feature for EEG-based emotion classification. In Neural Engineering (NER), 2013 6th International IEEE/EMBS Conference on. IEEE, 81--84.Google ScholarGoogle ScholarCross RefCross Ref
  13. Elia Gatti, Giandomenico Caruso, Monica Bordegoni, and Charles Spence. 2013. Can the feel of the haptic interaction modify a user’s emotional state?. In World Haptics Conference (WHC’13). IEEE, 247--252.Google ScholarGoogle ScholarCross RefCross Ref
  14. Antal Haans and Wijnand I. Jsselsteijn. 2006. Mediated social touch: A review of current research and future directions. Virtual Reality 9, 2-3, 149--159. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Ali Israr and Ivan Poupyrev. 2011. Control space of apparent haptic motion. In World Haptics Conference (WHC’11). IEEE, 457--462.Google ScholarGoogle ScholarCross RefCross Ref
  16. Tanimu Ahmed Jibril and Mardziah Hayati Abdullah. 2013. Relevance of emoticons in computer-mediated communication contexts: An overview. Asian Social Science 9, 4, 201.Google ScholarGoogle ScholarCross RefCross Ref
  17. Min-Ki Kim, Miyoung Kim, Eunmi Oh, and Sung-Phil Kim. 2013. A review on the computational methods for emotional state estimation from the human EEG. Computational and Mathematical Methods in Medicine, vol. 2013, 13 pages.Google ScholarGoogle ScholarCross RefCross Ref
  18. Franklin B. Krohn. 2004. A generational approach to using emoticons as nonverbal communication. Journal of Technical Writing and Communication 34, 4, 321--328.Google ScholarGoogle ScholarCross RefCross Ref
  19. Peter J. Lang, Margaret M. Bradley, and Bruce N. Cuthbert. 2008. International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical Report A-8. Gainesville, FL: University of Florida, USA.Google ScholarGoogle Scholar
  20. Paul Lemmens, Floris Crompvoets, Dirk Brokken, Jack Van Den Eerenbeemd, and Gert-Jan de Vries. 2009. A body-conforming tactile jacket to enrich movie viewing. In 3rd Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (World Haptics’09). IEEE, 7--12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Rodrigo Lentini, Beatrice Ionascu, Friederike A. Eyssel, Scandar Copti, and Mohamad Eid. 2016. Authoring tactile gestures: Case study for emotion stimulation. World Academy of Science, Engineering and Technology, International Journal of Computer, Electrical, Automation, Control and Information Engineering 10, 11, 1862--1867.Google ScholarGoogle Scholar
  22. Antonella Mazzoni and Nick Bryan-Kinns. 2015a. How does it feel like? An exploratory study of a prototype system to convey emotion through haptic wearable devices. In 7th International Conference on Intelligent Technologies for Interactive Entertainment (INTETAIN’15). IEEE, 64--68. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Troy McDaniel, Shantanu Bala, Jacob Rosenthal, Ramin Tadayon, Arash Tadayon, and Sethuraman Panchanathan. 2014. Affective haptics for enhancing access to social interactions for individuals who are blind. In International Conference on Universal Access in Human-Computer Interaction. Springer, 419--429.Google ScholarGoogle ScholarCross RefCross Ref
  24. Neurobehavioral Systems Inc. 2017. Presentation. Retrieved November 14, 2017 from https://www.neurobs.com/.Google ScholarGoogle Scholar
  25. Lauri Nummenmaa, Enrico Glerean, Riitta Hari, and Jari K. Hietanen. 2014. Bodily maps of emotions. Proceedings of the National Academy of Sciences 111, 2, 646--651.Google ScholarGoogle ScholarCross RefCross Ref
  26. Panagiotis C. Petrantonakis and Leontios J. Hadjileontiadis. 2011. A novel emotion elicitation index using frontal brain asymmetry for enhanced EEG-based emotion recognition. IEEE Transactions on Information Technology in Biomedicine 15, 5, 737--746. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Abu Saleh Md Mahfujur Rahman, S. K. Alamgir Hossain, and Abdulmotaleb El Saddik. 2010. Bridging the gap between virtual and real world by bringing an interpersonal haptic communication system in second life. In IEEE International Symposium on Multimedia (ISM’10). IEEE, 228--235. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Jukka Raisamo, Roope Raisamo, and Veikko Surakka. 2013. Comparison of saltation, amplitude modulation, and a hybrid method of vibrotactile stimulation. IEEE Transactions on Haptics 6, 4, 517--521. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Debra L. Roter, Richard M. Frankel, Judith A. Hall, and David Sluyter. 2006. The expression of emotion through nonverbal behavior in medical visits. Journal of General Internal Medicine 21, S1, S28--S34.Google ScholarGoogle ScholarCross RefCross Ref
  30. Katri Salminen, Veikko Surakka, Jani Lylykangas, Jukka Raisamo, Rami Saarinen, Roope Raisamo, Jussi Rantala, and Grigori Evreinov. 2008. Emotional and behavioral responses to haptic stimulation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1555--1562. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Louis A. Schmidt and Laurel J. Trainor. 2001. Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cognition 8 Emotion 15, 4, 487--500.Google ScholarGoogle Scholar
  32. Mohammad Soleymani, Maja Pantic, and Thierry Pun. 2012. Multimodal emotion recognition in response to videos. IEEE Transactions on Affective Computing 3, 2, 211--223. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Kai Sun, Junqing Yu, Yue Huang, and Xiaoqiang Hu. 2009. An improved valence-arousal emotion space for video affective content representation and recognition. In IEEE International Conference on Multimedia and Expo (ICME’09). IEEE, 566--569. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Dzmitry Tsetserukou, Alena Neviarouskaya, Helmut Prendinger, Mitsuru Ishizuka, and Susumu Tachi. 2010. iFeel_IM: Innovative real-time communication system with rich emotional and haptic channels. In CHI’10 Extended Abstracts on Human Factors in Computing Systems. ACM, 3031--3036. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Xiao-Wei Wang, Dan Nie, and Bao-Liang Lu. 2014. Emotional state classification from EEG data using machine learning approach. Neurocomputing 129, 94--106. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Joyce H. D. M. Westerink, Egon L. Van Den Broek, Marleen H. Schut, Jan Van Herk, and Kees Tuinenbreijer. 2008. Computing emotion awareness through galvanic skin response and facial electromyography. In Probing Experience. Springer, 149--162.Google ScholarGoogle Scholar
  37. Yongjae Yoo, Taekbeom Yoo, Jihyun Kong, and Seungmoon Choi. 2015. Emotional responses of tactile icons: Effects of amplitude, frequency, duration, and envelope. In IEEE World Haptics Conference (WHC’15). IEEE, 235--240.Google ScholarGoogle Scholar
  38. Xu Zhe and A. C. Boucouvalas. 2002. Text-to-emotion engine for real time Internet communication. In Proceedings of International Symposium on Communication Systems, Networks and DSPs. 164--168.Google ScholarGoogle Scholar
  39. Wei-Long Zheng, Jia-Yi Zhu, and Bao-Liang Lu. 2016. Identifying stable patterns over time for emotion recognition from EEG. Arxiv Preprint Arxiv:1601.02197.Google ScholarGoogle Scholar

Index Terms

  1. Implicit Emotion Communication: EEG Classification and Haptic Feedback

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM Transactions on Multimedia Computing, Communications, and Applications
            ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 14, Issue 1
            February 2018
            287 pages
            ISSN:1551-6857
            EISSN:1551-6865
            DOI:10.1145/3173554
            Issue’s Table of Contents

            Copyright © 2017 ACM

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 20 December 2017
            • Accepted: 1 August 2017
            • Revised: 1 June 2017
            • Received: 1 March 2017
            Published in tomm Volume 14, Issue 1

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article
            • Research
            • Refereed

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!