skip to main content
research-article

Exploiting unconscious user signals in multimodal human-computer interaction

Published:17 October 2013Publication History
Skip Abstract Section

Abstract

This article presents the idea of empathic stimulation that relies on the power and potential of unconsciously conveyed attentive and emotional information to facilitate human-machine interaction. Starting from a historical review of related work presented at past ACM Multimedia conferences, we discuss challenges that arise when exploiting unconscious human signals for empathic stimulation, such as the real-time analysis of psychological user states and the smooth adaptation of the human-machine interface based on this analysis. A classical application field that might benefit from the idea of unconscious human-computer interaction is the exploration of massive datasets.

References

  1. André, E. 2011. Experimental methodology in emotion-oriented computing. IEEE Perv. Comput. 10, 3, 54--57. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Buscher, G., Dengel, A., Biedert, R., and Elst, L. V. 2012. Attentive documents: Eye tracking as implicit feedback for information retrieval and beyond. ACM Trans. Interact. Intell. Syst. 1, 2, Article 9, 30 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Cohn, J. F. and Katz, G. S. 1998. Bimodal expression of emotion by face and voice. In Proceedings of the 6th ACM International Conference on Multimedia: Face/Gesture Recognition and their Applications (MULTIMEDIA'98). ACM, New York, NY, 41--44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Ekman, P. 1999. Basic emotions. In Handbook of Cognition and Emotion, Wiley, 45--60.Google ScholarGoogle Scholar
  5. Gilroy, S. W., Cavazza, M., Chaignon, R., Mäkelä, S.-M., Niranen, M., André, E., Vogt, T., Urbain, J., Billinghurst, M., Seichter, H., and Benayoun, M. 2008. E-tree: Emotionally driven augmented reality art. In Proceedings of the 16th ACM International Conference on Multimedia (MM'08). ACM, New York, NY, 945--948. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Jaimes, A., Sebe, N., and Gatica-Perez, D. 2006. Human-centered computing: a multimedia perspective. In Proceedings of the 14th Annual ACM International Conference on Multimedia (MULTIMEDIA'06). ACM, New York, NY, 855--864. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Kapoor, A. and Picard, R. W. 2005. Multimodal affect recognition in learning environments. In Proceedings of the 13th annual ACM International Conference on Multimedia (MULTIMEDIA'05). ACM, New York, NY, 677--682. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Lisetti, C. L. and Nasoz, F. 2002. MAUI: A multimodal affective user interface. In Proceedings of the 10th ACM International Conference on Multimedia (MULTIMEDIA'02). ACM, New York, NY, 161--170. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Mower, E., Mataric, M. J., and Narayanan, S. S. 2011. A framework for automatic human emotion classification using emotion profiles. IEEE Trans. Audio Speech Lang. Process. 19, 5, 1057--1070. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Nakatsu, R. 1998. Nonverbal information recognition and its application to communications. In Proceedings of the 6th ACM International Conference on Multimedia: Face/Gesture Recognition and their Applications (MULTIMEDIA'98). ACM, New York, NY, 2--9. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Pantic, M., Sebe, N., Cohn, J. F., and Huang, T. 2005. Affective multimodal human-computer interaction. In Proceedings of the 13th Annual ACM International Conference on Multimedia (MULTIMEDIA'05). ACM, New York, NY, 669--676. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Pentland, A. 2005. Socially aware media. In Proceedings of the 13th Annual ACM International Conference on Multimedia (MULTIMEDIA'05). ACM, New York, NY, 690--695. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Stiefelhagen, R., Yang, J., and Waibel, A. 1999. Modeling focus of attention for meeting indexing. In Proceedings of the 7th ACM International Conference on Multimedia (Part 1) (MULTIMEDIA'99). ACM, New York, NY, 3--10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Vinciarelli, A., Pantic, M., Bourlard, H., and Pentland, A. 2008. Social signal processing: State-of-the-art and future perspectives of an emerging domain. In Proceedings of the 16th ACM International Conference on Multimedia (MM'08). ACM, New York, NY, 1061--1070. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Vogt, T. and André, E. 2005. Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME'05). IEEE, Los Alamitos, CA, 474--477.Google ScholarGoogle Scholar
  16. Wagner, J., Lingenfelser, F., André, E., Mazzei, D., Tognetti, A., Lanatà, A., De Rossi, D., Betella, A., Zucca, R., Omedas, P., and Verschure, P. F. M. J. 2013. A sensing architecture for empathetic data systems. In Proceedings of the 4th Augmented Human International Conference. ACM, New York, NY, 96--99. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Exploiting unconscious user signals in multimodal human-computer interaction

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 9, Issue 1s
        Special Sections on the 20th Anniversary of ACM International Conference on Multimedia, Best Papers of ACM Multimedia 2012
        October 2013
        218 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/2523001
        Issue’s Table of Contents

        Copyright © 2013 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 17 October 2013
        • Revised: 1 July 2013
        • Accepted: 1 July 2013
        • Received: 1 May 2013
        Published in tomm Volume 9, Issue 1s

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!