Competitive affective gaming: winning with a smile

ABSTRACT

Human-computer interaction (HCI) is expanding towards natural modalities of human expression. Gestures, body movements and other affective interaction techniques can change the way computers interact with humans. In this paper, we propose to extend existing interaction paradigms by including facial expression as a controller in videogames. NovaEmötions is a multiplayer game where players score by acting an emotion through a facial expression. We designed an algorithm to offer an engaging interaction experience using the facial expression. Despite the novelty of the interaction method, our game scoring algorithm kept players engaged and competitive. A user study done with 46 users showed the success and potential for the usage of affective-based interaction in videogames, i.e., the facial expression as the sole controller in videogames. Moreover, we released a novel facial expression dataset with over 41,000 images. These face images were captured in a novel and realistic setting: users playing games where a player's facial expression has an impact on the game score.

References

1. Arapakis, I., Jose, J.M. and Gray, P.D. 2008. Affective feedback. ACM SIGIR (2008), 395--402.
2. Arapakis, I., Moshfeghi, Y., Joho, H., Ren, R., Hannah, D. and Jose, J.M. 2009. Integrating facial expressions into user profiling for the improvement of a multimodal recommender system. IEEE International Conference on Multimedia and Expo (ICME) (2009), 1440--1443.
3. Dahmane, M. and Meunier, J. 2011. Continuous emotion recognition using Gabor energy filters. Affective Computing and Intelligent Interaction (2011), 351--358.
4. Deriso, D., Susskind, J., Krieger, L. and Bartlett, M. 2012. Emotion Mirror: A Novel Intervention for Autism Based on Real-Time Expression Recognition. Computer Vision -- ECCV Workshops and Demonstrations. A. Fusiello, V. Murino, and R. Cucchiara, eds. Springer. 671--674.
5. Desurvire, H., Caplan, M. and Toth, J. 2004. Using heuristics to evaluate the playability of games. Human factors and computing systems (CHI) (Apr. 2004), 1509--1512.
6. Edwards, G.J., Taylor, C.J. and Cootes, T.F. 1998. Interpreting face images using active appearance models. IEEE International Conference on Automatic Face and Gesture Recognition (1998), 300--305.
7. Ekman, P. 1999. Basic Emotions. Handbook of Cognition and Emotion. John Wiley & Sons.
8. Ekman, P., Friesen, W. V. and Hager, J.C. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press.
9. EverQuest 2 Receives Facial Tracking Upgrade: 2012. http://www.tomshardware.com/news/SOEmote-EverQuest-SOE-MMORPG-Live-Driver,16701.html. Accessed: 2013-07--31.
10. EverQuest II: SOEmote Released: http://www.mmorpg.com/discussion2.cfm/thread/359738/. Accessed: 2013-07--31.
11. Iacobini, M., Gonsalves, T., Bianchi-Berthouze, N. and Frith, C. 2010. Emotional contagion in interactive art. Kansei Engineering and Emotion Research (2010), 1975--1984.
12. Kinect for Xbox 360: http://www.xbox.com/Kinect. Accessed: 2013-07--31.
13. Lang, P.J. 2008. International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical Report A-8, University of Florida, Gainesville (2008).
14. Lang, P.J. 1995. The emotion probe: Studies of motivation and attention. American psychologist. 50, 5 (1995), 372--385.
15. Littlewort, G. and Fasel, I. 2002. Fully automatic coding of basic expressions from video. INC MPLab Technical Report. (2002), 6.
16. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z. and Matthews, I. 2010. The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. IEEE Computer Vision and Pattern Recognition - Workshops (2010), 94--101.
17. MacQueen, J.B. 1967. Some Methods for Classification and Analysis of MultiVariate Observations. Berkeley Symposium on Mathematical Statistics and Probability (1967), 281--297.
18. Manjunath, B.S. and Ma, W.Y. 1996. Texture features for browsing and retrieval of image data. IEEE Trans on Pattern Analysis and Machine Intelligence. 18, 8 (1996), 837--842.
19. Mourão, A., Borges, P., Correia, N. and Magalhães, J. 2013. Facial Expression Recognition by Sparse Reconstruction with Robust Features. Image Analysis and Recognition (2013), 107--115.
20. Orvalho, V., Miranda, J. and Sousa, A. 2009. What a Feeling: Learning Facial Expressions and Emotions. Videojogos Conference (2009), 1--11.
21. Paiva, A., Costa, M., Chaves, R., Piedade, M., Mourão, D., Sobral, D., Höök, K., Andersson, G. and Bullock, A. 2003. SenToy: an affective sympathetic interface. International Journal of Human-Computer Studies. 59, 1--2 (2003), 227--235.
22. Pantic, M., Sebe, N., Cohn, J.F. and Huang, T. 2005. Affective multimodal human-computer interaction. ACM Multimedia (2005), 669--676.
23. Savva, N., Scarinzi, A. and Bianchi-Berthouze, N. 2012. Continuous Recognition of Player's Affective Body Expression as Dynamic Quality of Aesthetic Experience. IEEE Trans on Computational Intelligence and AI in Games. 4, 3 (2012), 199--212.
24. SOEmote Guide: 2012. http://soemote.com/guide/index.vm. Accessed: 2013-07--31.
25. Stoiber, N., Aubault, O., Seguier, R. and Breton, G. 2010. The mimic game: real-time recognition and imitation of emotional facial expressions. ACM SIGGRAPH Talks. ACM.
26. Tavares, G., Mourão, A. and Magalhães, J. 2013. Crowdsourcing for affective-interaction in computer games. Workshop on Crowdsourcing for Multimedia at ACM MM (2013).
27. The Daily Grind: Do you really want MMO innovation?: http://massively.joystiq.com/2012/08/16/the-daily-grind-do-you-really-want-mmo-innovation/. Accessed: 2013-07--31.
28. Tian, Y.L., Kanade, T. and Cohn, J.F. 2005. Facial expression analysis. Handbook of face recognition. S.Z. Li and A.K. Jain, eds. Springer. 247--275.
29. Valstar, M.F., Mehu, M., Jiang, B., Pantic, M. and Scherer, K. 2012. Meta-Analysis of the First Facial Expression Recognition Challenge. IEEE Trans on Systems, Man, and Cybernetics, Part B: Cybernetics. 42, 4 (2012), 966--979.
30. Viola, P. and Jones, M.J. 2004. Robust Real-Time Face Detection. International Journal of Computer Vision. 57, 2 (2004), 137--154.
31. Wang, N. and Marsella, S. 2006. Introducing EVG: An emotion evoking game. Intelligent Virtual Agents. 4133, (2006), 282--291.
32. Yeasin, M., Bullot, B. and Sharma, R. 2006. Recognition of facial expressions and measurement of levels of interest from video. IEEE Trans on Multimedia. 8, 3 (2006), 500--508.

Index Terms

• Applied computing

  • Computers in other domains

    • Personal computers and PC applications

      • Computer games

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Reconstruction

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Touch screens

• Information systems

  • Information systems applications

    • Multimedia information systems

      • Massively multiplayer online games

• Software and its engineering

  • Software organization and properties

    • Contextual software domains

      • Virtual worlds software

        • Interactive games

• Theory of computation

  • Theory and algorithms for application domains

    • Algorithmic game theory and mechanism design