Reinforcement learning utilizes proxemics: An avatar learns to manipulate the position of people in immersive virtual reality

Abstract

A reinforcement learning (RL) method was used to train a virtual character to move participants to a specified location. The virtual environment depicted an alleyway displayed through a wide field-of-view head-tracked stereo head-mounted display. Based on proxemics theory, we predicted that when the character approached within a personal or intimate distance to the participants, they would be inclined to move backwards out of the way. We carried out a between-groups experiment with 30 female participants, with 10 assigned arbitrarily to each of the following three groups: In the Intimate condition the character could approach within 0.38m and in the Social condition no nearer than 1.2m. In the Random condition the actions of the virtual character were chosen randomly from among the same set as in the RL method, and the virtual character could approach within 0.38m. The experiment continued in each case until the participant either reached the target or 7 minutes had elapsed. The distributions of the times taken to reach the target showed significant differences between the three groups, with 9 out of 10 in the Intimate condition reaching the target significantly faster than the 6 out of 10 who reached the target in the Social condition. Only 1 out of 10 in the Random condition reached the target. The experiment is an example of applied presence theory: we rely on the many findings that people tend to respond realistically in immersive virtual environments, and use this to get people to achieve a task of which they had been unaware. This method opens up the door for many such applications where the virtual environment adapts to the responses of the human participants with the aim of achieving particular goals.

References

1. Abbeel, P. and Ng, A. Y. 2004. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the 24th International Conference on Machine Learning (ICML'04). ACM, New York, 1.
2. Baas, J. M., Nugent, M., Lissek, S., Pine, D. S., and Grillon, C. 2004. Fear conditioning in virtual reality contexts: A new tool for the study of anxiety. Biol. Psych. 55, 11, 1056--1060.
3. Bailenson, J. N., Blascovich, J., Beall, A. C., and Loomis, J. M. 2003. Interpersonal distance in immersive virtual environments. Personality Soc. Psychol. Bull. 29, 7, 819--833.
4. Bailenson, J. N., Blascovich, J., Beall, A. C., and Loomis, J. M. 2001. Equilibrium theory revisited: Mutual gaze and personal space in virtual environments. Presence: Teleoper.Virtual Environ. 10, 6, 583--598.
5. Barfield, W. and Hendrix, C. 1995. The effect of update rate on the sense of presence within virtual environments. Virtual Reality: J. Virtual Reality Soc. 1, 1, 3--16.
6. Barfield, W. and Weghorst, S. 1993. The sense of presence within virtual environments: A conceptual framework. In Human-Computer Interaction: Software and Hardware Interfaces, G. Salvendy and M. Smith Eds, Elsevier, 699--704.
7. Carrozzino, M., Tecchia, F., Bacinelli, S., Cappelletti, C., and Bergamasco, M. 2005. Lowering the development time of multimodal interactive application: The real-life experience of the XVR project. In Proceedings of the ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE'05). ACM, New York, 270--273.
8. Collins, S., Ruina, A., Tedrake, R., and Wisse, M. 2005. Efficient bipedal robots based on passive dynamic walkers, Science 307, 5712.
9. Doya, K. 2007. Reinforcement learning: Computational theory and biological mechanisms. HFSP J. 1, 1, 30--40.
10. Draper, J. V., Kaber, D. B., and Usher, J. M. 1998. Telepresence. Hum. Factors 40, 3, 354--375.
11. Freeman, D., Pugh, K., Antley, A., Slater, M., Bebbington, P., Gittins, M., Dunn, G., Kuipers, E., Fowler, D., and Garety, P. 2008. Virtual reality study of paranoid thinking in the general population. British J. Psych. 192, 4, 258--263.
12. Friedman, D. and Gillies, M. 2005. Teaching virtual characters how to use body language. In Intelligent Virtual Agents, Lecture Notes in Computer Science, vol. 3663, Panayiotopoulos, et al. Eds., Springer, Berlin, 205--214.
13. Friedman, D., Steed, A., and Slater, M. 2007. Spatial social behavior in second life. In Proceedings of the 7th International Conference on intelligent Virtual Agents. C. Pelachaud, et al. Eds., Lecture Notes in Artificial Intelligence, vol. 4722. Springer, Berlin, 252--263.
14. Garau, M., Friedman, D., Widenfeld, H., Antley, A., Brogni, A., and Slater, M. 2008. Temporal and spatial variations in presence: Qualitative analysis of interviews from an experiment on breaks in presence. Presence: Teleoper. Virtual Environ. 17, 3, 293--309.
15. Gillies, M. and Spanlang, B. 2010. Comparing and evaluating real time character engines for virtual environments. Presence: Teleoper. Virtual Environ. 19, 2, 95--117.
16. Guye-Vuilleme, A., Capin, T. K., Pandzic, S., Magnenat Thalmann, N., and Thalmann, D. 1999. Nonverbal communication interface for collaborative virtual environments. Virtual Reality 4, 1, 49--59.
17. Hall, E. T. 1973. The Hidden Dimension, Leonardo.
18. Harris, S. R., Kemmerling, R. L., and North, M. M. 2002. Brief virtual reality therapy for public speaking anxiety. Cyberpsychol. Behav. 5, 6, 543--550.
19. Held, R. M. and Durlach, N. I. 1992. Telepresence. Presence: Teleoper. Virtual Environ. 1, 1, 109--112.
20. Hodges, L. F., Kooper, R., Meyer, T. C., Rothbaum, B. O., Opdyke, D., De Graaff, J. J., Williford, J. S., and North, M. M. 1995. Virtual environments for treating the fear of heights. Computer 28, 7, 27--34.
21. Hoffman, H. G., Richards, T., Coda, B., Richards, A., and Sharar, S. R. 2003. The illusion of presence in immersive virtual reality during an fMRI brain scan. Cyberpsychol. Behav. 6, 2, 127--131.
22. Hoffman, H. H., Richards, T. L., Coda, B., Bills, A. R., Blough, D., Richards, A. L., and Sharar, S. R. 2004. Modulation of thermal pain-related brain activity with virtual reality: Evidence from MRI. Neuroreport 15, 8, 1245--1248.
23. Ikemoto, L., Arikan, O., and Forsyth, D. 2005. Learning to move autonomously in a hostile world. In ACM SIGGRAPH Sketches (SIGGRAPH'05). J. Buhler Ed., ACM, New York, 46.
24. Krijn, M., Emmelkamp, P. M. G., Olafsson, R. P., and Biemond, R. 2004. Virtual reality exposure therapy of anxiety disorders: A review. Clinical Psychol. Rev. 24, 3, 259--281.
25. Krijn, M., Emmelkamp, P. M. G., Olafsson, R. P., Bouwman, M., Van Gerwen, L. J., Spinhoven, P., Schuemie, M. J., and Van Der Mast, C. A. P. G. 2007. Fear of flying treatment methods: Virtual reality exposure vs. cognitive behavioral therapy. Aviation Space Environ. Med. 78, 2, 121--128.
26. Kuhl, S. A., Thompson, W. B., and Creem-Regehr, S. H. 2009. HMD calibration and its effects on distance judgments. ACM Trans. Appl. Percept. 6, 3, 1--20.
27. Lee, S. J. and Popović, Z. 2010. Learning behavior styles with inverse reinforcement learning. ACM Trans. Graph. 29, 4. Paper: 122.
28. Lessiter, J., Freeman, J., Keogh, E., and Davidoff, J. 2001. A cross-media presence questionnaire: The ITC-sense of presence inventory. Presence: Teleoper. Virtual Environ. 10, 3, 282--297.
29. Llobera, J., Ruffini, G., and Slater, M. 2010. Proxemics with multiple dynamic characters in an immersive virtual environment. ACM Trans. Appl. Percept. 8, 1 (Oct.).
30. Lo, W. and Zwicker, M. 2008. Real-time planning for parameterized human motion. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, New York, 29--38.
31. McCann, J. and Pollard, N. 2007. Responsive characters from motion fragments. ACM Trans. Graph. 26, 3. Article 6.
32. Meehan, M., Insko, B., Whitton, M., and Brooks, F. P. 2002. Physiological measures of presence in stressful virtual environments. ACM Trans. Graph. 21, 3, 645--652.
33. Meehan, M., Razzaque, S., Whitton, M. C., and Brooks, F. P. 2003. Effect of latency on presence in stressful virtual environments. In Proceedings of the IEEE Virtual Reality 2003 (VR). IEEE, Los Alamitos, CA, 141--148.
34. Mine, M. R., Brooks, F. P., and Sequin, C. H. 1997. Moving objects in space: Exploiting proprioception in virtual-environment interaction. In Proceedings of the 24th Annual International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'97). ACM, New York, 19--26.
35. Mortensen, J., Yu, I., Khanna, P., Tecchia, F., Spanlang, B., Marino, G., and Slater, M. 2008. Real-time global illumination for virtual reality applications. IEEE Comput. Graph. Appl. 28, 6, 56--64.
36. Pan, X., Gillies, M., and Slater, M. 2008. Male bodily responses during an interaction with a virtual woman. In Intelligent Virtual Agents, Lecture Notes in Computer Science, vol. 5208, Springer, Berlin, 89--96.
37. Razzaque, S., Swapp, D., Slater, M., and Whitton, M. C. 2002. Redirected walking in place. In Proceedings of the Workshop on Virtual Environments, W. Stürzlinger and S. Müller Eds., ACM, New York. 123--130.
38. Regenbrecht, H. T., Schubert, T. W., and Friedmann, F. 1998. Measuring the sense of presence and its relations to fear of heights in virtual environments. Int. J. Hum.-Comput. Interact. 10, 3, 233--249.
39. Rey, B., Alcaòiz, M., Tembl, J., and Parkhutik, V. 2008. Brain activity and presence: A preliminary study in different immersive conditions using transcranial Doppler monitoring. In Proceedings of Presence. A. Spagnolli and L. Gamberini Eds., 209--218.
40. Rizzo, A., Pair, J., Mcnerney, P. J., Eastlund, E., Manson, B., Gratch, J., Hill, R., Roy, M., Swartout, B., Graap, K., and Wiederhold, B. 2005. From training to toy to treatment: Preliminary results from virtual reality therapy: Application for Iraq war military personnel with combat-related post-traumatic stress disorder. Cyberpsychol. Behav. 8, 4, 352--353.
41. Rothbaum, B. O. and Hodges, L. F. 1999. The use of virtual reality exposure in the treatment of anxiety disorders. Behav. Modification 23, 4, 507--525.
42. Rothbaum, B. O., Hodges, L., Smith, S., Lee, J. H., and Price, L. 2000. A controlled study of virtual reality exposure therapy for the fear of flying. J. Consult. Clinical Psychol. 68, 6, 1020--1026.
43. Sanchez-Vives, M. V. and Slater, M. 2005. From presence to consciousness through virtual reality. Nature Rev. Neurosci. 6, 4, 332--339.
44. Schubert, T., Friedmann, F., and Regenbrecht, H. 2001. The experience of presence: Factor analytic insights. Presence: Teleoper. Virtual Environ. 10, 3, 266--281.
45. Sheridan, T. B. 1992. Musings on telepresence and virtual presence. Presence: Teleoper. Virtual Environ. 1, 1, 120--126.
46. Shum, H. P., Komura, T., and Yamazaki, S. 2008. Simulating interactions of avatars in high dimensional state space. In Proceedings of the Symposium on Interactive 3D Graphics and Games (I3D'08). ACM, New York, 131--138.
47. Slater, M., Spanlang, B., and Corominas, D. 2010. Simulating virtual environments within virtual environments as the basis for a psychophysics of presence. ACM Trans. Graph. 29, 4. Article 92.
48. Slater, M. and Wilbur, S. 1997. A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence: Teleoper.Virtual Environ. 6, 6, 603--616.
49. Slater, M., Khanna, P., Mortensen, J., and Yu, I. 2009. Visual realism enhances realistic response in an immersive virtual environment. IEEE Comput. Graph. Appl. 29, 3, 76--84.
50. Sutton, R. and Barto, A. 1998. Reinforcement Learning: An Introduction: MIT Press, Cambridge, MA.
51. Treuille, A., Lee, Y., and Popović, Z. 2007. Near-optimal character animation with continuous control. ACM Trans.Graph. 26, 3. Article 7.
52. Usoh, M., Catena, E., Arman, S., and Slater, M. 2000. Using presence questionnaires in reality. Presence: Teleoper. Virtual Environ. 9, 5, 497--503.
53. Usoh, M., Arthur, K., Whitton, M. C., Bastos, R., Steed, A., Slater, M., and Brooks, F. P. 1999. Walking > walking-in-place > flying, in virtual environments. In Proceedings of the 26th International Annual Conference on Computer Graphics and interactive Techniques (SIGGRAPH'99). ACM, New York, 359--364.
54. Vigorito, C. M. 2007. Distributed path planning for mobile robots using a swarm of interacting reinforcement learners. In Proceedings of the 6th international Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS'07). ACM, New York, 1--8.
55. Wallach, H. S., Safir, M., and M. Bar-Zvi, M. 2009. Virtual reality cognitive behavior therapy for public speaking anxiety: A randomized clinical trial. Behav. Modification 33, 3, 314--338.
56. Wilcox, L. M., Allison, R. S., Elfassy, S., and Grelik, C. 2006. Personal space in virtual reality. ACM Trans. Appl. Percept. 3, 4, 412--428.
57. Witmer, B. G. and Singer, M. J. 1998. Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoper. Virtual Environ. 7, 3, 225--240.
58. Witmer, B. G., Jerome, C. J., and Singer M. J. 2005. The factor structure of the presence questionnaire. Presence: Teleoper. Virtual Environ. 14, 3, 298--312.
59. Wu, T., Butko, N. J., Ruvulo, P., Bartlett, M. S., and Movellan, J. R. 2009. Learning to make facial expressions. In Proceedings of the IEEE 8th International Conference on Development and Learning (DEVLRN). IEEE, Los Alamitos, CA, 1--6.
60. Zimmons, P. and Panter, A. 2003. The influence of rendering quality on presence and task performance in a virtual environment. In Proceedings of the IEEE Virtual Reality 2003 (VR). IEEE, Los Alamitos, CA, 293--294.

Index Terms

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

      • Virtual reality

  • Machine learning

• Hardware

  • Communication hardware, interfaces and storage

    • Signal processing systems