Achieving eye contact in a one-to-many 3D video teleconferencing system

Abstract

We present a set of algorithms and an associated display system capable of producing correctly rendered eye contact between a three-dimensionally transmitted remote participant and a group of observers in a 3D teleconferencing system. The participant's face is scanned in 3D at 30Hz and transmitted in real time to an autostereoscopic horizontal-parallax 3D display, displaying him or her over more than a 180° field of view observable to multiple observers. To render the geometry with correct perspective, we create a fast vertex shader based on a 6D lookup table for projecting 3D scene vertices to a range of subject angles, heights, and distances. We generalize the projection mathematics to arbitrarily shaped display surfaces, which allows us to employ a curved concave display surface to focus the high speed imagery to individual observers. To achieve two-way eye contact, we capture 2D video from a cross-polarized camera reflected to the position of the virtual participant's eyes, and display this 2D video feed on a large screen in front of the real participant, replicating the viewpoint of their virtual self. To achieve correct vertical perspective, we further leverage this image to track the position of each audience member's eyes, allowing the 3D display to render correct vertical perspective for each of the viewers around the device. The result is a one-to-many 3D teleconferencing system able to reproduce the effects of gaze, attention, and eye contact generally missing in traditional teleconferencing systems.

References

1. Argyle, M., and Cook, M. 1976. Gaze and Mutual Gaze. Cambridge University Press, London.
2. Chen, M. 2002. Leveraging the asymmetric sensitivity of eye contact for videoconference. In CHI '02: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, New York, NY, USA, 49--56.
3. Gemmell, J., Toyama, K., Zitnick, C., Kang, T., and Seitz, S. 2000. Gaze awareness for video-conferencing: a software approach. Multimedia, IEEE 7, 4 (Oct-Dec), 26--35.
4. Grayson, D. M., and Monk, A. F. 2003. Are you looking at me? Eye contact and desktop video conferencing. ACM Trans. Comput.-Hum. Interact. 10, 3, 221--243.
5. Gross, M., Würmlin, S., Naef, M., Lamboray, E., Spagno, C., Kunz, A., Koller-Meier, E., Svoboda, T., Gool, L. V., Lang, S., Strehlke, K., De Moere, A. V., and Staadt, O. 2003. blue-c: A spatially immersive display and 3d video portal for telepresence. ACM Transactions on Graphics 22, 3 (July), 819--827.
6. Ishii, H., Kobayashi, M., and Grudin, J. 1993. Integration of interpersonal space and shared workspace: Clearboard design and experiments. ACM Trans. Inf. Syst. 11, 4, 349--375.
7. Jerald, J., and Daily, M. 2002. Eye gaze correction for video-conferencing. In ETRA '02: Proceedings of the 2002 symposium on Eye tracking research & applications, ACM, New York, NY, USA, 77--81.
8. Jones, A., McDowall, I., Yamada, H., Bolas, M., and Debevec, P. 2007. Rendering for an interactive 360ř light field display. ACM Transactions on Graphics 26, 3 (July), 40:1--40:10.
9. Kajiya, J. T., and Kay, T. L. 1989. Rendering fur with three dimensional textures. In Computer Graphics (Proceedings of SIGGRAPH 89), 271--280.
10. Liu, J., Beldie, I. P., and Wöpking, M. 1995. A computational approach to establish eye-contact in videocommunication. In in Videocommunication, Int. Workshop on Stereoscopic and Three Dimentional Imaging, 229--234.
11. Matusik, W., and Pfister, H. 2004. 3D tv: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Transactions on Graphics 23, 3 (Aug.), 814--824.
12. Morris, E. 2004. The Fog of War: 13 questions and answers on the filmmaking of Errol Morris. FLM Magazine. http://www.errolmorris.com/content/eyecontact/interrotron.html.
13. Muhlbach, L., Bocker, M., and Prussog, A. 1995. Telepresence in videocommunications: a study on stereoscopy and individual eye contact. Human Factors 37, 2, 290--305.
14. Mukawa, N., Oka, T., Arai, K., and Yuasa, M. 2005. What is connected by mutual gaze?: user's behavior in video-mediated communication. In CHI '05: CHI '05 extended abstracts on Human factors in computing systems, ACM, New York, NY, USA, 1677--1680.
15. Nayar, S. K., Krishnan, G., Grossberg, M. D., and Raskar, R. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM Transactions on Graphics 25, 3 (July), 935--944.
16. Nguyen, D., and Canny, J. 2005. Multiview: spatially faithful group video conferencing. In CHI '05: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, New York, NY, USA, 799--808.
17. Nguyen, D. T., and Canny, J. 2007. Multiview: improving trust in group video conferencing through spatial faithfulness. In CHI '07: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, New York, NY, USA, 1465--1474.
18. Ostromoukhov, V. 2001. A simple and efficient error-diffusion algorithm. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 567--572.
19. Ott, M., Lewis, J. P., and Cox, I. 1993. Teleconferencing eye contract using a virtual camera. In CHI '93: INTERACT '93 and CHI '93 conference companion on Human factors in computing systems, ACM, New York, NY, USA, 109--110.
20. Perlin, K., Paxia, S., and Kollin, J. S. 2000. An autostereoscopic display. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, 319--326.
21. Poulin, P., and Fournier, A. 1990. A model for anisotropic reflection. In Computer Graphics (Proceedings of SIGGRAPH 90), 273--282.
22. Prussog, A., Muhlbach, L., and Bocker, M. 1994. Telepresence in videocommunications. In Proceedings of the Human Factors and Ergonomics Society 38th Annual Meeting, Human Factors and Ergonomics Society, Santa Monica, CA, USA, vol. 1, 180--184.
23. Quante, B., and Muhlbach, L. 1999. Eye-contact in multipoint videoconferencing. In Proceedings of the 17th International Symposium on Human Factors in Telecommunication.
24. Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., and Fuchs, H. 1998. The office of the future: A unified approach to image-based modeling and spatially immersive displays. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, 179--188.
25. Rees, J. 2008. Critics pan CNN's fake election holograms. New Zealand Herald (Nov 7).
26. Rose, D. A. D., and Clarke, P. M. 1995. A review of eye-to-eye videoconferencing techniques. BT technology journal 13, 4, 127--131.
27. Sandin, D. J., Margolis, T., Ge, J., Girado, J., Peterka, T., and Defanti, T. A. 2005. The varrier autostereoscopic virtual reality display. In SIGGRAPH '05: ACM SIGGRAPH 2005 Papers, ACM, New York, NY, USA, 894--903.
28. Sellen, A. J. 1995. Remote conversations: The effects of mediating talk with technology. Human Computer Interaction 10, 401--444.
29. Steinmeyer, J. 1999. The Science behind the Ghost: A Brief History of Pepper's Ghost. Hahne.
30. Taguchi, Y., Koike, T., Takahashi, K., and Naemura, T. 2009. Transcaip: A live 3d tv system using a camera array and an integral photography display with interactive control of viewing parameters. Accepted to IEEE Transactions on Visualization and Computer Graphics.
31. Vertegaal, R. 1999. The gaze groupware system: mediating joint attention in multiparty communication and collaboration. In CHI '99: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, New York, NY, USA, 294--301.
32. Viola, P., and Jones, M. J. 2004. Robust real-time face detection. International Journal of Computer Vision 57, 2, 137--154.
33. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Transactions on Graphics 24, 3 (Aug), 765--776.
34. Woods, E., Mason, P., and Billinghurst, M. 2003. Magicmouse: an inexpensive 6-degree-of-freedom mouse. In GRAPHITE 2003, 285--286.
35. Yang, R., and Zhang, Z. 2004. Eye gaze correction with stereovision for video-teleconferencing. Pattern Analysis and Machine Intelligence, IEEE Transactions on 26, 7 (July), 956--960.
36. Yang, J. C., Everett, M., Buehler, C., and Mcmillan, L. 2002. A real-time distributed light field camera. In Rendering Techniques 2002: 13th Eurographics Workshop on Rendering, 77--86.
37. Zhang, S., and Huang, P. 2006. High-resolution, real-time three-dimensional shape measurement. Optical Engineering 45, 12.
38. Zhang, Z. 2000. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22, 11, 1330--1334.
39. Zwicker, M., Matusik, W., Durand, F., and Pfister, H. 2006. Antialiasing for automultiscopic 3D displays. In Rendering Techniques 2006: 17th Eurographics Workshop on Rendering, 73--82.

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Graphics recognition and interpretation

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision tasks

        • Scene understanding

      • Image and video acquisition

        • 3D imaging

  • Computer graphics

    • Animation

• Information systems

  • World Wide Web

    • Web applications

      • Internet communications tools

        • Web conferencing