Real-time Projection of Lip Animation onto Face Masks using OmniProcam

ABSTRACT

This paper describes an OmniProcam system, which enables 360 degree horizontal projection by a fisheye lens with a coaxial procam in which the optical axes of the camera and projector are exactly matched. Combined with 2D marker recognition, the OmniProcam can display images onto screens at arbitrary positions in 3D space. As an example application, we developed a system which projects lip animation onto the user’s face masks for better communication at the physical meeting in current COVID-19 situation. The system recognizes the user’s speech, generates the lip animation using Lipsync, and projects the animation onto the user’s face masks.

References

1. Carbon Claus-Christian. 2020. Wearing Face Masks Strongly Confuses Counterparts in Reading Emotions. Vol. 11. 2526 pages.
2. Andreas Rene Fender, Hrvoje Benko, and Andy Wilson. 2017. MeetAlive: Room-Scale Omni-Directional Display System for Multi-User Content and Control Sharing. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (Brighton, United Kingdom) (ISS ’17). Association for Computing Machinery, New York, NY, USA, 106–115. https://doi.org/10.1145/3132272.3134117
3. Brett Jones, Rajinder Sodhi, Michael Murdock, Ravish Mehra, Hrvoje Benko, Andrew Wilson, Eyal Ofek, Blair MacIntyre, Nikunj Raghuvanshi, and Lior Shapira. 2014. RoomAlive: Magical Experiences Enabled by Scalable, Adaptive Projector-Camera Units. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology(Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 637–644. https://doi.org/10.1145/2642918.2647383

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Mixed / augmented reality