Mixed Reality Speaker Identification as an Accessibility Tool for Deaf and Hard of Hearing Users

ABSTRACT

People who are Deaf or Hard of Hearing (DHH) benefit from text captioning to understand audio, yet captions alone are often insufficient for the complex environment of a panel presentation, with rapid and unpredictable turn-taking among multiple speakers. It is challenging and tiring for DHH individuals to view captioned panel presentations, leading to feelings of misunderstanding and exclusion. In this work, we investigate the potential of Mixed Reality (MR) head-mounted displays for providing captioning with visual cues to indicate which person on the panel is speaking. For consistency in our experimental study, we simulate a panel presentation in virtual reality (VR) with various types of MR visual cues; in a study with 18 DHH participants, visual cues made it easier to identify speakers.

References

1. A. Brown, J. Turner, J. Patterson, A. Schmitz, M. Armstrong, M. Glancy. 2017. Subtitles in 360-degree Video. In 2017 ACM International Conference on Interactive Experiences for TV and Online Video (TVX '17 Adjunct). ACM, New York, NY, USA, 3-8. DOI: https://doi.org/10.1145/3084289.3089915
2. K. Kurzhals, E. Cetinkaya, Y. Hu, W. Wang, D. Weiskopf. 2017. Close to the Action: Eye-Tracking Evaluation of Speaker-Following Subtitles. In Proc. 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, NY, NY, USA, 6559-6568. DOI: https://doi.org/10.1145/3025453.3025772
3. R. Kushalnagar, G. Behm, K. Wolfe, P. Yeung, B. Dingman, S. Ali, A. Glasser, C. Ryan. 2017. RTTD-ID: Tracked Captions with Multiple Speakers for Deaf Students. In Proc. 2018 ASEE Annual Conference & Exposition (ASEE ‘18). Salt Lake City, UT, USA. https://peer.asee.org/30945
4. S. Rothe, K. Tran, H. Hußmann. 2018. Dynamic Subtitles in Cinematic Virtual Reality. In Proc. 2018 ACM International Conference on Interactive Experiences for TV and Online Video (TVX '18). ACM, New York, NY, USA, 209-214. DOI: https://doi.org/10.1145/3210825.3213556

Index Terms

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

      • Virtual reality