skip to main content
research-article

I Should Listen More: Real-time Sensing and Feedback of Non-Verbal Communication in Video Telehealth

Published:06 December 2017Publication History
Skip Abstract Section

Abstract

Video telehealth is growing to allow more clinicians to see patients from afar. As a result, clinicians, typically trained for in-person visits, must learn to communicate both health information and non-verbal affective signals to patients through a digital medium. We introduce a system called ReflectLive that senses and provides real-time feedback about non-verbal communication behaviors to clinicians so they can improve their communication behaviors. A user evaluation with 10 clinicians showed that the real-time feedback helped clinicians maintain better eye contact with patients and was not overly distracting. Clinicians reported being more aware of their non-verbal communication behaviors and reacted positively to summaries of their conversational metrics, motivating them to want to improve. Using ReflectLive as a probe, we also discuss the benefits and concerns around automatically quantifying the "soft skills" and complexities of clinician-patient communication, the controllability of behaviors, and the design considerations for how to present real-time and summative feedback to clinicians.

References

  1. Hiroyuki Adachi, Seiko Myojin, and Nobutaka Shimada. 2015. ScoringTalk: a tablet system scoring and visualizing conversation for balancing of participation. In SIGGRAPH Asia 2015 Mobile Graphics and Interactive Applications, 9. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Zia Agha, Debra L. Roter, and Ralph M. Schapira. 2009. An Evaluation of Patient-Physician Communication Style During Telemedicine Consultations. Journal of Medical Internet Research 11, 3: e36.Google ScholarGoogle ScholarCross RefCross Ref
  3. Mara S. Aruguete and Carlos A. Roberts. 2000. Gender, Affiliation, and Control in Physician--Patient Encounters. Sex Roles 42, 1--2: 107--118.Google ScholarGoogle ScholarCross RefCross Ref
  4. Madeline Balaam, Geraldine Fitzpatrick, Judith Good, and Eric Harris. 2011. Enhancing Interactional Synchrony with an Ambient Display. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 867--876. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Rashid L. Bashshur. 1997. Critical issues in telemedicine. Telemedicine Journal 3, 2: 113--126.Google ScholarGoogle ScholarCross RefCross Ref
  6. Rashid L. Bashshur, Jay H. Sanders, and Gary W. Shannon. 1997. Telemedicine: Theory and practice. Charles C. Thomas Springfield, IL.Google ScholarGoogle Scholar
  7. Rainer S. Beck, Rebecca Daughtridge, and Philip D. Sloane. 2002. Physician-patient communication in the primary care office: a systematic review. The Journal of the American Board of Family Practice 15, 1: 25--38.Google ScholarGoogle Scholar
  8. Jozien M. Bensing, Jan J. Kerssens, and Marja van der Pasch. 1995. Patient-directed gaze as a tool for discovering and handling psychosocial problems in general practice. Journal of nonverbal behavior 19, 4: 223--242.Google ScholarGoogle ScholarCross RefCross Ref
  9. Tony Bergstrom and Karrie Karahalios. 2007. Conversation Clock: Visualizing audio patterns in co-located groups. In Proceedings of the 40th Annual Hawaii International Conference on Social Systems (HICSS 2007),78--78. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. J. Beullens, J. J. Rethans, Jo Goedhuys, and Frank Buntinx. 1997. The use of standardized patients in research in general practice. Family Practice 14, 1: 58--62.Google ScholarGoogle ScholarCross RefCross Ref
  11. LouAnne E. Boyd, Alejandro Rangel, Helen Tomimbang, Andrea Conejo-Toledo, Kanika Patel, Monica Tentori, and Gillian R. Hayes. 2016. SayWAT: Augmenting Face-to-Face Conversations for Adults with Autism. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 4872--4883. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Jed R. Brubaker, Gina Venolia, and John C. Tang. 2012. Focusing on shared experiences: moving beyond the camera in video communication. In Proceedings of the Designing Interactive Systems Conference, 96--105. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Patrick S. Byrne and B. E. Long. 1976. Doctors talking to patients. A study of the verbal behaviour of general practitioners consulting in their surgeries. Exeter {England}: Royal College of General Practitioners, 1984, c1976.Google ScholarGoogle Scholar
  14. Byungki Byun, Anurag Awasthi, Philip A. Chou, Ashish Kapoor, Bongshin Lee, and Mary Czerwinski. 2011. Honest signals in video conferencing. In Proceedings of 2011 IEEE International Conference on Multimedia and Expo (ICME), 1--6. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 981--992. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Scott Carter, Laurent Denoue, and Matthew Cooper. 2015. Searching and Browsing Live, Web-based Meetings. In Proceedings of the 23rd ACM International Conference on Multimedia (MM '15), 791--792. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Charles S. Carver and Michael F. Scheier. 2012. Attention and self-regulation: A control-theory approach to human behavior.Springer Science & Business Media.Google ScholarGoogle Scholar
  18. Kathryn Charmaz. 2006. Constructing Grounded Theory: A Practical Guide to Qualitative Analysis. Sage Publications Ltd.Google ScholarGoogle Scholar
  19. Milton Chen. 2002. Leveraging the Asymmetric Sensitivity of Eye Contact for Videoconference. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02), 49--56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Yunan Chen, Victor Ngo, Sidney Harrison, and Victoria Duong. 2011. Unpacking exam-room computing: negotiating computer-use in patient-physician interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 3343--3352. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Ionut Damian, Chiew Seng (Sean) Tan, Tobias Baur, Johannes Schöning, Kris Luyten, and Elisabeth André. 2015. Augmenting Social Interactions: Realtime Behavioural Feedback Using Social Signal Processing Techniques. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 565--574. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Epstein Becker Green. 2016. 50-State Survey of Telemental/Telebehavioral Health.Google ScholarGoogle Scholar
  23. Joan Morris DiMicco, Katherine J. Hollenbach, Anna Pandolfo, and Walter Bender. 2007. The impact of increased awareness while face-to-face. Human--Computer Interaction 22, 1--2: 47--96. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Paul Dourish, Annette Adler, Victoria Bellotti, and Austin Henderson. 1996. Your place or mine? Learning from long-term use of audio-video communication. In Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work (CSCW '96)5, 1: 33--62. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. F. Daniel Duffy, Geoffrey H. Gordon, Gerald Whelan, Kathy Cole-Kelly, Richard Frankel, and others. 2004. Assessing competence in communication and interpersonal skills: the Kalamazoo II report. Academic Medicine 79, 6: 495--507.Google ScholarGoogle ScholarCross RefCross Ref
  26. Susan T. Fiske. 1993. Controlling other people: The impact of power on stereotyping. American Psychologist 48, 6: 621--628.Google ScholarGoogle ScholarCross RefCross Ref
  27. Howard S. Friedman. 1979. Nonverbal communication between patients and medical practitioners. Journal of Social Issues 35, 1: 82--99.Google ScholarGoogle ScholarCross RefCross Ref
  28. Chris Fullwood and Gwyneth Doherty-Sneddon. 2006. Effect of gazing at the camera during a video link on recall. Applied Ergonomics 37, 2: 167--175.Google ScholarGoogle ScholarCross RefCross Ref
  29. Dominik Giger, Jean-Charles Bazin, Claudia Kuster, Tiberiu Popa, and Markus Gross. 2014. Gaze correction with a single webcam. In Multimedia and Expo (ICME), 2014 IEEE International Conference on, 1--6.Google ScholarGoogle ScholarCross RefCross Ref
  30. Dehra Glueck, K. Myers, and C. L. Turvey. 2013. Establishing therapeutic rapport in telemental health. Telemental health: Clinical, technical and administrative foundations for evidence-based practice: 29--46.Google ScholarGoogle Scholar
  31. Andrea. L. Hartzler, Rupa A. Patel, Mary Czerwinski, Wanda Pratt, Asta Roseway, N. Chandrasekaran, and Anthony Back. 2014. Real-time feedback on nonverbal clinical communication. Methods of Information in Medicine 53, 5: 389--405.Google ScholarGoogle ScholarCross RefCross Ref
  32. Helen Ai He, Naomi Yamashita, Ari Hautasaari, Xun Cao, and Elaine M. Huang. 2017. Why Did They Do That?: Exploring Attribution Mismatches Between Native and Non-Native Speakers Using Videoconferencing. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), 297--309. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Stephen G. Henry, Andrea Fuhrel-Forbis, Mary A. M. Rogers, and Susan Eggly. 2012. Association between nonverbal communication during clinical interactions and outcomes: A systematic review and meta-analysis. Patient Education and Counseling 86, 3: 297--315.Google ScholarGoogle ScholarCross RefCross Ref
  34. John Heritage and Douglas W. Maynard. 2006. Communication in medical care: Interaction between primary care physicians and patients. Cambridge University Press.Google ScholarGoogle Scholar
  35. Clara E. Hill and James Gormally. 1977. Effects of reflection, restatement, probe, and nonverbal behaviors on client affect. Journal of Counseling Psychology 24, 2: 92.Google ScholarGoogle ScholarCross RefCross Ref
  36. Mohammadreza Hojat, Joseph S. Gonnella, Thomas J. Nasca, Salvatore Mangione, Michael Vergare, and Michael Magee. 2002. Physician empathy: definition, components, measurement, and relationship to gender and specialty. American Journal of Psychiatry 159, 9: 1563--1569.Google ScholarGoogle ScholarCross RefCross Ref
  37. Ellen A. Isaacs and John C. Tang. 1994. What video can and cannot do for collaboration: a case study. Multimedia Systems 2, 2: 63--73. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Matthew Jackson, Anne H. Anderson, Rachel McEwan, and Jim Mullin. 2000. Impact of Video Frame Rate on Communicative Behaviour in Two and Four Party Groups. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW '00), 11--20. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Donald J. Kiesler and Stephen M. Auerbach. 2003. Integrating measurement of control and affiliation in studies of physician-patient interaction: the interpersonal circumplex. Social Science & Medicine (1982) 57, 9: 1707--1722.Google ScholarGoogle Scholar
  40. Taemie Kim, Agnes Chang, Lindsey Holland, and Alex (Sandy) Pentland. 2008. Meeting Mediator: Enhancing Group Collaboration with Sociometric Feedback. In CHI '08 Extended Abstracts on Human Factors in Computing Systems (CHI EA '08), 3183--3188. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Ann King and Ruth B. Hoppe. 2013. "Best practice" for patient-centered communication: a narrative review. Journal of Graduate Medical Education 5, 3: 385--393.Google ScholarGoogle ScholarCross RefCross Ref
  42. Stuart T. Klapp, Martin D. Hill, John G. Tyler, Zeke E. Martin, Richard J. Jagacinski, and Mari Riess Jones. 1985. On marching to two different drummers: perceptual aspects of the difficulties. Journal of Experimental Psychology: Human Perception and Performance 11, 6: 814.Google ScholarGoogle ScholarCross RefCross Ref
  43. Olga Kulyk, Jimmy Wang, and Jacques Terken. 2005. Real-time feedback on nonverbal behaviour to enhance social dynamics in small group meetings. In International Workshop on Machine Learning for Multimodal Interaction, 150--161. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Claudia Kuster, Tiberiu Popa, Jean-Charles Bazin, Craig Gotsman, and Markus Gross. 2012. Gaze correction for home video conferencing. ACM Transactions on Graphics (TOG)31, 6: 174. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Wendy Levinson, Debra L. Roter, John P. Mullooly, Valerie T. Dull, and Richard M. Frankel. 1997. Physician-patient communication. The relationship with malpractice claims among primary care physicians and surgeons. Journal of the American Medical Association 277, 7: 553--559.Google ScholarGoogle ScholarCross RefCross Ref
  46. Chunfeng Liu, Renee L. Lim, Kathryn L. McCabe, Silas Taylor, and Rafael A. Calvo. 2016. A Web-Based Telehealth Training Platform Incorporating Automated Nonverbal Behavior Feedback for Teaching Communication Skills to Medical Students: A Randomized Crossover Study. Journal of Medical Internet Research 18, 9.Google ScholarGoogle ScholarCross RefCross Ref
  47. Xiao Liu, Yoshie Sawada, Takako Takizawa, Hiroko Sato, Mahito Sato, Hironosuke Sakamoto, Toshihiro Utsugi, Kunio Sato, Hiroyuki Sumino, Shinichi Okamura, and others. 2007. Doctor-Patient Communication: A Comparison between Telemedicine Consultation and Face-to-Face Consultation. Internal Medicine 46, 5: 227--232.Google ScholarGoogle ScholarCross RefCross Ref
  48. Marcial Losada, Pedro Sanchez, and Elizabeth E. Noble. 1990. Collaborative Technology and Group Process Feedback: Their Impact on Interactive Sequences in Meetings. In Proceedings of the 1990 ACM Conference on Computer-supported Cooperative Work (CSCW '90), 53--64. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Ludmila Marcinowicz, Jerzy Konstantynowicz, and Cezary Godlewski. 2010. Patients' perceptions of GP non-verbal communication: a qualitative study. British Journal General Practice 60, 571: 83--87.Google ScholarGoogle ScholarCross RefCross Ref
  50. Jennifer Marlow, Eveline van Everdingen, and Daniel Avrahami. 2016. Taking Notes or Playing Games?: Understanding Multitasking in Video Communication. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16), 1726--1737. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Marianne Schmid Mast. 2007. On the importance of nonverbal communication in the physician--patient interaction. Patient Education and Counseling 67, 3: 315--318.Google ScholarGoogle ScholarCross RefCross Ref
  52. Paul McLaren, Chris J. Ball, A. B. Summerfield, M, J. P. Watson, and M. Lipsedge. 1995. An evaluation of the use of interactive television in an acute psychiatric service. Journal of Telemedicine and Telecare 1, 2: 79--85.Google ScholarGoogle ScholarCross RefCross Ref
  53. Edward Alan Miller. 2003. The technical and interpersonal aspects of telemedicine: effects on doctor-patient communication. Journal of Telemedicine and Telecare 9, 1: 1--7.Google ScholarGoogle Scholar
  54. Martin Montgomery. 2013. An introduction to language and society. Routledge.Google ScholarGoogle Scholar
  55. L. M. Ong, J. C. de Haes, A. M. Hoos, and F. B. Lammes. 1995. Doctor-patient communication: a review of the literature. Social Science & Medicine (1982) 40, 7: 903.Google ScholarGoogle Scholar
  56. Michael Quinn Patton. 1990. Qualitative evaluation and research methods. SAGE Publications, Inc.Google ScholarGoogle Scholar
  57. Alex Pentland and Tracy Heibeck. 2010. Honest signals: how they shape our world.MIT press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Debra L. Roter, Richard M. Frankel, Judith A. Hall, and David Sluyter. 2006. The expression of emotion through nonverbal behavior in medical visits. Journal of General Internal Medicine 21, S1: S28--S34.Google ScholarGoogle ScholarCross RefCross Ref
  59. Debra Roter and Susan Larson. 2002. The Roter interaction analysis system (RIAS): utility and flexibility for analysis of medical interactions. Patient Education and Counseling 46, 4: 243--251.Google ScholarGoogle ScholarCross RefCross Ref
  60. Allison Sauppé and Bilge Mutlu. 2014. How Social Cues Shape Task Coordination and Communication. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '14), 97--108. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Abigail J. Sellen. 1992. Speech patterns in video-mediated conversations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 49--59. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Richard L. Street Jr, E. Jay Wheeler, and William T. McCaughan. 2000. Specialist--primary care provider--patient communication in telemedical consultations. Telemedicine Journal 6, 1: 45--54.Google ScholarGoogle ScholarCross RefCross Ref
  63. Richard Thaler and Cass Sunstein. 2008. Nudge: Improving decisions about health, wealth, and happiness. New Haven, CT: Yale University Press.Google ScholarGoogle Scholar
  64. Donald T. Tepper and Richard F. Haase. 1978. Verbal and nonverbal communication of facilitative conditions. Journal of Counseling Psychology 25, 1: 35.Google ScholarGoogle ScholarCross RefCross Ref
  65. Jacques Terken and Janienke Sturm. 2010. Multimodal support for social dynamics in co-located meetings. Personal and Ubiquitous Computing 14, 8: 703--714. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. American Telehealth Association. 2009. Evidence-Based Practice for Telemental Health.http://dev.americantelemed.org/docs/default-source/standards/evidence-based-practice-for-telemental-health.pdfGoogle ScholarGoogle Scholar
  67. American Telehealth Association. 2017. Executive Leadership Survey. http://thesource.americantelemed.org/resources/telemedicine-executive-leadership-surveyGoogle ScholarGoogle Scholar
  68. OSHA.https://www.osha.gov/SLTC/etools/computerworkstations/components_monitors.htmlGoogle ScholarGoogle Scholar
  69. VisageSDK. http://visagetechnologies.com/products-and-services/visagesdk/Google ScholarGoogle Scholar
  70. Hark.js. https://github.com/otalk/hark/Google ScholarGoogle Scholar

Index Terms

  1. I Should Listen More: Real-time Sensing and Feedback of Non-Verbal Communication in Video Telehealth

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!