Abstract

Video telehealth is growing to allow more clinicians to see patients from afar. As a result, clinicians, typically trained for in-person visits, must learn to communicate both health information and non-verbal affective signals to patients through a digital medium. We introduce a system called ReflectLive that senses and provides real-time feedback about non-verbal communication behaviors to clinicians so they can improve their communication behaviors. A user evaluation with 10 clinicians showed that the real-time feedback helped clinicians maintain better eye contact with patients and was not overly distracting. Clinicians reported being more aware of their non-verbal communication behaviors and reacted positively to summaries of their conversational metrics, motivating them to want to improve. Using ReflectLive as a probe, we also discuss the benefits and concerns around automatically quantifying the "soft skills" and complexities of clinician-patient communication, the controllability of behaviors, and the design considerations for how to present real-time and summative feedback to clinicians.
- Hiroyuki Adachi, Seiko Myojin, and Nobutaka Shimada. 2015. ScoringTalk: a tablet system scoring and visualizing conversation for balancing of participation. In SIGGRAPH Asia 2015 Mobile Graphics and Interactive Applications, 9. Google Scholar
Digital Library
- Zia Agha, Debra L. Roter, and Ralph M. Schapira. 2009. An Evaluation of Patient-Physician Communication Style During Telemedicine Consultations. Journal of Medical Internet Research 11, 3: e36.Google Scholar
Cross Ref
- Mara S. Aruguete and Carlos A. Roberts. 2000. Gender, Affiliation, and Control in Physician--Patient Encounters. Sex Roles 42, 1--2: 107--118.Google Scholar
Cross Ref
- Madeline Balaam, Geraldine Fitzpatrick, Judith Good, and Eric Harris. 2011. Enhancing Interactional Synchrony with an Ambient Display. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 867--876. Google Scholar
Digital Library
- Rashid L. Bashshur. 1997. Critical issues in telemedicine. Telemedicine Journal 3, 2: 113--126.Google Scholar
Cross Ref
- Rashid L. Bashshur, Jay H. Sanders, and Gary W. Shannon. 1997. Telemedicine: Theory and practice. Charles C. Thomas Springfield, IL.Google Scholar
- Rainer S. Beck, Rebecca Daughtridge, and Philip D. Sloane. 2002. Physician-patient communication in the primary care office: a systematic review. The Journal of the American Board of Family Practice 15, 1: 25--38.Google Scholar
- Jozien M. Bensing, Jan J. Kerssens, and Marja van der Pasch. 1995. Patient-directed gaze as a tool for discovering and handling psychosocial problems in general practice. Journal of nonverbal behavior 19, 4: 223--242.Google Scholar
Cross Ref
- Tony Bergstrom and Karrie Karahalios. 2007. Conversation Clock: Visualizing audio patterns in co-located groups. In Proceedings of the 40th Annual Hawaii International Conference on Social Systems (HICSS 2007),78--78. Google Scholar
Digital Library
- J. Beullens, J. J. Rethans, Jo Goedhuys, and Frank Buntinx. 1997. The use of standardized patients in research in general practice. Family Practice 14, 1: 58--62.Google Scholar
Cross Ref
- LouAnne E. Boyd, Alejandro Rangel, Helen Tomimbang, Andrea Conejo-Toledo, Kanika Patel, Monica Tentori, and Gillian R. Hayes. 2016. SayWAT: Augmenting Face-to-Face Conversations for Adults with Autism. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 4872--4883. Google Scholar
Digital Library
- Jed R. Brubaker, Gina Venolia, and John C. Tang. 2012. Focusing on shared experiences: moving beyond the camera in video communication. In Proceedings of the Designing Interactive Systems Conference, 96--105. Google Scholar
Digital Library
- Patrick S. Byrne and B. E. Long. 1976. Doctors talking to patients. A study of the verbal behaviour of general practitioners consulting in their surgeries. Exeter {England}: Royal College of General Practitioners, 1984, c1976.Google Scholar
- Byungki Byun, Anurag Awasthi, Philip A. Chou, Ashish Kapoor, Bongshin Lee, and Mary Czerwinski. 2011. Honest signals in video conferencing. In Proceedings of 2011 IEEE International Conference on Multimedia and Expo (ICME), 1--6. Google Scholar
Digital Library
- Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 981--992. Google Scholar
Digital Library
- Scott Carter, Laurent Denoue, and Matthew Cooper. 2015. Searching and Browsing Live, Web-based Meetings. In Proceedings of the 23rd ACM International Conference on Multimedia (MM '15), 791--792. Google Scholar
Digital Library
- Charles S. Carver and Michael F. Scheier. 2012. Attention and self-regulation: A control-theory approach to human behavior.Springer Science & Business Media.Google Scholar
- Kathryn Charmaz. 2006. Constructing Grounded Theory: A Practical Guide to Qualitative Analysis. Sage Publications Ltd.Google Scholar
- Milton Chen. 2002. Leveraging the Asymmetric Sensitivity of Eye Contact for Videoconference. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02), 49--56. Google Scholar
Digital Library
- Yunan Chen, Victor Ngo, Sidney Harrison, and Victoria Duong. 2011. Unpacking exam-room computing: negotiating computer-use in patient-physician interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 3343--3352. Google Scholar
Digital Library
- Ionut Damian, Chiew Seng (Sean) Tan, Tobias Baur, Johannes Schöning, Kris Luyten, and Elisabeth André. 2015. Augmenting Social Interactions: Realtime Behavioural Feedback Using Social Signal Processing Techniques. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 565--574. Google Scholar
Digital Library
- Epstein Becker Green. 2016. 50-State Survey of Telemental/Telebehavioral Health.Google Scholar
- Joan Morris DiMicco, Katherine J. Hollenbach, Anna Pandolfo, and Walter Bender. 2007. The impact of increased awareness while face-to-face. Human--Computer Interaction 22, 1--2: 47--96. Google Scholar
Digital Library
- Paul Dourish, Annette Adler, Victoria Bellotti, and Austin Henderson. 1996. Your place or mine? Learning from long-term use of audio-video communication. In Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work (CSCW '96)5, 1: 33--62. Google Scholar
Digital Library
- F. Daniel Duffy, Geoffrey H. Gordon, Gerald Whelan, Kathy Cole-Kelly, Richard Frankel, and others. 2004. Assessing competence in communication and interpersonal skills: the Kalamazoo II report. Academic Medicine 79, 6: 495--507.Google Scholar
Cross Ref
- Susan T. Fiske. 1993. Controlling other people: The impact of power on stereotyping. American Psychologist 48, 6: 621--628.Google Scholar
Cross Ref
- Howard S. Friedman. 1979. Nonverbal communication between patients and medical practitioners. Journal of Social Issues 35, 1: 82--99.Google Scholar
Cross Ref
- Chris Fullwood and Gwyneth Doherty-Sneddon. 2006. Effect of gazing at the camera during a video link on recall. Applied Ergonomics 37, 2: 167--175.Google Scholar
Cross Ref
- Dominik Giger, Jean-Charles Bazin, Claudia Kuster, Tiberiu Popa, and Markus Gross. 2014. Gaze correction with a single webcam. In Multimedia and Expo (ICME), 2014 IEEE International Conference on, 1--6.Google Scholar
Cross Ref
- Dehra Glueck, K. Myers, and C. L. Turvey. 2013. Establishing therapeutic rapport in telemental health. Telemental health: Clinical, technical and administrative foundations for evidence-based practice: 29--46.Google Scholar
- Andrea. L. Hartzler, Rupa A. Patel, Mary Czerwinski, Wanda Pratt, Asta Roseway, N. Chandrasekaran, and Anthony Back. 2014. Real-time feedback on nonverbal clinical communication. Methods of Information in Medicine 53, 5: 389--405.Google Scholar
Cross Ref
- Helen Ai He, Naomi Yamashita, Ari Hautasaari, Xun Cao, and Elaine M. Huang. 2017. Why Did They Do That?: Exploring Attribution Mismatches Between Native and Non-Native Speakers Using Videoconferencing. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), 297--309. Google Scholar
Digital Library
- Stephen G. Henry, Andrea Fuhrel-Forbis, Mary A. M. Rogers, and Susan Eggly. 2012. Association between nonverbal communication during clinical interactions and outcomes: A systematic review and meta-analysis. Patient Education and Counseling 86, 3: 297--315.Google Scholar
Cross Ref
- John Heritage and Douglas W. Maynard. 2006. Communication in medical care: Interaction between primary care physicians and patients. Cambridge University Press.Google Scholar
- Clara E. Hill and James Gormally. 1977. Effects of reflection, restatement, probe, and nonverbal behaviors on client affect. Journal of Counseling Psychology 24, 2: 92.Google Scholar
Cross Ref
- Mohammadreza Hojat, Joseph S. Gonnella, Thomas J. Nasca, Salvatore Mangione, Michael Vergare, and Michael Magee. 2002. Physician empathy: definition, components, measurement, and relationship to gender and specialty. American Journal of Psychiatry 159, 9: 1563--1569.Google Scholar
Cross Ref
- Ellen A. Isaacs and John C. Tang. 1994. What video can and cannot do for collaboration: a case study. Multimedia Systems 2, 2: 63--73. Google Scholar
Digital Library
- Matthew Jackson, Anne H. Anderson, Rachel McEwan, and Jim Mullin. 2000. Impact of Video Frame Rate on Communicative Behaviour in Two and Four Party Groups. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW '00), 11--20. Google Scholar
Digital Library
- Donald J. Kiesler and Stephen M. Auerbach. 2003. Integrating measurement of control and affiliation in studies of physician-patient interaction: the interpersonal circumplex. Social Science & Medicine (1982) 57, 9: 1707--1722.Google Scholar
- Taemie Kim, Agnes Chang, Lindsey Holland, and Alex (Sandy) Pentland. 2008. Meeting Mediator: Enhancing Group Collaboration with Sociometric Feedback. In CHI '08 Extended Abstracts on Human Factors in Computing Systems (CHI EA '08), 3183--3188. Google Scholar
Digital Library
- Ann King and Ruth B. Hoppe. 2013. "Best practice" for patient-centered communication: a narrative review. Journal of Graduate Medical Education 5, 3: 385--393.Google Scholar
Cross Ref
- Stuart T. Klapp, Martin D. Hill, John G. Tyler, Zeke E. Martin, Richard J. Jagacinski, and Mari Riess Jones. 1985. On marching to two different drummers: perceptual aspects of the difficulties. Journal of Experimental Psychology: Human Perception and Performance 11, 6: 814.Google Scholar
Cross Ref
- Olga Kulyk, Jimmy Wang, and Jacques Terken. 2005. Real-time feedback on nonverbal behaviour to enhance social dynamics in small group meetings. In International Workshop on Machine Learning for Multimodal Interaction, 150--161. Google Scholar
Digital Library
- Claudia Kuster, Tiberiu Popa, Jean-Charles Bazin, Craig Gotsman, and Markus Gross. 2012. Gaze correction for home video conferencing. ACM Transactions on Graphics (TOG)31, 6: 174. Google Scholar
Digital Library
- Wendy Levinson, Debra L. Roter, John P. Mullooly, Valerie T. Dull, and Richard M. Frankel. 1997. Physician-patient communication. The relationship with malpractice claims among primary care physicians and surgeons. Journal of the American Medical Association 277, 7: 553--559.Google Scholar
Cross Ref
- Chunfeng Liu, Renee L. Lim, Kathryn L. McCabe, Silas Taylor, and Rafael A. Calvo. 2016. A Web-Based Telehealth Training Platform Incorporating Automated Nonverbal Behavior Feedback for Teaching Communication Skills to Medical Students: A Randomized Crossover Study. Journal of Medical Internet Research 18, 9.Google Scholar
Cross Ref
- Xiao Liu, Yoshie Sawada, Takako Takizawa, Hiroko Sato, Mahito Sato, Hironosuke Sakamoto, Toshihiro Utsugi, Kunio Sato, Hiroyuki Sumino, Shinichi Okamura, and others. 2007. Doctor-Patient Communication: A Comparison between Telemedicine Consultation and Face-to-Face Consultation. Internal Medicine 46, 5: 227--232.Google Scholar
Cross Ref
- Marcial Losada, Pedro Sanchez, and Elizabeth E. Noble. 1990. Collaborative Technology and Group Process Feedback: Their Impact on Interactive Sequences in Meetings. In Proceedings of the 1990 ACM Conference on Computer-supported Cooperative Work (CSCW '90), 53--64. Google Scholar
Digital Library
- Ludmila Marcinowicz, Jerzy Konstantynowicz, and Cezary Godlewski. 2010. Patients' perceptions of GP non-verbal communication: a qualitative study. British Journal General Practice 60, 571: 83--87.Google Scholar
Cross Ref
- Jennifer Marlow, Eveline van Everdingen, and Daniel Avrahami. 2016. Taking Notes or Playing Games?: Understanding Multitasking in Video Communication. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16), 1726--1737. Google Scholar
Digital Library
- Marianne Schmid Mast. 2007. On the importance of nonverbal communication in the physician--patient interaction. Patient Education and Counseling 67, 3: 315--318.Google Scholar
Cross Ref
- Paul McLaren, Chris J. Ball, A. B. Summerfield, M, J. P. Watson, and M. Lipsedge. 1995. An evaluation of the use of interactive television in an acute psychiatric service. Journal of Telemedicine and Telecare 1, 2: 79--85.Google Scholar
Cross Ref
- Edward Alan Miller. 2003. The technical and interpersonal aspects of telemedicine: effects on doctor-patient communication. Journal of Telemedicine and Telecare 9, 1: 1--7.Google Scholar
- Martin Montgomery. 2013. An introduction to language and society. Routledge.Google Scholar
- L. M. Ong, J. C. de Haes, A. M. Hoos, and F. B. Lammes. 1995. Doctor-patient communication: a review of the literature. Social Science & Medicine (1982) 40, 7: 903.Google Scholar
- Michael Quinn Patton. 1990. Qualitative evaluation and research methods. SAGE Publications, Inc.Google Scholar
- Alex Pentland and Tracy Heibeck. 2010. Honest signals: how they shape our world.MIT press. Google Scholar
Digital Library
- Debra L. Roter, Richard M. Frankel, Judith A. Hall, and David Sluyter. 2006. The expression of emotion through nonverbal behavior in medical visits. Journal of General Internal Medicine 21, S1: S28--S34.Google Scholar
Cross Ref
- Debra Roter and Susan Larson. 2002. The Roter interaction analysis system (RIAS): utility and flexibility for analysis of medical interactions. Patient Education and Counseling 46, 4: 243--251.Google Scholar
Cross Ref
- Allison Sauppé and Bilge Mutlu. 2014. How Social Cues Shape Task Coordination and Communication. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '14), 97--108. Google Scholar
Digital Library
- Abigail J. Sellen. 1992. Speech patterns in video-mediated conversations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 49--59. Google Scholar
Digital Library
- Richard L. Street Jr, E. Jay Wheeler, and William T. McCaughan. 2000. Specialist--primary care provider--patient communication in telemedical consultations. Telemedicine Journal 6, 1: 45--54.Google Scholar
Cross Ref
- Richard Thaler and Cass Sunstein. 2008. Nudge: Improving decisions about health, wealth, and happiness. New Haven, CT: Yale University Press.Google Scholar
- Donald T. Tepper and Richard F. Haase. 1978. Verbal and nonverbal communication of facilitative conditions. Journal of Counseling Psychology 25, 1: 35.Google Scholar
Cross Ref
- Jacques Terken and Janienke Sturm. 2010. Multimodal support for social dynamics in co-located meetings. Personal and Ubiquitous Computing 14, 8: 703--714. Google Scholar
Digital Library
- American Telehealth Association. 2009. Evidence-Based Practice for Telemental Health.http://dev.americantelemed.org/docs/default-source/standards/evidence-based-practice-for-telemental-health.pdfGoogle Scholar
- American Telehealth Association. 2017. Executive Leadership Survey. http://thesource.americantelemed.org/resources/telemedicine-executive-leadership-surveyGoogle Scholar
- OSHA.https://www.osha.gov/SLTC/etools/computerworkstations/components_monitors.htmlGoogle Scholar
- VisageSDK. http://visagetechnologies.com/products-and-services/visagesdk/Google Scholar
- Hark.js. https://github.com/otalk/hark/Google Scholar
Index Terms
I Should Listen More: Real-time Sensing and Feedback of Non-Verbal Communication in Video Telehealth
Recommendations
Visual feedback on nonverbal communication: a design exploration with healthcare professionals
PervasiveHealth '13: Proceedings of the 7th International Conference on Pervasive Computing Technologies for HealthcareEffective nonverbal communication between patients and clinicians fosters both the delivery of empathic patient-centered care and positive patient outcomes. However, few efforts to develop tools for enhancing clinician communication have focused on ...
Leveraging visual feedback from social signal processing to enhance clinicians' nonverbal skills
CHI EA '13: CHI '13 Extended Abstracts on Human Factors in Computing SystemsNonverbal communication between patients and clinicians affects the delivery of empathic patient-centered care and patient outcomes. To be effective communicators, clinicians must appropriately encode, decode, and regulate nonverbal cues, such as speech ...
The Effect of Doctors-Patients' Verbal and Non verbal Communication on Patients' Satisfaction in Outpatient Installation of Mother and Child Centre RS Dr. Wahidin Sudirohusodo Makassar
ICHSM '18: Proceedings of the International Conference on Healthcare Service Management 2018The aim of the research was to determine the correlation between doctors-patients' verbal and non-verbal communication and patients' satisfaction in Outpatient Installation of Mother and Child Centre of DR. WAHIDIN SUDIROHUSODO MAKASSAR. The research ...






Comments