skip to main content
research-article
Public Access

Sochiatrist: Signals of Affect in Messaging Data

Authors Info & Claims
Published:15 October 2020Publication History
Skip Abstract Section

Abstract

Messaging is a common mode of communication, with conversations written informally between individuals. Interpreting emotional affect from messaging data can lead to a powerful form of reflection or act as a support for clinical therapy. Existing analysis techniques for social media commonly use LIWC and VADER for automated sentiment estimation. We correlate LIWC, VADER, and ratings from human reviewers with affect scores from 25 participants. We explore differences in how and when each technique is successful. Results show that human review does better than VADER, the best automated technique, when humans are judging positive affect ($r_s=0.45$ correlation when confident, $r_s=0.30$ overall). Surprisingly, human reviewers only do slightly better than VADER when judging negative affect ($r_s=0.38$ correlation when confident, $r_s=0.29$ overall). Compared to prior literature, VADER correlates more closely with PANAS scores for private messaging than public social media. Our results indicate that while any technique that serves as a proxy for PANAS scores has moderate correlation at best, there are some areas to improve the automated techniques by better considering context and timing in conversations.

References

  1. Marije aan het Rot, Koen Hogenelst, and Robert A. Schoevers. 2012. Mood disorders in everyday life: A systematic review of experience sampling and ecological momentary assessment studies. Clinical Psychology Review 32, 6 (2012), 510 -- 523. https://doi.org/10.1016/j.cpr.2012.05.007Google ScholarGoogle ScholarCross RefCross Ref
  2. Monica Anderson and Jingjing Jiang. 2018. Teens, Social Media & Technology 2018. https://www.pewresearch.org/ internet/2018/05/31/teens-social-media-technology-2018/Google ScholarGoogle Scholar
  3. Michael F Armey, Janis H Crowther, and Ivan W Miller. 2011. Changes in ecological momentary assessment reported affect associated with episodes of nonsuicidal self-injury. Behavior Therapy 42, 4 (2011), 579--588.Google ScholarGoogle ScholarCross RefCross Ref
  4. Natalya N Bazarova and Yoon Hyung Choi. 2014. Self-disclosure in social media: Extending the functional approach to disclosure motivations and characteristics on social network sites. Journal of Communication 64, 4 (2014), 635--657.Google ScholarGoogle ScholarCross RefCross Ref
  5. Natalya N Bazarova, Jessie G Taft, Yoon Hyung Choi, and Dan Cosley. 2013. Managing impressions and relationships on Facebook: Self-presentational and relational concerns revealed through the analysis of language style. Journal of Language and Social Psychology 32, 2 (2013), 121--141.Google ScholarGoogle ScholarCross RefCross Ref
  6. Asaf Beasley and Winter Mason. 2015. Emotional States vs. Emotional Words in Social Media. In Proceedings of the ACM Web Science Conference. ACM Press, Oxford, United Kingdom, 1--10. https://doi.org/10.1145/2786451.2786473Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Asaf Beasley, Winter Mason, and Eliot Smith. 2016. Inferring emotions and self-relevant domains in social media: Challenges and future directions. Translational Issues in Psychological Science 2, 3 (2016), 238--247. https://doi.org/10. 1037/tps0000086Google ScholarGoogle ScholarCross RefCross Ref
  8. Alex Cohen, Kyle Minor, Lauren Baillie, and Amanda Dahir. 2008. Clarifying the Linguistic Signature: Measuring Personality From Natural Speech. Journal of personality assessment 90 (12 2008), 559--63. https://doi.org/10.1080/ 00223890802388459Google ScholarGoogle Scholar
  9. Andrew Coles. 2012. iPhone backup database hashes: which filenames do they use? iPhone Backup Extractor: Recover Your Lost Data (2012). https://www.iphonebackupextractor.com/blog/iphone-backup-location-all-files-extension/Google ScholarGoogle Scholar
  10. Lorenzo Coviello, Yunkyu Sohn, Adam D.I. Kramer, Cameron Marlow, Massimo Franceschetti, Nicholas A. Christakis, and James H. Fowler. 2014. Detecting emotional contagion in massive social networks. PLoS One 9, 3 (12 3 2014). https://doi.org/10.1371/journal.pone.0090315Google ScholarGoogle Scholar
  11. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Enhanced Sentiment Learning Using Twitter Hashtags and Smileys. In Coling 2010: Posters. Coling 2010 Organizing Committee, Beijing, China, 241--249. https://www.aclweb.org/ anthology/C10--2028Google ScholarGoogle Scholar
  12. Munmun De Choudhury and Scott Counts. 2012. The nature of emotional expression in social media: measurement, inference and utility. Human Computer Interaction Consortium (HCIC) (2012).Google ScholarGoogle Scholar
  13. Munmun De Choudhury, Michael Gamon, and Scott Counts. 2012. Happy, nervous or surprised? classification of human affective states in social media. In Sixth International AAAI Conference on Weblogs and Social Media. 435--438.Google ScholarGoogle Scholar
  14. Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting Depression via Social Media. AAAI, 128--137.Google ScholarGoogle Scholar
  15. Peter Sheridan Dodds, Eric M. Clark, Suma Desu, Morgan R. Frank, Andrew J. Reagan, Jake Ryland Williams, Lewis Mitchell, Kameron Decker Harris, Isabel M. Kloumann, James P. Bagrow, Karine Megerdoomian, Matthew T. McMahon, Brian F. Tivnan, and Christopher M. Danforth. 2015. Human language reveals a universal positivity bias. Proceedings of the National Academy of Sciences 112, 8 (Feb. 2015), 2389--2394. https://doi.org/10.1073/pnas.1411678112Google ScholarGoogle ScholarCross RefCross Ref
  16. Fabon Dzogang, Stafford Lightman, and Nello Cristianini. 2017. Circadian mood variations in Twitter content. Brain and Neuroscience Advances 1 (Jan. 2017), 239821281774450. https://doi.org/10.1177/2398212817744501Google ScholarGoogle Scholar
  17. Carl E Fisher and Paul S Appelbaum. 2017. Beyond Googling: The Ethics of Using Patients? Electronic Footprints in Psychiatric Practice. Harvard Review of Psychiatry 25, 4 (2017), 170--179. Proc. ACM Hum.-Comput. Interact., Vol. 4, No. CSCW2, Article 111. Publication date: October 2020. 111:24 Talie Massachi et al.Google ScholarGoogle ScholarCross RefCross Ref
  18. Andrew J. Flanagin. 2005. IM Online: Instant Messaging Use Among College Students. Communication Research Reports 22, 3 (2005), 175--187. https://doi.org/10.1080/00036810500206966Google ScholarGoogle ScholarCross RefCross Ref
  19. Sharath Chandra Guntuku, Anneke Buffone, Kokil Jaidka, Johannes C Eichstaedt, and Lyle H Ungar. 2019. Understanding and measuring psychological stress using social media. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 13. 214--225.Google ScholarGoogle Scholar
  20. C.J. Hutto and Eric Gilbert. 2015. VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Proceedings of the 8th International Conference on Weblogs and Social Media, ICWSM 2014, 216--225.Google ScholarGoogle Scholar
  21. Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent Twitter Sentiment Classification. ACL, 151--160.Google ScholarGoogle Scholar
  22. Funda Kivran-Swaine and Mor Naaman. 2011. Network properties and social sharing of emotions in social awareness streams. In Proceedings of the ACM 2011 Conference on Computer supported cooperative work. 379--382.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Adam D.I. Kramer. 2010. An Unobtrusive Behavioral Model of 'Gross National Happiness'. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '10). Association for Computing Machinery, New York, NY, USA, 287--290. https://doi.org/10.1145/1753326.1753369Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Adam D.I. Kramer. 2012. The Spread of Emotion via Facebook. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI '12). Association for Computing Machinery, New York, NY, USA, 767--770. https://doi.org/10.1145/2207676.2207787Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Adam DI Kramer, Jamie E Guillory, and Jeffrey T Hancock. 2014. Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences 111, 24 (2014), 8788--8790.Google ScholarGoogle ScholarCross RefCross Ref
  26. Nicole Krämer and Stephan Winter. 2008. Impression Management 2.0: The Relationship of Self-Esteem, Extraversion, Self-Efficacy, and Self-Presentation Within Social Networking Sites. Journal of Media Psychology: Theories, Methods, and Applications 20 (01 2008), 106--. https://doi.org/10.1027/1864--1105.20.3.106Google ScholarGoogle Scholar
  27. Danielle Lottridge and Frank R Bentley. 2018. Let's Hate Together: How People Share News in Messaging, Social, and Public Networks. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--13.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Fiona Lyddy, Francesca Farina, James Hanney, Lynn Farrell, and Niamh Kelly O'Neill. 2014. An Analysis of Language in University Students? Text Messages. Journal of Computer-Mediated Communication 19, 3 (2014), 546--561. https: //doi.org/10.1111/jcc4.12045 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/jcc4.12045Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Yuanchao Ma, Bin Xu, Yin Bai, Guodong Sun, and Run Zhu. 2012. Daily mood assessment based on mobile phone sensing. In Wearable and implantable body sensor networks (BSN), 2012 ninth international conference on. IEEE, 142--147.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. David N. Miller. 2011. Positive Affect. Springer US, Boston, MA, 1121--1122. https://doi.org/10.1007/978-0--387--79061- 9_2193Google ScholarGoogle Scholar
  31. Margaret Mitchell, Kristy Hollingshead, and Glen Coppersmith. 2015. Quantifying the Language of Schizophrenia in Social Media. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. Association for Computational Linguistics, Denver, Colorado, 11--20. https: //doi.org/10.3115/v1/W15--1202Google ScholarGoogle ScholarCross RefCross Ref
  32. JA Naslund, KA Aschbrenner, LA Marsch, and SJ Bartels. 2016. The future of mental health care: peer-to-peer support and social media. Epidemiology and Psychiatric Sciences 25, 2 (2016), 113--122.Google ScholarGoogle ScholarCross RefCross Ref
  33. Kathleen O'Leary, Stephen M. Schueller, Jacob O. Wobbrock, and Wanda Pratt. 2018. 'Suddenly, We Got to Become Therapists for Each Other': Designing Peer Support Chats for Mental Health. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Article 331, 14 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Kenneth M Ovens and Gordon Morison. 2016. Forensic analysis of kik messenger on ios devices. Digital Investigation 17 (2016), 40--52.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Myoungouk Park, D.W. McDonald, and M. Cha. 2013. Perception differences between the depressed and non-depressed users in Twitter. Proceedings of the 7th International Conference on Weblogs and Social Media, ICWSM 2013 (01 2013), 476--485.Google ScholarGoogle Scholar
  36. Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. Transactions of the Association for Computational Linguistics 4 (2016), 61--74.Google ScholarGoogle Scholar
  37. James Pennebaker, Martha Francis, and Roger Booth. 1999. Linguistic inquiry and word count (LIWC). (01 1999).Google ScholarGoogle Scholar
  38. James W. Pennebaker, Ryan L. Boyd, Kayla Jordan, and Kate Blackburn. 2015. The Development and Psychometric Properties of LIWC2015. (2015). https://doi.org/10.15781/T29G6ZGoogle ScholarGoogle Scholar
  39. Megan Ranney, Kyler Lehrbach, Nicholas Scott, Nicole Nugent, Alison Riese, Jeff Huang, Grant Fong, and Rochelle Rosen. 2020. Insights into Adolescent Online Conflict through Qualitative Analysis of Online Messages. Proceedings of the 53rd Hawaii International Conference on System Sciences, 9. https://scholarspace.manoa.hawaii.edu/bitstream/ 10125/64209/0378.pdfGoogle ScholarGoogle ScholarCross RefCross Ref
  40. Andrew G. Reece and Christopher M. Danforth. 2017. Instagram photos reveal predictive markers of depression. EPJ Data Science 6, 1 (Dec. 2017), 1--12. https://doi.org/10.1140/epjds/s13688-017-0110-z Proc. ACM Hum.-Comput. Interact., Vol. 4, No. CSCW2, Article 111. Publication date: October 2020. Sochiatrist: Signals of Affect in Messaging Data 111:25Google ScholarGoogle Scholar
  41. Andrew G. Reece, Andrew J. Reagan, Katharina L. M. Lix, Peter Sheridan Dodds, Christopher M. Danforth, and Ellen J. Langer. 2017. Forecasting the onset and course of mental illness with Twitter data. Scientific Reports 7, 1 (Dec. 2017), 13006. https://doi.org/10.1038/s41598-017--12961--9Google ScholarGoogle Scholar
  42. Koustuv Saha, Larry Chan, Kaya De Barbaro, Gregory D Abowd, and Munmun De Choudhury. 2017. Inferring Mood Instability on Social Media by Leveraging Ecological Momentary Assessments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 95.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Sandra Servia-Rodríguez, Kiran K Rachuri, Cecilia Mascolo, Peter J Rentfrow, Neal Lathia, and Gillian M Sandstrom. 2017. Mobile sensing at the service of mental well-being: a large-scale longitudinal study. In Proceedings of the 26th International Conference on World Wide Web. 103--112.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Saul Shiffman, Arthur A. Stone, and Michael R. Hufford. 2008. Ecological Momentary Assessment. Annual Review of Clinical Psychology 4, 1 (2008), 1--32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415 arXiv:https://doi.org/10.1146/annurev.clinpsy.3.022806.091415 PMID: 18509902.Google ScholarGoogle Scholar
  45. Saul Shiffman, Arthur A Stone, and Michael R Hufford. 2008. Ecological momentary assessment. Annu. Rev. Clin. Psychol. 4 (2008), 1--32.Google ScholarGoogle ScholarCross RefCross Ref
  46. Aaron Smith and Monica Anderson. 2018. Social Media Use 2018: Demographics and Statistics. https://www. pewresearch.org/internet/2018/03/01/social-media-use-in-2018/Google ScholarGoogle Scholar
  47. Deborah M. Stringer. 2013. Negative Affect. Springer New York, New York, NY, 1303--1304. https://doi.org/10.1007/978- 1--4419--1005--9_606Google ScholarGoogle Scholar
  48. Yla R. Tausczik and James W. Pennebaker. 2010. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. Journal of Language and Social Psychology 29, 1 (March 2010), 24--54. https://doi.org/10.1177/ 0261927X09351676Google ScholarGoogle Scholar
  49. Auke Tellegen. 1985. Structures of mood and personality and their relevance to assessing anxiety, with an emphasis on self-report. (1985), 681--706.Google ScholarGoogle Scholar
  50. William Tov, Kok Leong Ng, Han Lin, and Lin Ge Qiu. 2013. Detecting well-being via computerized content analysis of brief diary entries. Psychological assessment 25 4 (2013), 1069--78.Google ScholarGoogle Scholar
  51. Timothy J Trull, Marika B Solhan, Sarah L Tragesser, Seungmin Jahng, Phillip K Wood, Thomas M Piasecki, and David Watson. 2008. Affective instability: measuring a core feature of borderline personality disorder with ecological momentary assessment. Journal of Abnormal Psychology 117, 3 (2008), 647.Google ScholarGoogle ScholarCross RefCross Ref
  52. Wei Wang, Ivan Hernandez, Daniel Newman, Jibo He, and Jiang Bian. 2016. Twitter Analysis: Studying US Weekly Trends in Work Stress and Emotion. Applied Psychology 65 (04 2016), 355--378. https://doi.org/10.1111/apps.12065Google ScholarGoogle Scholar
  53. David Watson, Lee Anna Clark, and Auke Tellegen. 1988. Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of Personality and Social Psychology 54, 6 (1988), 1063--1070.Google ScholarGoogle Scholar
  54. Yong Sook Yang, Gi Wook Ryu, and Mona Choi. 2019. Methodological strategies for ecological momentary assessment to evaluate mood and stress in adult patients using mobile phones: Systematic review. JMIR mHealth and uHealth 7, 4 (1 1 2019). https://doi.org/10.2196/11215 RGoogle ScholarGoogle Scholar

Index Terms

  1. Sochiatrist: Signals of Affect in Messaging Data

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image Proceedings of the ACM on Human-Computer Interaction
          Proceedings of the ACM on Human-Computer Interaction  Volume 4, Issue CSCW2
          CSCW
          October 2020
          2310 pages
          EISSN:2573-0142
          DOI:10.1145/3430143
          Issue’s Table of Contents

          Copyright © 2020 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 15 October 2020
          Published in pacmhci Volume 4, Issue CSCW2

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!