Abstract
Messaging is a common mode of communication, with conversations written informally between individuals. Interpreting emotional affect from messaging data can lead to a powerful form of reflection or act as a support for clinical therapy. Existing analysis techniques for social media commonly use LIWC and VADER for automated sentiment estimation. We correlate LIWC, VADER, and ratings from human reviewers with affect scores from 25 participants. We explore differences in how and when each technique is successful. Results show that human review does better than VADER, the best automated technique, when humans are judging positive affect ($r_s=0.45$ correlation when confident, $r_s=0.30$ overall). Surprisingly, human reviewers only do slightly better than VADER when judging negative affect ($r_s=0.38$ correlation when confident, $r_s=0.29$ overall). Compared to prior literature, VADER correlates more closely with PANAS scores for private messaging than public social media. Our results indicate that while any technique that serves as a proxy for PANAS scores has moderate correlation at best, there are some areas to improve the automated techniques by better considering context and timing in conversations.
- Marije aan het Rot, Koen Hogenelst, and Robert A. Schoevers. 2012. Mood disorders in everyday life: A systematic review of experience sampling and ecological momentary assessment studies. Clinical Psychology Review 32, 6 (2012), 510 -- 523. https://doi.org/10.1016/j.cpr.2012.05.007Google Scholar
Cross Ref
- Monica Anderson and Jingjing Jiang. 2018. Teens, Social Media & Technology 2018. https://www.pewresearch.org/ internet/2018/05/31/teens-social-media-technology-2018/Google Scholar
- Michael F Armey, Janis H Crowther, and Ivan W Miller. 2011. Changes in ecological momentary assessment reported affect associated with episodes of nonsuicidal self-injury. Behavior Therapy 42, 4 (2011), 579--588.Google Scholar
Cross Ref
- Natalya N Bazarova and Yoon Hyung Choi. 2014. Self-disclosure in social media: Extending the functional approach to disclosure motivations and characteristics on social network sites. Journal of Communication 64, 4 (2014), 635--657.Google Scholar
Cross Ref
- Natalya N Bazarova, Jessie G Taft, Yoon Hyung Choi, and Dan Cosley. 2013. Managing impressions and relationships on Facebook: Self-presentational and relational concerns revealed through the analysis of language style. Journal of Language and Social Psychology 32, 2 (2013), 121--141.Google Scholar
Cross Ref
- Asaf Beasley and Winter Mason. 2015. Emotional States vs. Emotional Words in Social Media. In Proceedings of the ACM Web Science Conference. ACM Press, Oxford, United Kingdom, 1--10. https://doi.org/10.1145/2786451.2786473Google Scholar
Digital Library
- Asaf Beasley, Winter Mason, and Eliot Smith. 2016. Inferring emotions and self-relevant domains in social media: Challenges and future directions. Translational Issues in Psychological Science 2, 3 (2016), 238--247. https://doi.org/10. 1037/tps0000086Google Scholar
Cross Ref
- Alex Cohen, Kyle Minor, Lauren Baillie, and Amanda Dahir. 2008. Clarifying the Linguistic Signature: Measuring Personality From Natural Speech. Journal of personality assessment 90 (12 2008), 559--63. https://doi.org/10.1080/ 00223890802388459Google Scholar
- Andrew Coles. 2012. iPhone backup database hashes: which filenames do they use? iPhone Backup Extractor: Recover Your Lost Data (2012). https://www.iphonebackupextractor.com/blog/iphone-backup-location-all-files-extension/Google Scholar
- Lorenzo Coviello, Yunkyu Sohn, Adam D.I. Kramer, Cameron Marlow, Massimo Franceschetti, Nicholas A. Christakis, and James H. Fowler. 2014. Detecting emotional contagion in massive social networks. PLoS One 9, 3 (12 3 2014). https://doi.org/10.1371/journal.pone.0090315Google Scholar
- Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Enhanced Sentiment Learning Using Twitter Hashtags and Smileys. In Coling 2010: Posters. Coling 2010 Organizing Committee, Beijing, China, 241--249. https://www.aclweb.org/ anthology/C10--2028Google Scholar
- Munmun De Choudhury and Scott Counts. 2012. The nature of emotional expression in social media: measurement, inference and utility. Human Computer Interaction Consortium (HCIC) (2012).Google Scholar
- Munmun De Choudhury, Michael Gamon, and Scott Counts. 2012. Happy, nervous or surprised? classification of human affective states in social media. In Sixth International AAAI Conference on Weblogs and Social Media. 435--438.Google Scholar
- Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting Depression via Social Media. AAAI, 128--137.Google Scholar
- Peter Sheridan Dodds, Eric M. Clark, Suma Desu, Morgan R. Frank, Andrew J. Reagan, Jake Ryland Williams, Lewis Mitchell, Kameron Decker Harris, Isabel M. Kloumann, James P. Bagrow, Karine Megerdoomian, Matthew T. McMahon, Brian F. Tivnan, and Christopher M. Danforth. 2015. Human language reveals a universal positivity bias. Proceedings of the National Academy of Sciences 112, 8 (Feb. 2015), 2389--2394. https://doi.org/10.1073/pnas.1411678112Google Scholar
Cross Ref
- Fabon Dzogang, Stafford Lightman, and Nello Cristianini. 2017. Circadian mood variations in Twitter content. Brain and Neuroscience Advances 1 (Jan. 2017), 239821281774450. https://doi.org/10.1177/2398212817744501Google Scholar
- Carl E Fisher and Paul S Appelbaum. 2017. Beyond Googling: The Ethics of Using Patients? Electronic Footprints in Psychiatric Practice. Harvard Review of Psychiatry 25, 4 (2017), 170--179. Proc. ACM Hum.-Comput. Interact., Vol. 4, No. CSCW2, Article 111. Publication date: October 2020. 111:24 Talie Massachi et al.Google Scholar
Cross Ref
- Andrew J. Flanagin. 2005. IM Online: Instant Messaging Use Among College Students. Communication Research Reports 22, 3 (2005), 175--187. https://doi.org/10.1080/00036810500206966Google Scholar
Cross Ref
- Sharath Chandra Guntuku, Anneke Buffone, Kokil Jaidka, Johannes C Eichstaedt, and Lyle H Ungar. 2019. Understanding and measuring psychological stress using social media. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 13. 214--225.Google Scholar
- C.J. Hutto and Eric Gilbert. 2015. VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Proceedings of the 8th International Conference on Weblogs and Social Media, ICWSM 2014, 216--225.Google Scholar
- Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent Twitter Sentiment Classification. ACL, 151--160.Google Scholar
- Funda Kivran-Swaine and Mor Naaman. 2011. Network properties and social sharing of emotions in social awareness streams. In Proceedings of the ACM 2011 Conference on Computer supported cooperative work. 379--382.Google Scholar
Digital Library
- Adam D.I. Kramer. 2010. An Unobtrusive Behavioral Model of 'Gross National Happiness'. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '10). Association for Computing Machinery, New York, NY, USA, 287--290. https://doi.org/10.1145/1753326.1753369Google Scholar
Digital Library
- Adam D.I. Kramer. 2012. The Spread of Emotion via Facebook. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI '12). Association for Computing Machinery, New York, NY, USA, 767--770. https://doi.org/10.1145/2207676.2207787Google Scholar
Digital Library
- Adam DI Kramer, Jamie E Guillory, and Jeffrey T Hancock. 2014. Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences 111, 24 (2014), 8788--8790.Google Scholar
Cross Ref
- Nicole Krämer and Stephan Winter. 2008. Impression Management 2.0: The Relationship of Self-Esteem, Extraversion, Self-Efficacy, and Self-Presentation Within Social Networking Sites. Journal of Media Psychology: Theories, Methods, and Applications 20 (01 2008), 106--. https://doi.org/10.1027/1864--1105.20.3.106Google Scholar
- Danielle Lottridge and Frank R Bentley. 2018. Let's Hate Together: How People Share News in Messaging, Social, and Public Networks. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--13.Google Scholar
Digital Library
- Fiona Lyddy, Francesca Farina, James Hanney, Lynn Farrell, and Niamh Kelly O'Neill. 2014. An Analysis of Language in University Students? Text Messages. Journal of Computer-Mediated Communication 19, 3 (2014), 546--561. https: //doi.org/10.1111/jcc4.12045 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/jcc4.12045Google Scholar
Digital Library
- Yuanchao Ma, Bin Xu, Yin Bai, Guodong Sun, and Run Zhu. 2012. Daily mood assessment based on mobile phone sensing. In Wearable and implantable body sensor networks (BSN), 2012 ninth international conference on. IEEE, 142--147.Google Scholar
Digital Library
- David N. Miller. 2011. Positive Affect. Springer US, Boston, MA, 1121--1122. https://doi.org/10.1007/978-0--387--79061- 9_2193Google Scholar
- Margaret Mitchell, Kristy Hollingshead, and Glen Coppersmith. 2015. Quantifying the Language of Schizophrenia in Social Media. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. Association for Computational Linguistics, Denver, Colorado, 11--20. https: //doi.org/10.3115/v1/W15--1202Google Scholar
Cross Ref
- JA Naslund, KA Aschbrenner, LA Marsch, and SJ Bartels. 2016. The future of mental health care: peer-to-peer support and social media. Epidemiology and Psychiatric Sciences 25, 2 (2016), 113--122.Google Scholar
Cross Ref
- Kathleen O'Leary, Stephen M. Schueller, Jacob O. Wobbrock, and Wanda Pratt. 2018. 'Suddenly, We Got to Become Therapists for Each Other': Designing Peer Support Chats for Mental Health. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Article 331, 14 pages.Google Scholar
Digital Library
- Kenneth M Ovens and Gordon Morison. 2016. Forensic analysis of kik messenger on ios devices. Digital Investigation 17 (2016), 40--52.Google Scholar
Digital Library
- Myoungouk Park, D.W. McDonald, and M. Cha. 2013. Perception differences between the depressed and non-depressed users in Twitter. Proceedings of the 7th International Conference on Weblogs and Social Media, ICWSM 2013 (01 2013), 476--485.Google Scholar
- Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. Transactions of the Association for Computational Linguistics 4 (2016), 61--74.Google Scholar
- James Pennebaker, Martha Francis, and Roger Booth. 1999. Linguistic inquiry and word count (LIWC). (01 1999).Google Scholar
- James W. Pennebaker, Ryan L. Boyd, Kayla Jordan, and Kate Blackburn. 2015. The Development and Psychometric Properties of LIWC2015. (2015). https://doi.org/10.15781/T29G6ZGoogle Scholar
- Megan Ranney, Kyler Lehrbach, Nicholas Scott, Nicole Nugent, Alison Riese, Jeff Huang, Grant Fong, and Rochelle Rosen. 2020. Insights into Adolescent Online Conflict through Qualitative Analysis of Online Messages. Proceedings of the 53rd Hawaii International Conference on System Sciences, 9. https://scholarspace.manoa.hawaii.edu/bitstream/ 10125/64209/0378.pdfGoogle Scholar
Cross Ref
- Andrew G. Reece and Christopher M. Danforth. 2017. Instagram photos reveal predictive markers of depression. EPJ Data Science 6, 1 (Dec. 2017), 1--12. https://doi.org/10.1140/epjds/s13688-017-0110-z Proc. ACM Hum.-Comput. Interact., Vol. 4, No. CSCW2, Article 111. Publication date: October 2020. Sochiatrist: Signals of Affect in Messaging Data 111:25Google Scholar
- Andrew G. Reece, Andrew J. Reagan, Katharina L. M. Lix, Peter Sheridan Dodds, Christopher M. Danforth, and Ellen J. Langer. 2017. Forecasting the onset and course of mental illness with Twitter data. Scientific Reports 7, 1 (Dec. 2017), 13006. https://doi.org/10.1038/s41598-017--12961--9Google Scholar
- Koustuv Saha, Larry Chan, Kaya De Barbaro, Gregory D Abowd, and Munmun De Choudhury. 2017. Inferring Mood Instability on Social Media by Leveraging Ecological Momentary Assessments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 95.Google Scholar
Digital Library
- Sandra Servia-Rodríguez, Kiran K Rachuri, Cecilia Mascolo, Peter J Rentfrow, Neal Lathia, and Gillian M Sandstrom. 2017. Mobile sensing at the service of mental well-being: a large-scale longitudinal study. In Proceedings of the 26th International Conference on World Wide Web. 103--112.Google Scholar
Digital Library
- Saul Shiffman, Arthur A. Stone, and Michael R. Hufford. 2008. Ecological Momentary Assessment. Annual Review of Clinical Psychology 4, 1 (2008), 1--32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415 arXiv:https://doi.org/10.1146/annurev.clinpsy.3.022806.091415 PMID: 18509902.Google Scholar
- Saul Shiffman, Arthur A Stone, and Michael R Hufford. 2008. Ecological momentary assessment. Annu. Rev. Clin. Psychol. 4 (2008), 1--32.Google Scholar
Cross Ref
- Aaron Smith and Monica Anderson. 2018. Social Media Use 2018: Demographics and Statistics. https://www. pewresearch.org/internet/2018/03/01/social-media-use-in-2018/Google Scholar
- Deborah M. Stringer. 2013. Negative Affect. Springer New York, New York, NY, 1303--1304. https://doi.org/10.1007/978- 1--4419--1005--9_606Google Scholar
- Yla R. Tausczik and James W. Pennebaker. 2010. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. Journal of Language and Social Psychology 29, 1 (March 2010), 24--54. https://doi.org/10.1177/ 0261927X09351676Google Scholar
- Auke Tellegen. 1985. Structures of mood and personality and their relevance to assessing anxiety, with an emphasis on self-report. (1985), 681--706.Google Scholar
- William Tov, Kok Leong Ng, Han Lin, and Lin Ge Qiu. 2013. Detecting well-being via computerized content analysis of brief diary entries. Psychological assessment 25 4 (2013), 1069--78.Google Scholar
- Timothy J Trull, Marika B Solhan, Sarah L Tragesser, Seungmin Jahng, Phillip K Wood, Thomas M Piasecki, and David Watson. 2008. Affective instability: measuring a core feature of borderline personality disorder with ecological momentary assessment. Journal of Abnormal Psychology 117, 3 (2008), 647.Google Scholar
Cross Ref
- Wei Wang, Ivan Hernandez, Daniel Newman, Jibo He, and Jiang Bian. 2016. Twitter Analysis: Studying US Weekly Trends in Work Stress and Emotion. Applied Psychology 65 (04 2016), 355--378. https://doi.org/10.1111/apps.12065Google Scholar
- David Watson, Lee Anna Clark, and Auke Tellegen. 1988. Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of Personality and Social Psychology 54, 6 (1988), 1063--1070.Google Scholar
- Yong Sook Yang, Gi Wook Ryu, and Mona Choi. 2019. Methodological strategies for ecological momentary assessment to evaluate mood and stress in adult patients using mobile phones: Systematic review. JMIR mHealth and uHealth 7, 4 (1 1 2019). https://doi.org/10.2196/11215 RGoogle Scholar
Index Terms
Sochiatrist: Signals of Affect in Messaging Data
Recommendations
Human vs. Automated Text Analysis: Estimating Positive and Negative Affect
HT '16: Proceedings of the 27th ACM Conference on Hypertext and Social MediaAutomated text analysis (ATA) has been a widely used tool for determining the sentiment of writing samples. However, it is unclear how ATA compares to human ratings of text when estimating affect. There are costs and benefits associated with each method,...
Unhappy developers: bad for themselves, bad for process, and bad for software product
ICSE-C '17: Proceedings of the 39th International Conference on Software Engineering CompanionRecent research in software engineering supports the "happy-productive" thesis, and the desire of flourishing happiness among programmers is often expressed by industry practitioners. Recent literature has suggested that a cost-effective way to foster ...
Consequences of unhappiness while developing software
SEmotion '17: Proceedings of the 2nd International Workshop on Emotion Awareness in Software EngineeringThe growing literature on affect among software developers mostly reports on the linkage between happiness, software quality, and developer productivity. Understanding the positive side of happiness - positive emotions and moods - is an attractive and ...






Comments