research-article

#thyghgapp: Instagram Content Moderation and Lexical Variation in Pro-Eating Disorder Communities

Authors:

Stevie Chancellor,

Jessica Annette Pater,

Munmun De ChoudhuryAuthors Info & Claims

CSCW '16: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing

Pages 1201 - 1213

https://doi.org/10.1145/2818048.2819963

Published: 27 February 2016 Publication History

Abstract

Pro-eating disorder (pro-ED) communities on social media encourage the adoption and maintenance of disordered eating habits as acceptable alternative lifestyles rather than threats to health. In particular, the social networking site Instagram has reacted by banning searches on several pro-ED tags and issuing content advisories on others. We pre-sent the first large-scale quantitative study investigating pro-ED communities on Instagram in the aftermath of moderation -- our dataset contains 2.5M posts between 2011 and 2014. We find that the pro-ED community has adopted non-standard lexical variations of moderated tags to circumvent these restrictions. In fact, increasingly complex lexical variants have emerged over time. Communities that use lexical variants show increased participation and support of pro-ED (15-30%). Finally, the tags associated with content on these variants express more toxic, self-harm, and vulnerable content. Despite Instagram's moderation strategies, pro-ED communities are active and thriving. We discuss the effectiveness of content moderation as an intervention for communities of deviant behavior.

References

[1]

Ronald L. Akers, Marvin D. Krohn, Lonn Lanza-Kaduce, and Marcia Radosevich. 1979. Social learning and deviant behavior: A specific test of a general theory. American Sociological Review, 636-655.

[2]

Jon Arcelus, Alex J Mitchell, Jackie Wales, and Sren Nielsen. 2011. Mortality rates in patients with anorexia nervosa and other eating disorders: a meta-analysis of 36 studies. Archv. Gen. Psychiatry 68, 7 (2011), 724–731.

[3]

David Bamman, Brendan O'Connor, and Noah Smith. 2012. Censorship and deletion practives in Chinese so-cial media. First Monday, 17(3).

[4]

David Bamman, Jacob Eisenstein and Tyler Schnoe-belen. 2014. Gender identity and lexical variation in so-cial media. Journal of Sociolinguistics, 18(2), 135-160.

[5]

Anna M Bardone-Cone and Kamila M Cass. 2006. In-vestigating the impact of pro-anorexia websites: A pilot study. European Eating Disorders Review 14, 4 (2006), 256–262.

[6]

Yvonne M. Bishop, Stephen E. Feinberg, and Paul W. Holland. 2007. Discrete multivariate analysis: theory and practice. Springer Science & Business Media.

[7]

Dina LG Borzekowski, Summer Schenk, Jenny L Wil-son, and Rebecka Peebles. 2010. e-Ana and e-Mia: A content analysis of pro–eating disorder web sites. Amer-ican journal of public health 100, 8 (2010), 1526.

[8]

Samuel Brody and Nicholas Diakopoulos. 2011. Cooooooooooooooo llllllllllllll!!!!!!!!!!!!!!: using word lengthening to detect sentiment in microblogs. In Proc. EMNLP. 562-570.

Digital Library

[9]

Jane Callaghan. 2013. Research in online spaces: 'Tum-blr' and eating 'disorder'. Symposium presented to: Children and Young People's Mental Health. The Uni-versity of Northampton.

[10]

Justine Costanza. Instagram 'Thinspo' Ban Won't Com-bat Pro-Eating Disorder Web Content. 2013. Retrieved November 15, 2014 from http://www.ibtimes.com/instagram-thinspo-ban-wont-combat-pro-eating-disorder-web-content-994366

[11]

Munmun De Choudhury. 2015. Anorexia on Tumblr: A Characterization Study. In Proc. Digital Health.

Digital Library

[12]

Munmun De Choudhury and Sushovan De. 2014. Men-tal Health Discourse on Reddit: Self-Disclosure, Social Support, and Anonymity. In Proc. ICWSM.

[13]

Lauren Duca. Can Thinspiration Really Be #Banned From Instagram?. 2013. Retrieved November 16, 2014 from http://www.huffingtonpost. com/lauren-duca/thinspiration-banned-from-instagram_b_3829155.html

[14]

Maeve Duggan, Nicole B. Ellison, Cliff Lampe, Aman-da Lenhart, and Mary Madden. 2015. Social Media Up-date 2014. Pew Research Internet Proj.

[15]

Jacob Eisenstein, Brendan O'Connor, Noah A. Smith, and Eric P. Xing. 2010. A latent variable model for geo-graphic lexical variation. In Proc. EMNLP.

[16]

Tobit Emmens and Andy Phippen. 2010. Evaluating Online Safety Programs. Harvard Berkman Center for Internet & Society.

[17]

Antonio M. Espinoza and Jedidiah R. Crandall. 2011. Work-in-progress: Automated named entity extraction for tracking censorship of current events. In USENIX Workshop on Free & Open Comm. on the Internet.

[18]

Facebook. Statement of Rights and Responsibilities. 2013. Retrieved January 10, 2015 from https://www.facebook.com/legal/terms

[19]

Rachel A Fleming-May and Laura E Miller. 2010. I'm scared to look. But I'm dying to know: Information seeking and sharing on Pro-Ana weblogs. In Proc. ASIST 47, 1 (2010), 1–9.

Digital Library

[20]

Stephan Gouws, Donald Metzler, Congxing Cai, and Eduard Hovy. 2011. Contextual bearing on linguistic variation in social media. In Proc. Workshop on LSM. 20-29.

Digital Library

[21]

Heba Hasan. Instagram bans Thinspo Content. 2012. Retrieved November 15, 2014 from http://newsfeed.time.com/2012/04/26/instagram-bans-thinspo-content/

[22]

Chaya Hiruncharoenvate, Zhiyuan Lin, and Eric Gilbert. "Algorithmically Bypassing Censorship on Sina Weibo with Nondeterministic Homophone Substitutions." 2015. In Proc. ICWSM.

[23]

Yuheng Hu, Kartik Talamadupula, & Subbarao Kambhampati. 2013. Dude, srsly?: The Surprisingly Formal Nature of Twitter's Language. In Proc. ICWSM.

[24]

Instagram. Instagram's New Guidelines Against Self-Harm Images & Accounts. 2012. Retrieved November 17, 2014 from http://blog.instagram.com/post/21454597658/ insta-grams-new-guidelines-against-self-harm

[25]

Gary King, Jennifer Pan, & Margaret E. Roberts. 2013. How censorship in China allows government criticism but silences collective expression. American Political Science Rev, 107(02), 326-343.

[26]

Farshad Kooti, Haeryun Yang, Meeyoung Cha, P. Krishna Gummadi, and Winter A. Mason. 2012. The Emergence of Conventions in Online Social Networks. In Proc. ICWSM.

Digital Library

[27]

Stephen P. Lewis and Alexis E. Arbuthnott. 2012. Searching for thinspiration: The nature of internet searches for pro- eating disorder websites. Cyberpsy-chology, Behavior, and Social Networking, 15(4), 200-204.

[28]

Rebecca MacKinnon. 2008. Flatter world and thicker walls? Blogs, censorship and civic discourse in China. Public Choice 134, 1-2 (2008), 31–46.

[29]

Jeanne B Martin. 2010. The development of ideal body image perceptions in the United States. Nutrition Today 45, 3 (2010), 98–110.

[30]

Gonzalo Navarro. 2001. A guided tour to approximate string matching. ACM comp. surveys, 33(1), 31-88.

Digital Library

[31]

Minsu Park, David W McDonald, and Meeyoung Cha. 2013. Perception Differences between the Depressed and Non-depressed Users in Twitter. In Proc. ICWSM.

[32]

Alice Philipson. Concerns raised over Instagram after app allows users to see photos promoting anorexia. 2013. Retrieved November 16, 2014 from http://www.telegraph.co.uk/technology/social-media/9775559/Concerns-raised-over-Instagram-after-app-allows-users-to-see-photos-promoting-anorexia.html

[33]

Hansen Andrew Schwartz, Johannes C. Eichstaedt., Margaret L. Kern, Lukasz Dziurzynski, Richard E. Lu-cas, Megha Agrawal, Gregory J. Park, et al. 2013. Char-acterizing Geographic Variation in Well-Being Using Tweets. In Proc. ICWSM.

[34]

Leslie Regan Shade. 2003. Weborexics: The Ethical Issues Surrounding Pro-Ana Websites. SIGCAS Comput. Soc. 33, 4 (Dec. 2003).

Digital Library

[35]

Jianbo Shi and Jitendra Malik. 2000. Normalized cuts and image segmentation. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(8), 888-905.

Digital Library

[36]

Clay Shirky. 2011. The political power of social media: Technology, the public sphere, and political change. Foreign affairs, 28-41.

[37]

Irina Shklovski and Nalini Kotamraju. 2011. Online contribution practices in countries that engage in Inter-net blocking and censorship. In Proc. CHI. 1109-1118.

Digital Library

[38]

Frédérique RE Smink, Daphne van Hoeken, and Hans W Hoek. 2012. Epidemiology of eating disorders: inci-dence, prevalence and mortality rates. Current psychia-try reports 14, 4 (2012), 406–414.

[39]

John R Suler and Wende L Phillips. 1998. The bad boys of cyberspace: Deviant behavior in a multimedia chat community. CyberPsychology & Behavior 1, 3 (1998), 275–294.

[40]

Zeynep Tufekci, & Christopher Wilson. 2012. Social media and the decision to participate in political protest: Observations from Tahrir Square. Journal of Comm, 62(2), 363-379.

[41]

Tracey D. Wade, Anna Keski-Rahkonen., & James I. Hudson. 2011. Epidemiology of Eating Disorders. In Textbook in Psychiatric Epidemiology. Wiley, 343–360.

[42]

Elad Yom-Tov, Luis Fernandez-Luque, Ingmar Weber, and P Steven Crain. 2012. Pro-Anorexia and Pro-Recovery Photo Sharing: A Tale of Two Warring Tribes. J Med Internet Res (2012).

[43]

Yafu Zhao and William Encinosa. 2009. Hospitaliza-tions for Eating Disorders from 1999-2006. Agency for Healthcare Policy and Research, Rockville, MD.

Cited By

Nova FShinde APfafman RHarrison ALogan Delaney CPater J(2025)Sentiment Analysis of #Meanspo Tweets: Humans vs. Automatic ClassificationsProceedings of the ACM on Human-Computer Interaction10.1145/37012079:1(1-26)Online publication date: 10-Jan-2025
https://dl.acm.org/doi/10.1145/3701207
KS R(2024)Analyzing Online Conversations on Reddit: A Study of Stress and Anxiety Through Topic Modeling and Sentiment AnalysisCureus10.7759/cureus.69030Online publication date: 9-Sep-2024
https://doi.org/10.7759/cureus.69030
Merhbene GPuttick AKurpicz-Briki M(2024)Investigating machine learning and natural language processing techniques applied for detecting eating disorders: a systematic literature reviewFrontiers in Psychiatry10.3389/fpsyt.2024.131952215Online publication date: 26-Mar-2024
https://doi.org/10.3389/fpsyt.2024.1319522
Show More Cited By

Recommendations

Detecting and Characterizing Eating-Disorder Communities on Social Media
WSDM '17: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining

Eating disorders are complex mental disorders and responsible for the highest mortality rate among mental illnesses. Recent studies reveal that user-generated content on social media provides useful information in understanding these disorders. Most ...
Sensitive Self-disclosures, Responses, and Social Support on Instagram: The Case of #Depression
CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

People can benefit from disclosing negative emotions or stigmatized facets of their identities, and psychologists have noted that imagery can be an effective medium for expressing difficult emotions. Social network sites like Instagram offer ...
"Notjustgirls": Exploring Male-related Eating Disordered Content across Social Media Platforms
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Eating disorders (EDs) are a worldwide public health concern that impact approximately 10% of the U.S. population. Our previous research characterized these behaviors across online spaces. These characterizations have used clinical terminology, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '16: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing

February 2016

1866 pages

ISBN:9781450335928

DOI:10.1145/2818048

General Chairs:
Darren Gergle
Northwestern University
,
Meredith Ringel Morris
Microsoft Research
,
Program Chairs:
Pernille Bjørn
University of Copenhagen
,
Joseph Konstan
University of Minnesota

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 February 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CSCW '16

Sponsor:

SIGCHI

CSCW '16: Computer Supported Cooperative Work and Social Computing

February 27 - March 2, 2016

California, San Francisco, USA

Acceptance Rates

CSCW '16 Paper Acceptance Rate 142 of 571 submissions, 25%;

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

198
Total Citations
View Citations
3,138
Total Downloads

Downloads (Last 12 months)409
Downloads (Last 6 weeks)35

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nova FShinde APfafman RHarrison ALogan Delaney CPater J(2025)Sentiment Analysis of #Meanspo Tweets: Humans vs. Automatic ClassificationsProceedings of the ACM on Human-Computer Interaction10.1145/37012079:1(1-26)Online publication date: 10-Jan-2025
https://dl.acm.org/doi/10.1145/3701207
KS R(2024)Analyzing Online Conversations on Reddit: A Study of Stress and Anxiety Through Topic Modeling and Sentiment AnalysisCureus10.7759/cureus.69030Online publication date: 9-Sep-2024
https://doi.org/10.7759/cureus.69030
Merhbene GPuttick AKurpicz-Briki M(2024)Investigating machine learning and natural language processing techniques applied for detecting eating disorders: a systematic literature reviewFrontiers in Psychiatry10.3389/fpsyt.2024.131952215Online publication date: 26-Mar-2024
https://doi.org/10.3389/fpsyt.2024.1319522
Nova FPfafman RLogan Delaney CPater J(2024)Unveiling the "Toxic" World of #Meanspo: Understanding Users' Emerging Online Eating Disorder Practices in X/TwitterProceedings of the ACM on Human-Computer Interaction10.1145/36870618:CSCW2(1-27)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687061
Rivera VWilkinson DAugusta ALi SRedmiles EStrohmayer A(2024)Safer Algorithmically-Mediated Offline Introductions: Harms and Protective BehaviorsProceedings of the ACM on Human-Computer Interaction10.1145/36869488:CSCW2(1-43)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686948
Baumer ETaylor ABrubaker JMcGee M(2024)Algorithmic SubjectivitiesACM Transactions on Computer-Human Interaction10.1145/366034431:3(1-34)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3660344
Hwang SKiene COng SShaw A(2024)Adopting Third-party Bots for Managing Online CommunitiesProceedings of the ACM on Human-Computer Interaction10.1145/36537078:CSCW1(1-26)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3653707
Huang ESarma AHwang SChandrasekharan EChancellor S(2024)Opportunities, tensions, and challenges in computational approaches to addressing online harassmentProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661623(1483-1498)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661623
Guluzade LSas C(2024)Functionality and User Review Analysis of Mobile Apps for Mindfulness Eating and Eating DisordersProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661521(1350-1371)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661521
Nigatu HRaji I(2024)“I Searched for a Religious Song in Amharic and Got Sexual Content Instead’’: Investigating Online Harm in Low-Resourced Languages on YouTube.Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658546(141-160)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658546
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten