skip to main content
research-article

Multimodal Multiplatform Social Media Event Summarization

Published:25 April 2018Publication History
Skip Abstract Section

Abstract

Social media platforms are turning into important news sources since they provide real-time information from different perspectives. However, high volume, dynamism, noise, and redundancy exhibited by social media data make it difficult to comprehend the entire content. Recent works emphasize on summarizing the content of either a single social media platform or of a single modality (either textual or visual). However, each platform has its own unique characteristics and user base, which brings to light different aspects of real-world events. This makes it critical as well as challenging to combine textual and visual data from different platforms. In this article, we propose summarization of real-world events with data stemming from different platforms and multiple modalities. We present the use of a Markov Random Fields based similarity measure to link content across multiple platforms. This measure also enables the linking of content across time, which is useful for tracking the evolution of long-running events. For the final content selection, summarization is modeled as a subset selection problem. To handle the complexity of the optimal subset selection, we propose the use of submodular objectives. Facets such as coverage, novelty, and significance are modeled as submodular objectives in a multimodal social media setting. We conduct a series of quantitative and qualitative experiments to illustrate the effectiveness of our approach compared to alternative methods.

References

  1. Majed AlRubaian, Muhammad Al-Qurishi, Mabrook Al-Rakhami, Sk Md Mizanur Rahman, and Atif Alamri. 2015. A multistage credibility analysis model for microblogs. In Proceedings of the IEEE/ACM International Conference on Advances in Social Network Analysis and Mining (ASONAM). ACM, New York, NY, 1434--1440. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Matthew A. Baum and Yuri M. Zhukov. 2015. Filtering revolution: Reporting bias in international newspaper coverage of the Libyan civil war. Journal of Peace Research 52, 3 (2015), 384–400.Google ScholarGoogle ScholarCross RefCross Ref
  3. Jingwen Bian, Yang Yang, and Tat-Seng Chua. 2013. Multimedia summarization for trending topics in microblogs. In Proceedings of the International Conference on Information and Knowledge Management (CIKM). ACM, 1807--1812. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Jingwen Bian, Yang Yang, Hanwang Zhang, and Tat-Seng Chua. 2015. Multimedia summarization for social events in microblog stream. IEEE Transactions on Multimedia 17, 2 (2015), 216--228.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. David M. Blei and Michael I. Jordan. 2003. Modeling annotated data. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 127--134. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research 3 (2003), 993--1022. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Zhiyuan Chen and Bing Liu. 2014. Topic modeling using topics from many domains, lifelong learning and big data. In Proceedings of the International Conference on Machine Learning (ICML). Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Lingyang Chu, Yanyan Zhang, Guorong Li, Shuhui Wang, Weigang Zhang, and Qingming Huang. 2016. Effective multimodality fusion framework for cross-media topic detection. IEEE Transactions on Circuits and Systems for Video Technology 26, 3 (2016), 556--569. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Charles L. A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan Büttcher, and Ian MacKinnon. 2008. Novelty and diversity in information retrieval evaluation. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 659--666. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Bin Cui, Anthony K. H. Tung, Ce Zhang, and Zhe Zhao. 2010. Multiple feature fusion for social media applications. In Proceedings of the ACM SIGMOD Conference of the Special Interest Group on Management of Data. ACM, 435--446. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Peng Cui, Wenwu Zhu, Tat-Seng Chua, and Ramesh Jain. 2016. Social-sensed multimedia computing. IEEE MultiMedia 23, 1 (2016), 92--96. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Manfred Del Fabro, Anita Sobe, and Laszlo Böszörmenyi. 2012. Summarization of real-life events based on community-contributed content. In Proceedings of the 4th International Conferences on Advances in Multimedia, MMEDIA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. J. L. Fleiss and others. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 5 (1971), 378--382.Google ScholarGoogle ScholarCross RefCross Ref
  14. Michael Gygli, Helmut Grabner, and Luc Van Gool. 2015. Video summarization by learning submodular mixtures of objectives. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3090--3098.Google ScholarGoogle ScholarCross RefCross Ref
  15. Aria Haghighi and Lucy Vanderwende. 2009. Exploring content models for multi-document summarization. In Proceedings of Human Language Technologies: The 2009 Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 362--370. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20, 4 (Oct. 2002), 422--446. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Gunhee Kim, Seungwhan Moon, and Leonid Sigal. 2015. Joint photo stream and blog post summarization and exploration. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 3081--3089.Google ScholarGoogle ScholarCross RefCross Ref
  18. Ross Kindermann and Laurie Shell. 1980. Markov random fields and their application. American Mathematical Society, volume 1.Google ScholarGoogle ScholarCross RefCross Ref
  19. Katrin Kirchhoff and Jeff Bilmes. 2014. Submodularity for data selection in machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 131--141.Google ScholarGoogle ScholarCross RefCross Ref
  20. Jure Leskovec, Andreas Krause, Carlos Guestrin, Christos Faloutsos, Jeanne VanBriesen, and Natalie Glance. 2007. Cost-effective outbreak detection in networks. In ACM SIGKDD’07. ACM, 420--429. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Bang Hui Lim, Dongyuan Lu, Tao Chen, and Min-Yen Kan. 2015. # mytweet via instagram: Exploring user behaviour across multiple social networks. In Proceedings of the IEEE/ACM International Conference on Advances in Social Network Analysis and Mining (ASONAM). ACM, 113--120. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out: Proceedings of the ACL-04 Workshop 8 (2004).Google ScholarGoogle Scholar
  23. Hui Lin and Jeff Bilmes. 2011. A class of submodular functions for document summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, 510--520. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. X. Lu, Z. Lin, H. Jin, J. Yang, and J. Z. Wang. 2015. Rating image aesthetics using deep learning. IEEE Transactions on Multimedia 17, 11 (Nov. 2015), 2021--2034.Google ScholarGoogle ScholarCross RefCross Ref
  25. Philip J. McParlane, Andrew James McMinn, and Joemon M. Jose. 2014. “Picture the scene...”; Visually summarising social media events. In Proceedings of the International Conference on Information and Knowledge Management (CIKM). ACM, 1459--1468. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Donald Metzler and W. Bruce Croft. 2005. A Markov random field model for term dependencies. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 472--479. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Michel Minoux. 1978. Accelerated greedy algorithms for maximizing submodular set functions. In Optimization Techniques. Springer, 234--243.Google ScholarGoogle Scholar
  28. Baharan Mirzasoleiman, Amin Karbasi, Rik Sarkar, and Andreas Krause. 2013. Distributed submodular maximization: Identifying representative elements in massive data. In Advances in Neural Information Processing Systems. 2049--2057. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. George L. Nemhauser, Laurence A. Wolsey, and Marshall L. Fisher. 1978. An analysis of approximations for maximizing submodular set functions I. Mathematical Programming 14, 1 (1978), 265--294. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Liqiang Nie, Xuemeng Song, and Tat-Seng Chua. 2016. Learning from multiple social networks. Synthesis Lectures on Information Concepts, Retrieval, and Services 8, 2 (2016), 1--118.Google ScholarGoogle ScholarCross RefCross Ref
  31. Liqiang Nie, Luming Zhang, Yi Yang, Meng Wang, Richang Hong, and Tat-Seng Chua. 2015. Beyond doctors: Future health prediction from multimedia and multimodal observations. In Proceedings of the 23rd ACM International Conference on Multimedia. ACM, 591--600. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Brendan O’Connor, Michel Krieger, and David Ahn. 2010. TweetMotif: Exploratory search and topic summarization for Twitter. In Proceedings of the International Conference on Weblogs and Social Media (ICWSM).Google ScholarGoogle Scholar
  33. Shengsheng Qian, Tianzhu Zhang, Richang Hong, and Changsheng Xu. 2015a. Cross-domain collaborative learning in social multimedia. In Proceedings of the 23rd ACM International Conference on Multimedia. ACM, 99--108. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Shengsheng Qian, Tianzhu Zhang, and Changsheng Xu. 2016. Multi-modal multi-view topic-opinion mining for social event analysis. In Proceedings of the 2016 ACM on Multimedia Conference. ACM, 2--11. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Shengsheng Qian, Tianzhu Zhang, Changsheng Xu, and M. Shamim Hossain. 2015b. Social event classification via boosted multimodal supervised latent dirichlet allocation. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 11, 2 (2015), 27. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Manos Schinas, Symeon Papadopoulos, Yiannis Kompatsiaris, and Pericles A. Mitkas. 2015. Visual event summarization on social media using topic modelling and graph-based ranking algorithms. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval. ACM, 203--210. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Rajiv Ratn Shah, Anwar Dilawar Shaikh, Yi Yu, Wenjing Geng, Roger Zimmermann, and Gangshan Wu. 2015. EventBuilder: Real-time multimedia event summarization by visualizing social media. In ACM Multimedia. ACM, 185--188. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Beaux Sharifi, Mark-Anthony Hutton, and Jugal Kalita. 2010. Summarizing microblogs automatically. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 685--688. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Pinaki Sinha, Sharad Mehrotra, and Ramesh Jain. 2011. Summarization of personal photologs using multidimensional content and context. In Proceedings of the ACM International Conference on Multimedia Retrieval. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Ruben Sipos, Adith Swaminathan, Pannaga Shivaswamy, and Thorsten Joachims. 2012. Temporal corpus summarization using submodular word coverage. In Proceedings of the International Conference on Information and Knowledge Management (CIKM). ACM, 754--763. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Xuemeng Song, Liqiang Nie, Luming Zhang, Mohammad Akbari, and Tat-Seng Chua. 2015. Multiple social network learning and its application in volunteerism tendency prediction. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 213--222. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. X. Tang, W. Luo, and X. Wang. 2013. Content-based photo quality assessment. IEEE Transactions on Multimedia 15, 8 (Dec. 2013), 1930--1943. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Sebastian Tschiatschek, Rishabh K. Iyer, Haochen Wei, and Jeff A. Bilmes. 2014. Learning mixtures of submodular functions for image collection summarization. In Advances in Neural Information Processing Systems. 1413--1421. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Zhiyu Wang, Peng Cui, Lexing Xie, Hao Chen, Wenwu Zhu, and Shiqiang Yang. 2012. Analyzing social media via event facets. In Proceedings of the 20th ACM International Conference on Multimedia. ACM, 1359--1360. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Zhiyu Wang, Peng Cui, Lexing Xie, Wenwu Zhu, Yong Rui, and Shiqiang Yang. 2014. Bilateral correspondence model for words-and-pictures association in multimedia-rich microblogs. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 10, 4 (2014), 34. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Kai Wei, Yuzong Liu, Katrin Kirchhoff, and Jeff A Bilmes. 2013. Using document summarization techniques for speech data subset selection. In Proceedings of the Conference North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT). 721--726.Google ScholarGoogle Scholar
  47. Ming Yan, Jitao Sang, Changsheng Xu, and M. Shamim Hossain. 2016. A unified video recommendation by cross-network user modeling. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 12, 4 (2016), 53. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. M. Zaharieva, M. D. Fabro, and M. Zeppelzauer. 2015. Cross-platform social event detection. IEEE MultiMedia 22, 3 (July 2015), 14--25.Google ScholarGoogle ScholarCross RefCross Ref
  49. Tianzhu Zhang, Shengsheng Qian, Changsheng Xu, and Jie Shao. 2016. Multi-modal event topic model for social event analysis. IEEE Transactions on Multimedia 18, 2 (2016), 233--246.Google ScholarGoogle ScholarCross RefCross Ref
  50. Tianzhu Zhang and Changsheng Xu. 2014. Cross-domain multi-event tracking via CO-PMHT. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 10, 4 (2014), 31. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Multimodal Multiplatform Social Media Event Summarization

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!