skip to main content
research-article

MMSUM Digital Twins: A Multi-view Multi-modality Summarization Framework for Sporting Events

Authors Info & Claims
Published:27 January 2022Publication History
Skip Abstract Section

Abstract

Sporting events generate a massive amount of traffic on social media with live moment-to-moment accounts as any given situation unfolds. The generated data are intensified by fans feelings, reactions, and subjective opinions towards what happens during the event, all of which are based on their individual points of view. Analyzing and summarizing this data will generate a comprehensive overview of the event in terms of how the event evolves and how fans react and view the event based on their perspectives. Previously, most of the summarization works ignore fan reactions and subjective opinions, and focus primarily on generating an objective-view summary. We believe that an effective and useful summary should consider human reactions, sentiment, and point of view, as opposed to simply describing what happens during the event. Accordingly, in this work, we propose MMSUM Digital Twins: a summarization framework that is capable of generating a multi-view multi-modal summary for sporting events in real-time. The proposed digital twins-based framework consists of four main components: sub-event recognition which detects the event’s key moments, tweet categorization, which determines which team the tweets’ writers support and assigns tweets to their teams, sentiment analysis to track fans’ state of mind, and image popularity prediction for selecting representative images. Furthermore, the MMSUM employs a visual-filtering model to address the issue of noisy images that inundate social media, compromising the summarization quality. We leverage the knowledge of sport fans to evaluate the generated multi-view summarization through an online user study. The experiment results confirm the effectiveness of our proposed approach for summarizing sporting events by considering multimedia data, sentiment, and subjective views of the event.

REFERENCES

  1. [1] Aloufi Samah and Saddik Abdulmotaleb El. 2018. Sentiment identification in football-specific tweets. IEEE Access 6 (2018), 7860978621.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Aloufi Samah, Zhu Shiai, and Saddik Abdulmotaleb El. 2017. On the prediction of Flickr image popularity by analyzing heterogeneous social sensory data. Sensors 17, 3 (2017), 631.Google ScholarGoogle Scholar
  3. [3] Bian Jingwen, Yang Yang, Zhang Hanwang, and Chua Tat-Seng. 2015. Multimedia summarization for social events in microblog stream. IEEE Transactions on Multimedia 17, 2 (Feb 2015), 216228. https://doi.org/10.1109/TMM.2014.2384912Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. [4] Bischke Benjamin, Borth Damian, and Dengel Andreas. 2019. Large-scale social multimedia analysis. In Big Data Analytics for Large-Scale Multimedia Search. Wiley, 157178.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Borth Damian, Ji Rongrong, Chen Tao, Breuel Thomas, and Chang Shih-Fu. 2013. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In Proceedings of the 21st ACM International Conference on Multimedia. 223232. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. [6] Chakrabarti Deepayan and Punera Kunal. 2011. Event summarization using tweets. In Proceedings of the 5th International AAAI Conference on Weblogs and Social Media.Google ScholarGoogle Scholar
  7. [7] Chua Freddy and Asur Sitaram. 2013. Automatic summarization of events from social media. In Proceedings of the 7th International AAAI Conference on Weblogs and Social Media.Google ScholarGoogle Scholar
  8. [8] Corney David, Dancausa Carlos Martin, and Goker Ayse. 2014. Two sides to every story: Subjective event summarization of sports events using Twitter. In ICMR 2014, 1st International Workshop on Social Multimedia and Storytelling (SoMuS 2014), CEUR Proceedings.Google ScholarGoogle Scholar
  9. [9] Saddik Abdulmotaleb El. 2018. Digital twins: The convergence of multimedia technologies. IEEE MultiMedia 25, 2 (2018), 8792.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Saddik Abdulmotaleb El, Badawi Hawazin, Velazquez Roberto Alejandro Martinez, Laamarti Fedwa, Diaz Rogelio Gámez, Bagaria Namrata, and Arteaga-Falconi Juan Sebastian. 2019. Dtwins: A digital twins ecosystem for health and well-being. MMTC Communications-Frontiers 14, 2 (2019), 3946.Google ScholarGoogle Scholar
  11. [11] Gratch Jonathan, Lucas Gale, Malandrakis Nikolaos, Szablowski Evan, Fessler Eli, and Nichols Jeffrey. 2015. GOAALLL!: Using sentiment in the World Cup to explore theories of emotion. In Proceedings of the 2015 International Conference on Affective Computing and Intelligent Interaction (ACII). 898903. https://doi.org/10.1109/ACII.2015.7344681 Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Guo Bin, Ouyang Yi, Zhang Cheng, Zhang Jiafan, Yu Zhiwen, Wu Di, and Wang Yu. 2017. CrowdStory: Fine-grained event storyline generation by fusion of multi-modal crowdsourced data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3, Article 55 (Sept. 2017), 19 pages. https://doi.org/10.1145/3130920 Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Hsieh Liang-Chi, Lee Ching-Wei, Chiu Tzu-Hsuan, and Hsu Winston. 2012. Live semantic sport highlight detection based on analyzing tweets of Twitter. In Proceedings of the 2012 IEEE International Conference on Multimedia and Expo. 949954. https://doi.org/10.1109/ICME.2012.135 Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Inouye David and Kalita Jugal K. 2011. Comparing Twitter summarization algorithms for multiple post summaries. In 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing. 298306. https://doi.org/10.1109/PASSAT/SocialCom.2011.31Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Jia Yangqing, Shelhamer Evan, Donahue Jeff, Karayev Sergey, Long Jonathan, Girshick Ross, Guadarrama Sergio, and Darrell Trevor. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM International Conference on Multimedia (MM’14). 675678. https://doi.org/10.1145/2647868.2654889 Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Krizhevsky Alex, Sutskever Ilya, and Hinton Geoffrey E.. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 10971105. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Kubo Mitsumasa, Sasano Ryohei, Takamura Hiroya, and Okumura Manabu. 2013. Generating live sports updates from Twitter by finding good reporters. In Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), Vol. 1. 527534. https://doi.org/10.1109/WI-IAT.2013.74 Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. [18] Lee Michael. 2017. NBA memes: The role of fan image macros within the online NBA fan community. Ph.D. Dissertation. Queensland University of Technology.Google ScholarGoogle Scholar
  19. [19] Lienhart Rainer W. and Hartmann Alexander. 2002. Classifying images on the web automatically. Journal of Electronic Imaging 11, 4 (2002), 445454. https://doi.org/10.1117/1.1502259Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Lin Chin-Yew. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. 7481. https://www.aclweb.org/anthology/W04-1013Google ScholarGoogle Scholar
  21. [21] Marcelino Gonçalo, Pinto Ricardo, and Magalhães João. 2018. Ranking news-quality multimedia. In Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval (Yokohama, Japan) (ICMR’18). 1018. https://doi.org/10.1145/3206025.3206053 Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Marcus Adam, Bernstein Michael S., Badar Osama, Karger David R., Madden Samuel, and Miller Robert C.. 2011. Twitinfo: Aggregating and visualizing microblogs for event exploration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI’11). 227236. https://doi.org/10.1145/1978942.1978975 Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] McParlane Philip J., McMinn Andrew James, and Jose Joemon M.. 2014. ‘Picture the Scene...’: Visually summarising social media events. In Proceedings of the 23rd ACM International Conference on Information and Knowledge Management (Shanghai, China) (CIKM’14). 14591468. https://doi.org/10.1145/2661829.2661923 Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Meladianos Polykarpos, Nikolentzos Giannis, Rousseau François, Stavrakas Yannis, and Vazirgiannis Michalis. 2015. Degeneracy-based real-time sub-event detection in twitter stream. In Proceedings of the 9th International AAAI Conference on Web and Social Media.Google ScholarGoogle Scholar
  25. [25] Meladianos Polykarpos, Xypolopoulos Christos, Nikolentzos Giannis, and Vazirgiannis Michalis. 2018. An optimization approach for sub-event detection and summarization in twitter. In Advances in Information Retrieval. 481493.Google ScholarGoogle Scholar
  26. [26] Nichols Jeffrey, Mahmud Jalal, and Drews Clemens. 2012. Summarizing sporting events using twitter. In Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces (Lisbon, Portugal) (IUI’12). 189198. https://doi.org/10.1145/2166966.2166999 Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. [27] Phuvipadawat Swit and Murata Tsuyosh. 2010. Breaking news detection and tracking in twitter. In Proceedings of the 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, Vol. 3. 120123. https://doi.org/10.1109/WI-IAT.2010.205 Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Qian Xueming, Li Mingdi, Ren Yayun, and Jiang Shuhui. 2018. Social media based event summarization by user-text-image co-clustering. Knowledge-Based Systems (2018). https://doi.org/10.1016/j.knosys.2018.10.028Google ScholarGoogle Scholar
  29. [29] Schinas Manos, Papadopoulos Symeon, Kompatsiaris Yiannis, and Mitkas Pericles A.. 2016. MGraph: Multimodal event summarization in social media using topic models and graph-based ranking. International Journal of Multimedia Information Retrieval 5, 1 (01 Mar 2016), 5169. https://doi.org/10.1007/s13735-015-0089-9Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Sharifi Beaux, Hutton Mark-Anthony, and Kalita Jugal. 2010. Summarizing microblogs automatically. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. 685688. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Shih Huang-Chia. 2018. A survey of content-aware video analysis for sports. IEEE Transactions on Circuits and Systems for Video Technology 28, 5 (2018), 12121231. https://doi.org/10.1109/TCSVT.2017.2655624Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Tagawa Yuuki and Shimada Kazutaka. 2018. Sports Game Summarization Based on Sub-events and Game-Changing Phrases. Springer International Publishing, 6580. https://doi.org/10.1007/978-3-319-70636-8_5Google ScholarGoogle Scholar
  33. [33] Tang Anthony and Boring Sebastian. 2012. # EpicPlay: Crowd-sourcing sports video highlights. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 15691572. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Lee Chris van der, Krahmer Emiel, and Wubben Sander. 2017. PASS: A Dutch data-to-text system for soccer, targeted towards specific audiences. In Proceedings of the 10th International Conference on Natural Language Generation. 95104. https://doi.org/10.18653/v1/W17-3513Google ScholarGoogle Scholar
  35. [35] Van Oorschot Guido, Van Erp Marieke, and Dijkshoorn Chris. 2012. Automatic extraction of soccer game events from twitter. In Proceedings of Detection, Representation and Exploitation of Events in the Semantic Web (DeRiVE 2012).Google ScholarGoogle Scholar
  36. [36] Wang Fei and Kan Min-Yen. 2006. NPIC: Hierarchical synthetic image classification using image search and generic features. In Proceedings of the 5th International Conference on Image and Video Retrieval. 473482. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Zhang Mei-Yan and Cai Wen-Yu. 2014. Multi-view video summarization algorithm for WMSN. In 2014 International Conference on Wireless Communication and Sensor Network. IEEE, 213216. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. [38] Zubiaga Arkaitz, Spina Damiano, Amigó Enrique, and Gonzalo Julio. 2012. Towards real-time summarization of scheduled events from twitter streams. In Proceedings of the 23rd ACM Conference on Hypertext and Social Media (HT’12). 319320. https://doi.org/10.1145/2309996.2310053 Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. MMSUM Digital Twins: A Multi-view Multi-modality Summarization Framework for Sporting Events

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM Transactions on Multimedia Computing, Communications, and Applications
            ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 1
            January 2022
            517 pages
            ISSN:1551-6857
            EISSN:1551-6865
            DOI:10.1145/3505205
            Issue’s Table of Contents

            Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 27 January 2022
            • Accepted: 1 April 2021
            • Revised: 1 March 2021
            • Received: 1 November 2020
            Published in tomm Volume 18, Issue 1

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article
            • Refereed

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          Full Text

          View this article in Full Text.

          View Full Text

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!