skip to main content
research-article

Using Multisensory Content to Impact the Quality of Experience of Reading Digital Books

Published:12 November 2021Publication History
Skip Abstract Section

Abstract

Multisensorial books enrich a story with either traditional multimedia content or sensorial effects. The main idea is to increase children’s interest in reading by enhancing their QoE while reading. Studies on enriched and/or augmented e-books also propose synchronizing additional content with text. However, they usually focus on audio, vídeo, images, or haptic feedback. In this work, we present MBook, a tool for presenting multisensorial books. It decouples the book’s textual content from the additional content, as well as its synchronization, and rendering. Thus, a change in the additional content or its synchronization does not require changes to the book’s content. To enable fine-grained synchronization, MBook captures the reading position using an eye-tracker. Experimental results with students within the 13- to 19-year-old age group point to MBook being able to provide good usability.

REFERENCES

  1. [1] Ademoye Oluwakemi A. and Ghinea Gheorghita. 2013. Information recall task impact in olfaction-enhanced multimedia. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 9, 3 (2013), 116. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Alam Kazi Masudul, Rahman Abu Saleh Md Mahfujur, and Saddik Abdulmotaleb El. 2013. Mobile haptic e-book system to support 3D immersive reading in ubiquitous environments. ACM Transactions on Multimedia Computing, Communications, and Applications 9, 4 (2013), 27. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Borgstrom Liam. 2011. Developing story-time: The importance of interactivity in encouraging childhood reading. Mousaion 29, 3 (2011), 193208.Google ScholarGoogle Scholar
  4. [4] Bottos Stephen and Balasingam Balakumar. 2019. Tracking the progression of reading through eye-gaze measurements. In 22th International Conference on Information Fusion (FUSION’19). 18.Google ScholarGoogle Scholar
  5. [5] Brooke John et al. 1996. SUS-A quick and dirty usability scale. Usability Evaluation in Industry 189, 194 (1996), 47.Google ScholarGoogle Scholar
  6. [6] Cecotti H.. 2016. A multimodal gaze-controlled virtual keyboard. IEEE Transactions on Human-Machine Systems 46, 4 (2016), 601606.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Fu Fong-Ling, Su Rong-Chang, and Yu Sheng-Chin. 2009. EGameFlow: A scale to measure learners’ enjoyment of e-learning games. Computers & Education 52, 1 (2009), 101112. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Ghinea Gheorghita, Timmerer Christian, Lin Weisi, and Gulliver Stephen R.. 2014. Mulsemedia: State of the art, perspectives, and challenges. ACM Transactions on Multimedia Computing, Communications, and Applications 11, 1s (2014), 17. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Gibson A., Webb A., and Stirling L.. 2018. Evaluation of a visual–tactile multimodal display for surface obstacle avoidance during walking. IEEE Transactions on Human-Machine Systems 48, 6 (2018), 604613.Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Gökbulut Burak and Yeniasır Mustafa. 2017. Analysis of children’s interest in books and their reading levels depending on the education status of family. Quality & Quantity 52 (2017), 235–245. DOI: https://doi.org/10.1007/s11135-017-0608-2Google ScholarGoogle Scholar
  11. [11] Gu Heng, Sanchez Susana, Kunze Kai, and Inami Masahiko. 2015. An augmented e-reader for multimodal literacy. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers. 353356. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Guedes Gustavo Paiva. 2018. Multisensorial books: Improving readers’ quality of experience. In 2018 XIII Latin American Conference on Learning Objects and Technology (LACLO’18).Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Heikkilä J., Alho K., Hyvönen H., and Tiippana K.. 2015. Audiovisual semantic congruency during encoding enhances memory performance. Experimental Psychology 62, 2 (2015), 123130. https://doi.org/10.1027/1618-3169/a000279Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Kovács P. T., Murray N., Rozinaj G., Sulema Y., and Rybárová R.. 2015. Application of immersive technologies for education: State of the art. In International Conference on Interactive Mobile Communication Technologies and Learning (IMCL’15). 283288. https://doi.org/10.1109/IMCTL.2015.7359604Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Lin Jim-Min, Chiou Che Wun, Lee Chiou-Yng, and Hsiao Jing-Rui. 2016. Supporting physical agents in an interactive e-book. In Genetic and Evolutionary Computing. Springer International Publishing, 243252. Google ScholarGoogle Scholar
  16. [16] Murray Niall, Ademoye Oluwakemi A., Ghinea Gheorghita, and Muntean Gabriel-Miro. 2017. A tutorial for olfaction-based multisensorial media application design and evaluation. ACM Computing Surveys 50, 5, Article 67 (Sept. 2017), 30 pages. https://doi.org/10.1145/3108243 Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Ribeiro Pedro, Michel Anna, Iurgel Ido, Ressel Christian, Sylla Cristina, and Müller Wolfgang. 2018. Designing a smart reading environment with and for children. In Proceedings of the 12th International Conference on Tangible, Embedded, and Embodied Interaction. 8893. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. [18] Ribeiro Pedro, Sylla Cristina Maria Santos Moreira Silva, Iurgel Ido Aharon, Müller Wolfgang, and Ressel Christian. 2017. STREEN: Designing smart environments for story reading with children. International Journal on Interaction Design & Architecture(s) 32 (2017), 84103.Google ScholarGoogle Scholar
  19. [19] Sanchez Susana, Dingler Tilman, Gu Heng, and Kunze Kai. 2016. Embodied reading: A multisensory experience. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 14591466. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. [20] Sánchez-Azqueta Carlos, Gimeno Cecilia, Celma Santiago, and Aldea Concepción. 2016. Enhanced eBooks in the teaching/learning process of electronics. In 2nd International Conference on Higher Education Advances. 8491.Google ScholarGoogle Scholar
  21. [21] Shams L. and Seitz A. R.. 2008. Benefits of multisensory learning. Trends in Cognitive Sciences 12, 11 (2008), 411417. https://doi.org/10.1016/j.tics.2008.07.006Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Spence Charles. 2020. The multisensory experience of handling and reading books. Multisensory Research 33, 8 (2020), 902–928. DOI: https://doi.org/10.1163/22134808-bja10015Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Tal Irina, Zou Longhao, Covaci Alexandra, Ibarrola Eva, Bratu Marilena, Ghinea Gheorghita, and Muntean Gabriel-Miro. 2019. Mulsemedia in telecommunication and networking education: A novel teaching approach that improves the learning process. IEEE Communications Magazine 57, 11 (2019), 6066.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Tung T., Gomez R., Kawahara T., and Matsuyama T.. 2014. Multiparty interaction understanding using smart multimodal digital signage. IEEE Transactions on Human-Machine Systems 44, 5 (2014), 625637.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Vieira Natália, Pinto Anderson, Silva Felipe, Okuno Helder, Amorim Iury, Ramos Taymison, Haddad Diego, Amorim Glauco, Guedes Gustavo Paiva, and Santos Joel dos. 2018. Evaluating the influence of mulsemedia content in reading. In Proceedings of the 24th Brazilian Symposium on Multimedia and the Web. ACM, 133136. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Yuan Zhenhui, Ghinea Gheorghita, and Muntean Gabriel-Miro. 2015. Beyond multimedia adaptation: Quality of experience-aware multi-sensorial media delivery. IEEE Transactions on Multimedia 17, 1 (2015), 104117.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Zou Longhao, Tal Irina, Covaci Alexandra, Ibarrola Eva, Ghinea Gheorghita, and Muntean Gabriel-Miro. 2017. Can multisensorial media improve learner experience? In Proceedings of the 8th ACM on Multimedia Systems Conference. 315320.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Using Multisensory Content to Impact the Quality of Experience of Reading Digital Books

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 17, Issue 4
      November 2021
      529 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3492437
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 12 November 2021
      • Revised: 1 March 2021
      • Accepted: 1 March 2021
      • Received: 1 August 2020
      Published in tomm Volume 17, Issue 4

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed
    • Article Metrics

      • Downloads (Last 12 months)137
      • Downloads (Last 6 weeks)9

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!