Investigating Representation of Text and Audio in Educational VR using Learning Outcomes and EEG

ABSTRACT

This paper reports findings from a between-subjects experiment that investigates how different learning content representations in virtual environments (VE) affect the process and outcomes of learning. Seventy-eight participants were subjected to an immersive virtual reality (VR) application, where they received identical instructional information, rendered in three different formats: as text in an overlay interface, as text embedded semantically in a virtual book, or as audio. Learning outcome measures, self-reports, and an electroencephalogram (EEG) were used to compare conditions. Results show that reading was superior to listening for the learning outcomes of retention, self-efficacy, and extraneous attention. Reading text from a virtual book was reported to be less cognitively demanding, compared to reading from an overlay interface. EEG analyses show significantly lower theta and higher alpha activation in the audio condition. The findings provide important considerations for the design of educational VR environments.

References

1. Pavlo D. Antonenko and Andreas Keil. 2017. Assessing Working Memory Dynamics with Electroencephalography. Routledge, 93--111. DOI: http://dx.doi.org/10.4324/9781315296258--7
2. Pavlo Pasha Antonenko, Fred Paas, Roland Grabner, and Tamara Gog. 2010. Using Electroencephalography to Measure Cognitive Load. Educational Psychology Review 22 (12 2010), 425--438. DOI: http://dx.doi.org/10.1007/s10648-010--9130-y
3. Paul Ayres. 2006. Using subjective measures to detect variations of intrinsic cognitive load within problems. Learning and Instruction 16 (6 2006), 389--400. DOI: http://dx.doi.org/10.1016/j.learninstruc.2006.09.001
4. Robert J. Barry, Adam R. Clarke, Stuart J. Johnstone, Christopher A. Magee, and Jacqueline A. Rushby. 2007. EEG differences between eyes-closed and eyes-open resting conditions. Clinical Neurophysiology 118, 12 (2007), 2765 -- 2773. DOI: http://dx.doi.org/10.1016/j.clinph.2007.07.028
5. C. Berka, D. J. Levendowski, M. N. Lumicao, A. Yau, G. Davis, V. T. Zivkovic, R. E. Olmstead, P. D. Tremoulet, and P. L. Craven. 2007. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Journal of Aviation Space and Environmental Medicine 78, 5 (May 2007), B231--244.
6. Johanna Bertram, Johannes Moskaliuk, and Ulrike Cress. 2015. Virtual training: Making reality work? Computers in Human Behavior 43 (02 2015), 284--292. DOI: http://dx.doi.org/10.1016/j.chb.2014.10.032
7. Doug A. Bowman, Ajith Sowndararajan, Eric D. Ragan, and Regis Kopper. 2009. Higher Levels of Immersion Improve Procedure Memorization Performance. In Proceedings of the 15th Joint Virtual Reality Eurographics Conference on Virtual Environments (JVRC'09). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 121--128. DOI: http://dx.doi.org/10.2312/EGVE/JVRC09/121--128
8. Fabio Buttussi and Luca Chittaro. 2017. Effects of Different Types of Virtual Reality Display on Presence and Learning in a Safety Training Scenario. IEEE Transactions on Visualization and Computer Graphics PP (01 2017), 1--1. DOI: http://dx.doi.org/10.1109/TVCG.2017.2653117
9. Leidy Castro-Meneses, Jan-Louis Kruger, and Stephen Doherty. 2019. Validating theta power as an objective measure of cognitive load in educational video. Educational Technology Research and Development (06 2019). DOI: http://dx.doi.org/10.1007/s11423-019-09681--4
10. Luca Chittaro and Fabio Buttussi. 2015. Assessing Knowledge Retention of an Immersive Serious Game vs. a Traditional Education Method in Aviation Safety. Visualization and Computer Graphics, IEEE Transactions on 21 (04 2015), 529--538. DOI: http://dx.doi.org/10.1109/TVCG.2015.2391853
11. Gabriele Cierniak, Katharina Scheiter, and Peter Gerjets. 2009. Explaining the split-attention effect: Is the reduction of extraneous cognitive load accompanied by an increase in germane cognitive load? Computers in Human Behavior 25 (03 2009), 315--324. DOI: http://dx.doi.org/10.1016/j.chb.2008.12.020
12. Cody Cox, Andrew House, Alex Lopez, and Gregory Pool. 2017. The Need for Conceptual Models of Technology in Training and Development: How Immersive Does Training Need to Be? Industrial and Organizational Psychology 10 (12 2017), 696--701. DOI: http://dx.doi.org/10.1017/iop.2017.78
13. Tilman Dingler, Kai Kunze, and Benjamin Outram. 2018. VR Reading UIs: Assessing Text Parameters for Reading in VR. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, NY, NY, USA, Article LBW094, 6 pages. DOI: http://dx.doi.org/10.1145/3170427.3188695
14. Poul Dourish. 2001. Where the Action is: The Foundations of Embodied Interaction. MIT Press, Cambridge, MA, USA.
15. EngageVR. 2019. EngageVR. https://engagevr.io/. (2019). [Online; accessed 01-09--2019].
16. Chris Fowler. 2014. Virtual reality and learning: Where is the pedagogy?: Learning activities in 3-D virtual worlds. British Journal of Educational Technology 46 (02 2014). DOI: http://dx.doi.org/10.1111/bjet.12135
17. Anthony Gallagher and Christopher Cates. 2004. Virtual Reality Training for the Operating Room and Cardiac Catheterisation Laboratory. Lancet 364 (10 2004), 1538--40. DOI: http://dx.doi.org/10.1016/S0140--6736(04)17278--4
18. Sarah Garcia, Ronald Kauer, Denis Laesker, Jason Nguyen, and Marvin Andujar. 2019. A Virtual Reality Experience for Learning Languages. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). ACM, NY, NY, USA, Article INT039, 4 pages. DOI: http://dx.doi.org/10.1145/3290607.3313253
19. D.M. Halliday, J.R. Rosenberg, A.M. Amjad, P. Breeze, B.A. Conway, and S.F. Farmer. 1995. A framework for the analysis of mixed time series/point process data-Theory and application to the study of physiological tremor, single motor unit discharges and electromyograms. Progress in Biophysics and Molecular Biology 64, 2 (1995), 237--278. DOI: http://dx.doi.org/10.1016/S0079--6107(96)00009-0
20. Daniel Harley, Aneesh P. Tarun, Daniel Germinario, and Ali Mazalek. 2017. Tangible VR: Diegetic Tangible Objects for Virtual Reality Narratives. In Proceedings of the 2017 Conference on Designing Interactive Systems (DIS '17). ACM, NY, NY, USA, 1253--1263. DOI: http://dx.doi.org/10.1145/3064663.3064680
21. Lasse Jensen and Flemming Konradsen. 2018. A review of the use of virtual reality head-mounted displays in education and training. Education and Information Technologies 23, 4 (2018), 1515--1529. DOI: http://dx.doi.org/10.1007/s10639-017--9676-0
22. Cheryl Johnson and Richard Mayer. 2012. An Eye Movement Analysis of the Spatial Contiguity Effect in Multimedia Learning. Journal of Experimental Psychology 18 (02 2012), 178--91. DOI: http://dx.doi.org/10.1037/a0026923
23. Mina Khan, Fernando Trujano, Ashris Choudhury, and Pattie Maes. 2018. Mathland: Playful Mathematical Learning in Mixed Reality. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, NY, NY, USA, Article D108, 4 pages. DOI: http://dx.doi.org/10.1145/3170427.3186499
24. Pankaj Khanna, Insu Yu, Jesper Mortensen, and Mel Slater. 2006. Presence in Response to Dynamic Visual Realism: A Preliminary Report of an Experiment Study. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '06). ACM, NY, NY, USA, 364--367. DOI: http://dx.doi.org/10.1145/1180495.1180569
25. David Kirsh. 2013. Embodied Cognition and the Magical Future of Interaction Design. ACM Transactions on Computer-Human Interaction 20, 1, Article 3 (April 2013), 30 pages. DOI: http://dx.doi.org/10.1145/2442106.2442109
26. George H. Klem, H. Lüders, Herbert H. Jasper, and C Elger. 1958. The ten-twenty electrode system of the International Federation. The International Federation of Clinical Neurophysiology. Electroencephalography and clinical neurophysiology. Supplement 52 (1958), 3--6.
27. Wolfgang Klimesch. 1996. Memory processes, brain oscillations and EEG synchronization. International Journal of Psychophysiology 24, 1 (1996), 61 -- 100. DOI: http://dx.doi.org/10.1016/S0167--8760(96)00057--8 New Advances in EEG and cognition.
28. Wolfgang Klimesch. 1999. EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Research Reviews 29, 2 (1999), 169 -- 195. DOI: http://dx.doi.org/10.1016/S0165-0173(98)00056--3
29. Labster. 2019. Labster Simulations. https://www.labster.com/simulations/. (2019). [Online; accessed 01-09--2019].
30. Colin Lankshear and Michele Knobel. 2003. New Literacies: Changing Knowledge and Classroom Learning (1 ed.). Open University Press.
31. Marc Erich Latoschik, Daniel Roth, Dominik Gall, Jascha Achenbach, Thomas Waltemate, and Mario Botsch. 2017. The Effect of Avatar Realism in Immersive Social Virtual Realities. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (VRST '17). ACM, NY, NY, USA, Article 39, 10 pages. DOI: http://dx.doi.org/10.1145/3139131.3139156
32. Robb Lindgren, Michael Tscholl, Shuai Wang, and Emily Johnson. 2016. Enhancing Learning and Engagement through Embodied Interaction within a Mixed Reality Simulation. Computers & Education 95, C (April 2016), 174--187. DOI: http://dx.doi.org/10.1016/j.compedu.2016.01.001
33. BF Lu, KT Lim, JM Zheng, and YY Cai. 2004. Learning Molecular Biology by VR Playing. In Proceedings of the 2004 ACM SIGGRAPH International Conference on Virtual Reality Continuum and Its Applications in Industry (VRCAI '04). ACM, NY, NY, USA, 168--172. DOI: http://dx.doi.org/10.1145/1044588.1044622
34. Guido Makransky, Stefan Borre-Gude, and Richard E. Mayer. 2019a. Motivational and cognitive benefits of training in immersive virtual reality based on multiple assessments. Journal of Computer Assisted Learning (2019). DOI: http://dx.doi.org/10.1111/jcal.12375
35. Guido Makransky and Gustav Bøg Petersen. 2019. Investigating the process of learning with desktop virtual reality: A structural equation modeling approach. Computers & Education 134 (2019), 15 -- 30. DOI: http://dx.doi.org/https://doi.org/10.1016/j.compedu.2019.02.002
36. Guido Makransky, Thomas S. Terkildsen, and Richard E. Mayer. 2019b. Role of subjective and objective measures of cognitive processing during learning in explaining the spatial contiguity effect. Learning and Instruction 61 (1 6 2019), 23--34. DOI: http://dx.doi.org/10.1016/j.learninstruc.2018.12.001
37. Guido Makransky, Thomas S. Terkildsen, and Richard E. Mayer. 2019c. Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learning and Instruction 60 (2019), 225 -- 236. DOI: http://dx.doi.org/10.1016/j.learninstruc.2017.12.007
38. Guido Makransky, Philip Wismer, and Richard E. Mayer. 2019d. A gender matching effect in learning with pedagogical agents in an immersive virtual reality science simulation. Journal of Computer Assisted Learning 35, 3 (2019), 349--358. DOI: http://dx.doi.org/10.1111/jcal.12335
39. Anne Mangen and Theresa Schilhab. 2012. An embodied view of reading: Theoretical considerations, empirical findings, and educational implications. Skriv! Les! (2012), 285--300.
40. Richard E. Mayer. 2014. The Cambridge Handbook of Multimedia Learning (2 ed.). Cambridge University Press. DOI: http://dx.doi.org/10.1017/CBO9781139547369
41. Richard E. Mayer and Richard B. Anderson. 1992. The Instructive Animation: Helping Students Build Connections between Words and Pictures in Multimedia Learning. Journal of Educational Psychology 84, 4 (1992), 444.
42. Zahira Merchant, Ernest T. Goetz, Lauren Cifuentes, Wendy Keeney-Kennicutt, and Trina J. Davis. 2014. Effectiveness of Virtual Reality-based Instruction on Students' Learning Outcomes in K-12 and Higher Education. Computers & Education 70 (Jan. 2014), 29--40. DOI: http://dx.doi.org/10.1016/j.compedu.2013.07.033
43. Oliver A. Meyer, Magnus K. Omdahl, and Guido Makransky. 2019. Investigating the effect of pre-training when learning through immersive virtual reality and video: A media and methods experiment. Computers & Education 140 (2019), 103603. DOI: http://dx.doi.org/10.1016/j.compedu.2019.103603
44. Tassos A. Mikropoulos and Antonis Natsis. 2011. Educational Virtual Environments: A Ten-year Review of Empirical Research (1999--2009). Computers & Educaction 56, 3 (April 2011), 769--780. DOI: http://dx.doi.org/10.1016/j.compedu.2010.10.020
45. Roxana Moreno and Richard E. Mayer. 2002. Verbal Redundancy in Multimedia Learning: When Reading Helps Listening. Journal of Educational Psychology 94, 1 (2002). DOI: http://dx.doi.org/10.1037/0022-0663.94.1.156
46. Johannes Moskaliuk, Johanna Bertram, and Ulrike Cress. 2012. Training in virtual environments: Putting theory into practice. Ergonomics 56 (12 2012). DOI: http://dx.doi.org/10.1080/00140139.2012.745623
47. Matthias Oberhauser and Daniel Dreyer. 2017. A Virtual Reality Flight Simulator for Human Factors Engineering. Cogn. Technol. Work 19, 2--3 (Sept. 2017), 263--277. DOI: http://dx.doi.org/10.1007/s10111-017-0421--7
48. Daria Osipova, Atsuko Takashima, Robert Oostenveld, Guillén Fernández, Eric Maris, and Ole Jensen. 2006. Theta and Gamma Oscillations Predict Encoding and Retrieval of Declarative Memory. Journal of Neuroscience 26, 28 (2006), 7523--7531. DOI: http://dx.doi.org/10.1523/JNEUROSCI.1948-06.2006
49. Fred Paas. 1992. Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: A Cognitive-Load Approach. Journal of Educational Psychology 84 (12 1992), 429--434. DOI: http://dx.doi.org/10.1037/0022-0663.84.4.429
50. Paul R Pintrich, David A F Smith, Teresa Garcia, and Wilbert James McKeachie. 1991. A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ). National Center for Research to Improve Postsecondary Teaching and Learning (1991).
51. Sébastien Puma, Nadine Matton, Pierre-Vincent Paubel, Eric Raufaste, and Radouane Yagoubi. 2017. Using theta and alpha band power to assess cognitive workload in multitasking environments. International journal of psychophysiology : official journal of the International Organization of Psychophysiology 123 (10 2017). DOI: http://dx.doi.org/10.1016/j.ijpsycho.2017.10.004
52. Pei-Luen Patrick Rau, Jian Zheng, Zhi Guo, and Jiaqi Li. 2018. Speed reading on virtual reality and augmented reality. Computers & Education 125 (2018), 240 -- 245. DOI: http://dx.doi.org/10.1016/j.compedu.2018.06.016
53. W. J. Ray and H. W. Cole. 1985. EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science 228, 4700 (May 1985), 750--752.
54. Giuseppe Riva. 2008. From virtual to real body: Virtual reality as embodied technology. Journal of Cyber Therapy and Rehabilitation 1 (01 2008), 7--22.
55. Katja Rogers, Jana Funke, Julian Frommel, Sven Stamm, and Michael Weber. 2019. Exploring Interaction Fidelity in Virtual Reality: Object Manipulation and Whole-Body Movements. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 414, 14 pages. DOI: http://dx.doi.org/10.1145/3290605.3300644
56. Gavriel Salomon. 1984. Television is easy and print is tough. The differential investment of mental effort in learning as a function of perceptions and attributions. Journal of Educational Psychology 76 (1984), 647--658. DOI: http://dx.doi.org/10.1037/0022-0663.76.4.647
57. Christian Scharinger. 2018. Fixation-related EEG frequency band power analysis: A promising methodology for studying instructional design effects of multimedia learning material. Frontline Learning Research 6 (12 2018), 57--71. DOI: http://dx.doi.org/10.14786/flr.v6i3.373
58. Jinsil Hwaryoung Seo, Brian Michael Smith, Margaret E. Cook, Erica R. Malone, Michelle Pine, Steven Leal, Zhikun Bai, and Jinkyo Suh. 2017. Anatomy Builder VR: Embodied VR Anatomy Learning Program to Promote Constructionist Learning. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). ACM, NY, NY, USA, 2070--2075. DOI: http://dx.doi.org/10.1145/3027063.3053148
59. Daniel Sjölie, Kenneth Bodin, Eva Elgh, Johan Eriksson, Lars-Erik Janlert, and Lars Nyberg. 2010. Effects of Interactivity and 3D-motion on Mental Rotation Brain Activity in an Immersive Virtual Environment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, NY, NY, USA, 869--878. DOI: http://dx.doi.org/10.1145/1753326.1753454
60. Charles Spence. 2009. Explaining the Colavita visual dominance effect. In Attention, Narayanan Srinivasan (Ed.). Progress in Brain Research, Vol. 176. Elsevier, 245 -- 258. DOI: http://dx.doi.org/10.1016/S0079--6123(09)17615-X
61. Misha Sra, Xuhai Xu, Aske Mottelson, and Pattie Maes. 2018. VMotion: Designing a Seamless Walking Experience in VR. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS '18). ACM, NY, NY, USA, 59--70. DOI: http://dx.doi.org/10.1145/3196709.3196792
62. John Sweller. 1994. Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction 4, 4 (1994), 295--312. DOI: http://dx.doi.org/10.1016/0959--4752(94)90003--5
63. Rustin Webster. 2015. Declarative knowledge acquisition in immersive virtual learning environments. Interactive Learning Environments (01 2015), 1--15. DOI: http://dx.doi.org/10.1080/10494820.2014.994533
64. William Winn. 2003. Learning in Artificial Environments: Embodiment, Embeddedness and Dynamic Adaptation. Technology, Instruction, Cognition and Learning 1 (01 2003).
65. Mihye Won, M. Mocerino, Kok-Sing Tang, David Treagust, and Roy Tasker. 2019. Interactive Immersive Virtual Reality to Enhance Students' Visualisation of Complex Molecules. Springer, Singapore, 51--64. DOI: http://dx.doi.org/10.1007/978--981--13--6998--8_4
66. Anna Wong, Wayne Leahy, Nadine Marcus, and John Sweller. 2012. Cognitive load theory, the transient information effect and e-learning. Learning and Instruction 22, 6 (2012), 449 -- 457. DOI: http://dx.doi.org/10.1016/j.learninstruc.2012.05.004

Index Terms

• Applied computing

  • Education

    • Interactive learning environments

• Human-centered computing

  • Human computer interaction (HCI)

    • Empirical studies in HCI

    • Interaction paradigms

      • Virtual reality