Exploring Machine Learning Approaches for Classifying Mental Workload using fNIRS Data from HCI Tasks

ABSTRACT

Functional Near-Infrared Spectroscopy (fNIRS) has shown promise for being potentially more suitable (than e.g. EEG) for brain-based Human Computer Interaction (HCI). While some machine learning approaches have been used in prior HCI work, this paper explores different approaches and configurations for classifying Mental Workload (MWL) from a continuous HCI task, to identify and understand potential limitations and data processing decisions. In particular, we investigate three overall approaches: a logistic regression method, a supervised shallow method (SVM), and a supervised deep learning method (CNN). We examine personalised and generalised models, as well as consider different features and ways of labelling the data. Our initial explorations show that generalised models can perform as well as personalised ones and that deep learning can be a suitable approach for medium size datasets. To provide additional practical advice for future brain-computer interaction systems, we conclude by discussing the limitations and data-preparation needs of different machine learning approaches. We also make recommendations for avenues of future work that are most promising for the machine learning of fNIRS data.

References

1. Daniel Afergan, Evan M Peck, Erin T Solovey, Andrew Jenkins, Samuel W Hincks, Eli T Brown, Remco Chang, and Robert JK Jacob. 2014. Dynamic difficulty using brain metrics of workload. In Proc. SIGCHI. ACM, 3797–3806.
2. Sangtae Ahn, Thien Nguyen, Hyojung Jang, Jae G Kim, and Sung C Jun. 2016. Exploring neuro-physiological correlates of drivers’ mental fatigue caused by sleep deprivation using simultaneous EEG, ECG, and fNIRS data. Frontiers in human neuroscience 10 (2016), 219.
3. Aurélien Appriou, Andrzej Cichocki, and Fabien Lotte. 2018. Towards Robust Neuroadaptive HCI: Exploring Modern Machine Learning Methods to Estimate Mental Workload From EEG Signals. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, LBW615.
4. Hasan Ayaz and Banu Onaral. 2005. Analytical software and stimulus-presentation platform to utilize, visualize and analyze near-infrared spectroscopy measures. Drexel University.
5. Jackson Beatty. 1982. Task-evoked pupillary responses, processing load, and the structure of processing resources.Psychological bulletin 91, 2 (1982), 276.
6. Chris Berka, Daniel J Levendowski, Michelle N Lumicao, Alan Yau, Gene Davis, Vladimir T Zivkovic, Richard E Olmstead, Patrice D Tremoulet, and Patrick L Craven. 2007. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, space, and environmental medicine 78, 5 (2007), B231–B244.
7. George E Billman. 2011. Heart rate variability–a historical perspective. Frontiers in physiology 2 (2011).
8. Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. 1992. A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory. ACM, 144–152.
9. SD Brennan. 1992. An experimental report on rating scale descriptor sets for the instantaneous self assessment (ISA) recorder. Portsmouth: DRA Maritime Command and Control Division. DRA Technical Memorandum (CAD5) 92017(1992).
10. Siyuan Chen and Julien Epps. 2013. Automatic classification of eye activity for cognitive load measurement with emotion interference. Computer methods and programs in biomedicine 110, 2(2013), 111–124.
11. C Collet, E Salvia, and C Petit-Boulanger. 2014. Measuring workload with electrodermal activity during common braking actions. Ergonomics 57, 6 (2014), 886–896.
12. Xu Cui, Signe Bray, Daniel M Bryant, Gary H Glover, and Allan L Reiss. 2011. A quantitative comparison of NIRS and fMRI across multiple cognitive tasks. Neuroimage 54, 4 (2011), 2808–2821.
13. Xu Cui, Signe Bray, and Allan L Reiss. 2010. Functional near infrared spectroscopy (NIRS) signal improvement based on negative correlation between oxygenated and deoxygenated hemoglobin dynamics. Neuroimage 49, 4 (2010), 3039–3046.
14. Lex Fridman, Bryan Reimer, Bruce Mehler, and William T. Freeman. 2018. Cognitive Load Estimation in the Wild. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems(CHI ’18). ACM, New York, NY, USA, Article 652, 9 pages. https://doi.org/10.1145/3173574.3174226
15. Audrey Girouard, Erin Treacy Solovey, Leanne M Hirshfield, Evan M Peck, Krysta Chauncey, Angelo Sassaroli, Sergio Fantini, and Robert JK Jacob. 2010. From brain signals to adaptive interfaces: using fNIRS in HCI. In Brain-Computer Interfaces. Springer, 221–237.
16. Audrey Girouard, Erin Treacy Solovey, and Robert JK Jacob. 2013. Designing a passive brain computer interface using real time classification of functional near-infrared spectroscopy. International Journal of Autonomous and Adaptive Communications Systems 6, 1(2013), 26–44.
17. Eija Haapalainen, SeungJun Kim, Jodi F Forlizzi, and Anind K Dey. 2010. Psycho-physiological measures for assessing cognitive load. In Proceedings of the 12th ACM international conference on Ubiquitous computing. ACM, 301–310.
18. Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human mental workload(1988).
19. Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998. Support vector machines. IEEE Intelligent Systems and their applications 13, 4(1998), 18–28.
20. Yoko Hoshi, Brian H Tsou, Vincent A Billock, Masato Tanosaki, Yoshinobu Iguchi, Miho Shimada, Toshikazu Shinba, Yoshifumi Yamada, and Ichiro Oda. 2003. Spatiotemporal characteristics of hemodynamic changes in the human lateral prefrontal cortex during working memory tasks. Neuroimage 20, 3 (2003), 1493–1504.
21. Shamsi T Iqbal, Xianjun Sam Zheng, and Brian P Bailey. 2004. Task-evoked pupillary response to mental workload in human-computer interaction. In CHI’04 extended abstracts on Human factors in computing systems. ACM, 1477–1480.
22. Jordan and Brennen. 1992. Instantaneous self-assessment of workload technique (ISA). Retrieved from http://www.skybrary.aero/bookshelf/books/1963.pdf. (1992).
23. CS Jordan. 1992. Experimental study of the effects of an instantaneous self assessment workload recorder on task performance. Report No. DRA/TM (CAD5)/92011. Farnborough: Defence Evaluation & Research Agency (1992).
24. Kristiyan Lukanov, Horia A Maior, and Max L Wilson. 2016. Using fNIRS in usability testing: understanding the effect of web form layout on mental workload. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4011–4016.
25. Horia A Maior, Matthew Pike, Sarah Sharples, and Max L Wilson. 2015. Examining the reliability of using fNIRS in realistic hci settings for spatial and verbal tasks. In Proceedings of CHI, Vol. 15. 3807–3816.
26. Horia A Maior, Matthew Pike, Max L Wilson, and Sarah Sharples. 2014. Continuous detection of workload overload: an FNIRS approach. In Contemporary Ergonomics and Human Factors 2014: Proceedings of the international conference on Ergonomics & Human Factors 2014, Southampton, UK, 7-10 April 2014. CRC Press, 450.
27. Horia A Maior, Max L Wilson, and Sarah Sharples. 2018. Workload Alerts - Using Physiological Measures of Mental Workload to Provide Feedback During Tasks. ACM Transactions on Computer-Human Interaction (TOCHI) 25, 2(2018), 9.
28. Adrian Cornelius Marinescu. 2018. Facial thermography for assessment of workload in safety critical environments. Ph.D. Dissertation. University of Nottingham.
29. Adrian Cornelius Marinescu, Sarah Sharples, Alastair Campbell Ritchie, Tomas Sánchez López, Michael McDowell, and Hervé P Morvan. 2017. Physiological parameter response to variation of mental workload. Human factors (2017), 0018720817733101.
30. Sandra P Marshall. 2007. Identifying cognitive state from eye metrics. Aviation, space, and environmental medicine 78, 5 (2007), B165–B175.
31. Peter Nickel and Friedhelm Nachreiner. 2003. Sensitivity and diagnosticity of the 0.1-Hz component of heart rate variability as an indicator of mental workload. Human Factors: The Journal of the Human Factors and Ergonomics Society 45, 4(2003), 575–590.
32. Ryota Nishiyori. 2016. fNIRS: An emergent method to document functional cortical activity during infant movements. Frontiers in psychology 7 (2016), 533.
33. Fred Paas, Alexander Renkl, and John Sweller. 2003. Cognitive load theory and instructional design: Recent developments. Educational psychologist 38, 1 (2003), 1–4.
34. Matthew Pike, Max L Wilson, Steve Benford, and Richard Ramchurn. 2016. # Scanners: A BCI Enhanced Cinematic Experience. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 293–296.
35. Matthew F Pike, Horia A Maior, Martin Porcheron, Sarah C Sharples, and Max L Wilson. 2014. Measuring the effect of think aloud protocols on workload using fNIRS. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 3807–3816.
36. Richard Ramchurn, Max L Wilson, Sarah Martindale, and Steve Benford. 2018. # Scanners 2-The MOMENT: A New Brain-Controlled Movie. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, D210.
37. Angelo Sassaroli, Feng Zheng, Leanne M Hirshfield, Audrey Girouard, Erin Treacy Solovey, Robert JK Jacob, and Sergio Fantini. 2008. Discrimination of mental workload levels in human subjects with functional near-infrared spectroscopy. Journal of Innovative Optical Health Sciences 1, 02 (2008), 227–237.
38. Sarah Sharples and Ted Megaw. 2015. Definition and Mesurement of Human Workload. In Evaluation of human work, John R Wilson and Sarah Sharples (Eds.). CRC Press.
39. Yu Shi, Natalie Ruiz, Ronnie Taib, Eric Choi, and Fang Chen. 2007. Galvanic skin response (GSR) as an index of cognitive load. In CHI’07 extended abstracts on Human factors in computing systems. ACM, 2651–2656.
40. Yoshihiro Shimomura, Takumi Yoda, Koji Sugiura, Akinori Horiguchi, Koichi Iwanaga, and Tetsuo Katsuura. 2008. Use of frequency domain analysis of skin conductance for evaluation of mental workload. Journal of physiological anthropology 27, 4 (2008), 173–177.
41. Erin Solovey, Paul Schermerhorn, Matthias Scheutz, Angelo Sassaroli, Sergio Fantini, and Robert Jacob. 2012. Brainput: enhancing interactive systems with streaming fNIRs brain input. In CHI. ACM, 2193–2202.
42. Erin Treacy Solovey, Audrey Girouard, Krysta Chauncey, Leanne M Hirshfield, Angelo Sassaroli, Feng Zheng, Sergio Fantini, and Robert JK Jacob. 2009. Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines. In Proc. UIST. ACM, 157–166.
43. Erin T Solovey, Marin Zec, Enrique Abdon Garcia Perez, Bryan Reimer, and Bruce Mehler. 2014. Classifying driver workload using physiological and driving performance data: two field studies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 4057–4066.
44. Kazushi Taoda, Masanori Kawamura, Kinzou Wakara, Yasuma Fukuchi, and Katsuo Nishiyama. 2001. Heart rate variability during long truck driving work. Journal of human ergology 30, 1-2 (2001), 235–240.
45. Andrew J Tattersall and Penelope S Foord. 1996. An experimental evaluation of instantaneous self-assessment as a measure of workload. Ergonomics 39, 5 (1996), 740–748.
46. Arno Villringer and Britton Chance. 1997. Non-invasive optical spectroscopy and imaging of human brain function. Trends in neurosciences 20, 10 (1997), 435–442.
47. Christopher D Wickens. 2008. Multiple resources and mental workload. The Journal of the Human Factors and Ergonomics Society 50, 3(2008), 449–455.
48. Beste F Yuksel, Kurt B Oleson, Lane Harrison, Evan M Peck, Daniel Afergan, Remco Chang, and Robert JK Jacob. 2016. Learn piano with BACh: An adaptive learning interface that adjusts task difficulty based on brain state. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5372–5384.
49. Yilu Zhang, Yuri Owechko, and Jing Zhang. 2004. Driver cognitive workload estimation: A data-driven perspective. In Intelligent Transportation Systems. Citeseer, 642–647.

Index Terms

• Computing methodologies

  • Machine learning

    • Machine learning approaches

• Human-centered computing

  • Human computer interaction (HCI)