Guohao Lan
ABSTRACT
In this work, we present GazeGraph, a system that leverages human gazes as the sensing modality for cognitive context sensing. GazeGraph is a generalized framework that is compatible with different eye trackers and supports various gaze-based sensing applications. It ensures high sensing performance in the presence of heterogeneity of human visual behavior, and enables quick system adaptation to unseen sensing scenarios with few-shot instances. To achieve these capabilities, we introduce the spatial-temporal gaze graphs and the deep learning-based representation learning method to extract powerful and generalized features from the eye movements for context sensing. Furthermore, we develop a few-shot gaze graph learning module that adapts the `learning to learn' concept from meta-learning to enable quick system adaptation in a data-efficient manner. Our evaluation demonstrates that GazeGraph outperforms the existing solutions in recognition accuracy by 45% on average over three datasets. Moreover, in few-shot learning scenarios, GazeGraph outperforms the transfer learning-based approach by 19% to 30%, while reducing the system adaptation time by 80%.
- A. Bulling and T. O. Zander, "Cognition-aware computing," IEEE Pervasive Computing, vol. 13, no. 3, pp. 80--83, 2014.Google Scholar
Cross Ref
- A. Bulling, D. Roggen, and G. Troester, "What's in the eyes for context-awareness?" IEEE Pervasive Computing, vol. 10, no. 2, pp. 48--57, 2010.Google ScholarDigital Library
- A. Bulling and D. Roggen, "Recognition of visual memory recall processes using eye movement analysis," in Proceedings of ACM UbiComp, 2011.Google Scholar
- K. Kassem, J. Salah, Y. Abdrabou, M. Morsy, R. El-Gendy, Y. Abdelrahman, and S. Abdennadher, "DiVA: Exploring the usage of pupil diameter to elicit valence and arousal," in Proceedings of ACM MUM, 2017.Google ScholarDigital Library
- B. Pfleging, D. K. Fekety, A. Schmidt, and A. L. Kun, "A model relating pupil diameter to mental workload and lighting conditions," in Proceedings of ACM CHI, 2016.Google Scholar
- A. Bulling, J. A. Ward, H. Gellersen, and G. Troster, "Eye movement analysis for activity recognition using electrooculography," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 4, pp. 741--753, 2010.Google ScholarDigital Library
- O. Augereau, C. L. Sanches, K. Kise, and K. Kunze, "Wordometer systems for everyday life," Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 1, no. 4, p. 123, 2018.Google ScholarDigital Library
- J. Karolus, P. W. Wozniak, L. L. Chuang, and A. Schmidt, "Robust gaze features for enabling language proficiency awareness," in Proceedings of ACM CHI, 2017.Google Scholar
- T. Kosch, M. Hassib, P. W. Wozniak, D. Buschek, and F. Alt, "Your eyes tell: Leveraging smooth pursuit for assessing cognitive workload," in Proceedings of ACM CHI, 2018.Google ScholarDigital Library
- A. T. Duchowski, K. Krejtz, I. Krejtz, C. Biele, A. Niedzielska, P. Kiefer, M. Raubal, and I. Giannopoulos, "The index of pupillary activity: Measuring cognitive load vis-à-vis task difficulty with pupil oscillation," in Proceedings of ACM CHI, 2018.Google ScholarDigital Library
- "Laptops that are integrated with eye tracking," https://gaming.tobii.com/products/laptops/.Google Scholar
- P. Norloff. Eye tracking technology is making new cars safer. [Online]. Available: https://eyegaze.com/eye-tracking-technology-is-making-new-cars-safer/Google Scholar
- Tobii. Eye tracking for driver safety. [Online]. Available: https://www.tobiipro.com/fields-of-use/psychology-and-neuroscience/customer-cases/audi-attitudes/Google Scholar
- N. Srivastava, J. Newn, and E. Velloso, "Combining low and mid-level gaze features for desktop activity recognition," Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 2, no. 4, p. 189, 2018.Google ScholarDigital Library
- K. Kunze, Y. Utsumi, Y. Shiga, K. Kise, and A. Bulling, "I know what you are reading: Recognition of document types using mobile eye tracking," in Proceedings of ACM ISWC, 2013.Google ScholarDigital Library
- Y. Li, P. Xu, D. Lagun, and V. Navalpakkam, "Towards measuring and inferring user interest from gaze," in Proceedings of ACM WWW, 2017.Google Scholar
- D. Lagun, C.-H. Hsieh, D. Webster, and V. Navalpakkam, "Towards better measurement of attention and satisfaction in mobile search," in Proceedings of ACM SIGIR, 2014.Google Scholar
- M. Sugiyama and A. J. Storkey, "Mixture regression for covariate shift," in Proceedings of NeurIPS, 2007.Google Scholar
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," in Proceedings of IEEE CVPR, 2015.Google Scholar
- S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. 8, pp. 1735--1780, 1997.Google ScholarDigital Library
- Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436--444, 2015.Google ScholarDigital Library
- C. Katsini, H. Opsis, Y. Abdrabou, G. E. Raptis, M. Khamis, and F. Alt, "The role of eye gaze in security and privacy applications: Survey and future HCI research directions," in Proceedings of ACM CHI, 2020.Google Scholar
- A. Liu, L. Xia, A. Duchowski, R. Bailey, K. Holmqvist, and E. Jain, "Differential privacy for eye-tracking data," in Proceedings of ACM ETRA, 2019.Google Scholar
- J. Steil, I. Hagestedt, M. X. Huang, and A. Bulling, "Privacy-aware eye tracking using differential privacy," in Proceedings of ACM ETRA, 2019.Google Scholar
- F.-F. Li, R. Fergus, and P. Perona, "One-shot learning of object categories," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 4, pp. 594--611, 2006.Google ScholarDigital Library
- W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, and J.-B. Huang, "A closer look at few-shot classification," in Proceedings of ICLR, 2019.Google Scholar
- S. Thrun, "Lifelong learning algorithms," in Learning to learn. Springer, 1998, pp. 181--209.Google Scholar
- C. Finn, P. Abbeel, and S. Levine, "Model-agnostic meta-learning for fast adaptation of deep networks," in Proceedings of ICML, 2017.Google Scholar
- S. Ravi and H. Larochelle, "Optimization as a model for few-shot learning," in Proceedings of ICLR, 2016.Google Scholar
- M. A. Jamal and G.-J. Qi, "Task agnostic meta-learning for few-shot learning," in Proceedings of IEEE CVPR, 2019.Google Scholar
- D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales, "Learning to generalize: Meta-learning for domain generalization" in Proceedings of AAAI, 2018.Google Scholar
- S. A. Rokni, M. Nourollahi, and H. Ghasemzadeh, "Personalized human activity recognition using convolutional neural networks," in Proceedings of AAAI, 2018.Google Scholar
- J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?" in Proceedings of NeurIPS, 2014.Google Scholar
- J. Steil and A. Bulling, "Discovery of everyday human activities from long-term visual behaviour using topic models," in Proceedings of the ACM UbiComp, 2015.Google Scholar
- P. Kiefer, I. Giannopoulos, and M. Raubal, "Using eye movements to recognize activities on cartographic maps," in Proceedings of ACM SIGSPATIAL, 2013.Google Scholar
- K. Kunze, K. Masai, M. Inami, Ö. Sacakli, M. Liwicki, A. Dengel, S. Ishimaru, and K. Kise, "Quantifying reading habits: counting how many words you read," in Proceedings of ACM UbiComp, 2015.Google ScholarDigital Library
- J. Nie, Y. Hu, Y. Wang, S. Xia, and X. Jiang, "SPIDERS: Low-cost wireless glasses for continuous in-situ bio-signal acquisition and emotion recognition" in Proceedings of IEEE/ACM IoTDI, 2020.Google Scholar
- H. Wu, J. Feng, X. Tian, E. Sun, Y. Liu, B. Dong, F. Xu, and S. Zhong, "EMO: Real-time emotion recognition from single-eye images for resource-constrained eyewear devices," in Proceedings of ACM MobiSys, 2020.Google Scholar
- W. L. Hamilton, R. Ying, and J. Leskovec, "Representation learning on graphs: Methods and applications," IEEE Data Engineering Bulletin, 2017.Google Scholar
- P. Cui, X. Wang, J. Pei, and W. Zhu, "A survey on network embedding," IEEE Transactions on Knowledge and Data Engineering, vol. 31, no. 5, pp. 833--852, 2019.Google Scholar
- V. Gligorijević, M. Barot, and R. Bonneau, "deepNF: Deep network fusion for protein function prediction," Bioinformatics, vol. 34, no. 22, pp. 3873--3881, 2018.Google Scholar
- D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams, "Convolutional networks on graphs for learning molecular fingerprints," in Proceedings of NeurIPS, 2015.Google Scholar
- L. Yao, C. Mao, and Y. Luo, "Graph convolutional networks for text classification," in Proceedings of AAAI, 2019.Google Scholar
- S. Yan, Y. Xiong, and D. Lin, "Spatial temporal graph convolutional networks for skeleton-based action recognition," in Proceedings of AAAI, 2018.Google Scholar
- A. Jain, A. R. Zamir, S. Savarese, and A. Saxena, "Structural-RNN: Deep learning on spatio-temporal graphs," in Proceedings of IEEE CVPR, 2016.Google Scholar
- A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell, "Meta-learning with latent embedding optimization," in Proceedings of ICLR, 2019.Google Scholar
- T. Gong, Y. Kim, J. Shin, and S.-J. Lee, "MetaSense: Few-shot adaptation to untrained conditions in deep mobile sensing," in Proceedings of ACM SenSys, 2019.Google Scholar
- "How to use eye tracking on your Alienware 17 R4," https://gaming.tobii.com/onboarding/alienware17-eye-tracking-how-to/.Google Scholar
- "Acer Predator 21x," https://gaming.tobii.com/product/acer-predator-21x/.Google Scholar
- "Magic Leap," https://www.magicleap.com/.Google Scholar
- "Tobii Pro VR integration," https://www.tobiipro.com/product-listing/vr-integration/.Google Scholar
- J. Sigut and S.-A. Sidha, "Iris center corneal reflection method for gaze tracking using visible light," IEEE Transactions on Biomedical Engineering, vol. 58, no. 2, pp. 411--419, 2010.Google ScholarCross Ref
- M. Kassner, W. Patera, and A. Bulling, "Pupil: An open source platform for pervasive eye tracking and mobile gaze-based interaction," in Proceedings of ACM UbiComp, 2014.Google ScholarDigital Library
- M. Tonsen, J. Steil, Y. Sugano, and A. Bulling, "InvisibleEye: Mobile eye tracking using multiple low-resolution cameras and learning-based gaze estimation," Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 1, no. 3, p. 106, 2017.Google ScholarDigital Library
- "Pupil Labs eye tracker," https://pupil-labs.com/.Google Scholar
- D. A. Robinson, "A method of measuring eye movement using a scleral search coil in a magnetic field," IEEE Transactions on Biomedical Electronics, vol. 10, no. 4, pp. 137--145, 1963.Google ScholarCross Ref
- A. T. Duchowski, "A breadth-first survey of eye-tracking applications," Behavior Research Methods, Instruments, & Computers, vol. 34, no. 4, pp. 455--470, 2002.Google ScholarCross Ref
- M. Khamis, F. Alt, and A. Bulling, "The past, present, and future of gaze-enabled handheld mobile devices: Survey and lessons learned," in Proceedings of ACM MobiHCI, 2018.Google ScholarDigital Library
- I. Goodfellow, Y. Bengio, and A. Courville, Deep learning. MIT press, 2016.Google ScholarDigital Library
- T. Plötz and Y. Guan, "Deep learning for human activity recognition in mobile computing," Computer, vol. 51, no. 5, pp. 50--59, 2018.Google ScholarCross Ref
- Y. Guan and T. Plötz, "Ensembles of deep LSTM learners for activity recognition using wearables," Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 1, no. 2, pp. 1--28, 2017.Google ScholarDigital Library
- Z. Jia, X. Lyu, W. Zhang, R. P. Martin, R. E. Howard, and Y. Zhang, "Continuous low-power ammonia monitoring using long short-term memory neural networks," in Proceedings of ACM SenSys, 2018.Google Scholar
- "SMI eye tracking glasses," https://imotions.com/hardware/smi-eye-tracking-glasses/.Google Scholar
- "Tobii Pro X2 eye tracker," https://www.tobiipro.com/product-listing/tobii-pro-x2-30/.Google Scholar
- E. N. Ussery, J. E. Fulton, D. A. Galuska, P. T. Katzmarzyk, and S. A. Carlson, "Joint prevalence of sitting time and leisure-time physical activity among US adults, 2015--2016," Jama, vol. 320, no. 19, pp. 2036--2038, 2018.Google ScholarCross Ref
- D. Lagun, C. Manzanares, S. M. Zola, E. A. Buffalo, and E. Agichtein, "Detecting cognitive impairment by eye movement analysis using automatic classification algorithms," Journal of Neuroscience Methods, vol. 201, no. 1, pp. 196--203, 2011.Google ScholarCross Ref
- A. García-Blanco, L. Salmerón, M. Perea, and L. Livianos, "Attentional biases toward emotional images in the different episodes of bipolar disorder: An eye-tracking study," Psychiatry Research, vol. 215, no. 3, pp. 628--633, 2014.Google ScholarCross Ref
- S. Wang, M. Jiang, X. M. Duchesne, E. A. Laugeson, D. P. Kennedy, R. Adolphs, and Q. Zhao, "Atypical visual saliency in autism spectrum disorder quantified through model-based eye tracking," Neuron, vol. 88, no. 3, pp. 604--616, 2015.Google Scholar
- K. Harezlak and P. Kasprowski, "Application of eye tracking in medicine: A survey, research issues and challenges," Computerized Medical Imaging and Graphics, vol. 65, pp. 176--190, 2018.Google Scholar
- "HP Omnicept," https://www8.hp.com/us/en/vr/reverb-g2-vr-headset-omnicept-edition.html.Google Scholar
- R. S. Khan, G. Tien, M. S. Atkins, B. Zheng, O. N. Panton, and A. T. Meneghetti, "Analysis of eye gaze: Do novice surgeons look at the same location as expert surgeons during a laparoscopic operation?" Surgical Endoscopy, vol. 26, no. 12, pp. 3536--3540, 2012.Google ScholarCross Ref
- A. Burova, J. Mäkelä, J. Hakulinen, T. Keskinen, H. Heinonen, S. Siltanen, and M. Turunen, "Utilizing VR and gaze tracking to develop AR solutions for industrial maintenance," in Proceedings of ACM CHI, 2020.Google Scholar
- T. Blascheck, K. Kurzhals, M. Raschke, M. Burch, D. Weiskopf, and T. Ertl, "State-of-the-art of visualization for eye tracking data," in Proceedings of EuroVis, 2014.Google Scholar
- P. Blignaut, "Visual span and other parameters for the generation of heatmaps," in Proceedings of ACM ETRA, 2010.Google Scholar
- D. Noton and L. Stark, "Scanpaths in eye movements during pattern perception," Science, vol. 171, no. 3968, pp. 308--311, 1971.Google Scholar
- H. Cai, V. W. Zheng, and K. C.-C. Chang, "A comprehensive survey of graph embedding: Problems, techniques, and applications," IEEE Transactions on Knowledge and Data Engineering, vol. 30, no. 9, pp. 1616--1637, 2018.Google ScholarDigital Library
- D. B. West et al., Introduction to graph theory. Prentice Hall Upper Saddle River, 2001, vol. 2.Google Scholar
- S. Eraslan, Y. Yesilada, and S. Harper, "Scanpath trend analysis on web pages: Clustering eye tracking scanpaths," ACM Transactions on the Web, vol. 10, no. 4, p. 20, 2016.Google ScholarDigital Library
- N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929--1958, 2014.Google ScholarDigital Library
- "Super Mario Bros game," https://www.classicgames.me/super-mario-bros.html.Google Scholar
- "Agario game," https://agar.io/.Google Scholar
- D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in Proceedings of ICLR, 2015.Google Scholar
- P. Olsson, "Real-time and offline filters for eye tracking," 2007, Master Thesis, KTH Royal Institute of Technology.Google Scholar
- K. Rayner and M. Castelhano, "Eye movements," Scholarpedia, vol. 2, no. 10, p. 3649, 2007.Google Scholar
- R. Fallahzadeh and H. Ghasemzadeh, "Personalization without user interruption: Boosting activity recognition in new subjects using unlabeled data," in Proceedings of ACM ICCPS, 2017.Google ScholarDigital Library
- "NVIDIA Jetson Nano," https://developer.nvidia.com/embedded/jetson-nano.Google Scholar
- M. Glushakov, Y. Zhang, Y. Han, T. J. Scargill, G. Lan, and M. Gorlatova, "Edge-based provisioning of holographic content for contextual and personalized augmented reality," in Proceedings of IEEE SmartEdge, 2020.Google Scholar
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Proceedings of NeurIPS, 2017.Google Scholar
- Y. Qin, D. Song, H. Chen, W. Cheng, G. Jiang, and G. W. Cottrell, "A dual-stage attention-based recurrent neural network for time series prediction," in Proceedings of IJCAI, 2017.Google Scholar
- I. Sutskever, O. Vinyals, and Q. V. Le, "Sequence to sequence learning with neural networks," in Proceedings of NeurIPS, 2014.Google Scholar
- Z. Che, X. He, K. Xu, and Y. Liu, "DECADE: A deep metric learning model for multivariate time series," in Proceedings of KDD Workshop on Miningand Learning from Time Series, 2017.Google Scholar
- S. Li, D. Hong, and H. Wang, "Relation inference among sensor time series in smart buildings with metric learning," in Proceedings of AAAI, 2020.Google Scholar
- A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine, "Meta-learning with implicit gradients" in Proceedings of NeurIPS, 2019.Google Scholar
- Q. Sun, Y. Liu, T.-S. Chua, and B. Schiele, "Meta-transfer learning for few-shot learning," in Proceedings of IEEE CVPR, 2019.Google Scholar
- T. N. Kipf and M. Welling, "Semi-supervised classification with graph convolutional networks," in Proceedings of ICLR, 2017.Google Scholar
Index Terms
GazeGraph: graph-based few-shot cognitive context sensing from human visual behavior
Recommendations
Few-Shot Adaptation for Multimedia Semantic Indexing
MM '18: Proceedings of the 26th ACM international conference on MultimediaWe propose a few-shot adaptation framework, which bridges zero-shot learning and supervised many-shot learning, for semantic indexing of image and video data. Few-shot adaptation provides robust parameter estimation with few training examples, by ...
Few-shot learning with unsupervised part discovery and part-aligned similarity
Highlights- We propose a novel unsupervised Part Discovery Network, which can learn discriminative and transferable part representations from unlabeled images for few-...
AbstractFew-shot learning aims to recognize novel concepts with only a few examples. To this end, previous studies resort to acquiring a strong inductive bias via meta-learning on a group of similar tasks, which however needs a large labeled ...
Can we improve meta-learning model in few-shot learning by aligning data distributions?
AbstractMeta-learning becomes a promising way to solve the few-shot learning problem in recent research. This paradigm mainly relies on hierarchical architecture and episodic training to achieve good generalization on the new learning task. ...
Comments