Abstract
Gaze estimation is a difficult task, even for humans. However, as humans, we are good at understanding a situation and exploiting it to guess the expected visual focus of attention of people, and we usually use this information to retrieve people’s gaze. In this article, we propose to leverage such situation-based expectation about people’s visual focus of attention to collect weakly labeled gaze samples and perform person-specific calibration of gaze estimators in an unsupervised and online way. In this context, our contributions are the following: (i) we show how task contextual attention priors can be used to gather reference gaze samples, which is a cumbersome process otherwise; (ii) we propose a robust estimation framework to exploit these weak labels for the estimation of the calibration model parameters; and (iii) we demonstrate the applicability of this approach on two human-human and human-robot interaction settings, namely conversation and manipulation. Experiments on three datasets validate our approach, providing insights on the priors effectiveness and on the impact of different calibration models, particularly the usefulness of taking head pose into account.
- [1] . 2017. Social eye gaze in human-robot interaction: A review. Journal of Human-Robot Interaction 6, 1 (2017), 25–53. Google Scholar
Digital Library
- [2] . 2011. Multiperson visual focus of attention from head pose and meeting contextual cues. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 1 (2011), 101–116. Google Scholar
Digital Library
- [3] . 2010. Computational Models for Multiparty Turn Taking.
Technical Report MSR-TR 2010-115 . Microsoft Research.Google Scholar - [4] . 2001. What can a mouse cursor tell us more? Correlation of eye/mouse movements on web browsing. In CHI’01 Extended Abstracts on Human Factors in Computing Systems. ACM, New York, NY, 281–282. Google Scholar
Digital Library
- [5] . 2020. GEDDnet: A network for gaze estimation with dilation and decomposition. arXiv preprint arXiv:2001.09284 (2020).Google Scholar
- [6] . 2014. EYEDIAP: A database for the development and evaluation of gaze estimation algorithms from RGB and RGB-D cameras. In Proceedings of the ACM Symposium on Eye Tracking Research and Applications. Google Scholar
Digital Library
- [7] . 2013. A semi-automated system for accurate gaze coding in natural dyadic interactions. In Proceedings of the International Conference on Multimodal Interaction. ACM, New York, NY. Google Scholar
Digital Library
- [8] . 2012. Gaze estimation from multimodal kinect data. In Proceedings of the Conference in Computer Vision and Pattern Recognition Workshop on Gesture Recognition.Google Scholar
Cross Ref
- [9] . 2016. Gaze estimation in the 3D space using RGB-D sensors. International Journal of Computer Vision 118 (2016), 194–216. Google Scholar
Digital Library
- [10] . 2006. General theory of remote gaze estimation using the pupil center and corneal reflections. IEEE Transactions on BioMedical Engineering 53, 6 (2006), 1124–1133.Google Scholar
Cross Ref
- [11] . 2014. Shorter gaze duration for happy faces in current but not remitted depression: Evidence from eye movements. Psychiatry Research 218, 1–2 (2014), 79–86.Google Scholar
Cross Ref
- [12] . 2001. Eye-hand coordination in object manipulation. Journal of Neuroscience 21, 17 (2001), 6917–6932.Google Scholar
Cross Ref
- [13] . 2014. Bottom-up and top-down attention: Different processes and overlapping neural systems. Neuroscientist 20, 5 (2014), 509–521.Google Scholar
Cross Ref
- [14] . 2009. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research 10 (2009), 1755–1758. Google Scholar
Digital Library
- [15] . 2011. Engagement-based multi-party dialog with a humanoid robot. InProceedings of the SIGDIAL 2011 Conference. Google Scholar
Digital Library
- [16] . 2009. The effect of head turn on the perception of gaze. Vision Research 49, 15 (2009), 1979–1993.Google Scholar
Cross Ref
- [17] . 2016. Eye tracking for everyone. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA.Google Scholar
Cross Ref
- [18] . 2009. Vision, eye movements, and natural behavior. Visual Neuroscience 26, 1 (2009), 51–62.Google Scholar
Cross Ref
- [19] . 2004. The influence of head contour and nose angle on the perception of eye-gaze direction. Perception & Psychophysics 66, 5 (2004), 752–771.Google Scholar
Cross Ref
- [20] . 2013. Beyond the tangent point: Gaze targets in naturalistic driving. Journal of Vision 13, 13 (2013), 11.Google Scholar
Cross Ref
- [21] . 2019. Differences in eye movement range based on age and gaze direction. Eye 33, 7 (2019), 1145–1151.Google Scholar
Cross Ref
- [22] . 2019. Learning to personalize in appearance-based gaze tracking. In Proceedings of the International Conference on Computer Vision Workshops.Google Scholar
Cross Ref
- [23] . 2021. A differential approach for gaze estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 3 (2021), 1092–1099.Google Scholar
- [24] . 1990. Perception of where a person is looking: Overestimation and underestimation of gaze direction. Tohoku Psychologica Folia 49 (1990), 33–41.Google Scholar
- [25] . 2017. Calibration in Eye Tracking Using Transfer Learning. Degree Project in Computer Science and Engineering, KTH.Google Scholar
- [26] . 2018. Tracking gaze and visual focus of attention of people involved in social interaction. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 11 (2018), 2711–2724.Google Scholar
Digital Library
- [27] . 2005. Eye gaze tracking techniques for interactive applications. Computer Vision and Image Understanding 98, 1 (2005), 4–24. Google Scholar
Digital Library
- [28] . 2019. Reducing calibration drift in mobile eye trackers by exploiting mobile phone usage. In Proceedings of the ACM Symposium on Eye Tracking Research and Applications. Google Scholar
Digital Library
- [29] . 2016. Training on the job: Behavioral analysis of job interviews in hospitality. In Proceedings of the International Conference on Multimodal Interactions. Google Scholar
Digital Library
- [30] . 2018. Facing employers and customers: What do gaze and expressions tell about soft skills? In Proceedings of the International Conference on Mobile and Ubiquitous Multimedia. ACM, New York, NY, 121–126. Google Scholar
Digital Library
- [31] . 2013. The influence of calibration method and eye physiology on eyetracking data quality. Behavior Research Methods 45, 1 (2013), 272–288.Google Scholar
Cross Ref
- [32] . 2014. Who will get the grant? A multimodal corpus for the analysis of conversational behaviours in group interviews. In Proceedings of the Workshop on Understanding and Modeling Multiparty, Multimodal Interactions. 27–32. Google Scholar
Digital Library
- [33] . 2013. Gaze patterns in turn-taking. In Proceedings of the Annual Conference of the International Speech Communication Association.Google Scholar
- [34] . 2018. Recurrent CNN for 3D gaze estimation using appearance and shape cues. In Proceedings of the 29th British Machine Vision Conference.Google Scholar
- [35] . 2018. Learning to find eye region landmarks for remote gaze estimation in unconstrained settings. In Proceedings of the ACM Symposium on Eye Tracking Research and Applications. Google Scholar
Digital Library
- [36] . 2013. Pursuit calibration: Making gaze calibration less tedious and more flexible. In Proceedings of the Symposium on User Interface Software and Technology. Google Scholar
Digital Library
- [37] . 2019. Task-embedded online eye-tracker calibration for improving robustness to head motion. In Proceedings of the ACM Symposium on Eye Tracking Research and Applications. Google Scholar
Digital Library
- [38] . 1984. Least median of squares regression. Journal of the American Statistical Association 79, 388 (1984), 871–880.Google Scholar
Cross Ref
- [39] . 2017. CalibMe: Fast and unsupervised eye tracker calibration for gaze-based pervasive human-computer interaction. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 2594–2605. Google Scholar
Digital Library
- [40] . 2015. Combining dynamic head pose-gaze mapping with the robot conversational state for attention recognition in human-robot interactions. Pattern Recognition Letters 66 (2015), 81–90. Google Scholar
Digital Library
- [41] . 2019. Eye, head and torso coordination during gaze shifts in virtual reality. ACM Transactions on Computer-Human Interaction 27, 1 (2019), 1–40. Google Scholar
Digital Library
- [42] . 2019. Gaze behaviour on interacted objects during hand interaction in virtual reality for eye tracking calibration. In Proceedings of the 11th ACM Symposium on Eye Tracking Research and Applications. Google Scholar
Digital Library
- [43] . 2020. ManiGaze: A dataset for evaluating remote gaze estimator in object manipulation situations. In Proceedings of the ACM Symposium on Eye Tracking Research and Applications. Google Scholar
Digital Library
- [44] . 2017. Towards the use of social interaction conventions as prior for gaze model adaptation. In Proceedings of the International Conference on Multimodal Interaction. ACM, New York, NY, 154–162. Google Scholar
Digital Library
- [45] . 2013. Gaze locking: Passive eye contact detection for human-object interaction. In Proceedings of the ACM Symposium on User Interface Software and Technology. ACM, New York, NY. Google Scholar
Digital Library
- [46] . 2014. Learning-by-synthesis for appearance-based 3D gaze estimation. In Proceedings of the Conference on Computer Vision and Pattern Recognition. 1821–1828. Google Scholar
Digital Library
- [47] . 2015. Appearance-based gaze estimation with online calibration from mouse operations. IEEE Transactions on Human-Machine Systems 45, 6 (2015), 750–760.Google Scholar
Cross Ref
- [48] . 2019. Generalizing eye tracking with Bayesian adversarial learning. In Proceedings of the Conference on Computer Vision and Pattern Recognition. 11907–11916.Google Scholar
Cross Ref
- [49] . 2018. HeadFusion: 360 degree head pose tracking combining 3D morphable model and 3D reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 11 (2018).Google Scholar
Digital Library
- [50] . 2019. Improving few-shot user-specific gaze adaptation via gaze redirection synthesis. In Proceedings of the Conference on Computer Vision and Pattern Recognition.Google Scholar
Cross Ref
- [51] . 2019. Evaluation of appearance-based methods and implications for gaze-based applications. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, New York, NY. Google Scholar
Digital Library
- [52] . 2015. Appearance-based gaze estimation in the wild. In Proceedings of the Conference on Computer Vision and Pattern Recognition. 4511–4520.Google Scholar
Cross Ref
- [53] . 2017. MPIIGaze: Real-world dataset and deep appearance-based gaze estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 1 (2017), 162–175.Google Scholar
Digital Library
Index Terms
Robust Unsupervised Gaze Calibration Using Conversation and Manipulation Attention Priors
Recommendations
A Probabilistic Approach to Online Eye Gaze Tracking Without Explicit Personal Calibration
Existing eye gaze tracking systems typically require an explicit personal calibration process in order to estimate certain person-specific eye parameters. For natural human computer interaction, such a personal calibration is often inconvenient and ...
Conversation scene analysis based on dynamic Bayesian network and image-based gaze detection
ICMI-MLMI '10: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal InteractionThis paper presents a probabilistic framework, which incorporates automatic image-based gaze detection, for inferring the structure of multiparty face-to-face conversations. This framework aims to infer conversation regimes and gaze patterns from the ...
Task-embedded online eye-tracker calibration for improving robustness to head motion
ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & ApplicationsRemote eye trackers are widely used for screen-based interactions. They are less intrusive than head mounted eye trackers, but are generally quite sensitive to head movement. This leads to the requirement for frequent recalibration, especially in ...






Comments