Blending camera and 77 GHz radar sensing for equitable, robust plethysmography

Abstract

With the resurgence of non-contact vital sign sensing due to the COVID-19 pandemic, remote heart-rate monitoring has gained significant prominence. Many existing methods use cameras; however previous work shows a performance loss for darker skin tones. In this paper, we show through light transport analysis that the camera modality is fundamentally biased against darker skin tones. We propose to reduce this bias through multi-modal fusion with a complementary and fairer modality - radar. Through a novel debiasing oriented fusion framework, we achieve performance gains over all tested baselines and achieve skin tone fairness improvements over the RGB modality. That is, the associated Pareto frontier between performance and fairness is improved when compared to the RGB modality. In addition, performance improvements are obtained over the radar-based method, with small trade-offs in fairness. We also open-source the largest multi-modal remote heart-rate estimation dataset of paired camera and radar measurements with a focus on skin tone representation.

References

1. Øyvind Aardal, Yoann Paichard, Sverre Brovoll, Tor Berger, Tor Sverre Lande, and Svein-Erik Hamran. 2012. Physical working principles of medical radar. IEEE Transactions on Biomedical Engineering 60, 4 (2012), 1142--1149.
2. Md Atiqur Rahman Ahad, Upal Mahbub, and Tauhidur Rahman. 2021. Contactless Human Activity Analysis. Springer.
3. Mostafa Alizadeh, George Shaker, João Carlos Martins De Almeida, Plinio Pelegrini Morita, and Safeddin Safavi-Naeini. 2019. Remote monitoring of human vital signs using mm-wave FMCW radar. IEEE Access 7 (2019), 54958--54968.
4. Sarah Alotaibi and William AP Smith. 2017. A biophysical 3D morphable model of face appearance. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 824--832.
5. R. Rox Anderson and John A. Parrish. 1981. The Optics of Human Skin. Journal of Investigative Dermatology 77, 1 (1981), 13--19.
6. Consumer Technology Association. 2018. Physical Activity Monitoring for Heart Rate, ANSI/CTA-2065.
7. Yunhao Ba, Zhen Wang, Kerim Doruk Karinca, Oyku Deniz Bozkurt, and Achuta Kadambi. 2021. Overcoming Difficulty in Obtaining Dark-skinned Subjects for Remote-PPG by Synthetic Augmentation. arXiv preprint arXiv:2106.06007 (2021).
8. Guha Balakrishnan, Fredo Durand, and John Guttag. 2013. Detecting pulse from head motions in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3430--3437.
9. Carina Barbosa Pereira, Michael Czaplik, Vladimir Blazek, Steffen Leonhardt, and Daniel Teichmann. 2018. Monitoring of cardiorespiratory signals using thermal imaging: a pilot study on healthy human subjects. Sensors 18, 5 (2018), 1541.
10. Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2017. Fairness in machine learning. Nips tutorial 1 (2017), 2017.
11. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 (2016), 4349--4357.
12. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77--91.
13. Toon Calders, Asim Karim, Faisal Kamiran, Wasif Ali, and Xiangliang Zhang. 2013. Controlling attribute effect in linear regression. In 2013 IEEE 13th international conference on data mining. IEEE, 71--80.
14. Weixuan Chen and Daniel McDuff. 2018. Deepphys: Video-based physiological measurement using convolutional attention networks. In Proceedings of the European Conference on Computer Vision (ECCV). 349--365.
15. Gerard De Haan and Vincent Jeanne. 2013. Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering 60, 10 (2013), 2878--2886.
16. Amy Diane Droitcour. 2006. Non-contact measurement of heart and respiration rates with a single-chip microwave doppler radar. Stanford University.
17. Amy D Droitcour, Olga Boric-Lubecke, Victor M Lubecke, Jenshan Lin, and Gregory TA Kovacs. 2004. Range correlation and I/Q performance benefits in single-chip silicon Doppler radars for noncontact cardiopulmonary monitoring. IEEE Transactions on Microwave Theory and Techniques 52, 3 (2004), 838--848.
18. Davide Giavarina. 2015. Understanding bland altman analysis. Biochemia medica 25, 2 (2015), 141--151.
19. Wei Han, Hui Chen, and Soujanya Poria. 2021. Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis. arXiv preprint arXiv:2109.00412 (2021).
20. S. W. Hasinoff, F. Durand, and W. T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 553--560. ISSN: 1063-6919.
21. Christophe Hurter and Daniel McDuff. 2017. Cardiolens: remote physiological monitoring in a mixed reality environment. In ACM siggraph 2017 emerging technologies. 1--2.
22. Takanori Igarashi, Ko Nishino, and Shree K Nayar. 2007. The appearance of human skin: A survey. Now Publishers Inc.
23. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. PMLR, 448--456.
24. Manjit Kaur and Dilbag Singh. 2020. Fusion of medical images using deep belief networks. Cluster Computing 23, 2 (2020), 1439--1453.
25. Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. 2019. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9012--9020.
26. Yoonkyoung Kim, Yosep Park, Jinman Kim, and Eui Chul Lee. 2018. Remote heart rate monitoring method using infrared thermal camera. International Journal of Engineering Research and Technology 11, 3 (2018), 493--500.
27. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
28. Eugene Lee, Evan Chen, and Chen-Yi Lee. 2020. Meta-rppg: Remote heart rate estimation using a transductive meta-learner. In European Conference on Computer Vision. Springer, 392--409.
29. Magdalena Lewandowska, Jacek Rumiński, Tomasz Kocejko, and Jędrzej Nowak. 2011. Measuring pulse rate with a webcam---a non-contact method for evaluating cardiac activity. In 2011 federated conference on computer science and information systems (FedCSIS). IEEE, 405--410.
30. James C Lin. 1975. Noninvasive microwave measurement of respiration. Proc. IEEE 63, 10 (1975), 1530--1530.
31. Wenjie Lv, Wangdong He, Xipeng Lin, and Jungang Miao. 2021. Non-Contact Monitoring of Human Vital Signs Using FMCW Millimeter Wave Radar in the 120 GHz Band. Sensors 21, 8 (2021), 2732.
32. Ewa Magdalena Nowara, Tim K Marks, Hassan Mansour, and Ashok Veeraraghavan. 2018. SparsePPG: Towards driver monitoring using camera-based vital signs estimation in near-infrared. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 1272--1281.
33. Kenta Matsumura, Sogo Toda, and Yuji Kato. 2020. RGB and near-infrared light reflectance/transmittance photoplethysmography for measuring heart rate during motion. IEEE Access 8 (2020), 80233--80242.
34. Cardiac Monitors. 2002. Heart Rate Meters, and Alarms. ANSI/AAMI Standard EC13 (2002).
35. Toshiaki Negishi, Shigeto Abe, Takemi Matsui, He Liu, Masaki Kurosawa, Tetsuo Kirimoto, and Guanghao Sun. 2020. Contactless vital signs measurement system using RGB-thermal image sensors and its clinical screening test on patients with seasonal influenza. Sensors 20, 8 (2020), 2171.
36. Benjamin W Nelson and Nicholas B Allen. 2019. Accuracy of consumer wearable heart rate measurement during an ecologically valid 24-hour period: intraindividual validation study. JMIR mHealth and uHealth 7, 3 (2019), e10828.
37. Xuesong Niu, Shiguang Shan, Hu Han, and Xilin Chen. 2019. Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation. IEEE Transactions on Image Processing 29 (2019), 2409--2423.
38. Ewa M Nowara, Daniel McDuff, and Ashok Veeraraghavan. 2020. A meta-analysis of the impact of skin tone and gender on non-contact photoplethysmography measurements. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 284--285.
39. Prashanth Pai and Mohammed Zafar Ali Khan. 2008. Comparison of SC and MRC receiver complexity for three antenna diversity systems. In 2008 24th Biennial Symposium on Communications. IEEE, 302--305.
40. Ming-Zher Poh, Daniel J McDuff, and Rosalind W Picard. 2010. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE transactions on biomedical engineering 58, 1 (2010), 7--11.
41. Lingyun Ren, Lingqin Kong, Farnaz Foroughian, Haofei Wang, Paul Theilmann, and Aly E Fathy. 2017. Comparison study of noncontact vital signs detection using a Doppler stepped-frequency continuous-wave radar and camera-based imaging photoplethysmography. IEEE Transactions on Microwave Theory and Techniques 65, 9 (2017), 3519--3529.
42. Lingyun Ren, Haofei Wang, Krishna Naishadham, Ozlem Kilic, and Aly E Fathy. 2016. Phase-based methods for heart rate detection using UWB impulse Doppler radar. IEEE Transactions on Microwave Theory and Techniques 64, 10 (2016), 3319--3331.
43. Silonie Sachdeva et al. 2009. Fitzpatrick skin typing: Applications in dermatology. Indian journal of dermatology, venereology and leprology 75, 1 (2009), 93.
44. Vikas Singh, Nishchal K Verma, Zeeshan Ul Islam, and Yan Cui. 2019. Feature learning using stacked autoencoder for shared and multimodal fusion of medical images. In Computational Intelligence: Theories, Applications and Future Directions-Volume I. Springer, 53--66.
45. Rencheng Song, Huan Chen, Juan Cheng, Chang Li, Yu Liu, and Xun Chen. 2021. PulseGAN: Learning to generate realistic pulse waveforms in remote photoplethysmography. IEEE Journal of Biomedical and Health Informatics 25, 5 (2021), 1373--1384.
46. Nitish Srivastava, Ruslan Salakhutdinov, et al. 2012. Multimodal Learning with Deep Boltzmann Machines.. In NIPS, Vol. 1. Citeseer, 2.
47. Wim Verkruysse, Lars O Svaasand, and J Stuart Nelson. 2008. Remote plethysmographic imaging using ambient light. Optics express 16, 26 (2008), 21434--21445.
48. Sahil Verma and Julia Rubin. 2018. Fairness definitions explained. In 2018 ieee/acm international workshop on software fairness (fairware). IEEE, 1--7.
49. Vytautas Vizbara. 2013. Comparison of green, blue and infrared light in wrist and forehead photoplethysmography. BIOMEDICAL ENGINEERING 2016 17, 1 (2013).
50. Wenjin Wang, Albertus C den Brinker, Sander Stuijk, and Gerard De Haan. 2016. Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering 64, 7 (2016), 1479--1491.
51. Zhen Wang, Yunhao Ba, Pradyumna Chari, Oyku Bozkurt, Gianna Brown, Parth Patwa, Niranjan Vaddi, Laleh Jalilian, and Achuta Kadambi. 2022. Synthetic Generation of Face Videos with Plethysmograph Physiology. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
52. Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. 2020. Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8919--8928.
53. Fokko P Wieringa, Frits Mastik, and Antonius FW van der Steen. 2005. Contactless multiple wavelength photoplethysmographic imaging: A first step toward "SpO 2 camera" technology. Annals of biomedical engineering 33, 8 (2005), 1034--1041.
54. Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, and William Freeman. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM transactions on graphics (TOG) 31, 4 (2012), 1--8.
55. Shuqiong Wu, Takuya Sakamoto, Kentaro Oishi, Toru Sato, Kenichi Inoue, Takeshi Fukuda, Kenji Mizutani, and Hiroyuki Sakai. 2019. Person-specific heart rate estimation with ultra-wideband radar using convolutional neural networks. IEEE Access 7 (2019), 168484--168494.
56. T Wu. 2003. Ppgi: New development in noninvasive and contactless diagnosis of dermal perfusion using near infrared light. J. GCPD eV 7, 1 (2003), 17--24.
57. Ting Wu, Vladimir Blazek, and Hans Juergen Schmitt. 2000. Photoplethysmography imaging: a new noninvasive and noncontact method for mapping of the dermal perfusion changes. In Optical Techniques and Instrumentation for the Measurement of Blood Composition, Structure, and Dynamics, Vol. 4163. International Society for Optics and Photonics, 62--70.
58. Zitong Yu, Xiaobai Li, and Guoying Zhao. 2019. Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. arXiv preprint arXiv:1905.02419 (2019).
59. Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. 2017. Fairness constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics. PMLR, 962--970.
60. Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 23, 10 (2016), 1499--1503.
61. Tianyue Zheng, Zhe Chen, Shujie Zhang, Chao Cai, and Jun Luo. 2021. MoRe-Fi: Motion-robust and Fine-grained Respiration Monitoring via Deep-Learning UWB Radar. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems. 111--124.

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation

      • Computational photography