Chris Xiaoxuan Lu
ABSTRACT
Robust and accurate trajectory estimation of mobile agents such as people and robots is a key requirement for providing spatial awareness for emerging capabilities such as augmented reality or autonomous interaction. Although currently dominated by optical techniques e.g., visual-inertial odometry these suffer from challenges with scene illumination or featureless surfaces. As an alternative, we propose milliEgo, a novel deep-learning approach to robust egomotion estimation which exploits the capabilities of low-cost mm Wave radar. Although mmWave radar has a fundamental advantage over monocular cameras of being metric i.e., providing absolute scale or depth, current single chip solutions have limited and sparse imaging resolution, making existing point-cloud registration techniques brittle. We propose a new architecture that is optimized for solving this challenging pose transformation problem. Secondly, to robustly fuse mmWave pose estimates with additional sensors, e.g. inertial or visual sensors we introduce a mixed attention approach to deep fusion. Through extensive experiments, we demonstrate our proposed system is able to achieve 1.3% 3D error drift and generalizes well to unseen environments. We also show that the neural architecture can be made highly efficient and suitable for real-time embedded applications.
- Farhad Aghili and Chun-Yi Su. 2016. Robust relative navigation by integration of ICP and adaptive Kalman filter using laser scanner and IMU. IEEE/ASME Transactions on Mechatronics 21, 4 (2016), 2015--2026.Google Scholar
Cross Ref
- Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. 2016. Learning to poke by poking: Experiential learning of intuitive physics. In Advances in neural information processing systems.Google Scholar
- Yasin Almalioglu, Mehmet Turan, Chris Xiaoxuan Lu, Niki Trigoni, and Andrew Markham. 2020. Milli-RIO: Ego-Motion Estimation with Low-Cost Millimetre-Wave Radar. IEEE Sensors Journal (2020).Google Scholar
- Oscar Alsing. 2018. Mobile Object Detection using TensorFlow Lite and Transfer Learning.Google Scholar
- Nicolas Alt, Patrick Rives, and Eckehard Steinbach. 2013. Reconstruction of transparent objects in unstructured scenes with a depth camera. In 2013 IEEE International Conference on Image Processing. 4131--4135.Google ScholarCross Ref
- Sofien Bouaziz, Andrea Tagliasacchi, and Mark Pauly. 2013. Sparse iterative closest point. In Computer graphics forum, Vol. 32. 113--123.Google Scholar
- Carlos Pena Caballero, Elifaleth Cantu, Jesus Rodriguez, Adolfo Gonzales, Osvaldo Castellanos, Angel Cantu, Megan Strait, Jae Son, and Dongchul Kim. [n. d.]. A Multiple Radar Approach for Automatic Target Recognition of Aircraft using Inverse Synthetic Aperture Radar. In 2018 1st International Conference on Data Intelligence and Security (ICDIS).Google Scholar
- Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian Reid, and John J Leonard. 2016. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on robotics (2016).Google Scholar
- Sarah H Cen and Paul Newman. 2018. Precise ego-motion estimation with millimeter-wave radar under diverse and challenging conditions. In 2018 IEEE International Conference on Robotics and Automation (ICRA).Google ScholarCross Ref
- Balasubramaniyan Chandrasekaran, Shruti Gangadhar, and James M Conrad. 2017. A survey of multisensor fusion techniques, architectures and methodologies. In SoutheastCon 2017. IEEE, 1--8.Google ScholarCross Ref
- Changhao Chen, Xiaoxuan Lu, Andrew Markham, and Niki Trigoni. 2018. Ionet: Learning to cure the curse of drift in inertial odometry. In Thirty-Second AAAI Conference on Artificial Intelligence.Google ScholarCross Ref
- Changhao Chen, Xiaoxuan Lu, Johan Wahlstrom, Andrew Markham, and Niki Trigoni. 2019. Deep Neural Network Based Inertial Odometry Using Low-cost Inertial Measurement Units. IEEE Transactions on Mobile Computing (2019).Google ScholarCross Ref
- Changhao Chen, Stefano Rosa, Yishu Miao, Chris Xiaoxuan Lu, Wei Wu, Andrew Markham, and Niki Trigoni. 2019. Selective sensor fusion for neural visual-inertial odometry. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
- Javier Civera, Oscar G Grasa, Andrew J Davison, and JMM Montiel. 2010. 1-Point RANSAC for extended Kalman filtering: Application to real-time structure from motion and visual odometry. Journal of field robotics 27, 5 (2010), 609--631.Google ScholarDigital Library
- Ronald Clark, Sen Wang, Hongkai Wen, Andrew Markham, and Niki Trigoni. 2017. Vinet: Visual-inertial odometry as a sequence-to-sequence learning problem. In Thirty-First AAAI Conference on Artificial Intelligence.Google Scholar
- George E Dahl, Tara N Sainath, and Geoffrey E Hinton. 2013. Improving deep neural networks for LVCSR using rectified linear units and dropout. In 2013 IEEE international conference on acoustics, speech and signal processing. 8609--8613.Google ScholarCross Ref
- Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2018. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 224--236.Google ScholarCross Ref
- Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. 2015. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision. 2758--2766.Google ScholarDigital Library
- Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Longyue Wang, Shuming Shi, and Tong Zhang. 2019. Dynamic layer aggregation for neural machine translation with routing-by-agreement. In AAAI.Google Scholar
- Andreas Eitel, Jost Tobias Springenberg, Luciano Spinello, Martin Riedmiller, and Wolfram Burgard. 2015. Multimodal deep learning for robust RGB-D object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 681--687.Google ScholarCross Ref
- Steven Gold, Anand Rangarajan, Chien-Ping Lu, Suguna Pappu, and Eric Mjolsness. 1998. New algorithms for 2D and 3D point matching: Pose estimation and correspondence. Pattern recognition 31, 8 (1998), 1019--1031.Google ScholarCross Ref
- Davide Guermandi, Qixian Shi, Andy Dewilde, Veerle Derudder, Ubaid Ahmad, Annachiara Spagnolo, Ilja Ocket, André Bourdoux, Piet Wambacq, Jan Craninckx, et al. 2017. A 79-GHz MIMO PMCW radar SoC in 28-nm CMOS. IEEE Journal of Solid-State Circuits 52, 10 (2017), 2613--2626.Google ScholarCross Ref
- Ankur Handa, Thomas Whelan, John McDonald, and Andrew J Davison. 2014. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In IEEE international conference on Robotics and automation (ICRA).Google ScholarCross Ref
- Gibson Hu, Shoudong Huang, Liang Zhao, Alen Alempijevic, and Gamini Dissanayake. 2012. A robust rgb-d slam algorithm. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. 1714--1719.Google ScholarCross Ref
- Daniel P Huttenlocher and Shimon Ullman. 1990. Recognizing solid objects by alignment with an image. International journal of computer vision 5, 2 (1990), 195--212.Google ScholarDigital Library
- Texas Instruments. [n. d.]. Automotive mmWave sensors. http://www.ti.com/sensors/mmwave/overview.htmlGoogle Scholar
- Texas Instruments. [n. d.]. mmWave sensors Overview. https://www.ti.com/sensors/mmwave-radar/overview.htmlGoogle Scholar
- Texas Instruments. [n. d.]. mmWave Training Series. https://training.ti.com/mmwave-training-seriesGoogle Scholar
- Jean-Pierre Joosting. [n. d.]. Radar enables heavy lifting drones to navigate in complex environments. https://www.eenewseurope.com/news/radar-enables-heavy-lifting-drones-navigate-complex-environments?from=singlemessage&isappinstalled=0#Google Scholar
- Deok-Hwa Kim and Jong-Hwan Kim. 2013. Image-Based ICP algorithm for visual odometry using a RGB-D sensor in a dynamic environment. In Robot Intelligence Technology and Applications 2012. 423--430.Google Scholar
- Andrew Kramer, Carl Stahoviak, Angel Santamaria-Navarro, Ali-akbar Aghamohammadi, and Christoffer Heckman. 2020. Radar-Inertial Ego-Velocity Estimation for Visually Degraded Environments. In IEEE International Conference on Robotics and Automation (ICRA).Google Scholar
- MARK LAPEDUS. 2017. Here Comes High-Res Car Radar. https://semiengineering.com/here-comes-high-res-car-radar/Google Scholar
- Stefan Leutenegger, Simon Lynen, Michael Bosse, Roland Siegwart, and Paul Furgale. 2015. Keyframe-based visual-inertial odometry using nonlinear optimization. The International Journal of Robotics Research 34, 3 (2015), 314--334.Google ScholarDigital Library
- Bo Li, Tianlei Zhang, and Tian Xia. 2016. Vehicle detection from 3d lidar using fully convolutional network. arXiv preprint arXiv:1608.07916 (2016).Google Scholar
- Ruihao Li, Sen Wang, Zhiqiang Long, and Dongbing Gu. 2018. Undeepvo: Monocular visual odometry through unsupervised deep learning. In 2018 IEEE international conference on robotics and automation (ICRA). 7286--7291.Google ScholarCross Ref
- Shengzhong Liu, Shuochao Yao, Jinyang Li, Dongxin Liu, Tianshi Wang, Huajie Shao, and Tarek Abdelzaher. 2020. GIobalFusion: A Global Attentional Deep Learning Framework for Multisensor Information Fusion. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--27.Google ScholarDigital Library
- Chris Xiaoxuan Lu, Yang Li, Peijun Zhao, Changhao Chen, Linhai Xie, Hongkai Wen, Rui Tan, and Niki Trigoni. 2018. Simultaneous localization and mapping with power network electromagnetic field. In Proceedings of the 24th annual international conference on mobile computing and networking (MobiCom).Google ScholarDigital Library
- Chris Xiaoxuan Lu, Stefano Rosa, Peijun Zhao, Bing Wang, Changhao Chen, John A. Stankovic, Niki Trigoni, and Andrew Markham. 2020. See Through Smoke: Robust Indoor Mapping with Low-cost mmWave Radar. In ACM International Conference on Mobile Systems, Applications, and Services (MobiSys).Google ScholarDigital Library
- Jonathan S Lu, Patrick Cabrol, Daniel Steinbach, and Ravikumar V Pragada. 2013. Measurement and characterization of various outdoor 60 GHz diffracted and scattered paths. In MILCOM.Google Scholar
- Dimitrios Lymberopoulos, Jie Liu, Xue Yang, Romit Roy Choudhury, Vlado Handziski, and Souvik Sen. 2015. A realistic evaluation and comparison of indoor location technologies: Experiences and lessons learned. In Proceedings of the 14th international conference on information processing in sensor networks.Google ScholarDigital Library
- George R MacCartney, Sijia Deng, Shu Sun, and Theodore S Rappaport. 2016. Millimeter-wave human blockage at 73 GHz with a simple double knife-edge diffraction model and extension for directional antennas. In IEEE Vehicular Technology Conference (VTC-Fall).Google ScholarCross Ref
- Babak Mamandipoor, Greg Malysa, Amin Arbabian, Upamanyu Madhow, and Karam Noujeim. 2014. 60 ghz synthetic aperture radar for short-range imaging: Theory and experiments. In ACSSC.Google Scholar
- Andriy Myronenko and Xubo Song. 2010. Point set registration: Coherent point drift. IEEE transactions on pattern analysis and machine intelligence 32, 12 (2010), 2262--2275.Google Scholar
- Austin Nicolai, Ryan Skeele, Christopher Eriksen, and Geoffrey A Hollinger. 2016. Deep learning for laser based odometry estimation. In RSS workshop Limits and Potentials of Deep Learning in Robotics.Google Scholar
- Chris Nuttall and Tim Bradshaw. [n. d.]. Google draws on old radar technology for its motion sensor Pixel 4 smartphone. https://www.ft.com/content/02c051ec-f005-11e9-ad1e-4367d8281195Google Scholar
- Yeong Sang Park, Young-Sik Shin, and Ayoung Kim. 2020. PhaRaO: Direct Radar Odometry using Phase Correlation. In IEEE ICRA.Google Scholar
- Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer. In ICML.Google Scholar
- Michail N Petsios, Emmanouil G Alivizatos, and Nikolaos K Uzunoglu. 2008. Solving the association problem for a multistatic range-only radar target tracker. Signal Processing (2008).Google Scholar
- Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y Ng. 2009. ROS: an open-source Robot Operating System. In ICRA workshop on open source software, Vol. 3. 5.Google Scholar
- Valentin Radu and Mahesh K Marina. 2013. HiMLoc: Indoor smartphone localization via activity aware pedestrian dead reckoning with selective crowdsourced WiFi fingerprinting. In International conference on indoor positioning and indoor navigation.Google ScholarCross Ref
- Jeffrey J Richardson and L Monika Moskal. 2011. Strengths and limitations of assessing forest density and spatial configuration with aerial LiDAR. Remote Sensing of Environment 115, 10 (2011), 2640--2651.Google ScholarCross Ref
- Peng Rong and Mihail L Sichitiu. 2006. Angle of arrival localization for wireless sensor networks. In SECON.Google Scholar
- ROS.org. [n. d.]. gmapping. http://wiki.ros.org/gmappingGoogle Scholar
- Muhamad Risqi U Saputra, Pedro Porto Buarque de Gusmao, Chris Xiaoxuan Lu, Yasin Almalioglu, Stefano Rosa, Changhao Chen, Johan Wahlstrom, Wei Wang, Andrew Markham, and Niki Trigoni. 2020. Deeptio: A deep thermal-inertial odometry with visual hallucination. IEEE Robotics and Automation Letters (2020).Google Scholar
- Muhamad Risqi U Saputra, Andrew Markham, and Niki Trigoni. 2018. Visual SLAM and structure from motion in dynamic environments: A survey. ACM Computing Surveys (CSUR) 51, 2 (2018).Google ScholarDigital Library
- Sheng Shen, Mahanth Gowda, and Romit Roy Choudhury. 2018. Closing the gaps in inertial motion tracking. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. 429--444.Google ScholarDigital Library
- Charles Spence and Jon Driver. 2004. Crossmodal space and crossmodal attention. Oxford University Press.Google Scholar
- Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Advances in neural information processing systems. 2377--2385.Google Scholar
- Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox. 2017. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
- Deepak Uttam and B Culshaw. 1985. Precision time domain reflectometry in optical fiber systems using a frequency modulated continuous wave ranging technique. Journal of Lightwave Technology (1985).Google Scholar
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems.Google Scholar
- Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni. 2017. Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks. In 2017 IEEE International Conference on Robotics and Automation (ICRA). 2043--2050.Google ScholarCross Ref
- Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni. 2018. End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks. The International Journal of Robotics Research (2018).Google Scholar
- Wei Wang, Muhamad Risqi U Saputra, Peijun Zhao, Pedro Gusmao, Bo Yang, Changhao Chen, Andrew Markham, and Niki Trigoni. 2019. DeepPCO: End-to-End Point Cloud Odometry through Deep Parallel Neural Network. International Conference on Intelligent Robots and Systems (2019).Google ScholarDigital Library
- Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local neural networks. In CVPR.Google Scholar
- DK Barton HR Ward. 1969. Handbook of radar measurement.Google Scholar
- Rob Weston, Sarah Cen, Paul Newman, and Ingmar Posner. 2018. Probably unknown: Deep inverse sensor modelling in radar. In ICRA.Google Scholar
- Zhuoling Xiao, Hongkai Wen, Andrew Markham, and Niki Trigoni. 2014. Lightweight map matching for indoor localisation using conditional random fields. In Proceedings of the 13th International Symposium on Information Processing in Sensor Networks.Google ScholarCross Ref
- Qiang Xu, Rong Zheng, and Steve Hranilovic. 2015. IDyLL: Indoor localization using inertial and light sensors on smartphones. In ACM International Joint Conference on Pervasive and Ubiquitous Computing.Google ScholarDigital Library
- Hongfei Xue, Wenjun Jiang, Chenglin Miao, Fenglong Ma, Shiyang Wang, Ye Yuan, Shuochao Yao, Aidong Zhang, and Lu Su. 2020. DeepMV: Multi-View Deep Learning for Device-Free Human Activity Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--26.Google ScholarDigital Library
- Hongfei Xue, Wenjun Jiang, Chenglin Miao, Ye Yuan, Fenglong Ma, Xin Ma, Yijiang Wang, Shuochao Yao, Wenyao Xu, Aidong Zhang, et al. 2019. DeepFusion: A Deep Learning Framework for the Fusion of Heterogeneous Sensory Data. In Proceedings of the Twentieth ACM International Symposium on Mobile Ad Hoc Networking and Computing. 151--160.Google ScholarDigital Library
- Yan Yan, Long Li, Guodong Xie, Changjing Bao, Peicheng Liao, Hao Huang, Yongxiong Ren, Nisar Ahmed, Zhe Wang, et al. 2016. Multipath effects in millimetre-wave wireless communication using orbital angular momentum multiplexing. Scientific reports 6 (2016), 33482.Google Scholar
- Heng Yang, Jingnan Shi, and Luca Carlone. 2020. TEASER: Fast and Certifiable Point Cloud Registration. arXiv preprint arXiv:2001.07715 (2020).Google Scholar
- Nan Yang, Rui Wang, Jorg Stuckler, and Daniel Cremers. 2018. Deep virtual stereo odometry: Leveraging deep depth prediction for monocular direct sparse odometry. In Proceedings of the European Conference on Computer Vision (ECCV). 817--833.Google ScholarCross Ref
- Shuochao Yao, Shaohan Hu, Yiran Zhao, Aston Zhang, and Tarek Abdelzaher. 2017. Deepsense: A unified deep learning framework for time-series mobile sensing data processing. In Proceedings of the 26th International Conference on World Wide Web.Google ScholarDigital Library
- Shuochao Yao, Yiran Zhao, Huajie Shao, Dongxin Liu, Shengzhong Liu, Yifan Hao, Ailing Piao, Shaohan Hu, Lu Su, and Tarek F Abdelzaher. 2019. SADeepSense: Self-Attention Deep Learning Framework for Heterogeneous On-Device Sensors in Internet of Things Applications. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications.Google Scholar
- Xiaoqing Ye, Jiamao Li, Hexiao Huang, Liang Du, and Xiaolin Zhang. 2018. 3d recurrent neural networks with context fusion for point cloud semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV).Google ScholarCross Ref
- Renjie Zhao, Timothy Woodford, Teng Wei, Qian Kun, and Xinyu Zhang. 2020. M-Cube: A Millimeter-Wave Massive MIMO Software Radio. In ACM MobiCom.Google Scholar
Index Terms
milliEgo: single-chip mmWave radar aided egomotion estimation via deep sensor fusion
Recommendations
Efficient, causal camera tracking in unprepared environments
This paper addresses the problem of tracking the 3D pose of a camera in space, using the images it acquires while moving freely in unmodeled, arbitrary environments. A novel feature-based approach for camera tracking is proposed, intended to facilitate ...
Estimation of the epipole using optical flow at antipodal points
We present algorithms for estimating the epipole or direction of translation of a moving camera. We use constraints arising from two points that are antipodal on the image sphere in order to decouple rotation from translation. One pair of antipodal ...
Multispectral Stereo Odometry
In this paper, we investigate the problem of visual odometry for ground vehicles based on the simultaneous utilization of multispectral cameras. It encompasses a stereo rig composed of an optical (visible) and thermal sensors. The novelty resides in the ...
Comments