Abstract
Artificial intelligence including deep learning and 3D reconstruction methods is changing the daily life of people. Now, an unmanned aerial vehicle that can move freely in the air and avoid harsh ground conditions has been commonly adopted as a suitable tool for 3D reconstruction. The traditional 3D reconstruction mission based on drones usually consists of two steps: image collection and offline post-processing. But there are two problems: one is the uncertainty of whether all parts of the target object are covered, and another is the tedious post-processing time. Inspired by modern deep learning methods, we build a telexistence drone system with an onboard deep learning computation module and a wireless data transmission module that perform incremental real-time dense reconstruction of urban cities by itself. Two technical contributions are proposed to solve the preceding issues. First, based on the popular depth fusion surface reconstruction framework, we combine it with a visual-inertial odometry estimator that integrates the inertial measurement unit and allows for robust camera tracking as well as high-accuracy online 3D scan. Second, the capability of real-time 3D reconstruction enables a new rendering technique that can visualize the reconstructed geometry of the target as navigation guidance in the HMD. Therefore, it turns the traditional path-planning-based modeling process into an interactive one, leading to a higher level of scan completeness. The experiments in the simulation system and our real prototype demonstrate an improved quality of the 3D model using our artificial intelligence leveraged drone system.
- C. E. Aguero, N. Koenig, I. Chen, H. Boyer, S. Peters, J. Hsu, B. Gerkey, et al. 2015. Inside the Virtual Robotics Challenge: Simulating real-time robotic disaster response. IEEE Transactions on Automation Science and Engineering 12, 2 (April 2015), 494–506. https://doi.org/10.1109/TASE.2014.2368997Google Scholar
Cross Ref
- D. Andersen, P. Villano, and V. Popescu. 2019. AR HMD guidance for controlled hand-held 3D acquisition. IEEE Transactions on Visualization and Computer Graphics 25, 11 (2019), 3073–3082.Google Scholar
Cross Ref
- Jiawen Chen, Dennis Bautembach, and Shahram Izadi. 2013. Scalable real-time volumetric surface reconstruction. ACM Transactions on Graphics 32, 4 (July 2013), Article 113, 16 pages. https://doi.org/10.1145/2461912.2461940 Google Scholar
Digital Library
- Long Chen, Wen Tang, Nigel W. John, Tao Ruan Wan, and Jian Jun Zhang. 2018. SLAM-based dense surface reconstruction in monocular minimally invasive surgery and its application to augmented reality. Computer Methods and Programs in Biomedicine 158 (2018), 135–146. https://doi.org/10.1016/j.cmpb.2018.02.006Google Scholar
Cross Ref
- Yang Chen and Gerard Medioni. 1992. Object modelling by registration of multiple range images. Image and Vision Computing 10, 3 (1992), 145–155. https://doi.org/10.1016/0262-8856(92)90066-C Google Scholar
Digital Library
- Alexandre Cherpillod, Stefano Mintchev, and Dario Floreano. 2017. Embodied flight with a drone. arXiv:1707.01788. http://arxiv.org/abs/1707.01788.Google Scholar
- Angela Dai. 2018. Using Generative Deep Learning to Create High-Quality Models from 3D Scans. Stanford University.Google Scholar
- Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. 2017. BundleFusion: Real-time globally consistent 3D reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics 36, 4 (2017), 1. Google Scholar
Digital Library
- Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-scale direct monocular SLAM. In Computer Vision—ECCV 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, Switzerland, 834–849. Google Scholar
Cross Ref
- Haoqiang Fan, Hao Su, and Leonidas J. Guibas. 2017. A point set generation network for 3D object reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 605–613.Google Scholar
Cross Ref
- Ramon A. Suarez Fernandez, Jose Luis Sanchez-Lopez, Carlos Sampedro, Hriday Bavle, Martin Molina, and Pascual Campoy. 2016. Natural user interfaces for human-drone multi-modal interaction. In Proceedings of the 2016 International Conference on Unmanned Aircraft Systems (ICUAS’16). IEEE, Los Alamitos, CA, 1013–1022.Google Scholar
Cross Ref
- Simon Fuhrmann and Michael Goesele. 2014. Floating scale surface reconstruction. ACM Transactions on Graphics 33, 4 (July 2014), Article 46, 11 pages. https://doi.org/10.1145/2601097.2601163 Google Scholar
Digital Library
- Yasutaka Furukawa and Carlos Hernandez. 2015. Multi-view stereo: A tutorial. Foundations and Trends in Computer Graphics and Vision 9, 1–2 (2015), 1–148. https://doi.org/10.1561/0600000052 Google Scholar
Digital Library
- Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multi-view stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 8 (2010), 1362–1376. Google Scholar
Digital Library
- Richard Hartley and Andrew Zisserman. 2003. Multiple View Geometry in Computer Vision. Cambridge University Press. Google Scholar
Digital Library
- Hirohiko Hayakawa, Charith Lasantha Fernando, M. H. D. Yamen Saraiji, Kouta Minamizawa, and Susumu Tachi. 2015. Telexistence drone: Design of a flight telexistence system for immersive aerial sports experience. In Proceedings of the 6th Augmented Human International Conference. 171–172. Google Scholar
Digital Library
- Roman Herrmann and Ludger Schmidt. 2018. Design and evaluation of a natural user interface for piloting an unmanned aerial vehicle. I-com 17, 1 (2018), 15–24.Google Scholar
Cross Ref
- Keita Higuchi and Jun Rekimoto. 2013. Flying head: A head motion synchronization mechanism for unmanned aerial vehicle control. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. 2029–2038. Google Scholar
Digital Library
- Wenqi Hu, Guo Zhan Lum, Massimo Mastrangeli, and Metin Sitti. 2018. Small-scale soft-bodied robot with multimodal locomotion. Nature 554, 7690 (Feb. 2018), 81–85. https://doi.org/10.1038/nature25443Google Scholar
Cross Ref
- Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Transactions on Graphics 32, 3 (July 2013), Article 29, 13 pages. https://doi.org/10.1145/2487228.2487237 Google Scholar
Digital Library
- T. Kersten and M. Lindstaedt. 2012. Potential of automatic 3D object reconstruction from multiple images for applications in architecture, cultural heritage and archaeology. International Journal of Heritage in the Digital Era 1, 3 (2012), 399–420.Google Scholar
Cross Ref
- Byung Hyung Kim, Minho Kim, and Sungho Jo. 2014. Quadcopter flight control using a low-cost hybrid interface with EEG-based classification and eye tracking. Computers in Biology and Medicine 51 (2014), 82–92. https://doi.org/10.1016/j.compbiomed.2014.04.020 Google Scholar
Digital Library
- Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics 36, 4 (2017), Article 78, 13 pages. Google Scholar
Digital Library
- Paul Koppel. 2015. Agisoft PhotoScan: Point Cloud Accuracy in Close Range Configuration. Koppel Engineering.Google Scholar
- Karl LaFleur, Kaitlin Cassady, Alexander Doud, Kaleb Shades, Eitan Rogin, and Bin He. 2013. Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface. Journal of Neural Engineering 10, 4 (2013), 046003. http://stacks.iop.org/1741-2552/10/i=4/a=046003.Google Scholar
Cross Ref
- Kok-Lim Low. 2004. Linear least-squares optimization for point-to-plane ICP surface registration. Chapel Hill, University of North Carolina 4, 10 (2004), 1–3.Google Scholar
- Huimin Lu, Yujie Li, Min Chen, Hyoungseop Kim, and Seiichi Serikawa. 2018. Brain intelligence: Go beyond artificial intelligence. Mobile Networks and Applications 23, 2 (2018), 368–375. Google Scholar
Digital Library
- Huimin Lu, Yujie Li, Shenglin Mu, Dong Wang, Hyoungseop Kim, and Seiichi Serikawa. 2017. Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet of Things Journal 5, 4 (2017), 2315–2322.Google Scholar
Cross Ref
- H. Lu, Y. Li, S. Mu, D. Wang, H. Kim, and S. Serikawa. 2018. Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet of Things Journal 5, 4 (2018), 2315–2322.Google Scholar
Cross Ref
- Jenifer Miehlbradt, Alexandre Cherpillod, Stefano Mintchev, Martina Coscia, Fiorenzo Artoni, Dario Floreano, and Silvestro Micera. 2018. Data-driven body–machine interface for the accurate control of drones. Proceedings of the National Academy of Sciences 115, 31 (2018), 7913–7918. https://doi.org/10.1073/pnas.1718648115arXiv:http://www.pnas.org/content/115/31/7913.full.pdf.Google Scholar
Cross Ref
- Christian Mostegel, Markus Rumpler, Friedrich Fraundorfer, and Horst Bischof. 2016. UAV-based autonomous image acquisition with multi-view stereo quality assurance by confidence prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1–10.Google Scholar
Cross Ref
- R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality. 127–136. https://doi.org/10.1109/ISMAR.2011.6092378 Google Scholar
Digital Library
- Matthias Nießner, Michael Zollhofer, Shahram Izadi, and Marc Stamminger. 2013. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics 32, 6 (Jan. 2013), Article 169, 11 pages. https://www.microsoft.com/en-us/research/publication/real-time-3d-reconstruction-at-scale-using-voxel-hashing/. Google Scholar
Digital Library
- S. Park, Y. Jung, and J. Bae. 2016. A tele-operation interface with a motion capture system and a haptic glove. In Proceedings of the 2016 13th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI’16). 544–549. https://doi.org/10.1109/URAI.2016.7625774Google Scholar
- Kevin Pfeil, Seng Lee Koh, and Joseph LaViola. 2013. Exploring 3D gesture metaphors for interaction with unmanned aerial vehicles. In Proceedings of the 2013 International Conference on Intelligent User Interfaces (IUI’13). ACM, New York, NY, 257–266. https://doi.org/10.1145/2449396.2449429 Google Scholar
Digital Library
- Corey Pittman and Joseph J. LaViola Jr.2014. Exploring head tracked head mounted displays for first person robot teleoperation. In Proceedings of the 19th International Conference on Intelligent User Interfaces (IUI’14). ACM, New York, NY, 323–328. https://doi.org/10.1145/2557500.2557527 Google Scholar
Digital Library
- Tong Qin, Peiliang Li, and Shaojie Shen. 2018. VINS-Mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics 34, 4 (2018), 1004–1020. Google Scholar
Digital Library
- B. Reitinger, C. Zach, and D. Schmalstieg. 2007. Augmented reality scouting for interactive 3D reconstruction. In Proceedings of the 2007 IEEE Virtual Reality Conference. 219–222. https://doi.org/10.1109/VR.2007.352485Google Scholar
Cross Ref
- Fabio Remondino, Luigi Barazzetti, Francesco Nex, Marco Scaioni, and Daniele Sarazzi. 2011. UAV photogrammetry for mapping and 3D modeling—Current status and future perspectives. International Archives of the Photogrammetry, Remote Sensing, and Spatial Information Sciences 38, 1 (2011), C22.Google Scholar
- Mike Roberts, Debadeepta Dey, Anh Truong, Sudipta Sinha, Shital Shah, Ashish Kapoor, Pat Hanrahan, and Neel Joshi. 2017. Submodular trajectory optimization for aerial 3D scanning. In Proceedings of the IEEE International Conference on Computer Vision. 5324–5333.Google Scholar
Cross Ref
- C. Rognon, S. Mintchev, F. Dell’Agnola, A. Cherpillod, D. Atienza, and D. Floreano. 2018. FlyJacket: An upper body soft exoskeleton for immersive drone control. IEEE Robotics and Automation Letters 3, 3 (July 2018), 2362–2369. https://doi.org/10.1109/LRA.2018.2810955Google Scholar
Cross Ref
- Andrea Sanna, Fabrizio Lamberti, Gianluca Paravati, and Federico Manuri. 2013. A Kinect-based natural interface for quadrotor control. Entertainment Computing 4, 3 (2013), 179–186. https://doi.org/10.1016/j.entcom.2013.01.001Google Scholar
Cross Ref
- Keng Hua Sing and Wei Xie. 2016. Garden: A mixed reality experience combining virtual reality and 3D reconstruction. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA’16). ACM, New York, NY, 180–183. https://doi.org/10.1145/2851581.2890370 Google Scholar
Digital Library
- Alvy Ray Smith. 1978. Color gamut transform pairs. ACM SIGGRAPH Computer Graphics 12, 3 (1978), 12–19. Google Scholar
Digital Library
- Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2006. Photo tourism: Exploring photo collections in 3D. In ACM SIGGRAPH 2006 Papers (SIGGRAPH’06). ACM, New York, NY, 835–846. https://doi.org/10.1145/1179352.1141964 Google Scholar
Digital Library
- X. Song, K. Mann, E. Allison, S. Yoon, H. Hila, A. Muller, and C. Gieder. 2016. A quadcopter controlled by brain concentration and eye blink. In Proceedings of the 2016 IEEE Signal Processing in Medicine and Biology Symposium (SPMB’16). 1–4. https://doi.org/10.1109/SPMB.2016.7846875Google Scholar
Cross Ref
- S. Tachi. 1998. Real-time remote robotics-toward networked telexistence. IEEE Computer Graphics and Applications 18, 6 (Nov. 1998), 6–9. https://doi.org/10.1109/38.734972 Google Scholar
Digital Library
- S. Tachi. 2016. Telexistence: Enabling humans to be virtually ubiquitous. IEEE Computer Graphics and Applications 36, 1 (Jan.–Feb. 2016), 8–14. https://doi.org/10.1109/MCG.2016.6Google Scholar
Digital Library
- Peng Wang, Daoyong Wang, Xiantao Zhang, Xin Li, Tao Peng, Huimin Lu, and Xinliang Tian. 2020. Numerical and experimental study on the maneuverability of an active propeller control based wave glider. Applied Ocean Research 104 (2020), 102369.Google Scholar
Cross Ref
- Ting Wang, Xiangjun Ji, Aiguo Song, Kurosh Madani, Amine Chohra, Huimin Lu, and Ramon Moreno. 2021. Output constrained and RBFNN-based position and adaptive force control for security tele-surgery. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 25 (2021), Article 61, 15 pages.Google Scholar
Digital Library
- Zhiying Wang and Daniele Perissin. 2012. Cosmo SkyMed AO projects—3D reconstruction and stability monitoring of the Three Gorges Dam. In Proceedings of the 2012 IEEE International Geoscience and Remote Sensing Symposium (IGARSS’12).Google Scholar
Cross Ref
- Changchang Wu. 2011. VisualSFM: A Visual Structure from Motion System. Retrieved August 1, 2021 from http://ccwu.me/vsfm/.Google Scholar
- X. Xia, C. Pun, D. Zhang, Y. Yang, H. Lu, H. Gao, and F. Xu. 2019. A 6-DOF telexistence drone controlled by a head mounted display. In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR’19). 1241–1242.Google Scholar
- Jingyu Yang, Jianhua Guo, Huanjing Yue, Zhiheng Liu, Haofeng Hu, and Kun Li. 2019. CDnet: CNN-based cloud detection for remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing 57, 8 (2019), 6195–6211.Google Scholar
Cross Ref
- Xiaolong Yang, Hongtao Wu, Yao Li, Shengzheng Kang, Bai Chen, Huimin Lu, Carman KM Lee, and Ping Ji. 2020. Dynamics and isotropic control of parallel mechanisms for vibration isolation. IEEE/ASME Transactions on Mechatronics 25, 4 (2020), 2027–2034.Google Scholar
Cross Ref
- Xianghua Ying and Zhanyi Hu. 2004. Can we consider central catadioptric cameras and fisheye cameras within a unified imaging model. In Proceedings of the European Conference on Computer Vision. 442–455.Google Scholar
Cross Ref
- Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. 2016. Fast global registration. In Proceedings of the European Conference on Computer Vision. 766–782.Google Scholar
Cross Ref
- J. Zillner, E. Mendez, and D. Wagner. 2018. Augmented reality remote collaboration with dense reconstruction. In Proceedings of the 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct’18). 38–39. https://doi.org/10.1109/ISMAR-Adjunct.2018.00028Google Scholar
Cross Ref
Index Terms
Virtual Reality Aided High-Quality 3D Reconstruction by Remote Drones
Recommendations
Procedurally generated virtual reality from 3D reconstructed physical space
VRST '16: Proceedings of the 22nd ACM Conference on Virtual Reality Software and TechnologyWe present a novel system for automatically generating immersive and interactive virtual reality (VR) environments using the real world as a template. The system captures indoor scenes in 3D, detects obstacles like furniture and walls, and maps walkable ...
Real-time mixed-reality telepresence via 3D reconstruction with HoloLens and commodity depth sensors
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal InteractionWe present a demo of low-cost mixed reality telepresence system that performs real-time 3D reconstruction of a person or an object and wirelessly transmits the reconstructions to Microsoft's HoloLens head mounted display at frame rates perceived as ...
Oculus Rift Application for Training Drone Pilots
SIMUTOOLS '17: Proceedings of the 10th EAI International Conference on Simulation Tools and TechniquesThe research described in this paper, focuses on a virtual reality headset system that integrates the Oculus Rift VR headset with a low cost Unmanned Aerial Vehicle (UAV) to allow for drone teleoperation and telepresence using the Robot Operating System ...






Comments