skip to main content
research-article

Virtual Reality Aided High-Quality 3D Reconstruction by Remote Drones

Authors Info & Claims
Published:14 September 2021Publication History
Skip Abstract Section

Abstract

Artificial intelligence including deep learning and 3D reconstruction methods is changing the daily life of people. Now, an unmanned aerial vehicle that can move freely in the air and avoid harsh ground conditions has been commonly adopted as a suitable tool for 3D reconstruction. The traditional 3D reconstruction mission based on drones usually consists of two steps: image collection and offline post-processing. But there are two problems: one is the uncertainty of whether all parts of the target object are covered, and another is the tedious post-processing time. Inspired by modern deep learning methods, we build a telexistence drone system with an onboard deep learning computation module and a wireless data transmission module that perform incremental real-time dense reconstruction of urban cities by itself. Two technical contributions are proposed to solve the preceding issues. First, based on the popular depth fusion surface reconstruction framework, we combine it with a visual-inertial odometry estimator that integrates the inertial measurement unit and allows for robust camera tracking as well as high-accuracy online 3D scan. Second, the capability of real-time 3D reconstruction enables a new rendering technique that can visualize the reconstructed geometry of the target as navigation guidance in the HMD. Therefore, it turns the traditional path-planning-based modeling process into an interactive one, leading to a higher level of scan completeness. The experiments in the simulation system and our real prototype demonstrate an improved quality of the 3D model using our artificial intelligence leveraged drone system.

References

  1. C. E. Aguero, N. Koenig, I. Chen, H. Boyer, S. Peters, J. Hsu, B. Gerkey, et al. 2015. Inside the Virtual Robotics Challenge: Simulating real-time robotic disaster response. IEEE Transactions on Automation Science and Engineering 12, 2 (April 2015), 494–506. https://doi.org/10.1109/TASE.2014.2368997Google ScholarGoogle ScholarCross RefCross Ref
  2. D. Andersen, P. Villano, and V. Popescu. 2019. AR HMD guidance for controlled hand-held 3D acquisition. IEEE Transactions on Visualization and Computer Graphics 25, 11 (2019), 3073–3082.Google ScholarGoogle ScholarCross RefCross Ref
  3. Jiawen Chen, Dennis Bautembach, and Shahram Izadi. 2013. Scalable real-time volumetric surface reconstruction. ACM Transactions on Graphics 32, 4 (July 2013), Article 113, 16 pages. https://doi.org/10.1145/2461912.2461940 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Long Chen, Wen Tang, Nigel W. John, Tao Ruan Wan, and Jian Jun Zhang. 2018. SLAM-based dense surface reconstruction in monocular minimally invasive surgery and its application to augmented reality. Computer Methods and Programs in Biomedicine 158 (2018), 135–146. https://doi.org/10.1016/j.cmpb.2018.02.006Google ScholarGoogle ScholarCross RefCross Ref
  5. Yang Chen and Gerard Medioni. 1992. Object modelling by registration of multiple range images. Image and Vision Computing 10, 3 (1992), 145–155. https://doi.org/10.1016/0262-8856(92)90066-C Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Alexandre Cherpillod, Stefano Mintchev, and Dario Floreano. 2017. Embodied flight with a drone. arXiv:1707.01788. http://arxiv.org/abs/1707.01788.Google ScholarGoogle Scholar
  7. Angela Dai. 2018. Using Generative Deep Learning to Create High-Quality Models from 3D Scans. Stanford University.Google ScholarGoogle Scholar
  8. Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. 2017. BundleFusion: Real-time globally consistent 3D reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics 36, 4 (2017), 1. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-scale direct monocular SLAM. In Computer Vision—ECCV 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, Switzerland, 834–849. Google ScholarGoogle ScholarCross RefCross Ref
  10. Haoqiang Fan, Hao Su, and Leonidas J. Guibas. 2017. A point set generation network for 3D object reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 605–613.Google ScholarGoogle ScholarCross RefCross Ref
  11. Ramon A. Suarez Fernandez, Jose Luis Sanchez-Lopez, Carlos Sampedro, Hriday Bavle, Martin Molina, and Pascual Campoy. 2016. Natural user interfaces for human-drone multi-modal interaction. In Proceedings of the 2016 International Conference on Unmanned Aircraft Systems (ICUAS’16). IEEE, Los Alamitos, CA, 1013–1022.Google ScholarGoogle ScholarCross RefCross Ref
  12. Simon Fuhrmann and Michael Goesele. 2014. Floating scale surface reconstruction. ACM Transactions on Graphics 33, 4 (July 2014), Article 46, 11 pages. https://doi.org/10.1145/2601097.2601163 Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Yasutaka Furukawa and Carlos Hernandez. 2015. Multi-view stereo: A tutorial. Foundations and Trends in Computer Graphics and Vision 9, 1–2 (2015), 1–148. https://doi.org/10.1561/0600000052 Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multi-view stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 8 (2010), 1362–1376. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Richard Hartley and Andrew Zisserman. 2003. Multiple View Geometry in Computer Vision. Cambridge University Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Hirohiko Hayakawa, Charith Lasantha Fernando, M. H. D. Yamen Saraiji, Kouta Minamizawa, and Susumu Tachi. 2015. Telexistence drone: Design of a flight telexistence system for immersive aerial sports experience. In Proceedings of the 6th Augmented Human International Conference. 171–172. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Roman Herrmann and Ludger Schmidt. 2018. Design and evaluation of a natural user interface for piloting an unmanned aerial vehicle. I-com 17, 1 (2018), 15–24.Google ScholarGoogle ScholarCross RefCross Ref
  18. Keita Higuchi and Jun Rekimoto. 2013. Flying head: A head motion synchronization mechanism for unmanned aerial vehicle control. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. 2029–2038. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Wenqi Hu, Guo Zhan Lum, Massimo Mastrangeli, and Metin Sitti. 2018. Small-scale soft-bodied robot with multimodal locomotion. Nature 554, 7690 (Feb. 2018), 81–85. https://doi.org/10.1038/nature25443Google ScholarGoogle ScholarCross RefCross Ref
  20. Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Transactions on Graphics 32, 3 (July 2013), Article 29, 13 pages. https://doi.org/10.1145/2487228.2487237 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. T. Kersten and M. Lindstaedt. 2012. Potential of automatic 3D object reconstruction from multiple images for applications in architecture, cultural heritage and archaeology. International Journal of Heritage in the Digital Era 1, 3 (2012), 399–420.Google ScholarGoogle ScholarCross RefCross Ref
  22. Byung Hyung Kim, Minho Kim, and Sungho Jo. 2014. Quadcopter flight control using a low-cost hybrid interface with EEG-based classification and eye tracking. Computers in Biology and Medicine 51 (2014), 82–92. https://doi.org/10.1016/j.compbiomed.2014.04.020 Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics 36, 4 (2017), Article 78, 13 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Paul Koppel. 2015. Agisoft PhotoScan: Point Cloud Accuracy in Close Range Configuration. Koppel Engineering.Google ScholarGoogle Scholar
  25. Karl LaFleur, Kaitlin Cassady, Alexander Doud, Kaleb Shades, Eitan Rogin, and Bin He. 2013. Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface. Journal of Neural Engineering 10, 4 (2013), 046003. http://stacks.iop.org/1741-2552/10/i=4/a=046003.Google ScholarGoogle ScholarCross RefCross Ref
  26. Kok-Lim Low. 2004. Linear least-squares optimization for point-to-plane ICP surface registration. Chapel Hill, University of North Carolina 4, 10 (2004), 1–3.Google ScholarGoogle Scholar
  27. Huimin Lu, Yujie Li, Min Chen, Hyoungseop Kim, and Seiichi Serikawa. 2018. Brain intelligence: Go beyond artificial intelligence. Mobile Networks and Applications 23, 2 (2018), 368–375. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Huimin Lu, Yujie Li, Shenglin Mu, Dong Wang, Hyoungseop Kim, and Seiichi Serikawa. 2017. Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet of Things Journal 5, 4 (2017), 2315–2322.Google ScholarGoogle ScholarCross RefCross Ref
  29. H. Lu, Y. Li, S. Mu, D. Wang, H. Kim, and S. Serikawa. 2018. Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet of Things Journal 5, 4 (2018), 2315–2322.Google ScholarGoogle ScholarCross RefCross Ref
  30. Jenifer Miehlbradt, Alexandre Cherpillod, Stefano Mintchev, Martina Coscia, Fiorenzo Artoni, Dario Floreano, and Silvestro Micera. 2018. Data-driven body–machine interface for the accurate control of drones. Proceedings of the National Academy of Sciences 115, 31 (2018), 7913–7918. https://doi.org/10.1073/pnas.1718648115arXiv:http://www.pnas.org/content/115/31/7913.full.pdf.Google ScholarGoogle ScholarCross RefCross Ref
  31. Christian Mostegel, Markus Rumpler, Friedrich Fraundorfer, and Horst Bischof. 2016. UAV-based autonomous image acquisition with multi-view stereo quality assurance by confidence prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1–10.Google ScholarGoogle ScholarCross RefCross Ref
  32. R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality. 127–136. https://doi.org/10.1109/ISMAR.2011.6092378 Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Matthias Nießner, Michael Zollhofer, Shahram Izadi, and Marc Stamminger. 2013. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics 32, 6 (Jan. 2013), Article 169, 11 pages. https://www.microsoft.com/en-us/research/publication/real-time-3d-reconstruction-at-scale-using-voxel-hashing/. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. S. Park, Y. Jung, and J. Bae. 2016. A tele-operation interface with a motion capture system and a haptic glove. In Proceedings of the 2016 13th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI’16). 544–549. https://doi.org/10.1109/URAI.2016.7625774Google ScholarGoogle Scholar
  35. Kevin Pfeil, Seng Lee Koh, and Joseph LaViola. 2013. Exploring 3D gesture metaphors for interaction with unmanned aerial vehicles. In Proceedings of the 2013 International Conference on Intelligent User Interfaces (IUI’13). ACM, New York, NY, 257–266. https://doi.org/10.1145/2449396.2449429 Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Corey Pittman and Joseph J. LaViola Jr.2014. Exploring head tracked head mounted displays for first person robot teleoperation. In Proceedings of the 19th International Conference on Intelligent User Interfaces (IUI’14). ACM, New York, NY, 323–328. https://doi.org/10.1145/2557500.2557527 Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Tong Qin, Peiliang Li, and Shaojie Shen. 2018. VINS-Mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics 34, 4 (2018), 1004–1020. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. B. Reitinger, C. Zach, and D. Schmalstieg. 2007. Augmented reality scouting for interactive 3D reconstruction. In Proceedings of the 2007 IEEE Virtual Reality Conference. 219–222. https://doi.org/10.1109/VR.2007.352485Google ScholarGoogle ScholarCross RefCross Ref
  39. Fabio Remondino, Luigi Barazzetti, Francesco Nex, Marco Scaioni, and Daniele Sarazzi. 2011. UAV photogrammetry for mapping and 3D modeling—Current status and future perspectives. International Archives of the Photogrammetry, Remote Sensing, and Spatial Information Sciences 38, 1 (2011), C22.Google ScholarGoogle Scholar
  40. Mike Roberts, Debadeepta Dey, Anh Truong, Sudipta Sinha, Shital Shah, Ashish Kapoor, Pat Hanrahan, and Neel Joshi. 2017. Submodular trajectory optimization for aerial 3D scanning. In Proceedings of the IEEE International Conference on Computer Vision. 5324–5333.Google ScholarGoogle ScholarCross RefCross Ref
  41. C. Rognon, S. Mintchev, F. Dell’Agnola, A. Cherpillod, D. Atienza, and D. Floreano. 2018. FlyJacket: An upper body soft exoskeleton for immersive drone control. IEEE Robotics and Automation Letters 3, 3 (July 2018), 2362–2369. https://doi.org/10.1109/LRA.2018.2810955Google ScholarGoogle ScholarCross RefCross Ref
  42. Andrea Sanna, Fabrizio Lamberti, Gianluca Paravati, and Federico Manuri. 2013. A Kinect-based natural interface for quadrotor control. Entertainment Computing 4, 3 (2013), 179–186. https://doi.org/10.1016/j.entcom.2013.01.001Google ScholarGoogle ScholarCross RefCross Ref
  43. Keng Hua Sing and Wei Xie. 2016. Garden: A mixed reality experience combining virtual reality and 3D reconstruction. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA’16). ACM, New York, NY, 180–183. https://doi.org/10.1145/2851581.2890370 Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Alvy Ray Smith. 1978. Color gamut transform pairs. ACM SIGGRAPH Computer Graphics 12, 3 (1978), 12–19. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2006. Photo tourism: Exploring photo collections in 3D. In ACM SIGGRAPH 2006 Papers (SIGGRAPH’06). ACM, New York, NY, 835–846. https://doi.org/10.1145/1179352.1141964 Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. X. Song, K. Mann, E. Allison, S. Yoon, H. Hila, A. Muller, and C. Gieder. 2016. A quadcopter controlled by brain concentration and eye blink. In Proceedings of the 2016 IEEE Signal Processing in Medicine and Biology Symposium (SPMB’16). 1–4. https://doi.org/10.1109/SPMB.2016.7846875Google ScholarGoogle ScholarCross RefCross Ref
  47. S. Tachi. 1998. Real-time remote robotics-toward networked telexistence. IEEE Computer Graphics and Applications 18, 6 (Nov. 1998), 6–9. https://doi.org/10.1109/38.734972 Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. S. Tachi. 2016. Telexistence: Enabling humans to be virtually ubiquitous. IEEE Computer Graphics and Applications 36, 1 (Jan.–Feb. 2016), 8–14. https://doi.org/10.1109/MCG.2016.6Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Peng Wang, Daoyong Wang, Xiantao Zhang, Xin Li, Tao Peng, Huimin Lu, and Xinliang Tian. 2020. Numerical and experimental study on the maneuverability of an active propeller control based wave glider. Applied Ocean Research 104 (2020), 102369.Google ScholarGoogle ScholarCross RefCross Ref
  50. Ting Wang, Xiangjun Ji, Aiguo Song, Kurosh Madani, Amine Chohra, Huimin Lu, and Ramon Moreno. 2021. Output constrained and RBFNN-based position and adaptive force control for security tele-surgery. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 25 (2021), Article 61, 15 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Zhiying Wang and Daniele Perissin. 2012. Cosmo SkyMed AO projects—3D reconstruction and stability monitoring of the Three Gorges Dam. In Proceedings of the 2012 IEEE International Geoscience and Remote Sensing Symposium (IGARSS’12).Google ScholarGoogle ScholarCross RefCross Ref
  52. Changchang Wu. 2011. VisualSFM: A Visual Structure from Motion System. Retrieved August 1, 2021 from http://ccwu.me/vsfm/.Google ScholarGoogle Scholar
  53. X. Xia, C. Pun, D. Zhang, Y. Yang, H. Lu, H. Gao, and F. Xu. 2019. A 6-DOF telexistence drone controlled by a head mounted display. In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR’19). 1241–1242.Google ScholarGoogle Scholar
  54. Jingyu Yang, Jianhua Guo, Huanjing Yue, Zhiheng Liu, Haofeng Hu, and Kun Li. 2019. CDnet: CNN-based cloud detection for remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing 57, 8 (2019), 6195–6211.Google ScholarGoogle ScholarCross RefCross Ref
  55. Xiaolong Yang, Hongtao Wu, Yao Li, Shengzheng Kang, Bai Chen, Huimin Lu, Carman KM Lee, and Ping Ji. 2020. Dynamics and isotropic control of parallel mechanisms for vibration isolation. IEEE/ASME Transactions on Mechatronics 25, 4 (2020), 2027–2034.Google ScholarGoogle ScholarCross RefCross Ref
  56. Xianghua Ying and Zhanyi Hu. 2004. Can we consider central catadioptric cameras and fisheye cameras within a unified imaging model. In Proceedings of the European Conference on Computer Vision. 442–455.Google ScholarGoogle ScholarCross RefCross Ref
  57. Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. 2016. Fast global registration. In Proceedings of the European Conference on Computer Vision. 766–782.Google ScholarGoogle ScholarCross RefCross Ref
  58. J. Zillner, E. Mendez, and D. Wagner. 2018. Augmented reality remote collaboration with dense reconstruction. In Proceedings of the 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct’18). 38–39. https://doi.org/10.1109/ISMAR-Adjunct.2018.00028Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Virtual Reality Aided High-Quality 3D Reconstruction by Remote Drones

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!