skip to main content
research-article
Public Access

Three Dimensional Reconstruction of Botanical Trees with Simulatable Geometry

Published:27 September 2021Publication History
Skip Abstract Section

Abstract

We tackle the challenging problem of creating full and accurate three dimensional reconstructions of botanical trees with the topological and geometric accuracy required for subsequent physical simulation, e.g. in response to wind forces. Although certain aspects of our approach would benefit from various improvements, our results exceed the state of the art especially in geometric and topological complexity and accuracy. Starting with two dimensional RGB image data acquired from cameras attached to drones, we create point clouds, textured triangle meshes, and a simulatable and skinned cylindrical articulated rigid body model. We discuss the pros and cons of each step of our pipeline, and in order to stimulate future research we make the raw and processed data from every step of the pipeline as well as the final geometric reconstructions publicly available.

Skip Supplemental Material Section

Supplemental Material

References

  1. Sameer Agarwal, Noah Snavely, Ian Simon, Steven M Seitz, and Richard Szeliski. 2009. Building rome in a day. In Computer Vision, 2009 IEEE 12th Int. Conf. on. IEEE, 72--79.Google ScholarGoogle ScholarCross RefCross Ref
  2. Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. 2017. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105 (2017).Google ScholarGoogle Scholar
  3. Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 2016. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1534--1543.Google ScholarGoogle ScholarCross RefCross Ref
  4. Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. 2000. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 417--424.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Jules Bloomenthal. 1985. Modeling the mighty maple. In ACM SIGGRAPH Computer Graphics, Vol. 19. ACM, 305--311.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Robert L Cook. 1984. Shade trees. ACM Siggraph Comput. Graph. 18, 3 (1984), 223--231.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, Vol. 1. IEEE, 886--893.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. M. Desbrun, M. Meyer, P. Schröder, and A. H. Barr. 1999. Implicit Fairing of Irregular Meshes using Diffusion and Curvature Flow. Comput. Graph. (SIGGRAPH Proc.) (1999), 317--324.Google ScholarGoogle Scholar
  9. Rolando Estrada, Carlo Tomasi, Scott C Schmidler, and Sina Farsiu. 2015. Tree topology estimation. IEEE Transactions on Pattern Analysis & Machine Intelligence 1 (2015), 1--1.Google ScholarGoogle Scholar
  10. Martin A Fischler and Robert C Bolles. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6 (1981), 381--395.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Alvaro Fuentes, Sook Yoon, Sang Cheol Kim, and Dong Sun Park. 2017. A robust deep-learning-based detector for real-time tomato plant diseases and pests recognition. Sensors 17, 9 (2017), 2022.Google ScholarGoogle ScholarCross RefCross Ref
  12. Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intelligence 32, 8 (2010), 1362--1376.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Jingwei Huang, Angela Dai, Leonidas J Guibas, and Matthias Nießner. 2017. 3DLite: towards commodity 3D scanning for content creation. ACM Trans. Graph. 36, 6 (2017), 203--1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Anil K Jain and Farshid Farrokhnia. 1991. Unsupervised texture segmentation using Gabor filters. Pattern recognition 24, 12 (1991), 1167--1186.Google ScholarGoogle Scholar
  15. Yifeng Jiang and C Karen Liu. 2018. Data-Augmented Contact Model for Rigid Body Simulation. arXiv preprint arXiv:1803.04019 (2018).Google ScholarGoogle Scholar
  16. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision. Springer, 694--711.Google ScholarGoogle ScholarCross RefCross Ref
  17. Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. 2018. Learning Category-Specific Mesh Reconstruction from Image Collections. In The European Conf. on Comput. Vision (ECCV). 371--386.Google ScholarGoogle ScholarCross RefCross Ref
  18. Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson Surface Reconstruction. In Proc. of the Fourth Eurographics Symp. on Geom. Processing (Cagliari, Sardinia, Italy) (SGP '06). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 61--70. http://dl.acm.org/citation.cfm?id=1281957.1281965Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Alina Kloss, Stefan Schaal, and Jeannette Bohg. 2017. Combining learned and analytical models for predicting action effects. arXiv preprint arXiv:1710.04102 (2017).Google ScholarGoogle Scholar
  20. Karl Krissian, Xunlei Wu, and Vincent Luboz. 2006. Smooth vasculature reconstruction with circular and elliptic cross sections. Medicine Meets Virtual Reality 14: Accelerating Change in Healthcare: Next Medical Toolkit 119 (2006), 273.Google ScholarGoogle Scholar
  21. Yangyan Li, Xiaochen Fan, Niloy J Mitra, Daniel Chamovitz, Daniel Cohen-Or, and Baoquan Chen. 2013. Analyzing growing plants from 4D point cloud data. ACM Transactions on Graphics (TOG) 32, 6 (2013), 1--10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. 2017. RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle Scholar
  23. Aristid Lindenmayer. 1968. Mathematical models for cellular interactions in development I. Filaments with one-sided inputs. Journal of theoretical biology 18, 3 (1968), 280--299.Google ScholarGoogle Scholar
  24. Yotam Livny, Feilong Yan, Matt Olson, Baoquan Chen, Hao Zhang, and Jihad El-sana. 2010. Automatic Reconstruction of Tree Skeletal Structures from Point Clouds. Proc. SIGGRAPH Asia 2010 29 (2010), 151:1--151:8. Issue 6.Google ScholarGoogle ScholarCross RefCross Ref
  25. C. Loop. 2001. Triangle mesh subdivision with bounded curvature and the convex hull property. Technical Report MSR-TR-2001-24. Microsoft Research.Google ScholarGoogle Scholar
  26. Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In European Conf. on Comput. Vision. Springer, 154--169.Google ScholarGoogle ScholarCross RefCross Ref
  27. Gellért Máttyus, Wenjie Luo, and Raquel Urtasun. 2017. Deeproadmapper: Extracting road topology from aerial images. In The IEEE International Conference on Computer Vision (ICCV).Google ScholarGoogle ScholarCross RefCross Ref
  28. Agata Mosinska, Pablo Márquez-Neila, Mateusz Koziński, and Pascal Fua. 2018. Beyond the Pixel-Wise Loss for Topology-Aware Delineation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle Scholar
  29. Pierre Moulon, Pascal Monasse, Romuald Perrot, and Renaud Marlet. 2016. OpenMVG: Open multiple view geometry. In International Workshop on Reproducible Research in Pattern Recognition. Springer, 60--74.Google ScholarGoogle Scholar
  30. Matthias Müller and Nuttapong Chentanez. 2011. Solid Simulation with Oriented Particles. ACM TOG 30, 4, Article 92 (2011), 10 pages.Google ScholarGoogle Scholar
  31. Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. 2017. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Trans. Graph. 36, 4 (2017), 41.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Przemyslaw Prusinkiewicz, Mark Hammel, Jim Hanan, and Radomír Měch. 1997. Visual models of plant development. In Handbook of formal languages. Springer, 535--597.Google ScholarGoogle Scholar
  33. Guoxiang Qu, Wenwei Zhang, Zhe Wang, Xing Dai, Jianping Shi, Junjun He, Fei Li, Xiulan Zhang, and Yu Qiao. 2018. StripNet: Towards Topology Consistent Strip Structure Segmentation. In Proceedings of the 26th ACM International Conference on Multimedia (Seoul, Republic of Korea) (MM '18). ACM, New York, NY, USA, 283--291. https://doi.org/10.1145/3240508.3240553Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Ed Quigley, Yue Yu, Jingwei Huang, Winnie Lin, and Ronald Fedkiw. 2018. Real-time Interactive Tree Animation. IEEE transactions on visualization and computer graphics 24, 5 (2018), 1717--1727.Google ScholarGoogle ScholarCross RefCross Ref
  35. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.Google ScholarGoogle ScholarCross RefCross Ref
  36. Johannes L Schönberger and Jan-Michael Frahm. 2016. Structure-from-motion revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4104--4113.Google ScholarGoogle ScholarCross RefCross Ref
  37. Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proc. of the IEEE Conf. on Comput. Vision and Pattern Recognition (CVPR). IEEE, 519--528.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Ondrej Stava, Sören Pirk, Julian Kratt, Baoquan Chen, Radomír Měch, Oliver Deussen, and Bedrich Benes. 2014. Inverse procedural modelling of trees. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 118--131.Google ScholarGoogle Scholar
  39. Ping Tan, Gang Zeng, Jingdong Wang, Sing Bing Kang, and Long Quan. 2007. Image-based tree modeling. In ACM Trans. Graph., Vol. 26. ACM, 87.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Gabriel Taubin. 1995. A signal processing approach to fair surface design. In Proc. of the 22nd annual conf. on Comput. graphics and interactive techniques. ACM, 351--358.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Carles Ventura, Jordi Pont-Tuset, Sergi Caelles, Kevis-Kokitsi Maninis, and Luc Van Gool. 2018. Iterative Deep Learning for Road Topology Extraction. In Proc. of the British Machine Vision Conf. (BMVC).Google ScholarGoogle Scholar
  42. Jason Weber and Joseph Penn. 1995. Creation and rendering of realistic trees. In Proc. 22nd Ann. Conf. Comput. Graph. Int. Tech. ACM, 119--128.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Jamie Wither, Frédéric Boudon, M-P Cani, and Christophe Godin. 2009. Structure from silhouettes: a new paradigm for fast sketch-based design of trees. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 541--550.Google ScholarGoogle Scholar
  44. Changchang Wu. 2011. VisualSFM: A visual structure from motion system. http://ccwu.me/vsfm/.Google ScholarGoogle Scholar
  45. Changchang Wu. 2013. Towards linear-time incremental structure from motion. In 3D Vision-3DV 2013, 2013 International Conference on. IEEE, 127--134.Google ScholarGoogle Scholar
  46. Ke Xie, Feilong Yan, Andrei Sharf, Oliver Deussen, Baoquan Chen, and Hui Huang. 2015. Tree Modeling with Real Tree-Parts Examples. IEEE TVCG 22, 12 (Dec 2015), 2608--2618.Google ScholarGoogle Scholar
  47. Weipeng Xu, Avishek Chatterjee, Michael Zollhöfer, Helge Rhodin, Dushyant Mehta, Hans-Peter Seidel, and Christian Theobalt. 2018. MonoPerfCap: Human performance capture from monocular video. ACM Transactions on Graphics (TOG) 37, 2 (2018), 27.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Tianfan Xue, Jiajun Wu, Zhoutong Zhang, Chengkai Zhang, Joshua B. Tenenbaum, and William T. Freeman. 2018. Seeing Tree Structure from Vibration. In The European Conf. on Comput. Vision (ECCV). 748--764.Google ScholarGoogle Scholar
  49. Yili Zhao and Jernej Barbič. 2013. Interactive Authoring of Simulation-Ready Plants. ACM Trans. Graph. 32, 4 (2013), 84:1--84:12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Ying Zheng, Steve Gu, Herbert Edelsbrunner, Carlo Tomasi, and Philip Benfey. 2011. Detailed reconstruction of 3D plant root shape. In Comput. Vision (ICCV), 2011 IEEE Int. Conf. on. IEEE, 2026--2033.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Jin Zhou, Ananya Das, Feng Li, and Baoxin Li. 2008. Circular generalized cylinder fitting for 3D reconstruction in endoscopic imaging based on MRF. In Comput. Vision and Pattern Recognition Workshops, 2008. CVPRW'08. IEEE Comput. Society Conf. on. IEEE, 1--8.Google ScholarGoogle Scholar
  52. Silvia Zuffi, Angjoo Kanazawa, and Michael J Black. 2018. Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape From Images. In Proc. of the IEEE Conf. on Comput. Vision and Pattern Recognition (CVPR). 3955--3963.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

(auto-classified)
  1. Three Dimensional Reconstruction of Botanical Trees with Simulatable Geometry

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques
            Proceedings of the ACM on Computer Graphics and Interactive Techniques  Volume 4, Issue 3
            September 2021
            268 pages
            EISSN:2577-6193
            DOI:10.1145/3488568
            Issue’s Table of Contents

            Copyright © 2021 ACM

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 27 September 2021
            Published in pacmcgit Volume 4, Issue 3

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article
            • Research
            • Refereed

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!