Abstract
We tackle the challenging problem of creating full and accurate three dimensional reconstructions of botanical trees with the topological and geometric accuracy required for subsequent physical simulation, e.g. in response to wind forces. Although certain aspects of our approach would benefit from various improvements, our results exceed the state of the art especially in geometric and topological complexity and accuracy. Starting with two dimensional RGB image data acquired from cameras attached to drones, we create point clouds, textured triangle meshes, and a simulatable and skinned cylindrical articulated rigid body model. We discuss the pros and cons of each step of our pipeline, and in order to stimulate future research we make the raw and processed data from every step of the pipeline as well as the final geometric reconstructions publicly available.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Three Dimensional Reconstruction of Botanical Trees with Simulatable Geometry
- Sameer Agarwal, Noah Snavely, Ian Simon, Steven M Seitz, and Richard Szeliski. 2009. Building rome in a day. In Computer Vision, 2009 IEEE 12th Int. Conf. on. IEEE, 72--79.Google Scholar
Cross Ref
- Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. 2017. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105 (2017).Google Scholar
- Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 2016. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1534--1543.Google Scholar
Cross Ref
- Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. 2000. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 417--424.Google Scholar
Digital Library
- Jules Bloomenthal. 1985. Modeling the mighty maple. In ACM SIGGRAPH Computer Graphics, Vol. 19. ACM, 305--311.Google Scholar
Digital Library
- Robert L Cook. 1984. Shade trees. ACM Siggraph Comput. Graph. 18, 3 (1984), 223--231.Google Scholar
Digital Library
- Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, Vol. 1. IEEE, 886--893.Google Scholar
Digital Library
- M. Desbrun, M. Meyer, P. Schröder, and A. H. Barr. 1999. Implicit Fairing of Irregular Meshes using Diffusion and Curvature Flow. Comput. Graph. (SIGGRAPH Proc.) (1999), 317--324.Google Scholar
- Rolando Estrada, Carlo Tomasi, Scott C Schmidler, and Sina Farsiu. 2015. Tree topology estimation. IEEE Transactions on Pattern Analysis & Machine Intelligence 1 (2015), 1--1.Google Scholar
- Martin A Fischler and Robert C Bolles. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6 (1981), 381--395.Google Scholar
Digital Library
- Alvaro Fuentes, Sook Yoon, Sang Cheol Kim, and Dong Sun Park. 2017. A robust deep-learning-based detector for real-time tomato plant diseases and pests recognition. Sensors 17, 9 (2017), 2022.Google Scholar
Cross Ref
- Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intelligence 32, 8 (2010), 1362--1376.Google Scholar
Digital Library
- Jingwei Huang, Angela Dai, Leonidas J Guibas, and Matthias Nießner. 2017. 3DLite: towards commodity 3D scanning for content creation. ACM Trans. Graph. 36, 6 (2017), 203--1.Google Scholar
Digital Library
- Anil K Jain and Farshid Farrokhnia. 1991. Unsupervised texture segmentation using Gabor filters. Pattern recognition 24, 12 (1991), 1167--1186.Google Scholar
- Yifeng Jiang and C Karen Liu. 2018. Data-Augmented Contact Model for Rigid Body Simulation. arXiv preprint arXiv:1803.04019 (2018).Google Scholar
- Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision. Springer, 694--711.Google Scholar
Cross Ref
- Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. 2018. Learning Category-Specific Mesh Reconstruction from Image Collections. In The European Conf. on Comput. Vision (ECCV). 371--386.Google Scholar
Cross Ref
- Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson Surface Reconstruction. In Proc. of the Fourth Eurographics Symp. on Geom. Processing (Cagliari, Sardinia, Italy) (SGP '06). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 61--70. http://dl.acm.org/citation.cfm?id=1281957.1281965Google Scholar
Digital Library
- Alina Kloss, Stefan Schaal, and Jeannette Bohg. 2017. Combining learned and analytical models for predicting action effects. arXiv preprint arXiv:1710.04102 (2017).Google Scholar
- Karl Krissian, Xunlei Wu, and Vincent Luboz. 2006. Smooth vasculature reconstruction with circular and elliptic cross sections. Medicine Meets Virtual Reality 14: Accelerating Change in Healthcare: Next Medical Toolkit 119 (2006), 273.Google Scholar
- Yangyan Li, Xiaochen Fan, Niloy J Mitra, Daniel Chamovitz, Daniel Cohen-Or, and Baoquan Chen. 2013. Analyzing growing plants from 4D point cloud data. ACM Transactions on Graphics (TOG) 32, 6 (2013), 1--10.Google Scholar
Digital Library
- Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. 2017. RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
- Aristid Lindenmayer. 1968. Mathematical models for cellular interactions in development I. Filaments with one-sided inputs. Journal of theoretical biology 18, 3 (1968), 280--299.Google Scholar
- Yotam Livny, Feilong Yan, Matt Olson, Baoquan Chen, Hao Zhang, and Jihad El-sana. 2010. Automatic Reconstruction of Tree Skeletal Structures from Point Clouds. Proc. SIGGRAPH Asia 2010 29 (2010), 151:1--151:8. Issue 6.Google Scholar
Cross Ref
- C. Loop. 2001. Triangle mesh subdivision with bounded curvature and the convex hull property. Technical Report MSR-TR-2001-24. Microsoft Research.Google Scholar
- Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In European Conf. on Comput. Vision. Springer, 154--169.Google Scholar
Cross Ref
- Gellért Máttyus, Wenjie Luo, and Raquel Urtasun. 2017. Deeproadmapper: Extracting road topology from aerial images. In The IEEE International Conference on Computer Vision (ICCV).Google Scholar
Cross Ref
- Agata Mosinska, Pablo Márquez-Neila, Mateusz Koziński, and Pascal Fua. 2018. Beyond the Pixel-Wise Loss for Topology-Aware Delineation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
- Pierre Moulon, Pascal Monasse, Romuald Perrot, and Renaud Marlet. 2016. OpenMVG: Open multiple view geometry. In International Workshop on Reproducible Research in Pattern Recognition. Springer, 60--74.Google Scholar
- Matthias Müller and Nuttapong Chentanez. 2011. Solid Simulation with Oriented Particles. ACM TOG 30, 4, Article 92 (2011), 10 pages.Google Scholar
- Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. 2017. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Trans. Graph. 36, 4 (2017), 41.Google Scholar
Digital Library
- Przemyslaw Prusinkiewicz, Mark Hammel, Jim Hanan, and Radomír Měch. 1997. Visual models of plant development. In Handbook of formal languages. Springer, 535--597.Google Scholar
- Guoxiang Qu, Wenwei Zhang, Zhe Wang, Xing Dai, Jianping Shi, Junjun He, Fei Li, Xiulan Zhang, and Yu Qiao. 2018. StripNet: Towards Topology Consistent Strip Structure Segmentation. In Proceedings of the 26th ACM International Conference on Multimedia (Seoul, Republic of Korea) (MM '18). ACM, New York, NY, USA, 283--291. https://doi.org/10.1145/3240508.3240553Google Scholar
Digital Library
- Ed Quigley, Yue Yu, Jingwei Huang, Winnie Lin, and Ronald Fedkiw. 2018. Real-time Interactive Tree Animation. IEEE transactions on visualization and computer graphics 24, 5 (2018), 1717--1727.Google Scholar
Cross Ref
- Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.Google Scholar
Cross Ref
- Johannes L Schönberger and Jan-Michael Frahm. 2016. Structure-from-motion revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4104--4113.Google Scholar
Cross Ref
- Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proc. of the IEEE Conf. on Comput. Vision and Pattern Recognition (CVPR). IEEE, 519--528.Google Scholar
Digital Library
- Ondrej Stava, Sören Pirk, Julian Kratt, Baoquan Chen, Radomír Měch, Oliver Deussen, and Bedrich Benes. 2014. Inverse procedural modelling of trees. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 118--131.Google Scholar
- Ping Tan, Gang Zeng, Jingdong Wang, Sing Bing Kang, and Long Quan. 2007. Image-based tree modeling. In ACM Trans. Graph., Vol. 26. ACM, 87.Google Scholar
Digital Library
- Gabriel Taubin. 1995. A signal processing approach to fair surface design. In Proc. of the 22nd annual conf. on Comput. graphics and interactive techniques. ACM, 351--358.Google Scholar
Digital Library
- Carles Ventura, Jordi Pont-Tuset, Sergi Caelles, Kevis-Kokitsi Maninis, and Luc Van Gool. 2018. Iterative Deep Learning for Road Topology Extraction. In Proc. of the British Machine Vision Conf. (BMVC).Google Scholar
- Jason Weber and Joseph Penn. 1995. Creation and rendering of realistic trees. In Proc. 22nd Ann. Conf. Comput. Graph. Int. Tech. ACM, 119--128.Google Scholar
Digital Library
- Jamie Wither, Frédéric Boudon, M-P Cani, and Christophe Godin. 2009. Structure from silhouettes: a new paradigm for fast sketch-based design of trees. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 541--550.Google Scholar
- Changchang Wu. 2011. VisualSFM: A visual structure from motion system. http://ccwu.me/vsfm/.Google Scholar
- Changchang Wu. 2013. Towards linear-time incremental structure from motion. In 3D Vision-3DV 2013, 2013 International Conference on. IEEE, 127--134.Google Scholar
- Ke Xie, Feilong Yan, Andrei Sharf, Oliver Deussen, Baoquan Chen, and Hui Huang. 2015. Tree Modeling with Real Tree-Parts Examples. IEEE TVCG 22, 12 (Dec 2015), 2608--2618.Google Scholar
- Weipeng Xu, Avishek Chatterjee, Michael Zollhöfer, Helge Rhodin, Dushyant Mehta, Hans-Peter Seidel, and Christian Theobalt. 2018. MonoPerfCap: Human performance capture from monocular video. ACM Transactions on Graphics (TOG) 37, 2 (2018), 27.Google Scholar
Digital Library
- Tianfan Xue, Jiajun Wu, Zhoutong Zhang, Chengkai Zhang, Joshua B. Tenenbaum, and William T. Freeman. 2018. Seeing Tree Structure from Vibration. In The European Conf. on Comput. Vision (ECCV). 748--764.Google Scholar
- Yili Zhao and Jernej Barbič. 2013. Interactive Authoring of Simulation-Ready Plants. ACM Trans. Graph. 32, 4 (2013), 84:1--84:12.Google Scholar
Digital Library
- Ying Zheng, Steve Gu, Herbert Edelsbrunner, Carlo Tomasi, and Philip Benfey. 2011. Detailed reconstruction of 3D plant root shape. In Comput. Vision (ICCV), 2011 IEEE Int. Conf. on. IEEE, 2026--2033.Google Scholar
Digital Library
- Jin Zhou, Ananya Das, Feng Li, and Baoxin Li. 2008. Circular generalized cylinder fitting for 3D reconstruction in endoscopic imaging based on MRF. In Comput. Vision and Pattern Recognition Workshops, 2008. CVPRW'08. IEEE Comput. Society Conf. on. IEEE, 1--8.Google Scholar
- Silvia Zuffi, Angjoo Kanazawa, and Michael J Black. 2018. Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape From Images. In Proc. of the IEEE Conf. on Comput. Vision and Pattern Recognition (CVPR). 3955--3963.Google Scholar
Cross Ref
Index Terms
(auto-classified)Three Dimensional Reconstruction of Botanical Trees with Simulatable Geometry
Recommendations
Learning to reconstruct botanical trees from single images
We introduce a novel method for reconstructing the 3D geometry of botanical trees from single photographs. Faithfully reconstructing a tree from single-view sensor data is a challenging and open problem because many possible 3D trees exist that fit the ...
Realistic surface geometry reconstruction using a hand-held RGB-D camera
In this paper, we have proposed a novel approach for the reconstruction of real object/scene with realistic surface geometry using a hand-held, low-cost, RGB-D camera. To achieve accurate reconstruction, the most important issues to consider are the ...
Single-View RGBD-Based Reconstruction of Dynamic Human Geometry
ICCVW '13: Proceedings of the 2013 IEEE International Conference on Computer Vision WorkshopsWe present a method for reconstructing the geometry and appearance of indoor scenes containing dynamic human subjects using a single (optionally moving) RGBD sensor. We introduce a framework for building a representation of the articulated scene ...






Comments