Image-based reconstruction and synthesis of dense foliage

Abstract

Flora is an element in many computer-generated scenes. But trees, bushes and plants have complex geometry and appearance, and are difficult to model manually. One way to address this is to capture models directly from the real world. Existing techniques have focused on extracting macro structure such as the branching structure of trees, or the structure of broad-leaved plants with a relatively small number of surfaces. This paper presents a finer scale technique to demonstrate for the first time the processing of densely leaved foliage - computation of 3D structure, plus extraction of statistics for leaf shape and the configuration of neighboring leaves. Our method starts with a mesh of a single exemplar leaf of the target foliage. Using a small number of images, point cloud data is obtained from multi-view stereo, and the exemplar leaf mesh is fitted non-rigidly to the point cloud over several iterations. In addition, our method learns a statistical model of leaf shape and appearance during the reconstruction phase, and a model of the transformations between neighboring leaves. This information is useful in two ways - to augment and increase leaf density in reconstructions of captured foliage, and to synthesize new foliage that conforms to a user-specified layout and density. The result of our technique is a dense set of captured leaves with realistic appearance, and a method for leaf synthesis. Our approach excels at reconstructing plants and bushes that are primarily defined by dense leaves and is demonstrated with multiple examples.

References

1. Ahrends, H. E., Etzold, S., Eugster, W., Buchmann, N., Jeanneret, F., and Wanner, H. 2009. Use of digital images to observe forest phenology and drought stress. In EGU General Assembly Conference Abstracts, vol. 11, 10886.
2. Anastacio, F., Sousa, M. C., Samavati, F., and Jorge, J. A. 2006. Modeling plant structures using concept sketches. In Proceedings of NPAR, 105--113.
3. Baranoski, G. V. G., and Rokne, J. G. 2001. Efficiently simulating scattering of light by leaves. The Visual Computer 17, 8, 491--505.
4. Beeler, T., Bickel, B., Sumner, R., Beardsley, P., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH 98).
5. Besl, P. J., and McKay, N. D. 1992. A method for registration of 3-d shapes. IEEE Trans. on PAMI 14, 2, 239--256.
6. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In SIGGRAPH, 187--194.
7. Blinn, J. F. 1977. Models of light reflection for computer synthesized pictures. SIGGRAPH Comput. Graph. 11, 2 (July), 192--198.
8. Bradley, D., Boubekeur, T., and Heidrich, W. 2008. Accurate multi-view reconstruction using robust binocular stereo and surface meshing. In Proc. CVPR.
9. Chen, X., Neubert, B., Xu, Y.-Q., Deussen, O., and Kang, S. B. 2008. Sketch-based tree modeling using markov random field. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 27, 109.
10. COBWEB. 2013. Citizen Observatory Web,edina.ac.uk.
11. Deussen, O., and Lintermann, B. 2005. Digital Design of Nature: Computer Generated Plants and Organics. Springer-Verlag.
12. Diener, J., Reveret, L., and Eugene, F. 2006. Hierarchical retargetting of 2d motion fields to the animation of 3d plant models. In Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, 187--195.
13. Fischler, M. A., and Bolles, R. C. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6, 381--395.
14. Goesele, M., Curless, B., and Seitz, S. M. 2006. Multi-view stereo revisited. In CVPR.
15. Greene, N. 1989. Voxel space automata: modeling with stochastic growth processes in voxel space. SIGGRAPH Comput. Graph. 23, 3, 175--184.
16. Habel, R., Kusternig, A., and Wimmer, M. 2007. Physically based real-time translucency for leaves. In Proc. Eurographics Symposium on Rendering, 253--263.
17. Jakob, W., 2012. Mitsuba renderer. www.mitsuba-renderer.org.
18. Li, C., Deussen, O., Song, Y.-Z., Willis, P., and Hall, P. 2011. Modeling and generating moving trees from video. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 127.
19. Lindenmayer, A. 1968. Mathematical models for cellular interactions in development ii. simple and branching filaments with two-sided inputs. Journal of Theoretical Biology 18, 3, 300--315.
20. Lintermann, B., and Deussen, O. 1999. Interactive modeling of plants. IEEE Comput. Graph. Appl. 19, 56--65.
21. Livny, Y., Yan, F., Olson, M., Chen, B., Zhang, H., and El-Sana, J. 2010. Automatic reconstruction of tree skeletal structures from point clouds. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 29, 151.
22. Livny, Y., Pirk, S., Cheng, Z., Yan, F., Deussen, O., Cohen-Or, D., and Chen, B. 2011. Texture-lobes for tree modelling. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 53.
23. Ma, W., Zha, H., Liu, J., Zhang, X., and Xiang, B. 2008. Image-based plant modeling by knowing leaves from their apexes. In Proc. ICPR.
24. Mundermann, L., MacMurchy, P., Pivovarov, J., and Prusinkiewicz, P. 2003. Modeling lobed leaves. In Computer Graphics International.
25. Neubert, B., Franken, T., and Deussen, O. 2007. Approximate image-based tree-modeling using particle flows. ACM Trans. Graphics (Proc. SIGGRAPH) 26, 88.
26. Okabe, M., Owada, S., and Igarashi, T. 2005. Interactive design of botanical trees using freehand sketches and example-based editing. Computer Graphics Forum 24, 3, 487--496.
27. Palubicki, W., Horel, K., Longay, S., Runions, A., Lane, B., Měch, R., and Prusinkiewicz, P. 2009. Self-organizing tree models for image synthesis. ACM Trans. Graph. (Proc. SIGGRAPH) 28, 58.
28. Papazov, C., and Burschka, D. 2011. An efficient ransac for 3d object recognition in noisy and occluded scenes. In Proceedings of the 10th Asian conference on Computer vision, 135--148.
29. Pirk, S., Niese, T., Deussen, O., and Neubert, B. 2012. Capturing and animating the morphogenesis of polygonal tree models. ACM Trans. Graph. 31, 6, 169:1--169:10.
30. Pirk, S., Stava, O., Kratt, J., Said, M. A. M., Neubert, B., Měch, R., Benes, B., and Deussen, O. 2012. Plastic trees: interactive self-adapting botanical tree models. ACM Trans. Graph. 31, 4, 50:1--50:10.
31. Prusinkiewicz, P., and Lindenmayer, A. 1990. The algorithmic beauty of plants. Springer-Verlag.
32. Prusinkiewicz, P., James, M., and Měch, R. 1994. Synthetic topiary. In SIGGRAPH '94, 351--358.
33. Quan, L., Tan, P., Zeng, G., Yuan, L., Wang, J., and Kang, S. B. 2006. Image-based plant modeling. ACM Trans. Graphics (Proc. SIGGRAPH) 25, 599--604.
34. Reche-Martinez, A., Martin, I., and Drettakis, G. 2004. Volumetric reconstruction and interactive rendering of trees from photographs. ACM Trans. Graphics (Proc. SIGGRAPH) 23, 720--727.
35. Schnabel, R., Wahl, R., and Klein, R. 2007. Efficient ransac for point-cloud shape detection. Computer Graphics Forum 26, 2, 214--226.
36. Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In CVPR.
37. Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: Exploring image collections in 3d. ACM Transactions on Graphics (Proc. of SIGGRAPH) 25, 3, 835--846.
38. Sonohat, G., Sinoquet, H., Kulandaivelu, V., Combes, D., and Lescourret, F. 2006. Three-dimensional reconstruction of partially 3d-digitized peach tree canopies. Tree Physiology 26, 3, 337--351.
39. Sorkine, O., Cohen-Or, D., Lipman, Y., Alexa, M., Rössl, C., and Seidel, H.-P. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, 175--184.
40. Strecha, C., Fransens, R., and Gool, L. V. 2006. Combined depth and outlier estimation in multi-view stereo. In CVPR.
41. Szeliski, R. 1999. A multi-view approach to motion and stereo. In CVPR.
42. Talton, J. O., Lou, Y., Lesser, S., Duke, J., Měch, R., and Koltun, V. 2011. Metropolis procedural modeling. ACM Trans. Graph. 30, 11:1--11:14.
43. Tan, P., Zeng, G., Wang, J., Kang, S. B., and Quan, L. 2007. Image-based tree modeling. ACM Trans. Graphics (Proc. SIGGRAPH) 26, 87.
44. Tan, P., Fang, T., Xiao, J., Zhao, P., and Quan, L. 2008. Single image tree modeling. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 27, 108.
45. Wilson, E. 2009. Ant Lovers Unite,www.npr.org.
46. Wither, J., Boudon, F., Cani, M.-P., and Godin, C. 2009. Structure from silhouettes: a new paradigm for fast sketch-based design of trees. Computer Graphics Forum 28, 2, 541--550.
47. Xu, H., Gossett, N., and Chen, B. 2007. Knowledge and heuristic-based modeling of laser-scanned trees. ACM Trans. Graphics 26, 19--31.

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Document scanning

      • Graphics recognition and interpretation

• Computing methodologies

  • Computer graphics

    • Image manipulation