Semantic decomposition and reconstruction of residential scenes from LiDAR data

Abstract

We present a complete system to semantically decompose and reconstruct 3D models from point clouds. Different than previous urban modeling approaches, our system is designed for residential scenes, which consist of mainly low-rise buildings that do not exhibit the regularity and repetitiveness as high-rise buildings in downtown areas. Our system first automatically labels the input into distinctive categories using supervised learning techniques. Based on the semantic labels, objects in different categories are reconstructed with domain-specific knowledge. In particular, we present a novel building modeling scheme that aims to decompose and fit the building point cloud into basic blocks that are block-wise symmetric and convex. This building representation and its reconstruction algorithm are flexible, efficient, and robust to missing data. We demonstrate the effectiveness of our system on various datasets and compare our building modeling scheme with other state-of-the-art reconstruction algorithms to show its advantage in terms of both quality and speed.

References

1. Akbarzadeh, A., Frahm, J.-M., Mordohai, P., Clipp, B., Engels, C., Gallup, D., Merrell, P., Phelps, M., Sinha, S., Talton, B., Wang, L., Yang, Q., Stewnius, H., Yang, R., Welch, G., Towles, H., Nistr, D., and Pollefeys, M. 2006. Towards urban 3d reconstruction from video. In 3DPVT.
2. Bokeloh, M., Berner, A., Wand, M., Seidel, H.-P., and Schilling, A. 2009. Symmetry detection using line features. Computer Graphics Forum (Proceedings of Eurographics).
3. Chauve, A.-L., Labatut, P., and Pons, J.-P. 2010. Robust piecewise-planar 3d reconstruction and completion from large-scale unstructured point data. In CVPR.
4. Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In SIGGRAPH, 11--20.
5. Dorninger, P., and Pfeifer, N. 2008. A comprehensive automated 3d approach for building extraction, reconstruction, and regularization from airborne laser scanning point clouds. Sensor 8, 7323--7343.
6. Frahm, J.-M., Georgel, P., Gallup, D., Johnson, T., Raguram, R., Wu, C., Jen, Y.-H., Dunn, E., Clipp, B., Lazebnik, S., and Pollefeys, M. 2010. Building rome on a cloudless day. In ECCV.
7. Frueh, C., and Zakhor, A. 2003. Constructing 3d city models by merging aerial and ground views. IEEE Computer Graphics and Applications 23, 6, 52--61.
8. Frueh, C., Jain, S., and Zakhor, A. 2005. Data processing algorithms for generating textured 3d building facade meshes from laser scans and camera images. Int. J. Comput. Vision 61, 159--184.
9. Golovinskiy, A., Kim, V., and Funkhouser, T. 2009. Shape-based recognition of 3d point clouds in urban environments. In ICCV.
10. Irschara, A., Zach, C., Klopschitz, M., and Bischof, H. 2012. Large-scale, dense city reconstruction from user-contributed photos. Computer Vision and Image Understanding 116, 1, 2--15.
11. Karantzalos, K., and Paragios, N. 2010. Large-scale building reconstruction through information fusion and 3-d priors. IEEE Transactions on Geoscienec and Remote Sensing 48, 5, 2283--2296.
12. Lafarge, F., and Mallet, C. 2011. Bulding large urban environment from unstructed point data. In International Conference on Computer Vision, 1068--1075.
13. Lafarge, F., Descombes, X., Zerubia, J., and Deseilligny, M. P. 2010. Structural approach for building reconstruction from a single dsm. ieee trans. PAMI 32, 1, 135--147.
14. Li, Y., Wu, X., Chrysanthou, Y., Sharf, A., Cohen-Or, D., and Mitra, N. J. 2011. Globfit: Consistently fitting primitives by discovering global relations. ACM Transactions on Graphics 30, 4.
15. Li, Y., Zheng, Q., Sharf, A., Cohen-Or, D., Chen, B., and Mitra, N. J. 2011. 2d-3d fusion for layer decomposition of urban facades. In ICCV.
16. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3d video stabilization. ACM Trans. Graph. 28, 3 (July), 44:1--44:9.
17. Müller, P., Wonka, P., Haegler, S., Ulmer, A., and Van Gool, L. 2006. Procedural modeling of buildings. ACM Trans. Graph. 25 (July), 614--623.
18. Musialski, P., Wonka, P., Aliaga, D. G., Wimmer, M., van Gool, L., and Purgathofer, W. 2012. A survey of urban reconstruction. In EUROGRAPHICS 2012 State of the Art Reports, Eurographics Association,
19. Nan, L., Sharf, A., Zhang, H., Cohen-Or, D., and Chen, B. 2010. Smartboxes for interactive urban reconstruction. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2010) 29, 4, Article 93.
20. Pauly, M., Mitra, N. J., Wallner, J., Pottmann, H., and Guibas, L. J. 2008. Discovering structural regularity in 3d geometry. ACM Trans. Graph. 27 (August), 43:1--43:11.
21. Poullis, C., and You, S. 2009. Automatic creation of massive virtual cities. In IEEE Virtual Reality Conference, 199--202.
22. Poullis, C., and You, S. 2009. Photorealistic large-scale urban city model reconstruction. IEEE Transaction on Visualization and Computer Graphics 15, 4, 654--669.
23. Schaffalitzky, F., and Zisserman, A. 1999. Geometric grouping of repeated elements within images. In Shape, Contour and Grouping in Computer Vision, 165--181.
24. Schnabel, R., Wahl, R., and Klein, R. 2007. Efficient ransac for point-cloud shape detection. Computer Graphics Forum 26, 2 (June), 214--226.
25. Shao, T., Xu, W., Zhou, K., Wang, J., Li, D., and Guo, B. 2012. An interactive approach to semantic modeling of indoor scenes with an rgbd camera. ACM Trans. Graph..
26. Sinha, S., Steedly, D., Szeliski, R., Agrawala, M., and Pollefeys, M. 2008. Interactive 3d architectural modeling from unordered photo collections. In SIGGRAPH Asia, 159:1--159:10.
27. Van Gool, L., Zeng, G., van den Borre, F., and Muller, P. 2007. Invited paper: Towards mass-produced building models. In PIA07, 209.
28. Vanegas, C. A., Aliaga, D. G., Wonka, P., Mueller, P., Waddell, P., and Watson, B. 2010. Modeling the appearance and behavior of urban spaces. Computer Graphics Forum 29, 1, 25--42. Also in Eurographics 2009 STAR (State-of-the-Art Report).
29. Vanegas, C. A., Aliaga, D. G., and Benes, B. 2012. Automatic extraction of manhattan-world building masses from 3d laser range scans. IEEE Transactions on Visualization and Computer Graphics 18, 10, 1627--1637.
30. Werner, T., and Zisserman, A. 2002. New techniques for automated architectural reconstruction from photographs. In ECCV.
31. Wu, C., Frahm, J.-M., and Pollefeys, M. 2010. Detecting large repetitive structures with salient boundaries. In ECCV.
32. Xiao, J., and Furukawa, Y. 2012. Reconstructing the world's museums. In ECCV.
33. Xiao, J., and Quan, L. 2009. Multiple view semantic segmentation for street view images. In ICCV.
34. Xiao, J., Fang, T., Tan, P., Ofek, P. Z. E., and Quan, L. 2008. Image-based façade modeling. ACM Trans. Graph. 27, 5, 1--10.
35. Xiao, J., Fang, T., Zhao, P., Lhuillier, M., and Quan, L. 2009. Image-based street-side city modeling. ACM Transactions on Grpahics 28, 5, 10--19.
36. Zhang, C., Wang, L., and Yang, R. 2010. Semantic segmentation of urban scenes using dense depth maps. In ECCV.
37. Zhou, Q.-Y., and Neumann, U. 2010. 2.5d dual contouring: a robust approach to creating building models from aerial lidar point clouds. In ECCV.
38. Zhou, Q., and Neumann, U. 2011. 2.5d building modeling with topology control. In CVPR.
39. Zhou, Q.-Y., and Neumann, U. 2012. 2.5d building modeling by discovering global regularities. In CVPR.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

      • Computer vision tasks

        • Scene understanding

      • Image and video acquisition

        • 3D imaging

  • Computer graphics

    • Shape modeling

• Theory of computation

  • Randomness, geometry and discrete structures