Reconstructing scenes with mirror and glass surfaces

Abstract

Planar reflective surfaces such as glass and mirrors are notoriously hard to reconstruct for most current 3D scanning techniques. When treated naïvely, they introduce duplicate scene structures, effectively destroying the reconstruction altogether. Our key insight is that an easy to identify structure attached to the scanner---in our case an AprilTag---can yield reliable information about the existence and the geometry of glass and mirror surfaces in a scene. We introduce a fully automatic pipeline that allows us to reconstruct the geometry and extent of planar glass and mirror surfaces while being able to distinguish between the two. Furthermore, our system can automatically segment observations of multiple reflective surfaces in a scene based on their estimated planes and locations. In the proposed setup, minimal additional hardware is needed to create high-quality results. We demonstrate this using reconstructions of several scenes with a variety of real mirrors and glass.

References

1. Sameer Agarwal, Keir Mierle, and Others. 2018. Ceres Solver, http://ceres-solver.org. (2018).
2. N. Arvanitopoulos, R. Achanta, and S. Süsstrunk. 2017. Single Image Reflection Suppression. In CVPR 2017. 1752--1760.
3. J. Balzer, D. Acevedo-Feliz, S. Soatto, S. Höfer, M. Hadwiger, and J. Beyerer. 2014. Cavlectometry: Towards Holistic Reconstruction of Large Mirror Objects. In 2nd International Conference on 3D Vision (3DV). 448--455.
4. J. Balzer, S. Höfer, and J. Beyerer. 2011. Multiview specular stereo reconstruction of large mirror surfaces. In CVPR 2011. 2537--2544.
5. L. G. Brown. 1992. A Survey of Image Registration Techniques. Computing Surveys 24, 4 (December 1992), 325--376.
6. Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3D: Learning from RGB-D Data in Indoor Environments. In 5th International Conference on 3D Vision (3DV).
7. Tongbo Chen, Michael Goesele, and Hans-Peter Seidel. 2006. Mesostructure from Specularity. In CVPR 2006, Vol. 2. 1825--1832.
8. Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.
9. A. DelPozo and S. Savarese. 2007. Detecting Specular Surfaces on Natural Images. In CVPR 2007.
10. Yuanyuan Ding and Jingyi Yu. 2008. Recovering shape characteristics on near-flat specular surfaces. In CVPR 2008.
11. J. Engel, V. Koltun, and D. Cremers. 2018. Direct Sparse Odometry. PAMI 40, 3 (2018), 611--625.
12. A. Fasano, M. Callieri, P. Cignoni, and R. Scopigno. 2003. Exploiting mirrors for laser stripe 3D scanning. In 3DIM 2003. 243--250.
13. Paul Foster, Zhenghong Sun, Jong Jin Park, and Benjamin Kuipers. 2013. VisAGGE: Visible angle grid for glass environments. In CVPR 2013. 2213--2220.
14. C. Godard, P. Hedman, W. Li, and G. J. Brostow. 2015. Multi-view Reconstruction of Highly Specular Surfaces in Uncontrolled Environments. In 3rd International Conference on 3D Vision (3DV). 19--27.
15. Ivo Ihrke, Kiriakos N. Kutulakos, Hendrik P. A. Lensch, Marcus Magnor, and Wolfgang Heidrich. 2010. Transparent and Specular Object Reconstruction. Computer Graphics Forum 29, 8 (2010), 2400--2426.
16. B. Jacquet, C. Häne, K. Köser, and M. Pollefeys. 2013. Real-World Normal Map Capture for Nearly Flat Reflective Surfaces. In CVPR 2013. 713--720.
17. Jun Jiang, Renato Miyagusuku, Atsushi Yamashita, and Hajime Asama. 2017. Glass Confidence Maps Building Based on Neural Networks Using Laser Range-Finders for Mobile Robots. In IEEE/SICE International Symposium on System Integration.
18. O. Kähler, V. Adrian Prisacariu, C. Yuheng Ren, X. Sun, P. Torr, and D. Murray. 2015. Very High Frame Rate Volumetric Integration of Depth Images on Mobile Devices. TVCG 21, 11 (Nov 2015), 1241--1250.
19. J. Kannala and S. S. Brandt. 2006. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. PAMI 28, 8 (Aug 2006), 1335--1340.
20. P.-F. Käshammer and A. Nüchter. 2015. Mirror identification and correction of 3D point clouds. The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 40, 5 (2015), 109.
21. U Klank, D. Carton, and M. Beetz. 2011. Transparent object detection and reconstruction on a mobile platform. In ICRA 2011. 5971--5978.
22. Rainer Koch, Stefan May, Patrick Murmann, and Andreas Nüchter. 2017b. Identification of Transparent and Specular Reflective Material in Laser Scans to Discriminate Affected Measurements for Faultless Robotic SLAM. Journal of Robotics and Autonomous Systems (JRAS) 87 (2017), 296--312.
23. R. Koch, S. May, and A. Nüchter. 2017a. Effective distinction of transparent and specular reflective objects in point clouds of a multi-echo laser scanner. In ICAR 2017. 566--571.
24. Brian Kulis and Michael I. Jordan. 2011. Revisiting k-means: New algorithms via Bayesian nonparametrics. arXiv preprint arXiv.1111.0352 (2011).
25. J. P. Lewis. 1995. Fast Normalized Cross-Correlation. In Vision Interface '95. Canadian Image Processing and Pattern Recognition Society.
26. M. Liu, R. Hartley, and M. Salzmann. 2015. Mirror Surface Reconstruction from a Single Image. PAMI 37, 4 (April 2015), 760--773.
27. William E. Lorensen and Harvey E. Cline. 1987. Marching cubes: A high resolution 3D surface construction algorithm. In SIGGRAPH. ACM, 163--169.
28. D. Miyazaki, M. Kagesawa, and K. Ikeuchi. 2004. Transparent surface modeling from a pair of polarization images. PAMI 26, 1 (2004), 73--82.
29. R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós. 2015. ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics 31, 5 (Oct 2015), 1147--1163.
30. R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In ISMAR 2011. 127--136.
31. Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013. Real-time 3D Reconstruction at Scale Using Voxel Hashing. ACM Trans. Graph. 32, 6, Article 169 (Nov. 2013), 11 pages.
32. Edwin Olson. 2011. AprilTag: A robust and flexible visual fiducial system. In ICRA 2011. 3400--3407.
33. Rui Rodrigues, João P. Barreto, and Urbano Nunes. 2010. Camera Pose Estimation Using Images of Planar Mirror Reflections. In ECCV 2010. 382--395.
34. YiChang Shih, D. Krishnan, F. Durand, and W. T. Freeman. 2015. Reflection removal using ghosting cues. In CVPR 2015. 3193--3201.
35. Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012. Image-based Rendering for Scenes with Reflections. ACM Trans. Graph. 31, 4, Article 100 (July 2012), 10 pages.
36. Julian Straub, Trevor Campbell, Jonathan P How, and John W Fisher. 2015. Small-variance nonparametric clustering on the hypersphere. In CVPR 2015. 334--342.
37. Marco Tarini, Hendrik P.A. Lensch, Michael Goesele, and Hans-Peter Seidel. 2005. 3D acquisition of mirroring objects using striped patterns. Graphical Models 67, 4 (2005), 233 -- 259.
38. Markus Unger, Thomas Pock, and Horst Bischof. 2008a. Interactive globally optimal image segmentation. Technical Report ICG-TR-08/02. Graz University of Technology.
39. Markus Unger, Thomas Pock, Werner Trobin, Daniel Cremers, and Horst Bischof. 2008b. TVSeg - Interactive Total Variation Based Image Segmentation. In BMVC 2008.
40. J. Wang and E. Olson. 2016. AprilTag 2: Efficient and robust fiducial detection. In IROS 2016. 4193--4198.
41. Qiaosong Wang, Haiting Lin, Yi Ma, Sing Bing Kang, and Jingyi Yu. 2015. Automatic Layer Separation using Light Field Imaging. CoRR abs/1506.04721 (2015). arXiv:1506.04721 http://arxiv.org/abs/1506.04721
42. Sven Wanner and Bastian Goldluecke. 2013. Reconstructing Reflective and Transparent Surfaces from Epipolar Plane Images. In GCPR 2013.
43. Tianfan Xue, Michael Rubinstein, Ce Liu, and William T. Freeman. 2015. A computational approach for obstruction-free photography. ACM Trans. Graph. 34, 4 (2015), 79:1--79:11.
44. Shao-Wen Yang and Chieh-Chih Wang. 2008. Dealing with laser scanner failure: Mirrors and windows. In ICRA 2008. 3009--3015.
45. S. W Yang and C. C. Wang. 2011. On Solving Mirror Reflection in LIDAR Sensing. IEEE/ASME Transactions on Mechatronics 16, 2 (April 2011), 255--265.
46. Y. Zhang, M. Ye, D. Manocha, and R. Yang. 2017. 3D Reconstruction in the Presence of Glass and Mirrors by Acoustic and Visual Fusion. PAMI (2017).
47. Zhengyou Zhang. 2000. A Flexible New Technique for Camera Calibration. PAMI 22, 11 (Nov. 2000), 1330--1334.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • 3D imaging

  • Computer graphics

    • Rendering

      • Reflectance modeling

    • Shape modeling