Scene reconstruction from high spatio-angular resolution light fields

Abstract

This paper describes a method for scene reconstruction of complex, detailed environments from 3D light fields. Densely sampled light fields in the order of 10⁹ light rays allow us to capture the real world in unparalleled detail, but efficiently processing this amount of data to generate an equally detailed reconstruction represents a significant challenge to existing algorithms. We propose an algorithm that leverages coherence in massive light fields by breaking with a number of established practices in image-based reconstruction. Our algorithm first computes reliable depth estimates specifically around object boundaries instead of interior regions, by operating on individual light rays instead of image patches. More homogeneous interior regions are then processed in a fine-to-coarse procedure rather than the standard coarse-to-fine approaches. At no point in our method is any form of global optimization performed. This allows our algorithm to retain precise object contours while still ensuring smooth reconstructions in less detailed areas. While the core reconstruction method handles general unstructured input, we also introduce a sparse representation and a propagation scheme for reliable depth estimates which make our algorithm particularly effective for 3D input, enabling fast and memory efficient processing of "Gigaray light fields" on a standard GPU. We show dense 3D reconstructions of highly detailed scenes, enabling applications such as automatic segmentation and image-based rendering, and provide an extensive evaluation and comparison to existing image-based reconstruction techniques.

References

1. Adelson, E. H., and Wang, J. Y. A. 1992. Single lens stereo with a plenoptic camera. IEEE PAMI 14, 2.
2. Ayvaci, A., Raptis, M., and Soatto, S. 2012. Sparse occlusion detection with optical flow. IJCV 97, 3.
3. Basha, T., Avidan, S., Hornung, A., and Matusik, W. 2012. Structure and motion from scene registration. In CVPR.
4. Beeler, T., Bickel, B., Beardsley, P. A., Sumner, B., and Gross, M. H. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graph. 29, 4.
5. Bishop, T. E., and Favaro, P. 2010. Full-resolution depth map estimation from an aliased plenoptic light field. In ACCV.
6. Bishop, T., Zanetti, S., and Favaro, P. 2009. Light field superresolution. In ICCP.
7. Bleyer, M., Rother, C., Kohli, P., Scharstein, D., and Sinha, S. 2011. Object stereo --- joint stereo matching and object segmentation. In CVPR.
8. Bolles, R. C., Baker, H. H., and Marimont, D. H. 1987. Epipolar-plane image analysis: An approach to determining structure from motion. IJCV 1, 1.
9. Buehler, C., Bosse, M., McMillan, L., Gortler, S. J., and Cohen, M. F. 2001. Unstructured lumigraph rendering. In SIGGRAPH.
10. Čech, J., and Šára, R. 2007. Efficient sampling of disparity space for fast and accurate matching. In CVPR.
11. Chai, J., Chan, S.-C., Shum, H.-Y., and Tong, X. 2000. Plenoptic sampling. In SIGGRAPH.
12. Chen, W.-C., Bouguet, J.-Y., Chu, M. H., and Grzeszczuk, R. 2002. Light field mapping: Efficient representation and hardware rendering of surface light fields. In SIGGRAPH.
13. Comaniciu, D., and Meer, P. 2002. Mean shift: A robust approach toward feature space analysis. IEEE PAMI 24, 5.
14. Criminisi, A., Kang, S. B., Swaminathan, R., Szeliski, R., and Anandan, P. 2005. Extracting layers and analyzing their specular properties using epipolar-plane-image analysis. CVIU 97, 1.
15. Davis, A., Levoy, M., and Durand, F. 2012. Unstructured light fields. Comput. Graph. Forum 31, 2.
16. Duda, R., Hart, P., and Stork, D. 1995. Pattern Classification and Scene Analysis, 2nd ed.
17. Fitzgibbon, A., Wexler, Y., and Zisserman, A. 2005. Image-based rendering using image-based priors. IJCV 63, 2.
18. Furukawa, Y., and Ponce, J. 2010. Accurate, dense, and robust multi-view stereopsis. IEEE PAMI 32, 8.
19. Furukawa, Y., Curless, B., Seitz, S. M., and Szeliski, R. 2010. Towards Internet-scale multi-view stereo. In CVPR.
20. Fusiello, A., Trucco, E., and Verri, A. 2000. A compact algorithm for rectification of stereo pairs. Mach. Vis. Appl. 12, 1. Geiger, A., Roser, M., and Urtasun, R. 2010. Efficient large-scale stereo matching. In ACCV.
21. Georgiev, T., and Lumsdaine, A. 2010. Reducing plenoptic camera artifacts. Comp. Graph. Forum 29, 6.
22. Goldlücke, B., and Magnor, M. 2003. Joint 3D-reconstruction and background separation in multiple views using graph cuts. In CVPR.
23. Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The Lumigraph. In SIGGRAPH.
24. Hirschmüller, H. 2005. Accurate and efficient stereo processing by semi-global matching and mutual information. In CVPR.
25. Humayun, A., Mac Aodha, O., and Brostow, G. 2011. Learning to find occlusion regions. In CVPR.
26. Isaksen, A., McMillan, L., and Gortler, S. J. 2000. Dynamically reparameterized light fields. In SIGGRAPH.
27. Kang, S. B., and Szeliski, R. 2004. Extracting view-dependent depth maps from a collection of images. IJCV 58, 2.
28. Kolmogorov, V., and Zabih, R. 2001. Computing visual correspondence with occlusions via graph cuts. In ICCV.
29. Levoy, M., and Hanrahan, P. 1996. Light field rendering. In SIGGRAPH.
30. Liang, C.-K., Lin, T.-H., Wong, B.-Y., Liu, C., and Chen, H. H. 2008. Programmable aperture photography: multiplexed light field acquisition. ACM Trans. Graph. 27, 3.
31. Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., and Hanrahan, P. 2005. Light field photography with a hand-held plenoptic camera. Comp. Sci. Techn. Rep. CSTR 2.
32. Rav-Acha, A., Shor, Y., and Peleg, S. 2004. Mosaicing with parallax using time warping. In IVR.
33. Rhemann, C., Hosni, A., Bleyer, M., Rother, C., and Gelautz, M. 2011. Fast cost-volume filtering for visual correspondence and beyond. In CVPR.
34. Scharstein, D., and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47, 1--3.
35. Schechner, Y. Y., and Kiryati, N. 2000. Depth from defocus vs. stereo: How different really are they? IJCV 39, 2.
36. Seitz, S. M., and Dyer, C. R. 1999. Photorealistic scene reconstruction by voxel coloring. IJCV 35, 2.
37. Seitz, S., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In CVPR.
38. Snavely, N., Seitz, S. M., and Szeliski, R. 2008. Modeling the world from Internet photo collections. IJCV 80, 2.
39. Stich, T., Tevs, A., and Magnor, M. A. 2006. Global depth from epipolar volumes--a general framework for reconstructing non-lambertian surfaces. In 3DPVT.
40. Sun, X., Mei, X., Jiao, S., Zhou, M., and Wang, H. 2011. Stereo matching with reliable disparity propagation. In 3DIMPVT.
41. Sylwan, S. 2010. The application of vision algorithms to visual effects production. In ACCV.
42. Szeliski, R., and Scharstein, D. 2002. Symmetric sub-pixel stereo matching. In ECCV.
43. Vaish, V., Levoy, M., Szeliski, R., Zitnick, C., and Kang, S. 2006. Reconstructing occluded surfaces using synthetic apertures: Stereo, focus and robust measures. In CVPR.
44. Veeraraghavan, A., Raskar, R., Agrawal, A. K., Mohan, A., and Tumblin, J. 2007. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph. 26, 3.
45. Vu, H.-H., Keriven, R., Labatut, P., and Pons, J.-P. 2009. Towards high-resolution large-scale multi-view stereo. In CVPR.
46. Wanner, S., and Goldlücke, B. 2012. Globally consistent depth labeling of 4D light fields. In CVPR.
47. Wanner, S., Fehr, J., and Jaehne, B. 2011. Generating EPI representations of 4D light fields with a single lens focused plenoptic camera. In IISVC.
48. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antúnez, E. R., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Trans. Graph. 24, 3.
49. Wood, D. N., Azuma, D. I., Aldinger, K., Curless, B., Duchamp, T., Salesin, D. H., and Stuetzle, W. 2000. Surface light fields for 3D photography. In SIGGRAPH.
50. Yu, Y., Ferencz, A., and Malik, J. 2001. Extracting objects from range and radiance images. IEEE TVCG 7, 4.
51. Zhang, C., and Chen, T. 2004. A self-reconfigurable camera array. In EGSR.
52. Zhu, Z., Xu, G., and Lin, X. 1999. Panoramic EPI generation and analysis of video from a moving platform with vibration. In CVPR.
53. Ziegler, R., Bucheli, S., Ahrenberg, L., Magnor, M. A., and Gross, M. H. 2007. A bidirectional light field - hologram transform. Comput. Graph. Forum 26, 3.
54. Zitnick, C. L., and Kang, S. B. 2007. Stereo for image-based rendering using image over-segmentation. IJCV 75, 1.
55. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23, 3.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

      • Computer vision representations

        • Image representations

      • Computer vision tasks

        • Scene understanding