OpenSurfaces: a richly annotated catalog of surface appearance

Abstract

The appearance of surfaces in real-world scenes is determined by the materials, textures, and context in which the surfaces appear. However, the datasets we have for visualizing and modeling rich surface appearance in context, in applications such as home remodeling, are quite limited. To help address this need, we present OpenSurfaces, a rich, labeled database consisting of thousands of examples of surfaces segmented from consumer photographs of interiors, and annotated with material parameters (reflectance, material names), texture information (surface normals, rectified textures), and contextual information (scene category, and object names).

Retrieving usable surface information from uncalibrated Internet photo collections is challenging. We use human annotations and present a new methodology for segmenting and annotating materials in Internet photo collections suitable for crowdsourcing (e.g., through Amazon's Mechanical Turk). Because of the noise and variability inherent in Internet photos and novice annotators, designing this annotation engine was a key challenge; we present a multi-stage set of annotation tasks with quality checks and validation. We demonstrate the use of this database in proof-of-concept applications including surface retexturing and material and image browsing, and discuss future uses. OpenSurfaces is a public resource available at http://opensurfaces.cs.cornell.edu/.

References

1. Adelson, E. H. 2001. On seeing stuff: the perception of materials by humans and machines. Proc. SPIE Human Vision and Electronic Imaging 4299.
2. Ben-Artzi, A., Overbeck, R., and Ramamoorthi, R. 2006. Real-time BRDF editing in complex lighting. In SIGGRAPH Conf. Proc.
3. Brainard, D. H., Brunt, W., and Speigle, J. 1997. Color constancy in the nearly natural image. J. of the Optical Society of America 14, 9.
4. Cgal, Computational Geometry Algorithms Library. http://www.cgal.org/.
5. Chen, X., Golovinskiy, A., and Funkhouser, T. 2009. A benchmark for 3D mesh segmentation. In SIGGRAPH Conf. Proc.
6. Cole, F., Sanik, K., DeCarlo, D., Finkelstein, A., Funkhouser, T., Rusinkiewicz, S., and Singh, M. 2009. How well do line drawings depict shape? In SIGGRAPH Conf. Proc.
7. Dana, K., Van-Ginneken, B., Nayar, S., and Koenderink, J. 1999. Reflectance and texture of real world surfaces. ACM Transactions on Graphics 18, 1.
8. Debevec, P. 1998. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In SIGGRAPH Conf. Proc.
9. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and FeiFei, L. 2009. ImageNet: A large-scale hierarchical image database. In Proc. Comp. Vision and Pattern Recognition.
10. Dror, R., Adelson, E. H., and Willsky, A. 2001. Estimating surface reflectance properties from images under unknown illumination. In Proc. SPIE Human Vision and Electronic Imaging.
11. Endres, I., Farhadi, A., Hoiem, D., and Forsyth, D. 2010. The benefits and challenges of collecting richer object annotations. In Workshop on Advancing Computer Vision with Humans in the Loop.
12. Feng, C., Deng, F., and Kamat, V. R. 2010. Semi-automatic 3D reconstruction of piecewise planar building models from single image. In Int. Conf. on Construction Appl. of Virtual Reality.
13. Fleming, R. W., Dror, R. O., and Adelson, E. H. 2003. Real-world illumination and the perception of surface reflectance properties. J. of Vision 3, 5.
14. Fleming, R. W., Torralba, A., and Adelson, E. H. 2004. Specular reflections and the perception of shape. J. of Vision 4, 9.
15. Geisler-Moroder, D., and Dür, A. 2010. A new Ward BRDF model with bounded albedo. In Proc. Eurographics Symp. on Rendering.
16. Gingold, Y., Shamir, A., and Cohen-Or, D. 2012. Micro perceptual human computation. ACM Transactions on Graphics 31, 5.
17. Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. In SIGGRAPH Conf. Proc., 4:1--4:7.
18. Hu, D., Bo, L., and Ren, X. 2011. Toward robust material recognition for everyday objects. In Proc. British Machine Vision Conf.
19. Juracek, J. 1996. Surfaces: Visual Research for Artists and Designers. Norton.
20. Karsch, K., Hedau, V., Forsyth, D., and Hoiem, D. 2011. Rendering synthetic objects into legacy photographs. In SIGGRAPH Asia Conf. Proc.
21. Kerr, W. B., and Pellacini, F. 2010. Toward evaluating material design interface paradigms for novice users. ACM Transactions on Graphics 29, 4.
22. Koenderink, J. J., Doorn, A. J. V., and Kappers, A. M. L. 1992. Surface perception in pictures. Perception & Psychophysics.
23. Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. In SIGGRAPH Conf. Proc.
24. Liu, Y., Lin, W.-C., and Hays, J. 2004. Near regular texture analysis and manipulation. In SIGGRAPH Conf. Proc.
25. Liu, C., Sharan, L., Adelson, E., and Rosenholtz, R. 2010. Exploring features in a Bayesian framework for material recognition. In Proc. Comp. Vision and Pattern Recognition.
26. Marge, M., Banerjee, S., and Rudnicky, A. I. 2010. Using the Amazon Mechanical Turk for transcription of spoken language. In Int. Conf. on Acoustics, Speech, and Signal Processing.
27. Matusik, W., Pfister, H., Brand, M., and McMillan, L. 2003. A data-driven reflectance model. ACM Transactions on Graphics 22, 3.
28. Ngan, A., Durand, F., and Matusik, W. 2005. Experimental analysis of BRDF models. In Proc. Eurographics Symp. on Rendering.
29. Pellacini, F., Ferwerda, J. A., and Greenberg, D. P. 2000. Toward a psychophysically-based light reflection model for image synthesis. In SIGGRAPH Conf. Proc.
30. Ramanarayanan, G., Ferwerda, J., Walter, B., and Bala, K. 2007. Visual equivalence: Towards a new standard for image fidelity. In SIGGRAPH Conf. Proc.
31. Reinhard, E., Stark, M., Shirley, P., and Ferwerda, J. 2002. Photographic tone reproduction for digital images. In SIGGRAPH Conf. Proc.
32. Ren, P., Wang, J., Snyder, J., Tong, X., and Guo, B. 2011. Pocket reflectometry. In SIGGRAPH Conf. Proc.
33. Romeiro, F., and Zickler, T. 2010. Blind reflectometry. In Proc. European Conf. on Comp. Vision.
34. Rubinstein, M., Gutierrez, D., Sorkine, O., and Shamir, A. 2010. A comparative study of image retargeting. In SIGGRAPH Asia Conf. Proc.
35. Russell, B. C., Torralba, A., Murphy, K. P., and Freeman, W. T. 2008. LabelMe: A database and web-based tool for image annotation. Int. J. of Computer Vision 77, 1--3.
36. Sharan, L., Rosenholtz, R., and Adelson, E. H. 2009. Material perception: What can you see in a brief glance? J. of Vision 9, 8.
37. Tardif, J.-P. 2009. Non-iterative approach for fast and accurate vanishing point detection. In Proc. Int. Conf. on Comp. Vision.
38. Toldo, R., and Fusiello, A. 2008. Robust multiple structures estimation with J-linkage. In Proc. European Conf. on Comp. Vision.
39. Torralba, A., Fergus, R., and Freeman, W. T. 2008. 80 million tiny images: A large data set for nonparametric object and scene recognition. Trans. on Pattern Analysis and Machine Intelligence 30, 11.
40. Vangorp, P., Laurijssen, J., and Dutré, P. 2007. The influence of shape on the perception of material reflectance. ACM Transactions on Graphics 26, 3.
41. von Gioi, R. G., Jakubowicz, J., Morel, J.-M., and Randall, G. 2010. LSD: A fast line segment detector with a false detection control. Trans. on Pattern Analysis and Machine Intelligence 32, 4.
42. Walter, B., Khungurn, P., and Bala, K. 2012. Bidirectional lightcuts. In SIGGRAPH Conf. Proc.
43. Ward, G. 1992. Measuring and modeling anisotropic reflection. In SIGGRAPH Conf. Proc.
44. Welinder, P., Branson, S., Belongie, S., and Perona, P. 2010. The multidimensional wisdom of crowds. In Proc. Neural Information Processing Systems.
45. Weyrich, T., Lawrence, J., Lensch, H. P. A., Rusinkiewicz, S., and Zickler, T. 2009. Principles of appearance acquisition and representation. Foundations and Trends in Computer Graphics and Vision 4, 2.
46. Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. 2010. SUN database: Large-scale scene recognition from abbey to zoo. In Proc. Comp. Vision and Pattern Recognition.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

      • Computer vision representations

        • Appearance and texture representations

      • Computer vision tasks

        • Scene understanding