The effect of shape and illumination on material perception: model and applications

Abstract

Material appearance hinges on material reflectance properties but also surface geometry and illumination. The unlimited number of potential combinations between these factors makes understanding and predicting material appearance a very challenging task. In this work, we collect a large-scale dataset of perceptual ratings of appearance attributes with more than 215,680 responses for 42,120 distinct combinations of material, shape, and illumination. The goal of this dataset is twofold. First, we analyze for the first time the effects of illumination and geometry in material perception across such a large collection of varied appearances. We connect our findings to those of the literature, discussing how previous knowledge generalizes across very diverse materials, shapes, and illuminations. Second, we use the collected dataset to train a deep learning architecture for predicting perceptual attributes that correlate with human judgments. We demonstrate the consistent and robust behavior of our predictor in various challenging scenarios, which, for the first time, enables estimating perceived material attributes from general 2D images. Since our predictor relies on the final appearance in an image, it can compare appearance properties across different geometries and illumination conditions. Finally, we demonstrate several applications that use our predictor, including appearance reproduction using 3D printing, BRDF editing by integrating our predictor in a differentiable renderer, illumination design, or material recommendations for scene design.

References

1. Wendy J Adams, Gizem Kucukoglu, Michael S Landy, and Rafał K Mantiuk. 2018. Naturally glossy: Gloss perception, illumination statistics, and tone mapping. Journal of Vision 18, 13 (2018), 4--4.
2. Alan Agresti. 2003. Categorical data analysis. Vol. 482. John Wiley & Sons.
3. Barton L Anderson. 2011. Visual perception of materials and surfaces. Current Biology 21, 24 (2011), R978--R983.
4. Barton L Anderson and Juno Kim. 2009. Image statistics do not explain the perception of gloss and lightness. Journal of Vision 9, 11 (2009), 10--10.
5. Teun Baar, Sepideh Samadzadegan, Hans Brettel, Philipp Urban, and Maria V Ortiz Segovia. 2014. Printing gloss effects in a 2.5 D system. In Measuring, Modeling, and Reproducing Material Appearance, Vol. 9018. International Society for Optics and Photonics, 90180M.
6. Jacob Beck and Slava Prazdny. 1981. Highlights and the perception of glossiness. Perception & Psychophysics 30, 4 (1981), 407--410.
7. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material Recognition in the Wild with the Materials in Context Database. Computer Vision and Pattern Recognition (2015).
8. Julia Berzhanskaya, Gurumurthy Swaminathan, Jacob Beck, and Ennio Mingolla. 2005. Remote Effects of Highlights on Gloss Perception. Perception 34, 5 (2005), 565--575.
9. J Bieron and P Peers. 2020. An adaptive brdf fitting metric. In Computer Graphics Forum, Vol. 39. 59--74.
10. Adrien Bousseau, Emmanuelle Chapoulie, Ravi Ramamoorthi, and Maneesh Agrawala. 2011. Optimizing environment maps for material depiction. In Computer Graphics Forum, Vol. 30. 1171--1180.
11. Adrien Bousseau, James P O'shea, Frédo Durand, Ravi Ramamoorthi, and Maneesh Agrawala. 2013. Gloss perception in painterly and cartoon rendering. ACM Trans. Graph. 32, 2 (2013), 1--13.
12. Paul Brossier, Juan Pablo Bello, and Mark D Plumbley. 2004. Real-time temporal segmentation of note objects in music signals. In Proceedings of ICMC 2004, the 30th Annual International Computer Music Conference.
13. Alice C Chadwick and RW Kentridge. 2015. The perception of gloss: A review. Vision Research 109 (2015), 221--235.
14. Rune Haubo B Christensen. 2018. Cumulative link models for ordinal regression with the R package ordinal. Submitted in J. Stat. Software (2018).
15. Douglas W Cunningham and Christian Wallraven. 2011. Experimental design: From user studies to psychophysics. CRC Press.
16. CW Dawson, Nick J Mount, Robert J Abrahart, and John Louis. 2014. Sensitivity analysis for comparison, validation and physical legitimacy of neural network-based hydrological models. Journal of Hydroinformatics 16, 2 (2014), 407--424.
17. Johanna Delanoy, Manuel Lagunas, Ignacio Galve, Diego Gutierrez, Ana Serrano, Roland Fleming, and Belen Masia. 2020. The Role of Objective and Subjective Measures in Material Similarity Learning. In ACM SIGGRAPH Posters. Article 51, 2 pages.
18. Katja Doerschner, Laurence T. Maloney, and Huseyin Boyaci. 2010. Perceived glossiness in high dynamic range scenes. Journal of Vision 10, 9, Article 11 (2010).
19. Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016. Toward a learning science for complex crowdsourcing tasks. In Human Factors in Computing Systems. 2623--2634.
20. Ron O. Dror, Alan S. Willsky, and Edward H. Adelson. 2004. Statistical characterization of real-world illumination. Journal of Vision 4 (2004), 821--837.
21. Jonathan Dupuy and Wenzel Jakob. 2018. An adaptive parameterization for efficient material acquisition and rendering. ACM Trans. Graph. 37, 6 (2018), 1--14.
22. Frédo Durand and Julie Dorsey. 2002. Fast bilateral filtering for the display of high-dynamic-range images. In Proc. ACM SIGGRAPH. 257--266.
23. Willemijn Elkhuizen, Tessa Essers, Yu Song, Jo Geraedts, Clemens Weijkamp, Joris Dik, and Sylvia Pont. 2019. Gloss, Color, and Topography Scanning for Reproducing a Painting's Appearance Using 3D Printing. Journal on Computing and Cultural Heritage (JOCCH) 12, 4 (2019), 1--22.
24. Franz Faul. 2019. The influence of Fresnel effects on gloss perception. Journal of Vision 19, 13 (2019), 1--39.
25. Jiří Filip. 2015. Analyzing and predicting anisotropic effects of BRDFs. In Proc. ACM Symposium on Applied Perception. 25--32.
26. J. Filip and R. Vávra. 2014. Template-Based Sampling of Anisotropic BRDFs. Computer Graphics Forum 33, 7 (2014), 91--99.
27. Roland W Fleming. 2014. Visual perception of materials and their properties. Vision Research 94 (2014), 62--75.
28. Roland W Fleming. 2017. Material perception. Annual Review of Vision Science 3 (2017), 365--388.
29. Roland W. Fleming and Heinrich H. Bülthoff. 2005. Low-Level Image Cues in the Perception of Translucent Materials. ACM Trans. Appl. Percept. 2, 3 (2005), 346--382.
30. Roland W. Fleming, Ron O. Dror, and Edward H. Adelson. 2003. Real-world illumination and the perception of surface reflectance properties. Journal of Vision 3, 5 (2003), 3--3.
31. Roland W Fleming, Shin'ya Nishida, and Karl R Gegenfurtner. 2015. Perception of material properties. Vision Research 115 (2015), 157--62.
32. Roland W Fleming and Katherine R Storrs. 2019. Learning to see stuff. Current Opinion in Behavioral Sciences 30 (2019), 100--108.
33. Adria Fores, James Ferwerda, and Jinwei Gu. 2012. Toward a perceptually based metric for BRDF modeling. In Proc. Color and Imaging Conference. 142--148.
34. A. Gilchrist, Kossyfidis C., Bonato F., and et al. 1999. An anchoring theory of lightness perception. Psychol Rev. 106, 4 (1999), 795--834.
35. Ioannis Gkioulekas, Bruce Walter, Edward H Adelson, Kavita Bala, and Todd Zickler. 2015. On the appearance of translucent edges. In Computer Vision and Pattern Recognition. 5528--5536.
36. Dar'ya Guarnera, Giuseppe Claudio Guarnera, Matteo Toscani, Mashhuda Glencross, Baihua Li, Jon Yngve Hardeberg, and Karl R Gegenfurtner. 2018. Perceptually validated cross-renderer analytical BRDF parameter remapping. IEEE Transactions on Visualization and Computer Graphics 26, 6 (2018), 2258--2272.
37. Sabrina Hansmann-Roth and Pascal Mamassian. 2017. A Glossy Simultaneous Contrast: Conjoint Measurements of Gloss and Lightness. i-Perception 8, 1 (2017).
38. Vlastimil Havran, Jiri Filip, and Karol Myszkowski. 2016. Perceptually motivated BRDF comparison using single image. In Computer Graphics Forum, Vol. 35. 1--12.
39. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition. 770--778.
40. Jeffrey Heer and Michael Bostock. 2010. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In Human Factors in Computing Systems. 203--212.
41. Bingyang Hu, Jie Guo, Yanjun Chen, Mengtian Li, and Yanwen Guo. 2020. DeepBRDF: A Deep Representation for Manipulating Measured BRDF. In Computer Graphics Forum, Vol. 39. 157--166.
42. R.S. Hunter. 1937. Methods of Determining Gloss. Part of Journal of Research of the National Bureau of Standards 18 (1937), 19--39.
43. R.S. Hunter and R.W. Harold. 1987. The measurement of appearance (2nd ed.). Wiley, New York.
44. Juno Kim, Phillip J. Marlow, and Barton L. Anderson. 2012. The dark side of gloss. Nature Neuroscience 15, 11 (2012), 1590--1595.
45. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980 (2014).
46. Manuel Lagunas, Sandra Malpica, Ana Serrano, Elena Garces, Diego Gutierrez, and Belen Masia. 2019. A Similarity Measure for Material Appearance. ACM Trans. Graph. 38, 4 (2019).
47. Manuel Lagunas, Ana Serrano, Diego Gutierrez, and Belen Masia. 2021. The joint role of geometry and illumination on material recognition. Journal of Vision 21, 2 (2021).
48. Y-X. Landy, M.S. Landy, and L.T. Maloney. 2008. Conjoint measurement of gloss and surface texture. Psychological Science 19, 2 (2008), 196--204.
49. Guillaume Lavoué, Nicolas Bonneel, Jean-Philippe Farrugia, and Cyril Soler. 2021. Perceptual quality of BRDF approximations: dataset and metrics. Computer Graphics Forum 40, 2 (2021).
50. Frédéric B Leloup, Michael R Pointer, Philip Dutré, and Peter Hanselaer. 2010. Geometry of illumination, luminance contrast, and gloss perception. J. Opt. Soc. Am. A 27, 9 (2010), 2046--2054.
51. A Luongo, V Falster, MB Doest, MM Ribo, ER Eiriksson, DB Pedersen, and JR Frisvad. 2019. Microstructure Control in 3D Printing with Digital Light Processing. In Computer Graphics Forum, Vol. 39. 347--359.
52. Phillip J Marlow and Barton L Anderson. 2013. Generative constraints on image cues for perceived gloss. Journal of Vision 13, 14 (2013), 2--2.
53. Phillip J Marlow, Juno Kim, and Barton L Anderson. 2012. The perception and misperception of specular surface reflectance. Current Biology 22, 20 (2012), 1909--1913.
54. Wojciech Matusik. 2003. A data-driven reflectance model. Ph.D. Dissertation. Massachusetts Institute of Technology.
55. Wojciech Matusik, Boris Ajdin, Jinwei Gu, Jason Lawrence, Hendrik P. A. Lensch, Fabio Pellacini, and Szymon Rusinkiewicz. 2009. Printing Spatially-Varying Reflectance. ACM Trans. Graph. 28, 5 (2009), 1--9.
56. Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-Driven Reflectance Model. ACM Trans. Graph. 22, 3 (2003), 759--769.
57. Peter McCullagh. 1980. Regression models for ordinal data. Journal of the Royal Statistical Society: Series B (Methodological) 42, 2 (1980), 109--127.
58. Isamu Motoyoshi and Hiroaki Matoba. 2012. Variability in constancy of the perceived surface reflectance across different illumination statistics. Vision Research 53, 1 (2012), 30--39.
59. Isamu Motoyoshi, Shin'ya Nishida, Lavanya Sharan, and Edward H Adelson. 2007. Image statistics and the perception of surface qualities. Nature 447, 7141 (2007), 206--209.
60. Addy Ngan, Frédo Durand, and Wojciech Matusik. 2006. Image-driven Navigation of Analytical BRDF Models. Rendering Techniques (2006), 399--407.
61. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A retargetable forward and inverse renderer. ACM Trans. Graph. 38, 6 (2019), 1--17.
62. Shin'ya Nishida and Mikio Shinya. 1998. Use of image-based information in judgments of surface-reflectance properties. J. Opt. Soc. Am. A 15, 12 (1998), 2951--2965.
63. Jum C Nunnally. 1994. Psychometric theory 3E. Tata McGraw-hill education.
64. Maria Olkkonen and David H Brainard. 2010. Perceived glossiness and lightness under real-world illumination. Journal of Vision 10, 9 (2010), 5--5.
65. Fabio Pellacini, James A Ferwerda, and Donald P Greenberg. 2000. Toward a psychophysically-based light reflection model for image synthesis. In Proc. ACM SIGGRAPH. 55--64.
66. Thiago Pereira and Szymon Rusinkiewicz. 2012. Gamut mapping spatially varying reflectance with an improved BRDF similarity metric. In Computer Graphics Forum, Vol. 31. 1557--1566.
67. Michal Piovarči, Michael Wessely, Michał Jagielski, Marc Alexa, Wojciech Matusik, and Piotr Didyk. 2017. Directional screens. In Proceedings of the 1st Annual ACM Symposium on Computational Fabrication. ACM, 1.
68. Michal Piovarči, Michael Foshey, Vahid Babaei, Szymon Rusinkiewicz, Wojciech Matusik, and Piotr Didyk. 2020. Towards Spatially Varying Gloss Reproduction for 3D Printing. ACM Trans. Graph. 39, 6, Article 206 (2020).
69. Sylvia C Pont and Susan F te Pas. 2006. Material --- Illumination Ambiguities and the Perception of Solid Objects. Perception 35, 10 (2006), 1331--1350.
70. Ganesh Ramanarayanan, James Ferwerda, Bruce Walter, and Kavita Bala. 2007. Visual equivalence: towards a new standard for image fidelity. ACM Trans. Graph. 26, 3 (2007), 76--es.
71. Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. 2002. Photographic tone reproduction for digital images. In Proc. ACM SIGGRAPH. 267--276.
72. Olivier Rouiller, Bernd Bickel, Jan Kautz, Wojciech Matusik, and Marc Alexa. 2013. 3D-printing spatially varying BRDFs. IEEE Computer Graphics and Applications 33, 6 (2013), 48--57.
73. Sepideh Samadzadegan, Teun Baar, Philipp Urban, Maria V Ortiz Segovia, and Jana Blahová. 2015. Controlling colour-printed gloss by varnish-halftones. In Measuring, Modeling, and Reproducing Material Appearance 2015, Vol. 9398. International Society for Optics and Photonics, 93980V.
74. A. C. Schmid, P. Barla, and K. Doerschner. 2020. Material category determined by specular reflection structure mediates the processing of image features for perceived gloss. bioRxiv (2020).
75. Gabriel Schwartz and Ko Nishino. 2019. Recognizing material properties from images. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 8 (2019), 1981--1995.
76. Ana Serrano, Diego Gutierrez, Karol Myszkowski, Hans-Peter Seidel, and Belen Masia. 2016. An intuitive control space for material appearance. ACM Trans. Graph. 35, 6 (2016).
77. Lavanya Sharan, Yuanzhen Li, Isamu Motoyoshi, Shin'ya Nishida, and Edward H Adelson. 2008. Image statistics for surface reflectance perception. J. Opt. Soc. Am. A 25, 4 (2008), 846--865.
78. Lavanya Sharan, Ruth Rosenholtz, and Edward H Adelson. 2014. Accuracy and speed of material categorization in real-world images. Journal of Vision 14, 9 (2014), 12--12.
79. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014).
80. Katherine R Storrs and Roland W Fleming. 2020. Unsupervised Learning Predicts Human Perception and Misperception of Specular Surface Reflectance. bioRxiv (2020).
81. Tiancheng Sun, Henrik Wann Jensen, and Ravi Ramamoorthi. 2018. Connecting measured BRDFs to analytic brdfs by data-driven diffuse-specular separation. ACM Trans. Graph. 37, 6 (2018), 1--15.
82. Tiancheng Sun, Ana Serrano, Diego Gutierrez, and Belen Masia. 2017. Attribute-preserving gamut mapping of measured BRDFs. In Computer Graphics Forum, Vol. 36. 47--54.
83. Alejandro Sztrajman, Jaroslav Křivánek, Alexander Wilkie, and Tim Weyrich. 2019. Image-based Remapping of Spatially-Varying Material Appearance. Journal of Computer Graphics Techniques (JCGT) 8, 4 (2019), 1--30.
84. James T. Todd, J. Farley Norman, and Ennio Mingolla. 2004. Lightness Constancy in the Presence of Specular Highlights. Psychological Science 15, 1 (2004), 33--39.
85. Matteo Toscani, Dar'ya Guarnera, Giuseppe Claudio Guarnera, Jon Yngve Hardeberg, and Karl R Gegenfurtner. 2020. Three perceptual dimensions for specular and diffuse reflection. ACM Transactions on Applied Perception (TAP) 17, 2 (2020), 1--26.
86. Matteo Toscani and Matteo Valsecchi. 2019. Lightness Discrimination Depends More on Bright Rather Than Shaded Regions of Three-Dimensional Objects. i-Perception 10, 6 (2019), 1--10.
87. Matteo Toscani, Matteo Valsecchi, and Karl R Gegenfurtner. 2017. Lightness perception for matte and glossy complex shapes. Vision Research 131 (2017), 82--95.
88. TS Trowbridge and Karl P Reitz. 1975. Average irregularity representation of a rough surface for ray reflection. J. Opt. Soc. Am. A 65, 5 (1975), 531--536.
89. Peter Vangorp, Pascal Barla, and Roland W Fleming. 2017. The perception of hazy gloss. Journal of Vision 17, 5 (2017), 19--19.
90. Peter Vangorp and Philip Dutré. 2008. Shape-dependent gloss correction. In Proc. Applied Perception in Graphics and Visualization. 123--130.
91. Peter Vangorp, Jurgen Laurijssen, and Philip Dutré. 2007. The influence of shape on the perception of material reflectance. In Proc. ACM SIGGRAPH. 77:1--77:9.
92. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces. Rendering Techniques (2007), 195--206.
93. Gregory J. Ward. 1992. Measuring and Modeling Anisotropic Reflection. Proc. ACM SIGGRAPH 26, 2 (1992), 265--272.
94. Peter Welinder, Steve Branson, Pietro Perona, and Serge Belongie. 2010. The multidimensional wisdom of crowds. Advances in Neural Information Processing Systems 23 (2010), 2424--2432.
95. G. Wendt, F. Faul, V. Ekroll, and R. Mausfeld. 2010. Disparity, motion, and color information improve gloss constancy performance. Journal of Vision 10, 9 (2010).
96. Tim Weyrich, Pieter Peers, Wojciech Matusik, and Szymon Rusinkiewicz. 2009. Fabricating microgeometry for custom surface reflectance. ACM Trans. Graph. 28, 3 (2009), 32.
97. Christiane B Wiebel, Matteo Toscani, and Karl R Gegenfurtner. 2015. Statistical correlates of perceived gloss in natural images. Vision Research 115 (2015), 175--187.
98. Josh Wills, Sameer Agarwal, David Kriegman, and Serge Belongie. 2009. Toward a perceptual space for gloss. ACM Trans. Graph. 28, 4 (2009), 1--15.
99. Haoyu Xu, Zhenqi Han, Songlin Feng, Han Zhou, and Yuchun Fang. 2018. Foreign object debris material recognition based on convolutional neural networks. Eurasip Journal on Image and Video Processing 2018, 1 (2018), 1--10.
100. Fan Zhang, Huib de Ridder, Pascal Barla, and Sylvia Pont. 2020a. Effects of light map orientation and shape on the visual perception of canonical materials. Journal of Vision 20, 4, Article 13 (2020), 18 pages.
101. Fan Zhang, Huib de Ridder, Pascal Barla, and Sylvia Pont. 2020b. A systematic approach to testing and predicting light-material interactions. Journal of Vision 19, 4, Article 11 (2020), 22 pages.
102. Fan Zhang, Huib de Ridder, and Sylvia Pont. 2015. The influence of lighting on visual perception of material qualities. In Human Vision and Electronic Imaging, Vol. SPIE 9394. 239--248.
103. Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian Material Synthesis. ACM Trans. Graph. 37, 4, Article 76 (2018).

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision representations

        • Appearance and texture representations

  • Computer graphics

    • Graphics systems and interfaces

      • Perception