A similarity measure for material appearance

Abstract

We present a model to measure the similarity in appearance between different materials, which correlates with human similarity judgments. We first create a database of 9,000 rendered images depicting objects with varying materials, shape and illumination. We then gather data on perceived similarity from crowdsourced experiments; our analysis of over 114,840 answers suggests that indeed a shared perception of appearance similarity exists. We feed this data to a deep learning architecture with a novel loss function, which learns a feature space for materials that correlates with such perceived appearance similarity. Our evaluation shows that our model outperforms existing metrics. Last, we demonstrate several applications enabled by our metric, including appearance-based search for material suggestions, database visualization, clustering and summarization, and gamut mapping.

References

1. Edward H Adelson. 2001. On seeing stuff: the perception of materials by humans and machines. In Human vision and electronic imaging VI, Vol. 4299. International Society for Optics and Photonics, 1--13.
2. Sameer Agarwal, Josh Wills, Lawrence Cayton, Gert Lanckriet, David Kriegman, and Serge Belongie. 2007. Generalized non-metric multidimensional scaling. In Conference on artifical intelligence and statistics. 11--18.
3. Barton L Anderson. 2011. Visual perception of materials and surfaces. Current biology 21, 24 (2011), R978--R983.
4. Amit Banerjee and Rajesh N Dave. 2004. Validating clusters using the Hopkins statistic. In 2004 IEEE International conference on fuzzy systems (IEEE Cat. No. 04CH37542), Vol. 1. IEEE, 149--153.
5. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2013. Opensurfaces: A richly annotated catalog of surface appearance. ACM Transactions on graphics (TOG) 32, 4 (2013), 111.
6. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material recognition in the wild with the materials in context database. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 3479--3487.
7. A. C. Chadwick and R. W. Kentridge. 2015. The perception of gloss: A review. Vision research 109 (2015), 221 -- 235.
8. Kang Chen, Kun Xu, Yizhou Yu, Tian-Yi Wang, and Shi-Min Hu. 2015. Magic decorator: automatic material suggestion for indoor digital scenes. ACM Transactions on graphics (TOG) 34, 6 (2015), 232.
9. De Cheng, Yihong Gong, Sanping Zhou, Jinjun Wang, and Nanning Zheng. 2016. Person re-identification by multi-channel parts-based cnn with improved triplet loss function. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 1335--1344.
10. D. W. Cunningham, C. Wallraven, R. W. Fleming, and W. Strasser. 2007. Perceptual Reparameterization of Material Properties. In Proceedings of the third Eurographics conference on computational aesthetics in graphics, visualization and imaging (Computational Aesthetics'07). 89--96.
11. Kristin J. Dana, Bram van Ginneken, Shree K. Nayar, and Jan J. Koenderink. 1999. Reflectance and Texture of Real-world Surfaces. ACM Transactions on graphics (TOG) 18, 1 (Jan. 1999), 1--34.
12. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). Ieee, 248--255.
13. Katja Doerschner, Roland W. Fleming, Ozgur Yilmaz, Paul R. Schrater, Bruce Hartung, and Daniel Kersten. 2011. Visual Motion and the Perception of Surface Material. Current biology 21, 23 (2011), 2010 -- 2016.
14. Julie Dorsey, Holly Rushmeier, and François Sillion. 2010. Digital modeling of material appearance. Elsevier.
15. Jonathan Dupuy and Wenzel Jakob. 2018. An Adaptive Parameterization for Efficient Material Acquisition and Rendering. ACM Transactions on graphics (TOG).
16. Jirí Filip and Radomír Vávra. 2014. Template-based sampling of anisotropic BRDFs. In Computer graphics forum, Vol. 33. Wiley Online Library, 91--99.
17. Roland W. Fleming. 2014. Visual perception of materials and their properties. Vision research 94 (2014), 62 -- 75.
18. Roland W Fleming. 2017. Material perception. Annual review of vision science 3 (2017), 365--388.
19. Roland W Fleming, Ron O Dror, and Edward H Adelson. 2003. Real-world illumination and the perception of surface reflectance properties. Journal of vision 3, 5 (2003), 3--3.
20. Roland W. Fleming, Shin'ya Nishida, and Karl R. Gegenfurtner. 2015. Perception of material properties. Vision research 115 (2015), 157 -- 162.
21. Adria Fores, James Ferwerda, and Jinwei Gu. 2012. Toward a perceptually based metric for BRDF modeling. In Color and imaging conference, Vol. 2012. Society for Imaging Science and Technology, 142--148.
22. Jiyang Gao and Ram Nevatia. 2018. Revisiting Temporal Modeling for Video-based Person ReID. arXiv preprint arXiv:1805.02104 (2018).
23. Elena Garces, Aseem Agarwala, Diego Gutierrez, and Aaron Hertzmann. 2014. A Similarity Measure for Illustration Style. ACM transactions on graphics (TOG, proc. SIGGRAPH) 33, 4 (2014).
24. Stamatios Georgoulis, Vincent Vanweddingen, Marc Proesmans, and Luc Van Gool. 2017. Material Classification under Natural Illumination Using Reflectance Maps. In IEEE Winter conference on applications of computer vision (WACV). IEEE, 244--253.
25. Ioannis Gkioulekas, Bruce Walter, Edward H Adelson, Kavita Bala, and Todd Zickler. 2015. On the appearance of translucent edges. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 5528--5536.
26. Ioannis Gkioulekas, Bei Xiao, Shuang Zhao, Edward H Adelson, Todd Zickler, and Kavita Bala. 2013. Understanding the role of phase function in translucent appearance. ACM Transactions on graphics (TOG) 32, 5 (2013), 147.
27. Vlastimil Havran, Jiri Filip, and Karol Myszkowski. 2016. Perceptually Motivated BRDF Comparison using Single Image. Computer graphics forum (2016).
28. Eric Hayman, Barbara Caputo, Mario Fritz, and Jan-Olof Eklundh. 2004. On the significance of real-world conditions for material classification. In European conference on computer vision (ECCV). Springer, 253--266.
29. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 770--778.
30. Yun-Xian Ho, Michael S Landy, and Laurence T Maloney. 2006. How direction of illumination affects visually perceived surface roughness. Journal of vision 6, 5 (2006), 8--8.
31. Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely Connected Convolutional Networks.. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Vol. 1. 3.
32. Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.
33. Adrian Jarabo, Hongzhi Wu, Julie Dorsey, Holly Rushmeier, and Diego Gutierrez. 2014. Effects of Approximate Filtering on the Appearance of Bidirectional Texture Functions. IEEE Transactions on visualization and computer graphics 20, 6 (2014).
34. Maurice Kendall and Jean D. Gibbons. 1990. Rank Correlation Methods (5 ed.). A Charles Griffin Title.
35. M G Kendall and B Babington-Smith. 1940. On the Method of Paired Comparisons. Biometrica 31 (1940), 324--345.
36. Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, and Jan Kautz. 2017. A lightweight approach for on-the-fly reflectance estimation. In Proceedings of the international conference on computer vision (ICCV). 20--28.
37. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
38. Jaroslav Křivánek, James A. Ferwerda, and Kavita Bala. 2010. Effects of global illumination approximations on material appearance. ACM transactions on graphics (TOG, proc. SIGGRAPH) 29, 4, Article 112 (July 2010), 112:1--112:10 pages.
39. Manuel Lagunas, Elena Garces, and Diego Gutierrez. 2018. Learning icons appearance similarity. Multimedia tools and applications (2018), 1--19.
40. Thomas Leung and Jitendra Malik. 2001. Representing and recognizing the visual appearance of materials using three-dimensional textons. International journal of computer vision 43, 1 (2001), 29--44.
41. Stephen Lombardi and Ko Nishino. 2012. Reflectance and natural illumination from a single image. In European conference on computer vision (ECCV). Springer, 582--595.
42. Feng Lu, Xiaowu Chen, Imari Sato, and Yoichi Sato. 2018. SymPS: BRDF symmetry guided photometric stereo for shape and light source estimation. IEEE transactions on pattern analysis and machine intelligence 40, 1 (2018), 221--234.
43. Zhaoliang Lun, Evangelos Kalogerakis, and Alla Sheffer. 2015. Elements of style: learning perceptual shape style similarity. ACM Transactions on graphics (TOG) 34, 4 (2015), 84.
44. Laurence T Maloney and David H Brainard. 2010. Color and material perception: Achievements and challenges. Journal of vision 10, 9 (2010), 19--19.
45. Rafał Mantiuk, Scott Daly, and Louis Kerofsky. 2008. Display adaptive tone mapping. In ACM Transactions on graphics (TOG), Vol. 27. ACM, 68.
46. Wojciech Matusik, Boris Ajdin, Jinwei Gu, Jason Lawrence, Hendrik P. A. Lensch, Fabio Pellacini, and Szymon Rusinkiewicz. 2009. Printing Spatially-Varying Reflectance. ACM transactions on graphics (TOG, proc. SIGGRAPH Asia) 28, 5 (Dec. 2009).
47. Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-Driven Reflectance Model. ACM Transactions on graphics (TOG) 22, 3 (July 2003), 759--769.
48. Brian McFee and Gert Lanckriet. 2011. Learning Multi-modal Similarity. Journal of machine learning research 12 (2011), 491--523.
49. Leland McInnes and John Healy. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018).
50. M Mylo, M Giesel, Q Zaidi, M Hullin, and R Klein. 2017. Appearance bending: A perceptual editing paradigm for data-driven material models. Vision, modeling and visualization. The Eurographics association (2017).
51. Addy Ngan, Frédo Durand, and Wojciech Matusik. 2005. Experimental Analysis of BRDF Models. In Eurographics Symposium on Rendering (2005). The Eurographics association.
52. Addy Ngan, Frédo Durand, and Wojciech Matusik. 2006. Image-driven Navigation of Analytical BRDF Models.. In Rendering Techniques. 399--407.
53. Fabio Pellacini, James A. Ferwerda, and Donald P. Greenberg. 2000. Toward a Psychophysically-based Light Reflection Model for Image Synthesis. In Proceedings of the 27th annual conference on computer graphics and interactive techniques (SIGGRAPH '00). 55--64.
54. Thiago Pereira and Szymon Rusinkiewicz. 2012. Gamut mapping spatially varying reflectance with an improved BRDF similarity metric. In Computer graphics forum, Vol. 31. Wiley Online Library, 1557--1566.
55. Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. 2019. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237 (2019).
56. Michael Rubinstein, Diego Gutierrez, Olga Sorkine, and Ariel Shamir. 2010. A Comparative Study of Image Retargeting. ACM transactions on graphics (TOG, proc. SIGGRAPH Asia) 29, 6 (2010), 160:1--160:10.
57. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 3 (2015), 211--252.
58. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 815--823.
59. M. Schultz and T. Joachims. 2003. Learning a Distance Metric from Relative Comparisons. In Advances in neural information processing systems.
60. Gabriel Schwartz and Ko Nishino. 2016. Material Recognition from Local Appearance in Global Context. arXiv preprint arXiv:1611.09394 (2016).
61. Gabriel Schwartz and Ko Nishino. 2018. Recognizing Material Properties from Images. arXiv preprint arXiv:1801.03127 (2018).
62. Ana Serrano, Diego Gutierrez, Karol Myszkowski, Hans-Peter Seidel, and Belen Masia. 2016. An Intuitive Control Space for Material Appearance. ACM Transactions on graphics (TOG) 35, 6, Article 186 (Nov. 2016), 186:1--186:12 pages.
63. Lavanya Sharan, Ruth Rosenholtz, and Edward Adelson. 2009. Material perception: What can you see in a brief glance? Journal of vision 9, 8 (2009), 784--784.
64. Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014. CNN features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) workshops. 806--813.
65. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
66. Cyril Soler, Kartic Subr, and Derek Nowrouzezahrai. 2018. A Versatile Parameterization for Measured Material Manifolds. In Computer graphics forum, Vol. 37. Wiley Online Library, 135--144.
67. Tiancheng Sun, Henrik Wann Jensen, and Ravi Ramamoorthi. 2018. Connecting Measured BRDFs to Analytic BRDFs by Data-driven Diffuse-specular Separation. In ACM transactions on graphics (TOG, proc. SIGGRAPH Asia). Article 273, 273:1--273:15 pages.
68. Tiancheng Sun, Ana Serrano, Diego Gutierrez, and Belen Masia. 2017. Attribute-preserving Gamut Mapping of Measured BRDFs. Computer graphics forum 36, 4 (July 2017).
69. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2818--2826.
70. Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir, and Adam Tauman Kalai. 2011. Adaptively Learning the Crowd Kernel. In Proceedings of the 28th international conference on machine learning (ICML'11). 673--680.
71. William Thompson, Roland Fleming, Sarah Creem-Regehr, and Jeanine Kelly Stefanucci. 2011. Visual Perception from a Computer Graphics Perspective (1st ed.). A. K. Peters, Ltd., Natick, MA, USA.
72. Jan Jaap R Van Assen, Pascal Barla, and Roland W Fleming. 2018. Visual features in the perception of liquids. Current biology 28, 3 (2018), 452--458.
73. Laurens Van Der Maaten and Kilian Weinberger. 2012. Stochastic triplet embedding. In Machine learning for signal processing (MLSP), 2012 IEEE International Workshop on. IEEE, 1--6.
74. Peter Vangorp, Pascal Barla, and Roland W Fleming. 2017. The perception of hazy gloss. Journal of vision 17, 5 (2017), 19--19.
75. Peter Vangorp, Jurgen Laurijssen, and Philip Dutré. 2007. The Influence of Shape on the Perception of Material Reflectance. ACM Transactions on graphics (TOG) 26, 3, Article 77 (July 2007).
76. Raquel Vidaurre, Dan Casas, Elena Garces, and Jorge Lopez-Moreno. 2019. BRDF Estimation of Complex Materials with Nested Learning. In IEEE Winter conference on applications of computer vision (WACV).
77. Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600--612.
78. Peter Welinder, Steve Branson, Pietro Perona, and Serge J Belongie. 2010. The multidimensional wisdom of crowds. In Advances in neural information processing systems. 2424--2432.
79. Josh Wills, Sameer Agarwal, David Kriegman, and Serge Belongie. 2009. Toward a Perceptual Space for Gloss. ACM Transactions on graphics (TOG) 28, 4, Article 103 (Sept. 2009), 103:1--103:15 pages.
80. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks?. In Advances in neural information processing systems. 3320--3328.
81. Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). IEEE Computer Society, 586--595.
82. Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian Material Synthesis. ACM Transactions on graphics (TOG) 37, 4, Article 76 (July 2018), 76:1--76:14 pages.

Index Terms

• Computer systems organization

  • Architectures

    • Other architectures

      • Neural networks

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision representations

        • Appearance and texture representations

  • Computer graphics

    • Graphics systems and interfaces

      • Perception