Gaussian material synthesis

Abstract

We present a learning-based system for rapid mass-scale material synthesis that is useful for novice and expert users alike. The user preferences are learned via Gaussian Process Regression and can be easily sampled for new recommendations. Typically, each recommendation takes 40-60 seconds to render with global illumination, which makes this process impracticable for real-world workflows. Our neural network eliminates this bottleneck by providing high-quality image predictions in real time, after which it is possible to pick the desired materials from a gallery and assign them to a scene in an intuitive manner. Workflow timings against Disney's "principled" shader reveal that our system scales well with the number of sought materials, thus empowering even novice users to generate hundreds of high-quality material models without any expertise in material modeling. Similarly, expert users experience a significant decrease in the total modeling time when populating a scene with materials. Furthermore, our proposed solution also offers controllable recommendations and a novel latent space variant generation step to enable the real-time fine-tuning of materials without requiring any domain expertise.

References

1. Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance modeling by neural texture synthesis. ACM Transactions on Graphics 35, 4 (2016), 65.
2. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2013. Practical SVBRDF capture in the frequency domain. ACM Transactions on Graphics 32, 4 (2013), 110--1.
3. Miika Aittala, Tim Weyrich, Jaakko Lehtinen, et al. 2015. Two-shot SVBRDF capture for stationary materials. ACM Transactions on Graphics 34, 4 (2015), 110--1.
4. Ken Anjyo and JP Lewis. 2011. RBF interpolation and Gaussian process regression through an RKHS formulation. Journal of Math-for-Industry 3, 6 (2011), 63--71.
5. Melinos Averkiou, Vladimir G Kim, Youyi Zheng, and Niloy J Mitra. 2014. Shapesynth: Parameterizing model collections for coupled shape exploration and synthesis. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 125--134.
6. Steve Bako, Thijs Vogels, Brian Mcwilliams, Mark Meyer, Jan NováK, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Transactions on Graphics 36, 4 (2017), 97.
7. Mikhail Belkin and Partha Niyogi. 2003. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation 15, 6 (2003), 1373--1396.
8. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2013. OpenSurfaces: A richly annotated catalog of surface appearance. ACM Transactions on Graphics 32, 4 (2013), 111.
9. Aner Ben-Artzi, Ryan Overbeck, and Ravi Ramamoorthi. 2006. Real-time BRDF editing in complex lighting. In ACM Transactions on Graphics, Vol. 25. ACM, 945--954.
10. Manuel Blum and Martin A Riedmiller. 2013. Optimization of Gaussian process hyper-parameters using Rprop.. In ESANN. 339--344.
11. Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. 2017. Optimizing the Latent Space of Generative Networks. arXiv preprint arXiv:1707.05776 (2017).
12. Brent Burley and Walt Disney Animation Studios. 2012. Physically-based Shading at Disney. In ACM SIGGRAPH, Vol. 2012. 1--7.
13. Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. 1995. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing 16, 5 (1995), 1190--1208.
14. Neill DF Campbell and Jan Kautz. 2014. Learning a manifold of fonts. ACM Transactions on Graphics 33, 4 (2014), 91.
15. Kang Chen, Kun Xu, Yizhou Yu, Tian-Yi Wang, and Shi-Min Hu. 2015. Magic decorator: automatic material suggestion for indoor digital scenes. ACM Transactions on Graphics 34, 6 (2015), 232.
16. Ewen Cheslack-Postava, Rui Wang, Oskar Akerlund, and Fabio Pellacini. 2008. Fast, realistic lighting and material design using nonlinear cut approximation. In ACM Transactions on Graphics, Vol. 27. ACM, 128.
17. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015).
18. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. 249--256.
19. He He and Wan-Chi Siu. 2011. Single image super-resolution using Gaussian process regression. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 449--456.
20. Geoffrey E Hinton and Ruslan R Salakhutdinov 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504--507.
21. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A machine learning approach for filtering Monte Carlo noise. ACM Transactions on Graphics 34, 4 (2015), 122--1.
22. Simon Kallweit, Thomas Müller, Brian McWilliams, Markus Gross, and Jan Novák. 2017. Deep Scattering: Rendering Atmospheric Clouds with Radiance-Predicting Neural Networks. ACM Transactions on Graphics (Proc. of Siggraph Asia) 36, 6, Article 231 (Nov. 2017), 11 pages.
23. Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. 2007. Active learning with gaussian processes for object categorization. In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. IEEE, 1--8.
24. Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1725--1732.
25. William B Kerr and Fabio Pellacini. 2010. Toward evaluating material design interface paradigms for novice users. In ACM Transactions on Graphics, Vol. 29. ACM, 35.
26. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv.1412.6980 (2014).
27. Yuki Koyama, Daisuke Sakamoto, and Takeo Igarashi. 2014. Crowd-powered parameter analysis for visual design exploration. In Proceedings of the 27th annual ACM symposium on User Interface Software and Technology. ACM, 65--74.
28. Eric PF Lafortune, Sing-Choong Foo, Kenneth E Torrance, and Donald P Greenberg. 1997. Non-linear approximation of reflectance functions. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques. ACM Press/Addison-Wesley Publishing Co., 117--126.
29. Neil D Lawrence. 2004. Gaussian process latent variable models for visualisation of high dimensional data. In Advances in Neural Information Processing Systems. 329--336.
30. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278--2324.
31. Joel Lehman and Kenneth O Stanley. 2008. Exploiting Open-Endedness to Solve Problems Through the Search for Novelty.. In ALIFE. 329--336.
32. Joel Lehman and Kenneth O Stanley. 2011. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation. ACM, 211--218.
33. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, Nov (2008), 2579--2605.
34. David JC MacKay. 1996. Bayesian methods for backpropagation networks. In Models of Neural Networks III. Springer, 211--254.
35. Joe Marks, Brad Andalman, Paul A Beardsley, William Freeman, Sarah Gibson, Jessica Hodgins, Thomas Kang, Brian Mirtich, Hanspeter Pfister, Wheeler Ruml, et al. 1997. Design galleries: A general approach to setting parameters for computer graphics and animation. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 389--400.
36. Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber. 2011. Stacked convolutional auto-encoders for hierarchical feature extraction. Artificial Neural Networks and Machine Learning-ICANN 2011 (2011), 52--59.
37. Wojciech Matusik. 2003. A data-driven reflectance model. Ph.D. Dissertation. Massachusetts Institute of Technology.
38. Leo Miyashita, Kota Ishihara, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2016. Zoe-Matrope: A system for physical material design. In ACM SIGGRAPH 2016 Emerging Technologies. ACM, 24.
39. Martin Fodslette Møller. 1993. A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks 6, 4 (1993), 525--533.
40. Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, H-P Seidel, and Tobias Ritschel. 2017. Deep Shading: Convolutional Neural Networks for Screen Space Shading. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 65--78.
41. Radford M Neal. 2012. Bayesian learning for neural networks. Vol. 118. Springer Science & Business Media.
42. Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. 2015. Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision. 1520--1528.
43. Steven J Nowlan and Geoffrey E Hinton. 1992. Simplifying neural networks by soft weight-sharing. Neural Computation 4, 4 (1992), 473--493.
44. Augustus Odena, Vincent Dumoulin, and Chris Olah. 2016. Deconvolution and Checkerboard Artifacts. Distill (2016).
45. Peiran Ren, Jiaping Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013. Global illumination with radiance regression functions. ACM Transactions on Graphics 32, 4 (2013), 130.
46. Martin Riedmiller and Heinrich Braun. 1992. RPROP-A fast adaptive learning algorithm. In Proc. of ISCIS VII), Universitat. Citeseer.
47. Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, and Hao Li. 2016. Photorealistic Facial Texture Inference Using Deep Neural Networks. arXiv preprint arXiv:1612.00523 (2016).
48. Pỳnar Satỳlmỳs, Thomas Bashford-Rogers, Alan Chalmers, and Kurt Debattista. 2017. A Machine-Learning-Driven Sky Model. IEEE Computer Graphics and Applications 37, 1 (2017), 80--91.
49. Pradeep Sen and Soheil Darabi. 2012. On filtering the noise from the random parameters in Monte Carlo rendering. ACM Transactions on Graphics 31, 3 (2012), 18--1.
50. Ana Serrano, Diego Gutierrez, Karol Myszkowski, Hans-Peter Seidel, and Belen Masia. 2016. An intuitive control space for material appearance. ACM Transactions on Graphics 35, 6 (2016), 186.
51. Maria Shugrina, Jingwan Lu, and Stephen Diverdi. 2017. Playful palette: an interactive parametric color mixer for artists. ACM Transactions on Graphics 36, 4 (2017), 61.
52. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15, 1 (2014), 1929--1958.
53. Xin Sun, Kun Zhou, Yanyun Chen, Stephen Lin, Jiaoying Shi, and Baining Guo. 2007. Interactive relighting with dynamic BRDFs. ACM Transactions on Graphics 26, 3 (2007), 27.
54. Joshua B Tenenbaum, Vin De Silva, and John C Langford. 2000. A global geometric framework for nonlinear dimensionality reduction. Science 290, 5500 (2000), 2319--2323.
55. Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research 11, Dec (2010), 3371--3408.
56. Jack M Wang, David J Fleet, and Aaron Hertzmann. 2008. Gaussian process dynamical models for human motion. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (2008), 283--298.
57. Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. 2010. Deconvolutional networks. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2528--2535.
58. Hui Zou and Trevor Hastie. 2005. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67, 2 (2005), 301--320.

Index Terms

• Computing methodologies

  • Computer graphics

    • Rendering

      • Ray tracing

  • Machine learning

    • Machine learning approaches

      • Kernel methods

        • Gaussian processes

      • Neural networks