Neural complex luminaires: representation and rendering

Abstract

Complex luminaires, such as grand chandeliers, can be extremely costly to render because the light-emitting sources are typically encased in complex refractive geometry, creating difficult light paths that require many samples to evaluate with Monte Carlo approaches. Previous work has attempted to speed up this process, but the methods are either inaccurate, require the storage of very large lightfields, and/or do not fit well into modern path-tracing frameworks. Inspired by the success of deep networks, which can model complex relationships robustly and be evaluated efficiently, we propose to use a machine learning framework to compress a complex luminaire's lightfield into an implicit neural representation. Our approach can easily plug into conventional renderers, as it works with the standard techniques of path tracing and multiple importance sampling (MIS). Our solution is to train three networks to perform the essential operations for evaluating the complex luminaire at a specific point and view direction, importance sampling a point on the luminaire given a shading location, and blending to determine the transparency of luminaire queries to properly composite them with other scene elements. We perform favorably relative to state-of-the-art approaches and render final images that are close to the high-sample-count reference with only a fraction of the computation and storage costs, with no need to store the original luminaire geometry and materials.

References

1. Ian Ashdown. 1995. Near-Field Photometry: Measuring and Modeling Complex 3-D Light Sources. In ACM SIGGRAPH Course Notes. 1--15.
2. Ian Ashdown and Ron Rykowski. 1998. Making Near-Field Photometry Practical. Journal of the Illuminating Engineering Society of North America 27, 1 (1998), 67--79.
3. Steve Bako, Mark Meyer, Tony DeRose, and Pradeep Sen. 2019. Offline Deep Importance Sampling for Monte Carlo Path Tracing. Computer Graphics Forum (2019).
4. Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle. 2017. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Transactions on Graphics 36, 4 (July 2017).
5. Mojtaba Bemana, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. 2019. Neural View-Interpolation for Sparse LightField Video. arXiv preprint arXiv:1910.13921 (2019).
6. Chakravarty R. A. Chaitanya, Anton Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Noisy Monte Carlo Image Sequences using a Recurrent Autoencoder. ACM Transactions on Graphics (July 2017).
7. Chuo-Ling Chang, Xiaoging Zhu, Prashant Ramanathan, and Bernd Girod. 2003. Inter-View Wavelet Compression of Light Fields with Disparity-Compensated Lifting. In Proc. of VCIP. SPIE, 694--706.
8. Chuo-Ling Chang, Xiaoqing Zhu, Prashant Ramanathan, and Bernd Girod. 2006. Light Field Compression Using Disparity-Compensated Lifting and Shape Adaptation. IEEE Transactions on Image Processing 15, 4 (2006), 793--806.
9. Wei-Chao Chen, Jean-Yves Bouguet, Michael H. Chu, and Radek Grzeszczuk. 2002. Light Field Mapping: Efficient Representation and Hardware Rendering of Surface Light Fields. ACM Trans. Graph. 21, 3 (2002), 447--456.
10. Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. 2020. Overfit Neural Networks as a Compact Shape Representation. arXiv:2009.09808 [cs.GR]
11. SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. 2018. Neural scene representation and rendering. Science 360, 6394 (2018), 1204--1210.
12. Liangsheng Ge, Beibei Wang, Lu Wang, and Nicolas Holzschuch. 2018. A Compact Representation for Multiple Scattering in Participating Media using Neural Networks. In ACM SIGGRAPH 2018 Talks. Vancouver, Canada, 1--2.
13. Iliyan Georgiev, Jaroslav Křivánek, Tomáš Davidovič, and Philipp Slusallek. 2012. Light transport simulation with vertex connection and merging. ACM Trans. Graph. 31, 6 (2012), 192:1--192:10.
14. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The Lumigraph. In Proc. of SIGGRAPH. 43--54.
15. Jonathan Granskog, Fabrice Rousselle, Marios Papas, and Jan Novák. 2020. Compositional Neural Scene Representations for Shading Inference. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020).
16. Toshiya Hachisuka, Shinji Ogaki, and Henrik Wann Jensen. 2008. Progressive Photon Mapping. ACM Trans. Graph. 27, 5 (2008), 130:1--130:8.
17. Toshiya Hachisuka, Jacopo Pantaleoni, and Henrik Wann Jensen. 2012. A path space extension for robust light transport simulation. ACM Trans. Graph. 31, 6 (2012), 191:1--191:10.
18. Wolfgang Heidrich, Jan Kautz, Philipp Slusallek, and Hans-Peter Seidel. 1998. Canned Lightsources. In Rendering techniques' 98. 293--300.
19. Wenzel Jakob. 2010. Mitsuba Physically Based Renderer. http://www.mitsubarenderer.org
20. Wenzel Jakob and Steve Marschner. 2012. Manifold Exploration: A Markov Chain Monte Carlo Technique for Rendering Scenes with Difficult Specular Transport. ACM Transactions on Graphics (TOG) 31, 4, Article 58 (2012), 58:1--58:13 pages.
21. James T. Kajiya. 1986. The Rendering Equation. SIGGRAPH Comput. Graph. 20, 4 (Aug. 1986), 143--150.
22. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A Machine Learning Approach for Filtering Monte Carlo Noise. ACM Transactions on Graphics 34, 4, Article 122 (July 2015), 12 pages.
23. Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based View Synthesis for Light Field Cameras. ACM Transactions on Graphics 35, 6, Article 193 (Nov. 2016), 10 pages.
24. Simon Kallweit, Thomas Müller, Brian McWilliams, Markus Gross, and Jan Novák. 2017. Deep Scattering: Rendering Atmospheric Clouds with Radiance-Predicting Neural Networks. ACM Transactions on Graphics 36, 6, Article 231 (Nov. 2017), 11 pages.
25. S. Kniep, S. Häring, and M. Magnor. 2009. Efficient and Accurate Rendering of Complex Light Sources. Computer Graphics Forum 28, 4 (June 2009), 1073--1081.
26. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097--1105.
27. Alexandr Kuznetsov, Miloš Hašan, Zexiang Xu, Ling-Qi Yan, Bruce Walter, Nima Khademi Kalantari, Steve Marschner, and Ravi Ramamoorthi. 2019. Learning Generative Models for Rendering Specular Microgeometry. ACM Transactions on Graphics 38, 6, Article 225 (Nov. 2019), 14 pages.
28. Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2Noise: Learning Image Restoration without Clean Data. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80.
29. Marc Levoy and Pat Hanrahan. 1996. Light Field Rendering. Proc. of SIGGRAPH (1996).
30. Heqi Lu, Romain Pacanowski, and Xavier Granier. 2015. Position-Dependent Importance Sampling of Light Field Luminaires. IEEE Transactions on Visualization and Computer Graphics 21, 2 (feb 2015), 241--251.
31. Albert Mas, Ignacio Martín, and Gustavo Patow. 2008. Compression and Importance Sampling of Near-Field Light Sources. Computer Graphics Forum 27, 8 (Dec. 2008), 2013--2027.
32. Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. ACM Transactions on Graphics (2019).
33. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV 2020.
34. Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural importance sampling. ACM Transactions on Graphics 38, 5 (2019).
35. Julius Muschaweck. 2011. What's in a ray set: moving towards a unified ray set format, In Illumination Optics II. Proc. SPIE 8170.
36. P.Y. Ngai. 1987. On near-field photometry. Journal of the Illuminating Engineering Society 16, 2 (1987), 129--136.
37. NVIDIA. 2021. NVIDIA TensorRT. https://developer.nvidia.com/tensorrt
38. Gilles Rainer, Wenzel Jakob, Abhijeet Ghosh, and Tim Weyrich. 2019. Neural BTF Compression and Interpolation. In Computer Graphics Forum, Vol. 38. 235--244.
39. Peiran Ren, Yue Dong, Stephen Lin, Xin Tong, and Baining Guo. 2015. Image based relighting using neural networks. ACM Transactions on Graphics 34, 4 (2015), 111.
40. Peiran Ren, Jinpeng Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013. Global Illumination with Radiance Regression Functions. ACM Transactions on Graphics 32 (July 2013).
41. Todd Saemish, P. Ericson, G. Hauser, E. Gibson, R. Heinisch, C. Loch, and IESNA. 2002. ANSI/IESNA Standard File Format for the Electronic Transfer of Photometric Data and Related Information.
42. Pỳnar Satỳlmỳs, Thomas Bashford-Rogers, Alan Chalmers, and Kurt Debattista. 2017. A machine-learning-driven sky model. IEEE computer graphics and applications 37, 1 (2017), 80--91.
43. Eric Veach. 1997. Robust Monte Carlo Methods for Light Transport Simulation. Ph.D. Dissertation. Stanford University.
44. Eric Veach and Leonidas J. Guibas. 1997. Metropolis Light Transport. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., USA, 65--76.
45. Edgar Velázquez-Armendáriz, Zhao Dong, Bruce Walter, and Donald P. Greenberg. 2015. Complex Luminaires: Illumination and Appearance Rendering. ACM Transactions on Graphics 34, 3 (May 2015), 26:1--26:15.
46. Delio Vicini, Vladlen Koltun, and Wenzel Jakob. 2019. A Learned Shape-Adaptive Subsurface Scattering Model. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 38, 4 (July 2019), 126:1--126:15.
47. Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Transactions on Graphics 37, 4, Article 124 (2018), 124:1--124:15 pages.
48. Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, and Ravi Ramamoorthi. 2017. Light Field Video Capture Using a Learning-Based Hybrid Imaging System. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 36, 4 (2017).
49. Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. 2018. Deep image-based relighting from optimal sparse samples. ACM Transactions on Graphics 37, 4 (2018), 126.
50. Quan Zheng and Matthias Zwicker. 2019. Learning to importance sample in primary sample space. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 169--179.
51. Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian material synthesis. ACM Transactions on Graphics 37, 4 (2018), 76:1--76:14.

Index Terms

• Computing methodologies

  • Computer graphics

    • Rendering