Real-time neural radiance caching for path tracing

Abstract

We present a real-time neural radiance caching method for path-traced global illumination. Our system is designed to handle fully dynamic scenes, and makes no assumptions about the lighting, geometry, and materials. The data-driven nature of our approach sidesteps many difficulties of caching algorithms, such as locating, interpolating, and updating cache points. Since pretraining neural networks to handle novel, dynamic scenes is a formidable generalization challenge, we do away with pretraining and instead achieve generalization via adaptation, i.e. we opt for training the radiance cache while rendering. We employ self-training to provide low-noise training targets and simulate infinite-bounce transport by merely iterating few-bounce training updates. The updates and cache queries incur a mild overhead---about 2.6ms on full HD resolution---thanks to a streaming implementation of the neural network that fully exploits modern hardware. We demonstrate significant noise reduction at the cost of little induced bias, and report state-of-the-art, real-time performance on a number of challenging scenarios.

References

1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. http://tensorflow.org/
2. Michael Abrash. 1997. Quake's lighting model: Surface caching. In Graphics Programming Black Book. Coriolis Group, Chapter 68, 1245--1256.
3. Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D. Fairchild. 2020. FLIP: A Difference Evaluator for Alternating Images. Proc. ACM Comput. Graph. Interact. Tech. 3, 2, Article 15 (Aug. 2020), 23 pages.
4. Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. 2020. Second Order Optimization Made Practical. arXiv:2002.09018 (Feb. 2020).
5. James Arvo. 1986. Backward Ray Tracing. In In ACM SIGGRAPH '86 Course Notes - Developments in Ray Tracing. 259--263.
6. Colin Barré-Brisebois. 2017. A Certain Slant of Light: Past, Present and Future Challenges of Global Illumination in Games. In Open problems in real-time rendering. ACM SIGGRAPH 2017 Courses.
7. Philippe Bekaert, Philipp Slusallek, Ronald Cools, Vlastimil Havran, and Hans-Peter Seidel. 2003. A custom designed Density Estimation Method for Light Transport. Technical Report. Max-Planck-Institut für Informatik, Saarbrücken, Germany.
8. Laurent Belcour, Cyril Soler, Kartic Subr, Nicolas Holzschuch, and Fredo Durand. 2013. 5D Covariance Tracing for Efficient Defocus and Motion Blur. ACM Trans. Graph. 32, 3, Article Article 31 (July 2013), 18 pages.
9. Nir Benty, Kai-Hwa Yao, Petrik Clarberg, Lucy Chen, Simon Kallweit, Tim Foley, Matthew Oakes, Conor Lavelle, and Chris Wyman. 2020. The Falcor Rendering Framework. https://github.com/NVIDIAGameWorks/Falcor https://github.com/NVIDIAGameWorks/Falcor.
10. Nikolaus Binder, Sascha Fricke, and Alexander Keller. 2018. Fast Path Space Filtering by Jittered Spatial Hashing. In ACM SIGGRAPH 2018 Talks (SIGGRAPH '18). Association for Computing Machinery, New York, NY, USA, Article 71, 2 pages.
11. Benedikt Bitterli, Chris Wyman, Matt Pharr, Peter Shirley, Aaron Lefohn, and Wojciech Jarosz. 2020. Spatiotemporal reservoir resampling for real-time ray tracing with dynamic direct lighting. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020). https://doi.org/10/gg8xc7
12. Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2018. The Design and Evolution of Disney's Hyperion Renderer. ACM Trans. Graph. 37, 3, Article 33 (July 2018), 22 pages.
13. Cyril Crassin, Fabrice Neyret, Miguel Sainz, Simon Green, and Elmar Eisemann. 2011. Interactive Indirect Illumination Using Voxel Cone Tracing. Computer Graphics Forum 30, 7 (2011), 1921--1930.
14. Carsten Dachsbacher and Marc Stamminger. 2005. Reflective Shadow Maps. In Proceedings of the 2005 Symposium on Interactive 3D Graphics and Games (I3D '05). Association for Computing Machinery, New York, NY, USA, 203--231.
15. Ken Dahm and Alexander Keller. 2018. Learning Light Transport the Reinforced Way. In Monte Carlo and Quasi-Monte Carlo Methods, Art B. Owen and Peter W. Glynn (Eds.). Springer International Publishing, 181--195.
16. William J. Dally, Yatish Turakhia, and Song Han. 2020. Domain-Specific Hardware Accelerators. Commun. ACM 63, 7 (June 2020), 48--57.
17. Johannes Deligiannis and Jan Schmid. 2019. "It Just Works" Ray-Traced Reflections in 'Battlefield V'. In Game Developers Conference.
18. Addis Dittebrandt, Johannes Hanika, and Carsten Dachsbacher. 2020. Temporal Sample Reuse for Next Event Estimation and Path Guiding for Real-Time Path Tracing. In Eurographics Symposium on Rendering - DL-only Track, Carsten Dachsbacher and Matt Pharr (Eds.). The Eurographics Association.
19. Renaud Adrien Dubouchet, Laurent Belcour, and Derek Nowrouzezahrai. 2017. Frequency Based Radiance Cache for Rendering Animations. In Proceedings of the Eurographics Symposium on Rendering: Experimental Ideas & Implementations (EGSR '17). Eurographics Association, Goslar, DEU, 41--53.
20. Boris Ginsburg, Patrice Castonguay, Oleksii Hrinchuk, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Huyen Nguyen, and Jonathan M. Cohen. 2019. Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks. arXiv:1905.11286 (June 2019).
21. Gene Greger, Peter Shirley, Philip M. Hubbard, and Donald P. Greenberg. 1998. The irradiance volume. IEEE Computer Graphics and Applications 18, 2 (1998), 32--43.
22. Vineet Gupta, Tomer Koren, and Yoram Singer. 2018. Shampoo: Preconditioned Stochastic Tensor Optimization. arXiv:1802.09568 (Aug. 2018).
23. Jon Hasselgren, Jacob Munkberg, Marco Salvi, Anjul Patney, and Aaron Lefohn. 2020. Neural Temporal Adaptive Sampling and Denoising. Computer Graphics Forum 39, 2 (2020), 147--155.
24. Paul S. Heckbert. 1990. Adaptive Radiosity Textures for Bidirectional Ray Tracing. SIGGRAPH Comput. Graph. 24, 4 (Sept. 1990), 145--154.
25. Sebastian Herholz, Oskar Elek, Jens Schindel, Jaroslav Křivánek, and Hendrik P. A. Lensch. 2018. A Unified Manifold Framework for Efficient BRDF Sampling based on Parametric Mixture Models. In Eurographics Symposium on Rendering - Experimental Ideas & Implementations, Wenzel Jakob and Toshiya Hachisuka (Eds.). The Eurographics Association.
26. Sebastian Herholz, Oskar Elek, Jiří Vorba, Hendrik Lensch, and Jaroslav Křivánek. 2016. Product Importance Sampling for Light Transport Path Guiding. Computer Graphics Forum (2016).
27. Pedro Hermosilla, Sebastian Maisch, Tobias Ritschel, and Timo Ropinski. 2019. Deep-learning the Latent Space of Light Transport. Computer Graphics Forum 38, 4 (2019).
28. John T. Hooker. 2016. Volumetric Global Illumination at Treyarch. In Advances in real-time rendering, part I. ACM SIGGRAPH 2016 Courses.
29. Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. 2020. Meta-Learning in Neural Networks: A Survey. arXiv:2004.05439 (April 2020).
30. Michał Iwanicki and Peter-Pike Sloan. 2017. Precomputed Lighting in Call of Duty: Infinite Warfare. In Advances in Real-Time Rendering, Part I (ACM SIGGRAPH 2017 Courses). Association for Computing Machinery, New York, NY, USA, Article 7a, 1 pages.
31. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging Weights Leads to Wider Optima and Better Generalization. arXiv:1803.05407 (March 2018).
32. Wojciech Jarosz, Craig Donner, Matthias Zwicker, and Henrik Wann Jensen. 2008. Radiance Caching for Participating Media. ACM Trans. Graph. 27, 1, Article 7 (March 2008), 11 pages.
33. Henrik Wann Jensen. 1995. Importance Driven Path Tracing using the Photon Map. In Rendering Techniques. Springer Vienna, Vienna, 326--335.
34. Henrik Wann Jensen. 1996. Global Illumination using Photon Maps. In Rendering Techniques '96, Xavier Pueyo and Peter Schröder (Eds.). Springer Vienna, Vienna, 21--30.
35. Giulio Jiang and Bernhard Kainz. 2021. Deep radiance caching: Convolutional autoencoders deeper in ray tracing. Computers & Graphics 94 (2021), 22 -- 31.
36. James T. Kajiya. 1986. The Rendering Equation. Computer Graphics 20 (1986), 143--150.
37. Simon Kallweit, Thomas Müller, Brian McWilliams, Markus Gross, and Jan Novák. 2017. Deep Scattering: Rendering Atmospheric Clouds with Radiance-Predicting Neural Networks. ACM Trans. Graph. 36, 6, Article 231 (Nov. 2017), 11 pages.
38. Anton Kaplanyan and Carsten Dachsbacher. 2010. Cascaded Light Propagation Volumes for Real-Time Indirect Illumination. In Proceedings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D '10). Association for Computing Machinery, New York, NY, USA, 99--107.
39. Alexander Keller. 1997. Instant Radiosity. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'97). ACM Press/Addison-Wesley Publishing Co., USA, 49--56.
40. Alexander Keller and Ken Dahm. 2019. Integral Equations and Machine Learning. Mathematics and Computers in Simulation 161 (2019), 2--12.
41. Alexander Keller, Timo Viitanen, Colin Barré-Brisebois, Christoph Schied, and Morgan McGuire. 2019. Are We Done with Ray Tracing?. In ACM SIGGRAPH 2019 Courses (SIGGRAPH'19). Association for Computing Machinery, New York, NY, USA, Article 3, 381 pages.
42. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 (June 2014).
43. Jaroslav Křivánek and Pascal Gautron. 2009. Practical Global Illumination with Irradiance Caching. Morgan & Claypool.
44. Jaroslav Křivánek, Pascal Gautron, Sumanta Pattanaik, and Kadi Bouatouch. 2005. Radiance caching for efficient global illumination computation. IEEE Transactions on Visualization and Computer Graphics 11, 5 (2005), 550--561.
45. Eric P. Lafortune and Yves D. Willems. 1995. A 5D Tree to Reduce the Variance of Monte Carlo Ray Tracing. In Proc. EGWR. 11--20.
46. Sébastien Lagarde and Charles de Rousiers. 2014. Moving Frostbite to Physically Based Rendering 3.0. In ACM SIGGRAPH Courses: Physically Based Shading in Theory and Practice, Chapter 10 (SIGGRAPH '14). ACM, New York, NY, USA.
47. Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2Noise: Learning Image Restoration without Clean Data. arXiv:1803.04189 (March 2018).
48. Zander Majercik, Jean-Philippe Guertin, Derek Nowrouzezahrai, and Morgan McGuire. 2019. Dynamic Diffuse Global Illumination with Ray-Traced Irradiance Fields. Journal of Computer Graphics Techniques (JCGT) 8, 2 (5 June 2019), 1--30. http://jcgt.org/published/0008/02/01/
49. Julio Marco, Adrian Jarabo, Wojciech Jarosz, and Diego Gutierrez. 2018. Second-Order Occlusion-Aware Volumetric Radiance Caching. ACM Trans. Graph. 37, 2, Article 20 (July 2018), 14 pages.
50. Sam Martin and Per Einarsson. 2010. A Real-Time Radiosity Architecture for Video Games. SIGGRAPH 2010 Course: Advances in Real-Time Rendering in 3D Graphics and Games.
51. Morgan McGuire, Mike Mara, Derek Nowrouzezahrai, and David Luebke. 2017. Real-Time Global Illumination Using Precomputed Light Field Probes. In Proceedings of the 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D '17). Association for Computing Machinery, New York, NY, USA, Article 2, 11 pages.
52. Johannes Meng, Marios Papas, Ralf Habel, Carsten Dachsbacher, Steve Marschner, Markus Gross, and Wojciech Jarosz. 2015. Multi-Scale Modeling and Rendering of Granular Materials. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 34, 4 (July 2015). https://doi.org/10/gfzndr
53. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.
54. Martin Mittring. 2007. Finding next Gen: CryEngine 2. In ACM SIGGRAPH 2007 Courses (SIGGRAPH '07). Association for Computing Machinery, New York, NY, USA, 97--121.
55. Pierre Moreau, Matt Pharr, and Petrik Clarberg. 2019. Dynamic Many-Light Sampling for Real-Time Ray Tracing. In High-Performance Graphics 2019 - Short Papers, Strasbourg, France, July 8-10, 2019, Markus Steinberger and Tim Foley (Eds.). Eurographics Association, 21--26.
56. Thomas Müller. 2021. Tiny CUDA Neural Network Framework. https://github.com/nvlabs/tiny-cuda-nn.
57. Thomas Müller, Markus Gross, and Jan Novák. 2017. Practical Path Guiding for Efficient Light-Transport Simulation. Computer Graphics Forum 36, 4 (June 2017), 91--100.
58. Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural Importance Sampling. ACM Trans. Graph. 38, 5, Article 145 (Oct. 2019), 19 pages.
59. Thomas Müller, Fabrice Rousselle, Alexander Keller, and Jan Novák. 2020. Neural Control Variates. ACM Trans. Graph. 39, 6, Article 243 (Nov. 2020), 19 pages.
60. Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, and Tobias Ritschel. 2017. Deep Shading: Convolutional Neural Networks for Screen-Space Shading. 36, 4 (2017).
61. Christopher Oat. 2005. Irradiance Volumes for Games. In Game Developers Conference.
62. Jacopo Pantaleoni. 2020. Online path sampling control with progressive spatio-temporal filtering. arXiv:2005.07547 (May 2020).
63. Hauke Rehfeld, Tobias Zirr, and Carsten Dachsbacher. 2014. Clustered Pre-Convolved Radiance Caching. In Proceedings of the 14th Eurographics Symposium on Parallel Graphics and Visualization (PGV '14). Eurographics Association, Goslar, DEU, 25--32.
64. Peiran Ren, Jiaping Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013. Global Illumination with Radiance Regression Functions. ACM Trans. Graph. 32, 4, Article 130 (July 2013), 12 pages.
65. Tobias Ritschel, Carsten Dachsbacher, Thorsten Grosch, and Jan Kautz. 2012. The State of the Art in Interactive Global Illumination. Computer Graphics Forum 31, 1 (2012), 160--188.
66. Tobias Ritschel, Thomas Engelhardt, Thorsten Grosch, Hans-Peter Seidel, Jan Kautz, and Carsten Dachsbacher. 2009a. Micro-Rendering for Scalable, Parallel Final Gathering. ACM Trans. Graph. 28, 5 (Dec. 2009), 1--8.
67. Tobias Ritschel, Thorsten Grosch, Min H. Kim, Hans-Peter Seidel, Carsten Dachsbacher, and Jan Kautz. 2008. Imperfect Shadow Maps for Efficient Computation of Indirect Illumination. ACM Trans. Graph. 27, 5, Article 129 (Dec. 2008), 8 pages.
68. Tobias Ritschel, Thorsten Grosch, and Hans-Peter Seidel. 2009b. Approximating Dynamic Global Illumination in Image Space. In Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games (I3D '09). Association for Computing Machinery, New York, NY, USA, 75--82.
69. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2011. Adaptive Sampling and Reconstruction Using Greedy Error Minimization. ACM Trans. Graph. 30, 6, Article 159 (Dec. 2011), 12 pages.
70. Daniel Scherzer, Chuong H. Nguyen, Tobias Ritschel, and Hans-Peter Seidel. 2012. Pre-Convolved Radiance Caching. Comput. Graph. Forum 31, 4 (June 2012), 1391--1397.
71. Dario Seyb, Peter-Pike Sloan, Ari Silvennoinen, Michał Iwanicki, and Wojciech Jarosz. 2020. The design and evolution of the UberBake light baking system. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020).
72. Ari Silvennoinen and Jaakko Lehtinen. 2017. Real-Time Global Illumination by Pre-computed Local Reconstruction from Sparse Radiance Probes. ACM Trans. Graph. 36, 6, Article 230 (Nov. 2017), 13 pages.
73. Peter-Pike Sloan, Jan Kautz, and John Snyder. 2002. Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Lighting Environments. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '02). Association for Computing Machinery, New York, NY, USA, 527--536.
74. Tiago Sousa, Nickolay Kasyan, and Nicolas Schulz. 2011. Secrets of CryENGINE 3 Graphics Technology. In Advances in Real-Time Rendering in Games: Part I. ACM SIGGRAPH 2011 Courses.
75. Eric Tabellion and Arnauld Lamorlette. 2004. An Approximate Global Illumination System for Computer Generated Films. In ACM SIGGRAPH 2004 Papers (SIGGRAPH '04). Association for Computing Machinery, New York, NY, USA, 469--476.
76. Sergios Theodoridis. 2008. Pattern Recognition. Elsevier.
77. Kostas Vardis, Georgios Papaioannou, and Anastasios Gkaravelis. 2014. Real-time Radiance Caching using Chrominance Compression. Journal of Computer Graphics Techniques (JCGT) 3, 4 (16 December 2014), 111--131. http://jcgt.org/published/0003/04/06/
78. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.03762 (June 2017).
79. Eric Veach and Leonidas J. Guibas. 1995. Optimally Combining Sampling Techniques for Monte Carlo Rendering. In Proc. SIGGRAPH. 419--428.
80. Jiří Vorba, Johannes Hanika, Sebastian Herholz, Thomas Müller, Jaroslav Křivánek, and Alexander Keller. 2019. Path Guiding in Production. In ACM SIGGRAPH Courses. ACM, New York, NY, USA, 18:1--18:77.
81. Jiří Vorba, Ondřej Karlík, Martin Šik, Tobias Ritschel, and Jaroslav Křivánek. 2014. On-line Learning of Parametric Mixture Models for Light Transport Simulation. ACM Trans. Graph. 33, 4 (Aug. 2014).
82. Gregory J. Ward, Francis M. Rubinstein, and Robert D. Clear. 1988. A Ray Tracing Solution for Diffuse Interreflection. In Proceedings of the 15th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '88). Association for Computing Machinery, New York, NY, USA, 85--92.
83. Yangyang Zhao, Laurent Belcour, and Derek Nowrouzezahrai. 2019. View-Dependent Radiance Caching. In Proceedings of the 45th Graphics Interface Conference on Proceedings of Graphics Interface 2019 (GI'19). Canadian Human-Computer Communications Society, Waterloo, CAN, Article 22, 9 pages.

Index Terms

• Computing methodologies

  • Computer graphics

    • Rendering

      • Ray tracing

  • Machine learning

    • Learning paradigms

      • Reinforcement learning

      • Supervised learning

        • Supervised learning by regression

    • Machine learning approaches

      • Neural networks