Monte Carlo estimators for differential light transport

Abstract

Physically based differentiable rendering algorithms propagate derivatives through realistic light transport simulations and have applications in diverse areas including inverse reconstruction and machine learning. Recent progress has led to unbiased methods that can simultaneously compute derivatives with respect to millions of parameters. At the same time, elementary properties of these methods remain poorly understood.

Current algorithms for differentiable rendering are constructed by mechanically differentiating a given primal algorithm. While convenient, such an approach is simplistic because it leaves no room for improvement. Differentiation produces major changes in the integrals that occur throughout the rendering process, which indicates that the primal and differential algorithms should be decoupled so that the latter can suitably adapt.

This leads to a large space of possibilities: consider that even the most basic Monte Carlo path tracer already involves several design choices concerning the techniques for sampling materials and emitters, and their combination, e.g. via multiple importance sampling (MIS). Differentiation causes a veritable explosion of this decision tree: should we differentiate only the estimator, or also the sampling technique? Should MIS be applied before or after differentiation? Are specialized derivative sampling strategies of any use? How should visibility-related discontinuities be handled when millions of parameters are differentiated simultaneously? In this paper, we provide a taxonomy and analysis of different estimators for differential light transport to provide intuition about these and related questions.

References

1. Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In Proceedings of Computer Vision and Pattern Recognition (CVPR), IEEE.
2. Sai Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased Warped-Area Sampling for Differentiable Rendering. ACM Transactions on Graphics 39, 6 (2020), 245:1--245:18.
3. Petr Beckmann and Andre Spizzichino. 1987. The scattering of electromagnetic waves from rough surfaces. Norwood (1987).
4. Laurent Belcour, Cyril Soler, Kartic Subr, Nicolas Holzschuch, and Fredo Durand. 2013. 5D Covariance Tracing for Efficient Defocus and Motion Blur. ACM Transactions on Graphics 32, 3 (July 2013).
5. Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.
6. Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2018. Inverse Transport Networks. arXiv preprint arXiv:1809.10820 (2018).
7. Min Chen and James Arvo. 2000. Theory and Application of Specular Path Perturbation. ACM Transactions on Graphics 19, 4 (Oct. 2000), 246--278.
8. Robert L Cook and Kenneth E. Torrance. 1982. A reflectance model for computer graphics. ACM Transactions on Graphics (ToG) 1, 1 (1982), 7--24.
9. Luc Devroye. 1986. Non-Uniform Random Variate Generation. Springer-Verlag.
10. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An evaluation of computational imaging techniques for heterogeneous inverse scattering. In European Conference on Computer Vision. Springer, 685--701.
11. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse Volume Rendering with Material Dictionaries. ACM Transactions on Graphics 32, 6, Article 162 (Nov. 2013).
12. Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. Vol. 105. SIAM.
13. Carole K. Hayakawa, Jerome Spanier, Frédéric Bevilacqua, Andrew K. Dunn, Joon S. You, Bruce J. Tromberg, and Vasan Venugopalan. 2001. Perturbation Monte Carlo methods to solve inverse photon migration problems in heterogeneous tissues. Opt. Lett. 26, 17 (Sep 2001), 1335--1337.
14. Paul S Heckbert. 1989. Fundamentals of texture mapping and image warping. Master's thesis.
15. Eric Heitz and Eugene D'Eon. 2014. Importance Sampling Microfacet-Based BSDFs using the Distribution of Visible Normals. Computer Graphics Forum 33, 4 (July 2014), 103--112.
16. Binh-Son Hua, Adrien Gruson, Victor Petitjean, Matthias Zwicker, Derek Nowrouzezahrai, Elmar Eisemann, and Toshiya Hachisuka. 2019. A Survey on Gradient-Domain Rendering. Computer Graphics Forum 38, 2 (2019), 455--472.
17. Homan Igehy. 1999. Tracing Ray Differentials. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 99). 179--186.
18. Wenzel Jakob and Steve Marschner. 2012. Manifold Exploration: A Markov Chain Monte Carlo Technique for Rendering Scenes with Difficult Specular Transport. ACM Transactions on Graphics 31, 4 (July 2012).
19. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
20. Jaroslav Krivánek, Pascal Gautron, Sumanta Pattanaik, and Kadi Bouatouch. 2005. Radiance caching for efficient global illumination computation. IEEE Transactions on Visualization and Computer Graphics 11, 5 (2005), 550--561.
21. Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular Primitives for High-Performance Differentiable Rendering. ACM Transactions on Graphics 39, 6 (2020).
22. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1--222:11.
23. Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. 2019. Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction. CoRR abs/1901.05567 (2019). arXiv:1901.05567 http://arxiv.org/abs/1901.05567
24. Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In European Conference on Computer Vision. Springer.
25. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38, 6 (Dec. 2019).
26. Iván Lux and Lázló Koblinger. 1990. Monte Carlo Particle Transport Methods: Neutron and Photon Calculations. CRC Press, Boston.
27. Don Mitchell and Pat Hanrahan. 1992. Illumination from curved reflectors. In Proceedings of the 19th annual conference on Computer graphics and interactive techniques. 283--291.
28. Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020).
29. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38, 6 (Dec. 2019).
30. Art B. Owen. 2013. Monte Carlo theory, methods and examples. https://statweb.stanford.edu/~owen/mc/
31. Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich. 2010. OptiX: A General Purpose Ray Tracing Engine. ACM Transactions on Graphics 29, 4, Article 66 (July 2010), 13 pages.
32. Felix Petersen, Amit H. Bermano, Oliver Deussen, and Daniel Cohen-Or. 2019. Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer. CoRR abs/1903.11149 (2019). arXiv:1903.11149 http://arxiv.org/abs/1903.11149
33. Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically Based Rendering: From Theory to Implementation (3rd ed.) (3rd ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. 1266 pages.
34. Ravi Ramamoorthi, Dhruv Mahajan, and Peter Belhumeur. 2007. A first-order analysis of lighting, shading, and shadows. ACM Transactions on Graphics (TOG) 26, 1 (2007).
35. Helge Rhodin, Nadia Robertini, Christian Richardt, Hans-Peter Seidel, and Christian Theobalt. 2015. A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation. In Proceedings of ICCV 2015.
36. Kenneth E Torrance and Ephraim M Sparrow. 1967. Theory for off-specular reflection from roughened surfaces. Josa 57, 9 (1967), 1105--1114.
37. T. S. Trowbridge and K. P. Reitz. 1975. J. Opt. Soc. Am. 65, 5 (May 1975), 531--536.
38. Eric Veach and Leonidas J. Guibas. 1995. Optimally Combining Sampling Techniques for Monte Carlo Rendering. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). Association for Computing Machinery, New York, NY, USA, 419--428.
39. Darko Veberic. 2010. Having Fun with Lambert W(x) Function. CoRR abs/1003.1628 (2010). arXiv:1003.1628 http://arxiv.org/abs/1003.1628
40. Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021. Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time. Transactions on Graphics (Proceedings of SIGGRAPH) 40, 4 (Aug. 2021).
41. Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (Grenoble, France) (EGSR'07). Eurographics Association, Goslar, DEU, 195--206.
42. G J Ward and P S Heckbert. 1992. Irradiance gradients. Technical Report. Lawrence
43. Berkeley Lab., CA (United States); Ecole Polytechnique Federale, Lausanne (Switzerland); Technische Hogeschool Delft (Netherlands). Dept. of Technical Mathematics and Informatics.
44. Tizian Zeltner, Iliyan Georgiev, and Wenzel Jakob. 2020. Specular Manifold Sampling for Rendering High-Frequency Caustics and Glints. Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020).
45. Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM Transactions on Graphics 39, 4 (2020), 143:1--143:19.
46. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A Differential Theory of Radiative Transfer. ACM Transactions on Graphics 38, 6 (2019), 227:1--227:16.
47. Shaung Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling Scattering Parameters for Rendering Anisotropic Media. ACM Transactions on Graphics 35, 6 (2016).

Index Terms

• Computing methodologies

  • Computer graphics

    • Rendering