Path-space differentiable rendering of participating media

Abstract

Physics-based differentiable rendering---which focuses on estimating derivatives of radiometric detector responses with respect to arbitrary scene parameters---has a diverse array of applications from solving analysis-by-synthesis problems to training machine-learning pipelines incorporating forward-rendering processes. Unfortunately, existing general-purpose differentiable rendering techniques lack either the generality to handle volumetric light transport or the flexibility to devise Monte Carlo estimators capable of handling complex geometries and light transport effects.

In this paper, we bridge this gap by showing how generalized path integrals can be differentiated with respect to arbitrary scene parameters. Specifically, we establish the mathematical formulation of generalized differential path integrals that capture both interfacial and volumetric light transport. Our formulation allows the development of advanced differentiable rendering algorithms capable of efficiently handling challenging geometric discontinuities and light transport phenomena such as volumetric caustics.

We validate our method by comparing our derivative estimates to those generated using the finite differences. Further, to demonstrate the effectiveness of our technique, we compare both differentiable rendering and inverse rendering performance with state-of-the-art methods.

References

1. Sai Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased warped-area sampling for differentiable rendering. ACM Trans. Graph. 39, 6 (2020), 245:1--245:18.
2. Eva Cerezo, Frederic Pérez, Xavier Pueyo, Francisco J Seron, and François X Sillion. 2005. A survey on participating media rendering techniques. The Visual Computer 21, 5 (2005), 303--328.
3. Subrahmanyan Chandrasekhar. 1960. Radiative transfer. Courier Corporation.
4. Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2020. Towards Learning-based Inverse Subsurface Scattering. In 2020 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--12.
5. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse volume rendering with material dictionaries. ACM Trans. Graph. 32, 6 (2013), 162:1--162:13.
6. John Michael Hammersley and JG Mauldon. 1956. General principles of antithetic variates. In Mathematical proceedings of the Cambridge philosophical society, Vol. 52. Cambridge University Press, 476--481.
7. Wenzel Jakob and Steve Marschner. 2012a. Manifold Exploration: A Markov Chain Monte Carlo technique for rendering scenes with difficult specular transport. ACM Trans.Graph. 31, 4 (2012), 58:1--58:13 pages.
8. Wenzel Jakob and Steve Marschner. 2012b. Manifold exploration: A Markov chain Monte Carlo technique for rendering scenes with difficult specular transport. ACM Trans. Graph. 31, 4 (2012), 58:1--58:13.
9. James T. Kajiya. 1986. The Rendering Equation. In SIGGRAPH '86. 143--150.
10. James T. Kajiya and Brian P Von Herzen. 1984. Ray tracing volume densities. SIGGRAPH Comput. Graph. 18, 3 (1984), 165--174.
11. Csaba Kelemen, László Szirmay-Kalos, György Antal, and Ferenc Csonka. 2002. A simple and robust mutation strategy for the metropolis light transport algorithm. In Computer Graphics Forum, Vol. 21. Wiley Online Library, 531--540.
12. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
13. Eric P Lafortune and Yves D Willems. 1996. Rendering participating media with bidirectional path tracing. In Rendering techniques' 96. Springer, 91--100.
14. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo ray tracing through edge sampling. ACM Trans. Graph. 37, 6 (2018), 222:1--222:11.
15. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. 38, 6 (2019), 228:1--228:14.
16. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A retargetable forward and inverse renderer. ACM Trans. Graph. 38, 6 (2019), 203:1--203:17.
17. Jan Novák, Iliyan Georgiev, Johannes Hanika, and Wojciech Jarosz. 2018. Monte Carlo methods for volumetric light transport simulation. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 551--576.
18. Mark Pauly, Thomas Kollig, and Alexander Keller. 2000. Metropolis light transport for participating media. In Rendering Techniques 2000. Springer, 11--22.
19. Osborne Reynolds. 1903. Papers on mechanical and physical subjects: the sub-mechanics of the universe. Vol. 3. The University Press.
20. Hang Si. 2015. TetGen, a Delaunay-based quality tetrahedral mesh generator. ACM Transactions on Mathematical Software (TOMS) 41, 2 (2015), 1--36.
21. Jerome Spanier and Ely M Gelbard. 1969. Monte Carlo principles and neutron transport problems. The Addison-Wesley Publishing Company.
22. Denis Sumin, Tobias Rittig, Vahid Babaei, Thomas Nindel, Alexander Wilkie, Piotr Didyk, Bernd Bickel, Jaroslav Křivánek, Karol Myszkowski, and Tim Weyrich. 2019. Geometry-aware scattering compensation for 3D printing. ACM Trans. Graph. 38, 4 (2019), 111:1--111:14.
23. Jean-Marc Thiery, Pooran Memari, and Tamy Boubekeur. 2018. Mean value coordinates for quad cages in 3D. ACM Trans. Graph. 37, 6 (2018), 229:1--229:14.
24. Eric Veach. 1997. Robust Monte Carlo methods for light transport simulation. Vol. 1610. Stanford University PhD thesis.
25. Eric Veach and Leonidas Guibas. 1995. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques. Springer, 145--167.
26. Eric Veach and Leonidas J. Guibas. 1997. Metropolis light transport. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., 65--76.
27. E Woodcock, T Murphy, P Hemmings, and S Longworth. 1965. Techniques used in the GEM code for Monte Carlo neutronics calculations in reactors and other systems of complex geometry. In Proc. Conf. Applications of Computing Methods to Reactor Problems, Vol. 557.
28. Cheng Zhang, Zhao Dong, Michael Doggett, and Shuang Zhao. 2021. Antithetic sampling for Monte Carlo differentiable rendering. ACM Trans. Graph. 40, 4 (2021), 77:1--77:12.
29. Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-space differentiable rendering. ACM Trans. Graph. 39, 4 (2020), 143:1--143:19.
30. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A differential theory of radiative transfer. ACM Trans. Graph. 38, 6 (2019), 227:1--227:16.
31. Shuang Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling scattering parameters for rendering anisotropic media. ACM Trans. Graph. 35, 6 (2016), 166:1--166:11.

Index Terms

• Computing methodologies

  • Computer graphics

    • Rendering