Differentiable signed distance function rendering

Abstract

Physically-based differentiable rendering has recently emerged as an attractive new technique for solving inverse problems that recover complete 3D scene representations from images. The inversion of shape parameters is of particular interest but also poses severe challenges: shapes are intertwined with visibility, whose discontinuous nature introduces severe bias in computed derivatives unless costly precautions are taken. Shape representations like triangle meshes suffer from additional difficulties, since the continuous optimization of mesh parameters cannot introduce topological changes.

One common solution to these difficulties entails representing shapes using signed distance functions (SDFs) and gradually adapting their zero level set during optimization. Previous differentiable rendering of SDFs did not fully account for visibility gradients and required the use of mask or silhouette supervision, or discretization into a triangle mesh.

In this article, we show how to extend the commonly used sphere tracing algorithm so that it additionally outputs a reparameterization that provides the means to compute accurate shape parameter derivatives. At a high level, this resembles techniques for differentiable mesh rendering, though we show that the SDF representation admits a particularly efficient reparameterization that outperforms prior work. Our experiments demonstrate the reconstruction of (synthetic) objects without complex regularization or priors, using only a per-pixel RGB loss.

References

1. David Adalsteinsson and James A. Sethian. 1995. A Fast Level Set Method for Propagating Interfaces. J. Comput. Phys. 118, 2 (1995), 269--277.
2. David Adalsteinsson and James A. Sethian. 1999. The Fast Construction of Extension Velocities in Level Set Methods. J. Comput. Phys. 148, 1 (1999), 2--22.
3. Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, Haggai Maron, and Yaron Lipman. 2019. Controlling neural level sets. In Advances in Neural Information Processing Systems (NeurIPS). 2032--2041.
4. Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
5. Sai Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased Warped-Area Sampling for Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 39, 6 (2020), 245:1--245:18.
6. Sai Bangaru, Jesse Michel, Kevin Mu, Gilbert Bernstein, Tzu-Mao Li, and Jonathan Ragan-Kelley. 2021. Systematically Differentiating Parametric Discontinuities. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 107 (2021), 107:1--107:17.
7. Brent Burley. 2012. Physically-based shading at Disney. SIGGRAPH Course Notes. Practical physically-based shading in film and game production. (2012), 1--27.
8. Forrester Cole, Kyle Genova, Avneesh Sud, Daniel Vlasic, and Zhoutong Zhang. 2021. Differentiable Surface Rendering via Non-Differentiable Sampling. (2021). arXiv:2108.04886
9. Brian Curless and Marc Levoy. 1996. A Volumetric Method for Building Complex Models from Range Images. In SIGGRAPH Comput. Graph. 303--312.
10. Miles Detrixhe, Frédéric Gibou, and Chohong Min. 2013. A parallel fast sweeping method for the eikonal equation. J. Comput. Phys. 237 (2013), 46--55.
11. Akio Doi and Akio Koide. 1991. An Efficient Method of Triangulating Equi-Valued Surfaces by Using Tetrahedral Cells. IEICE Transactions on Information and Systems 1, 74 (1991), 214--224.
12. Pau Gargallo, Emmanuel Prados, and Peter Sturm. 2007. Minimizing the Reprojection Error in Surface Reconstruction from Images. The IEEE International Conference on Computer Vision (ICCV) (2007), 1--8.
13. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An evaluation of computational imaging techniques for heterogeneous inverse scattering. In European Conference on Computer Vision. Springer, 685--701.
14. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse Volume Rendering with Material Dictionaries. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 32, 6, Article 162 (Nov. 2013), 13 pages.
15. José Gomes and Olivier Faugeras. 2000. Reconciling Distance Functions and Level Sets. Journal of Visual Communication and Image Representation 11, 2 (2000), 209--223.
16. John C. Hart. 1996. Sphere tracing: A geometric method for the antialiased ray tracing of implicit surfaces. The Visual Computer 12, 10 (1 Jan. 1996), 527--545.
17. Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. 2020. SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
18. James T. Kajiya. 1986. The Rendering Equation. In SIGGRAPH Comput. Graph. 143--150.
19. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
20. Benjamin Keinert, Henry Schäfer, Johann Korndörfer, Urs Ganse, and Marc Stamminger. 2014. Enhanced Sphere Tracing. In Smart Tools and Apps for Graphics - Eurographics Italian Chapter Conference. The Eurographics Association.
21. Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching Real Fabrics with Micro-Appearance Models. ACM Trans. Graph. 35, 1, Article 1 (Dec. 2015), 26 pages.
22. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR).
23. Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular Primitives for High-Performance Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 39, 6 (2020).
24. A. Laurentini. 1994. The visual hull concept for silhouette-based image understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence 16, 2 (1994), 150--162.
25. Shingyu Leung and Jianliang Qian. 2006. An adjoint state method for three-dimensional transmission traveltime tomography using first-arrivals. Communications in Mathematical Sciences 4, 1 (2006), 249 -- 266.
26. Chunming Li, Chenyang Xu, Changfeng Gui, and M.D. Fox. 2005. Level set evolution without re-initialization: a new variational formulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 430--436.
27. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1--222:11.
28. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. The IEEE International Conference on Computer Vision (ICCV) (Oct. 2019).
29. Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. 2020. DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
30. Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural Volumes: Learning Dynamic Renderable Volumes from Images. ACM Trans. Graph. (Proc. SIGGRAPH) 38, 4, Article 65 (July 2019), 14 pages.
31. Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In European Conference on Computer Vision (ECCV). Springer, 154--169.
32. William E. Lorensen and Harvey E. Cline. 1987. Marching Cubes: A High Resolution 3D Surface Construction Algorithm. In SIGGRAPH Comput. Graph. 163--169.
33. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (Dec. 2019).
34. Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. 2004. Fluid control using the adjoint method. In ACM Trans. Graph. (Proc. SIGGRAPH), Vol. 23. ACM, 449--456.
35. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In European Conference on Computer Vision (ECCV).
36. Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Mueller, and Sanja Fidler. 2022. Extracting Triangular 3D Models, Materials, and Lighting From Images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
37. Ken Museth. 2017. Novel Algorithm for Sparse and Parallel Fast Sweeping: Efficient Computation of Sparse Signed Distance Fields. In ACM SIGGRAPH Talks. Article 74.
38. Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE International Symposium on Mixed and Augmented Reality. 127--136.
39. Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. 2021. Large Steps in Inverse Rendering of Geometry. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 40, 6 (Dec. 2021).
40. Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. 2020. Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
41. Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013. Real-Time 3D Reconstruction at Scale Using Voxel Hashing. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 32, 6, Article 169 (Nov. 2013), 11 pages.
42. Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4, Article 146 (July 2020), 15 pages.
43. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (Nov. 2019), 17 pages.
44. Stanley Osher and James A. Sethian. 1988. Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations. J. Comput. Phys. 79, 1 (1988), 12--49.
45. Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically Based Rendering: From Theory to Implementation (3rd ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
46. Edoardo Remelli, Artem Lukoianov, Stephan Richter, Benoit Guillard, Timur Bagautdinov, Pierre Baque, and Pascal Fua. 2020. MeshSDF: Differentiable Iso-Surface Extraction. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33.
47. Osborne Reynolds. 1903. Papers on mechanical and physical subjects: the sub-mechanics of the universe, Vol. 3. The University Press.
48. Darius Rückert, Linus Franke, and Marc Stamminger. 2021. Adop: Approximate differentiable one-pixel point rendering. (2021). arXiv:2110.06635
49. James A. Sethian. 1996. A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences 93, 4 (1996), 1591--1595.
50. James A. Sethian. 1999. Fast Marching Methods. SIAM Rev. 41, 2 (1999), 199--235.
51. Dario Seyb, Alec Jacobson, Derek Nowrouzezahrai, and Wojciech Jarosz. 2019. Nonlinear sphere tracing for rendering deformed signed distance fields. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (Nov. 2019).
52. Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. 2021. Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis. In Advances in Neural Information Processing Systems (NeurIPS).
53. Jos Stam and Ryan Schmidt. 2011. On the Velocity of an Implicit Surface. ACM Trans. Graph. 30, 3, Article 21 (May 2011), 7 pages.
54. Yen-Hsi Richard Tsai, Li-Tien Cheng, Stanley Osher, Paul Burchard, and Guillermo Sapiro. 2004. Visibility and its dynamics in a PDE based implicit framework. J. Comput. Phys. 199, 1 (2004), 260--290.
55. Eric Veach and Leonidas J. Guibas. 1995. Optimally Combining Sampling Techniques for Monte Carlo Rendering. In SIGGRAPH Comput. Graph. Association for Computing Machinery, New York, NY, USA, 419--428.
56. Delio Vicini, Wenzel Jakob, and Anton Kaplanyan. 2021a. A Non-Exponential Transmittance Model for Volumetric Scene Representations. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 4 (Aug. 2021), 136:1--136:16.
57. Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021b. Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 4 (Aug. 2021), 108:1--108:14.
58. Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In Advances in Neural Information Processing Systems (NeurIPS).
59. Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. 2021. Neural Fields in Visual Computing and Beyond. arXiv:2111.11426
60. Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. 2021. Volume Rendering of Neural Implicit Surfaces. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 34.
61. Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. 2020. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33.
62. Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung. 2019. Differentiable Surface Splatting for Point-based Geometry Processing. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (2019).
63. Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022. Plenoxels: Radiance Fields without Neural Networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
64. Tizian Zeltner, Sébastien Speierer, Iliyan Georgiev, and Wenzel Jakob. 2021. Monte Carlo Estimators for Differential Light Transport. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 4 (Aug. 2021), 78:1--78:16.
65. Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4, Article 143 (July 2020), 19 pages.
66. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A Differential Theory of Radiative Transfer. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6, Article 227 (Nov. 2019), 16 pages.
67. Cheng Zhang, Zihan Yu, and Shuang Zhao. 2021. Path-Space Differentiable Rendering of Participating Media. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 4 (2021), 76:1--76:15.
68. Hongkai Zhao. 2004. A fast sweeping method for eikonal equations. Math. Comp. 74 (2004), 603--627. Issue 250.
69. Hong-Kai Zhao, S. Osher, and R. Fedkiw. 2001. Fast surface reconstruction using the level set method. In Proceedings IEEE Workshop on Variational and Level Set Methods in Computer Vision. 194--201.
70. Yang Zhou, Lifan Wu, Ravi Ramamoorthi, and Ling-Qi Yan. 2021. Vectorization for Fast, Analytic, and Differentiable Visibility. ACM Trans. Graph. 40, 3, Article 27 (July 2021), 21 pages.

Index Terms

• Computing methodologies

  • Computer graphics

    • Rendering