Abstract
We propose, DeepPRT, a deep convolutional neural network to compactly encapsulate the radiance transfer of a freely deformable object for rasterization in real-time.
With pre-computation of radiance transfer (PRT) we can store complex light interactions appropriate to the shape of a given object at each surface point for subsequent real-time rendering via fast linear algebra evaluation against the viewing direction and distant light environment. However, performing light transport projection into an efficient basis representation, such as Spherical Harmonics (SH), requires a numerical Monte Carlo integration computation, limiting usage to rigid only objects or highly constrained deformation sequences. The bottleneck, when considering freely deformable objects, is the heavy memory requirement to wield all pre-computations in rendering with global illumination results. We present a compact representation of PRT for deformable objects with fixed memory consumption, which solves diverse non-linear deformations and is shown to be effective beyond the input training set. Specifically, a U-Net is trained to predict the coefficients of the transfer function (SH coefficients in this case), for a given animation's shape query each frame in real-time.
We contribute deep learning of PRT within a parametric surface space representation via geometry images using harmonic mapping with a texture space filling energy minimization variant. This surface representation facilitates the learning procedure, removing irrelevant, deformation invariant information; and supports standard convolution operations. Finally, comparisons with ground truth and a recent linear morphable-model method is provided.
- Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of 3D Faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '99). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 187--194. Google Scholar
Digital Library
- Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael M. Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. CoRR abs/1605.06437 (2016). arXiv:1605.06437 http://arxiv.org/abs/1605.06437Google Scholar
- Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2016. Geometric deep learning: going beyond Euclidean data. CoRR abs/1611.08097 (2016). arXiv:1611.08097 http://arxiv.org/abs/1611.08097Google Scholar
- Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder. ACM Trans. Graph. 36, 4, Article 98 (July 2017), 12 pages. Google Scholar
Digital Library
- Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. 2017. A Survey of Model Compression and Acceleration for Deep Neural Networks. CoRR abs/1710.09282 (2017). arXiv:1710.09282 http://arxiv.org/abs/1710.09282Google Scholar
- François Chollet et al. 2015. Keras. https://keras.io.Google Scholar
- Michael F. Cohen, John Wallace, and Pat Hanrahan. 1993. Radiosity and Realistic Image Synthesis. Academic Press Professional, Inc., San Diego, CA, USA. Google Scholar
Digital Library
- Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. 2017. Deformable Convolutional Networks. CoRR abs/1703.06211 (2017). arXiv:1703.06211 http://arxiv.org/abs/1703.06211Google Scholar
- Nicki Skafte Detlefsen, Oren Freifeld, and Søren Hauberg. 2018. Deep Diffeomorphic Transformer Networks. Conference on Computer Vision and Pattern Recognition (CVPR) (2018).Google Scholar
- James Eells and J. H. Sampson. 1964. Harmonic mappings of Riemannian manifords. The Johns Hopkins University Press. 86:109--160 pages.Google Scholar
- Wei-Wen Feng, Liang Peng, Yuntao Jia, and Yizhou Yu. 2007. Large-scale Data Management for PRT-based Real-time Rendering of Dynamically Skinned Models. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR'07). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 23--34. Google Scholar
Digital Library
- David Gu. 2018. Harmonic Map. https://www3.cs.stonybrook.edu/~gu/tutorial/HarmonicMap.html Accessed: 28-11-2018.Google Scholar
- Xianfeng Gu, Steven J Gortler, and Hugues Hoppe. 2002. Geometry images. ACM Transactions on Graphics (TOG) 21, 3 (2002), 355--361. Google Scholar
Digital Library
- Song Han, Huizi Mao, and William J. Dally. 2015. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. CoRR abs/1510.00149 (2015). arXiv:1510.00149 http://arxiv.org/abs/1510.00149Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. CoRR abs/1512.03385 (2015). arXiv:1512.03385 http://arxiv.org/abs/1512.03385Google Scholar
- Pedro Hermosilla, Sebastian Maisch, Tobias Ritschel, and Timo Ropinski. 2018. Deep-learning the Latent Space of Light Transport. arXiv abs/1811.04756 (2018).Google Scholar
- M. Horowitz. 2014. 1.1 Computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC). 10--14.Google Scholar
Cross Ref
- Doug L. James and Kayvon Fatahalian. 2003. Precomputing Interactive Dynamic Deformable Scenes. ACM Trans. Graph. 22, 3 (July 2003), 879--887. Google Scholar
Digital Library
- A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. 2014. Large-Scale Video Classification with Convolutional Neural Networks. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. 1725--1732. Google Scholar
Digital Library
- Jan Kautz, Peter-Pike Sloan, and John Snyder. 2002. Fast, Arbitrary BRDF Shading for Low-frequency Lighting Using Spherical Harmonics. In Proceedings of the 13th Eurographics Workshop on Rendering (EGRW'02). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 291--296. http://dl.acm.org/citation.cfm?id=581896.581934 Google Scholar
Digital Library
- Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014). arXiv:1412.6980 http://arxiv.org/abs/1412.6980Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1 (NIPS'12). Curran Associates Inc., USA, 1097--1105. http://dl.acm.org/citation.cfm?id=2999134.2999257 Google Scholar
Digital Library
- Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436--444.Google Scholar
- Yunze Man, Yangsibo Huang, Junyi Feng, Xi Li, and Fei Wu. 2018. Deep Q Learning Driven CT Pancreas Segmentation with Geometry-Aware U-Net.Google Scholar
- Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, Vladimir G. Kim, and Yaron Lipman. 2017. Convolutional Neural Networks on Surfaces via Seamless Toric Covers. ACM Trans. Graph. 36, 4, Article 71 (July 2017), 10 pages. Google Scholar
Digital Library
- Jonathan Masci, Davide Boscaini, Michael M. Bronstein, and Pierre Vandergheynst. 2015. ShapeNet: Convolutional Neural Networks on Non-Euclidean Manifolds. CoRR abs/1501.06297 (2015). arXiv:1501.06297 http://arxiv.org/abs/1501.06297Google Scholar
- Jonathan Masci, Emanuele Rodolà, Davide Boscaini, Michael M. Bronstein, and Hao Li. 2016. Geometric Deep Learning. In SIGGRAPH ASIA 2016 Courses (SA '16). ACM, New York, NY, USA, Article 1, 50 pages. Google Scholar
Digital Library
- Federico Monti. 2018. Geometric Deep Learning. http://geometricdeeplearning.com/ Accessed: 26-11-2018.Google Scholar
- Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, and Tobias Ritschel. 2017. Deep Shading: Convolutional Neural Networks for Screen-Space Shading. 36, 4 (2017). Google Scholar
Digital Library
- Ren Ng, Ravi Ramamoorthi, and Pat Hanrahan. 2003. All-frequency Shadows Using Non-linear Wavelet Lighting Approximation. ACM Trans. Graph. 22, 3 (July 2003), 376--381. Google Scholar
Digital Library
- Emil Praun and Hugues Hoppe. 2003. Spherical Parametrization and Remeshing. ACM Trans. Graph. 22, 3 (July 2003), 340--349. Google Scholar
Digital Library
- Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. CoRR abs/1505.04597 (2015). arXiv:1505.04597 http://arxiv.org/abs/1505.04597Google Scholar
- A. Schneider, S. SchÃűnborn, B. Egger, L. Frobeen, and T. Vetter. 2017. Efficient Global Illumination for Morphable Models. In 2017 IEEE International Conference on Computer Vision (ICCV). 3885--3893.Google Scholar
- Ayan Sinha, Jing Bai, and Karthik Ramani. 2016. Deep learning 3D shape surfaces using geometry images. In European Conference on Computer Vision. Springer, 223--240.Google Scholar
Cross Ref
- Peter-Pike Sloan, Jan Kautz, and John Snyder. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In ACM Transactions on Graphics (TOG), Vol. 21. ACM, 527--536. Google Scholar
Digital Library
- Peter-Pike Sloan, Ben Luna, and John Snyder. 2005. Local, Deformable Precomputed Radiance Transfer. ACM, 1216--1224. https://www.microsoft.com/en-us/research/publication/local-deformable-precomputed-radiance-transfer/ Google Scholar
Digital Library
- Peter-Pike Sloan and Ari Silvennoinen. 2018. Directional Lightmap Encoding Insights. In SIGGRAPH Asia 2018 Technical Briefs (SA '18). ACM, New York, NY, USA, Article 12, 3 pages. Google Scholar
Digital Library
- Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. 2014. Striving for Simplicity: The All Convolutional Net. CoRR abs/1412.6806 (2014). arXiv:1412.6806 http://arxiv.org/abs/1412.6806Google Scholar
- Manu Mathew Thomas and Angus Graeme Forbes. 2017. Deep Illumination: Approximating Dynamic Global Illumination with Generative Adversarial Network. CoRR abs/1710.09834 (2017). arXiv:1710.09834 http://arxiv.org/abs/1710.09834Google Scholar
- Zhirong Wu, S. Song, A. Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and J. Xiao. 2015. 3D ShapeNets: A deep representation for volumetric shapes. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1912--1920.Google Scholar
Index Terms
Deep Precomputed Radiance Transfer for Deformable Objects
Recommendations
Real-time global illumination by precomputed local reconstruction from sparse radiance probes
We present a direct-to-indirect transport technique that enables accurate real-time rendering of indirect illumination in mostly static scenes of complexity on par with modern games while supporting fully dynamic lights, cameras and diffuse surface ...
Local, deformable precomputed radiance transfer
Precomputed radiance transfer (PRT) captures realistic lighting effects from distant, low-frequency environmental lighting but has been limited to static models or precomputed sequences. We focus on PRT for local effects such as bumps, wrinkles, or ...
Local, deformable precomputed radiance transfer
SIGGRAPH '05: ACM SIGGRAPH 2005 PapersPrecomputed radiance transfer (PRT) captures realistic lighting effects from distant, low-frequency environmental lighting but has been limited to static models or precomputed sequences. We focus on PRT for local effects such as bumps, wrinkles, or ...






Comments