Abstract
In this paper, we transfer machine learning techniques previously applied to denoising surface-only Monte Carlo renderings to path-traced visualizations of medical volumetric data. In the domain of medical imaging, path-traced videos turned out to be an efficient means to visualize and understand internal structures, in particular for less experienced viewers such as students or patients. However, the computational demands for the rendering of high-quality path-traced videos are very high due to the large number of samples necessary for each pixel. To accelerate the process, we present a learning-based technique for denoising path-traced videos of volumetric data by increasing the sample count per pixel; both through spatial (integrating neighboring samples) and temporal filtering (reusing samples over time). Our approach uses a set of additional features and a loss function both specifically designed for the volumetric case. Furthermore, we present a novel network architecture tailored for our purpose, and introduce reprojection of samples to improve temporal stability and reuse samples over frames. As a result, we achieve good image quality even from severely undersampled input images, as visible in the teaser image.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Neural Denoising for Path Tracing of Medical Volumetric Data
- Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph. 36, 4 (2017), 97--1.Google Scholar
Digital Library
- Pablo Bauszat, Martin Eisemann, and Marcus Magnor. 2011. Guided image filtering for interactive high-quality global illumination. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 1361--1368.Google Scholar
- Benedikt Bitterli, Fabrice Rousselle, Bochang Moon, José A Iglesias-Guitián, David Adler, Kenny Mitchell, Wojciech Jarosz, and Jan Novák. 2016. Nonlinearly weighted first-order regression for denoising Monte Carlo renderings. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 107--117.Google Scholar
- Chakravarty R Alla Chaitanya, Anton S Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1--12.Google Scholar
Digital Library
- Per Christensen, Julian Fong, Jonathan Shade, Wayne Wooten, Brenden Schubert, Andrew Kensler, Stephen Friedman, Charlie Kilpatrick, Cliff Ramshaw, Marc Bannister, et al. 2018. Renderman: An advanced path-tracing architecture for movie rendering. ACM Transactions on Graphics (TOG) 37, 3 (2018), 1--21.Google Scholar
Digital Library
- Dorin Comaniciu, Klaus Engel, Bogdan Georgescu, and Tommaso Mansi. 2016. Shaping the future through innovations: From medical imaging to precision medicine.Google Scholar
- Holger Dammertz, Daniel Sewtz, Johannes Hanika, and Hendrik PA Lensch. 2010. Edge-avoiding À-Trous wavelet transform for fast global illumination filtering. In Proceedings of the Conference on High Performance Graphics. Citeseer, 67--75.Google Scholar
- Evelyn Dappa, Kai Higashigaito, Jürgen Fornaro, Sebastian Leschka, Simon Wildermuth, and Hatem Alkadhi. 2016. Cinematic rendering -- an alternative to volume rendering for 3D computed tomography imaging. Insights into Imaging 7, 6 (2016), 849--856.Google Scholar
Cross Ref
- Laura Fink, Sing Chun Lee, Jie Ying Wu, Xingtong Liu, Tianyu Song, Yordanka Velikova, Marc Stamminger, Nassir Navab, and Mathias Unberath. 2019. LumiPath-Towards Real-Time Physically-Based Rendering on Embedded Devices. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 673--681.Google Scholar
- Iliyan Georgiev, Thiago Ize, Mike Farnsworth, Ramón Montoya-Vozmediano, Alan King, Brecht Van Lommel, Angel Jimenez, Oscar Anson, Shinji Ogaki, Eric Johnston, et al. 2018. Arnold: A brute-force production path tracer. ACM Transactions on Graphics (TOG) 37, 3 (2018), 1--12.Google Scholar
Digital Library
- Michaël Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Frédo Durand. 2019. Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.Google Scholar
Digital Library
- Philip A Glemser, Klaus Engel, David Simons, Johann Steffens, Heinz-Peter Schlemmer, and Berk Orakcioglu. 2018. A new approach for photorealistic visualization of rendered computed tomography images. World neurosurgery 114 (2018), e283-e292.Google Scholar
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.Google Scholar
- Jie Guo, Mengtian Li, Quewei Li, Yuting Qiang, Bingyang Hu, Yanwen Guo, and Ling-Qi Yan. 2019. GradNet: unsupervised deep screened poisson reconstruction for gradient-domain rendering. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--13.Google Scholar
Digital Library
- Intel®. 2019. Open Image Denoise. https://openimagedenoise.github.io/index.htmlGoogle Scholar
- Henrik Wann Jensen. 2001. Realistic image synthesis using photon mapping. AK Peters/CRC Press.Google Scholar
- Alexia Jolicoeur-Martineau. 2018. The relativistic discriminator: a key element missing from standard GAN. arXiv preprint arXiv:1807.00734 (2018).Google Scholar
- Daniel Jönsson, Erik Sundén, Anders Ynnerman, and Timo Ropinski. 2012. A Survey of Volumetric Illumination Techniques for Interactive Volume Rendering. Computer Graphics Forum 33, 1 (2012), 27--51.Google Scholar
Digital Library
- Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A machine learning approach for filtering Monte Carlo noise. ACM Trans. Graph. 34, 4 (2015), 122--1.Google Scholar
Digital Library
- Simon Kallweit, Thomas Müller, Brian Mcwilliams, Markus Gross, and Jan Novák. 2017. Deep scattering: Rendering atmospheric clouds with radiance-predicting neural networks. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1--11.Google Scholar
Digital Library
- B Karis. 2014. High quality temporal anti-aliasing. Advances in Real-Time Rendering for Games, SIGGRAPH Courses (2014).Google Scholar
- Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. 2019. Deep convolutional reconstruction for gradient-domain rendering. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.Google Scholar
Digital Library
- Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Advances in neural information processing systems. 3581--3589.Google Scholar
Digital Library
- Matias Koskela, Kalle Immonen, Markku Mäkitalo, Alessandro Foi, Timo Viitanen, Pekka Jääskeläinen, Heikki Kultala, and Jarmo Takala. 2019. Blockwise Multi-Order Feature Regression for Real-Time Path-Tracing Reconstruction. ACM Transactions on Graphics (TOG) 38, 5 (2019), 1--14.Google Scholar
Digital Library
- Jaroslav Krivanek and Pascal Gautron. 2009. Practical global illumination with irradiance caching. Synthesis lectures on computer graphics and animation 4, 1 (2009), 1--148.Google Scholar
Cross Ref
- Thomas Kroes, Frits H. Post, and Charl P. Botha. 2012. Exposure Render: An Interactive Photo-Realistic Volume Rendering Framework. PLOS ONE 7, 7 (07 2012), 1--10.Google Scholar
- Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. 2019. Noise2void-learning denoising from single noisy images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2129--2137.Google Scholar
Cross Ref
- Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2noise: Learning image restoration without clean data. arXiv preprint arXiv:1803.04189 (2018).Google Scholar
- Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chuang. 2012. SURE-based optimization for adaptive sampling and reconstruction. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1--9.Google Scholar
Digital Library
- Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions on Graphics (TOG) 38, 4 (2019), 65.Google Scholar
Digital Library
- Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, Vol. 30. 3.Google Scholar
- Jana Martschinke, Stefan Hartnagel, Benjamin Keinert, Klaus Engel, and Marc Stamminger. 2019. Adaptive Temporal Sampling for Volumetric Path Tracing of Medical Data. In Computer Graphics Forum, Vol. 38. Wiley Online Library.Google Scholar
- Soham Uday Mehta, Brandon Wang, Ravi Ramamoorthi, and Fredo Durand. 2013. Axis-aligned filtering for interactive physically-based diffuse indirect lighting. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1--12.Google Scholar
Digital Library
- Bochang Moon, Nathan Carr, and Sung-Eui Yoon. 2014. Adaptive rendering based on weighted local regression. ACM Transactions on Graphics (TOG) 33, 5 (2014), 1--14.Google Scholar
Digital Library
- Jan Novák, Iliyan Georgiev, Johannes Hanika, and Wojciech Jarosz. 2018. Monte Carlo methods for volumetric light transport simulation. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 551--576.Google Scholar
- Mikhail Panin and Sergey Nikolenko. 2019. Faster RPNN: Rendering Clouds with Latent Space Light Probes. In SIGGRAPH Asia 2019 Technical Briefs. 21--24.Google Scholar
- Steven G Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, et al. 2010. OptiX: a general purpose ray tracing engine. Acm transactions on graphics (tog) 29, 4 (2010), 1--13.Google Scholar
- Bui Tuong Phong. 1975. Illumination for Computer Generated Pictures. Commun. ACM 18, 6 (June 1975), 311--317. https://doi.org/10.1145/360825.360839Google Scholar
Digital Library
- Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).Google Scholar
- Saiprasad Ravishankar and Yoram Bresler. 2010. MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE transactions on medical imaging 30, 5 (2010), 1028--1041.Google Scholar
- Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.Google Scholar
Cross Ref
- Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2012. Adaptive rendering with non-local means filtering. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1--11.Google Scholar
Digital Library
- Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust denoising using feature and color information. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 121--130.Google Scholar
- Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul Patney, Chakravarty R Alla Chaitanya, John Burgess, Shiqiu Liu, Carsten Dachsbacher, Aaron Lefohn, and Marco Salvi. 2017. Spatiotemporal variance-guided filtering: real-time reconstruction for path-traced global illumination. In Proceedings of High Performance Graphics. 1--12.Google Scholar
Digital Library
- Eric Veach and Leonidas Guibas. 1995a. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques. Springer, 145--167.Google Scholar
- Eric Veach and Leonidas J Guibas. 1995b. Optimally combining sampling techniques for Monte Carlo rendering. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 419--428.Google Scholar
Digital Library
- D Vicini, David Adler, Jan Novák, Fabrice Rousselle, and Brent Burley. 2019. Denoising Deep Monte Carlo Renderings. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 316--327.Google Scholar
- Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--15.Google Scholar
Digital Library
- Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600--612.Google Scholar
Digital Library
- Sebastian Weiss, Mengyu Chu, Nils Thuerey, and Rüdiger Westermann. 2019. Volumetric isosurface rendering with deep learning-based super-resolution. arXiv preprint arXiv.1906.06520 (2019).Google Scholar
- Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. 2016. Loss functions for image restoration with neural networks. IEEE Transactions on computational imaging 3, 1 (2016), 47--57.Google Scholar
Cross Ref
- Henning Zimmer, Fabrice Rousselle, Wenzel Jakob, Oliver Wang, David Adler, Wojciech Jarosz, Olga Sorkine-Hornung, and Alexander Sorkine-Hornung. 2015. Path-space motion estimation and decomposition for robust animation filtering. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 131--142.Google Scholar
- Matthias Zwicker, Wojciech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and S-E Yoon. 2015. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. In Computer graphics forum, Vol. 34. Wiley Online Library, 667--681.Google Scholar
Digital Library
Index Terms
Neural Denoising for Path Tracing of Medical Volumetric Data
Recommendations
Interactive Path Tracing and Reconstruction of Sparse Volumes
We combine state-of-the-art techniques into a system for high-quality, interactive rendering of participating media. We leverage unbiased volume path tracing with multiple scattering, temporally stable neural denoising and NanoVDB [Museth 2021], a fast, ...
A Hardware Acceleration Method for Volumetric Ray Tracing
VIS '95: Proceedings of the 6th conference on Visualization '95In this paper we present an acceleration method for volumetric ray tracing which utilizes standard graphics hardware without compromising image accuracy. The graphics hardware is employed to identify those segments of each ray that could possibly ...
Rendering forest scenes in real-time
EGSR'04: Proceedings of the Fifteenth Eurographics conference on Rendering TechniquesForests are crucial for scene realism in applications such as light simulators. This paper proposes a new representation allowing for the real-time rendering of realistic forests covering an arbitrary terrain. It lets us produce dense forests ...






Comments