skip to main content
research-article

Neural Denoising for Path Tracing of Medical Volumetric Data

Published:26 August 2020Publication History
Skip Abstract Section

Abstract

In this paper, we transfer machine learning techniques previously applied to denoising surface-only Monte Carlo renderings to path-traced visualizations of medical volumetric data. In the domain of medical imaging, path-traced videos turned out to be an efficient means to visualize and understand internal structures, in particular for less experienced viewers such as students or patients. However, the computational demands for the rendering of high-quality path-traced videos are very high due to the large number of samples necessary for each pixel. To accelerate the process, we present a learning-based technique for denoising path-traced videos of volumetric data by increasing the sample count per pixel; both through spatial (integrating neighboring samples) and temporal filtering (reusing samples over time). Our approach uses a set of additional features and a loss function both specifically designed for the volumetric case. Furthermore, we present a novel network architecture tailored for our purpose, and introduce reprojection of samples to improve temporal stability and reuse samples over frames. As a result, we achieve good image quality even from severely undersampled input images, as visible in the teaser image.

Skip Supplemental Material Section

Supplemental Material

3406181.mp4

Presentation Video

References

  1. Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph. 36, 4 (2017), 97--1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Pablo Bauszat, Martin Eisemann, and Marcus Magnor. 2011. Guided image filtering for interactive high-quality global illumination. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 1361--1368.Google ScholarGoogle Scholar
  3. Benedikt Bitterli, Fabrice Rousselle, Bochang Moon, José A Iglesias-Guitián, David Adler, Kenny Mitchell, Wojciech Jarosz, and Jan Novák. 2016. Nonlinearly weighted first-order regression for denoising Monte Carlo renderings. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 107--117.Google ScholarGoogle Scholar
  4. Chakravarty R Alla Chaitanya, Anton S Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1--12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Per Christensen, Julian Fong, Jonathan Shade, Wayne Wooten, Brenden Schubert, Andrew Kensler, Stephen Friedman, Charlie Kilpatrick, Cliff Ramshaw, Marc Bannister, et al. 2018. Renderman: An advanced path-tracing architecture for movie rendering. ACM Transactions on Graphics (TOG) 37, 3 (2018), 1--21.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Dorin Comaniciu, Klaus Engel, Bogdan Georgescu, and Tommaso Mansi. 2016. Shaping the future through innovations: From medical imaging to precision medicine.Google ScholarGoogle Scholar
  7. Holger Dammertz, Daniel Sewtz, Johannes Hanika, and Hendrik PA Lensch. 2010. Edge-avoiding À-Trous wavelet transform for fast global illumination filtering. In Proceedings of the Conference on High Performance Graphics. Citeseer, 67--75.Google ScholarGoogle Scholar
  8. Evelyn Dappa, Kai Higashigaito, Jürgen Fornaro, Sebastian Leschka, Simon Wildermuth, and Hatem Alkadhi. 2016. Cinematic rendering -- an alternative to volume rendering for 3D computed tomography imaging. Insights into Imaging 7, 6 (2016), 849--856.Google ScholarGoogle ScholarCross RefCross Ref
  9. Laura Fink, Sing Chun Lee, Jie Ying Wu, Xingtong Liu, Tianyu Song, Yordanka Velikova, Marc Stamminger, Nassir Navab, and Mathias Unberath. 2019. LumiPath-Towards Real-Time Physically-Based Rendering on Embedded Devices. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 673--681.Google ScholarGoogle Scholar
  10. Iliyan Georgiev, Thiago Ize, Mike Farnsworth, Ramón Montoya-Vozmediano, Alan King, Brecht Van Lommel, Angel Jimenez, Oscar Anson, Shinji Ogaki, Eric Johnston, et al. 2018. Arnold: A brute-force production path tracer. ACM Transactions on Graphics (TOG) 37, 3 (2018), 1--12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Michaël Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Frédo Durand. 2019. Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Philip A Glemser, Klaus Engel, David Simons, Johann Steffens, Heinz-Peter Schlemmer, and Berk Orakcioglu. 2018. A new approach for photorealistic visualization of rendered computed tomography images. World neurosurgery 114 (2018), e283-e292.Google ScholarGoogle Scholar
  13. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.Google ScholarGoogle Scholar
  14. Jie Guo, Mengtian Li, Quewei Li, Yuting Qiang, Bingyang Hu, Yanwen Guo, and Ling-Qi Yan. 2019. GradNet: unsupervised deep screened poisson reconstruction for gradient-domain rendering. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--13.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Intel®. 2019. Open Image Denoise. https://openimagedenoise.github.io/index.htmlGoogle ScholarGoogle Scholar
  16. Henrik Wann Jensen. 2001. Realistic image synthesis using photon mapping. AK Peters/CRC Press.Google ScholarGoogle Scholar
  17. Alexia Jolicoeur-Martineau. 2018. The relativistic discriminator: a key element missing from standard GAN. arXiv preprint arXiv:1807.00734 (2018).Google ScholarGoogle Scholar
  18. Daniel Jönsson, Erik Sundén, Anders Ynnerman, and Timo Ropinski. 2012. A Survey of Volumetric Illumination Techniques for Interactive Volume Rendering. Computer Graphics Forum 33, 1 (2012), 27--51.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A machine learning approach for filtering Monte Carlo noise. ACM Trans. Graph. 34, 4 (2015), 122--1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Simon Kallweit, Thomas Müller, Brian Mcwilliams, Markus Gross, and Jan Novák. 2017. Deep scattering: Rendering atmospheric clouds with radiance-predicting neural networks. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1--11.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. B Karis. 2014. High quality temporal anti-aliasing. Advances in Real-Time Rendering for Games, SIGGRAPH Courses (2014).Google ScholarGoogle Scholar
  22. Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. 2019. Deep convolutional reconstruction for gradient-domain rendering. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Advances in neural information processing systems. 3581--3589.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Matias Koskela, Kalle Immonen, Markku Mäkitalo, Alessandro Foi, Timo Viitanen, Pekka Jääskeläinen, Heikki Kultala, and Jarmo Takala. 2019. Blockwise Multi-Order Feature Regression for Real-Time Path-Tracing Reconstruction. ACM Transactions on Graphics (TOG) 38, 5 (2019), 1--14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Jaroslav Krivanek and Pascal Gautron. 2009. Practical global illumination with irradiance caching. Synthesis lectures on computer graphics and animation 4, 1 (2009), 1--148.Google ScholarGoogle ScholarCross RefCross Ref
  26. Thomas Kroes, Frits H. Post, and Charl P. Botha. 2012. Exposure Render: An Interactive Photo-Realistic Volume Rendering Framework. PLOS ONE 7, 7 (07 2012), 1--10.Google ScholarGoogle Scholar
  27. Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. 2019. Noise2void-learning denoising from single noisy images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2129--2137.Google ScholarGoogle ScholarCross RefCross Ref
  28. Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2noise: Learning image restoration without clean data. arXiv preprint arXiv:1803.04189 (2018).Google ScholarGoogle Scholar
  29. Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chuang. 2012. SURE-based optimization for adaptive sampling and reconstruction. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1--9.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions on Graphics (TOG) 38, 4 (2019), 65.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, Vol. 30. 3.Google ScholarGoogle Scholar
  32. Jana Martschinke, Stefan Hartnagel, Benjamin Keinert, Klaus Engel, and Marc Stamminger. 2019. Adaptive Temporal Sampling for Volumetric Path Tracing of Medical Data. In Computer Graphics Forum, Vol. 38. Wiley Online Library.Google ScholarGoogle Scholar
  33. Soham Uday Mehta, Brandon Wang, Ravi Ramamoorthi, and Fredo Durand. 2013. Axis-aligned filtering for interactive physically-based diffuse indirect lighting. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1--12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Bochang Moon, Nathan Carr, and Sung-Eui Yoon. 2014. Adaptive rendering based on weighted local regression. ACM Transactions on Graphics (TOG) 33, 5 (2014), 1--14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Jan Novák, Iliyan Georgiev, Johannes Hanika, and Wojciech Jarosz. 2018. Monte Carlo methods for volumetric light transport simulation. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 551--576.Google ScholarGoogle Scholar
  36. Mikhail Panin and Sergey Nikolenko. 2019. Faster RPNN: Rendering Clouds with Latent Space Light Probes. In SIGGRAPH Asia 2019 Technical Briefs. 21--24.Google ScholarGoogle Scholar
  37. Steven G Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, et al. 2010. OptiX: a general purpose ray tracing engine. Acm transactions on graphics (tog) 29, 4 (2010), 1--13.Google ScholarGoogle Scholar
  38. Bui Tuong Phong. 1975. Illumination for Computer Generated Pictures. Commun. ACM 18, 6 (June 1975), 311--317. https://doi.org/10.1145/360825.360839Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).Google ScholarGoogle Scholar
  40. Saiprasad Ravishankar and Yoram Bresler. 2010. MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE transactions on medical imaging 30, 5 (2010), 1028--1041.Google ScholarGoogle Scholar
  41. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.Google ScholarGoogle ScholarCross RefCross Ref
  42. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2012. Adaptive rendering with non-local means filtering. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1--11.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust denoising using feature and color information. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 121--130.Google ScholarGoogle Scholar
  44. Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul Patney, Chakravarty R Alla Chaitanya, John Burgess, Shiqiu Liu, Carsten Dachsbacher, Aaron Lefohn, and Marco Salvi. 2017. Spatiotemporal variance-guided filtering: real-time reconstruction for path-traced global illumination. In Proceedings of High Performance Graphics. 1--12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Eric Veach and Leonidas Guibas. 1995a. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques. Springer, 145--167.Google ScholarGoogle Scholar
  46. Eric Veach and Leonidas J Guibas. 1995b. Optimally combining sampling techniques for Monte Carlo rendering. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 419--428.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. D Vicini, David Adler, Jan Novák, Fabrice Rousselle, and Brent Burley. 2019. Denoising Deep Monte Carlo Renderings. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 316--327.Google ScholarGoogle Scholar
  48. Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--15.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600--612.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Sebastian Weiss, Mengyu Chu, Nils Thuerey, and Rüdiger Westermann. 2019. Volumetric isosurface rendering with deep learning-based super-resolution. arXiv preprint arXiv.1906.06520 (2019).Google ScholarGoogle Scholar
  51. Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. 2016. Loss functions for image restoration with neural networks. IEEE Transactions on computational imaging 3, 1 (2016), 47--57.Google ScholarGoogle ScholarCross RefCross Ref
  52. Henning Zimmer, Fabrice Rousselle, Wenzel Jakob, Oliver Wang, David Adler, Wojciech Jarosz, Olga Sorkine-Hornung, and Alexander Sorkine-Hornung. 2015. Path-space motion estimation and decomposition for robust animation filtering. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 131--142.Google ScholarGoogle Scholar
  53. Matthias Zwicker, Wojciech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and S-E Yoon. 2015. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. In Computer graphics forum, Vol. 34. Wiley Online Library, 667--681.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Neural Denoising for Path Tracing of Medical Volumetric Data

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!