Andrea Weidlich
ABSTRACT
Compared to path tracing, spectral rendering is still often considered to be a niche application used mainly to produce optical wave effects like dispersion or diffraction. And while over the last years more and more people started exploring the potential of spectral image synthesis, it is still widely assumed to be only of importance in high-quality offline applications associated with long render times and high visual fidelity.
While it is certainly true that describing light interactions in a spectral way is a necessity for predictive rendering, its true potential goes far beyond that. Used correctly, not only will it guarantee colour fidelity, but it will also simplify workflows for all sorts of applications.
Wētā Digital's renderer Manuka showed that there is a place for a spectral renderer in a production environment and how workflows can be simplified if the whole pipeline adapts. Picking up from the course last year, we want to continue the discussion we started as we firmly believe that spectral data is the future in content production. The authors feel enthusiastic about more people being aware of the advantages that spectral rendering and spectral workflows bring and share the knowledge we gained over many years. The novel workflows emerged during the adaptation of spectral techniques at a number of large companies are introduced to a wide audience including technical directors, artists and researchers. However, while last year's course concentrated primarily on the algorithmic sides of spectral image synthesis, this year we want to focus on the practical aspects.
We will draw examples from virtual production, digital humans over spectral noise reduction to image grading, therefore showing the usage of spectral data enhancing each and every single part of the image pipeline.
- Catherine Cooksey, David Allen, and Benjamin Tsai. 2017. Reference Data Set of Human Skin Reflectance. Google Scholar
Cross Ref
- Weta Digital. 2022. Physlight. PhyslightGoogle Scholar
- R Donaldson. 1954. Spectrophotometry of fluorescent pigments. British Journal of Applied Physics 5, 6 (jun 1954), 210--214. Google ScholarCross Ref
- Luca Fascione, Johannes Hanika, Mark Leone, Marc Droske, Jorge Schwarzhaupt, Tomas Davidovic, Andrea Weidlich, and Johannes Meng. 2018. Manuka: A Batch-Shading Architecture for Spectral Path Tracing in Movie Production. ACM Transactions on Graphics 37 (08 2018), 1--18. Google ScholarDigital Library
- Alban Fichet, Romain Pacanowski, and Alexander Wilkie. 2021. An OpenEXR Layout for Spectral images. Journal of Computer Graphics Techniques (JCGT) 10, 3 (29 September 2021), 1--18. http://jcgt.org/published/0010/03/01/Google Scholar
- Qingqin Hua, Alban Fichet, and Alexander Wilkie. 2021. A Compact Representation for Fluorescent Spectral Data. In Eurographics Symposium on Rendering - DL-only Track, Adrien Bousseau and Morgan McGuire (Eds.). The Eurographics Association. Google ScholarCross Ref
- Longqian Huang, Ruichen Luo, Xu Liu, and Xiang Hao. 2022. Spectral imaging with deep learning. Light: Science & Applications 11 (03 2022), 61. Google ScholarCross Ref
- Wenzel Jakob and Johannes Hanika. 2019. A Low-Dimensional Function Space for Efficient Spectral Upsampling. Computer Graphics Forum (Proceedings of Eurographics) 38, 2 (March 2019).Google ScholarCross Ref
- Alisa Jung, A. Wilkie, J. Hanika, W. Jakob, and C. Dachsbacher. 2019. Wide Gamut Spectral Upsampling with Fluorescence. Computer Graphics Forum 38 (07 2019), 87--96. Google ScholarCross Ref
- Lars König, Alisa Jung, and Carsten Dachsbacher. 2020. Improving Spectral Upsampling with Fluorescence. In Workshop on Material Appearance Modeling, Reinhard Klein and Holly Rushmeier (Eds.). The Eurographics Association. Google ScholarCross Ref
- Tom Lister, Philip A. Wright, and Paul H. Chappell. 2012. Optical properties of human skin. Journal of Biomedical Optics 17, 9 (2012), 1 -- 15. Google ScholarCross Ref
- Ian Mallett and Cem Yuksel. 2019. Spectral Primary Decomposition for Rendering with sRGB Reflectance. In Eurographics Symposium on Rendering - DL-only and Industry Track, Tamy Boubekeur and Pradeep Sen (Eds.). The Eurographics Association. Google ScholarCross Ref
- Johannes Meng, Florian Simon, Johannes Hanika, and Carsten Dachsbacher. 2015. Physically meaningful rendering using tristimulus colours. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 31--40.Google Scholar
- Arash Mirhashemi. 2018. Introducing spectral moment features in analyzing the SpecTex hyperspectral texture database. Machine Vision and Applications 29 (04 2018). Google ScholarDigital Library
- Hisanari Otsu, Masafumi Yamamoto, and Toshiya Hachisuka. 2018. Reproducing spectral reflectances from tristimulus colours. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 370--381.Google Scholar
- Mark S. Peercy. 1993. Linear Color Representations for Full Speed Spectral Rendering. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (Anaheim, CA) (SIGGRAPH '93). Association for Computing Machinery, New York, NY, USA, 191--198. Google ScholarDigital Library
- Scyllarus. 2016. HSZ Data Format. https://scyllarus.data61.csiro.au/data/hsz-data-format/Google Scholar
- Brian Smits. 1999. An RGB-to-spectrum conversion for reflectances. Journal of Graphics Tools 4, 4 (1999), 11--22.Google ScholarDigital Library
- Yinlong Sun, M.S. Drew, and F.D. Fracchia. 1999. Representing spectral functions by a composite model of smooth and spiky components for efficient full-spectrum photorealism. In Proceedings Workshop on Photometric Modeling for Computer Vision and Graphics (Cat. No.PR00271). 4--11. Google ScholarCross Ref
- Lucia Tódová, Alexander Wilkie, and Luca Fascione. 2021. Moment-based Constrained Spectral Uplifting. In Eurographics Symposium on Rendering - DL-only Track, Adrien Bousseau and Morgan McGuire (Eds.). The Eurographics Association. Google ScholarCross Ref
- Mark van de Ruit and Elmar Eisemann. 2021. A multi-pass method for accelerated spectral sampling. Computer Graphics Forum 40 (2021).Google Scholar
- Andrea Weidlich, Alex Forsythe, Scott Dyer, Thomas Mansencal, Johannes Hanika, Alexander Wilkie, Luke Emrose, and Anders Langlands. 2021. Spectral Imaging in Production: Course Notes Siggraph 2021. In ACM SIGGRAPH 2021 Courses (Virtual Event, USA) (SIGGRAPH '21). Association for Computing Machinery, New York, NY, USA, Article 14, 90 pages. Google ScholarDigital Library
- Rex West, Iliyan Georgiev, Adrien Gruson, and Toshiya Hachisuka. 2020. Continuous Multiple Importance Sampling. ACM Trans. Graph. 39, 4, Article 136 (jul 2020), 12 pages. Google ScholarDigital Library
- A. Wilkie, S. Nawaz, M. Droske, A. Weidlich, and J. Hanika. 2014. Hero Wavelength Spectral Sampling. In Proceedings of the 25th Eurographics Symposium on Rendering (Lyon, France) (EGSR '14). Eurographics Association, Goslar, DEU, 123--131.Google Scholar
- Alexander Wilkie, Robert Tobler, and Werner Purgathofer. 2000. Raytracing of Dispersion Effects in Transparent Materials.Google Scholar
- Chris Wyman, Peter-Pike Sloan, and Peter Shirley. 2013. Simple Analytic Approximations to the CIE XYZ Color Matching Functions. Journal of Computer Graphics Techniques (JCGT) 2, 2 (12 July 2013), 1--11. http://jcgt.org/published/0002/02/01/Google Scholar
- X-Rite. 2022. AxF Data Format. https://www.xrite.com/appearance-exchange-format-axfGoogle Scholar
- [n.d.]. In-camera VFX Camera Color Calibration. https://docs.unrealengine.com/4.27/en-US/WorkingWithMedia/IntegratingMedia/InCameraVFX/InCameraVFXCameraCalibration/Google Scholar
- Carlos F Borges. 1991. Trichromatic approximation for computer graphics illumination models. In Proceedings of the 18th annual conference on Computer graphics and interactive techniques (SIGGRAPH '91). 101--104.Google ScholarDigital Library
- Paul Debevec. 1998. Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-Based Graphics with Global Illumination and High Dynamic Range Photography. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '98). Association for Computing Machinery, New York, NY, USA, 189--198. Google ScholarDigital Library
- Paul Debevec, Andreas Wenger, Chris Tchou, Andrew Gardner, Jamie Waese, and Tim Hawkins. 2002. A Lighting Reproduction Approach to Live-Action Compositing. ACM Trans. Graph. 21, 3 (Jul 2002), 547--556. Google ScholarDigital Library
- Jonathan Erland. 2011. Chromatic Chaos: Implications of Newly Introduced Forms of Stage-light. National Association of Broadcasters (NAB) Conference.Google Scholar
- Nahum Gat and CA Torrance. 1998. Real-time multi-and hyper-spectral imaging for remote sensing and machine vision: an overview. In Proc. 1998 ASAE Annual International Mtg. Citeseer.Google Scholar
- Pierre-Loïc Hamon, James Harmer, Stuart Penn, and Nicolas Scapel. 2014. Gravity: Motion control and face integration. In ACM SIGGRAPH 2014 Talks. 1--1.Google ScholarDigital Library
- Jay Holben. 2020. The Mandalorian: This Is the Way. https://ascmag.com/articles/the-mandalorianGoogle Scholar
- Jack Holm, Tom Maier, Paul Debevec, Chloe LeGendre, Joshua Pines, Jonathan Erland, George Joblove, Scott Dyer, Blake Sloan, Joe di Gennaro, et al. 2016. A cinematographic spectral similarity index. In SMPTE 2016 Annual Technical Conference and Exhibition. SMPTE, 1--36.Google ScholarCross Ref
- Jun Jiang, Dengyu Liu, Jinwei Gu, and Sabine Süsstrunk. 2013. What is the space of spectral sensitivity functions for digital color cameras?. In 2013 IEEE Workshop on Applications of Computer Vision (WACV). IEEE, 168--179.Google ScholarDigital Library
- Noah Kadner. 2021a. 1899 Wraps Innovative Virtual Production. https://ascmag.com/articles/1899-wraps-virtual-productionGoogle Scholar
- Noah Kadner. 2021b. Color Fidelity in LED Volumes. https://ascmag.com/articles/color-fidelity-in-led-volumesGoogle Scholar
- Noah Kadner. 2021c. On The Walls: Virtual Production for Series Shooting. https://ascmag.com/articles/on-the-wallsGoogle Scholar
- Chloe LeGendre, Lukas Lepicovsky, and Paul Debevec. 2022. Jointly Optimizing Color Rendition and In-Camera Backgrounds in an RGB Virtual Production Stage. (arXiv.org) (2022).Google Scholar
- Chloe LeGendre, Xueming Yu, and Paul Debevec. 2017. Optimal LED selection for multi-spectral lighting reproduction. Electronic Imaging 2017, 8 (2017), 25--32.Google ScholarCross Ref
- Chloe LeGendre, Xueming Yu, Dai Liu, Jay Busch, Andrew Jones, Sumanta Pattanaik, and Paul Debevec. 2016. Practical multispectral lighting reproduction. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--11.Google ScholarDigital Library
- Calvin S McCamy, Harold Marcus, James G Davidson, et al. 1976. A color-rendition chart. J. App. Photog. Eng 2, 3 (1976), 95--99.Google Scholar
- Muhammad Uzair, Arif Mahmood, Faisal Shafait, Christian Nansen, and Ajmal Mian. 2015. Is spectral reflectance of the face a reliable biometric? Optics express 23, 12 (2015), 15160--15173.Google Scholar
- Andrea Weidlich, Alex Forsythe, Scott Dyer, Thomas Mansencal, Johannes Hanika, Alexander Wilkie, Luke Emrose, and Anders Langlands. 2021. Spectral imaging in production: course notes Siggraph 2021. In ACM SIGGRAPH 2021 Courses. 1--90.Google ScholarDigital Library
- Andreas Wenger, Tim Hawkins, and Paul Debevec. 2003. Optimizing color matching in a lighting reproduction system for complex subject and illuminant spectra. In Rendering Techniques. Citeseer, 249--259.Google Scholar
- Carlos Aliaga, Christophe Hery, and Mengqi Xia. 2022. Estimation of Spectral Biophysical Skin Properties from Captured RGB Albedo.Google Scholar
- Sarah Alotaibi and William AP Smith. 2017. A biophysical 3D morphable model of face appearance. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 824--832.Google ScholarCross Ref
- Gladimir VG Baranoski and Aravind Krishnaswamy. 2010. Light and skin interactions: simulations for computer graphics applications. Morgan Kaufmann.Google Scholar
- Alexey N Bashkatov, Elina A Genina, and Valery V Tuchin. 2011. Optical properties of skin, subcutaneous, and muscle tissues: a review. Journal of Innovative Optical Health Sciences 4, 01 (2011), 9--38.Google ScholarCross Ref
- Benedikt Bitterli, Srinath Ravichandran, Thomas Müller, Magnus Wrenninge, Jan Novák, Steve Marschner, and Wojciech Jarosz. 2018. A Radiative Transfer Framework for Non-Exponential Media. ACM Trans. Graph. 37, 6, Article 225 (Dec. 2018), 17 pages.Google ScholarDigital Library
- Tenn F Chen, Gladimir VG Baranoski, Bradley W Kimmel, and Erik Miranda. 2015. Hyper-spectral modeling of skin appearance. ACM Transactions on Graphics (TOG) 34, 3 (2015), 1--14.Google ScholarDigital Library
- Blender Online Community. 2020. Blender: a 3D modelling and rendering package. (2020). http://www.blender.orgGoogle Scholar
- Carolin Czekalla, Karl Heinz Schönborn, Jürgen Lademann, and Martina C Meinke. 2019. Noninvasive determination of epidermal and stratum corneum thickness in vivo using two-photon microscopy and optical coherence tomography: Impact of body area, age, and gender. Skin pharmacology and physiology 32, 3 (2019), 142--150.Google Scholar
- Philip E Dennison, Kerry Q Halligan, and Dar A Roberts. 2004. A comparison of error metrics and constraints for multiple endmember spectral mixture analysis and spectral angle mapper. Remote Sensing of Environment 93, 3 (2004), 359--367.Google ScholarCross Ref
- Craig Donner and Henrik Wann Jensen. 2006. A Spectral BSSRDF for Shading Human Skin. Rendering techniques 2006 (2006), 409--418.Google Scholar
- Eugene d' Eon. 2019. A Reciprocal Formulation of Nonexponential Radiative Transfer. 2: Monte Carlo Estimation and Diffusion Approximation. Journal of Computational and Theoretical Transport 48, 6 (Sep 2019), 201--262.Google Scholar
- Eugene d'Eon. 2021. A Hitchhiker's Guide to Multiple Scattering. (2021).Google Scholar
- Thomas B Fitzpatrick. 1988. The validity and practicality of sun-reactive skin types I through VI. Archives of dermatology 124, 6 (1988), 869--871.Google Scholar
- Lou Gevaux, Cyprien Adnet, Pierre Séroul, Raphael Clerc, Alain Trémeau, Jean Luc Perrot, and Mathieu Hébert. 2019. Three-dimensional maps of human skin properties on full face with shadows using 3-D hyperspectral imaging. Journal of biomedical optics 24, 6 (2019), 066002.Google ScholarCross Ref
- Lou Gevaux, Jordan Gierschendorf, Juliette Rengot, Marie Cherel, Pierre Séroul, Alex Nkengne, Julie Robic, Alain Trémeau, and Mathieu Hébert. 2021. Real-time skin chromophore estimation from hyperspectral images using a neural network. Skin Research and Technology 27, 2 (2021), 163--177.Google ScholarCross Ref
- Yuliya Gitlina, Giuseppe Claudio Guarnera, Daljit Singh Dhillon, Jan Hansen, Alexandros Lattas, Dinesh Pai, and Abhijeet Ghosh. 2020. Practical Measurement and Reconstruction of Spectral Skin Reflectance. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 75--89.Google Scholar
- Louis G Henyey and Jesse L Greenstein. 1941. Diffuse radiation in the galaxy. The Astrophysical Journal 93 (1941), 70--83.Google ScholarCross Ref
- Jose A. Iglesias-Guitian, Carlos Aliaga, Adrian Jarabo, and Diego Gutierrez. 2015. A Biophysically-Based Model of the Optical Properties of Skin Aging. Computer Graphics Forum (EUROGRAPHICS 2015) 34, 2 (2015).Google Scholar
- Steven L Jacques. 2013. Optical properties of biological tissues: a review. Physics in Medicine & Biology 58, 11 (2013), R37.Google ScholarCross Ref
- Steven L Jacques and Daniel J McAuliffe. 1991. The melanosome: threshold temperature for explosive vaporization and internal absorption coefficient during pulsed laser irradiation. Photochemistry and photobiology 53, 6 (1991), 769--775.Google Scholar
- Wenzel Jakob and Johannes Hanika. 2019. A Low-Dimensional Function Space for Efficient Spectral Upsampling. Computer Graphics Forum (Proceedings of Eurographics) 38, 2 (March 2019).Google ScholarCross Ref
- Adrian Jarabo, Carlos Aliaga, and Diego Gutierrez. 2018. A radiative transfer framework for spatially-correlated materials. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--13.Google ScholarDigital Library
- Nirmal Keshava and John F Mustard. 2002. Spectral unmixing. IEEE signal processing magazine 19, 1 (2002), 44--57.Google ScholarCross Ref
- Aravind Krishnaswamy and Gladimir VG Baranoski. 2004. A biophysically-based spectral model of light interaction with human skin. In Computer Graphics Forum, Vol. 23. Wiley Online Library, 331--340.Google Scholar
- Tom Lister, Philip A. Wright, and Paul H. Chappell. 2012. Optical properties of human skin. Journal of Biomedical Optics 17, 9 (2012), 1 -- 15.Google ScholarCross Ref
- Ian Mallett and Cem Yuksel. 2019. Spectral Primary Decomposition for Rendering with RGB Reflectance. In Eurographics Symposium on Rendering (EGSR 2019) (Strasbourg, France). The Eurographics Association.Google Scholar
- Igor V Meglinski and Stephen J Matcher. 2002. Quantitative assessment of skin layers absorption and skin reflectance spectra simulation in the visible and near-infrared spectral regions. Physiological measurement 23, 4 (2002), 741.Google Scholar
- Scott Prahl and Steven L Jacques. [n.d.]. Tabulated Molar Extinction Coefficient for Hemoglobin in Water. https://omlc.org/spectra/hemoglobin/summary.html.Google Scholar
- Iyad Salam Saidi. 1992. Transcutaneous optical measurement of hyperbilirubinemia in neonates. Ph.D. Dissertation.Google Scholar
- Tadeusz Sarna and Harold A Swartz. 2006. The physical properties of melanins. The pigmentary system: physiology and pathophysiology (2006), 311--341.Google Scholar
- MJC Van Gemert, Steven L Jacques, HJCM Sterenborg, and WM Star. 1989. Skin optics. IEEE Transactions on biomedical engineering 36, 12 (1989), 1146--1154.Google ScholarCross Ref
- Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Micro-facet Models for Refraction through Rough Surfaces. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (Grenoble, France) (EGSR'07). Eurographics Association, Goslar, DEU, 195--206.Google Scholar
- Magnus Wrenninge, Ryusuke Villemin, and Christophe Hery. 2017. Path traced subsurface scattering using anisotropic phase functions and non-exponential free flights. Technical Report. Technical Memo.Google Scholar
- Evgeny Zherebtsov, Viktor Dremin, Alexey Popov, Alexander Doronin, Daria Kurakina, Mikhail Kirillin, Igor Meglinski, and Alexander Bykov. 2019. Hyperspectral imaging of human skin aided by artificial neural networks. Biomedical optics express 10, 7 (2019), 3545--3559.Google Scholar
- George Zonios, Julie Bykowski, and Nikiforos Kollias. 2001. Skin melanin, hemoglobin, and light scattering properties can be quantitatively assessed in vivo using diffuse reflectance spectroscopy. Journal of Investigative Dermatology 117, 6 (2001), 1452--1457.Google ScholarCross Ref
- Benedikt Bitterli, Chris Wyman, Matt Pharr, Peter Shirley, Aaron Lefohn, and Wojciech Jarosz. 2020. Spatiotemporal reservoir resampling for real-time ray tracing with dynamic direct lighting. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020). Google ScholarCross Ref
- Guillaume Boissé. 2021. World-space spatiotemporal reservoir reuse for ray-traced Global illumination. Association for Computing Machinery, New York, NY, USA. Google ScholarDigital Library
- Jakub Boksansky, Paula Jukarainen, and Chris Wyman. 2021. Rendering Many Lights with Grid-Based Reservoirs. Apress, Berkeley, CA, 351--365. Google ScholarCross Ref
- Daqi Lin, Markus Kettunen, Benedikt Bitterli, Jacopo Pantaleoni, Cem Yuksel, and Chris Wyman. 2022. Generalized Resampled Importance Sampling: Foundations of ReSTIR. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2022) 41, 4, Article 75 (07 2022), 23 pages. Google ScholarDigital Library
- Daqi Lin, Chris Wyman, and Cem Yuksel. 2021. Fast Volume Rendering with Spatiotemporal Reservoir Resampling. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2021) 40, 6, Article 279 (dec 2021), 18 pages. Google ScholarDigital Library
- Y. Ouyang, S. Liu, M. Kettunen, M. Pharr, and Jacopo Pantaleoni. 2021. ReSTIR GI: Path Resampling for Real-Time Path Tracing. Computer Graphics Forum 40 (12 2021), 17--29. Google ScholarCross Ref
- Justin Talbot, David Cline, and Parris Egbert. 2005. Importance Resampling for Global Illumination. Eurographics Symposium on Rendering, 139--146. Google ScholarCross Ref
- M. van de Ruit and E. Eisemann. 2021. A multi-pass method for accelerated spectral sampling. Computer Graphics Forum 40, 7 (2021), 141--148. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14408 Google ScholarCross Ref
- Rex West, Iliyan Georgiev, Adrien Gruson, and Toshiya Hachisuka. 2020. Continuous Multiple Importance Sampling. ACM Trans. Graph. 39, 4, Article 136 (July 2020), 12 pages. Google ScholarDigital Library
- A Wilkie, S Nawaz, Marc Droske, A Weidlich, and J Hanika. 2014. Hero Wavelength Spectral Sampling. Computer Graphics Forum 33 (07 2014).Google Scholar
Recommendations
Spectral imaging in production: course notes Siggraph 2021
SIGGRAPH '21: ACM SIGGRAPH 2021 CoursesUnlike path tracing, spectral rendering is still not widely used in production although it has been around for more than thirty years. Traditionally connected to spectral effects such as dispersion and interference, spectral rendering - and, more ...
Practical Spectral Photography
We introduce a low-cost and compact spectral imaging camera design based on unmodified consumer cameras and a custom camera objective. The device can be used in a high-resolution configuration that measures the spectrum of a column of an imaged scene ...
Content-Independent Multi-Spectral Display Using Superimposed Projections
Many works focus on multi-spectral capture and analysis, but multi-spectral display still remains a challenge. Most prior works on multi-primary displays use ad-hoc narrow band primaries that assure a larger color gamut, but cannot assure a good ...
Comments