Spectral imaging in production: course notes Siggraph 2021

ABSTRACT

Unlike path tracing, spectral rendering is still not widely used in production although it has been around for more than thirty years. Traditionally connected to spectral effects such as dispersion and interference, spectral rendering - and, more importantly, the use of spectral data in general - is predominantly a way to guarantee colour fidelity. Additionally, with the rise of path tracing and the growing use of LED lights on-set as well as the recent shift to LED walls in virtual production, it becomes increasingly evident that the traditional way of seeing colour and light as RGB triplets is insufficient if colour accuracy is required.

The purpose of the course is two-fold. First and foremost, we want to share what we learned on our way towards a spectral image pipeline. We will talk about the unique opportunities and challenges the use of spectral data brings in a modern production pipeline and our motivation to build a spectral renderer. Since spectral data influences every step of the pipeline, the course will go beyond rendering aspects. We will discuss data acquisition and will shed some light on how to tackle the special problem of LED lights in production as well as its practical usage.

The second aim of the course is to build a community. We want to see the topic evolve over the next few years and connect people to shape the future together until spectral imaging is as ubiquitous as path tracing is in production.

References

1. Steven Bergner, Mark S. Drew, and Torsten Möller. 2009. A Tool to Create Illuminant and Reflectance Spectra for Light-Driven Graphics and Visualization. ACM Trans. Graph. 28, 1, Article 5 (Feb. 2009), 11 pages.
2. Luca Fascione, Johannes Hanika, Mark Leone, Marc Droske, Jorge Schwarzhaupt, Tomas Davidovic, Andrea Weidlich, and Johannes Meng. 2018. Manuka: A Batch-Shading Architecture for Spectral Path Tracing in Movie Production. ACM Transactions on Graphics 37 (08 2018), 1--18.
3. Wenzel Jakob and Johannes Hanika. 2019. A Low?Dimensional Function Space for Efficient Spectral Upsampling. Computer Graphics Forum 38 (05 2019), 147--155.
4. Christoph Peters, Sebastian Merzbach, Johannes Hanika, and Carsten Dachsbacher. 2019. Using Moments to Represent Bounded Signals for Spectral Rendering. ACM Trans. Graph. 38, 4, Article 136 (July 2019), 14 pages.
5. Brian Smits. 2000. An RGB to Spectrum Conversion for Reflectances. Journal of Graphics Tools 4 (06 2000).
6. Guenter Wyszecki and W.S. Stiles. 1982. Color science : concepts and methods, quantitative data and formulae (2nd ed. ed.). Wiley, New York.
7. Academy 2013. Procedure P-2013-001 - Recommended Procedures for the Creation and Use of Digital Camera System Input Device Transforms (IDTs). Procedure. The Academy of Motion Picture Arts and Sciences, Science and Technology Council, & Academy Color Encoding System (ACES) Project Subcommittee. http://j.mp/P-2013-001
8. Graham D Finlayson and Garrett M Johnson. 2016. Extended Linear Color Correction. In Color and Imaging Conference, Vol. 2016. Society for Imaging Science and Technology, 168--173.
9. Graham D Finlayson, Michal Mackiewicz, and Anya Hurlbert. 2015. Color correction using root-polynomial regression. IEEE Transactions on Image Processing 24, 5 (2015), 1460--1470.
10. ITU BT.709-6 2015. Parameter values for the HDTV standards for production and international programme exchange. Recommendation. International Telecommunications Union.
11. Hideyasu Kuniba and Roy S Berns. 2009. Spectral sensitivity optimization of color image sensors considering photon shot noise. Journal of Electronic Imaging 18, 2 (2009), 023002.
12. Jon S McElvain and Walter Gish. 2013. Camera color correction using two-dimensional transforms. In Color and Imaging Conference, Vol. 2013. Society for Imaging Science and Technology, 250--256.
13. Noboru Ohta and Alan R Robertson. 2005. Colorimetry: Fundamentals and Applications. Vol. 10. John Wiley & Sons, Ltd, Chichester, UK., Chapter Measurement and calculation of colorimetric values, 153--174.
14. SMPTE ST2065-1 2021. Academy Color Encoding Specification (ACES). Standard. Society of Motion Picture and Television Engineers.
15. Kevin E Spaulding, Geoffrey J Woolfe, and Edward J Giorgianni. 2000. `Reference input/output medium metric RGB color encodings. Proc. IS&T PICS (2000), 155--163.
16. A Stockman and Brainard. 2010. The Optical Society of America Handbook of Optics, Volume III: Vision and Vision Optics. Vol. Volume III: Vision and Vision Optics. McGraw Hill, New York, Chapter Colorimetry.
17. Andrew Stockman, Donald I. A. MacLeod, and Nancy E. Johnson. 1993. Spectral sensitivities of the human cones. J. Opt. Soc. Am. A 10, 12 (Dec 1993), 2491--2521.
18. R Von Luther. 1927. Aus dem gebiet der farbreizmetrik. Zeitschrift fur Technishe Physik 12 (1927), 540--558.
19. G. Wyszecki and W.S. Stiles. 2000. Color Science: Concepts and Methods, Quantitative Data and Formulae. Wiley.
20. ANSI and IES Color Committee. 2018. ANSI/IES TM-30-18 - IES Method for Evaluating Light Source Color Rendition. ANSI/IES.
21. CIE. 1987. 17-22-107 colour rendering, <of a light source>. http://cie.co.at/eilvterm/17-22-107
22. CIE Division 1.1995. CIE 013.3-1995 Method of Measuring and Specifying Colour Rendering of Light Sources.
23. CIE TC 1-90. 2017. CIE 2017 colour fidelity index for accurate scientific use. Number 224 in Technical report / CIE. CIE Central Bureau, Vienna.
24. Wendy Davis and Yoshiro Ohno. 2010. Color quality scale. Optical Engineering 49, 3 (March 2010), 033602.
25. Kevin A. G. Smet, Aurelien David, and Lorne Whitehead. 2016. Why Color Space Uniformity and Sample Set Spectral Uniformity Are Essential for Color Rendering Measures. LEUKOS 12, 1-2 (April 2016), 39--50.
26. The Academy of Motion Picture Arts and Sciences. 2019. Academy Spectral Similarity Index (SSI): Overview., 7 pages.
27. The Academy of Motion Picture Arts and Sciences, Science and Technology Council, and Academy Color Encoding System (ACES) Project Subcommittee. 2015. Procedure P-2013-001 - Recommended Procedures for the Creation and Use of Digital Camera System Input Device Transforms (IDTs)., 29 pages. http://j.mp/P-2013-001
28. Joscha Borné. 2014. Konvertierung von RGB nach Spektrum mittels Mean-Value-Koordinaten. Master's thesis, Karlsruhe Institute of Technology.
29. Scott Burns. 2019. Chromatic Adaptation Transform by Spectral Reconstruction. Color Research & Applications 44, 5 (2019), 682--693.
30. Scott Burns. 2020a. Fast RGB to Spectrum Conversion for Reflectances. http://scottburns.us/fast-rgb-to-spectrum-conversion-for-reflectances/
31. Scott Burns. 2020b. Numerical methods for smoothest reflectance reconstruction. Color Research & Applications 45, 1 (2020).
32. Andrew S. Glassner. 1989. How to derive a spectrum from an RGB triplet. IEEE Computer Graphics and Applications 9, 4 (1989), 95--99.
33. Toshiya Hachisuka and Henrik Wann Jensen. 2009. Stochastic Progressive Photon Mapping. In ACM SIGGRAPH Asia 2009 Papers (Yokohama, Japan). Article 141, 8 pages.
34. Ville Heikkinen, Reiner Lenz, Tuija Jetsu, Jussi Parkkinen, Markku Hauta-Kasari, and Timo Jääskeläinen. 2008. Evaluation and unification of some methods for estimating reflectance spectra from RGB images. J. Opt. Soc. Am. A 25, 10 (2008), 2444--2458.
35. Wenzel Jakob and Johannes Hanika. 2019. A Low-Dimensional Function Space for Efficient Spectral Upsampling. Computer Graphics Forum 38, 2 (2019).
36. Alisa Jung, Alexander Wilkie, Johannes Hanika, Wenzel Jakob, and Carsten Dachsbacher. 2019. Wide Gamut Spectral Upsampling with Fluorescence. Computer Graphics Forum (Proceedings of Eurographics Symposium on Rendering) 38, 4 (2019).
37. Lars König, Alisa Jung, and Carsten Dachsbacher. 2020. Improving Spectral Upsampling with Fluorescence. In Workshop on Material Appearance Modeling.
38. David L. MacAdam. 1935. Maximum Visual Efficiency of Colored Materials. J. Opt. Soc. Am. 25, 11 (1935).
39. Ian Mallett and Cem Yuksel. 2019. Spectral Primary Decomposition for Rendering with sRGB Reflectance. In EGSR '19 Proceedings of the 30th Eurographics Symposium on Rendering.
40. Johannes Meng, Florian Simon, Johannes Hanika, and Carsten Dachsbacher. 2015. Physically Meaningful Rendering using Tristimulus Colours. Computer Graphics Forum 34, 4 (2015).
41. Hisanari Otsu, Masafumi Yamamoto, and Toshiya Hachisuka. 2018. Reproducing Spectral Reflectances From Tristimulus Colours. Computer Graphics Forum 37, 6 (2018), 370--381.
42. Christoph Peters, Sebastian Merzbach, Johannes Hanika, and Carsten Dachsbacher. 2019a. Spectral Rendering with the Bounded MESE and sRGB Data. In MAM2019: Eurographics Workshop on Material Appearance Modeling.
43. Christoph Peters, Sebastian Merzbach, Johannes Hanika, and Carsten Dachsbacher. 2019b. Using Moments to Represent Bounded Signals for Spectral Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 38, 4 (2019), 136:1--136:14.
44. Brian Smits. 1999. An RGB-to-spectrum Conversion for Reflectances. Journal of Graphics Tools 4, 4 (1999), 11--22.
45. Alban Fichet, Romain Pacanowski, and Alexander Wilkie. 2021. An OpenEXR Layout for Spectral Images. Journal of Computer Graphics Techniques (JCGT) (2021).
46. Matthias B. Hullin, Johannes Hanika, Boris Ajdin, Hans-Peter Seidel, Jan Kautz, and Hendrik P. A. Lensch. 2010. Acquisition and Analysis of Bispectral Bidirectional Reflectance and Reradiation Distribution Functions. ACM Trans. Graph. 29, 4, Article 97 (July 2010), 7 pages.
47. Alisa Jung, Alexander Wilkie, Johannes Hanika, Wenzel Jakob, and Carsten Dachsbacher. 2019. Wide gamut spectral upsampling with fluorescence. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 87--96.
48. Michal Mojzík, Alban Fichet, and Alexander Wilkie. 2018. Handling Fluorescence in a Unidirectional Spectral Path Tracer. Comput. Graph. Forum 37, 4 (2018), 77--94.
49. Christoph Peters, Sebastian Merzbach, Johannes Hanika, and Carsten Dachsbacher. 2019. Using moments to represent bounded signals for spectral rendering. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--14.
50. Hua Qingqin, Alban Fichet, and Alexander Wilkie. 2021. A Compact Representation for Fluorescent Spectral Data. In Eurographics Symposium on Rendering (EGSR). Eurographics Association.
51. Alexander Wilkie. 2018. The Advanced Rendering Toolkit. http://cgg.mff.cuni.cz/ART.
52. Laurent Belcour and Pascal Barla. 2017. A Practical Extension to Microfacet Theory for the Modeling of Varying Iridescence. ACM Transactions on Graphics 36, 4 (July 2017), 65.
53. Matt Jen-Yuan Chiang, Peter Kutz, and Brent Burley. 2016. Practical and Controllable Subsurface Scattering for Production Path Tracing. In ACM SIGGRAPH 2016 Talks (Anaheim, California) (SIGGRAPH '16). Association for Computing Machinery, New York, NY, USA, Article 49, 2 pages.
54. Per H. Christensen. 2015. An Approximate Reflectance Profile for Efficient Subsurface Scattering. In ACM SIGGRAPH 2015 Talks (Los Angeles, California) (SIGGRAPH '15). Association for Computing Machinery, New York, NY, USA, Article 25, 1 pages.
55. Robert L. Cook, John Halstead, Maxwell Planck, and David Ryu. 2007. Stochastic Simplification of Aggregate Detail. ACM Trans. Graph. 26, 3 (July 2007), 79--es.
56. Jeppe Revall Frisvad, Toshiya Hachisuka, and Thomas Kim Kjeldsen. 2015. Directional Dipole Model for Subsurface Scattering. ACM Trans. Graph. 34, 1, Article 5 (Dec. 2015), 12 pages.
57. Daniel Heckenberg, Luke Emrose, Matthew Reid, Michael Balzer, Antoine Roille, and Max Liani. 2017. Rendering the Darkness: Glimpse on the LEGO Batman Movie. In ACM SIGGRAPH 2017 Talks (Los Angeles, California) (SIGGRAPH '17). Association for Computing Machinery, New York, NY, USA, Article 8, 2 pages.
58. Matthias B. Hullin, Johannes Hanika, and Wolfgang Heidrich. 2012. Polynomial Optics: A Construction Kit for Efficient Ray-Tracing of Lens Systems. Computer Graphics Forum (Proceedings of EGSR 2012) 31, 4 (July 2012).
59. Henrik Wann Jensen, Stephen R. Marschner, Marc Levoy, and Pat Hanrahan. 2001. A Practical Model for Subsurface Light Transport. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01). Association for Computing Machinery, New York, NY, USA, 511--518.
60. Alexander Keller, Pascal Grittmann, Jiří Vorba, Iliyan Georgiev, Martin Šik, Eugene d'Eon, Pascal Gautron, Petr Vévoda, and Ivo Kondapaneni. 2020. Advances in Monte Carlo Rendering: The Legacy of Jaroslav KřIváNek. In ACM SIGGRAPH 2020 Courses (Virtual Event, USA) (SIGGRAPH '20). Association for Computing Machinery, New York, NY, USA, Article 3, 366 pages.
61. Alan King, Christopher Kulla, Alejandro Conty, and Marcos Fajardo. 2013. BSSRDF Importance Sampling. In ACM SIGGRAPH 2013 Talks (Anaheim, California) (SIGGRAPH '13). Association for Computing Machinery, New York, NY, USA, Article 48, 1 pages.
62. Jaroslav Křivánek and Eugene d'Eon. 2014. A Zero-variance-based Sampling Scheme for Monte Carlo Subsurface Scattering. In ACM SIGGRAPH 2014 Talks (Vancouver, Canada) (SIGGRAPH '14). ACM, New York, NY, USA, Article 66, 1 pages.
63. Johannes Meng, Johannes Hanika, and Carsten Dachsbacher. 2016. Improving the Dwivedi Sampling Scheme. Computer Graphics Forum (Proceedings of Eurographics Symposium on Rendering) 35, 4 (2016), 37--44.
64. Jan Novák, Iliyan Georgiev, Johannes Hanika, Jaroslav Křivánek, and Wojciech Jarosz. 2018. Monte Carlo Methods for Physically Based Volume Rendering. In SIGGRAPH 2018 Courses (Vancouver, British Columbia, Canada). Article 14, 1 pages.
65. Jan Novák, Iliyan Georgiev, Johannes Hanika, and Wojciech Jarosz. 2018. Monte Carlo methods for volumetric light transport simulation. Computer Graphics Forum (Proceedings of Eurographics - State of the Art Reports) 37, 2 (May 2018). https://doi.org/10/gd2jqq
66. Michal Radziszewski, Krzysztof Boryczko, and Witold Alda. 2009. An Improved Technique for Full Spectral Rendering. J. WSCG 17, 1-3 (2009), 9--16. http://wscg.zcu.cz/WSCG2009/Papers_2009/!_2009_J_WSCG_No_1-3.zip
67. Charles Schmidt and Brian Budge. 2002. Simple Nested Dielectrics in Ray Traced Images. Journal of Graphics Tools 7 (01 2002).
68. Eric Veach. 1998. Robust Monte Carlo Methods for Light Transport Simulation. Ph.D. Dissertation. Stanford, CA, USA. Advisor(s) Guibas, Leonidas J. AAI9837162.
69. A. Wilkie, S. Nawaz, M. Droske, A. Weidlich, and J. Hanika. 2014. Hero Wavelength Spectral Sampling. In Proceedings of the 25th Eurographics Symposium on Rendering (Lyon, France) (EGSR '14). Eurographics Association, Goslar, DEU, 123--131.
70. Luca Fascione, Johannes Hanika, Mark Leone, Marc Droske, Jorge Schwarzhaupt, Tomáš Davidovič, Andrea Weidlich, and Johannes Meng. 2018. Manuka: A Batch-Shading Architecture for Spectral Path Tracing in Movie Production. ACM Trans. Graph. 37, 3, Article 31 (Aug. 2018), 18 pages.
71. Wenzel Jakob and Johannes Hanika. 2019. A Low-Dimensional Function Space for Efficient Spectral Upsampling. Computer Graphics Forum (Proceedings of Eurographics) 38, 2 (March 2019).
72. Anders Langlands and Luca Fascione. 2020. PhysLight: An End-to-End Pipeline for Scene-Referred Lighting. In ACM SIGGRAPH 2020 Talks (Virtual Event, USA) (SIGGRAPH '20). Association for Computing Machinery, New York, NY, USA, Article 19, 2 pages.
73. Christoph Peters, Sebastian Merzbach, Johannes Hanika, and Carsten Dachsbacher. 2019. Using Moments to Represent Bounded Signals for Spectral Rendering. ACM Trans. Graph. 38, 4, Article 136 (July 2019), 14 pages.
74. Brian Smits. 2000. An RGB to Spectrum Conversion for Reflectances. Journal of Graphics Tools 4 (06 2000).
75. A. Wilkie, S. Nawaz, M. Droske, A. Weidlich, and J. Hanika. 2014. Hero Wavelength Spectral Sampling. Computer Graphics Forum 33, 4 (2014), 123--131. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.12419

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

    • Image manipulation

    • Rendering