A gradient-based framework for 3D print appearance optimization

Abstract

In full-color inkjet 3D printing, a key problem is determining the material configuration for the millions of voxels that a printed object is made of. The goal is a configuration that minimises the difference between desired target appearance and the result of the printing process. So far, the techniques used to find such a configuration have relied on domain-specific methods or heuristic optimization, which allowed only a limited level of control over the resulting appearance.

We propose to use differentiable volume rendering in a continuous material-mixture space, which leads to a framework that can be used as a general tool for optimising inkjet 3D printouts. We demonstrate the technical feasibility of this approach, and use it to attain fine control over the fabricated appearance, and high levels of faithfulness to the specified target.

References

1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. http://tensorflow.org/ Software available from tensorflow.org.
2. Navid Ansari, Omid Alizadeh-Mousavi, Hans-Peter Seidel, and Vahid Babaei. 2020. Mixed integer ink selection for spectral reproduction. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 39, 6 (Nov. 2020), 255:1--255:16.
3. Thomas Auzinger, Wolfgang Heidrich, and Bernd Bickel. 2018. Computational design of nanostructural color for additive manufacturing. ACM Transactions on Graphics (Proc. SIGGRAPH) 37, 4 (July 2018), 159:1--159:16.
4. Vahid Babaei, Kiril Vidimče, Michael Foshey, Alexandre Kaspar, Piotr Didyk, and Wojciech Matusik. 2017. Color Contoning for 3D Printing. ACM Transactions on Graphics (Proc. SIGGRAPH) 36, 4 (July 2017), 124:1--124:15.
5. Atılım Günes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2017. Automatic Differentiation in Machine Learning: a Survey. Journal of Machine Learning Research 18, 1 (2017), 5595--5637.
6. Alan Brunton, Can Ates Arikan, Tejas Madan Tanksale, and Philipp Urban. 2018. 3D Printing Spatially Varying Color and Translucency. ACM Transactions on Graphics (Proc. SIGGRAPH) 37, 4 (July 2018), 157:1--157:13.
7. Alan Brunton, Can Ates Arikan, and Philipp Urban. 2015. Pushing the Limits of 3D Color Printing: Error Diffusion with Translucent Materials. ACM Transactions on Graphics 35, 1 (Dec. 2015), 4:1--4:13.
8. Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2020. Towards Learning-based Inverse Subsurface Scattering. In 2020 IEEE International Conference on Computational Photography, ICCP 2020, Saint Louis, MO, USA, April 24-26, 2020. IEEE, New York, NY, USA, 1--12.
9. Guan-Hao Chen, Chun-Ling Yang, Lai-Man Po, and Sheng-Li Xie. 2006. Edge-Based Structural Similarity for Image Quality Assessment. In IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, Vol. 2. IEEE, New York, NY, USA, 933--936.
10. Yue Dong, Jiaping Wang, Fabio Pellacini, Xin Tong, and Baining Guo. 2010. Fabricating spatially-varying subsurface scattering. ACM Transactions on Graphics (Proc. SIGGRAPH) 29, 4 (July 2010), 62:1--62:10.
11. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research 12, Jul (2011), 2121--2159.
12. John Duchi, Shai Shalev-Shwartz, Yoram Singer, and Tushar Chandra. 2008. Efficient Projections onto the L1-Ball for Learning in High Dimensions. In 25th International Conference on Machine Learning (Helsinki, Finland) (ICML '08). Association for Computing Machinery, New York, NY, USA, 272--279.
13. Oskar Elek, Denis Sumin, Ran Zhang, Tim Weyrich, Karol Myszkowski, Bernd Bickel, Alexander Wilkie, and Jaroslav Křivánek. 2017. Scattering-aware Texture Reproduction for 3D Printing. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 36, 6 (Nov. 2017), 241:1--241:15.
14. Roland W Fleming and Heinrich H Bülthoff. 2005. Low-level image cues in the perception of translucent materials. ACM Transactions on Applied Perception (TAP) 2, 3 (2005), 346--382.
15. J. R. Frisvad, S. A. Jensen, J. S. Madsen, A. Correia, L. Yang, S. K. S. Gregersen, Y. Meuret, and P.-E. Hansen. 2020. Survey of Models for Acquiring the Optical Properties of Translucent Materials. Computer Graphics Forum 39, 2 (2020), 729--755. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14023
16. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An Evaluation of Computational Imaging Techniques for Heterogeneous Inverse Scattering. In European Conference on Computer Vision. Springer, Berlin, Germany, 685--701.
17. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse volume rendering with material dictionaries. ACM Transactions on Graphics (TOG) 32, 6 (2013), 162:1--162:13.
18. Miloš Hašan, Martin Fuchs, Wojciech Matusik, Hanspeter Pfister, and Szymon Rusinkiewicz. 2010. Physical Reproduction of Materials with Specified Subsurface Scattering. ACM Trans. Graph. 29, 4, Article 61 (July 2010), 10 pages.
19. Roman Hochuli, Samuel Powell, Simon Arridge, and Ben Cox. 2016. Quantitative photoacoustic tomography using forward and adjoint Monte Carlo models of radiance. Journal of biomedical optics 21, 12 (2016), 126004.
20. Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching Real Fabrics with Micro-Appearance Models. ACM Trans. Graph. 35, 1 (2015), 1--1.
21. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable monte carlo ray tracing through edge sampling. ACM Transactions on Graphics (TOG) 37, 6 (July 2018), 125:1--125:12.
22. M Ronnier Luo, Guihua Cui, and Bryan Rigg. 2001. The development of the CIE 2000 colour-difference formula: CIEDE2000. Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 26, 5 (2001), 340--350.
23. A. Luongo, V. Falster, M. B. Doest, M. M. Ribo, E. R. Eiriksson, D. B. Pedersen, and J. R. Frisvad. 2020. Microstructure Control in 3D Printing with Digital Light Processing. Computer Graphics Forum 39, 1 (2020), 347--359. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.13807
24. M. Magnor, G. Kindlmann, N. Duric, and C. Hansen. 2004. Constrained inverse volume rendering for planetary nebulae. In IEEE Visualization 2004. IEEE, New York, NY, USA, 83--90.
25. K McLaren. 1976. XIII - The development of the CIE 1976 (L* a* b*) uniform colour space and colour-difference formula. Journal of the Society of Dyers and Colourists 92, 9 (1976), 338--341.
26. Peter Morovič, Ján Morovič, Ingeborg Tastl, Melanie Gottwals, and Gary Dispoto. 2019. Co-optimization of color and mechanical properties by volumetric voxel control. Struct Multidisc Optim 60, 3 (Sept. 2019), 895--908.
27. Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020), 146:1--146:15.
28. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38, 6 (Dec. 2019), 203:1--203:17.
29. Isabel Molina Orihuela and Mehran Ebrahimi. 2019. An Efficient Algorithm for Computing the Derivative of Mean Structural Similarity Index Measure. In Image Analysis and Recognition, Fakhri Karray, Aurélio Campilho, and Alfred Yu (Eds.). Springer International Publishing, Cham, 55--66.
30. Marios Papas, Christian Regg, Wojciech Jarosz, Bernd Bickel, Philip Jackson, Wojciech Matusik, Steve Marschner, and Markus Gross. 2013. Fabricating translucent materials using continuous pigment mixtures. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1--12.
31. Michal Piovarči, Michael Foshey, Vahid Babaei, Szymon Rusinkiewicz, Wojciech Matusik, and Piotr Didyk. 2020. Towards spatially varying gloss reproduction for 3D printing. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1--13.
32. Jens Preiss, Felipe Fernandes, and Philipp Urban. 2014. Color-image quality assessment: From prediction to optimization. IEEE Transactions on Image Processing 23, 3 (2014), 1366--1378.
33. Olivier Rouiller, Bernd Bickel, Jan Kautz, Wojciech Matusik, and Marc Alexa. 2013. 3D-printing spatially varying BRDFs. IEEE computer graphics and applications 33, 6 (2013), 48--57.
34. Gaurav Sharma, Wencheng Wu, and Edul N Dalal. 2005. The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 30, 1 (2005), 21--30.
35. Kfir Shem-Tov, Sai Praveen Bangaru, Anat Levin, and Ioannis Gkioulekas. 2020. Towards Reflectometry from Interreflections. In 2020 IEEE International Conference on Computational Photography (ICCP). IEEE, New York, NY, USA, 1--12.
36. Liang Shi, Vahid Babaei, Changil Kim, Michael Foshey, Yuanming Hu, Pitchaya Sitthi-Amorn, Szymon Rusinkiewicz, and Wojciech Matusik. 2018. Deep multispectral painting reproduction via multi-layer, custom-ink printing. ACM Trans. Graph. 37, 6 (Dec. 2018), 1--15.
37. Denis Sumin, Tobias Rittig, Vahid Babaei, Thomas Nindel, Alexander Wilkie, Piotr Didyk, Bernd Bickel, Jaroslav Křivánek, Karol Myszkowski, and Tim Weyrich. 2019. Geometry-Aware Scattering Compensation for 3D Printing. ACM Transactions on Graphics (Proc. SIGGRAPH) 38, 4 (July 2019), 111:1--111:14.
38. Philipp Urban, Tejas Madan Tanksale, Alan Brunton, Bui Minh Vu, and Shigeki Nakauchi. 2019. Redefining A in RGBA: Towards a Standard for Graphical 3D Printing. ACM Trans. Graph. 38, 3 (June 2019), 21:1--21:14.
39. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600--612.
40. Zhou Wang, Eero P Simoncelli, and Alan C Bovik. 2003. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, Vol. 2. IEEE, New York, NY, USA, 1398--1402.
41. Douglas R Wyman, Michael S Patterson, and Brian C Wilson. 1989a. Similarity relations for anisotropic scattering in Monte Carlo simulations of deeply penetrating neutral particles. J. Comput. Phys. 81, 1 (1989), 137--150.
42. Douglas R Wyman, Michael S Patterson, and Brian C Wilson. 1989b. Similarity relations for the interaction parameters in radiation transport. Applied optics 28, 24 (1989), 5243--5249.
43. Bei Xiao, Bruce Walter, Ioannis Gkioulekas, Todd Zickler, Edward Adelson, and Kavita Bala. 2014. Looking against the light: How perception of translucency depends on lighting direction. Journal of vision 14, 3 (2014), 17--17.
44. Bei Xiao, Shuang Zhao, Ioannis Gkioulekas, Wenyan Bi, and Kavita Bala. 2020. Effect of geometric sharpness on translucent material perception. Journal of vision 20, 7 (2020), 10--10.
45. Wufeng Xue, Lei Zhang, Xuanqin Mou, and Alan C Bovik. 2013. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE Transactions on Image Processing 23, 2 (2013), 684--695.
46. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A differential theory of radiative transfer. ACM Transactions on Graphics (TOG) 38, 6 (Nov. 2019), 227:1--227:16.
47. Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. 2011. FSIM: A feature similarity index for image quality assessment. IEEE transactions on Image Processing 20, 8 (2011), 2378--2386.
48. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. IEEE Conf. Comp. Vision & Pat. Rec. (CVPR). IEEE, New York, NY, USA, 586--595.
49. Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. 2016a. Loss functions for image restoration with neural networks. IEEE Transactions on computational imaging 3, 1 (2016), 47--57.
50. Shuang Zhao, Ravi Ramamoorthi, and Kavita Bala. 2014. High-order similarity relations in radiative transfer. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1--12.
51. Shuang Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016b. Downsampling scattering parameters for rendering anisotropic media. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1--11.
52. Quan Zheng, Vahid Babaei, Gordon Wetzstein, Hans-Peter Seidel, Matthias Zwicker, and Gurprit Singh. 2020. Neural Light Field 3D Printing. ACM Trans. Graph. 39, 6, Article 207 (Nov. 2020), 12 pages.

Index Terms

• Applied computing

  • Operations research

    • Computer-aided manufacturing

• Computing methodologies

  • Computer graphics

    • Rendering

    • Shape modeling

      • Volumetric models