Joint neural phase retrieval and compression for energy- and computation-efficient holography on the edge

Abstract

Recent deep learning approaches have shown remarkable promise to enable high fidelity holographic displays. However, lightweight wearable display devices cannot afford the computation demand and energy consumption for hologram generation due to the limited onboard compute capability and battery life. On the other hand, if the computation is conducted entirely remotely on a cloud server, transmitting lossless hologram data is not only challenging but also result in prohibitively high latency and storage.

In this work, by distributing the computation and optimizing the transmission, we propose the first framework that jointly generates and compresses high-quality phase-only holograms. Specifically, our framework asymmetrically separates the hologram generation process into high-compute remote encoding (on the server), and low-compute decoding (on the edge) stages. Our encoding enables light weight latent space data, thus faster and efficient transmission to the edge device. With our framework, we observed a reduction of 76% computation and consequently 83% in energy cost on edge devices, compared to the existing hologram generation methods. Our framework is robust to transmission and decoding errors, and approach high image fidelity for as low as 2 bits-per-pixel, and further reduced average bit-rates and decoding time for holographic videos.

References

1. Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D. Fairchild. 2020. FLIP: A Difference Evaluator for Alternating Images. In Proceedings of the ACM on Computer Graphics and Interactive Techniques, Vol. 3. Article 15, 15:1--15:23 pages.
2. Lasse F. Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. 2020. Carbon-tracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models. ICML Workshop on Challenges in Deploying and monitoring Machine Learning Systems.
3. Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. 2018. Variational image compression with a scale hyperprior. In International Conference on Learning Representations (ICLR).
4. Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. 2016. End-to-end optimization of nonlinear transform codes for perceptual quality. In Picture Coding Symposium (PCS). IEEE Signal Processing Society, 1--5.
5. Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. 2017. End-to-end optimized image compression. In International Conference on Learning Representations (ICLR).
6. Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. 2020. CompressAI: a PyTorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029 (2020).
7. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv preprint arXiv:1308.3432 (2013).
8. Stephen A Benton and V Michael Bove Jr. 2008. Holographic imaging. John Wiley & Sons.
9. Lokesh Boominathan, Mayug Maniparambil, Honey Gupta, Rahul Baburajan, and Kaushik Mitra. 2018. Phase retrieval for Fourier Ptychography under varying amount of measurements. arXiv preprint arXiv:1805.03593 (2018).
10. Praneeth Chakravarthula, Yifan Peng, Joel Kollin, Henry Fuchs, and Felix Heide. 2019. Wirtinger Holography for Near-Eye Displays. ACM Transactions on Graphics (TOG) 38, 6, Article 213 (2019).
11. Praneeth Chakravarthula, Ethan Tseng, Henry Fuchs, and Felix Heide. 2022. Hogel-free Holography. ACM Transactions on Graphics (TOG) (2022).
12. Praneeth Chakravarthula, Ethan Tseng, Tarun Srivastava, Henry Fuchs, and Felix Heide. 2020a. Learned hardware-in-the-loop phase retrieval for holographic near-eye displays. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1--18.
13. Praneeth Chakravarthula, Ethan Tseng, Tarun Srivastava, Henry Fuchs, and Felix Heide. 2020b. Learned Hardware-in-the-Loop Phase Retrieval for Holographic near-Eye Displays. ACM Transactions on Graphics (TOG) 39, 6, Article 186 (2020).
14. Praneeth Chakravarthula, Zhan Zhang, Okan Tursun, Piotr Didyk, Qi Sun, and Henry Fuchs. 2021. Gaze-Contingent Retinal Speckle Suppression for Perceptually-Matched Foveated Holographic Displays. IEEE Transactions on Visualization and Computer Graphics 27, 11 (2021), 4194--4203.
15. Rick H-Y Chen and Timothy D Wilkinson. 2009. Computer generated hologram from point cloud using graphics processor. Applied optics 48, 36 (2009), 6841--6850.
16. Yinbo Chen, Sifei Liu, and Xiaolong Wang. 2021. Learning Continuous Image Representation with Local Implicit Image Function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
17. Mathew J Cherukara, Youssef SG Nashed, and Ross J Harder. 2018. Real-time coherent diffraction inversion using deep generative networks. Scientific reports 8, 1 (2018), 1--8.
18. Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, and Gordon Wetzstein. 2021. Neural 3D Holography: Learning Accurate Wave Propagation Models for 3D Holographic Virtual and Augmented Reality Displays. ACM Trans. Graph. (SIGGRAPH Asia) (2021).
19. Thomas M. Cover and Joy A. Thomas. 2006. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA.
20. Steffen Czolbe, Oswin Krause, Ingemar Cox, and Christian Igel. 2020. A Loss Function for Generative Neural Networks Based on Watson's Perceptual Model. Advances in Neural Information Processing Systems 33, 2051--2061.
21. Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. 2021. On the Effectiveness of Weight-Encoded Neural Implicit 3D Shapes. arXiv preprint arXiv:2009.09808 (2021).
22. Jarek Duda. 2014. Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding. arXiv preprint arXiv:1311.2540 (2014).
23. M Hossein Eybposh, Nicholas W Caira, Mathew Atisa, Praneeth Chakravarthula, and Nicolas C Pégard. 2020. DeepCGH: 3D computer-generated holography using deep learning. Optics Express 28, 18 (2020), 26636--26650.
24. Alexandre Goy, Kwabena Arthur, Shuai Li, and George Barbastathis. 2018. Low photon count phase retrieval using deep learning. Physical review letters 121, 24 (2018).
25. Robert M Gray. 2011. Entropy and information theory. Springer Science & Business Media.
26. Yueyu Hu, Wenhan Yang, Zhan Ma, and Jiaying Liu. 2021. Learning end-to-end lossy image compression: A benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
27. Shuming Jiao, Zhi Jin, Chenliang Chang, Changyuan Zhou, Wenbin Zou, and Xia Li. 2018. Compression of Phase-Only Holograms with JPEG Standard and Deep Learning. Applied Sciences 8, 8, Article 1258 (2018).
28. Michael R. Kellman, Emrah Bostan, Nicole A. Repina, and Laura Waller. 2019. Physics-Based Learned Design: Optimized Coded-Illumination for Quantitative Phase Imaging. IEEE Transactions on Computational Imaging 5, 3 (2019), 344--353.
29. Zachary David Cleary Kemp. 2018. Propagation based phase retrieval of simulated intensity measurements using artificial neural networks. Journal of Optics 20, 4 (2018), 045606.
30. Hwi Kim, Joonku Hahn, and Byoungho Lee. 2008. Mathematical modeling of triangle-mesh-modeled three-dimensional surface objects for digital holography. Applied optics 47, 19 (2008), D117--D127.
31. Seung-Cheol Kim and Eun-Soo Kim. 2008. Effective generation of digital holograms of three-dimensional objects using a novel look-up table method. Appl. Opt. 47, 19 (Jul 2008), D55--D62.
32. Xiangbo Li, Mahmoud Darwich, Magdy Bayoumi, and Mohsen Amini Salehi. 2020. Cloud-Based Video Streaming Services: A Survey. arXiv preprint arXiv:2011.14976 (2020).
33. Robert LiKamWa, Zhen Wang, Aaron Carroll, Felix Xiaozhu Lin, and Lin Zhong. 2014. Draining Our Glass: An Energy and Heat Characterization of Google Glass. In Proceedings of 5th Asia-Pacific Workshop on Systems. ACM New York, NY, Article 10.
34. Siwei Ma, Xinfeng Zhang, Chuanmin Jia, Zhenghui Zhao, Shiqi Wang, and Shanshe Wang. 2020. Image and Video Compression With Neural Networks: A Review. IEEE Transactions on Circuits and Systems for Video Technology 30, 6 (2020), 1683--1698.
35. Andrew Maimone, Andreas Georgiou, and Joel S. Kollin. 2017. Holographic Near-Eye Displays for Virtual and Augmented Reality. ACM Transactions on Graphics (TOG) 36, 4, Article 85 (2017).
36. Rafał K. Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney. 2021. FovVideoVDP: A Visible Difference Predictor for Wide Field-of-View Video. ACM Transactions on Graphics (TOG) 40, 4, Article 49 (2021).
37. Julien N.P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. 2021. ACORN: Adaptive Coordinate Networks for Neural Representation. ACM Transactions on Graphics (TOG) 40, 4, Article 58 (2021).
38. Nobuyuki Masuda, Tomoyoshi Ito, Takashi Tanaka, Atsushi Shiraki, and Takashige Sugie. 2006. Computer generated holography using a graphics processing unit. Optics Express 14, 2 (2006), 603--608.
39. Kyoji Matsushima. 2005. Computer-generated holograms for three-dimensional surface objects with shade and texture. Applied optics 44, 22 (2005), 4607--4614.
40. Kyoji Matsushima and Tomoyoshi Shimobaba. 2009. Band-Limited Angular Spectrum Method for Numerical Simulation of Free-Space Propagation in Far and Near Fields. Optics express 17, 22 (2009), 19662--19673.
41. Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson. 2020. High-Fidelity Generative Image Compression. In Advances in Neural Information Processing Systems, Vol. 33. 11913--11924.
42. David Minnen, Johannes Ballé, and George Toderici. 2018. Joint Autoregressive and Hierarchical Priors for Learned Image Compression. In Advances in neural information processing systems. 10794--10803.
43. Y. Ogihara and Y. Sakamoto. 2015. Fast calculation method of a CGH for a patch model using a point-based method. Applied Optics 54, 1 (2015), A76--A83.
44. Nitish Padmanaban, Yifan Peng, and Gordon Wetzstein. 2019. Holographic near-eye displays based on overlap-add stereograms. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--13.
45. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems. 8024--8035.
46. Yifan Peng, Suyeon Choi, Nitish Padmanaban, and Gordon Wetzstein. 2020. Neural Holography with Camera-in-the-Loop Training. ACM Transactions on Graphics (TOG) 39, 6, Article 185 (2020).
47. Christoph Petz and Marcus Magnor. 2003. Fast hologram synthesis for 3D geometry models using graphics hardware. In Proc. SPIE 5005, Practical Holography XVII and Holographic Materials IX. 266--275.
48. Jorma Rissanen and Glen Langdon. 1981. Universal modeling and coding. IEEE Transactions on Information Theory 27, 1 (1981), 12--23.
49. Yair Rivenson, Yibo Zhang, Harun Günaydın, Da Teng, and Aydogan Ozcan. 2018. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light: Science & Applications 7, 2 (2018), 17141.
50. Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer, and Wojciech Matusik. 2021. Towards real-time photorealistic 3D holography with deep neural networks. Nature 591, 7849 (2021), 234--239.
51. Tomoyoshi Shimobaba, Nobuyuki Masuda, and Tomoyoshi Ito. 2009. Simple and fast calculation algorithm for computer-generated hologram with wavefront recording plane. Optics letters 34, 20 (2009), 3133--3135.
52. K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations (ICLR).
53. David Taubman and Michael Marcellin. 2013. JPEG2000 Image Compression Fundamentals, Standards and Practice. Springer Publishing Company, Incorporated.
54. Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. 2017. Lossy Image Compression with Compressive Autoencoders. In International Conference on Learning Representations (ICLR).
55. Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, et al. 2017. NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
56. Gregory K Wallace. 1992. The JPEG still picture compression standard. IEEE Transactions on Consumer Electronics 38, 1 (1992), xviii--xxxiv.
57. Haiqiang Wang, Ioannis Katsavounidis, Jiantong Zhou, Jeonghoon Park, Shawmin Lei, Xin Zhou, Man-On Pun, Xin Jin, Ronggang Wang, Xu Wang, Yun Zhang, Jiwu Huang, Sam Kwong, and Kuo C.-C. Jay. 2017. VideoSet: A large-scale compressed video quality dataset based on JND measurement. Journal of Visual Communication and Image Representation 46 (2017), 292--302.
58. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612.
59. Z. Wang, E. P. Simoncelli, and A. C. Bovik. 2003. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, Vol. 2. 1398--1402.
60. Hao Zhang, Liangcai Cao, and Guofan Jin. 2017. Computer-generated hologram with occlusion effect using layer-based processing. Appl. Opt. 56, 13 (May 2017), F138--F143.
61. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 586--595.

Index Terms

• Hardware

  • Communication hardware, interfaces and storage

    • Displays and imagers