Accommodative holography: improving accommodation response for perceptually realistic holographic displays

Abstract

Holographic displays have gained unprecedented attention as next-generation virtual and augmented reality applications with recent achievements in the realization of a high-contrast image through computer-generated holograms (CGHs). However, these holograms show a high energy concentration in a limited angular spectrum, whereas the holograms with uniformly distributed angular spectrum suffer from a severe speckle noise in the reconstructed images. In this study, we claim that these two physical phenomena attributed to the existing CGHs significantly limit the support of accommodation cues, which is known as one of the biggest advantages of holographic displays. To support the statement, we analyze and evaluate various CGH algorithms with contrast gradients - a change of contrast over the change of the focal diopter of the eye - simulated based on the optical configuration of the display system and human visual perception models. We first introduce two approaches to improve monocular accommodation response in holographic viewing experience; optical and computational approaches to provide holographic images with sufficient contrast gradients. We design and conduct user experiments with our prototype of holographic near-eye displays, validating the deficient support of accommodation cues in the existing CGH algorithms and demonstrating the feasibility of the proposed solutions with significant improvements on accommodative gains.

References

1. Eirikur Agustsson and Radu Timofte. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 126--135.
2. Kurt Akeley, Simon J. Watt, Ahna Reza Girshick, and Martin S. Banks. 2004. A Stereo Display Prototype with Multiple Focal Distances. ACM Trans. Graph. 23, 3 (2004), 804--813.
3. Martin S Banks, Wilson S Geisler, and Patrick J Bennett. 1987. The physical limits of grating visibility. Vision research 27, 11 (1987), 1915--1924.
4. Peter GJ Barten. 1999. Contrast sensitivity of the human eye and its effects on image quality. SPIE press.
5. Stephen A Benton and V Michael Bove Jr. 2008. Holographic imaging. John Wiley & Sons.
6. Rafal Bogacz, Eric Brown, Jeff Moehlis, Philip Holmes, and Jonathan D Cohen. 2006. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychological Review 113, 4 (2006), 700.
7. SE Broomfield, MAA Neil, EGS Paige, and GG Yang. 1992. Programmable binary phase-only optical device based on ferroelectric liquid crystal SLM. Electronics Letters 28, 1 (1992), 26--28.
8. Praneeth Chakravarthula, Yifan Peng, Joel Kollin, Henry Fuchs, and Felix Heide. 2019. Wirtinger Holography for Near-Eye Displays. 38, 6, Article 213 (2019), 13 pages.
9. Praneeth Chakravarthula, Ethan Tseng, Tarun Srivastava, Henry Fuchs, and Felix Heide. 2020. Learned Hardware-in-the-Loop Phase Retrieval for Holographic near-Eye Displays. ACM Trans. Graph. 39, 6, Article 186 (2020), 18 pages.
10. Praneeth Chakravarthula, Zhan Zhang, Okan Tursun, Piotr Didyk, Qi Sun, and Henry Fuchs. 2021. Gaze-contingent retinal speckle suppression for perceptually-matched foveated holographic displays. IEEE Transactions on Visualization and Computer Graphics 27, 11 (2021), 4194--4203.
11. Chenliang Chang, Kiseung Bang, Gordon Wetzstein, Byoungho Lee, and Liang Gao. 2020. Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective. Optica 7, 11 (2020), 1563--1578.
12. Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, and Gordon Wetzstein. 2021. Neural 3D Holography: Learning Accurate Wave Propagation Models for 3D Holographic Virtual and Augmented Reality Displays. ACM Trans. Graph. 40, 6, Article 240 (2021), 12 pages.
13. Steven A. Cholewiak, Gordon D. Love, Pratul P. Srinivasan, Ren Ng, and Martin S. Banks. 2017. Chromablur: Rendering Chromatic Eye Aberration Improves Accommodation and Realism. ACM Trans. Graph. 36, 6, Article 210 (2017), 12 pages.
14. Bruce G Cumming and Stuart J Judge. 1986. Disparity-induced and blur-induced convergence eye movement and accommodation in the monkey. Journal of Neurophysiology 55, 5 (1986), 896--914.
15. Vincent R Curtis, Nicholas W Caira, Jiayi Xu, Asha Gowda Sata, and Nicolas C Pégard. 2021. DCGH: Dynamic computer generated holography for speckle-free, high fidelity 3D displays. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). IEEE, 1--9.
16. Scott J Daly. 1992. Visible differences predictor: an algorithm for the assessment of image fidelity. In Human Vision, Visual Processing, and Digital Display III, Vol. 1666. International Society for Optics and Photonics, 2--15.
17. Antonio J Del Águila-Carrasco, Iván Marín-Franch, Paula Bernal-Molina, José J Esteve-Taboada, Philip B Kruger, Robert Montés-Micó, and Norberto López-Gil. 2017. Accommodation responds to optical vergence and not defocus blur alone. Investigative ophthalmology & visual science 58, 3 (2017), 1758--1763.
18. Yuanbo Deng and Daping Chu. 2017. Coherence properties of different light sources and their effect on the image sharpness and speckle of holographic displays. Scientific Reports 7, 1 (2017), 1--12.
19. James R Fienup. 1982. Phase retrieval algorithms: a comparison. Applied Optics 21, 15 (1982), 2758--2769.
20. Edgar F Fincham. 1951. The accommodation reflex and its stimulus. The British Journal of Ophthalmology 35, 7 (1951), 381.
21. Ralph W Gerchberg. 1972. A practical algorithm for the determination of phase from image and diffraction plane pictures. Optik 35 (1972), 237--246.
22. Joseph W Goodman. 2005. Introduction to Fourier optics. Roberts and Company Publishers.
23. Joseph W Goodman. 2007. Speckle phenomena in optics: theory and applications. Roberts and Company Publishers.
24. Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1--10.
25. David M Hoffman, Ahna R Girshick, Kurt Akeley, and Martin S Banks. 2008. Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. Journal of Vision 8, 3 (2008), 33--33.
26. Ryoichi Horisaki, Ryosuke Takagi, and Jun Tanida. 2018. Deep-learning-generated holography. Applied Optics 57, 14 (2018), 3859--3863.
27. Chung-Kai Hsueh and Alexander A Sawchuk. 1978. Computer-generated double-phase holograms. Applied Optics 17, 24 (1978), 3874--3883.
28. Fu-Chung Huang, David P Luebke, and Gordon Wetzstein. 2015. The light field stereoscope.. In SIGGRAPH emerging technologies. 24--1.
29. Changwon Jang, Kiseung Bang, Gang Li, and Byoungho Lee. 2018. Holographic Near-Eye Display with Expanded Eye-Box. ACM Trans. Graph. 37, 6, Article 195 (dec 2018), 14 pages.
30. Changwon Jang, Kiseung Bang, Seokil Moon, Jonghyun Kim, Seungjae Lee, and Byoungho Lee. 2017. Retinal 3D: Augmented Reality near-Eye Display via Pupil-Tracked Light Field Projection on Retina. ACM Trans. Graph. 36, 6, Article 190 (2017), 13 pages.
31. Donald H Kelly. 1979. Motion and vision. II. Stabilized spatio-temporal threshold surface. Josa 69, 10 (1979), 1340--1349.
32. Dongyeon Kim, Seung-Woo Nam, Kiseung Bang, Byounghyo Lee, Seungjae Lee, Youngmo Jeong, Jong-Mo Seo, and Byoungho Lee. 2021. Vision-correcting holographic display: evaluation of aberration correcting hologram. Biomedical Optics Express 12, 8 (2021), 5179--5195.
33. Robert Konrad, Nitish Padmanaban, Keenan Molner, Emily A. Cooper, and Gordon Wetzstein. 2017. Accommodation-Invariant Computational near-Eye Displays. ACM Trans. Graph. 36, 4, Article 88 (2017), 12 pages.
34. George-Alex Koulieris, Bee Bui, Martin S. Banks, and George Drettakis. 2017. Accommodation and Comfort in Head-Mounted Displays. ACM Trans. Graph. 36, 4, Article 87 (2017), 11 pages.
35. Brooke Krajancich, Petr Kellnhofer, and Gordon Wetzstein. 2021. A Perceptual Model for Eccentricity-dependent Spatio-temporal Flicker Fusion and its Applications to Foveated Graphics. ACM Trans. Graph. 40 (2021). Issue 4.
36. Philip B Kruger, Steven Mathews, Karan R Aggarwala, and Nivian Sanchez. 1993. Chromatic aberration and ocular focus: Fincham revisited. Vision Research 33, 10 (1993), 1397--1411.
37. Grace Kuo, Laura Waller, RenNg, and AndrewMaimone. 2020. High Resolution éTendue Expansion for Holographic Displays. ACM Trans. Graph. 39, 4, Article 66 (2020), 14 pages.
38. Byounghyo Lee, Dongyeon Kim, Seungjae Lee, Chun Chen, and Byoungho Lee. 2022. High-contrast, speckle-free, true 3D holography via binary CGH optimization. Scientific Reports 12, 1 (18 Feb 2022), 2811.
39. Byounghyo Lee, Dongheon Yoo, Jinsoo Jeong, Seungjae Lee, Dukho Lee, and Byoungho Lee. 2020c. Wide-angle speckleless DMD holographic display using structured illumination with temporal multiplexing. Optics Letters 45, 8 (2020), 2148--2151.
40. Juhyun Lee, Jinsoo Jeong, Jaebum Cho, Dongheon Yoo, Byounghyo Lee, and Byoungho Lee. 2020a. Deep neural network for multi-depth hologram generation and its training strategy. Optics Express 28, 18 (2020), 27137--27154.
41. Seungjae Lee, Youngjin Jo, Dongheon Yoo, Jaebum Cho, Dukho Lee, and Byoungho Lee. 2019. Tomographic near-eye displays. Nature communications 10, 1 (2019), 1--10.
42. Seungjae Lee, Dongyeon Kim, Seung-Woo Nam, Byounghyo Lee, Jaebum Cho, and Byoungho Lee. 2020b. Light source optimization for partially coherent holographic displays with consideration of speckle contrast, resolution, and depth of field. Scientific Reports 10, 1 (2020), 1--12.
43. Wai Hon Lee. 1970. Sampled Fourier transform hologram generated by computer. Applied Optics 9, 3 (1970), 639--643.
44. Gang Li, Dukho Lee, Youngmo Jeong, Jaebum Cho, and Byoungho Lee. 2016. Holographic display for see-through augmented reality using mirror-lens holographic optical element. Optics Letters 41, 11 (2016), 2486--2489.
45. Yongjun Lim, Keehoon Hong, Hwi Kim, Hyun-Eui Kim, Eun-Young Chang, Soohyun Lee, Taeone Kim, Jeho Nam, Hyon-Gon Choo, Jinwoong Kim, et al. 2016. 360-degree tabletop electronic holographic display. Optics Express 24, 22 (2016), 24999--25009.
46. Kevin J MacKenzie, David M Hoffman, and Simon J Watt. 2010. Accommodation to multiple-focal-plane displays: Implications for improving stereoscopic displays and for accommodation control. Journal of Vision 10, 8 (2010), 22--22.
47. Andrew Maimone, Andreas Georgiou, and Joel S Kollin. 2017. Holographic near-eye displays for virtual and augmented reality. ACM Transactions on Graphics (Tog) 36, 4 (2017), 1--16.
48. Rafał Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in All Luminance Conditions. ACM Trans. Graph. 30, 4, Article 40 (2011), 14 pages.
49. Rafał K. Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney. 2021. FovVideoVDP: A Visible Difference Predictor for Wide Field-of-View Video. ACM Trans. Graph. 40, 4, Article 49 (2021), 19 pages.
50. Theodore G Martens and Kenneth N Ogle. 1959. Observations on accommodative convergence: Especially its nonlinear relationships. American Journal of Ophthalmology 47, 1 (1959), 455--463.
51. Steven Mathews and Philip B Kruger. 1994. Spatiotemporal transfer function of human accommodation. Vision Research 34, 15 (1994), 1965--1980.
52. Arian Mehrfard, Javad Fotouhi, Giacomo Taylor, Tess Forster, Nassir Navab, and Bernhard Fuerst. 2019. A comparative analysis of virtual reality head-mounted display systems. arXiv preprint arXiv:1912.02913 (2019).
53. Rafael Navarro, Pablo Artal, and David R Williams. 1993. Modulation transfer of the human eye as a function of retinal eccentricity. JOSA A 10, 2 (1993), 201--212.
54. Aya Nozaki, Masaya Mitobe, Fumio Okuyama, and Yuji Sakamoto. 2017. Dynamic visual responses of accommodation and vergence to electro-holographic images. Optics Express 25, 4 (2017), 4542--4551.
55. Ryuichi Ohara, Masanobu Kurita, Takuo Yoneyama, Fumio Okuyama, and Yuji Sakamoto. 2015. Response of accommodation and vergence to electro-holographic images. Applied Optics 54, 4 (2015), 615--621.
56. International Commission on Non-Ionizing Radiation Protection et al. 1996. Guidelines on limits of exposure to laser radiation of wavelengths between 180 nm and 1,000 μm. Health Physics 71, 5 (1996), 804--819.
57. DA Owens. 1980. A comparison of accommodative responsiveness and contrast sensitivity for sinusoidal gratings. Vision Research 20, 2 (1980), 159--167.
58. Nitish Padmanaban, Robert Konrad, Tal Stramer, Emily A Cooper, and Gordon Wetzstein. 2017. Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays. Proceedings of the National Academy of Sciences 114, 9 (2017), 2183--2188.
59. Jongchan Park, KyeoReh Lee, and YongKeun Park. 2019. Ultrathin wide-angle large-area digital 3D holographic display using a non-periodic photon sieve. Nature Communications 10, 1 (2019), 1--8.
60. Yifan Peng, Suyeon Choi, Jonghyun Kim, and Gordon Wetzstein. 2021. Speckle-free holography with partially coherent light sources and camera-in-the-loop calibration. Science Advances 7, 46 (2021), eabg5040.
61. Yifan Peng, Suyeon Choi, Nitish Padmanaban, and Gordon Wetzstein. 2020. Neural Holography with Camera-in-the-Loop Training. ACM Trans. Graph. 39, 6, Article 185 (nov 2020), 14 pages.
62. Maria Perez-Ortiz and Rafal K Mantiuk. 2017. A practical guide and software for analysing pairwise comparison experiments. arXiv preprint arXiv:1712.03686 (2017).
63. Sowmya Ravikumar, Kurt Akeley, and Martin S Banks. 2011. Creating effective focus cues in multi-plane 3D displays. Optics Express 19, 21 (2011), 20940--20952.
64. Stephan Reichelt, Ralf Häussler, Gerald Fütterer, and Norbert Leister. 2010. Depth cues in human visual perception and their realization in 3D displays. In Three-Dimensional Imaging, Visualization, and Display 2010 and Display Technologies and Applications for Defense, Security, and Avionics IV, Vol. 7690. International Society for Optics and Photonics, 76900B.
65. Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer, and Wojciech Matusik. 2021. Towards real-time photorealistic 3D holography with deep neural networks. Nature 591, 7849 (2021), 234--239.
66. Yasuhiro Takaki and Naoya Okada. 2009. Hologram generation by horizontal scanning of a high-speed spatial light modulator. Applied Optics 48, 17 (2009), 3255--3260.
67. Yasuhiro Takaki and Masahito Yokouchi. 2012. Accommodation measurements of horizontally scanning holographic display. Optics Express 20, 4 (2012), 3918--3931.
68. Larry N Thibos, Xin Hong, Arthur Bradley, and Xu Cheng. 2002. Statistical variation of aberration structure and image quality in a normal population of healthy eyes. JOSA A 19, 12 (2002), 2329--2348.
69. Christopher W Tyler and Russell D Hamer. 1990. Analysis of visual modulation sensitivity. IV. Validity of the Ferry-Porter law. JOSA A 7, 4 (1990), 743--758.
70. Koki Wakunami, Po-Yuan Hsieh, Ryutaro Oi, Takanori Senoh, Hisayuki Sasaki, Yasuyuki Ichihashi, Makoto Okui, Yi-Pai Huang, and Kenji Yamamoto. 2016. Projection-type see-through holographic three-dimensional display. Nature Communications 7, 1 (2016), 1--7.
71. Andrew B Watson. 2013. A formula for the mean human optical modulation transfer function as a function of pupil size. Journal of Vision 13, 6 (2013), 18--18.
72. Gordon Wetzstein, Douglas R Lanman, Matthew Waggener Hirsch, and Ramesh Raskar. 2012. Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. (2012).
73. Han-Ju Yeom, Hee-Jae Kim, Seong-Bok Kim, HuiJun Zhang, BoNi Li, Yeong-Min Ji, Sang-Hoo Kim, and Jae-Hyeung Park. 2015. 3D holographic head mounted display using holographic optical elements with astigmatism aberration compensation. Optics Express 23, 25 (2015), 32025--32034.