Iterative Depth Warping

Abstract

This article presents an iterative backward-warping technique and its applications. It predictively synthesizes depth buffers for novel views. Our solution is based on a fixed-point iteration that converges quickly in practice. Unlike the previous techniques, our solution is a pure backward warping without using bidirectional sources. To efficiently seed the iterative process, we also propose a tight bounding method for motion vectors. Non-convergent depth holes are inpainted via deep depth buffers. Our solution works well with arbitrarily distributed motion vectors under moderate motions. Many scenarios can benefit from our depth warping. As an application, we propose a highly scalable image-based occlusion-culling technique, achieving a significant speed-up compared to the state of the art. We also demonstrate the benefit of our solution in multi-view soft-shadow generation.

References

1. Dmitry Andreev. 2010. Real-time frame rate up-conversion for video games: Or how to get from 30 to 60 fps for free. In Proceedings of the ACM SIGGRAPH 2010 Talks. ACM, 16.
2. Thaddeus Beier and Shawn Neely. 1992. Feature-based image metamorphosis. Proc. ACM SIGGRAPH 26, 2 (1992), 35--42.
3. Jiří Bittner, Michael Wimmer, Harald Piringer, and Werner Purgathofer. 2004. Coherent hierarchical culling: Hardware occlusion queries made useful. Comput. Graph. Forum 23, 3 (2004), 615--624.
4. Huw Bowles, Kenny Mitchell, Robert W. Sumner, Jeremy Moore, and Markus Gross. 2012. Iterative image warping. Comput. Graph. Forum 31, 2pt1 (2012), 237--246.
5. Shenchang Eric Chen and Lance Williams. 1993. View interpolation for image synthesis. In Proceedings of the ACM SIGGRAPH. ACM, 279--288.
6. Adam Cichocki. 2017. Advances in Real-Time Rendering in Games. ACM SIGGRAPH Courses.
7. R. Cunniff, M. Craighead, D. Ginsburg, K. Lefebvre, B. Licea-Kane, and N. Triantos. 2001. ARB Occlusion Query. Technical Report. NVIDIA and ATI.
8. X. Décoret. 2005. N-buffers for efficient depth map query. In Proceedings of the Eurographics Conference. Wiley Online Library, 393--400.
9. Piotr Didyk, Elmar Eisemann, Tobias Ritschel, Karol Myszkowski, and Hans-Peter Seidel. 2010a. Perceptually-motivated real-time temporal upsampling of 3D content for high-refresh-rate displays. Comput. Graph. Forum 29, 2 (2010), 713--722.
10. Piotr Didyk, Tobias Ritschel, Elmar Eisemann, Karol Myszkowski, and Hans-Peter Seidel. 2010b. Adaptive image-space stereo view synthesis. In Proceedings of the Conference on Vision, Modeling, and Visualization. 299--306.
11. Hawar Doghramachi and Jean-Normand Bucci. 2017. Deferred+: Next-Gen culling and rendering for dawn engine. In GPU Zen: Advanced Rendering Techniques, Wolfgang Engel (Ed.). Bowker Identifier Services.
12. Elmar Eisemann and Xavier Décoret. 2007. Visibility sampling on GPU and applications. Compu. Graph. Forum 26, 3 (2007), 535--544.
13. Cass Everitt. 2001. Interactive Order-Independent Transparency. NVIDIA Corporation.
14. Randima Fernando. 2005. Percentage-closer soft shadows. In ACM SIGGRAPH 2005 Sketches. ACM, 35.
15. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The lumigraph. In Proceedings of the ACM SIGGRAPH Conference. ACM, 43--54.
16. Stefan Gottschalk, Ming C. Lin, and Dinesh Manocha. 1996. OBBTree: A hierarchical structure for rapid interference detection. In Proceedings of the ACM SIGGRAPH Conference. ACM, 171--180.
17. Michael Guthe, Ákos Balázs, and Reinhard Klein. 2006. Near optimal hierarchical culling: Performance driven use of hardware occlusion queries. In Proceedings of the Eurographics Symposium on Rendering. 207--214.
18. Ulrich Haar and Sebastian Aaltonen. 2015. Advances in Real-Time Rendering in Games. ACM SIGGRAPH Courses.
19. Yun-Suk Kang and Yo-Sung Ho. 2010. High-quality multi-view depth generation using multiple color and depth cameras. In Proceedings of the International Conference on Multimedia and Expo. IEEE, 1405--1410.
20. Nickolay Kasyan, Nicolas Schulz, and Tiago Sousa. 2011. Secrets of CryENGINE 3 Graphics Technology. ACM SIGGRAPH Courses.
21. James T. Klosowski and Claudio T. Silva. 2000. The prioritized-layered projection algorithm for visible set estimation. IEEE Trans. Vis. Comput. Graph. 6, 2 (2000), 108--123.
22. Christoph Kubisch and Markus Tavenrath. 2014. OpenGL 4.4 Scene Rendering Techniques. GPU Technology Conference.
23. Sungkil Lee, Elmar Eisemann, and Hans-Peter Seidel. 2010. Real-time lens blur effects and focus control. ACM Trans. Graph. 29, 4 (2010), 65:1--7.
24. Sungkil Lee, Gerard Jounghyun Kim, and Seungmoon Choi. 2008. Real-time depth-of-field rendering using point splatting on per-pixel layers. Comput. Graph. Forum 27, 7 (2008), 1955--1962.
25. Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the ACM SIGGRAPH Conference. ACM, 31--42.
26. David Luebke and Chris Georges. 1995. Portals and mirrors: Simple, fast evaluation of potentially visible sets. In Proceedings of the ACM Symposium on Interactive 3D Graphics. ACM, 105--ff.
27. William R. Mark, Leonard McMillan, and Gary Bishop. 1997. Post-rendering 3D warping. In Proceedings of the Symposium on Interactive 3D Graphics. ACM, 7--ff.
28. Oliver Mattausch, Jiří Bittner, A. Jaspe, E. Gobbetti, Michael Wimmer, and Renato Pajarola. 2015. CHC+RT: Coherent hierarchical culling for ray tracing. Comput. Graph. Forum 34, 2 (2015), 537--548.
29. Oliver Mattausch, Jiří Bittner, and Michael Wimmer. 2008. CHC++: Coherent hierarchical culling revisited. Comput. Graph. Forum 27, 2 (2008), 221--230.
30. Oliver Mattausch, Daniel Scherzer, and Michael Wimmer. 2010. High-quality screen-space ambient occlusion using temporal coherence. Comput. Graph. Forum 29, 8 (2010), 2492--2503.
31. Leonard McMillan and Gary Bishop. 1995. Plenoptic modeling: An image-based rendering system. In Proceedings of the ACM SIGGRAPH Conference. ACM, 39--46.
32. Leonard McMillan Jr. 1997. An Image-Based Approach to Three-dimensional Computer Graphics. Ph.D. Dissertation. University of North Carolina Chapel Hill.
33. Diego Nehab, Pedro V. Sander, Jason Lawrence, Natalya Tatarchuk, and John R. Isidoro. 2007. Accelerating real-time shading with reverse reprojection caching. In Proceedings of the Conference on Graphics Hardware, Vol. 41. 61--62.
34. Ivan Neulander. 2008. Pismo: parallax-interpolated shadow map occlusion. ACM SIGGRAPH 2008 talks.
35. Matthias Nießner and Charles Loop. 2012. Patch-based occlusion culling for hardware tessellation. In Proceedings of the Computer Graphics International.
36. Christoph Peters and Reinhard Klein. 2015. Moment shadow mapping. In Proceedings of the Symposium on Interactive 3D Graphics and Games. ACM, 7--14.
37. Gilberto Rosado and Rainbow Studios. 2007. Motion blur as a post-processing effect. In GPU Gems 3, Hubert Nguyen (Ed.). Addison-Wesley Professional, 69--117.
38. Daniel Scherzer, Stefan Jeschke, and Michael Wimmer. 2007. Pixel-correct shadow maps with temporal reprojection and shadow test confidence. In Proceedings of the Eurographics Symposium on Rendering. Eurographics Association, 45--50.
39. Daniel Scherzer, Michael Schwärzler, Oliver Mattausch, and Michael Wimmer. 2009. Real-time soft shadows using temporal coherence. In Proceedings of the International Symposium on Visual Computing. Springer, 13--24.
40. Daniel Scherzer, Lei Yang, Oliver Mattausch, Diego Nehab, Pedro V. Sander, Michael Wimmer, and Elmar Eisemann. 2012. Temporal coherence methods in real-time rendering. Comput. Graph. Forum 31, 8 (2012), 2378--2408.
41. Michael Schwarz and Marc Stamminger. 2008. Quality scalability of soft shadow mapping. In Graphics Interface 2008. 147--154.
42. W Rrenl Seales, Greg Welch, and Christopher O. Jaynes. 1999. Real-time depth warping for 3-d scene reconstruction. In Proceedings of the IEEE Aerospace Conference, Vol. 3. IEEE, 413--419.
43. S. M. Seitz and C. R. Dyer. 1996. View morphing. In Proceedings of the ACM SIGGRAPH Conference. ACM, 21--30.
44. Li Shen, Jieqing Feng, and Baoguang Yang. 2013. Exponential soft shadow mapping. Comput. Graph. Forum 32, 4 (2013), 107--116.
45. Erik Sintorn, Elmar Eisemann, and Ulf Assarsson. 2008. Sample based visibility for soft shadows using alias-free shadow maps. Comput. Graph. Forum 27, 4 (2008), 1285--1292.
46. Pitchaya Sitthi-amorn, Jason Lawrence, Lei Yang, Pedro V. Sander, and Diego Nehab. 2008. An improved shading cache for modern GPUs. In Proceedings of the Symposium on Graphics Hardware. Eurographics Association, 95--101.
47. Sundar Vedula, Simon Baker, and Takeo Kanade. 2002. Spatio-temporal view interpolation. In Proceedings of the Rendering Techniques. 65--76.
48. Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Proc. 13, 4 (2004), 600--612.
49. Lance Williams. 1983. Pyramidal parametrics. Proc. ACM SIGGRAPH 17, 3 (1983), 1--11.
50. Peter Wonka, Michael Wimmer, and Dieter Schmalstieg. 2000. Visibility preprocessing with occluder fusion for urban walkthroughs. In Proceedings of the Eurographics Workshop on Rendering. 71--82.
51. Baoguang Yang, Zhao Dong, Jieqing Feng, Hans-Peter Seidel, and Jan Kautz. 2010. Variance soft shadow mapping. Comput. Graph. Forum 29, 7 (2010), 2127--2134.
52. Lei Yang, Yu-Chiu Tse, Pedro V. Sander, Jason Lawrence, Diego Nehab, Hugues Hoppe, and Clara L Wilkins. 2011. Image-based bidirectional scene reprojection. ACM Trans. Graph. 30, 6 (2011), 150:1--10.
53. Xuan Yu, Rui Wang, and Jingyi Yu. 2010. Real-time depth of field rendering via dynamic light field generation and filtering. Comput. Graph. Forum 29, 7 (2010), 2099--2107.
54. Sveta Zinger, Luat Do, and P. H. N. de With. 2010. Free-viewpoint depth image based rendering. J. Vis. Commun. Image Represent. 21, 5 (2010), 533--541.
55. Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. 2002. EWA splatting. IEEE Trans. Vis. Comput. Graph. 8, 3 (2002), 223--238.

Index Terms

• Computing methodologies

  • Computer graphics

    • Rendering

      • Rasterization

      • Visibility