An Omnistereoscopic Video Pipeline for Capture and Display of Real-World VR

Abstract

In this article, we describe a complete pipeline for the capture and display of real-world Virtual Reality video content, based on the concept of omnistereoscopic panoramas. We address important practical and theoretical issues that have remained undiscussed in previous works. On the capture side, we show how high-quality omnistereo video can be generated from a sparse set of cameras (16 in our prototype array) instead of the hundreds of input views previously required. Despite the sparse number of input views, our approach allows for high quality, real-time virtual head motion, thereby providing an important additional cue for immersive depth perception compared to static stereoscopic video. We also provide an in-depth analysis of the required camera array geometry in order to meet specific stereoscopic output constraints, which is fundamental for achieving a plausible and fully controlled VR viewing experience. Finally, we describe additional insights on how to integrate omnistereo video panoramas with rendered CG content. We provide qualitative comparisons to alternative solutions, including depth-based view synthesis and the Facebook Surround 360 system. In summary, this article provides a first complete guide and analysis for reimplementing a system for capturing and displaying real-world VR, which we demonstrate on several real-world examples captured with our prototype.

References

1. Aseem Agarwala, Maneesh Agrawala, Michael Cohen, David Salesin, and Richard Szeliski. 2006. Photographing long scenes with multi-viewpoint panoramas. TOG (SIGGRAPH) 25, 3 (2006), 853--861.
2. Robert Anderson, David Gallup, Jonathan T. Barron, Janne Kontkanen, Noah Snavely, Carlos Hernandez Esteban, Sameer Agarwal, and Steven M. Seitz. 2016. Jump: Virtual reality video. TOG (SIGGRAPH Asia) 35, 6 (2016), 198:1--198:13. http://dl.acm.org/citation.cfm?id=2980257.
3. Clemens Birklbauer and Oliver Bimber. 2014. Panorama light-field imaging. CGF (Eurographics) 33, 2 (2014), 43--52.
4. Matthew Brown and David G. Lowe. 2007. Automatic panoramic image stitching using invariant features. IJCV 74, 1 (2007), 59--73.
5. Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert. 2004. High accuracy optical flow estimation based on a theory for warping. In ECCV. 25--36.
6. Vincent Chapdelaine-Couture and Sébastien Roy. 2013. The omnipolar camera: A new approach to stereo immersive capture. In ICCP. 1--9.
7. Vincent Couture, Michael S. Langer, and Sébastien Roy. 2011. Panoramic stereo video textures. In ICCV. 1251--1258.
8. Alexei A. Efros and William T. Freeman. 2001. Image quilting for texture synthesis and transfer. In SIGGRAPH. 341--346. http://portal.acm.org/citation.cfm?id=383259.383296
9. Simon Fuhrmann, Fabian Langguth, and Michael Goesele. 2014. MVE -- A multi-view reconstruction environment. In Eurographics Workshop on Graphics and Cultural Heritage.
10. Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multiview stereopsis. TPAMI 32, 8 (2010), 1362--1376.
11. Silvano Galliani, Katrin Lasinger, and Konrad Schindler. 2015. Massively parallel multiview stereopsis by surface normal diffusion. In ICCV. 873--881.
12. Rajiv Gupta and Richard I. Hartley. 1997. Linear pushbroom cameras. TPAMI 19, 9 (1997), 963--975.
13. Richard Hartley and Andrew Zisserman. 2004. Multiple View Geometry in Computer Vision. Cambridge University Press.
14. Peter Hedman, Suhib Alsisan, Richard Szeliski, and Johannes Kopf. 2017. Casual 3D photography. TOG (SIGGRAPH Asia) 36, 6 (2017), 234:1--234:15.
15. Heiko Hirschmüller. 2008. Stereo processing by semiglobal matching and mutual information. TPAMI 30, 2 (2008), 328--341.
16. Hiroshi Ishiguro, Masashi Yamamoto, and Saburo Tsuji. 1992. Omni-directional stereo. TPAMI 14, 2 (1992), 257--262.
17. Jiaya Jia and Chi-Keung Tang. 2008. Image stitching using structure deformation. TPAMI 30, 4 (2008), 617--631.
18. Sing Bing Kang, Richard Szeliski, and Matthew Uyttendaele. 2004. Seamless Stitching using Multi-Perspective Plane Sweep. Technical Report MSR-TR-2004-48. Microsoft Research.
19. Johannes Kopf, Billy Chen, Richard Szeliski, and Michael Cohen. 2010. Street slide: Browsing street level imagery. TOG (SIGGRAPH) 29, 4 (2010), 96:1--8.
20. Johannes Kopf, Matthew Uyttendaele, Oliver Deussen, and Michael F. Cohen. 2007. Capturing and viewing gigapixel images. TOG (SIGGRAPH) 26, 3 (2007), 93.
21. Jungjin Lee, Bumki Kim, Kyehyun Kim, Younghui Kim, and Jun-yong Noh. 2016. Rich360: Optimized spherical representation from structured panoramic camera arrays. TOG (SIGGRAPH) 35, 4 (2016), 63.
22. Kevin Matzen, Michael F. Cohen, Bryce Evans, Johannes Kopf, and Richard Szeliski. 2017. Low-cost 360 stereo photography and video capture. ACM Trans. Graph. 36, 4, Article 148 (2017), 12 pages.
23. Shmuel Peleg and Moshe Ben-Ezra. 1999. Stereo panorama with a single camera. In CVPR. 1395--1401.
24. Shmuel Peleg, Moshe Ben-Ezra, and Yael Pritch. 2001. Omnistereo: Panoramic stereo imaging. TPAMI 23, 3 (2001), 279--290.
25. Federico Perazzi, Alexander Sorkine-Hornung, Henning Zimmer, Peter Kaufmann, Oliver Wang, S. Watson, and Markus H. Gross. 2015. Panoramic video from unstructured camera arrays. CGF (Eurographics) 34, 2 (2015), 57--68.
26. Alex Rav-Acha, Giora Engel, and Shmuel Peleg. 2008. Minimal aspect distortion (MAD) mosaicing of long scenes. IJCV 78, 2--3 (2008), 187--206.
27. Christian Richardt, Yael Pritch, Henning Zimmer, and Alexander Sorkine-Hornung. 2013. Megastereo: Constructing high-resolution stereo panoramas. In CVPR. 1256--1263.
28. Augusto Román and Hendrik P. A. Lensch. 2006. Automatic multiperspective images. In Eurographics Symposium on Rendering Techniques (EGSR). 83--92.
29. Steven M. Seitz and Jiwon Kim. 2002. The space of all stereo images. IJCV 48, 1 (2002), 21--38.
30. Heung-Yeung Shum and Li-wei He. 1999. Rendering with concentric mosaics. In SIGGRAPH. 299--306.
31. Harry Shum and Sing Bing Kang. 2000. Review of image-based rendering techniques. In Visual Communications and Image Processing. 2--13.
32. Heung-Yeung Shum, King To Ng, and Shing-Chow Chan. 2005. A virtual reality system using the concentric mosaic: Construction, rendering, and data compression. IEEE Transactions on Multimedia 7, 1 (2005), 85--95.
33. Heung-Yeung Shum and Richard Szeliski. 2000. Systems and experiment paper: Construction of panoramic image mosaics with global and local alignment. IJCV 36, 2 (2000), 101--130.
34. Andreas Simon, Randall C. Smith, and Richard R. Pawlicki. 2004. Omnistereo for panoramic virtual environment display systems. In IEEE VR. 67--74.
35. Richard Szeliski. 2006. Image alignment and stitching: A tutorial. Foundations and Trends in Computer Graphics and Vision 2, 1 (2006), 1--104.
36. Fan Zhang and Feng Liu. 2014. Parallax-tolerant image stitching. In CVPR. 3262--3269.
37. Fan Zhang and Feng Liu. 2015. Casual stereoscopic panorama stitching. In CVPR. 2002--2010.
38. Guofeng Zhang, Jiaya Jia, Tien-Tsin Wong, and Hujun Bao. 2009. Consistent depth maps recovery from a video sequence. TPAMI 31, 6 (2009), 974--988.
39. Zhengyou Zhang. 2000. A flexible new technique for camera calibration. TPAMI 22, 11 (2000), 1330--1334.
40. Enliang Zheng, Rahul Raguram, Pierre Fite Georgel, and Jan-Michael Frahm. 2011. Efficient generation of multi-perspective panoramas. In International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT). 86--92.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • Computational photography