Abstract
We present a new algorithm for image-based simplification of complex scenes for virtual reality (VR). The algorithm transforms geometrically-detailed environments into a layered quad tile representation that is optimized for a specified viewing region and is renderable on low-power mobile-class VR devices. A novel constrained optimization formulation ensures that the scene can be rendered within a predetermined compute budget, with limits on both primitive count and fill rate. Furthermore, we introduce a new method for texturing from point samples of the original scene geometry that generates high-quality silhouettes without the drawbacks of traditional point splatting.
The resulting representation achieves a visual fidelity that was previously impossible on mobile graphics hardware; our algorithm can typically generate a high-quality representation of visually-rich scenes with billions of triangles using just 72k triangles and a single high-resolution texture map (with generally only about 50% more texels than a stereo panorama). The effectiveness of the approach is demonstrated with a set of challenging test cases.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, View-Region Optimized Image-Based Scene Simplification
- Sameer Agarwal, Keir Mierle, and Others. 2018. Ceres Solver. (2018). http://ceres-solver.orgGoogle Scholar
- Alcon Interactive. 2018. Blade Runner: Revelations. (2018). https://play.google.com/store/apps/details?id=com.alconinteractive.brvrGoogle Scholar
- Robert Anderson, David Gallup, Jonathan T. Barron, Janne Kontkanen, Noah Snavely, Carlos Hernández, Sameer Agarwal, and Steven M. Seitz. 2016. Jump: Virtual Reality Video. ACM Trans. Graph. 35, 6, Article 198, 13 pages. Google Scholar
Digital Library
- C. Andujar, P. Brunet, A. Chica, I. Navazo, J. Rossignac, and A. Vinacua. 2004. Computing Maximal Tiles and Application to Impostor-Based Simplification. Computer Graphics Forum 23, 3 (2004), 401--410.Google Scholar
Cross Ref
- S. Behrendt, C. Colditz, O. Franzke, J. Kopf, and O. Deussen. 2005. Realistic real-time rendering of landscapes using billboard clouds. Computer Graphics Forum (2005).Google Scholar
- Marius Bjørge, Sam Martin, Sandeep Kakarlapudi, and Jan-Harald Fredriksen. 2014. Efficient Rendering with Tile Local Storage. In ACM SIGGRAPH 2014 Talks (SIGGRAPH '14). Article 51, 1 pages. Google Scholar
Digital Library
- Kevin Boos, David Chu, and Eduardo Cuervo. 2016. FlashBack: Immersive Virtual Reality on Mobile Devices via Rendering Memoization. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '16). 291--304. Google Scholar
Digital Library
- David Cohen-Steiner, Pierre Alliez, and Mathieu Desbrun. 2004. Variational Shape Approximation. ACM Trans. Graph. 23, 3 (Aug. 2004), 905--914. Google Scholar
Digital Library
- Cyril Crassin, Fabrice Neyret, Sylvain Lefebvre, and Elmar Eisemann. 2009. GigaVoxels: Ray-guided Streaming for Efficient and Detailed Voxel Rendering. In Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games (I3D '09). 15--22. Google Scholar
Digital Library
- Philippe Decaudin and Fabrice Neyret. 2009. Volumetric Billboards. Computer Graphics Forum 28, 8 (2009), 2079--2089.Google Scholar
- Xavier Décoret, Frédo Durand, François X. Sillion, and Julie Dorsey. 2003. Billboard Clouds for Extreme Model Simplification. ACM Trans. Graph. 22, 3 (July 2003), 689--696. Google Scholar
Digital Library
- Hugh Everett. 1963. Generalized Lagrange Multiplier Method for Solving Problems of Optimum Allocation of Resources. Operations Research 11, 3 (1963), 399--417. Google Scholar
Digital Library
- Marshall L. Fisher. 2004. The Lagrangian Relaxation Method for Solving Integer Programming Problems. Management Science 50, 12 (2004), 1861--1871. Google Scholar
Digital Library
- Anton L. Fuhrmann, Eike Umlauf, and Stephan Mantler. 2005. Extreme Model Simplification for Forest Rendering. In Proceedings of the First Eurographics Conference on Natural Phenomena (NPH'05). Eurographics Association, 57--67. Google Scholar
Digital Library
- Michael Garland and Paul S. Heckbert. 1997. Surface Simplification Using Quadric Error Metrics. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). 209--216. Google Scholar
Digital Library
- Enrico Gobbetti and Fabio Marton. 2005. Far Voxels: A Multiresolution Framework for Interactive Rendering of Huge Complex 3D Models on Commodity Graphics Platforms. In ACM SIGGRAPH 2005 Papers (SIGGRAPH '05). 878--885. Google Scholar
Digital Library
- Google Inc. 2018. Expeditions. (2018). https://play.google.com/store/apps/details?id=com.google.vr.expeditionsGoogle Scholar
- Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The Lumigraph. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96). 43--54. Google Scholar
Digital Library
- J. Huang, Z. Chen, D. Ceylan, and H. Jin. 2017. 6-DOF VR videos with a single 360-camera. In 2017 IEEE Virtual Reality (VR). 37--44.Google Scholar
- H. Ishiguro, M. Yamamoto, and S. Tsuji. 1990. Omni-directional stereo for making global map. In Proceedings Third International Conference on Computer Vision. 540--547.Google Scholar
- Stefan Jeschke and Michael Wimmer. 2002. Textured Depth Meshes for Real-time Rendering of Arbitrary Scenes. In Eurographics Workshop on Rendering, P. Debevec and S. Gibson (Eds.). The Eurographics Association. Google Scholar
Digital Library
- Alexander Kharlamov, Iain Cantlay, and Yury Stepanenko. 2007. Next-Generation SpeedTree Rendering. In GPU Gems 3 (first ed.), Hubert Nguyen (Ed.). Addison-Wesley Professional.Google Scholar
- Babis Koniaris, Ivan Huerta, Maggie Kosek Karen Darragh, Charles Malleson, Joanna Jamrozy, Nick Swafford, Jose Guitian, Bochang Moon, Ali Israr, and Kenny Mitchell. 2016. IRIDiuM: Immersive Rendered Interactive Deep Media. In ACM SIGGRAPH 2016 VR Village (SIGGRAPH '16). Article 11, 2 pages. Google Scholar
Digital Library
- Babis Koniaris, Maggie Kosek, David Sinclair, and Kenny Mitchell. 2017. Real-time Rendering with Compressed Animated Light Fields. In Proceedings of the 43rd Graphics Interface Conference (GI '17). 33--40. Google Scholar
Digital Library
- Martin Kraus and Thomas Ertl. 2002. Adaptive Texture Maps. In Proceedings of the ACM SIGGRAPH/EUROGRAPHICS Conference on Graphics Hardware (HWWS '02). Eurographics Association, 7--15. Google Scholar
Digital Library
- K.N. Kutulakos and S.M. Seitz. 1999. A Theory of Shape by Space Carving. In The Proceedings of the Seventh IEEE International Conference on Computer Vision. 307--314.Google Scholar
- Dylan Lacewell, David Edwards, Peter Shirley, and William B. Thompson. 2006. Stochastic Billboard Clouds for Interactive Foliage Rendering. Journal of Graphics Tools 11 (1 2006), 1--12.Google Scholar
- Marc Levoy and Pat Hanrahan. 1996. Light Field Rendering. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96). 31--42. Google Scholar
Digital Library
- Bruno Lévy, Sylvain Petitjean, Nicolas Ray, and Jérome Maillot. 2002. Least Squares Conformal Maps for Automatic Texture Atlas Generation. ACM Trans. Graph. 21, 3 (July 2002), 362--371. Google Scholar
Digital Library
- A. Lodi, S. Martello, and D. Vigo. 2004. Models and Bounds for Two-Dimensional Level Packing Problems. Journal of Combinatorial Optimization 8, 3 (9 2004), 363--379.Google Scholar
Cross Ref
- B. Luo, F. Xu, C. Richardt, and J. H. Yong. 2018. Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax. IEEE Transactions on Visualization and Computer Graphics (2018). Google Scholar
Digital Library
- Manufactura K4. 2015. Asia -- Far East Environment. https://assetstore.unity.com/packages/3d/environments/asia-far-east-environment-21298. (2015). Accessed on 2018-01-22.Google Scholar
- Harald Niederreiter. 1992. Random Number Generation and Quasi-Monte Carlo Methods. Society for Industrial and Applied Mathematics. Google Scholar
Digital Library
- J. Nystad, A. Lassen, A. Pomianowski, S. Ellis, and T. Olson. 2012. Adaptive Scalable Texture Compression. In Proceedings of the Fourth ACM SIGGRAPH/Eurographics Conference on High-Performance Graphics (EGGH-HPG'12). Eurographics Association, 105--114. Google Scholar
Digital Library
- S. Peleg, M. Ben-Ezra, and Y. Pritch. 2001. Omnistereo: panoramic stereo imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 3 (Mar 2001), 279--290. Google Scholar
Digital Library
- Eric Penner and Li Zhang. 2017. Soft 3D Reconstruction for View Synthesis. ACM Trans. Graph. 36, 6, Article 235 (Nov. 2017), 11 pages. Google Scholar
Digital Library
- Hanspeter Pfister, Matthias Zwicker, Jeroen van Baar, and Markus Gross. 2000. Surfels: Surface Elements As Rendering Primitives. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '00). 335--342. Google Scholar
Digital Library
- Matt Pharr. 2017. Mobile VR: Challenges and Opportunities. (2017). http://pharr.org/matt/mobilevr.pdf ACM I3D Keynote.Google Scholar
- Thomas Porter and Tom Duff. 1984. Compositing Digital Images. SIGGRAPH Comput. Graph. 18, 3 (Jan. 1984), 253--259. Google Scholar
Digital Library
- Liu Ren, Hanspeter Pfister, and Matthias Zwicker. 2002. Object Space EWA Surface Splatting: A Hardware Accelerated Approach to High Quality Point Rendering. Computer Graphics Forum 21, 3 (2002), 461--470.Google Scholar
Cross Ref
- C. Richardt, Y. Pritch, H. Zimmer, and A. Sorkine-Hornung. 2013. Megastereo: Constructing High-Resolution Stereo Panoramas. In 2013 IEEE Conference on Computer Vision and Pattern Recognition. 1256--1263. Google Scholar
Digital Library
- Takafumi Saito and Tokiichiro Takahashi. 1991. NC Machining with G-buffer Method. In Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '91). 207--216. Google Scholar
Digital Library
- Gernot Schaufler. 1998. Image-based Object Representation by Layered Impostors. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '98). 99--104. Google Scholar
Digital Library
- Mark Segal, Carl Korobkin, Rolf van Widenfelt, Jim Foran, and Paul Haeberli. 1992. Fast Shadows and Lighting Effects Using Texture Mapping. In Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '92). 249--252. Google Scholar
Digital Library
- Jonathan Shade, Steven Gortler, Li-Wei He, and Richard Szeliski. 1998. Layered Depth Images. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '98). 231--242. Google Scholar
Digital Library
- Unity Technologies. 2015. The Blacksmith Environments. https://assetstore.unity.com/packages/3d/environments/asia-far-east-environment-21298. (2015). Accessed on 2018-01-22.Google Scholar
- Alex Vlachos. 2016. Advanced VR Rendering Performance. (2016). https://alex.vlachos.com/graphics/Alex_Vlachos_Advanced_VR_Rendering_Performance_GDC2016.pdf GDC Talk.Google Scholar
- Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. Trans. Img. Proc. 13, 4 (April 2004), 600--612. Google Scholar
Digital Library
- Andrew Wilson and Dinesh Manocha. 2003. Simplifying Complex Environments Using Incremental Textured Depth Meshes. In ACM SIGGRAPH 2003 Papers (SIGGRAPH '03). 678--688. Google Scholar
Digital Library
- Daniel N. Wood, Daniel I. Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H. Salesin, and Werner Stuetzle. 2000. Surface Light Fields for 3D Photography. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '00). 287--296. Google Scholar
Digital Library
- Matthias Zwicker, Hanspeter Pfister, Jeroen van Baar, and Markus Gross. 2001. Surface Splatting. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01). 371--378. Google Scholar
Digital Library
Index Terms
View-Region Optimized Image-Based Scene Simplification
Recommendations
Simplification of Moving 3D Scene Data on GPU
ICETE 2016: Proceedings of the 13th International Joint Conference on e-Business and TelecommunicationsReal-time large scale continuous image and geometry based data visualization, with an uninterrupted content
delivery, quality and rendering, on home and mobile devices is difficult or even mostly impossible because of
the low processing capabilities of ...
Image-based bidirectional scene reprojection
We introduce a method for increasing the framerate of real-time rendering applications. Whereas many existing temporal upsampling strategies only reuse information from previous frames, our bidirectional technique reconstructs intermediate frames from a ...
Physically-Based Editing of Indoor Scene Lighting from a Single Image
Computer Vision – ECCV 2022AbstractWe present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. This is an extremely challenging problem that requires modeling complex light transport, and disentangling HDR ...






Comments