skip to main content
research-article

Hybrid Image-based Rendering for Free-view Synthesis

Authors Info & Claims
Published:28 April 2021Publication History
Skip Abstract Section

Abstract

Image-based rendering (IBR) provides a rich toolset for free-viewpoint navigation in captured scenes. Many methods exist, usually with an emphasis either on image quality or rendering speed. In this paper we identify common IBR artifacts and combine the strengths of different algorithms to strike a good balance in the speed/quality tradeoff. First, we address the problem of visible color seams that arise from blending casually-captured input images by explicitly treating view-dependent effects. Second, we compensate for geometric reconstruction errors by refining per-view information using a novel clustering and filtering approach. Finally, we devise a practical hybrid IBR algorithm, which locally identifies and utilizes the rendering method best suited for an image region while retaining interactive rates. We compare our method against classical and modern (neural) approaches in indoor and outdoor scenes and demonstrate superiority in quality and/or speed.

Skip Supplemental Material Section

Supplemental Material

References

  1. Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, and Michael Cohen. 2004. Interactive Digital Photomontage. In ACM SIGGRAPH 2004 Papers (Los Angeles, California) (SIGGRAPH '04). Association for Computing Machinery, New York, NY, USA, 294--302. https://doi.org/10.1145/1186562.1015718Google ScholarGoogle Scholar
  2. Sebastien Bonopera, Peter Hedman, Jerome Esnault, Siddhant Prakash, Simon Rodriguez, Theo Thonat, Mehdi Benadel, Gaurav Chaurasia, Julien Philip, and George Drettakis. 2020. sibr: A System for Image Based Rendering. https://sibr.gitlabpages.inria.fr/Google ScholarGoogle Scholar
  3. Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001. Unstructured Lumigraph Rendering. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01). Association for Computing Machinery, New York, NY, USA, 425--432. https://doi.org/10.1145/383259.383309Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. R. O. Cayon, A. Djelouah, and G. Drettakis. 2015. A Bayesian Approach for Selective Image-Based Rendering Using Superpixels. In 2015 International Conference on 3D Vision. 469--477. https://doi.org/10.1109/3DV.2015.59Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Gaurav Chaurasia, Sylvain Duchene, Olga Sorkine-Hornung, and George Drettakis. 2013. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics (TOG) 32, 3 (2013), 30.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Shenchang Eric Chen. 1995. QuickTime VR: An Image-Based Approach to Virtual Environment Navigation. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). Association for Computing Machinery, New York, NY, USA, 29--38. https://doi.org/10.1145/218380.218395Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Shenchang Eric Chen and Lance Williams. 1993. View Interpolation for Image Synthesis. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (Anaheim, CA) (SIGGRAPH '93). Association for Computing Machinery, New York, NY, USA, 279--288. https://doi.org/10.1145/166117.166153Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Paul Debevec, Yizhou Yu, and George Borshukov. 1998. Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping. In Rendering Techniques '98, George Drettakis and Nelson Max (Eds.). Springer Vienna, Vienna, 105--116.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. M. Eisemann, B. De Decker, M. Magnor, P. Bekaert, E. De Aguiar, N. Ahmed, C. Theobalt, and A. Sellent. 2008. Floating Textures. Computer Graphics Forum 27, 2 (2008), 409--418. https://doi.org/10.1111/j.1467--8659.2008.01138.x arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467--8659.2008.01138.xGoogle ScholarGoogle ScholarCross RefCross Ref
  10. John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. 2019. DeepView: View Synthesis With Learned Gradient Descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle ScholarCross RefCross Ref
  11. Michael Goesele, Jens Ackermann, Simon Fuhrmann, Carsten Haubold, Ronny Klowsky, Drew Steedly, and Richard Szeliski. 2010. Ambient Point Clouds for View Interpolation. In ACM SIGGRAPH 2010 Papers (Los Angeles, California) (SIGGRAPH '10). Association for Computing Machinery, New York, NY, USA, Article 95, 6 pages. https://doi.org/10.1145/1833349.1778832Google ScholarGoogle Scholar
  12. M. Goesele, N. Snavely, B. Curless, H. Hoppe, and S. M. Seitz. 2007. Multi-View Stereo for Community Photo Collections. In 2007 IEEE 11th International Conference on Computer Vision. 1--8. https://doi.org/10.1109/ICCV.2007.4408933Google ScholarGoogle Scholar
  13. Daniel B Goldman. 2010. Vignette and exposure calibration and compensation. IEEE transactions on pattern analysis and machine intelligence 32, 12 (2010), 2276--2288.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The Lumigraph. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96). Association for Computing Machinery, New York, NY, USA, 43--54. https://doi.org/10.1145/237170.237200Google ScholarGoogle Scholar
  15. Yoav HaCohen, Eli Shechtman, Dan B Goldman, and Dani Lischinski. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM transactions on graphics (TOG) 30, 4 (2011), 1--10.Google ScholarGoogle Scholar
  16. Yoav HaCohen, Eli Shechtman, Dan B Goldman, and Dani Lischinski. 2013. Optimizing color consistency in photo collections. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1--10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. 2018. Deep Blending for Free-Viewpoint Image-Based Rendering. ACM Trans. Graph. 37, 6, Article 257 (Dec. 2018), 15 pages. https://doi.org/10.1145/3272127.3275084Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel Brostow. 2016. Scalable inside-out image-based rendering. ACM Transactions on Graphics (TOG) 35, 6 (2016), 231.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Jingwei Huang, Angela Dai, Leonidas J Guibas, and Matthias Nießner. 2017. 3Dlite: towards commodity 3D scanning for content creation. ACM Trans. Graph. 36, 6 (2017), 203--1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. M. Jancosek and T. Pajdla. 2011. Multi-view reconstruction preserving weakly-supported surfaces. In CVPR 2011. 3121--3128. https://doi.org/10.1109/CVPR.2011.5995693Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. 2019. E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles. arXiv:1906.03973 [cs.CV]Google ScholarGoogle Scholar
  22. Seon Joo Kim and Marc Pollefeys. 2008. Robust radiometric calibration and vignetting correction. IEEE transactions on pattern analysis and machine intelligence 30, 4 (2008), 562--576.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017. Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction. ACM Trans. Graph. 36, 4, Article 78 (July 2017), 13 pages. https://doi.org/10.1145/3072959.3073599Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Vladlen Koltun. 2020. Towards Photorealism. (2020). https://youtu.be/Rd0nBO6--bM 42nd German Conference on Pattern Recognition (DAGM GCPR 2020), the 25th International Symposium on Vision, Modeling and Visualization (VMV 2020) and the 10th Eurographics Workshop on Visual Computing for Biology and Medicine (VCBM 2020).Google ScholarGoogle Scholar
  25. Johannes Kopf, Fabian Langguth, Daniel Scharstein, Richard Szeliski, and Michael Goesele. 2013. Image-based rendering in the gradient domain. ACM Transactions on Graphics (TOG) 32, 6 (2013), 199.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. 2003. Graphcut textures: image and video synthesis using graph cuts. ACM Transactions on Graphics (TOG) 22, 3 (2003), 277--286.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. V. Lempitsky and D. Ivanov. 2007. Seamless Mosaicing of Image-Based Texture Maps. In 2007 IEEE Conference on Computer Vision and Pattern Recognition. 1--6. https://doi.org/10.1109/CVPR.2007.383078Google ScholarGoogle Scholar
  28. Marc Levoy and Pat Hanrahan. 1996. Light Field Rendering. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96). Association for Computing Machinery, New York, NY, USA, 31--42. https://doi.org/10.1145/237170.237199Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural Volumes: Learning Dynamic Renderable Volumes from Images. ACM Trans. Graph. 38, 4, Article 65 (July 2019), 14 pages. https://doi.org/10.1145/3306346.3323020Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. A. Loza, L. Mihaylova, N. Canagarajah, and D. Bull. 2006. Structural Similarity-Based Object Tracking in Video Sequences. In 2006 9th International Conference on Information Fusion. 1--6. https://doi.org/10.1109/ICIF.2006.301574Google ScholarGoogle Scholar
  31. Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. 2021. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. arXiv:2008.02268 [cs.CV]Google ScholarGoogle Scholar
  32. Leonard McMillan and Gary Bishop. 1995. Plenoptic Modeling: An Image-Based Rendering System. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). Association for Computing Machinery, New York, NY, USA, 39--46. https://doi.org/10.1145/218380.218398Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Moustafa Meshry, Dan B. Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla. 2019. Neural Rerendering in the Wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle ScholarCross RefCross Ref
  34. Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. ACM Trans. Graph. 38, 4, Article 29 (July 2019), 14 pages. https://doi.org/10.1145/3306346.3322980Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Computer Vision - ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 405--421.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Pierre Moulon, Pascal Monasse, Romuald Perrot, and Renaud Marlet. 2017. OpenMVG: Open Multiple View Geometry. In Reproducible Research in Pattern Recognition, Bertrand Kerautret, Miguel Colom, and Pascal Monasse (Eds.). Springer International Publishing, Cham, 60--74.Google ScholarGoogle Scholar
  37. Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson Image Editing. In ACM SIGGRAPH 2003 Papers. Association for Computing Machinery, New York, NY, USA, 313--318. https://doi.org/10.1145/1201775.882269Google ScholarGoogle Scholar
  38. Capturing Reality. 2018. RealityCapture reconstruction software. https://www.capturingreality.com/Product.Google ScholarGoogle Scholar
  39. Gernot Riegler and Vladlen Koltun. 2020. Free View Synthesis. In Computer Vision - ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 623--640.Google ScholarGoogle ScholarCross RefCross Ref
  40. Simon Rodriguez, Siddhant Prakash, Peter Hedman, and George Drettakis. 2020. Image-Based Rendering of Cars Using Semantic Labels and Approximate Reflection Flow. Proc. ACM Comput. Graph. Interact. Tech. 3, 1, Article 6 (April 2020), 17 pages. https://doi.org/10.1145/3384535Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Johannes L. Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. 2016. Pixelwise View Selection for Unstructured Multi-View Stereo. In Computer Vision - ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 501--518.Google ScholarGoogle ScholarCross RefCross Ref
  42. Heung-Yeung Shum, Shing-Chow Chan, and Sing Bing Kang. 2008. Image-based rendering. Springer Science & Business Media.Google ScholarGoogle Scholar
  43. Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012. Image-based rendering for scenes with reflections. ACM Transactions on Graphics (TOG) 31, 4 (2012), 100--1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Niessner, Gordon Wetzstein, and Michael Zollhofer. 2019. DeepVoxels: Learning Persistent 3D Feature Embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle ScholarCross RefCross Ref
  45. Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2006. Photo Tourism: Exploring Photo Collections in 3D. In ACM SIGGRAPH 2006 Papers (Boston, Massachusetts) (SIGGRAPH '06). Association for Computing Machinery, New York, NY, USA, 835--846. https://doi.org/10.1145/1179352.1141964Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Pratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. 2019. Pushing the Boundaries of View Extrapolation With Multiplane Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle ScholarCross RefCross Ref
  47. A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualla, T. Simon, J. Saragih, M. Nießner, R. Pandey, S. Fanello, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, E. Shechtman, D. B Goldman, and M. Zollhöfer. 2020. State of the Art on Neural Rendering. Computer Graphics Forum 39, 2 (2020), 701--727. https://doi.org/10.1111/cgf.14022 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14022Google ScholarGoogle Scholar
  48. Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred Neural Rendering: Image Synthesis Using Neural Textures. ACM Trans. Graph. 38, 4, Article 66 (July 2019), 12 pages. https://doi.org/10.1145/3306346.3323035Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, and Matthias Nießner. 2018. IGNOR: Image-guided Neural Object Rendering. CoRR abs/1811.10720 (2018). arXiv:1811.10720 http://arxiv.org/abs/1811.10720Google ScholarGoogle Scholar
  50. Michael Waechter, Nils Moehrle, and Michael Goesele. 2014. Let There Be Color! Large-Scale Texturing of 3D Reconstructions. In Computer Vision - ECCV 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 836--850.Google ScholarGoogle ScholarCross RefCross Ref
  51. Edward Zhang, Michael F Cohen, and Brian Curless. 2016. Emptying, refurnishing, and relighting indoor spaces. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1--14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Qian-Yi Zhou and Vladlen Koltun. 2014. Color map optimization for 3D reconstruction with consumer depth cameras. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1--10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. 2018. Stereo Magnification: Learning View Synthesis using Multiplane Images. arXiv:1805.09817 [cs.CV]Google ScholarGoogle Scholar

Index Terms

  1. Hybrid Image-based Rendering for Free-view Synthesis

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques
            Proceedings of the ACM on Computer Graphics and Interactive Techniques  Volume 4, Issue 1
            April 2021
            274 pages
            EISSN:2577-6193
            DOI:10.1145/3463840
            Issue’s Table of Contents

            Copyright © 2021 ACM

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 28 April 2021
            Published in pacmcgit Volume 4, Issue 1

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article
            • Research
            • Refereed
          • Article Metrics

            • Downloads (Last 12 months)24
            • Downloads (Last 6 weeks)2

            Other Metrics

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!