skip to main content
research-article

Image-Based Rendering of Cars using Semantic Labels and Approximate Reflection Flow

Published:04 May 2020Publication History
Skip Abstract Section

Abstract

Image-Based Rendering (IBR) has made impressive progress towards highly realistic, interactive 3D navigation for many scenes, including cityscapes. However, cars are ubiquitous in such scenes; multi-view stereo reconstruction provides proxy geometry for IBR, but has difficulty with shiny car bodies, and leaves holes in place of reflective, semi-transparent windows on cars. We present a new approach allowing free-viewpoint IBR of cars based on an approximate analytic reflection flow computation on curved windows. Our method has three components: a refinement step of reconstructed car geometry guided by semantic labels, that provides an initial approximation for missing window surfaces and a smooth completed car hull; an efficient reflection flow computation using an ellipsoid approximation of the curved car windows that runs in real-time in a shader and a reflection/background layer synthesis solution. These components allow plausible rendering of reflective, semi-transparent windows in free viewpoint navigation. We show results on several scenes casually captured with a single consumer-level camera, demonstrating plausible car renderings with significant improvement in visual quality over previous methods.

Skip Supplemental Material Section

Supplemental Material

References

  1. Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, and Michael Cohen. 2004. Interactive digital photomontage. In ACM Transactions on Graphics (TOG), Vol. 23. ACM, 294--302.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001. Unstructured lumigraph rendering. In Proceedings of the 28th annual conference on Computer Graphics and Interactive Techniques. ACM, 425--432.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Gaurav Chaurasia, Sylvain Duchene, Olga Sorkine-Hornung, and George Drettakis. 2013. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics (TOG) 32, 3 (2013), 30.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2017. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 4 (2017), 834--848.Google ScholarGoogle ScholarCross RefCross Ref
  5. Paul Debevec, Yizhou Yu, and George Borshukov. 1998. Efficient view-dependent image-based rendering with projective texture-mapping. In Rendering Techniques' 98. Springer, 105--116.Google ScholarGoogle Scholar
  6. Pau Estalella, Ignacio Martin, George Drettakis, Dani Tost, Olivier Devillers, and Frédéric Cazals. 2005. Accurate interactive specular reflections on curved objects. In Proceedings of Vision, Modeling and Visualization. Eurographics Association. http://www-sop.inria.fr/reves/Basilic/2005/EMDTDC05Google ScholarGoogle Scholar
  7. Clement Godard, Peter Hedman, Wenbin Li, and Gabriel J. Brostow. 2015. Multi-view reconstruction of highly specular surfaces in uncontrolled environments. In International Conference on 3D Vision. IEEE, 19--27.Google ScholarGoogle Scholar
  8. Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, and Steven M. Seitz. 2007. Multi-view stereo for community photo collections. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 1--8.Google ScholarGoogle Scholar
  9. Steven J Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F Cohen. 1996. The lumigraph. In Proceedings of the 23rd annual conference on Computer Graphics and Interactive Techniques. ACM, 43--54.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Christian Hane, Nikolay Savinov, and Marc Pollefeys. 2014. Class specific 3d object shape priors using surface normals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 652--659.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. 2018. Deep blending for free-viewpoint image-based rendering. ACM Transactions on Graphics (TOG) 37, 6 (November 2018). http://www-sop.inria.fr/reves/Basilic/2018/HPPFDB18Google ScholarGoogle Scholar
  12. Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel Brostow. 2016. Scalable inside-out image-based rendering. ACM Transactions on Graphics (TOG) 35, 6 (2016), 231.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Ivo Ihrke, Kiriakos N. Kutulakos, Hendrik P.A. Lensch, Marcus Magnor, and Wolfgang Heidrich. 2008. State of the art in transparent and specular object reconstruction. In Eurographics State Of The Art Report (STAR). Eurographics Association.Google ScholarGoogle Scholar
  14. Ivo Ihrke, Kiriakos N. Kutulakos, Hendrik P.A. Lensch, Marcus Magnor, and Wolfgang Heidrich. 2010. Transparent and specular object reconstruction. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 2400--2426.Google ScholarGoogle Scholar
  15. Bastien Jacquet, Christian Hane, Kevin Koser, and Marc Pollefeys. 2013. Real-world normal map capture for nearly flat reflective surfaces. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 713--720.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Johannes Kopf, Fabian Langguth, Daniel Scharstein, Richard Szeliski, and Michael Goesele. 2013. Image-based rendering in the gradient domain. ACM Transactions on Graphics (TOG) 32, 6 (2013), 199.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. 2003. Graphcut textures: image and video synthesis using graph cuts. ACM Transactions on Graphics (TOG) 22, 3 (2003), 277--286.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Anat Levin and Yair Weiss. 2007. User assisted separation of reflections from a single image using a sparsity prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 9 (2007), 1647--1654.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the 23rd annual conference on Computer Graphics and Interactive Techniques. ACM, 31--42.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. David G. Lowe. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2 (2004), 91--110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) 34, 4 (2019).Google ScholarGoogle Scholar
  22. Eyal Ofek and Ari Rappoport. 1998. Interactive reflections on curved objects. In Proceedings of the 25th annual conference on Computer Graphics and Interactive Techniques. ACM, 333--342.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Eric Penner and Li Zhang. 2017. Soft 3D reconstruction for view synthesis. ACM Transactions on Graphics (TOG) 36, 6 (2017), 235.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Capturing Reality. 2018. RealityCapture reconstruction software. https://www.capturingreality.com/Product.Google ScholarGoogle Scholar
  25. Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. 2011. ORB: an efficient alternative to SIFT or SURF. In Proceedings of the IEEE International Conference on Computer Vision, Vol. 11. IEEE, 2564--2571.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-motion revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE.Google ScholarGoogle ScholarCross RefCross Ref
  27. Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. 2016. Pixelwise view selection for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision. Springer.Google ScholarGoogle ScholarCross RefCross Ref
  28. Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012. Image-based rendering for scenes with reflections. ACM Transactions on Graphics (TOG) 31, 4 (2012), 100--1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Richard Szeliski, Shai Avidan, and Padmanabhan Anandan. 2000. Layer extraction from multiple images containing reflections and transparency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1. IEEE, 246--253.Google ScholarGoogle ScholarCross RefCross Ref
  30. Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred neural rendering: image synthesis using neural textures. ACM Transactions on Graphics (TOG) (2019).Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, and Matthias Nießner. 2020. Image-guided neural object rendering. In International Conference on Learning Representations. ICLR. https://openreview.net/forum?id=Hyg9anEFPSGoogle ScholarGoogle Scholar
  32. Vaibhav Vaish, Marc Levoy, Richard Szeliski, C. Lawrence Zitnick, and Sing Bing Kang. 2006. Reconstructing occluded surfaces using synthetic apertures: stereo, focus and robust measures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 2. IEEE, 2331--2338.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Thomas Whelan, Michael Goesele, Steven J. Lovegrove, Julian Straub, Simon Green, Richard Szeliski, Steven Butterfield, Shobhit Verma, and Richard Newcombe. 2018. Reconstructing scenes with mirror and glass surfaces. ACM Transactions on Graphics (TOG) 37, 4 (2018), 102.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Turner Whitted. 1980. An improved illumination model for shaded display. Commun. ACM 23, 6 (June 1980), 343--349.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Patrick Wieschollek, Orazio Gallo, Jinwei Gu, and Jan Kautz. 2018. Separating reflection and transmission images in the wild. In Proceedings of the European Conference on Computer Vision. Springer, 89--104.Google ScholarGoogle ScholarCross RefCross Ref
  36. Bojian Wu, Yang Zhou, Yiming Qian, Minglun Cong, and Hui Huang. 2018. Full 3D reconstruction of transparent objects. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--11.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Tianfan Xue, Michael Rubinstein, Ce Liu, and William T. Freeman. 2015. A computational approach for obstruction-free photography. ACM Transactions on Graphics (TOG) 34, 4 (2015), 79.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Sid Yingze Bao, Manmohan Chandraker, Yuanqing Lin, and Silvio Savarese. 2013. Dense object reconstruction with semantic priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1264--1271.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene parsing through ADE20K dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 633--641.Google ScholarGoogle ScholarCross RefCross Ref
  40. Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2019. Semantic understanding of scenes through the ADE20K dataset. International Journal of Computer Vision 127, 3 (2019), 302--321.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Image-Based Rendering of Cars using Semantic Labels and Approximate Reflection Flow

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques
          Proceedings of the ACM on Computer Graphics and Interactive Techniques  Volume 3, Issue 1
          Apr 2020
          161 pages
          EISSN:2577-6193
          DOI:10.1145/3395964
          Issue’s Table of Contents

          Copyright © 2020 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 4 May 2020
          Published in pacmcgit Volume 3, Issue 1

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!