Abstract
Image-Based Rendering (IBR) has made impressive progress towards highly realistic, interactive 3D navigation for many scenes, including cityscapes. However, cars are ubiquitous in such scenes; multi-view stereo reconstruction provides proxy geometry for IBR, but has difficulty with shiny car bodies, and leaves holes in place of reflective, semi-transparent windows on cars. We present a new approach allowing free-viewpoint IBR of cars based on an approximate analytic reflection flow computation on curved windows. Our method has three components: a refinement step of reconstructed car geometry guided by semantic labels, that provides an initial approximation for missing window surfaces and a smooth completed car hull; an efficient reflection flow computation using an ellipsoid approximation of the curved car windows that runs in real-time in a shader and a reflection/background layer synthesis solution. These components allow plausible rendering of reflective, semi-transparent windows in free viewpoint navigation. We show results on several scenes casually captured with a single consumer-level camera, demonstrating plausible car renderings with significant improvement in visual quality over previous methods.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Image-Based Rendering of Cars using Semantic Labels and Approximate Reflection Flow
- Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, and Michael Cohen. 2004. Interactive digital photomontage. In ACM Transactions on Graphics (TOG), Vol. 23. ACM, 294--302.Google Scholar
Digital Library
- Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001. Unstructured lumigraph rendering. In Proceedings of the 28th annual conference on Computer Graphics and Interactive Techniques. ACM, 425--432.Google Scholar
Digital Library
- Gaurav Chaurasia, Sylvain Duchene, Olga Sorkine-Hornung, and George Drettakis. 2013. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics (TOG) 32, 3 (2013), 30.Google Scholar
Digital Library
- Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2017. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 4 (2017), 834--848.Google Scholar
Cross Ref
- Paul Debevec, Yizhou Yu, and George Borshukov. 1998. Efficient view-dependent image-based rendering with projective texture-mapping. In Rendering Techniques' 98. Springer, 105--116.Google Scholar
- Pau Estalella, Ignacio Martin, George Drettakis, Dani Tost, Olivier Devillers, and Frédéric Cazals. 2005. Accurate interactive specular reflections on curved objects. In Proceedings of Vision, Modeling and Visualization. Eurographics Association. http://www-sop.inria.fr/reves/Basilic/2005/EMDTDC05Google Scholar
- Clement Godard, Peter Hedman, Wenbin Li, and Gabriel J. Brostow. 2015. Multi-view reconstruction of highly specular surfaces in uncontrolled environments. In International Conference on 3D Vision. IEEE, 19--27.Google Scholar
- Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, and Steven M. Seitz. 2007. Multi-view stereo for community photo collections. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 1--8.Google Scholar
- Steven J Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F Cohen. 1996. The lumigraph. In Proceedings of the 23rd annual conference on Computer Graphics and Interactive Techniques. ACM, 43--54.Google Scholar
Digital Library
- Christian Hane, Nikolay Savinov, and Marc Pollefeys. 2014. Class specific 3d object shape priors using surface normals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 652--659.Google Scholar
Digital Library
- Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. 2018. Deep blending for free-viewpoint image-based rendering. ACM Transactions on Graphics (TOG) 37, 6 (November 2018). http://www-sop.inria.fr/reves/Basilic/2018/HPPFDB18Google Scholar
- Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel Brostow. 2016. Scalable inside-out image-based rendering. ACM Transactions on Graphics (TOG) 35, 6 (2016), 231.Google Scholar
Digital Library
- Ivo Ihrke, Kiriakos N. Kutulakos, Hendrik P.A. Lensch, Marcus Magnor, and Wolfgang Heidrich. 2008. State of the art in transparent and specular object reconstruction. In Eurographics State Of The Art Report (STAR). Eurographics Association.Google Scholar
- Ivo Ihrke, Kiriakos N. Kutulakos, Hendrik P.A. Lensch, Marcus Magnor, and Wolfgang Heidrich. 2010. Transparent and specular object reconstruction. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 2400--2426.Google Scholar
- Bastien Jacquet, Christian Hane, Kevin Koser, and Marc Pollefeys. 2013. Real-world normal map capture for nearly flat reflective surfaces. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 713--720.Google Scholar
Digital Library
- Johannes Kopf, Fabian Langguth, Daniel Scharstein, Richard Szeliski, and Michael Goesele. 2013. Image-based rendering in the gradient domain. ACM Transactions on Graphics (TOG) 32, 6 (2013), 199.Google Scholar
Digital Library
- Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. 2003. Graphcut textures: image and video synthesis using graph cuts. ACM Transactions on Graphics (TOG) 22, 3 (2003), 277--286.Google Scholar
Digital Library
- Anat Levin and Yair Weiss. 2007. User assisted separation of reflections from a single image using a sparsity prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 9 (2007), 1647--1654.Google Scholar
Digital Library
- Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the 23rd annual conference on Computer Graphics and Interactive Techniques. ACM, 31--42.Google Scholar
Digital Library
- David G. Lowe. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2 (2004), 91--110.Google Scholar
Digital Library
- Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) 34, 4 (2019).Google Scholar
- Eyal Ofek and Ari Rappoport. 1998. Interactive reflections on curved objects. In Proceedings of the 25th annual conference on Computer Graphics and Interactive Techniques. ACM, 333--342.Google Scholar
Digital Library
- Eric Penner and Li Zhang. 2017. Soft 3D reconstruction for view synthesis. ACM Transactions on Graphics (TOG) 36, 6 (2017), 235.Google Scholar
Digital Library
- Capturing Reality. 2018. RealityCapture reconstruction software. https://www.capturingreality.com/Product.Google Scholar
- Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. 2011. ORB: an efficient alternative to SIFT or SURF. In Proceedings of the IEEE International Conference on Computer Vision, Vol. 11. IEEE, 2564--2571.Google Scholar
Digital Library
- Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-motion revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE.Google Scholar
Cross Ref
- Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. 2016. Pixelwise view selection for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision. Springer.Google Scholar
Cross Ref
- Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012. Image-based rendering for scenes with reflections. ACM Transactions on Graphics (TOG) 31, 4 (2012), 100--1.Google Scholar
Digital Library
- Richard Szeliski, Shai Avidan, and Padmanabhan Anandan. 2000. Layer extraction from multiple images containing reflections and transparency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1. IEEE, 246--253.Google Scholar
Cross Ref
- Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred neural rendering: image synthesis using neural textures. ACM Transactions on Graphics (TOG) (2019).Google Scholar
Digital Library
- Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, and Matthias Nießner. 2020. Image-guided neural object rendering. In International Conference on Learning Representations. ICLR. https://openreview.net/forum?id=Hyg9anEFPSGoogle Scholar
- Vaibhav Vaish, Marc Levoy, Richard Szeliski, C. Lawrence Zitnick, and Sing Bing Kang. 2006. Reconstructing occluded surfaces using synthetic apertures: stereo, focus and robust measures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 2. IEEE, 2331--2338.Google Scholar
Digital Library
- Thomas Whelan, Michael Goesele, Steven J. Lovegrove, Julian Straub, Simon Green, Richard Szeliski, Steven Butterfield, Shobhit Verma, and Richard Newcombe. 2018. Reconstructing scenes with mirror and glass surfaces. ACM Transactions on Graphics (TOG) 37, 4 (2018), 102.Google Scholar
Digital Library
- Turner Whitted. 1980. An improved illumination model for shaded display. Commun. ACM 23, 6 (June 1980), 343--349.Google Scholar
Digital Library
- Patrick Wieschollek, Orazio Gallo, Jinwei Gu, and Jan Kautz. 2018. Separating reflection and transmission images in the wild. In Proceedings of the European Conference on Computer Vision. Springer, 89--104.Google Scholar
Cross Ref
- Bojian Wu, Yang Zhou, Yiming Qian, Minglun Cong, and Hui Huang. 2018. Full 3D reconstruction of transparent objects. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--11.Google Scholar
Digital Library
- Tianfan Xue, Michael Rubinstein, Ce Liu, and William T. Freeman. 2015. A computational approach for obstruction-free photography. ACM Transactions on Graphics (TOG) 34, 4 (2015), 79.Google Scholar
Digital Library
- Sid Yingze Bao, Manmohan Chandraker, Yuanqing Lin, and Silvio Savarese. 2013. Dense object reconstruction with semantic priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1264--1271.Google Scholar
Digital Library
- Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene parsing through ADE20K dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 633--641.Google Scholar
Cross Ref
- Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2019. Semantic understanding of scenes through the ADE20K dataset. International Journal of Computer Vision 127, 3 (2019), 302--321.Google Scholar
Digital Library
Index Terms
Image-Based Rendering of Cars using Semantic Labels and Approximate Reflection Flow
Recommendations
Sample-Based Cameras for Feed Forward Reflection Rendering
This paper presents sample-based cameras for rendering high quality reflections on convex reflectors at interactive rates. The method supports change of view, moving objects and reflectors, higher order reflections, view-dependent lighting of reflected ...
Accelerating voxel-based terrain rendering with keyframe-free image-based rendering
VG'01: Proceedings of the 2001 Eurographics conference on Volume GraphicsWe propose a voxel-based terrain rendering method which incorporates a novel keyframe-free image-based rendering algorithm and a new heuristic ray coherence raycasting algorithm. The current image is generated by warping the previous image with a ...
Image-based rendering of diffuse, specular and glossy surfaces from a single image
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniquesIn this paper, we present a new method to recover an approximation of the bidirectional reflectance distribution function (BRDF) of the surfaces present in a real scene. This is done from a single photograph and a 3D geometric model of the scene. The ...






Comments