Abstract
High-quality normal maps are important intermediates for representing complex shapes. In this paper, we propose an interactive system for generating normal maps with the help of deep learning techniques. Utilizing the Generative Adversarial Network (GAN) framework, our method produces high quality normal maps with sketch inputs. In addition, we further enhance the interactivity of our system by incorporating user-specified normals at selected points. Our method generates high quality normal maps in real time. Through comprehensive experiments, we show the effectiveness and robustness of our method. A thorough user study indicates the normal maps generated by our method achieve a lower perceptual difference from the ground truth compared to the alternative methods.
- M. Arjovsky and L. Bottou. 2017. Towards Principled Methods for Training Generative Adversarial Networks. ArXiv e-prints (Jan. 2017). arXiv:1701.04862Google Scholar
- M. Arjovsky, S. Chintala and L. Bottou. 2017. Wasserstein GAN. ArXiv e-prints (Jan. 2017). arXiv:1701.07875Google Scholar
- Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. 2005. A Non-Local Algorithm for Image Denoising. In CVPR '05. 60--65. Google Scholar
Digital Library
- Christopher B. Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 2016. 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. In ECCV '16. 628--644.Google Scholar
Cross Ref
- Doug DeCarlo, Adam Finkelstein, Szymon Rusinkiewicz, and Anthony Santella. 2003. Suggestive Contours for Conveying Shape. ACM Trans. Graph. 22, 3 (July 2003), 848--855. Google Scholar
Digital Library
- Emily L Denton, Soumith Chintala, arthur szlam, and Rob Fergus. 2015. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. In Advances in Neural Information Processing Systems 28. 1486--1494. Google Scholar
Digital Library
- Alexei A. Efros and William T. Freeman. 2001. Image Quilting for Texture Synthesis and Transfer. In SIGGRAPH '01. 341--346. Google Scholar
Digital Library
- David Eigen and Rob Fergus. 2015. Predicting Depth, Surface Normals and Semantic Labels With a Common Multi-Scale Convolutional Architecture. In ICCV '15. Google Scholar
Digital Library
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). 2672--2680. Google Scholar
Digital Library
- X. Han, C. Gao, and Y. Yu. 2017. DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling. ACM Trans. Graph. 36, 4 (July 2017). Google Scholar
Digital Library
- Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In SIGGRAPH '01. 327--340. Google Scholar
Digital Library
- G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the Dimensionality of Data with Neural Networks. Science 313, 5786 (2006), 504--507.Google Scholar
- Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and Locally Consistent Image Completion. ACM Trans. Graph. 36, 4 (July 2017), 107:1--107:14. Google Scholar
Digital Library
- Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In CVPR '17. 5967--5976.Google Scholar
- Scott F. Johnston. 2002. Lumo: Illumination for Cel Animation. In NPAR '02. 45--52. Google Scholar
Digital Library
- Evangelos Kalogerakis, Aaron Hertzmann, and Karan Singh. 2010. Learning 3D Mesh Segmentation and Labeling. ACM Trans. Graph. 29, 4 (July 2010), 102:1--102:12. Google Scholar
Digital Library
- T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim. 2017. Learning to Discover Cross-Domain Relations with Generative Adversarial Networks. ArXiv e-prints (March 2017). arXiv:1703.05192Google Scholar
- Changjian Li, Hao Pan, Yang Liu, Xin Tong, Alla Sheffer, and Wenping Wang. 2017. BendSketch: Modeling Freeform Surfaces Through 2D Sketching. ACM Trans. Graph. 36, 4 (July 2017), 125:1--125:14. Google Scholar
Digital Library
- Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 2017. 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks. CoRR abs/1707.06375 (2017). arXiv:1707.06375Google Scholar
- M. Mirza and S. Osindero. 2014. Conditional Generative Adversarial Nets. ArXiv e-prints (nov 2014). arXiv:1411.1784Google Scholar
- Luke Olsen, Faramarz F. Samavati, Mario Costa Sousa, and Joaquim A. Jorge. 2009. Sketch-based modeling: A survey. Computers 8 Graphics 33, 1 (2009), 85--103. Google Scholar
Digital Library
- Hao Pan, Yang Liu, Alla Sheffer, Nicholas Vining, Chang-Jian Li, and Wenping Wang. 2015. Flow Aligned Surfacing of Curve Networks. ACM Trans. Graph. 34, 4 (July 2015), 127:1--127:10. Google Scholar
Digital Library
- Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. 2016. Context Encoders: Feature Learning by Inpainting. In CVPR '16. 2536--2544.Google Scholar
- Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI '15. 234--241.Google Scholar
- T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. 2016. Improved Techniques for Training GANs. ArXiv e-prints (June 2016). arXiv:1606.03498Google Scholar
- Ryan Schmidt, Azam Khan, Karan Singh, and Gord Kurtenbach. 2009. Analytic Drawing of 3D Scaffolds. ACM Trans. Graph. 28, 5 (Dec. 2009), 149:1--149:10. Google Scholar
Digital Library
- Cloud Shao, Adrien Bousseau, Alla Sheffer, and Karan Singh. 2012. CrossShade: Shading Concept Sketches Using Cross-section Curves. ACM Trans. Graph. 31, 4 (July 2012), 45:1--45:11. Google Scholar
Digital Library
- Daniel Sýkora, Ladislav Kavan, Martin Čadík, Ondřej Jamriška, Alec Jacobson, Brian Whited, Maryann Simmons, and Olga Sorkine-Hornung. 2014. Ink-and-ray: Bas-relief Meshes for Adding Global Illumination Effects to Hand-drawn Characters. ACM Trans. Graph. 33, 2 (April 2014), 16:1--16:15. Google Scholar
Digital Library
- Xiaolong Wang, David Fouhey, and Abhinav Gupta. 2015. Designing Deep Networks for Surface Normal Estimation. In CVPR '15.Google Scholar
- Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. 2016. Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. In NIPS '16. 82--90. Google Scholar
Digital Library
- Tai-Pang Wu, Chi-Keung Tang, Michael S. Brown, and Heung-Yeung Shum. 2007. ShapePalettes: interactive normal transfer via sketching. In SIGGRAPH '07. 44. Google Scholar
Digital Library
- Q. Xu, Y. Gingold, and K. Singh. 2015. Inverse Toon Shading: Interactive Normal Field Modeling with Isophotes. In SBIM '15. 15--25. Google Scholar
Digital Library
- Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. 2016. Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision. (12 2016).Google Scholar
- Z. Yi, H. Zhang, P. Tan, and M. Gong. 2017. DualGAN: Unsupervised Dual Learning for Image-to-Image Translation. ArXiv e-prints (April 2017). arXiv:1704.02510Google Scholar
- H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. Metaxas. 2016b. StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks. ArXiv e-prints (Dec. 2016). arXiv:1612.03242Google Scholar
- Richard Zhang, Phillip Isola and Alexei A. Efros. 2016a. Colorful Image Colorization. In ECCV '16. 649--666.Google Scholar
- Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017. Real-Time User-Guided Image Colorization with Learned Deep Priors. ACM Trans. Graph. 9, 4 (2017). Google Scholar
Digital Library
- J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. 2017. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ArXiv e-prints (March 2017). arXiv:1703.10593Google Scholar
Index Terms
Interactive Sketch-Based Normal Map Generation with Deep Neural Networks
Recommendations
Robust flow-guided neural prediction for sketch-based freeform surface modeling
Sketching provides an intuitive user interface for communicating free form shapes. While human observers can easily envision the shapes they intend to communicate, replicating this process algorithmically requires resolving numerous ambiguities. ...
Sketch2normal: deep networks for normal map generation
SA '17: SIGGRAPH Asia 2017 PostersNormal maps are of great importance for many 2D graphics applications such as surface editing, re-lighting, texture mapping and 2D shading etc. Automatically inferring normal map is highly desirable for graphics designers. Many researchers have ...
‘CADSketchNet’ - An Annotated Sketch dataset for 3D CAD Model Retrieval with Deep Neural Networks▪
Highlights- The goal of this paper is to create a sketch dataset that is suitable for developing deep learning-based solutions to the problem of search and retrieval in ...
Graphical abstractDisplay Omitted
AbstractOngoing advancements in the fields of 3D modelling and digital archiving have led to an outburst in the amount of data stored digitally. Consequently, several retrieval systems have been developed depending on the type of data stored ...






Comments