skip to main content
research-article

Interactive Sketch-Based Normal Map Generation with Deep Neural Networks

Authors Info & Claims
Published:25 July 2018Publication History
Skip Abstract Section

Abstract

High-quality normal maps are important intermediates for representing complex shapes. In this paper, we propose an interactive system for generating normal maps with the help of deep learning techniques. Utilizing the Generative Adversarial Network (GAN) framework, our method produces high quality normal maps with sketch inputs. In addition, we further enhance the interactivity of our system by incorporating user-specified normals at selected points. Our method generates high quality normal maps in real time. Through comprehensive experiments, we show the effectiveness and robustness of our method. A thorough user study indicates the normal maps generated by our method achieve a lower perceptual difference from the ground truth compared to the alternative methods.

References

  1. M. Arjovsky and L. Bottou. 2017. Towards Principled Methods for Training Generative Adversarial Networks. ArXiv e-prints (Jan. 2017). arXiv:1701.04862Google ScholarGoogle Scholar
  2. M. Arjovsky, S. Chintala and L. Bottou. 2017. Wasserstein GAN. ArXiv e-prints (Jan. 2017). arXiv:1701.07875Google ScholarGoogle Scholar
  3. Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. 2005. A Non-Local Algorithm for Image Denoising. In CVPR '05. 60--65. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Christopher B. Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 2016. 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. In ECCV '16. 628--644.Google ScholarGoogle ScholarCross RefCross Ref
  5. Doug DeCarlo, Adam Finkelstein, Szymon Rusinkiewicz, and Anthony Santella. 2003. Suggestive Contours for Conveying Shape. ACM Trans. Graph. 22, 3 (July 2003), 848--855. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Emily L Denton, Soumith Chintala, arthur szlam, and Rob Fergus. 2015. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. In Advances in Neural Information Processing Systems 28. 1486--1494. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Alexei A. Efros and William T. Freeman. 2001. Image Quilting for Texture Synthesis and Transfer. In SIGGRAPH '01. 341--346. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. David Eigen and Rob Fergus. 2015. Predicting Depth, Surface Normals and Semantic Labels With a Common Multi-Scale Convolutional Architecture. In ICCV '15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). 2672--2680. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. X. Han, C. Gao, and Y. Yu. 2017. DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling. ACM Trans. Graph. 36, 4 (July 2017). Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In SIGGRAPH '01. 327--340. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the Dimensionality of Data with Neural Networks. Science 313, 5786 (2006), 504--507.Google ScholarGoogle Scholar
  13. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and Locally Consistent Image Completion. ACM Trans. Graph. 36, 4 (July 2017), 107:1--107:14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In CVPR '17. 5967--5976.Google ScholarGoogle Scholar
  15. Scott F. Johnston. 2002. Lumo: Illumination for Cel Animation. In NPAR '02. 45--52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Evangelos Kalogerakis, Aaron Hertzmann, and Karan Singh. 2010. Learning 3D Mesh Segmentation and Labeling. ACM Trans. Graph. 29, 4 (July 2010), 102:1--102:12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim. 2017. Learning to Discover Cross-Domain Relations with Generative Adversarial Networks. ArXiv e-prints (March 2017). arXiv:1703.05192Google ScholarGoogle Scholar
  18. Changjian Li, Hao Pan, Yang Liu, Xin Tong, Alla Sheffer, and Wenping Wang. 2017. BendSketch: Modeling Freeform Surfaces Through 2D Sketching. ACM Trans. Graph. 36, 4 (July 2017), 125:1--125:14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 2017. 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks. CoRR abs/1707.06375 (2017). arXiv:1707.06375Google ScholarGoogle Scholar
  20. M. Mirza and S. Osindero. 2014. Conditional Generative Adversarial Nets. ArXiv e-prints (nov 2014). arXiv:1411.1784Google ScholarGoogle Scholar
  21. Luke Olsen, Faramarz F. Samavati, Mario Costa Sousa, and Joaquim A. Jorge. 2009. Sketch-based modeling: A survey. Computers 8 Graphics 33, 1 (2009), 85--103. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Hao Pan, Yang Liu, Alla Sheffer, Nicholas Vining, Chang-Jian Li, and Wenping Wang. 2015. Flow Aligned Surfacing of Curve Networks. ACM Trans. Graph. 34, 4 (July 2015), 127:1--127:10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. 2016. Context Encoders: Feature Learning by Inpainting. In CVPR '16. 2536--2544.Google ScholarGoogle Scholar
  24. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI '15. 234--241.Google ScholarGoogle Scholar
  25. T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. 2016. Improved Techniques for Training GANs. ArXiv e-prints (June 2016). arXiv:1606.03498Google ScholarGoogle Scholar
  26. Ryan Schmidt, Azam Khan, Karan Singh, and Gord Kurtenbach. 2009. Analytic Drawing of 3D Scaffolds. ACM Trans. Graph. 28, 5 (Dec. 2009), 149:1--149:10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Cloud Shao, Adrien Bousseau, Alla Sheffer, and Karan Singh. 2012. CrossShade: Shading Concept Sketches Using Cross-section Curves. ACM Trans. Graph. 31, 4 (July 2012), 45:1--45:11. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Daniel Sýkora, Ladislav Kavan, Martin Čadík, Ondřej Jamriška, Alec Jacobson, Brian Whited, Maryann Simmons, and Olga Sorkine-Hornung. 2014. Ink-and-ray: Bas-relief Meshes for Adding Global Illumination Effects to Hand-drawn Characters. ACM Trans. Graph. 33, 2 (April 2014), 16:1--16:15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Xiaolong Wang, David Fouhey, and Abhinav Gupta. 2015. Designing Deep Networks for Surface Normal Estimation. In CVPR '15.Google ScholarGoogle Scholar
  30. Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. 2016. Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. In NIPS '16. 82--90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Tai-Pang Wu, Chi-Keung Tang, Michael S. Brown, and Heung-Yeung Shum. 2007. ShapePalettes: interactive normal transfer via sketching. In SIGGRAPH '07. 44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Q. Xu, Y. Gingold, and K. Singh. 2015. Inverse Toon Shading: Interactive Normal Field Modeling with Isophotes. In SBIM '15. 15--25. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. 2016. Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision. (12 2016).Google ScholarGoogle Scholar
  34. Z. Yi, H. Zhang, P. Tan, and M. Gong. 2017. DualGAN: Unsupervised Dual Learning for Image-to-Image Translation. ArXiv e-prints (April 2017). arXiv:1704.02510Google ScholarGoogle Scholar
  35. H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. Metaxas. 2016b. StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks. ArXiv e-prints (Dec. 2016). arXiv:1612.03242Google ScholarGoogle Scholar
  36. Richard Zhang, Phillip Isola and Alexei A. Efros. 2016a. Colorful Image Colorization. In ECCV '16. 649--666.Google ScholarGoogle Scholar
  37. Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017. Real-Time User-Guided Image Colorization with Learned Deep Priors. ACM Trans. Graph. 9, 4 (2017). Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. 2017. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ArXiv e-prints (March 2017). arXiv:1703.10593Google ScholarGoogle Scholar

Index Terms

  1. Interactive Sketch-Based Normal Map Generation with Deep Neural Networks

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!