Abstract
Sketch-based modeling strives to bring the ease and immediacy of drawing to the 3D world. However, while drawings are easy for humans to create, they are very challenging for computers to interpret due to their sparsity and ambiguity. We propose a data-driven approach that tackles this challenge by learning to reconstruct 3D shapes from one or more drawings. At the core of our approach is a deep convolutional neural network (CNN) that predicts occupancy of a voxel grid from a line drawing. This CNN provides an initial 3D reconstruction as soon as the user completes a single drawing of the desired shape. We complement this single-view network with an updater CNN that refines an existing prediction given a new drawing of the shape created from a novel viewpoint. A key advantage of our approach is that we can apply the updater iteratively to fuse information from an arbitrary number of viewpoints, without requiring explicit stroke correspondences between the drawings. We train both CNNs by rendering synthetic contour drawings from hand-modeled shape collections as well as from procedurally-generated abstract shapes. Finally, we integrate our CNNs in an interactive modeling system that allows users to seamlessly draw an object, rotate it to see its 3D reconstruction, and refine it by re-drawing from another vantage point using the 3D reconstruction as guidance.
- Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models. In ACM symposium on User Interface Software and Technology (UIST). 151--160. 2, 4, 10 Google Scholar
Digital Library
- H. Barrow and J. Tenenbaum. 1981. Interpreting Line Drawings as Three-Dimensional Surfaces. Artificial Intelligence 17 (1981). 3 Google Scholar
Digital Library
- Y. Y. Boykov and M. P. Jolly. 2001. Interactive graph cuts for optimal boundary 8 region segmentation of objects in N-D images. In IEEE International Conference on Computer Vision (ICCV), Vol. 1. 19Google Scholar
- Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. 2016. Human pose estimation with iterative error feedback. (2016). 4, 8Google Scholar
- Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. Technical Report. 9Google Scholar
- Qifeng Chen and Vladlen Koltun. 2017. Photographic Image Synthesis with Cascaded Refinement Networks. In IEEE International Conference on Computer Vision (ICCV). 2, 5Google Scholar
- Tao Chen, Zhe Zhu, Ariel Shamir, Shi-Min Hu, and Daniel Cohen-Or. 2013. 3-Sweep: Extracting Editable Objects from a Single Photo. ACM Transactions on Graphics (Proc. SIGGRAPH) 32, 6 (2013). 4 Google Scholar
Digital Library
- Christopher Bongsoo Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 2016. 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. In European Conference on Computer Vision (ECCV). 628--644. 5Google Scholar
Cross Ref
- Forrester Cole, Phillip Isola, William T Freeman, Frédo Durand, and Edward H Adelson. 2012. Shapecollage: occlusion-aware, example-based shape interpretation. In European Conference on Computer Vision (ECCV). Springer, 665--678. 4 Google Scholar
Digital Library
- Frederic Cordier, Hyewon Seo, Mahmoud Melkemi, and Nickolas S. Sapidis. 2013. Inferring Mirror Symmetric 3D Shapes from Sketches. Computer Aided Design 45, 2 (Feb. 2013), 301--311. 3 Google Scholar
Digital Library
- Frederic Cordier, Karan Singh, Yotam Gingold, and Marie-Paule Cani. 2016. Sketch-based Modeling. In SIGGRAPH ASIA Courses. ACM, Article 18, 222 pages. 3 Google Scholar
Digital Library
- Doug DeCarlo, Adam Finkelstein, Szymon Rusinkiewicz, and Anthony Santella. 2003. Suggestive Contours for Conveying Shape. ACM Transactions on Graphics (Proc. SIGGRAPH) 22, 3 (July 2003), 848--855. 10 Google Scholar
Digital Library
- A. Dosovitskiy, J. T. Springenberg, M. Tatarchenko, and T. Brox. 2016. Learning to Generate Chairs, Tables and Cars with Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) (2016). 5 Google Scholar
Digital Library
- David Eigen and Rob Fergus. 2015. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In IEEE International Conference on Computer Vision (ICCV). 2650--2658. 4, 6, 7, 16, 17 Google Scholar
Digital Library
- Koos Eissen and Roselien Steur. 2011. Sketching: The Basics. Bis Publishers. 10Google Scholar
- Mathias Eitz, James Hays, and Marc Alexa. 2012a. How Do Humans Sketch Objects? ACM Transactions on Graphics (Proc. SIGGRAPH) 31, 4 (2012), 44:1--44:10. 9 Google Scholar
Digital Library
- Mathias Eitz, Ronald Richter, Tamy Boubekeur, Kristian Hildebrand, and Marc Alexa. 2012b. Sketch-Based Shape Retrieval. ACM Transactions on Graphics (Proc. SIGGRAPH) 31, 4 (2012), 31:1--31:10. 2, 4, 8 Google Scholar
Digital Library
- Haoqiang Fan, Hao Su, and Leonidas Guibas. 2017. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. IEEE Computer Vision and Pattern Recognition (CVPR) (2017). 2, 5, 17Google Scholar
- Jean-Dominique Favreau, Florent Lafarge, and Adrien Bousseau. 2016. Fidelity vs. Simplicity: a Global Approach to Line Drawing Vectorization. ACM Transactions on Graphics (Proc. SIGGRAPH) (2016). 3 Google Scholar
Digital Library
- Thomas Funkhouser, Patrick Min, Michael Kazhdan, Joyce Chen, Alex Halderman, David Dobkin, and David Jacobs. 2003. A Search Engine for 3D Models. ACM Transactions on Graphics 22, 1 (Jan. 2003), 83--105. 4 Google Scholar
Digital Library
- Yotam Gingold, Takeo Igarashi, and Denis Zorin. 2009. Structured Annotations for 2D-to-3D Modeling. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 28, 5 (2009). 2, 4 Google Scholar
Digital Library
- Giorgio Gori, Alla Sheffer, Nicholas Vining, Enrique Rosales, Nathan Carr, and Tao Ju. 2017. FlowRep: Descriptive Curve Networks for Free-Form Design Shapes. ACM Transaction on Graphics (Proc. SIGGRAPH) 36, 4 (2017). 19 Google Scholar
Digital Library
- X. Han, C. Gao, and Y. Yu. 2017. DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling. ACM Transactions on Graphics (Proc. SIGGRAPH) 36, 4 (July 2017). 5 Google Scholar
Digital Library
- Christian Häne, Shubham Tulsiani, and Jitendra Malik. 2017. Hierarchical Surface Prediction for 3D Object Reconstruction. In International Conference on 3D Vision (3DV). 13, 17Google Scholar
Cross Ref
- Aaron Hertzmann and Denis Zorin. 2000. Illustrating smooth surfaces. In SIGGRAPH. ACM Press/Addison-Wesley Publishing Co., 517--526. 10, 18 Google Scholar
Digital Library
- Haibin Huang, Evangelos Kalogerakis, Ersin Yumer, and Radomir Mech. 2016. Shape Synthesis from Sketches via Procedural Models and Convolutional Networks. IEEE Transactions on Visualization and Computer Graphics (TVCG) 22, 10 (2016), 1. 2, 4, 5, 9Google Scholar
Digital Library
- Emmanuel Iarussi, David Bommes, and Adrien Bousseau. 2015. BendFields: Regularized Curvature Fields from Rough Concept Sketches. ACM Transactions on Graphics (2015). 3 Google Scholar
Digital Library
- Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. 1999. Teddy: A Sketching Interface for 3D Freeform Design. In SIGGRAPH. 409--416. 3, 4, 11 Google Scholar
Digital Library
- Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of The 32nd International Conference on Machine Learning. 448--456. 11 Google Scholar
Digital Library
- Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. IEEE Computer Vision and Pattern Recognition (CVPR) (2017). 7, 19Google Scholar
- Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang. 2017. SurfaceNet: An End-To-End 3D Neural Network for Multiview Stereopsis. In IEEE International Conference on Computer Vision (ICCV). 5Google Scholar
Cross Ref
- Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia. ACM, 675--678. 18 Google Scholar
Digital Library
- Thouis R. Jones, Frédo Durand, and Mathieu Desbrun. 2003. Non-Iterative, Feature-Preserving Mesh Smoothing. ACM Transactions on Graphics (Proc. SIGGRAPH) (2003). 12 Google Scholar
Digital Library
- Amaury Jung, Stefanie Hahmann, Damien Rohmer, Antoine Begault, Laurence Boissieux, and Marie-Paule Cani. 2015. Sketching Folds: Developable Surfaces from Non-Planar Silhouettes. ACM Transactions on Graphics (2015). 3 Google Scholar
Digital Library
- Evangelos Kalogerakis, Derek Nowrouzezahrai, Simon Breslav, and Aaron Hertzmann. 2012. Learning Hatching for Pen-and-Ink Illustration of Surfaces. ACM Transactions on Graphics 31, 1 (2012). 19 Google Scholar
Digital Library
- Alex Kendall and Yarin Gal. 2017. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?. In Advances in Neural Information Processing Systems 30 (NIPS). 14Google Scholar
- Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014). 11Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (NIPS). 1097--1105. 4 Google Scholar
Digital Library
- Jeehyung Lee and Thomas Funkhouser. 2008. Sketch-Based Search and Composition of3D Models. In EUROGRAPHICS Workshop on Sketch-Based Interfaces and Modeling (SBIM). 4 Google Scholar
Digital Library
- Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. 2017. GRASS: Generative Recursive Autoencoders for Shape Structures. ACM Transactions on Graphics (Proc. SIGGRAPH) 36, 4 (2017). 5 Google Scholar
Digital Library
- Yangyan Li, Xiaokun Wu, Yiorgos Chrysanthou, Andrei Sharf, Daniel Cohen-Or, and Niloy J. Mitra. 2011. GlobFit: Consistently Fitting Primitives by Discovering Global Relations. ACM Transactions on Graphics (proc. SIGGRAPH) 30, 4 (2011). 19 Google Scholar
Digital Library
- H Lipson and M Shpitalni. 1996. Optimization-based reconstruction of a 3D object from a single freehand line drawing. Computer-Aided Design 28, 8 (1996), 651--663. 2, 3Google Scholar
Cross Ref
- H. Lipson and M. Shpitalni. 2000. Conceptual Design and Analysis by Sketching. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 14, 5 (Nov. 2000), 391--401. 4 Google Scholar
Digital Library
- J. Liu, F. Yu, and T. Funkhouser. 2017. Interactive 3D Modeling with a Generative Adversarial Network. In International Conference on 3D Vision (3DV). 5Google Scholar
- Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully Convolutional Networks for Semantic Segmentation. IEEE Computer Vision and Pattern Recognition (CVPR) (2015). 4Google Scholar
- William E. Lorensen and Harvey E. Cline. 1987. Marching Cubes: A High Resolution 3D Surface Construction Algorithm. SIGGRAPH 21, 4 (Aug. 1987), 163--169. 12 Google Scholar
Digital Library
- Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 2017. 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks. In International Conference on 3D Vision (3DV). 5, 17Google Scholar
Cross Ref
- J. Malik and D. Maydan. 1989. Recovering three-dimensional shape from a single image of curved objects. IEEE Pattern Analysis and Machine Intelligence (PAMI) 11, 6 (1989), 555--566. 3 Google Scholar
Digital Library
- Worthy N. Martin and J. K. Aggarwal. 1983. Volumetric Descriptions of Objects from Multiple Views. IEEE Transactions on Pattern Analysis Machine Intelligence 5, 2 (Feb. 1983), 150--158. 16 Google Scholar
Digital Library
- Patrick Min. 2016. Binvox 3D mesh voxelizer. http://www.google.com/search?q=binvox. (2016). Accessed: 2016-11-01. 10Google Scholar
- Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Marc Alexa. 2007. FiberMesh: designing freeform surfaces with 3D curves. ACM Transactions on Graphics (Proc. SIGGRAPH) 26, Article 41 (2007). Issue 3. 4 Google Scholar
Digital Library
- A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune. 2016. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in Neural Information Processing Systems (NIPS). 5 Google Scholar
Digital Library
- Gen Nishida, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Adrien Bousseau. 2016. Interactive Sketching of Urban Procedural Models. ACM Transactions on Graphics (Proc. SIGGRAPH) (2016). 2, 4, 5, 9, 13 Google Scholar
Digital Library
- F. S. Nooruddin and G. Turk. 2003. Simplification and repair of polygonal models using volumetric techniques. IEEE Transactions on Visualization and Computer Graphics (TVCG) 9, 2 (April 2003), 191--205. 10 Google Scholar
Digital Library
- Markus Oberweger, Paul Wohlhart, and Vincent Lepetit. 2015. Training a feedback loop for hand pose estimation. In IEEE International Conference on Computer Vision (ICCV). 3316--3324. 4, 8 Google Scholar
Digital Library
- L. Olsen, F.F. Samavati, M.C. Sousa, and J. Jorge. 2009. Sketch-Based Modeling: A Survey. Computers 8 Graphics 33 (2009). Issue 1. 3 Google Scholar
Digital Library
- Gunay Orbay and Levent Burak Kara. 2012. Sketch-Based Surface Design Using Malleable Curve Networks. Computers 8 Graphics 36, 8 (2012), 916--929. 4 Google Scholar
Digital Library
- Eunbyung Park, Jimei Yang, Ersin Yumer, Duygu Ceylan, and Alexander C. Berg. 2017. Transformation-Grounded Image Generation Network for Novel 3D View Synthesis. IEEE Computer Vision and Pattern Recognition (CVPR) (2017). 5, 6Google Scholar
- Gernot Riegler, Ali Osman Ulusoy, Horst Bischof, and Andreas Geiger. 2017. OctNetFusion: Learning Depth Fusion from Data. In International Conference on 3D Vision (3DV). 13, 17Google Scholar
Cross Ref
- Alec Rivers, Frédo Durand, and Takeo Igarashi. 2010. 3D Modeling with Silhouettes. ACM Transactions on Graphics (Proc. SIGGRAPH) 29, 4, Article 109 (2010), 8 pages. 2, 4, 13 Google Scholar
Digital Library
- Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention - MICCAI. 234--241. 7Google Scholar
- Takafumi Saito and Tokiichiro Takahashi. 1990. Comprehensible Rendering of 3-D Shapes. SIGGRAPH 24, 4 (1990), 197--206. 10 Google Scholar
Digital Library
- Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies. ACM Transactions on Graphics (Proc. SIGGRAPH) (2016). 5 Google Scholar
Digital Library
- Patsorn Sangkloy, Jingwan Lu, Chen Fang, FIsher Yu, and James Hays. 2017. Scribbler: Controlling Deep Image Synthesis with Sketch and Color. IEEE Computer Vision and Pattern Recognition (CVPR) (2017). 5, 6Google Scholar
- Ryan Schmidt, Azam Khan, Karan Singh, and Gord Kurtenbach. 2009. Analytic drawing of 3D scaffolds. In ACM Transactions on Graphics (TOG), Vol. 28. ACM, 149. 3, 11, 18 Google Scholar
Digital Library
- Cloud Shao, Adrien Bousseau, Alla Sheffer, and Karan Singh. 2012. CrossShade: shading concept sketches using cross-section curves. ACM Transactions on Graphics (Proc. SIGGRAPH) 31, 4 (2012). 3, 10, 18 Google Scholar
Digital Library
- Alex Shtof, Alexander Agathos, Yotam Gingold, Ariel Shamir, and Daniel Cohen-Or. 2013. Geosemantic Snapping for Sketch-Based Modeling. Computer Graphics Forum 32, 2 (2013), 245--253. 4Google Scholar
Cross Ref
- Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup. ACM Transactions on Graphics (Proc. SIGGRAPH) 35, 4 (2016). 5 Google Scholar
Digital Library
- Suraj Srinivas, Ravi Kiran Sarvadevabhatla, Konda Reddy Mopuri, Nikita Prabhu, Srinivas S. S. Kruthiventi, and R. Venkatesh Babu. 2016. A Taxonomy of Deep Convolutional Neural Nets for Computer Vision. Vol. 2. 36. 3Google Scholar
- Daniel Sýkora, Ladislav Kavan, Martin Čadik, Ondřej Jamriška, Alec Jacobson, Brian Whited, Maryann Simmons, and Olga Sorkine-Hornung. 2014. Ink-and-Ray: Bas-Relief Meshes for Adding Global Illumination Effects to Hand-Drawn Characters. ACM Transactions on Graphics 33 (2014). 4 Google Scholar
Digital Library
- Fang Wang, Le Kang, and Yi Li. 2015b. Sketch-based 3d shape retrieval using convolutional neural networks. In IEEE Computer Vision and Pattern Recognition (CVPR). 1875--1883. 5Google Scholar
- Xiaolong Wang, David Fouhey, and Abhinav Gupta. 2015a. Designing deep networks for surface normal estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 539--547. 4Google Scholar
Cross Ref
- Yunhai Wang, Shmulik Asafi, Oliver van Kaick, Hao Zhang, Daniel Cohen-Or, and Baoquan Chen. 2012. Active Co-Analysis of a Set of Shapes. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) (2012). Issue 4. 9 Google Scholar
Digital Library
- Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua B Tenenbaum. 2016. Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. In Neural Information Processing Systems (NIPS). 5 Google Scholar
Digital Library
- Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A Deep Representation for Volumetric Shape Modeling. IEEE Computer Vision and Pattern Recognition (CVPR) (2015). 5Google Scholar
- Xiaohua Xie, Kai Xu, Niloy J Mitra, Daniel Cohen-Or, Wenyong Gong, Qi Su, and Baoquan Chen. 2013. Sketch-to-Design: Context-Based Part Assembly. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 233--245. 2, 4Google Scholar
- Baoxuan Xu, William Chang, Alla Sheffer, Adrien Bousseau, James McCrae, and Karan Singh. 2014. True2Form: 3D Curve Networks from 2D Sketches via Selective Regularization. ACM Transactions on Graphics (Proc. SIGGRAPH) 33, 4 (2014). 2, 3, 10, 19 Google Scholar
Digital Library
- Qiuying Xu, Yotam Gingold, and Karan Singh. 2015. Inverse Toon Shading: Interactive Normal Field Modeling with Isophotes. In Proceedings of Sketch-Based Interfaces and Modeling (SBIM). 3 Google Scholar
Digital Library
- Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. 2016. Attribute2Image: Conditional Image Generation from Visual Attributes. In European Conference on Computer Vision (ECCV). Springer, 776--791. 5Google Scholar
Cross Ref
- Robert C. Zeleznik, Kenneth P. Herndon, and John F. Hughes. 1996. SKETCH: An Interface for Sketching 3D Scenes. In SIGGRAPH. ACM, 163--170. 2, 3 Google Scholar
Digital Library
- Youyi Zheng, Han Liu, Julie Dorsey, and Niloy Mitra. 2016. SMART CANVAS: Context-inferred Interpretation of Sketches for Preparatory Design Studies. Computer Graphics Forum 35, 2 (2016), 37--48. 4 Google Scholar
Digital Library
Index Terms
3D Sketching using Multi-View Deep Volumetric Prediction
Recommendations
Lifting freehand concept sketches into 3D
We present the first algorithm capable of automatically lifting real-world, vector-format, industrial design sketches into 3D. Targeting real-world sketches raises numerous challenges due to inaccuracies, use of overdrawn strokes, and construction ...
OpenSketch: a richly-annotated dataset of product design sketches
Product designers extensively use sketches to create and communicate 3D shapes and thus form an ideal audience for sketch-based modeling, non-photorealistic rendering and sketch filtering. However, sketching requires significant expertise and time, ...
Symmetry-driven 3D Reconstruction from Concept Sketches
SIGGRAPH '22: ACM SIGGRAPH 2022 Conference ProceedingsConcept sketches, ubiquitously used in industrial design, are inherently imprecise yet highly effective at communicating 3D shape to human observers. We present a new symmetry-driven algorithm for recovering designer-intended 3D geometry from concept ...






Comments