Real-time data-driven interactive rough sketch inking

Abstract

We present an interactive approach for inking, which is the process of turning a pencil rough sketch into a clean line drawing. The approach, which we call the Smart Inker, consists of several "smart" tools that intuitively react to user input, while guided by the input rough sketch, to efficiently and naturally connect lines, erase shading, and fine-tune the line drawing output. Our approach is data-driven: the tools are based on fully convolutional networks, which we train to exploit both the user edits and inaccurate rough sketch to produce accurate line drawings, allowing high-performance interactive editing in real-time on a variety of challenging rough sketch images. For the training of the tools, we developed two key techniques: one is the creation of training data by simulation of vague and quick user edits; the other is a line normalization based on learning from vector data. These techniques, in combination with our sketch-specific data augmentation, allow us to train the tools on heterogeneous data without actual user interaction. We validate our approach with an in-depth user study, comparing it with professional illustration software, and show that our approach is able to reduce inking time by a factor of 1.8X, while improving the results of amateur users.

References

1. Maher Ahmed and Rabab Ward. 2002. A rotation invariant rule-based thinning algorithm for character recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 12 (2002), 1672--1678.
2. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: As-natural-as-possible Sketching System for Creating 3D Curve Models. In ACM Symposium on User Interface Software and Technology. 151--160.
3. Houssem Chatbri and Keisuke Kameyama. 2014. Using scale space filtering to make thinning algorithms robust against noise in sketch images. Pattern Recognition Letters 42 (2014), 1--10.
4. Jiazhou Chen, Gaël Guennebaud, Pascal Barla, and Xavier Granier. 2013. Non-Oriented MLS Gradient Fields. Computer Graphics Forum 32, 8 (2013), 98--109.
5. David H Douglas and Thomas K Peucker. 1973. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica: The International Journal for Geographic Information and Geovisualization 10, 2 (1973), 112--122.
6. Charles R Dyer and Azriel Rosenfeld. 1979. Thinning algorithms for gray-scale pictures. IEEE Transactions on Pattern Analysis and Machine Intelligence 1 (1979), 88--89.
7. Mathias Eitz, James Hays, and Marc Alexa. 2012. How Do Humans Sketch Objects? ACM Transactions on Graphics (Proceedings of SIGGRAPH) 31, 4 (2012), 44:1--44:10.
8. Jean-Dominique Favreau, Florent Lafarge, and Adrien Bousseau. 2016. Fidelity vs. Simplicity: a Global Approach to Line Drawing Vectorization. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 35, 4 (2016).
9. Jakub Fišer, Paul Asente, Stephen Schiller, and Daniel Sýkora. 2015. ShipShape: A Drawing Beautification Assistant. In Workshop on Sketch-Based Interfaces and Modeling. 49--57.
10. Cindy Grimm and Pushkar Joshi. 2012. Just Drawlt: A 3D Sketching System. In nternational Symposium on Sketch-Based Interfaces and Modeling. 121--130.
11. David Ha and Douglas Eck. 2018. A Neural Representation of Sketch Drawings. In International Conference on Learning Representations.
12. Xavier Hilaire and Karl Tombre. 2006. Robust and accurate vectorization of line drawings. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 6 (2006), 890--904.
13. Takeo Igarashi, Satoshi Matsuoka, Sachiko Kawachiya, and Hidehiko Tanaka. 1997. Interactive Beautification: A Technique for Rapid Geometric Design. In ACM Symposium on User Interface Software and Technology. 105--114.
14. Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning.
15. Jang and Roland T. Chin. 1990. Analysis of thinning algorithms using mathematical morphology. IEEE Transactions on Pattern Analysis and Machine Intelligence 12, 6 (1990), 541--551.
16. Louisa Lam, Seong-Whan Lee, and Ching Y Suen. 1992. Thinning methodologies-a comprehensive survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 9 (1992), 869--885.
17. David Lindlbauer, Michael Haller, Mark S. Hancock, Stacey D. Scott, and Wolfgang Stuerzlinger. 2013. Perceptual grouping: selection assistance for digital sketching. In International Conference on Interactive Tabletops and Surfaces. 51--60.
18. Xueting Liu, Tien-Tsin Wong, and Pheng-Ann Heng. 2015. Closure-aware Sketch Simplification. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 34, 6 (2015), 168:1--168:10.
19. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50--60.
20. Gary Martin, Steve Rude, and Terry Austin. 1997. The Art of Comic Book Inking. Dark Horse Comics.
21. Ana Maria Mendonca and Aurelio Campilho. 2006. Segmentation of retinal blood vessels by combining the detection of centerlines and morphological reconstruction. IEEE Transactions on Medical Imaging 25, 9 (2006), 1200--1213.
22. Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning. 807--814.
23. Gioacchino Noris, Alexander Hornung, Robert W Sumner, Maryann Simmons, and Markus Gross. 2013. Topology-driven Vectorization of Clean Line Drawings. ACM Transactions on Graphics 32, 1 (2013), 4:1--4:11.
24. Günay Orbay and Levent Burak Kara. 2011. Beautification of Design Sketches Using Trainable Stroke Clustering and Curve Fitting. IEEE Transactions on Visualization and Computer Graphics 17, 5 (2011), 694--708.
25. Urs Ramer. 1972. An iterative procedure for the polygonal approximation of plane curves. Computer graphics and image processing 1, 3 (1972), 244--256.
26. Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. 2017. Scribbler: Controlling Deep Image Synthesis with Sketch and Color. In IEEE Conference on Computer Vision and Pattern Recognition.
27. Amit Shesh and Baoquan Chen. 2008. Efficient and Dynamic Simplification of Line Drawings. Computer Graphics Forum 27, 2 (2008), 537--545.
28. Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In IEEE Conference on Computer Vision and Pattern Recognition.
29. Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2018. Mastering Sketching: Adversarial Augmentation for Structured Prediction. ACM Transactions on Graphics 37, 1 (2018).
30. Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 35, 4 (2016).
31. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15 (2014), 1929--1958.
32. Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. 2015. Efficient object localization using convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition.
33. Jun Xie, Holger Winnemöller, Wilmot Li, and Stephen Schiller. 2017. Interactive Vectorization. In ACM CHI Conference on Human Factors in Computing Systems.
34. Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. arXiv preprint arXiv:1212.5701 (2012).
35. Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017. Real-Time User-Guided Image Colorization with Learned Deep Priors. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 9, 4 (2017).
36. TY Zhang and Ching Y. Suen. 1984. A fast parallel algorithm for thinning digital patterns. Commun. ACM 27, 3 (1984), 236--239.

Index Terms

• Applied computing

  • Arts and humanities

    • Fine arts

• Computing methodologies

  • Computer graphics

    • Image manipulation

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction techniques

      • Gestural input