SurfaceBrush: from virtual reality drawings to manifold surfaces

Abstract

Popular Virtual Reality (VR) tools allow users to draw varying-width, ribbonlike 3D brush strokes by moving a hand-held controller in 3D space. Artists frequently use dense collections of such strokes to draw virtual 3D shapes. We propose SurfaceBrush, a surfacing method that converts such VR drawings into user-intended manifold free-form 3D surfaces, providing a novel approach for modeling 3D shapes. The inputs to our method consist of dense collections of artist-drawn stroke ribbons described by the positions and normals of their central polylines, and ribbon widths. These inputs are highly distinct from those handled by existing surfacing frameworks and exhibit different sparsity and error patterns, necessitating a novel surfacing approach. We surface the input stroke drawings by identifying and leveraging local coherence between nearby artist strokes. In particular, we observe that strokes intended to be adjacent on the artist imagined surface often have similar tangent directions along their respective polylines. We leverage this local stroke direction consistency by casting the computation of the user-intended manifold surface as a constrained matching problem on stroke polyline vertices and edges. We first detect and smoothly connect adjacent similarly-directed sequences of stroke edges producing one or more manifold partial surfaces. We then complete the surfacing process by identifying and connecting adjacent similarly directed edges along the borders of these partial surfaces. We confirm the usability of the SurfaceBrush interface and the validity of our drawing analysis via an observational study. We validate our stroke surfacing algorithm by demonstrating an array of manifold surfaces computed by our framework starting from a range of inputs of varying complexity, and by comparing our outputs to reconstructions computed using alternative means.

References

1. Fatemeh Abbasinejad, Pushkar Joshi, and Nina Amenta. 2012. Surface Patches from Unorganized Space Curves. In Proc. Symposium on Computational Geometry. 417--418.
2. Judith Amores and Jaron Lanier. 2017. HoloARt: Painting with Holograms in Mixed Reality. In Proc. Human Factors in Computing Systems. 421--424.
3. Haim Avron, Andrei Sharf, Chen Greif, and Daniel Cohen-Or. 2010. ℓ₁ Sparsee Reconstruction of Sharp Point Set Surfaces. ACM Trans. Graph. (2010), 135:1--135:12.
4. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: As-natural-as-possible Sketching System for Creating 3D Curve Models. In Proc. Symposium on User Interface Software and Technology. 151--160.
5. Nikhil Bansal, Avrim Blum, and Shuchi Chawla. 2004. Correlation clustering. Machine Learning 56, 1--3 (2004), 89--113.
6. Gavin Barill, Neil G. Dickson, Ryan Schmidt, David I. W. Levin, and Alec Jacobson. 2018. Fast Winding Numbers for Soups and Clouds. ACM Trans. Graph. (2018), 43:1--43:12.
7. Matthew Berger, Andrea Tagliasacchi, Lee M. Seversky, Pierre Alliez, Gaël Guennebaud, Joshua A. Levine, Andrei Sharf, and Claudio T. Silva. 2017. A Survey of Surface Reconstruction from Point Clouds. Comput. Graph. Forum 36, 1 (2017), 301--329.
8. F. Bernardini, J. Mittleman, H. Rushmeier, C. Silva, and G. Taubin. 1999. The ball-pivoting algorithm for surface reconstruction. IEEE Transactions on Visualization and Computer Graphics 5, 4 (1999), 349--359.
9. Mikhail Bessmeltsev, Caoyu Wang, Alla Sheffer, and Karan Singh. 2012. Design-Driven Quadrangulation of Closed 3D Curves. ACM Trans.Grap. 31, 5 (2012).
10. CGAL. 2018. CGAL The Computational Geometry Algorithms Library. https://www.cgal.org/
11. DesignSpace. 2018. DesignSpace VR. http://www.designspacevr.org/
12. Tamal K. Dey and Samrat Goswami. 2003. Tight Cocone: A Water-tight Surface Reconstructor. J. Comput. Inf. Sci. Eng. 3, 4 (2003), 302--307.
13. Holger Diehl, Franz Müller, and Udo Lindemann. 2004. From raw 3D-Sketches to exact CAD product models Concept for an assistant-system. In Sketch Based Interfaces and Modeling.
14. Herbert Edelsbrunner and Ernst P. Mücke. 1994. Three-dimensional Alpha Shapes. ACM Trans. Graph. 13, 1 (1994), 43--72.
15. Mark Finch and Hugues Hoppe. 2011. Freeform Vector Graphics with Controlled Thin-Plate Splines. ACM Trans. Graph. 30, 6 (2011).
16. Michele Fiorentino, Raffaele de Amicis, Giuseppe Monno, and Andre Stork. 2002. Spacedesign: A Mixed Reality Workspace for Aesthetic Industrial Design. In Proc. Symposium on Mixed and Augmented Reality.
17. Hongbo Fu, Shizhe Zhou, Ligang Liu, and Niloy J. Mitra. 2011. Animated Construction of Line Drawings. ACM Trans. Graph. (2011), 133:1--133:10.
18. Simon Fuhrmann and Michael Goesele. 2014. Floating Scale Surface Reconstruction. ACM Transactions on Graphics 33 (2014), 1--11.
19. Kun Gao and Alyn Rockwood. 2005. Multi-sided attribute based modeling. Mathematics of Surfaces XI (2005), 219--232.
20. GoogleBlocks. 2018. Google Blocks. https://vr.google.com/blocks/
21. GravitySketch. 2018. Gravity Sketch. https://www.gravitysketch.com/
22. Cindy Grimm and Pushkar Joshi. 2012. Just DrawIt: A 3D Sketching System. In Proc. Symposium on Sketch-Based Interfaces and Modeling. 121--130.
23. Tovi Grossman, Ravin Balakrishnan, Gordon Kurtenbach, George Fitzmaurice, Azam Khan, and Bill Buxton. 2002. Creating Principal 3D Curves with Digital Tape Drawing. In Proc. Human Factors in Computing Systems. 121--128.
24. Zhiyang Huang, Ming Zou, Nathan Carr, and Tao Ju. 2017. Topology-controlled Reconstruction of Multi-labelled Domains from Cross-sections. ACM Trans. Graph. 36, 4 (2017), 76:1--76:12.
25. J.H. Israel, E. Wiese, M. Mateescu, C. Zöllner, and R. Stark. 2009. Investigating three-dimensional sketching for early conceptual design---Results from expert discussions and user studies. Computers and Graphics (2009), 462 -- 473.
26. B. Jackson and D. F. Keefe. 2016. Lift-Off: Using Reference Imagery and Freehand Sketching to Create 3D Models in VR. IEEE Trans. on Visualization and Computer Graphics 22, 4 (2016), 1442--1451.
27. Tao Ju. 2004. Robust Repair of Polygonal Models. ACM Trans. Graph. (2004), 888--895.
28. R. Karp. 1972. Reducibility among combinatorial problems. In Complexity of Computer Computations. Plenum Press, 85--103.
29. Michael Kazhdan and Hugues Hoppe. 2013. Screened Poisson Surface Reconstruction. ACM Trans. Graph. 32, 3 (2013), 29:1--29:13.
30. D. Keefe, R. Zeleznik, and D. Laidlaw. 2007. Drawing on Air: Input Techniques for Controlled 3D Line Illustration. IEEE Transactions on Visualization and Computer Graphics 13, 5 (2007), 1067--1081.
31. Daniel F. Keefe, Daniel Acevedo Feliz, Tomer Moscovich, David H. Laidlaw, and Joseph J. LaViola, Jr. 2001. CavePainting: A Fully Immersive 3D Artistic Medium and Interactive Experience. In Proc. Symposium on Interactive 3D Graphics. 85--93.
32. Margret Keuper, Evgeny Levinkov, Nicolas Bonneel, Guillaume Lavoué, Thomas Brox, and Bjorn Andres. 2015. Efficient decomposition of image and mesh graphs by lifted multicuts. In Proc. ICCV. 1751--1759.
33. Yongkwan Kim, Sang-Gyun An, Joon Hyub Lee, and Seok-Hyung Bae. 2018. Agile 3D Sketching with Air Scaffolding. In Proc. Human Factors in Computing Systems. 238:1--238:12.
34. Kodon. 2018. TenkLabs Kodon. https://www.tenklabs.com/kodon
35. Ravikrishna Kolluri, Jonathan Richard Shewchuk, and James F. O'Brien. 2004. Spectral Surface Reconstruction from Noisy Point Clouds. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP '04). 11--21.
36. Jung-hoon Kwon, Han-wool Choi, Jeong-in Lee, and Young-Ho Chai. 2005. Free-Hand Stroke Based NURBS Surface for Sketching and Deforming 3D Contents. In Proc. Pacific-Rim Conference on Advances in Multimedia Information Processing. 315--326.
37. Changjian Li, Hao Pan, Yang Liu, Xin Tong, Alla Sheffer, and Wenping Wang. 2017. BendSketch: Modeling Freeform Surfaces Through 2D Sketching. ACM Trans. Graph. (2017), 125:1--125:14.
38. Chenxi Liu, Enrique Rosales, and Alla Sheffer. 2018. StrokeAggregator: Consolidating Raw Sketches into Artist-Intended Curve Drawings. ACM Trans. Graph. 37, 4 (2018).
39. Xueting Liu, Tien-Tsin Wong, and Pheng-Ann Heng. 2015. Closure-aware Sketch Simplification. ACM Trans. Graph. (2015), 168:1--168:10.
40. A. Nasri, M. Sabin, and Z. Yasseen. 2009. Filling N-Sided Regions by Quad Meshes for Subdivision Surfaces. Computer Graphics Forum 28, 6 (2009), 1644--1658.
41. Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Marc Alexa. 2007. FiberMesh: Designing Freeform Surfaces with 3D Curves. ACM Trans. Graph. 26, 3 (2007).
42. G. Noris, D. Sýkora, A. Shamir, S. Coros, B. Whited, M. Simmons, A. Hornung, M. Gross, and R. Sumner. 2012. Smart Scribbles for Sketch Segmentation. Comput. Graph. Forum (2012), 2516--2527.
43. OculusMedium. 2016. OculusMedium. https://oculus.com/medium/
44. L. Olsen, F.F. Samavati, M.C. Sousa, and J. Jorge. 2009. Sketch-Based Modeling: A Survey. Computers & Graphics 33 (2009). Issue 1.
45. A. C. Öztireli, G. Guennebaud, and M. Gross. 2009. Feature Preserving Point Set Surfaces based on Non-Linear Kernel Regression. Computer Graphics Forum (2009).
46. PaintLab. 2018. PaintLab VR. http://paintlabvr.com/
47. Hao Pan, Yang Liu, Alla Sheffer, Nicholas Vining, Chang-Jian Li, and Wenping Wang. 2015. Flow Aligned Surfacing of Curve Networks. ACM Trans. Graph. 34, 4 (2015), 127:1--127:10.
48. Poly. 2018. Tilt Brush sketches. https://poly.google.com/tiltbrush
49. E. Sachs, A. Roberts, and D. Stoops. 1991. 3-Draw: a tool for designing 3D shapes. IEEE Computer Graphics and Applications 11, 6 (1991), 18--26.
50. S. Schaefer, J. Warren, and D. Zorin. 2004. Lofting curve networks using subdivision surfaces. Proc. Symposium on Geometry Processing (2004), 103.
51. Steven Schkolne, Michael Pruett, and Peter Schröder. 2001. Surface Drawing: Creating Organic 3D Shapes with the Hand and Tangible Tools. In Proc. Human Factors in Computing Systems. 261--268.
52. S. Schkolne and P. Schroeder. 1999. Surface Drawing. Caltech Department of Computer Science Technical Report CS-TR-99-03.
53. Ryan Schmidt, Azam Khan, Karan Singh, and Gord Kurtenbach. 2009. Analytic Drawing of 3D Scaffolds. ACM Trans. Graph. 28, 5 (2009), 149:1--149:10.
54. Cloud Shao, Adrien Bousseau, Alla Sheffer, and Karan Singh. 2012. CrossShade: Shading Concept Sketches Using Cross-Section Curves. ACM Trans. Graph. 31, 4 (2012).
55. ShapeLab. 2018. ShapeLab. https://store.steampowered.com/app/571890/ShapeLab/
56. Ojaswa Sharma and Nidhi Agarwal. 2016. 3D Surface Reconstruction from Unorganized Sparse Cross Sections. In Proc. Graphics Interface. 33--40.
57. Chen Shen, James F. O'Brien, and Jonathan R. Shewchuk. 2004. Interpolating and Approximating Implicit Surfaces from Polygon Soup. ACM Trans. Graph. (2004), 896--904.
58. Sketchfab. 2018. Tilt Brush. https://sketchfab.com/tags/tiltbrush
59. Thomas F. Stahovich, Eric J. Peterson, and Hanlung Lin. 2014. An efficient, classification-based approach for grouping pen strokes into objects. Computers & Graphics (2014), 14 -- 30.
60. Shun'ichi Tano, T. Kodera, Takashi Nakashima, I. Kawano, K. Nakanishi, G. Hamagishi, M. Inoue, A. Watanabe, T. Okamoto, K. Kawagoe, K. Kaneko, T. Hotta, and M. Tatsuoka. 2003. Godzilla: Seamless 2D and 3D Sketch Environment for Reflective and Creative Design Work. In INTERACT.
61. Shun'ichi Tano, Shinya Yamamoto, Junko Ichino, Tomonori Hashiyama, and Mitsuru Iwata. 2013. Truly Useful 3D Drawing System for Professional Designer by "Life-Sized and Operable" Feature and New Interaction. In Human-Computer Interaction - INTERACT 2013. 37--55.
62. TiltBrush. 2018. Google TiltBrush. https://tiltbrush.com/
63. Anil Usumezbas, Ricardo Fabbri, and Benjamin B. Kimia. 2017. The Surfacing of Multiview 3D Drawings via Lofting and Occlusion Reasoning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 4560--4569.
64. T. Várady, Alyn Rockwood, and P. Salvi. 2011. Transfinite surface interpolation over irregular n-sided domains. Computer-Aided Design iv (2011).
65. A. Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory 13, 2 (1967), 260--269.
66. Ruimin Wang, Ligang Liu, Zhouwang Yang, Kang Wang, Wen Shan, Jiansong Deng, and Falai Chen. 2016. Construction of Manifolds via Compatible Sparse Representations. ACM Trans. Graph. (2016), 14:1--14:10.
67. Gerold Wesche and Hans-Peter Seidel. 2001. FreeDrawer: A Free-form Sketching System on the Responsive Workbench. In Proc. Virtual Reality Software and Technology. 167--174.
68. Jun Xing, Hsiang-Ting Chen, and Li-Yi Wei. 2014. Autocomplete Painting Repetitions. ACM Trans. Graph. (2014), 172:1--172:11.
69. Shiyao Xiong, Juyong Zhang, Jianmin Zheng, Jianfei Cai, and Ligang Liu. 2014. Robust Surface Reconstruction via Dictionary Learning. ACM Trans. Graph. (2014), 201:1--201:12.
70. Baoxuan Xu, William Chang, Alla Sheffer, Adrien Bousseau, James McCrae, and Karan Singh. 2014. True2Form: 3D Curve Networks from 2D Sketches via Selective Regularization. ACM Trans. Graph. 33, 4 (2014), 131:1--131:13.
71. Ming Zou, Michelle Holloway, Nathan Carr, and Tao Ju. 2015. Topology-constrained Surface Reconstruction from Cross-sections. ACM Trans. Graph. 34, 4 (2015), 128:1--128:10.

Index Terms

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

      • Virtual reality

    • Shape modeling

      • Mesh geometry models