NeuralTailor: reconstructing sewing pattern structures from 3D point clouds of garments

Abstract

The fields of SocialVR, performance capture, and virtual try-on are often faced with a need to faithfully reproduce real garments in the virtual world. One critical task is the disentanglement of the intrinsic garment shape from deformations due to fabric properties, physical forces, and contact with the body. We propose to use a garment sewing pattern, a realistic and compact garment descriptor, to facilitate the intrinsic garment shape estimation. Another major challenge is a high diversity of shapes and designs in the domain. The most common approach for Deep Learning on 3D garments is to build specialized models for individual garments or garment types. We argue that building a unified model for various garment designs has the benefit of generalization to novel garment types, hence covering a larger design domain than individual models would. We introduce NeuralTailor, a novel architecture based on point-level attention for set regression with variable cardinality, and apply it to the task of reconstructing 2D garment sewing patterns from the 3D point cloud garment models. Our experiments show that NeuralTailor successfully reconstructs sewing patterns and generalizes to garment types with pattern topologies unseen during training.

References

1. Seungbae Bang, Maria Korosteleva, and Sung-Hee Lee. 2021. Estimating Garment Patterns from Static Scan Data. Computer Graphics Forum 40, 6 (2021), 273--287. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14272
2. Aric Bartle, Alla Sheffer, Vladimir G. Kim, Danny M. Kaufman, Nicholas Vining, and Floraine Berthouzoz. 2016. Physics-driven pattern adjustment for direct 3D garment editing. ACM Transactions on Graphics 35, 4 (2016), 50--1.
3. Hugo Bertiche, Meysam Madadi, and Sergio Escalera. 2020. CLOTH3D: Clothed 3D Humans. In European Conference on Computer Vision, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Virtual, 344--359.
4. Lukas Biewald. 2020. Experiment Tracking with Weights and Biases. https://www.wandb.com/ Software available from wandb.com.
5. Remi Brouet, Alla Sheffer, Laurence Boissieux, and Marie-Paule Cani. 2012. Design preserving garment transfer. ACM Transactions on Graphics 31, 4 (7 2012), 1--11.
6. Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. 2020. DeepSVG: A hierarchical generative network for vector graphics animation. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 16351--16361.
7. Siddhartha Chaudhuri, Iit Bombay ERSIN YUMER, Adobe Research HAO ZHANG, Jun Li, Kai Xu, Ersin Yumer, and Hao Zhang. 2017. GRASS: Generative Recursive Autoencoders for Shape Struc-tures. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1--14.
8. Xiaowu Chen, Bin Zhou, Feixiang Lu, Lin Wang, Lang Bi, and Ping Tan. 2015. Garment modeling with a depth camera. ACM Transactions on Graphics 34, 6 (2015), 1--12.
9. Enric Corona, Albert Pumarola, Guillem Alenya, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2021. SMPLicit: Topology-Aware Generative Model for Clothed People. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Virtual, 11875--11885.
10. Hein Daanen and Sung Ae Hong. 2008. Made-to-measure pattern development based on 3D whole body scans. International Journal of Clothing Science and Technology 20, 1 (2008), 15--25.
11. Philippe Decaudin, Dan Julius, Jamie Wither, Laurence Boissieux, Alla Sheffer, and Marie Paule Cani. 2006. Virtual garments: A fully geometric approach for clothing design. Computer Graphics Forum 25, 3 (2006), 625--634.
12. Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.
13. Chihiro Goto and Nobuyuki Umetani. 2021. Data-driven Garment Pattern Estimation from 3D Geometries. In Eurographics 2021 - Short Papers. The Eurographics Association, 17--20.
14. Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. 2020. Deep Learning for 3D Point Clouds: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (12 2020), 1--1.
15. David Ha and Douglas Eck. 2018. A neural representation of sketch drawings. In 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings.
16. Nils Hasler, Bodo Rosenhahn, and Hans Peter Seidel. 2007. Reverse engineering garments. In International Conference on Computer Vision/Computer Graphics Collaboration Techniques and Applications. Springer, Berlin, Heidelberg, 200--211.
17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (11 1997), 1735--1780.
18. Takeo Igarashi, Tomer Moscovich, and John F. Hughes. 2005. As-rigid-as-possible shape manipulation. ACM Transactions on Graphics (TOG) 24, 3 (7 2005), 1134--1141.
19. Moon-Hwan Jeong, Dong-Hoon Han, and Hyeong-Seok Ko. 2015. Garment capture from a photograph. Computer Animation and Virtual Worlds 26, 3--4 (2015), 291--300.
20. Boyi Jiang, Juyong Zhang, Yang Hong, Jinhao Luo, Ligang Liu, and Hujun Bao. 2020. BCNet: Learning Body and Cloth Shape from a Single Image. In Computer Vision - ECCV 2020. Springer International Publishing, Cham, 18--35.
21. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR). International Conference on Learning Representations, ICLR, San-Diego, USA, abs/1412.6980. https://arxiv.org/abs/1412.6980v9
22. Maria Korosteleva and Sung-Hee Lee. 2021a. Dataset of 3D Garments with Sewing Patterns.
23. Maria Korosteleva and Sung-Hee Lee. 2021b. Generating Datasets of 3D Garments with Sewing Patterns. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung (Eds.), Vol. 1. Virtual.
24. Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. 2019. Multi-Garment Net: Learning to Dress 3D People from Images. In The IEEE International Conference on Computer Vision (ICCV). IEEE, Seoul, Republic of Korea, 5420--5430.
25. Minchen Li, Alla Sheffer, Eitan Grinspun, and Nicholas Vining. 2018. FoldSketch: Enriching garments with physically reproducible folds. ACM Transactions on Graphics 37, 4 (8 2018), 1--13.
26. Kaixuan Liu, Xianyi Zeng, Pascal Bruniaux, Xuyuan Tao, Xiaofeng Yao, Victoria Li, and Jianping Wang. 2018. 3D interactive garment pattern-making technology. CAD Computer Aided Design 104 (11 2018), 113--124.
27. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-moll, and Michael J Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM transactions on graphics (TOG) 34, 6 (2015), 1--16.
28. Mickaël Ly, Romain Casati, Florence Bertails-Descoubes, Mélina Skouras, and Laurence Boissieux. 2018. Inverse Elastic Shell Design with Contact and Friction. ACM Transaction on Graphics (TOG) 37, 6 (2018), 1--16.
29. Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J Black. 2020. Learning to Dress 3D People in Generative Clothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Virtual, 6469--6478. https://cape.is.tue.mpg.de.
30. Qianli Ma, Jinlong Yang, Siyu Tang, and Michael J Black. 2021. The Power of Points for Modeling Humans in Clothing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Virtual, 10974--10984.
31. Andre Martins and Ramon Astudillo. 2016. From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 1614--1623. https://proceedings.mlr.press/v48/martins16.html
32. Yuwei Meng, Charlie C.L. Wang, and Xiaogang Jin. 2012. Flexible shape control for automatic resizing of apparel products. CAD Computer Aided Design 44, 1 (2012), 68--76.
33. Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas J. Guibas. 2019. StructureNet: Hierarchical Graph Networks for 3D Shape Generation. ACM Transactions on Graphics (TOG) 38, 6 (8 2019), 1--19. http://arxiv.org/abs/1908.00575
34. Juan Montes, Bernhard Thomaszewski, Sudhir Mudur, and Tiberiu Popa. 2020. Computational Design of Skintight Clothing. ACM Trans. Graph 39, 4 (2020), 12.
35. Vacit Oguz Yazici, Abel Gonzalez-Garcia, Arnau Ramisa, Bartlomiej Twardowski, and Joost van de Weijer. 2020. Orderless Recurrent Models for Multi-label Classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Virtual, 13440--13449.
36. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024--8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
37. Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. 2020. TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Virtual, 7365--7375. http://arxiv.org/abs/2003.04583
38. Dominic Roberts, Ara Danielyan, Hang Chu, Mani Golparvar-Fard, and David Forsyth. 2021. LSD-StructureNet: Modeling Levels of Structural Detail in 3D Part Hierarchies. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, Virtual, 5836--5845. http://arxiv.org/abs/2108.13459
39. Igor Santesteban, Miguel A. Otaduy, and Dan Casas. 2019. Learning-Based Animation of Clothing for Virtual Try-On. Computer Graphics Forum 38, 2 (2019), 355--366.
40. Igor Santesteban, Nils Thuerey, Miguel A Otaduy, and Dan Casas. 2021. Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 11763--11773.
41. Matthew Schultz and Thorsten Joachims. 2003. Learning a Distance Metric from Relative Comparisons. Advances in neural information processing systems 16 (2003), 41--48.
42. Nicholas Sharp and Keenan Crane. 2018. Variational surface cutting. ACM Transactions on Graphics 37, 4 (2018), 1--13.
43. Yu Shen, Junbang Liang, and Ming C. Lin. 2020. GAN-Based Garment Generation Using Sewing Pattern Images. In Computer Vision-ECCV 2020: 16th European Conference, Vol. 16. Springer International Publishing, Glasgow, UK, 225--247.
44. Leslie N. Smith. 2018. A Disciplined Approach to Neural Network Hyper-Parameters: PART 1 - Learning Rate, Batch Size, Momentum, and Weight Decay. https://github.com/lnsmith54/hyperParam1.
45. Zhaoqi Su, Tao Yu, Yangang Wang, Yipeng Li, and Yebin Liu. 2020. DeepCloth : Neural Garment Representation for Shape and Style Editing. arXiv preprint arXiv:2011.14619 (2020), 1--10.
46. Garvita Tiwari, Bharat Lal Bhatnagar, Tony Tung, and Gerard Pons-Moll. 2020. SIZER: A Dataset and Model for Parsing 3D Clothing and Learning Size Sensitive 3D Clothing. http://arxiv.org/abs/2007.11610
47. Charlie C.L. Wang, Yu Wang, and Matthew M.F. Yuen. 2003. Feature based 3D garment design through 2D sketches. CAD Computer Aided Design 35, 7 (6 2003), 659--672.
48. Charlie C.L. Wang, Yu Wang, and Matthew M.F. Yuen. 2005. Design automation for customized apparel products. CAD Computer Aided Design 37, 7 (6 2005), 675--691.
49. Huamin Wang. 2018. Rule-free sewing pattern adjustment with precision and efficiency. ACM Transactions on Graphics 37, 4 (2018), 1--13.
50. Jin Wang, Guodong Lu, Weilong Li, Long Chen, and Yoshiyuki Sakaguti. 2009. Interactive 3D garment design with constrained contour curves and style curves. CAD Computer Aided Design 41, 9 (9 2009), 614--625.
51. Kai Wang, Paul Guerrero, Vladimir Kim, Siddhartha Chaudhuri, Minhyuk Sung, and Daniel Ritchie. 2021. The Shape Part Slot Machine: Contact-based Reasoning for Generating 3D Shapes from Parts. arXiv preprint arXiv:2112.00584 (2021), 1--19. http://arxiv.org/abs/2112.00584
52. Tuanfeng Y. Wang, Duygu Ceylan, Jovan Popović, and Niloy J. Mitra. 2018a. Learning a shared shape space for multimodal garment design. ACM Transactions on Graphics 37, 6 (12 2018), 1--13.
53. Tuanfeng Y Wang and MiHoYo Inc. 2019. Learning an Intrinsic Garment Space for Interactive Authoring of Garment Animation. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--12.
54. Yizhi Wang and Zhouhui Lian. 2021. DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1--15. http://arxiv.org/abs/2110.06688
55. Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. 2018b. Dynamic Graph CNN for Learning on Point Clouds. ACM Transactions on Graphics 38, 5 (1 2018), Article 146. http://arxiv.org/abs/1801.07829
56. Katja Wolff, Philipp Herholz, Verena Ziegler, Frauke Link, Nico Brügel, and Olga Sorkine-Hornung. 2021. 3D Custom Fit Garment Design with Body Movement. arXiv preprint arXiv:2102.05462 (2021), 1--12. http://arxiv.org/abs/2102.05462
57. Zhijie Wu, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2019. SagNet: Structure-aware generative network for 3D-shape modeling. ACM Transactions on Graphics 38, 4 (7 2019), 1--14.
58. Shan Yang, Zherong Pan, Tanya Amert, Ke Wang, Licheng Yu, Tamara Berg, and Ming C. Lin. 2018. Physics-inspired garment recovery from a single-view image. ACM Transactions on Graphics 37, 5 (11 2018), 1--14.
59. Yang Yunchu and Zhang Weiyuan. 2007. Prototype garment pattern flattening based on individual 3D virtual dummy. International Journal of Clothing Science and Technology 19, 5 (2007), 334--348.
60. Ilya Zakharkin, Kirill Mazur, Artur Grigorev, and Victor Lempitsky. 2021. Point-Based Modeling of Human Clothing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Virtual, 14718--14727. http://arxiv.org/abs/2104.08230
61. Heming Zhu, Yu Cao, Hang Jin, Weikai Chen, Dong Du, Zhangye Wang, Shuguang Cui, and Xiaoguang Han. 2020. Deep Fashion3D: A Dataset and Benchmark for 3D Garment Reconstruction from Single Images. http://arxiv.org/abs/2003.12753https://kv2000.github.io/2020/03/25/deepFashion3DRevisited/

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Reconstruction

        • Shape inference

      • Computer vision representations

        • Shape representations

  • Computer graphics

    • Shape modeling

      • Shape analysis