Abstract
Regularization is used to avoid overfitting when training a neural network; unfortunately, this reduces the attainable level of detail hindering the ability to capture high-frequency information present in the training data. Even though various approaches may be used to re-introduce high-frequency detail, it typically does not match the training data and is often not time coherent. In the case of network inferred cloth, these sentiments manifest themselves via either a lack of detailed wrinkles or unnaturally appearing and/or time incoherent surrogate wrinkles. Thus, we propose a general strategy whereby high-frequency information is procedurally embedded into low-frequency data so that when the latter is smeared out by the network the former still retains its high-frequency detail. We illustrate this approach by learning texture coordinates which when smeared do not in turn smear out the high-frequency detail in the texture itself but merely smoothly distort it. Notably, we prescribe perturbed texture coordinates that are subsequently used to correct the over-smoothed appearance of inferred cloth, and correcting the appearance from multiple camera views naturally recovers lost geometric information.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Recovering Geometric Information with Learned Texture Perturbations
- [n.d.]. PhysBAM: physically based animation. http://physbam.stanford.edu.Google Scholar
- Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. 2019a. Learning to reconstruct people in clothing from a single RGB camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1175--1186.Google Scholar
Cross Ref
- Thiemo Alldieck Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. 2018a. Detailed human avatars from monocular video. In 2018 International Conference on 3D Vision (3DV). IEEE, 98--109.Google Scholar
- Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. 2018b. Video based reconstruction of 3d people models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8387--8397.Google Scholar
Cross Ref
- Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Marcus Magnor. 2019b. Tex2Shape: Detailed Full Human Body Geometry from a Single Image. In Proceedings of the International Conference on Computer Vision (ICCV). IEEE.Google Scholar
Cross Ref
- Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: shape completion and animation of people. In ACM transactions on graphics (TOG), Vol. 24. ACM, 408--416.Google Scholar
- Alexandru O Bălan and Michael J Black. 2008. The naked truth: Estimating body shape under clothing. In European Conference on Computer Vision. Springer, 15--29.Google Scholar
Digital Library
- David Baraff and Andrew Witkin. 1998. Large steps in cloth simulation. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. ACM, 43--54.Google Scholar
Digital Library
- David Baraff, Andrew Witkin, and Michael Kass. 2003. Untangling cloth. In ACM Transactions on Graphics (TOG), Vol. 22. ACM, 862--870.Google Scholar
Digital Library
- Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. In Advances in Neural Information Processing Systems. 3189--3197.Google Scholar
- Derek Bradley, Tamy Boubekeur, and Wolfgang Heidrich. 2008a. Accurate multi-view reconstruction using robust binocular stereo and surface meshing. In 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1--8.Google Scholar
Cross Ref
- Derek Bradley, Tiberiu Popa, Alla Sheffer, Wolfgang Heidrich, and Tamy Boubekeur. 2008b. Markerless garment capture. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 99.Google Scholar
Digital Library
- Robert Bridson, Ronald Fedkiw, and John Anderson. 2002. Robust treatment of collisions, contact and friction for cloth animation. In ACM Transactions on Graphics (ToG), Vol. 21. ACM, 594--603.Google Scholar
Digital Library
- Robert Bridson, Sebastian Marino, and Ronald Fedkiw. 2003. Simulation of clothing with folds and wrinkles. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, 28--36.Google Scholar
Digital Library
- Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine 34, 4 (2017), 18--42.Google Scholar
- Thomas Buffet, Damien Rohmer, Loic Barthe, Laurence Boissieux, and Marie-Paule Cani. 2019. Implicit untangling: A robust solution for modeling layered clothing. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.Google Scholar
Digital Library
- Matthew Cong, Michael Bao, Jane L E, Kiran S Bhat, and Ronald Fedkiw. 2015. Fully automatic generation of anatomical face simulation models. In Proceedings of the 14th ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, 175--183.Google Scholar
Digital Library
- R Daněřek, Endri Dibra, Cengiz Öztireli, Remo Ziegler, and Markus Gross. 2017. Deepgarment: 3d garment shape estimation from a single image. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 269--280.Google Scholar
- Edilson De Aguiar, Leonid Sigal, Adrien Treuille, and Jessica K Hodgins. 2010. Stable spaces for real-time clothing. In ACM Transactions on Graphics (TOG), Vol. 29. ACM, 106.Google Scholar
Digital Library
- Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. 2016. A learned representation for artistic style. arXiv preprint arXiv:1610.07629 (2016).Google Scholar
- James D Foley, Foley Dan Van, Andries Van Dam, Steven K Feiner, John F Hughes, J Hughes, and Edward Angel. 1996. Computer graphics: principles and practice. Vol. 12110. Addison-Wesley Professional.Google Scholar
- Jean-Sébastien Franco, Marc Lapierre, and Edmond Boyer. 2006. Visual shapes of silhouette sets. In Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06). IEEE, 397--404.Google Scholar
Digital Library
- Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2015. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576 (2015).Google Scholar
- Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2414--2423.Google Scholar
Cross Ref
- Zhenglin Geng, Daniel Johnson, and Ronald Fedkiw. 2020. Coercing machine learning to output physically accurate results. J. Comput. Phys. 406 (2020), 109099.Google Scholar
Cross Ref
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT press.Google Scholar
Digital Library
- Peng Guan, Loretta Reiss, David A Hirshberg, Alexander Weiss, and Michael J Black. 2012. DRAPE: DRessing Any PErson. ACM Trans. Graph. 31, 4 (2012), 35--1.Google Scholar
- Erhan Gundogdu, Victor Constantin, Amrollah Seifoddini, Minh Dang, Mathieu Salzmann, and Pascal Fua. 2019. Garnet: A two-stream network for fast and accurate 3d cloth draping. In Proceedings of the IEEE International Conference on Computer Vision. 8739--8748.Google Scholar
Cross Ref
- Agrim Gupta, Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2017. Characterizing and improving stability in neural style transfer. In Proceedings of the IEEE International Conference on Computer Vision. 4067--4076.Google Scholar
Cross Ref
- Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt. 2019. Livecap: Real-time human performance capture from monocular video. ACM Transactions on Graphics (TOG) 38, 2 (2019), 14.Google Scholar
Digital Library
- Fabian Hahn, Bernhard Thomaszewski, Stelian Coros, Robert W Sumner, Forrester Cole, Mark Meyer, Tony DeRose, and Markus Gross. 2014. Subspace clothing simulation using adaptive bases. ACM Transactions on Graphics (TOG) 33, 4 (2014), 105.Google Scholar
Digital Library
- Richard I Hartley and Peter Sturm. 1997. Triangulation. Computer vision and image understanding 68, 2 (1997), 146--157. Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or. 2020. Deep geometric texture synthesis. arXiv preprint arXiv:2007.00074 (2020).Google Scholar
- Peng Huang, Margara Tejera, John Collomosse, and Adrian Hilton. 2015. Hybrid skeletal-surface motion graphs for character animation from 4d performance capture. ACM Transactions on Graphics (ToG) 34, 2 (2015), 17.Google Scholar
Digital Library
- Boyi Jiang, Juyong Zhang, Yang Hong, Jinhao Luo, Ligang Liu, and Hujun Bao. 2020. BCNet: Learning Body and Cloth Shape from A Single Image. arXiv preprint arXiv:2004.00214 (2020).Google Scholar
- Ning Jin, Yilin Zhu, Zhenglin Geng, and Ronald Fedkiw. 2020. A pixel-based framework for data-driven clothing. In Proceedings of the 19th ACM SIGGRAPH / Eurographics Symposium on Computer Animation, Vol. 39. Association for Computing Machinery.Google Scholar
Digital Library
- Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision. Springer, 694--711.Google Scholar
Cross Ref
- Ron Kimmel and James A Sethian. 1998. Computing geodesic paths on manifolds. Proceedings of the national academy of Sciences 95, 15 (1998), 8431--8435.Google Scholar
Cross Ref
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- Ilya Kostrikov, Zhongshi Jiang, Daniele Panozzo, Denis Zorin, and Joan Bruna. 2018. Surface networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2540--2548.Google Scholar
Cross Ref
- Zorah Lahner, Daniel Cremers, and Tony Tung. 2018. Deepwrinkles: Accurate and realistic clothing modeling. In Proceedings of the European Conference on Computer Vision (ECCV). 667--684.Google Scholar
Digital Library
- Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015. SMPL: A skinned multi-person linear model. ACM transactions on graphics (TOG) 34, 6 (2015), 248.Google Scholar
Digital Library
- Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, Vladimir G Kim, and Yaron Lipman. 2017. Convolutional neural networks on surfaces via seamless toric covers. ACM Trans. Graph. 36, 4 (2017), 71--1.Google Scholar
Digital Library
- Steve Marschner and Peter Shirley. 2015. Fundamentals of computer graphics. CRC Press.Google Scholar
- Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. 2015. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the IEEE international conference on computer vision workshops. 37--45.Google Scholar
Digital Library
- Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).Google Scholar
- Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. 2017. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5115--5124.Google Scholar
Cross Ref
- Matthias Müller and Nuttapong Chentanez. 2010. Wrinkle meshes. In Proceedings of the 2010 ACM SIGGRAPH/Eurographics symposium on computer animation. Eurographics Association, 85--92.Google Scholar
Digital Library
- Ryota Natsume, Shunsuke Saito, Zeng Huang, Weikai Chen, Chongyang Ma, Hao Li, and Shigeo Morishima. 2019. Siclope: Silhouette-based clothed people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4480--4490.Google Scholar
Cross Ref
- Alexandros Neophytou and Adrian Hilton. 2014. A layered model of human body and garment deformation. In 2014 2nd International Conference on 3D Vision, Vol. 1. IEEE, 171--178.Google Scholar
Digital Library
- Hayato Onizuka, Zehra Hayirci, Diego Thomas, Akihiro Sugimoto, Hideaki Uchiyama, and Rin-ichiro Taniguchi. 2020. TetraTSDF: 3D human reconstruction from a single image with a tetrahedral outer shell. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6011--6020.Google Scholar
Cross Ref
- Stanley Osher and Ronald Fedkiw. 2002. Level Set Methods and Dynamic Implicit Surfaces. Springer, New York.Google Scholar
- Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).Google Scholar
- Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. 2020. Tailornet: Predicting clothing in 3d as a function of human pose, shape and garment style. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7365--7375.Google Scholar
Cross Ref
- Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. 2019. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10975--10985.Google Scholar
Cross Ref
- Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael J Black. 2017. ClothCap: Seamless 4D clothing capture and retargeting. ACM Transactions on Graphics (TOG) 36, 4 (2017), 73.Google Scholar
Digital Library
- Tiberiu Popa, Quan Zhou, Derek Bradley, Vladislav Kraevoy, Hongbo Fu, Alla Sheffer, and Wolfgang Heidrich. 2009. Wrinkling captured garments using space-time data-driven deformation. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 427--435.Google Scholar
- Nadia Robertini, Edilson De Aguiar, Thomas Helten, and Christian Theobalt. 2014. Efficient multi-view performance capture of fine-scale surface detail. In 2014 2nd International Conference on 3D Vision, Vol. 1. IEEE, 5--12.Google Scholar
Digital Library
- Damien Rohmer, Tiberiu Popa, Marie-Paule Cani, Stefanie Hahmann, and Alla Sheffer. 2010. Animation wrinkling: augmenting coarse cloth simulations with realistic-looking wrinkles. In ACM Transactions on Graphics (TOG), Vol. 29. ACM, 157.Google Scholar
Cross Ref
- Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. 2019. PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization. In Proceedings of the International Conference on Computer Vision (ICCV). IEEE.Google Scholar
Cross Ref
- Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. 2020. PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 84--93.Google Scholar
Cross Ref
- Artsiom Sanakoyeu, Dmytro Kotovenko, Sabine Lang, and Bjorn Ommer. 2018. A style-aware content loss for real-time hd style transfer. In Proceedings of the European Conference on Computer Vision (ECCV). 698--714.Google Scholar
Cross Ref
- Igor Santesteban, Miguel A Otaduy, and Dan Casas. 2019. Learning-Based Animation of Clothing for Virtual Try-On. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 355--366.Google Scholar
- Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE Transactions on Neural Networks 20, 1 (2008), 61--80. Bernhard Scholkopf and Alexander J Smola. 2001. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press.Google Scholar
Digital Library
- Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), Vol. 1. IEEE, 519--528.Google Scholar
- Andrew Selle, Jonathan Su, Geoffrey Irving, and Ronald Fedkiw. 2008. Robust high-resolution cloth using parallelism, history-based collisions, and accurate friction. IEEE transactions on visualization and computer graphics 15, 2 (2008), 339--350.Google Scholar
- Qingyang Tan, Lin Gao, Yu-Kun Lai, and Shihong Xia. 2018. Variational autoencoders for deforming 3d mesh models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5841--5850.Google Scholar
Cross Ref
- Gul Varol, Duygu Ceylan, Bryan Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, and Cordelia Schmid. 2018. Bodynet: Volumetric inference of 3d human body shapes. In Proceedings of the European Conference on Computer Vision (ECCV). 20--36.Google Scholar
Cross Ref
- Daniel Vlasic, Ilya Baran, Wojciech Matusik, and Jovan Popović. 2008. Articulated mesh animation from multi-view silhouettes. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 97.Google Scholar
Digital Library
- Huamin Wang, Florian Hecht, Ravi Ramamoorthi, and James F O'Brien. 2010. Example-based wrinkle synthesis for clothing animation. In Acm Transactions on Graphics (TOG), Vol. 29. ACM, 107.Google Scholar
Digital Library
- Tuanfeng Y Wang, Duygu Ceylan, Jovan Popović, and Niloy J Mitra. 2018. Learning a shared shape space for multimodal garment design. In SIGGRAPH Asia 2018 Technical Papers. ACM, 203.Google Scholar
- Chenglei Wu, Kiran Varanasi, and Christian Theobalt. 2012. Full body performance capture under uncontrolled and varying illumination: A shading-based approach. In European Conference on Computer Vision. Springer, 757--770.Google Scholar
Digital Library
- Weipeng Xu, Avishek Chatterjee, Michael Zollhöfer, Helge Rhodin, Dushyant Mehta, Hans-Peter Seidel, and Christian Theobalt. 2018. Monoperfcap: Human performance capture from monocular video. ACM Transactions on Graphics (ToG) 37, 2 (2018), 27.Google Scholar
Digital Library
- Jinlong Yang, Jean-Sébastien Franco, Franck Hétroy-Wheeler, and Stefanie Wuhrer. 2016. Estimation of human body shape in motion with wide clothing. In European Conference on Computer Vision. Springer, 439--454.Google Scholar
Cross Ref
- Jinlong Yang, Jean-Sébastien Franco, Franck Hétroy-Wheeler, and Stefanie Wuhrer. 2018. Analyzing clothing layer deformation statistics of 3d human motions. In Proceedings of the European Conference on Computer Vision (ECCV). 237--253.Google Scholar
Digital Library
- Tao Yu, Zerong Zheng, Yuan Zhong, Jianhui Zhao, Qionghai Dai, Gerard Pons-Moll, and Yebin Liu. 2019. SimulCap: Single-View Human Performance Capture with Cloth Simulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
Cross Ref
- Chao Zhang, Sergi Pujades, Michael J Black, and Gerard Pons-Moll. 2017. Detailed, accurate, human shape estimation from clothed 3D scan sequences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4191--4200.Google Scholar
Cross Ref
Index Terms
Recovering Geometric Information with Learned Texture Perturbations
Recommendations
An art-directed wrinkle system for CG character clothing and skin
We present a kinematic system for creating art-directed clothing and skin wrinkles on CG characters used in the production of computer-animated feature films. This system employs a curve-based method for generating wrinkles on reference poses, which are ...
Art-directed costumes at pixar: design, tailoring, and simulation in production
SA '18: SIGGRAPH Asia 2018 CoursesCostumes are an important part of character design, acting, storytelling, and visual appeal in animation. However, it is challenging to achieve art-directed natural-looking motion and detail in CG animated clothing, due to technology, workflow, and ...
Exploring physical information cloth on a multitouch table
ITS '11: Proceedings of the ACM International Conference on Interactive Tabletops and SurfacesWe expand multitouch tabletop information exploration by placing 2D information on a physically-based cloth in a shallow 3D viewing environment. Instead of offering 2D information on a rigid window or screen, we place our information on a soft flexible ...






Comments