Rana Hanocka
ABSTRACT
The irrefutable success of deep learning on images and text has sparked significant interest in its applicability to 3D geometric data. Instead of covering a breadth of alternative geometric representations (e.g., implicit functions, volumetric, and point clouds), this course aims to take a deep dive into the discrete mesh representation, the most popular representation for shapes in computer graphics.
In this course, we provide different ways of covering aspects of deep learning on meshes for the virtual audience. Our course videos outline the key challenges of using deep learning on irregular mesh representation and the key ideas on how to combine machine learning with classic geometry processing to build better geometric learning algorithms. This course note complements the course videos by providing a brief history from image convolution to mesh convolutions and extended discussion on important works on this subject. Lastly, our course website (https://anintroductiontodeeplearningonmeshes.github.io/) offers a toy dataset, mesh MNIST, and some hands-on exercises to cover the actual implementation details. Our goal is to provide a permanent virtual resource that contains a combination of theoretical and practical aspects, that enables easily incorporating deep learning in geometry processing research.
Supplemental Material
Available for Download
- Naveed Akhtar and Ajmal Mian. 2018. Threat of adversarial attacks on deep learning in computer vision: A survey. Ieee Access 6 (2018), 14410--14430.Google Scholar
Digital Library
- Pravin Bhat, Stephen Ingram, and Greg Turk. 2004. Geometric Texture Synthesis by Example. In Second Eurographics Symposium on Geometry Processing, Nice, France, July 8-10, 2004 (ACM International Conference Proceeding Series), Jean-Daniel Boissonnat and Pierre Alliez (Eds.), Vol. 71. Eurographics Association, 41--44.Google ScholarDigital Library
- Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael M. Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (Eds.). 3189--3197.Google Scholar
- Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric Deep Learning: Going beyond Euclidean data. IEEE Signal Process. Mag. 34, 4 (2017), 18--42.Google ScholarCross Ref
- Yutong Feng, Yifan Feng, Haoxuan You, Xibin Zhao, and Yue Gao. 2019. MeshNet: Mesh Neural Network for 3D Shape Representation. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019. AAAI Press, 8279--8286.Google ScholarDigital Library
- Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. 2020. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods 17, 2 (2020), 184--192.Google ScholarCross Ref
- Shunwang Gong, Lei Chen, Michael M. Bronstein, and Stefanos Zafeiriou. 2019. SpiralNet++: A Fast and Highly Efficient Mesh Convolution Operator. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019. IEEE, 4141--4148.Google ScholarCross Ref
- Igor Guskov, Wim Sweldens, and Peter Schröder. 1999. Multiresolution Signal Processing for Meshes. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1999, Los Angeles, CA, USA, August 8-13, 1999, Warren N. Waggenspack (Ed.). ACM, 325--334.Google ScholarDigital Library
- Oshri Halimi, Or Litany, Emanuele Rodolà, Alexander M. Bronstein, and Ron Kimmel. 2019. Unsupervised Learning of Dense Shape Correspondence. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 4370--4379.Google Scholar
- Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2019. MeshCNN: A Network with an Edge. ACM Trans. Graph. 38, 4, Article 90 (July 2019), 12 pages.Google ScholarDigital Library
- Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. 2020. Point2Mesh: A Self-Prior for Deformable Meshes. ACM Trans. Graph. 39, 4, Article 126 (2020).Google ScholarDigital Library
- Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or. 2020. Deep geometric texture synthesis. ACM Transactions on Graphics (TOG) 39, 4 (2020), 108--1.Google ScholarDigital Library
- Alec Jacobson, Zhigang Deng, Ladislav Kavan, and John P. Lewis. 2014. Skinning: real-time shape deformation. In Special Interest Group on Computer Graphics and Interactive Techniques Conference, SIGGRAPH '14, Vancouver, Canada, August 10-14, 2014, Courses. ACM, 24:1.Google Scholar
- Misha Kazhdan, Ming Chuang, Szymon Rusinkiewicz, and Hugues Hoppe. 2020. Poisson Surface Reconstruction with Envelope Constraints. Comput. Graph. Forum 39, 5 (2020), 173--182.Google ScholarCross Ref
- Michael M. Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson surface reconstruction. In Proceedings of the Fourth Eurographics Symposium on Geometry Processing, Cagliari, Sardinia, Italy, June 26-28, 2006 (ACM International Conference Proceeding Series), Alla Sheffer and Konrad Polthier (Eds.), Vol. 256. Eurographics Association, 61--70.Google Scholar
- Alon Lahav and Ayellet Tal. 2020. MeshWalker: deep mesh understanding by random walks. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1--13.Google ScholarDigital Library
- Isaak Lim, Alexander Dielen, Marcel Campen, and Leif Kobbelt. 2018. A Simple Approach to Intrinsic Correspondence Learning on Unstructured 3D Meshes. In Computer Vision - ECCV 2018 Workshops - Munich, Germany, September 8-14, 2018, Proceedings, Part III (Lecture Notes in Computer Science), Laura Leal-Taixé and Stefan Roth (Eds.), Vol. 11131. Springer, 349--362.Google Scholar
- Or Litany, Tal Remez, Emanuele Rodolà, Alexander M. Bronstein, and Michael M. Bronstein. 2017. Deep Functional Maps: Structured Prediction for Dense Shape Correspondence. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society, 5660--5668.Google Scholar
- Hsueh-Ti Derek Liu, Vladimir G. Kim, Siddhartha Chaudhuri, Noam Aigerman, and Alec Jacobson. 2020. Neural Subdivision. 39, 4, Article 124 (July 2020), 16 pages. Google ScholarDigital Library
- Zhiyuan Liu and Jie Zhou. 2020. Introduction to graph neural networks. Synthesis Lectures on Artificial Intelligence and Machine Learning 14, 2 (2020), 1--127.Google ScholarCross Ref
- Charles Loop. 1987. Smooth subdivision surfaces based on triangles. Master's thesis, University of Utah, Department of Mathematics (1987).Google Scholar
- Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, Vladimir G. Kim, and Yaron Lipman. 2017. Convolutional neural networks on surfaces via seamless toric covers. ACM Trans. Graph. 36, 4 (2017), 71:1--71:10.Google ScholarDigital Library
- Jonathan Masci, Davide Boscaini, Michael M. Bronstein, and Pierre Vandergheynst. 2015. Geodesic Convolutional Neural Networks on Riemannian Manifolds. In 2015 IEEE International Conference on Computer Vision Workshop, ICCV Workshops 2015, Santiago, Chile, December 7-13, 2015. IEEE Computer Society, 832--840.Google ScholarDigital Library
- Jesús R Nieto and Antonio Susín. 2013. Cage based deformations: a survey. In Deformation models. Springer, 75--99.Google Scholar
- Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas J. Guibas. 2012. Functional maps: a flexible representation of maps between shapes. ACM Trans. Graph. 31, 4 (2012), 30:1--30:11.Google ScholarDigital Library
- Maks Ovsjanikov, Etienne Corman, Michael M. Bronstein, Emanuele Rodolà, Mirela Ben-Chen, Leonidas J. Guibas, Frédéric Chazal, and Alexander M. Bronstein. 2017. Computing and processing correspondences with functional maps. In Special Interest Group on Computer Graphics and Interactive Techniques Conference, SIGGRAPH 2017, Los Angeles, CA, USA, July 30 - August 3, 2017, Courses. ACM, 5:1--5:62.Google Scholar
- Adrien Poulenard and Maks Ovsjanikov. 2018. Multi-directional geodesic neural networks via equivariant convolution. ACM Trans. Graph. 37, 6 (2018), 236:1--236:14.Google ScholarDigital Library
- Marie-Julie Rakotosaona and Maks Ovsjanikov. 2020. Intrinsic Point Cloud Interpolation via Dual Latent Space Navigation. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II (Lecture Notes in Computer Science), Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.), Vol. 12347. Springer, 655--672.Google ScholarDigital Library
- Gemot Riegler, Ali Osman Ulusoy, and Andreas Geiger. 2017. OctNet: Learning Deep 3D Representations at High Resolutions. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, 6620--6629.Google Scholar
- Jean-Michel Roufosse, Abhishek Sharma, and Maks Ovsjanikov. 2019. Unsupervised Deep Learning for Structured Shape Matching. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV2 019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 1617--1627.Google Scholar
- Ariel Shamir. 2008. A survey on Mesh Segmentation Techniques. Comput. Graph. Forum 27, 6 (2008), 1539--1556.Google ScholarCross Ref
- Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. 2020. DiffusionNet: Discretization Agnostic Learning on Surfaces. arXiv preprint arXiv:2012.00888 (2020).Google Scholar
- Justin Solomon, Andy Nguyen, Adrian Butscher, Mirela Ben-Chen, and Leonidas J. Guibas. 2012. Soft Maps Between Surfaces. Comput. Graph. Forum 31, 5 (2012), 1617--1626.Google ScholarDigital Library
- Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller. 2015. Multi-view convolutional neural networks for 3d shape recognition. In Proc. ICCV.Google Scholar
- Jian Sun, Maks Ovsjanikov, and Leonidas J. Guibas. 2009. A Concise and Provably Informative Multi-Scale Signature Based on Heat Diffusion. Comput. Graph. Forum 28, 5 (2009), 1383--1392.Google ScholarCross Ref
- Johan W. H. Tangelder and Remco C. Veltkamp. 2008. A survey of content based 3D shape retrieval methods. Multim. Tools Appl. 39, 3 (2008), 441--471.Google ScholarDigital Library
- Federico Tombari, Samuele Salti, and Luigi di Stefano. 2010. Unique Signatures of Histograms for Local Surface Description. In Computer Vision - ECCV 2010, 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part III (Lecture Notes in Computer Science), Kostas Daniilidis, Petros Maragos, and Nikos Paragios (Eds.), Vol. 6313. Springer, 356--369.Google ScholarCross Ref
- Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition. 9446--9454.Google Scholar
- Amir Vaxman, Marcel Campen, Olga Diamanti, David Bommes, Klaus Hildebrandt, Mirela Ben-Chen, and Daniele Panozzo. 2016. Directional field synthesis, design, and processing. In SIGGRAPH ASIA 2016, Macao, December 5-8, 2016 - Courses, Niloy J. Mitra (Ed.). ACM, 15:1--15:30.Google ScholarDigital Library
- Yifan Wang, Noam Aigerman, Vladimir G. Kim, Siddhartha Chaudhuri, and Olga Sorkine-Hornung. 2020. Neural Cages for Detail-Preserving 3D Deformations. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. IEEE, 72--80.Google Scholar
- Ruben Wiersma, Elmar Eisemann, and Klaus Hildebrandt. 2020. Cnns on surfaces using rotation-equivariant features. ACM Transactions on Graphics (TOG) 39, 4 (2020), 92--1.Google ScholarDigital Library
- Francis Williams, Teseo Schneider, Claudio T. Silva, Denis Zorin, Joan Bruna, and Daniele Panozzo. 2019. Deep Geometric Prior for Surface Reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 10130--10139.Google Scholar
- Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A deep representation for volumetric shapes. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015. IEEE Computer Society, 1912--1920.Google Scholar
- Kun Zhou, Xin Huang, Xi Wang, Yiying Tong, Mathieu Desbrun, Baining Guo, and Heung-Yeung Shum. 2006. Mesh quilting for geometric texture synthesis. ACM Trans. Graph. 25, 3 (2006), 690--697.Google ScholarDigital Library
- Denis Zorin. 2007. Subdivision on arbitrary meshes: algorithms and theory. In Mathematics and Computation in Imaging Science and Information Processing. World Scientific, 1--46.Google Scholar
Index Terms
An introduction to deep learning on meshes
Recommendations
Biorthogonal wavelet construction for hybrid quad/triangle meshes
Ever since its introduction by Stam and Loop, the quad/triangle subdivision scheme, which is a generalization of the well-known Catmull–Clark subdivision and Loop subdivision, has attracted a great deal of interest due to its flexibility of allowing ...
5-6-7 meshes
GI '12: Proceedings of Graphics Interface 2012We introduce a new type of meshes called 5-6-7 meshes. For many mesh processing tasks, low- or high-valence vertices are undesirable. At the same time, it is not always possible to achieve complete vertex valence regularity, i.e., to only have valence-6 ...
Comments