HodgeNet: learning spectral geometry on triangle meshes

Abstract

Constrained by the limitations of learning toolkits engineered for other applications, such as those in image processing, many mesh-based learning algorithms employ data flows that would be atypical from the perspective of conventional geometry processing. As an alternative, we present a technique for learning from meshes built from standard geometry processing modules and operations. We show that low-order eigenvalue/eigenvector computation from operators parameterized using discrete exterior calculus is amenable to efficient approximate backpropagation, yielding spectral per-element or per-mesh features with similar formulas to classical descriptors like the heat/wave kernel signatures. Our model uses few parameters, generalizes to high-resolution meshes, and exhibits performance and time complexity on par with past work.

References

1. James Atwood and Don Towsley. 2016. Diffusion-convolutional neural networks. In Proc. NeurIPS. 1993--2001.
2. Oscar Kin-Chung Au, Youyi Zheng, Menglin Chen, Pengfei Xu, and Chiew-Lan Tai. 2011. Mesh segmentation with concavity-aware fields. IEEE Transactions on Visualization and Computer Graphics 18, 7 (2011), 1125--1134.
3. Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers. 2011. The wave kernel signature: A quantum mechanical approach to shape analysis. In Proc. ICCV workshops. 1626--1633.
4. Mirela Ben-Chen, Adrian Butscher, Justin Solomon, and Leonidas Guibas. 2010. On discrete Killing vector fields and patterns on surfaces. In Computer Graphics Forum, Vol. 29. 1701--1711.
5. Davide Boscaini, Davide Eynard, Drosos Kourounis, and Michael Bronstein. 2015a. Shape-from-operator: Recovering shapes from intrinsic operators. In Computer Graphics Forum, Vol. 34. 265--274.
6. Davide Boscaini, Jonathan Masci, Simone Melzi, Michael Bronstein, Umberto Castellani, and Pierre Vandergheynst. 2015b. Learning class-specific descriptors for deformable shapes using localized spectral convolutional networks. In Computer Graphics Forum, Vol. 34. 13--23.
7. Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. In Proc. NeurIPS.
8. Alexander Bronstein, Michael Bronstein, Leonidas Guibas, and Maks Ovsjanikov. 2011. Shape Google: Geometric words and expressions for invariant shape retrieval. ACM Transactions on Graphics 30, 1 (2011), 1--20.
9. Michael Bronstein and Alexander Bronstein. 2010. Shape recognition with spectral distances. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 5 (2010), 1065--1071.
10. Michael Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric deep learning: going beyond Euclidean data. IEEE Signal Processing Magazine 34, 4 (2017), 18--42.
11. Michael Bronstein and Iasonas Kokkinos. 2010. Scale-invariant heat kernel signatures for non-rigid shape recognition. In Proc. CVPR. 1704--1711.
12. Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2013. Spectral networks and locally connected networks on graphs. arXiv:1312.6203 (2013).
13. Max Budninskiy, Ameera Abdelaziz, Yiying Tong, and Mathieu Desbrun. 2020. Laplacian-optimized diffusion for semi-supervised learning. Computer Aided Geometric Design 79 (2020), 101864.
14. Ken Chan and Justin Wan. 2013. Reconstruction of missing cells by a Killing energy minimizing nonrigid image registration. In International Conference of the IEEE Engineering in Medicine and Biology Society. 3000--3003.
15. Yoni Choukroun, Gautam Pai, and Ron Kimmel. 2018. Sparse approximation of 3D meshes using the spectral geometry of the Hamiltonian operator. Journal of Mathematical Imaging and Vision 60, 6 (2018), 941--952.
16. Sebastian Claici, Mikhail Bessmeltsev, Scott Schaefer, and Justin Solomon. 2017. Isometry-aware preconditioning for mesh parameterization. In Computer Graphics Forum, Vol. 36. 37--47.
17. Etienne Corman, Justin Solomon, Mirela Ben-Chen, Leonidas Guibas, and Maks Ovsjanikov. 2017. Functional characterization of intrinsic and extrinsic geometry. ACM Transactions on Graphics 36, 2 (2017), 1--17.
18. Luca Cosmo, Mikhail Panine, Arianna Rampini, Maks Ovsjanikov, Michael Bronstein, and Emanuele Rodolà. 2019. Isospectralization, or how to hear shape, style, and correspondence. In Proc. CVPR. 7529--7538.
19. Fernando de Goes, Mathieu Desbrun, and Yiying Tong. 2016. Vector field processing on triangle meshes. In ACM SIGGRAPH 2016 Courses. 1--49.
20. Pim de Haan, Maurice Weiler, Taco Cohen, and Max Welling. 2020. Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs. arXiv:2003.05425 (2020).
21. Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. Proc. NeurIPS 29 (2016), 3844--3852.
22. Mathieu Desbrun, Anil Hirani, Melvin Leok, and Jerrold E Marsden. 2005. Discrete exterior calculus. arXiv preprint math/0508341 (2005).
23. Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov. 2020. Deep geometric functional maps: Robust feature learning for shape correspondence. In Proc. CVPR. 8592--8601.
24. Moshe Eliasof and Eran Treister. 2020. DiffGCN: Graph convolutional networks via differential operators and algebraic multigrid pooling. Proc. NeurIPS (2020).
25. Yutong Feng, Yifan Feng, Haoxuan You, Xibin Zhao, and Yue Gao. 2019. MeshNet: Mesh neural network for 3D shape representation. In Proc. AAAI, Vol. 33. 8279--8286.
26. Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. 2018. SplineCNN: Fast geometric deep learning with continuous B-spline kernels. In Proc. CVPR. 869--877.
27. Jun Gao, Wenzheng Chen, Tommy Xiang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. 2020. Learning deformable tetrahedral meshes for 3D reconstruction. Proc. NeurIPS 33 (2020).
28. Xianfeng Gu, Steven Gortler, and Hugues Hoppe. 2002. Geometry images. In Proc. SIGGRAPH. 355--361.
29. Niv Haim, Nimrod Segol, Heli Ben-Hamu, Haggai Maron, and Yaron Lipman. 2019. Surface networks via general covers. In Proc. ICCV. 632--641.
30. Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2019. MeshCNN: A network with an edge. ACM Transactions on Graphics 38, 4 (2019), 1--12.
31. Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. 2020. Point2Mesh: A self-prior for deformable meshes. ACM Transactions on Graphics 39, 4 (July 2020).
32. Wenchong He, Zhe Jiang, Chengming Zhang, and Arpan Man Sainju. 2020. CurvaNet: Geometric deep learning based on directional curvature for 3D shape analysis. In Proc. KDD. 2214--2224.
33. Mikael Henaff, Joan Bruna, and Yann LeCun. 2015. Deep convolutional networks on graph-structured data. arXiv:1506.05163 (2015).
34. Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or. 2020. Deep geometric texture synthesis. ACM Transactions on Graphics 39, 4 (July 2020).
35. Klaus Hildebrandt, Christian Schulz, Christoph von Tycowicz, and Konrad Polthier. 2012. Modal shape analysis beyond Laplacian. (2012).
36. Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkhouser, Matthias Nießner, and Leonidas Guibas. 2019. TextureNet: Consistent local parametrizations for learning from high-resolution signals on meshes. In Proc. CVPR. 4440--4449.
37. Qi-Xing Huang, Martin Wicke, Bart Adams, and Leonidas Guibas. 2009. Shape decomposition using modal analysis. In Computer Graphics Forum, Vol. 28. 407--416.
38. Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 2017. 3D shape segmentation with projective convolutional networks. In Proc. CVPR. 3779--3788.
39. Thomas Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proc. ICLR.
40. Ilya Kostrikov, Zhongshi Jiang, Daniele Panozzo, Denis Zorin, and Joan Bruna. 2018. Surface networks. In Proc. CVPR. 2540--2548.
41. Artiom Kovnatsky, Michael Bronstein, Alexander Bronstein, and Ron Kimmel. 2011. Photometric heat kernel signatures. In International Conference on Scale Space and Variational Methods in Computer Vision. 616--627.
42. Alon Lahav and Ayellet Tal. 2020. MeshWalker: Deep mesh understanding by random walks. arXiv:2006.05353 (2020).
43. Ron Levie, Federico Monti, Xavier Bresson, and Michael Bronstein. 2018. CayleyNets: Graph convolutional neural networks with complex rational spectral filters. IEEE Transactions on Signal Processing 67, 1 (2018), 97--109.
44. Z. Lian, A. Godil, B. Bustos, M. Daoudi, J. Hermans, S. Kawamura, Y. Kurita, G. Lavoué, H. V. Nguyen, R. Ohbuchi, Y. Ohkita, Y. Ohishi, F. Porikli, M. Reuter, I. Sipiran, D. Smeets, P. Suetens, H. Tabia, and D. Vandermeulen. 2011. SHREC '11 Track: Shape Retrieval on Non-rigid 3D Watertight Meshes. In Eurographics Workshop on 3D Object Retrieval, H. Laga, T. Schreck, A. Ferreira, A. Godil, I. Pratikakis, and R. Veltkamp (Eds.). The Eurographics Association.
45. Isaak Lim, Alexander Dielen, Marcel Campen, and Leif Kobbelt. 2018. A simple approach to intrinsic correspondence learning on unstructured 3D meshes. In Proc. ECCV.
46. Or Litany, Tal Remez, Emanuele Rodolà, Alex Bronstein, and Michael Bronstein. 2017. Deep functional maps: Structured prediction for dense shape correspondence. In Proc. ICCV. 5659--5667.
47. Roee Litman and Alexander Bronstein. 2013. Learning spectral descriptors for deformable shape correspondence. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 1 (2013), 171--180.
48. Hsueh-Ti Derek Liu, Alec Jacobson, and Keenan Crane. 2017. A Dirac operator for extrinsic shape analysis. In Computer Graphics Forum, Vol. 36. 139--149.
49. Hsueh-Ti Derek Liu, Vladimir Kim, Siddhartha Chaudhuri, Noam Aigerman, and Alec Jacobson. 2020. Neural subdivision. ACM Transactions on Graphics 39, 4 (July 2020).
50. Rong Liu and Hao Zhang. 2007. Mesh segmentation via spectral embedding and contour analysis. In Computer Graphics Forum, Vol. 26. 385--394.
51. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations. https://openreview.net/forum?id=Bkg6RiCqY7
52. A.L. Maas, A.Y. Hannun, and A.Y. Ng. 2013. Rectifier Nonlinearities Improve Neural Network Acoustic Models. In Proceedings of the International Conference on Machine Learning. Atlanta, Georgia.
53. Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, Vladimir Kim, and Yaron Lipman. 2017. Convolutional neural networks on surfaces via seamless toric covers. ACM Transactions on Graphics 36, 4 (2017).
54. Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. 2015. Geodesic convolutional neural networks on Riemannian manifolds. In Proc. ICCV. 37--45.
55. Francesco Milano, Antonio Loquercio, Antoni Rosinol, Davide Scaramuzza, and Luca Carlone. 2020. Primal-Dual Mesh Convolutional Neural Networks. In Proc. NeurIPS.
56. Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael Bronstein. 2017. Geometric deep learning on graphs and manifolds using mixture model CNNs. In Proc. CVPR.
57. Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. 2012. Functional maps: A flexible representation of maps between shapes. ACM Transactions on Graphics 31, 4 (2012), 1--11.
58. Maks Ovsjanikov, Quentin Mérigot, Facundo Mémoli, and Leonidas Guibas. 2010. One point isometric matching with the heat kernel. In Computer Graphics Forum, Vol. 29. 1555--1564.
59. Maks Ovsjanikov, Jian Sun, and Leonidas Guibas. 2008. Global intrinsic symmetries of shapes. In Computer Graphics Forum, Vol. 27. 1341--1348.
60. Hao Pan, Shilin Liu, Yang Liu, and Xin Tong. 2018. Convolutional neural networks on 3D surfaces using parallel frames. arXiv:1808.04952 (2018).
61. Adrien Poulenard and Maks Ovsjanikov. 2018. Multi-directional geodesic neural networks via equivariant convolution. ACM Transactions on Graphics 37, 6 (2018), 1--14.
62. Yi-Ling Qiao, Lin Gao, Paul Rosin, Yu-Kun Lai, Xilin Chen, et al. 2020. Learning on 3D meshes with Laplacian encoding and pooling. IEEE Transactions on Visualization and Computer Graphics (2020).
63. Dan Raviv, Michael Bronstein, Alexander Bronstein, and Ron Kimmel. 2010. Volumetric heat kernel signatures. In Proceedings of the ACM Workshop on 3D Object Retrieval. 39--44.
64. Dan Raviv, Michael Bronstein, Alexander Bronstein, Ron Kimmel, and Nir Sochen. 2011. Affine-invariant diffusion geometry for the analysis of deformable 3D shapes. In CVPR 2011. 2361--2367.
65. Martin Reuter, Franz-Erich Wolter, and Niklas Peinecke. 2006. Laplace-Beltrami spectra as 'shape-DNA' of surfaces and solids. Computer-Aided Design 38, 4 (2006), 342--366.
66. Raif Rustamov. 2007. Laplace-Beltrami eigenfunctions for deformation invariant shape representation. In Symposium on Geometry Processing. 225--233.
67. Lisa Schneider, Annika Niemann, Oliver Beuing, Bernhard Preim, and Sylvia Saalfeld. 2020. MedMeshCNN-Enabling MeshCNN for medical surface models. arXiv:2009.04893 (2020).
68. Stefan Schonsheck, Bin Dong, and Rongjie Lai. 2018. Parallel transport convolution: A new tool for convolutional neural networks on manifolds. arXiv:1805.07857 (2018).
69. Jonas Schult, Francis Engelmann, Theodora Kontogianni, and Bastian Leibe. 2020. DualConvMesh-Net: Joint geodesic and Euclidean convolutions on 3D meshes. In Proc. CVPR. 8612--8622.
70. Konstantinos Sfikas, Theoharis Theoharis, and Ioannis Pratikakis. 2017. Exploiting the PANORAMA representation for convolutional neural network classification and retrieval. Proc. 3DOR 6 (2017), 7.
71. Abhishek Sharma and Maks Ovsjanikov. 2020. Weakly supervised deep functional map for shape matching. In Proc. NeurIPS.
72. Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. 2020. Diffusion is all you need for learning on surfaces. arXiv:2012.00888 (2020).
73. Nicholas Sharp and Maks Ovsjanikov. 2020. PointTriNet: Learned triangulation of 3D point sets. In Proc. ECCV.
74. Nicholas Sharp, Yousuf Soliman, and Keenan Crane. 2019. The vector heat method. ACM Transactions on Graphics 38, 3 (2019), 1--19.
75. Baoguang Shi, Song Bai, Zhichao Zhou, and Xiang Bai. 2015. DeepPano: Deep panoramic representation for 3-d shape recognition. IEEE Signal Processing Letters 22, 12 (2015), 2339--2343.
76. Amit Singer and H.-T. Wu. 2012. Vector diffusion maps and the connection Laplacian. Communications on Pure and Applied Mathematics 65, 8 (2012), 1067--1144.
77. Ayan Sinha, Jing Bai, and Karthik Ramani. 2016. Deep learning 3D shape surfaces using geometry images. In Proc. ECCV. 223--240.
78. Miroslava Slavcheva, Maximilian Baust, Daniel Cremers, and Slobodan Ilic. 2017. Killing-Fusion: Non-rigid 3D reconstruction without correspondences. In Proc. CVPR. 1386--1395.
79. Justin Solomon, Mirela Ben-Chen, Adrian Butscher, and Leonidas Guibas. 2011a. AsKilling-as-possible vector fields for planar deformation. In Computer Graphics Forum, Vol. 30. 1543--1552.
80. Justin Solomon, Mirela Ben-Chen, Adrian Butscher, and Leonidas Guibas. 2011b. Discovery of intrinsic primitives on triangle meshes. In Computer Graphics Forum, Vol. 30. 365--374.
81. An Ping Song, Xin Yi Di, Xiao Kang Xu, and Zi Heng Song. 2020. MeshGraphNet: An effective 3D polygon mesh recognition With topology reconstruction. IEEE Access 8 (2020), 205181--205189.
82. M. Spagnuolo, M. Bronstein, A. Bronstein, and A. Ferreira. 2012. Affine-invariant photometric heat kernel signatures. Eurographics (2012), 39--46.
83. Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multi-view convolutional neural networks for 3D shape recognition. In Proc. ICCV. 945--953.
84. Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. 2009. A concise and provably informative multi-scale signature based on heat diffusion. In Computer Graphics Forum, Vol. 28. 1383--1392.
85. Zhiyu Sun, Ethan Rooke, Jerome Charton, Yusen He, Jia Lu, and Stephen Baek. 2020. ZerNet: Convolutional neural networks on arbitrary surfaces via Zernike local tangent space estimation. In Computer Graphics Forum.
86. Michael Tao, Justin Solomon, and Adrian Butscher. 2016. Near-isometric level set tracking. In Computer Graphics Forum, Vol. 35. 65--77.
87. Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian-Yi Zhou. 2018. Tangent convolutions for dense prediction in 3D. In Proc. CVPR. 3887--3896.
88. Nitika Verma, Edmond Boyer, and Jakob Verbeek. 2018. FeastNet: Feature-steered graph convolutions for 3D shape analysis. In Proc. CVPR. 2598--2606.
89. Daniel Vlasic, Ilya Baran, Wojciech Matusik, and Jovan Popović. 2008. Articulated mesh animation from multi-view silhouettes. In ACM SIGGRAPH 2008 papers. 1--9.
90. Hao Wang, Tong Lu, Oscar Kin-Chung Au, and Chiew-Lan Tai. 2014. Spectral 3D mesh segmentation with a novel single segmentation field. Graphical Models 76, 5 (2014), 440--456.
91. Pengyu Wang, Yuan Gan, Panpan Shui, Fenggen Yu, Yan Zhang, Songle Chen, and Zhengxing Sun. 2018b. 3D shape segmentation via shape fully convolutional networks. Computers & Graphics 70 (2018), 128--139.
92. Yunhai Wang, Shmulik Asafi, Oliver Van Kaick, Hao Zhang, Daniel Cohen-Or, and Baoquan Chen. 2012. Active co-analysis of a set of shapes. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1--10.
93. Yu Wang, Mirela Ben-Chen, Iosif Polterovich, and Justin Solomon. 2018a. Steklov spectral geometry for extrinsic shape analysis. ACM Transactions on Graphics 38, 1 (2018), 1--21.
94. Yu Wang, Vladimir Kim, Michael Bronstein, and Justin Solomon. 2019. Learning geometric operators on meshes. In Representation Learning on Graphs and Manifolds (ICLR workshop).
95. Yu Wang and Justin Solomon. 2019. Intrinsic and extrinsic operators for shape analysis. In Handbook of Numerical Analysis. Vol. 20. 41--115.
96. Lingyu Wei, Qixing Huang, Duygu Ceylan, Etienne Vouga, and Hao Li. 2016. Dense human body correspondences using convolutional networks. In Proc. CVPR. 1544--1553.
97. Ruben Wiersma, Elmar Eisemann, and Klaus Hildebrandt. 2020. CNNs on surfaces using rotation-equivariant features. ACM Transactions on Graphics 39, 4 (2020).
98. Haotian Xu, Ming Dong, and Zichun Zhong. 2017. Directionally convolutional networks for 3D shape segmentation. In Proc. ICCV. 2698--2707.
99. Yuqi Yang, Shilin Liu, Hao Pan, Yang Liu, and Xin Tong. 2020. PFCNN: Convolutional neural networks on 3D surfaces using parallel frames. In Proc. CVPR. 13578--13587.
100. Zi Ye, Olga Diamanti, Chengcheng Tang, Leonidas Guibas, and Tim Hoffmann. 2018. A unified discrete framework for intrinsic and extrinsic Dirac operators for geometry processing. In Computer Graphics Forum, Vol. 37. 93--106.
101. Li Yi, Hao Su, Xingwen Guo, and Leonidas Guibas. 2017. SpecSyncCNN: Synchronized spectral CNN for 3D shape segmentation. In Proc. CVPR. 2282--2290.
102. Wei Zeng, Ren Guo, Feng Luo, and Xianfeng Gu. 2012. Discrete heat kernel determines discrete Riemannian metric. Graphical Models 74, 4 (2012), 121--129.

Index Terms

• Computing methodologies

  • Computer graphics

    • Shape modeling

      • Mesh geometry models

      • Shape analysis

  • Machine learning

    • Machine learning approaches

      • Neural networks