Ruben Wiersma
Ahmad Nasikun
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Learning from 3D point-cloud data has rapidly gained momentum, motivated by the success of deep learning on images and the increased availability of 3D data. In this paper, we aim to construct anisotropic convolution layers that work directly on the surface derived from a point cloud. This is challenging because of the lack of a global coordinate system for tangential directions on surfaces. We introduce DeltaConv, a convolution layer that combines geometric operators from vector calculus to enable the construction of anisotropic filters on point clouds. Because these operators are defined on scalar- and vector-fields, we separate the network into a scalar- and a vector-stream, which are connected by the operators. The vector stream enables the network to explicitly represent, evaluate, and process directional information. Our convolutions are robust and simple to implement and match or improve on state-of-the-art approaches on several benchmarks, while also speeding up training and inference.
Supplemental Material
Available for Download
- Matan Atzmon, Haggai Maron, Yaron Lipman, Atzmon Matan, Maron Haggai, and Lipman Yaron. 2018. Point convolutional neural networks by extension operators. ACM Trans. Graph. 37, 4 (2018), 71:1--71:12.Google Scholar
Digital Library
- Yizhak Ben-Shabat, Michael Lindenbaum, and Anath Fischer. 2018. 3DmFV: Three-Dimensional Point Cloud Classification in Real-Time Using Convolutional Neural Networks. IEEE Robotics and Automation Letters 3 (2018), 3145--3152.Google ScholarCross Ref
- Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. NeurIPS (2016).Google Scholar
- Alexandre Boulch. 2020. ConvPoint: Continuous convolutions for point cloud processing. Comput. Graph. Forum 88 (2020), 24--34.Google ScholarCross Ref
- Christopher Brandt, Leonardo Scandolo, Elmar Eisemann, and Klaus Hildebrandt. 2017. Spectral Processing of Tangential Vector Fields. Computer Graphics Forum 36, 6 (2017), 338--353.Google ScholarDigital Library
- Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. 2021. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 (2021).Google Scholar
- Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric Deep Learning: Going beyond Euclidean data. IEEE Signal Process. Mag. 34 (2017), 18--42.Google ScholarCross Ref
- Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral Networks and Locally Connected Networks on Graphs. ICLR (2014).Google Scholar
- Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012.Google Scholar
- Jintai Chen, Biwen Lei, Qingyu Song, Haochao Ying, Danny Z Chen, and Jian Wu. 2020. A Hierarchical Graph Network for 3D Object Detection on Point Clouds. CVPR (2020).Google Scholar
- Christopher Choy, JunYoung Gwak, and Silvio Savarese. 2019. 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR (2019).Google Scholar
- Taco Cohen, Mario Geiger, Jonas Koehler, and Max Welling. 2018. Spherical CNNs. ICLR (2018).Google Scholar
- Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. 2019. Gauge Equivariant Convolutional Networks and the Icosahedral CNN. ICML (2019).Google Scholar
- Keenan Crane, Fernando de Goes, Mathieu Desbrun, and Peter Schroder. 2013a. Digital Geometry Processing with Discrete Exterior Calculus. SIGGRAPH Asia 2013 Courses (2013), 7:1--7:126.Google Scholar
- Keenan Crane, Clarisse Weischedel, and Max Wardetzky. 2013b. Geodesics in heat: A new approach to computing distance based on heat flow. ACM Trans. Graph. 32, 5 (2013), 152:1--152:11.Google ScholarDigital Library
- Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR (2017).Google Scholar
- Fernando de Goes, Mathieu Desbrun, and Yiying Tong. 2016. Vector Field Processing on Triangle Meshes. SIGGRAPH Asia 2016 Courses (2016), 27:1--27:49.Google Scholar
- Pim de Haan, Maurice Weiler, Taco Cohen, and Max Welling. 2021. Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs. ICLR (2021).Google Scholar
- C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L. Guibas. 2021. Vector Neurons: A General Framework for SO(3)-Equivariant Networks. In ICCV.Google Scholar
- Miguel Dominguez, Rohan Dhamdhere, Atir Petkar, Saloni Jain, Shagan Sah, and Raymond Ptucha. 2018. General-Purpose Deep Point Cloud Feature Extractor. WACV (2018).Google Scholar
- Moshe Eliasof and Eran Treister. 2020. DiffGCN: Graph Convolutional Networks via Differential Operators and Algebraic Multigrid Pooling. NeurIPS (2020).Google Scholar
- Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. 2017. Learning SO(3) Equivariant Representations with Spherical CNNs. ECCV (2017).Google Scholar
- Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. 2019. Hypergraph neural networks. AAAI (2019).Google Scholar
- Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. ICLR Workshop on Representation Learning on Graphs and Manifolds (2019).Google Scholar
- Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. 2018. SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernels. CVPR (2018).Google Scholar
- Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. 2020. SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks. NeurIPS (2020).Google Scholar
- Jan E Gerken, Jimmy Aronsson, Oscar Carlsson, Hampus Linander, Fredrik Ohlsson, Christoffer Petersson, and Daniel Persson. 2021. Geometric deep learning and equivariant neural networks. arXiv preprint arXiv:2105.13926 (2021).Google Scholar
- Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. 2021. Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline. ICML (2021).Google Scholar
- Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 2018. 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks. CVPR (2018).Google Scholar
- Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. 2020. Deep Learning for 3D Point Clouds: A Survey. IEEE TPAMI 43 (2020), 4338--4364.Google ScholarDigital Library
- Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2019. MeshCNN: A Network with an Edge. ACM Trans. Graph. 38, 4 (2019), 90:1--90:12.Google ScholarDigital Library
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. CVPR (2016).Google Scholar
- Pedro Hermosilla, Tobias Ritschel, Pere-Pau Vázquez, Àlvar Vinacua, and Timo Ropinski. 2018. Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. ACM Trans. Graph. 37, 6 (2018), 235:1--235:12.Google ScholarDigital Library
- Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Yu, and Sai-Kit Yeung. 2016. SceneNN: A Scene Meshes Dataset with aNNotations. 3DV (2016).Google Scholar
- Binh-Son Hua, Minh-Khoi Tran, and Sai-Kit Yeung. 2018. Pointwise Convolutional Neural Networks. CVPR (2018).Google Scholar
- Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML (2015).Google ScholarDigital Library
- Chiyu Max Jiang, Jingwei Huang, Karthik Kashinath, Prabhat, Philip Marcus, and Matthias Nießner. 2019. Spherical CNNs on Unstructured Grids. ICLR (2019).Google Scholar
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.Google Scholar
- Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. ICLR (2017).Google Scholar
- Ilya Kostrikov, Zhongshi Jiang, Daniele Panozzo, Denis Zorin, and Joan Bruna. 2018. Surface Networks. CVPR (2018).Google Scholar
- Alon Lahav and A. Tal. 2020. MeshWalker: deep mesh understanding by random walks. ACM Trans. Graph. 39, 6 (2020), 263:1--263:13.Google ScholarDigital Library
- Shiyi Lan, Ruichi Yu, Gang Yu, and L Davis. 2019. Modeling Local Geometric Structure of 3D Point Clouds Using Geo-CNN. CVPR (2019).Google Scholar
- Eric-Tuan Le, Iasonas Kokkinos, and Niloy J Mitra. 2020. Going Deeper With Lean Point Networks. CVPR (2020).Google Scholar
- Huan Lei, Naveed Akhtar, and Ajmal Mian. 2019. Octree guided CNN with Spherical Kernels for 3D Point Clouds. CVPR (2019).Google Scholar
- Yangyan Li, Bu Rui, Mingchao Sun, Wei Wu, Xinhan Di, Baoquan Chen, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. 2018. PointCNN: Convolution on x-transformed points. NeurIPS (2018).Google Scholar
- Zhouhui et al. Lian. 2011. SHREC '11 Track: Shape Retrieval on Non-rigid 3D Watertight Meshes. Eurographics Workshop on 3D Object Retrieval (01 2011), 79--88.Google Scholar
- Jian Liang and Hongkai Zhao. 2013. Solving Partial Differential Equations on Point Clouds. SIAM J. Sci. Comput. 35 (2013), A1461--A1486.Google ScholarCross Ref
- Liqiang Lin, Pengdi Huang, Chi-Wing Fu, Kai Xu, Hao Zhang, and Hui Huang. 2020. One Point is All You Need: Directional Attention Point for Feature Learning. arXiv preprint arXiv:2012.06257 (2020).Google Scholar
- Hsueh-Ti Derek Liu, Alec Jacobson, and Keenan Crane. 2017. A Dirac Operator for Extrinsic Shape Analysis. Comput. Graph. Forum 36, 5 (2017), 139--149.Google ScholarDigital Library
- Jinxian Liu, Bingbing Ni, Caiyuan Li, Jiancheng Yang, and Qi Tian. 2019c. Dynamic Points Agglomeration for Hierarchical Point Sets Learning. ICCV (2019).Google Scholar
- Weiping Liu, Jia Sun, Wanyi Li, Ting Hu, and Peng Wang. 2019d. Deep Learning on Point Clouds and Its Application: A Survey. Sens 19 (2019), 4188.Google ScholarCross Ref
- Yongcheng Liu, Bin Fan, Gaofeng Meng, Jiwen Lu, Shiming Xiang, and Chunhong Pan. 2019a. DensePoint: Learning densely contextual representation for efficient point cloud processing. ICCV (2019).Google Scholar
- Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. 2019b. Relation-Shape Convolutional Neural Network for Point Cloud Analysis. CVPR (2019).Google Scholar
- Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. 2020. A Closer Look at Local Aggregation Operators in Point Cloud Analysis. ECCV (2020).Google Scholar
- Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR (2017).Google Scholar
- Tao Lu, Limin Wang, and Gangshan Wu. 2021. CGA-Net: Category Guided Aggregation for Point Cloud Semantic Segmentation. CVPR (2021).Google Scholar
- Francesco Milano, Antonio Loquercio, Antoni Rosinol, Davide Scaramuzza, and Luca Carlone. 2020. Primal-Dual Mesh Convolutional Neural Networks. NeurIPS (2020).Google Scholar
- Thomas W. Mitchel, Vladimir G. Kim, and Michael Kazhdan. 2021. Field Convolutions for Surface CNNs. ICCV (2021).Google Scholar
- Andrew Nealen. 2004. An As-Short-as-possible introduction to the least squares, weighted least squares and moving least squares methods for scattered data approximation and interpolation. URL: http://www.nealen.com/projects/mls/asapmls.pdf 1 (2004).Google Scholar
- B. O'Neill. 1983. Semi-Riemannian Geometry With Applications to Relativity. Elsevier Science.Google Scholar
- Guanghua Pan, Jun Wang, Rendong Ying, and Peilin Liu. 2018. 3DTI-Net: Learn Inner Transform Invariant 3D Geometry Features using Dynamic GCN. arXiv preprint arXiv:1812.06254 (2018).Google Scholar
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. NeurIPS (2019).Google Scholar
- P Perona and J Malik. 1990. Scale-space and edge detection using anisotropic diffusion. IEEE TPAMI 12, 7 (1990), 629--639.Google ScholarDigital Library
- Adrien Poulenard and Maks Ovsjanikov. 2018. Multi-directional geodesic neural networks via equivariant convolution. ACM Trans. Graph. 37, 6 (2018), 236:1--236:14.Google ScholarDigital Library
- Adrien Poulenard, Marie-Julie Rakotosaona, Yann Ponty, and Maks Ovsjanikov. 2019. Effective Rotation-invariant Point CNN with Spherical Harmonics kernels. 3DV (2019).Google Scholar
- Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017a. PointNet: Deep learning on point sets for 3D classification and segmentation. CVPR (2017).Google Scholar
- Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017b. PointNet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS (2017).Google Scholar
- Shi Qiu, Saeed Anwar, and Nick Barnes. 2021a. Dense-Resolution Network for Point Cloud Classification and Segmentation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (2021).Google ScholarCross Ref
- Shi Qiu, Saeed Anwar, and Nick Barnes. 2021b. Geometric Back-projection Network for Point Cloud Classification. arXiv preprint arXiv:1911.12885 (2021).Google Scholar
- Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. 2021. Diffusion-Net: Discretization Agnostic Learning on Surfaces. ACM Trans. Graph. 41, 3 (2021), 27:1--27:16.Google Scholar
- Yiru Shen, Chen Feng, Yaoqing Yang, and Dong Tian. 2018. Mining point cloud local structures by kernel correlation and graph pooling. CVPR (2018).Google Scholar
- Martin Simonovsky and Nikos Komodakis. 2017. Dynamic edge-conditioned filters in convolutional neural networks on graphs. CVPR (2017).Google Scholar
- Dmitriy Smirnov and Justin Solomon. 2021. HodgeNet: Learning Spectral Geometry on Triangle Meshes. ACM Trans. Graph. 40, 4 (2021), 166:1--166:11.Google ScholarDigital Library
- Xiao Sun, Zhouhui Lian, and Jianguo Xiao. 2019. SRINet: Learning Strictly Rotation-Invariant Representations for Point Cloud Classification and Segmentation. ACM International Conference on Multimedia (2019), 980--988.Google ScholarDigital Library
- Gusi Te, Wei Hu, Amin Zheng, and Zongming Guo. 2018. RGCNN: Regularized graph CNN for point cloud segmentation. ACM International Conference on Multimedia (2018), 746--754.Google ScholarDigital Library
- Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. 2019. KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV (2019).Google Scholar
- Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. 2018. Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds. arXiv preprint arXiv:1802.08219 (2018).Google Scholar
- Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung. 2019. Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data. ICCV (2019).Google Scholar
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. NeurIPS (2017).Google Scholar
- Chu Wang, Babak Samari, and Kaleem Siddiqi. 2018. Local spectral graph convolution for point set feature learning. ECCV (2018).Google Scholar
- Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. 2019. Dynamic graph CNN for learning on point clouds. ACM Trans. Graph. 38, 5 (2019), 146:1--146:12.Google ScholarDigital Library
- Max Wardetzky. 2006. Discrete Differential Operators on Polyhedral Surfaces - Convergence and Approximation. Ph.D. Dissertation. Freie Universiät Berlin.Google Scholar
- Joachim Weickert. 1998. Anisotropic diffusion in image processing. Vol. 1.Google Scholar
- Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. 2021. Coordinate Independent Convolutional Networks - Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds. arXiv preprint arXiv:2106.06020 (2021).Google Scholar
- Ruben Wiersma, Elmar Eisemann, and Klaus Hildebrandt. 2020. CNNs on surfaces using rotation-equivariant features. ACM Trans. Graph. 4 (2020), 92:1--92:12.Google Scholar
- Wenxuan Wu, Zhongang Qi, and Li Fuxin. 2019. PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR (2019).Google Scholar
- Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A deep representation for volumetric shapes. CVPR (2015).Google Scholar
- Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. 2021. Walk in the Cloud: Learning Curves for Point Clouds Shape Analysis. ICCV (2021).Google Scholar
- Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. 2021a. PAConv: Position Adaptive Convolution With Dynamic Kernel Assembling on Point Clouds. CVPR (2021).Google Scholar
- Mutian Xu, Junhao Zhang, Zhipeng Zhou, Mingye Xu, Xiaojuan Qi, and Yu Qiao. 2021b. Learning Geometry-Disentangled Representation for Complementary Understanding of 3D Object Point Cloud. AAAI (2021).Google Scholar
- Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. 2018. SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters. ECCV (2018).Google Scholar
- Jiancheng Yang, Qiang Zhang, B Ni, L Li, J Liu, Mengdie Zhou, and Q Tian. 2019. Modeling Point Clouds With Self-Attention and Gumbel Subset Sampling. CVPR (2019).Google Scholar
- Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. 2016. A Scalable Active Framework for Region Annotation in 3D Shape Collections. ACM Trans. Graph. 35, 6 (2016), 210:1--210:12.Google ScholarDigital Library
- Cheng Zhang, Haocheng Wan, Shengqiang Liu, Xinyi Shen, and Zizhao Wu. 2021. PVT: Point-Voxel Transformer for 3D Deep Learning. arXiv preprint arXiv:2108.06076 (2021).Google Scholar
- Kuangen Zhang, Ming Hao, Jing Wang, Clarence W de Silva, and Chenglong Fu. 2019. Linked Dynamic Graph CNN: Learning on Point Cloud via Linking Hierarchical Features. arXiv preprint arXiv:1904.10014 (2019).Google Scholar
- Yingxue Zhang and Michael Rabbat. 2018. A Graph-CNN for 3D point cloud classification. ICASSP (2018), 6279--6283.Google Scholar
- Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia. 2019. PointWeb: Enhancing Local Neighborhood Features for Point Cloud Processing. CVPR (2019).Google Scholar
- Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip H.S. Torr, and Vladlen Koltun. 2021. Point Transformer. ICCV (2021).Google Scholar
Index Terms
DeltaConv: anisotropic operators for geometric deep learning on point clouds
Recommendations
Self-Sampling for Neural Point Cloud Consolidation
We introduce a novel technique for neural point cloud consolidation which learns from only the input point cloud. Unlike other point up-sampling methods which analyze shapes via local patches, in this work, we learn from global subsets. We repeatedly self-...
Orienting point clouds with dipole propagation
Establishing a consistent normal orientation for point clouds is a notoriously difficult problem in geometry processing, requiring attention to both local and global shape characteristics. The normal direction of a point is a function of the local ...
CurvaNet: Geometric Deep Learning based on Directional Curvature for 3D Shape Analysis
KDD '20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data MiningOver the last decade, deep learning research has achieved tremendous success in computer vision and natural language processing. The current widely successful deep learning models are largely based on convolution and pooling operations on a Euclidean ...
Comments