Fast Parallel Algorithms for Euclidean Minimum Spanning Tree and Hierarchical Spatial Clustering

This paper presents new parallel algorithms for generating Euclidean minimum spanning trees and spatial clustering hierarchies (known as HDBSCAN$^*$ in the literature). Our approach is based on generating a well-separated pair decomposition (WSPD) followed by using Kruskal's minimum spanning tree algorithm and bichromatic closest pair computations. We introduce a new notion of well-separation to reduce the work and space of our WSPD-based algorithm forHDBSCAN$^*$. We also present a parallel approximate algorithm for HDBSCAN$^*$ (and OPTICS) based on a recent sequential algorithm by Gan and Tao. Finally, we give a new parallel divide-and-conquer algorithm for computing the dendrogramand reachability plots, which are used in visualizing clusters of different scale that arise for both EMST and HDBSCAN$^*$.We present highly-optimized implementations of our algorithms that achieve state-of-the-art performance. For the WSPD-based algorithms, we propose a memory optimization that requires only a subset of all possible well-separated pairs to be computed and materialized, leading to savings in both space (up to 10x) and time (up to 8x). Our experiments on large real-world and synthetic data sets using a 48-core machine show that our fastest algorithms outperform the best serial algorithms for the problems by 11.13--55.89x, and existing parallel algorithms by at least an order of magnitude.Not only are our algorithms efficient in practice, but we also prove strong theoretical bounds on their work (number of operations) and depth (parallel time). We show that all of our algorithms have polylogarithmic depth, making them highly parallel, and have work bounds that match their sequential counterparts.