Heterogeneous distributed big data clustering on sparse grids
Summary: Clustering is an important task in data mining that has become more challenging due to the ever-increasing size of available datasets. To cope with these big data scenarios, a high-performance clustering approach is required. Sparse grid clustering is a density-based clustering method that uses a sparse grid density estimation as its central building block. The underlying density estimation approach enables the detection of clusters with non-convex shapes and without a predetermined number of clusters. In this work, we introduce a new distributed and performance-portable variant of the sparse grid clustering algorithm that is suited for big data settings. Our computed kernels were implemented in OpenCL to enable portability across a wide range of architectures. For distributed environments, we added a manager-worker scheme that was implemented using MPI. In experiments on two supercomputers, Piz Daint and Hazel Hen, with up to 100 million data points in a ten-dimensional dataset, we show the performance and scalability of our approach. The dataset with 100 million data points was clustered in 1198s using 128 nodes of Piz Daint. This translates to an overall performance of 352 TFLOPS. On the node-level, we provide results for two GPUs, Nvidia's Tesla P100 and the AMD FirePro W8100, and one processor-based platform that uses Intel Xeon E5-2680v3 processors. In these experiments, we achieved between 43\% and 66\% of the peak performance across all computed kernels and devices, demonstrating the performance portability of our approach.
- Elephant against Goliath: performance of big data versus high-performance computing DBSCAN clustering implementations
- A distributed framework for trimmed kernel \(k\)-means clustering
- scientific article; zbMATH DE number 5540352
- DGDCT: A Distributed Grid-Density Based Algorithm for Intrinsic Cluster Detection over Massive Spatial Data
- The fast clustering algorithm for the big data based on K-means
- A local search approximation algorithm for \(k\)-means clustering
- A new subspace-based algorithm for efficient spatially adaptive sparse grid regression, classification and multi-evaluation
- Finite element thin plate splines in density estimation
- From data to uncertainty: an efficient integrated data-driven sparse grid approach to propagate uncertainty
- scientific article; zbMATH DE number 6381735 (Why is no real title available?)
- scientific article; zbMATH DE number 3168214 (Why is no real title available?)
- Locality-sensitive hashing scheme based on p-stable distributions
- Spatially adaptive sparse grids for high-dimensional data-driven problems
- A distributed framework for trimmed kernel \(k\)-means clustering
- Fast sparse grid operations using the unidirectional principle: a generalized and unified framework
- UP-DPC: ultra-scalable parallel density peak clustering
- Deep FC-IIWO fuzzy clustering: an optimized fuzzy clustering approach for big data clustering
This page was built for publication: Heterogeneous distributed big data clustering on sparse grids
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2312425)
