Simultaneous estimation of cluster number and feature sparsity in high‐dimensional cluster analysis

DOI10.1111/BIOM.13449zbMath1520.62273arXiv1909.01930OpenAlexW3132798262MaRDI QIDQ6079473

Yu-Jia Li, George C. Tseng, Unnamed Author, Chien-Wei Lin

Publication date: 30 October 2023

Published in: Biometrics (Search for Journal in Brave)

Abstract: Estimating the number of clusters (K) is a critical and often difficult task in cluster analysis. Many methods have been proposed to estimate K, including some top performers using resampling approach. When performing cluster analysis in high-dimensional data, simultaneous clustering and feature selection is needed for improved interpretation and performance. To our knowledge, none has investigated simultaneous estimation of K and feature selection in an exploratory cluster analysis. In this paper, we propose a resampling method to meet this gap and evaluate its performance under the sparse K-means clustering framework. The proposed target function balances between sensitivity and specificity of clustering evaluation of pairwise subjects from clustering of full and subsampled data. Through extensive simulations, the method performs among the best over classical methods in estimating K in low-dimensional data. For high-dimensional simulation data, it also shows superior performance to simultaneously estimate K and feature sparsity parameter. Finally, we evaluated the methods in four microarray, two RNA-seq, one SNP and two non-omics datasets. The proposed method achieves better clustering accuracy with fewer selected predictive genes in almost all real applications.

Full work available at URL: https://arxiv.org/abs/1909.01930

zbMATH Keywords

feature selection\(K\)-meansestimate number of clusterssparse \(K\)-means

Mathematics Subject Classification ID

Applications of statistics to biology and medical sciences; meta analysis (62P10)

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Estimating the Number of Clusters in a Data Set Via the Gap Statistic
• Selection of the number of clusters via the bootstrap method
• Classification and clustering of sequencing data using a Poisson model
• Silhouettes: a graphical aid to the interpretation and validation of cluster analysis
• Resampling Method for Unsupervised Estimation of Cluster Validity
• A Criterion for Determining the Number of Groups in a Data Set Using Sum-of-Squares Clustering
• Model-Based Gaussian and Non-Gaussian Clustering
• Finding the Number of Clusters in a Dataset
• Stability-Based Validation of Clustering Solutions
• A Framework for Feature Selection in Clustering
• Tight Clustering: A Resampling‐Based Approach for Identifying Stable and Tight Patterns in Data

Related Items (2)

Simultaneous Cluster Structure Learning and Estimation of Heterogeneous Graphs for Matrix-Variate fMRI Data ⋮ Outcome-guided disease subtyping by generative model and weighted joint likelihood in transcriptomic applications

This page was built for publication: Simultaneous estimation of cluster number and feature sparsity in high‐dimensional cluster analysis